Update README.md

README.md

@@ -1,3 +1,112 @@
 ---
-license: mit
+language: 
+  - pt
+thumbnail: "Portugues BERT for the Legal Domain"
+tags:
+  - bert
+  - pytorch
+license: "mit"
+widget:
+ - text: "O advogado apresentou [MASK] ao juíz."
 ---
+
+# Legal_BERTimbau
+
+## Introduction
+
+Legal_BERTimbau Large is a fine-tuned BERT model based on [BERTimbau](https://huggingface.co/neuralmind/bert-base-portuguese-cased) Large.
+
+"BERTimbau Base is a pretrained BERT model for Brazilian Portuguese that achieves state-of-the-art performances on three downstream NLP tasks: Named Entity Recognition, Sentence Textual Similarity and Recognizing Textual Entailment. It is available in two sizes: Base and Large.
+
+For further information or requests, please go to [BERTimbau repository](https://github.com/neuralmind-ai/portuguese-bert/)."
+
+The performance of Language Models can change drastically when there is a domain shift between training and test data. In order create a Portuguese Language Model adapted to a Legal domain, the original BERTimbau model was submitted to a fine-tuning stage where it was performed 1 "PreTraining" epoch over 30 000 legal Portuguese Legal documents available online.
+
+
+## Available models
+
+| Model                                    | Arch.      | #Layers | #Params |
+| ---------------------------------------- | ---------- | ------- | ------- |
+| `rufimelo/Legal-BERTimbau-base`  | BERT-Base	|12	|110M|
+| `rufimelo/Legal-BERTimbau-large` | BERT-Large | 24      | 335M    |
+
+## Usage
+
+```python
+from transformers import AutoTokenizer, AutoModelForMaskedLM
+
+tokenizer = AutoTokenizer.from_pretrained("rufimelo/Legal-BERTimbau-base")
+
+model = AutoModelForMaskedLM.from_pretrained("rufimelo/Legal-BERTimbau-base")
+```
+
+### Masked language modeling prediction example
+
+```python
+from  transformers  import  pipeline
+from  transformers  import  AutoTokenizer, AutoModelForMaskedLM
+
+tokenizer = AutoTokenizer.from_pretrained("rufimelo/Legal-BERTimbau-base")
+model = AutoModelForMaskedLM.from_pretrained("rufimelo/Legal-BERTimbau-base")
+
+pipe = pipeline('fill-mask', model=model, tokenizer=tokenizer)
+pipe('O advogado apresentou [MASK] para o juíz')
+# [{'score': 0.5034703612327576, 
+#'token': 8190, 
+#'token_str': 'recurso', 
+#'sequence': 'O advogado apresentou recurso para o juíz'}, 
+#{'score': 0.07347951829433441, 
+#'token': 21973, 
+#'token_str': 'petição', 
+#'sequence': 'O advogado apresentou petição para o juíz'}, 
+#{'score': 0.05165359005331993, 
+#'token': 4299, 
+#'token_str': 'resposta', 
+#'sequence': 'O advogado apresentou resposta para o juíz'}, 
+#{'score': 0.04611917585134506,
+#'token': 5265, 
+#'token_str': 'exposição', 
+#'sequence': 'O advogado apresentou exposição para o juíz'}, 
+#{'score': 0.04068068787455559, 
+#'token': 19737, 'token_str': 
+#'alegações', 
+#'sequence': 'O advogado apresentou alegações para o juíz'}]
+
+```
+
+### For BERT embeddings
+
+```python
+import  torch
+from  transformers  import  AutoModel
+
+model = AutoModel.from_pretrained('rufimelo/Legal-BERTimbau-base')
+input_ids = tokenizer.encode('O advogado apresentou recurso para o juíz', return_tensors='pt')
+
+with  torch.no_grad():
+	outs = model(input_ids)
+	encoded = outs[0][0, 1:-1]
+	
+#tensor([[ 0.0328, -0.4292, -0.6230, ..., -0.3048, -0.5674, 0.0157], 
+#[-0.3569, 0.3326, 0.7013, ..., -0.7778, 0.2646, 1.1310], 
+#[ 0.3169, 0.4333, 0.2026, ..., 1.0517, -0.1951, 0.7050], 
+#..., 
+#[-0.3648, -0.8137, -0.4764, ..., -0.2725, -0.4879, 0.6264], 
+#[-0.2264, -0.1821, -0.3011, ..., -0.5428, 0.1429, 0.0509], 
+#[-1.4617, 0.6281, -0.0625, ..., -1.2774, -0.4491, 0.3131]])
+```
+
+## Citation
+
+If you use this work, please cite BERTimbau's work:
+
+```bibtex
+@inproceedings{souza2020bertimbau,
+  author    = {F{\'a}bio Souza and
+               Rodrigo Nogueira and
+               Roberto Lotufo},
+  title     = {{BERT}imbau: pretrained {BERT} models for {B}razilian {P}ortuguese},
+  booktitle = {9th Brazilian Conference on Intelligent Systems, {BRACIS}, Rio Grande do Sul, Brazil, October 20-23 (to appear)},
+  year      = {2020}
+}
+```