Update README.md

README.md

@@ -14,9 +14,9 @@ widget:
 ---
 
 # Biomedical language model for Spanish
+Biomedical pretrained language model for Spanish. For more details about the corpus, the pretraining and the evaluation, read the paper below.
 
 ## BibTeX  citation
-
 If you use any of these resources (datasets or models) in your work, please cite our latest paper:
 
 ```bibtex
@@ -30,9 +30,13 @@ If you use any of these resources (datasets or models) in your work, please cite
 }
 ```
 
-## Model and tokenization
+## Tokenization and model pretraining
+
 This model is a [RoBERTa-based](https://github.com/pytorch/fairseq/tree/master/examples/roberta) model trained on a
-**biomedical**  corpus collected from several sources (see next section).
+**biomedical** corpus in Spanish collected from several sources (see next section). 
+The training corpus has been tokenized using a byte version of [Byte-Pair Encoding (BPE)](https://github.com/openai/gpt-2)
+used in the original [RoBERTA](https://github.com/pytorch/fairseq/tree/master/examples/roberta) model with a vocabulary size of 52,000 tokens. The pretraining consists of a masked language model training at the subword level following the approach employed for the RoBERTa base model with the same hyperparameters as in the original work. The training lasted a total of 48 hours with 16 NVIDIA V100 GPUs of 16GB DDRAM, using Adam optimizer with a peak learning rate of 0.0005 and an effective batch size of 2,048 sentences.
+
 
 ## Training corpora and preprocessing