readme

README.md

@@ -1,6 +1,7 @@
 ---
 language: en
 tags:
+- xlm-roberta
 - long context
 pipeline_tag: fill-mask
 ---
@@ -16,7 +17,7 @@ pipeline_tag: fill-mask
 * [Tasks](#tasks)
 * [Training global tokens](#training-global-tokens)
 
-This model is adapted from [XLM-roberta-base](https://huggingface.co/xlm-roberta-base) model without additional pretraining yet. It uses the same number of parameters/layers and the same tokenizer. 
+This model is adapted from [XLM-RoBERTa-base](https://huggingface.co/xlm-roberta-base) model without additional pretraining yet. It uses the same number of parameters/layers and the same tokenizer. 
 
 
 This model can handle long sequences but faster and more efficiently than Longformer or BigBird (from Transformers) and relies on Local + Sparse + Global attention (LSG).
@@ -148,3 +149,24 @@ for name, param in model.named_parameters():
     else:
         param.required_grad = True
 ```
+
+**XLM-RoBERTa**
+```
+@article{DBLP:journals/corr/abs-2105-00572,
+  author    = {Naman Goyal and
+               Jingfei Du and
+               Myle Ott and
+               Giri Anantharaman and
+               Alexis Conneau},
+  title     = {Larger-Scale Transformers for Multilingual Masked Language Modeling},
+  journal   = {CoRR},
+  volume    = {abs/2105.00572},
+  year      = {2021},
+  url       = {https://arxiv.org/abs/2105.00572},
+  eprinttype = {arXiv},
+  eprint    = {2105.00572},
+  timestamp = {Wed, 12 May 2021 15:54:31 +0200},
+  biburl    = {https://dblp.org/rec/journals/corr/abs-2105-00572.bib},
+  bibsource = {dblp computer science bibliography, https://dblp.org}
+}
+```