Update README.md

README.md

@@ -1,3 +1,77 @@
 ---
 license: cc0-1.0
+language:
+- mt
+tags:
+- MaltBERTa
+- MaCoCu
 ---
+
+# Model description
+
+**XLMR-MaltBERTa** is a large pre-trained language model trained on Maltese texts. It was created by continuing training from the [XLM-RoBERTa-large](https://huggingface.co/xlm-roberta-large) model. It was developed as part of the [MaCoCu](https://macocu.eu/) project. The main developer is [Rik van Noord](https://www.rikvannoord.nl/) from the University of Groningen.
+
+XLMR-MaltBERTa was trained on 3.2GB of text, which is equal to 439M tokens. It was trained for 50,000 steps with a batch size of 1,024. It uses the same vocabulary as the original XLMR-large model. The model is trained on the same data as [MaltBERTa](https://huggingface.co/RVN/MaltBERTa), but this model was trained from scratch using the RoBERTa architecture.
+
+The training and fine-tuning procedures are described in detail on our [Github repo](https://github.com/macocu/LanguageModels).
+
+# How to use
+
+```python
+from transformers import AutoTokenizer, AutoModel, TFAutoModel
+
+tokenizer = AutoTokenizer.from_pretrained("RVN/XLMR-MaltBERTa")
+model = AutoModel.from_pretrained("RVN/XLMR-MaltBERTa") # PyTorch
+model = TFAutoModel.from_pretrained("RVN/XLMR-MaltBERTa") # Tensorflow
+```
+
+# Data
+
+For training, we used all Maltese data that was present in the [MaCoCu](https://macocu.eu/), Oscar and mc4 corpora. After de-duplicating the data, we were left with a total of 3.2GB of text.
+
+# Benchmark performance
+
+We tested the performance of MaltBERTa on the UPOS and XPOS benchmark of the [Universal Dependencies](https://universaldependencies.org/) project.  We compare performance to the strong multi-lingual models XLMR-base and XLMR-large, though note that Maltese was not one of the training languages for those models. We also compare to the recently introduced Maltese language models [BERTu](https://huggingface.co/MLRS/BERTu), [mBERTu](https://huggingface.co/MLRS/mBERTu) and our own [MaltBERTa](https://huggingface.co/RVN/MaltBERTa). For details regarding the fine-tuning procedure you can checkout our [Github](https://github.com/macocu/LanguageModels).
+
+Scores are averages of three runs. We use the same hyperparameter settings for all models.
+
+|                 | **UPOS** | **UPOS** | **XPOS** | **XPOS** |
+|-----------------|:--------:|:--------:|:--------:|:--------:|
+|                 |  **Dev** | **Test** |  **Dev** | **Test** |
+| **XLM-R-base**  |   93.6   |   93.2   |   93.4   |   93.2   |
+| **XLM-R-large** |   94.9   |   94.4   |   95.1   |   94.7   |
+| **BERTu**       |   97.5   |   97.6   |   95.7   |   95.8   |
+| **mBERTu**      |   **97.7**   |   97.8   |   97.9   |   98.1   |
+| **MaltBERTa**   |   95.7   |   95.8   |   96.1   |   96.0   |
+| **XLMR-MaltBERTa**   |   **97.7**   |   **98.1**  |   **98.1**  |   **98.2**   |
+
+# Citation
+
+If you use this model, please cite the following paper:
+
+```bibtex
+@inproceedings{non-etal-2022-macocu,
+    title = "{M}a{C}o{C}u: Massive collection and curation of monolingual and bilingual data: focus on under-resourced languages",
+    author = "Ba{\~n}{\'o}n, Marta  and
+      Espl{\`a}-Gomis, Miquel  and
+      Forcada, Mikel L.  and
+      Garc{\'\i}a-Romero, Cristian  and
+      Kuzman, Taja  and
+      Ljube{\v{s}}i{\'c}, Nikola  and
+      van Noord, Rik  and
+      Sempere, Leopoldo Pla  and
+      Ram{\'\i}rez-S{\'a}nchez, Gema  and
+      Rupnik, Peter  and
+      Suchomel, V{\'\i}t  and
+      Toral, Antonio  and
+      van der Werff, Tobias  and
+      Zaragoza, Jaume",
+    booktitle = "Proceedings of the 23rd Annual Conference of the European Association for Machine Translation",
+    month = jun,
+    year = "2022",
+    address = "Ghent, Belgium",
+    publisher = "European Association for Machine Translation",
+    url = "https://aclanthology.org/2022.eamt-1.41",
+    pages = "303--304"
+}
+```