Commit
•
6cf5dfd
1
Parent(s):
3acf6c4
Create README.md
Browse files
README.md
ADDED
|
@@ -0,0 +1,13 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
language:
|
| 3 |
+
- uk
|
| 4 |
+
tags:
|
| 5 |
+
- ukrainian
|
| 6 |
+
widget:
|
| 7 |
+
- text: "Тарас Шевченко – великий український <mask>."
|
| 8 |
+
license: mit
|
| 9 |
+
---
|
| 10 |
+
This is a smaller version of the [XLM-RoBERTa](https://huggingface.co/xlm-roberta-base) model with only Ukrainian and some English embeddings left.
|
| 11 |
+
|
| 12 |
+
* The original model has 470M parameters, with 384M of them being input and output embeddings.
|
| 13 |
+
* After shrinking the `sentencepiece` vocabulary from 250K to 31K (top 25K Ukrainian tokens and top English tokens) the number of model parameters reduced to 134M parameters, and model size reduced from 1GB to 400MB.
|