README.md

metadata

language:
  - uk
tags:
  - ukrainian
widget:
  - text: Тарас Шевченко – великий український <mask>.
license: mit

This is a smaller version of the XLM-RoBERTa model with only Ukrainian and some English embeddings left.

• The original model has 470M parameters, with 384M of them being input and output embeddings.
• After shrinking the sentencepiece vocabulary from 250K to 31K (top 25K Ukrainian tokens and top English tokens) the number of model parameters reduced to 134M parameters, and model size reduced from 1GB to 400MB.