|
--- |
|
language: ro |
|
license: mit |
|
datasets: |
|
- oscar |
|
- wikipedia |
|
--- |
|
|
|
# Romanian DistilBERT |
|
|
|
This repository contains the uncased Romanian DistilBERT (named Distil-BERT-base-ro in the paper). The teacher model used for distillation is: [dumitrescustefan/bert-base-romanian-cased-v1](https://huggingface.co/dumitrescustefan/bert-base-romanian-cased-v1). |
|
|
|
The model was introduced in [this paper](https://arxiv.org/abs/2112.12650). The adjacent code can be found |
|
[here](https://github.com/racai-ai/Romanian-DistilBERT). |
|
|
|
## Usage |
|
|
|
```python |
|
from transformers import AutoTokenizer, AutoModel |
|
|
|
# load the tokenizer and the model |
|
tokenizer = AutoTokenizer.from_pretrained("racai/distilbert-base-romanian-cased") |
|
model = AutoModel.from_pretrained("racai/distilbert-base-romanian-cased") |
|
|
|
# tokenize a test sentence |
|
input_ids = tokenizer.encode("Aceasta este o propoziție de test.", add_special_tokens=True, return_tensors="pt") |
|
|
|
# run the tokens trough the model |
|
outputs = model(input_ids) |
|
|
|
print(outputs) |
|
``` |
|
|
|
## Model Size |
|
|
|
It is 35% smaller than its teacher `bert-base-romanian-cased-v1`. |
|
|
|
| Model | Size (MB) | Params (Millions) | |
|
|--------------------------------|:---------:|:----------------:| |
|
| bert-base-romanian-cased-v1 | 477 | 124 | |
|
| distilbert-base-romanian-cased | 312 | 81 | |
|
|
|
## Evaluation |
|
|
|
We evaluated the model in comparison with its teacher on 5 Romanian tasks: |
|
|
|
- **UPOS**: Universal Part of Speech (F1-macro) |
|
- **XPOS**: Extended Part of Speech (F1-macro) |
|
- **NER**: Named Entity Recognition (F1-macro) |
|
- **SAPN**: Sentiment Anlaysis - Positive vs Negative (Accuracy) |
|
- **SAR**: Sentiment Analysis - Rating (F1-macro) |
|
- **DI**: Dialect identification (F1-macro) |
|
- **STS**: Semantic Textual Similarity (Pearson) |
|
|
|
| Model | UPOS | XPOS | NER | SAPN | SAR | DI | STS | |
|
|--------------------------------|:----:|:----:|:---:|:----:|:---:|:--:|:---:| |
|
| bert-base-romanian-cased-v1 | 98.00 | 96.46 | 85.88 | 98.07 | 79.61 | 95.58 | 80.30 | |
|
| distilbert-base-romanian-cased | 97.97 | 97.08 | 83.35 | 98.20 | 80.51 | 96.31 | 80.57 | |
|
|
|
### BibTeX entry and citation info |
|
```bibtex |
|
@article{avram2021distilling, |
|
title={Distilling the Knowledge of Romanian BERTs Using Multiple Teachers}, |
|
author={Andrei-Marius Avram and Darius Catrina and Dumitru-Clementin Cercel and Mihai Dascălu and Traian Rebedea and Vasile Păiş and Dan Tufiş}, |
|
journal={ArXiv}, |
|
year={2021}, |
|
volume={abs/2112.12650} |
|
} |
|
``` |
|
|
