|
--- |
|
language: ar |
|
datasets: |
|
- wikipedia |
|
- OSIAN |
|
- 1.5B Arabic Corpus |
|
- OSCAR Arabic Unshuffled |
|
widget: |
|
- text: " جاب ليا [MASK] ." |
|
--- |
|
|
|
|
|
|
|
|
|
|
|
**DarijaBERT** is the first BERT model for the Moroccan Arabic dialect called “Darija”. It is based on the same architecture as BERT-base, but without the Next Sentence Prediction (NSP) objective. This model was trained on a total of ~3 Million sequences of Darija dialect representing 691MB of text or a total of ~100M tokens. |
|
|
|
The model was trained on a dataset issued from three different sources: |
|
* Stories written in Darija scrapped from a dedicated website |
|
* Youtube comments from 40 different Moroccan channels |
|
* Tweets crawled based on a list of Darija keywords. |
|
|
|
More details about DarijaBert are available in the dedicated GitHub [repository](https://github.com/AIOXLABS/DBert) |
|
|
|
**Loading the model** |
|
|
|
The model can be loaded directly using the Huggingface library: |
|
|
|
```python |
|
from transformers import AutoTokenizer, AutoModel |
|
DBERT_tokenizer = AutoTokenizer.from_pretrained("Kamel/DarijaBERT") |
|
DBERT_Bert_model = AutoModel.from_pretrained("Kamel/DarijaBERT") |
|
``` |
|
|
|
**Acknowledgments** |
|
|
|
We gratefully acknowledge Google’s TensorFlow Research Cloud (TRC) program for providing us with free Cloud TPUs. |
|
|
|
|
|
|
