adding Yoruba BERT
Browse files- README.md +47 -0
- config.json +30 -0
- pytorch_model.bin +3 -0
- special_tokens_map.json +1 -0
- tokenizer_config.json +1 -0
- training_args.bin +3 -0
- vocab.txt +0 -0
README.md
ADDED
@@ -0,0 +1,47 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
Hugging Face's logo
|
2 |
+
---
|
3 |
+
language: yo
|
4 |
+
datasets:
|
5 |
+
- Bible, JW300, [Menyo-20k](https://huggingface.co/datasets/menyo20k_mt), [Yoruba Embedding corpus](https://huggingface.co/datasets/yoruba_text_c3) and [CC-Aligned](https://opus.nlpl.eu/), Wikipedia, news corpora (BBC Yoruba, VON Yoruba, Asejere, Alaroye), and other small datasets curated from friends.
|
6 |
+
---
|
7 |
+
# bert-base-multilingual-cased-finetuned-yoruba
|
8 |
+
## Model description
|
9 |
+
**bert-base-multilingual-cased-finetuned-yoruba** is a **Yoruba BERT** model obtained by fine-tuning **bert-base-multilingual-cased** model on Yorùbá language texts. It provides **better performance** than the multilingual BERT on text classification and named entity recognition datasets.
|
10 |
+
|
11 |
+
Specifically, this model is a *bert-base-multilingual-cased* model that was fine-tuned on Yorùbá corpus.
|
12 |
+
## Intended uses & limitations
|
13 |
+
#### How to use
|
14 |
+
You can use this model with Transformers *pipeline* for masked token prediction.
|
15 |
+
```python
|
16 |
+
from transformers import AutoTokenizer, AutoModelForTokenClassification
|
17 |
+
from transformers import pipeline
|
18 |
+
tokenizer = AutoTokenizer.from_pretrained("")
|
19 |
+
model = AutoModelForTokenClassification.from_pretrained("")
|
20 |
+
nlp = pipeline("", model=model, tokenizer=tokenizer)
|
21 |
+
example = "Emir of Kano turban Zhang wey don spend 18 years for Nigeria"
|
22 |
+
ner_results = nlp(example)
|
23 |
+
print(ner_results)
|
24 |
+
```
|
25 |
+
#### Limitations and bias
|
26 |
+
This model is limited by its training dataset of entity-annotated news articles from a specific span of time. This may not generalize well for all use cases in different domains.
|
27 |
+
## Training data
|
28 |
+
This model was fine-tuned on on JW300 Yorùbá corpus and [Menyo-20k](https://huggingface.co/datasets/menyo20k_mt) dataset
|
29 |
+
|
30 |
+
## Training procedure
|
31 |
+
This model was trained on a single NVIDIA V100 GPU
|
32 |
+
|
33 |
+
## Eval results on Test set (F-score)
|
34 |
+
Dataset|F1-score
|
35 |
+
-|-
|
36 |
+
|
37 |
+
Yoruba GV NER |86.26
|
38 |
+
MasakhaNER |75.76
|
39 |
+
BBC Yoruba |91.75
|
40 |
+
|
41 |
+
### BibTeX entry and citation info
|
42 |
+
By David Adelani
|
43 |
+
```
|
44 |
+
|
45 |
+
```
|
46 |
+
|
47 |
+
|
config.json
ADDED
@@ -0,0 +1,30 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
{
|
2 |
+
"_name_or_path": "bert-base-multilingual-cased",
|
3 |
+
"architectures": [
|
4 |
+
"BertForMaskedLM"
|
5 |
+
],
|
6 |
+
"attention_probs_dropout_prob": 0.1,
|
7 |
+
"directionality": "bidi",
|
8 |
+
"gradient_checkpointing": false,
|
9 |
+
"hidden_act": "gelu",
|
10 |
+
"hidden_dropout_prob": 0.1,
|
11 |
+
"hidden_size": 768,
|
12 |
+
"initializer_range": 0.02,
|
13 |
+
"intermediate_size": 3072,
|
14 |
+
"layer_norm_eps": 1e-12,
|
15 |
+
"max_position_embeddings": 512,
|
16 |
+
"model_type": "bert",
|
17 |
+
"num_attention_heads": 12,
|
18 |
+
"num_hidden_layers": 12,
|
19 |
+
"pad_token_id": 0,
|
20 |
+
"pooler_fc_size": 768,
|
21 |
+
"pooler_num_attention_heads": 12,
|
22 |
+
"pooler_num_fc_layers": 3,
|
23 |
+
"pooler_size_per_head": 128,
|
24 |
+
"pooler_type": "first_token_transform",
|
25 |
+
"position_embedding_type": "absolute",
|
26 |
+
"transformers_version": "4.3.2",
|
27 |
+
"type_vocab_size": 2,
|
28 |
+
"use_cache": true,
|
29 |
+
"vocab_size": 119547
|
30 |
+
}
|
pytorch_model.bin
ADDED
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
1 |
+
version https://git-lfs.github.com/spec/v1
|
2 |
+
oid sha256:1fd8d904fbd91faefa02c18ee1c36564e2a7332889d4437b462b0ad03910a99b
|
3 |
+
size 711988242
|
special_tokens_map.json
ADDED
@@ -0,0 +1 @@
|
|
|
|
|
1 |
+
{"unk_token": "[UNK]", "sep_token": "[SEP]", "pad_token": "[PAD]", "cls_token": "[CLS]", "mask_token": "[MASK]"}
|
tokenizer_config.json
ADDED
@@ -0,0 +1 @@
|
|
|
|
|
1 |
+
{"do_lower_case": false, "unk_token": "[UNK]", "sep_token": "[SEP]", "pad_token": "[PAD]", "cls_token": "[CLS]", "mask_token": "[MASK]", "tokenize_chinese_chars": true, "strip_accents": null, "model_max_length": 512, "name_or_path": "bert-base-multilingual-cased"}
|
training_args.bin
ADDED
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
1 |
+
version https://git-lfs.github.com/spec/v1
|
2 |
+
oid sha256:f7208c95e83a8c6c168de1505adabce7f51277e3067207441ffbb842e12312fd
|
3 |
+
size 2095
|
vocab.txt
ADDED
The diff for this file is too large to render.
See raw diff
|
|