ibraheemmoosa
commited on
Commit
·
c800e9f
1
Parent(s):
83798db
Add evaluation results
Browse files
README.md
CHANGED
@@ -127,7 +127,22 @@ The details of the sentence order prediction example generation procedure for ea
|
|
127 |
The model was pretrained on TPUv3-8 for 1M steps. We have checkpoints available every 10k steps. We will upload these in the future.
|
128 |
|
129 |
## Evaluation results
|
130 |
-
We evaluated this model on the [IndicGLUE](https://huggingface.co/datasets/indic_glue) benchmark dataset.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
131 |
|
132 |
## Intended uses & limitations
|
133 |
|
|
|
127 |
The model was pretrained on TPUv3-8 for 1M steps. We have checkpoints available every 10k steps. We will upload these in the future.
|
128 |
|
129 |
## Evaluation results
|
130 |
+
We evaluated this model on the Indo-Aryan subset of languages (Panjabi, Oriya, Assamese, Bangla, Hindi, Marathi, Gujarati) from the [IndicGLUE](https://huggingface.co/datasets/indic_glue) benchmark dataset. We report the mean and standard deviation of nine fine-tuning runs for this model.
|
131 |
+
|
132 |
+
### IndicGLUE
|
133 |
+
Task | mBERT | XLM-R | IndicBERT-Base | XLMIndic-Base-Uniscript (This Model)
|
134 |
+
-----| ----- | ----- | ------ | -------
|
135 |
+
Wikipedia Section Title Prediction| 71.90 | 65.45 | 69.40 | **81.78 ± 0.60**
|
136 |
+
Article Genre Classification | 88.64 | 96.61 | 97.72 | **98.70 ± 0.29**
|
137 |
+
Named Entity Recognition (F1-score) | 71.29 | 62.18 | 56.69 | **89.85 ± 1.14**
|
138 |
+
BBC Hindi News Article Classification | 60.55 | 75.52 | 74.60 | **77.28 ± 1.50**
|
139 |
+
Soham Bangla News Article Classification | 80.23 | 87.6 | 78.45 | **93.22 ± 0.49**
|
140 |
+
INLTK Gujarati Headlines | - | - | **92.91** | 90.41 ± 0.69
|
141 |
+
INLTK Marathi Headlines | - | - | **94.30** | 92.21 ± 0.23
|
142 |
+
IITP Hindi Product Reviews Sentiment Classification | 74.57 | **78.97** | 71.32 | 76.33 ± 0.84
|
143 |
+
IITP Hindi Movie Reviews Sentiment Classification | 56.77 | 61.61 | 59.03 | **65.91 ± 2.20**
|
144 |
+
MIDAS Hindi Discourse Classification | 71.20 | **79.94** | 78.44 | 78.39 ± 0.33
|
145 |
+
Cloze Style Question Answering | - | - | 37.16 | **41.54**
|
146 |
|
147 |
## Intended uses & limitations
|
148 |
|