Spaces:
Runtime error
Runtime error
Update src/about.py
Browse files- src/about.py +8 -4
src/about.py
CHANGED
@@ -42,6 +42,8 @@ linguistic skills and their level of bias, ethics, and trustworthiness. **These
|
|
42 |
explaining the data and performance of relevent models.**
|
43 |
|
44 |
Note: **We plan to release an evaluation framework soon in which the details and methods of evaluation are specified.**
|
|
|
|
|
45 |
"""
|
46 |
|
47 |
# Which evaluations are you running? how can people reproduce what you have?
|
@@ -51,10 +53,10 @@ For now, the only competitive open language models capable of properly speaking
|
|
51 |
There are only a few capable multilingual LLMs in Persian that derive their main knowledge from English. A Persian LLM is almost an imagination right now as there doesn't exist
|
52 |
that many models being expert in Persian in the first place.
|
53 |
|
54 |
-
Our
|
55 |
This benchmark can also be used by multilingual researchers to measure how well their model performs in a language like Persian.
|
56 |
|
57 |
-
We use our own framework to evaluate the models on the following benchmarks (TO BE RELEASED SOON)
|
58 |
### Tasks
|
59 |
- PeKA: Persian Knowledge Assesment (0-shot) - a set of multiple-choice questions that tests the level of native knowledge in Persian language in more 15 domains and categories: From art to history and geography, cinema, tv, sports, law and medicine, and much more.
|
60 |
- PersBETS: Persian Bias Ethics Toxicity and Skills (0-shot) - a test of model's capability in linguistic skills such as Grammar and Praphrasing, and also questions examining the bias, ethics, and toxicity of the model.
|
@@ -64,13 +66,15 @@ We use our own framework to evaluate the models on the following benchmarks (TO
|
|
64 |
- <a href="https://arxiv.org/abs/2012.06154" target="_blank"> ParsiNLU QQP </a> (max[0,2,5,10]-shot) - task of deciding whether a whether two given questions are paraphrases of each other or not.
|
65 |
|
66 |
For all these evaluations, a higher score is a better score.
|
|
|
67 |
We chose these benchmarks for now, but several other benchmarks are going to be added later to help us perform a more thorough examination of models.
|
|
|
68 |
The last two benchmarks, ParsiNLU NLI and ParsiNLU QQP are evaluated in different few-shot settings and then the maximum score is returned as the final evaluation.
|
69 |
-
We argue that is indeed a fair evaluation method since many
|
70 |
in small shots. We wish to not hold this against the model by trying to measure performances in different settings and take the maximum score achieved .
|
71 |
|
72 |
## REPRODUCIBILITY
|
73 |
-
(TO BE COMPLETED)
|
74 |
"""
|
75 |
|
76 |
EVALUATION_QUEUE_TEXT = """
|
|
|
42 |
explaining the data and performance of relevent models.**
|
43 |
|
44 |
Note: **We plan to release an evaluation framework soon in which the details and methods of evaluation are specified.**
|
45 |
+
|
46 |
+
Note: The two closed benchmarks are supposed to change drastically in the next two weeks, so this leaderboard's numeric values may need modifications once the framework and the corresponding data are made open.
|
47 |
"""
|
48 |
|
49 |
# Which evaluations are you running? how can people reproduce what you have?
|
|
|
53 |
There are only a few capable multilingual LLMs in Persian that derive their main knowledge from English. A Persian LLM is almost an imagination right now as there doesn't exist
|
54 |
that many models being expert in Persian in the first place.
|
55 |
|
56 |
+
Our goal is to provide a benchmark on diverse domains and tasks that provide insights on how much is the gap between current Persian LLMs and the SOTA multilingual models right now in different grounds.
|
57 |
This benchmark can also be used by multilingual researchers to measure how well their model performs in a language like Persian.
|
58 |
|
59 |
+
We use our own framework to evaluate the models on the following benchmarks (TO BE RELEASED SOON).
|
60 |
### Tasks
|
61 |
- PeKA: Persian Knowledge Assesment (0-shot) - a set of multiple-choice questions that tests the level of native knowledge in Persian language in more 15 domains and categories: From art to history and geography, cinema, tv, sports, law and medicine, and much more.
|
62 |
- PersBETS: Persian Bias Ethics Toxicity and Skills (0-shot) - a test of model's capability in linguistic skills such as Grammar and Praphrasing, and also questions examining the bias, ethics, and toxicity of the model.
|
|
|
66 |
- <a href="https://arxiv.org/abs/2012.06154" target="_blank"> ParsiNLU QQP </a> (max[0,2,5,10]-shot) - task of deciding whether a whether two given questions are paraphrases of each other or not.
|
67 |
|
68 |
For all these evaluations, a higher score is a better score.
|
69 |
+
|
70 |
We chose these benchmarks for now, but several other benchmarks are going to be added later to help us perform a more thorough examination of models.
|
71 |
+
|
72 |
The last two benchmarks, ParsiNLU NLI and ParsiNLU QQP are evaluated in different few-shot settings and then the maximum score is returned as the final evaluation.
|
73 |
+
We argue that is indeed a fair evaluation method since many light-weight models (around ~7B and less) can have a pooor in-context learning and thus they perform better
|
74 |
in small shots. We wish to not hold this against the model by trying to measure performances in different settings and take the maximum score achieved .
|
75 |
|
76 |
## REPRODUCIBILITY
|
77 |
+
The parameters used for evaluation along with instructions and prompts will be available once the framework is release. (TO BE COMPLETED)
|
78 |
"""
|
79 |
|
80 |
EVALUATION_QUEUE_TEXT = """
|