|
--- |
|
datasets: |
|
- teknium/OpenHermes-2.5 |
|
--- |
|
This is an ExLlamaV2 quantized model in 4bpw of [feeltheAGI/yi-super-9B](https://huggingface.co/feeltheAGI/yi-super-9B) using the default calibration dataset. |
|
|
|
# Original Model card: |
|
|
|
 |
|
|
|
YI-9B-Super |
|
|
|
YI-9B-Super is an YI-9B model that has been further fine-tuned with OpenHermes-2.5 dataset. |
|
|
|
|
|
Results on some benchmarks : |
|
|
|
| Tasks |Version| Filter |n-shot| Metric | Value | |Stderr| |
|
|---------------------------------------|-------|----------------|------|-----------|------:|---|-----:| |
|
|truthfulqa |N/A |none | 0|rouge1_max |47.1011|± |0.8016| |
|
|hellaswag | 1|none |None |acc | 0.5758|± |0.0049| |
|
| | |none |None |acc_norm | 0.7639|± |0.0042| |
|
|gsm8k_cot | 3|strict-match |8 |exact_match| 0.5262|± |0.0138| |
|
| | |flexible-extract|8 |exact_match| 0.6027|± |0.0135| |
|
|gsm8k | 3|strict-match |5 |exact_match| 0.6073|± |0.0135| |
|
| | |flexible-extract|5 |exact_match| 0.6126|± |0.0134| |
|
|
|
|
|
|
|
| Groups |Version|Filter|n-shot| Metric | Value | |Stderr| |
|
|------------------|-------|------|------|-----------|------:|---|-----:| |
|
|truthfulqa |N/A |none | 0|rouge1_max |47.1011|± |0.8016| |
|
| | |none | 0|bleu_max |21.9476|± |0.7162| |
|
| | |none | 0|rouge2_acc | 0.3293|± |0.0165| |
|
| | |none | 0|bleu_acc | 0.3635|± |0.0168| |
|
| | |none | 0|rouge1_acc | 0.3892|± |0.0171| |
|
| | |none | 0|rougeL_acc | 0.3782|± |0.0170| |
|
| | |none | 0|bleu_diff |-2.3953|± |0.6292| |
|
| | |none | 0|rouge2_diff|-4.6929|± |0.9130| |
|
| | |none | 0|rougeL_diff|-4.2677|± |0.8034| |
|
| | |none | 0|acc | 0.4040|± |0.0113| |
|
| | |none | 0|rouge1_diff|-3.8975|± |0.7966| |
|
| | |none | 0|rougeL_max |43.7954|± |0.8145| |
|
| | |none | 0|rouge2_max |32.3573|± |0.9094| |
|
|mmlu |N/A |none | 0|acc | 0.6726|± |0.0037| |
|
| - humanities |N/A |none |None |acc | 0.6043|± |0.0067| |
|
| - other |N/A |none |None |acc | 0.7306|± |0.0077| |
|
| - social_sciences|N/A |none |None |acc | 0.7741|± |0.0074| |
|
| - stem |N/A |none |None |acc | 0.6181|± |0.0083| |
