We released Abel-7B-002, resulting in a stronger (35% improvement on GSM8K, 126% improvement on MATH) and more generalized model, achieving the best performance among all 7B models (80.44 on GSM8K, 29.46 on MATH)
| Model | GSM8k | MATH | MathQA | SVAMP | SCQ5K-EN | ARC-E | ARC-C | HellaSwag | MMLU |
|---|---|---|---|---|---|---|---|---|---|
| Abel-7B-002 | 80.44 | 29.46 | 69.78 | 77.67 | 55.95 | 77.67 | 55.05 | 77.72 | 61.19 |
| Abel-7B-001 | 59.74 | 13 | 1.21 | 57.67 | 9.3 | 53.32 | 38.97 | 63.51 | 40.59 |
| MetaMath-Mistral-7B | 77.7 | 28.2 | 33.94 | 79.33 | 37.6 | 78.48 | 51.93 | 76.44 | 61.93 |
| Qwen-7b | 47.84 | 9.34 | 27.44 | 53 | 40.05 | 74.97 | 53.05 | 86.85 | 57.98 |
| Mistral-7b | 37.83 | 9.06 | 25.73 | 63 | 39.6 | 76.83 | 53.22 | 76.31 | 64.05 |
| Yi-6b | 32.6 | 5.78 | 26.98 | 55.67 | 35.5 | 73.66 | 49.53 | 68.97 | 64.02 |
| LLaMA2-7b | 12.96 | 2.78 | 11.52 | 44 | 28.24 | 71.12 | 46.61 | 71.32 | 46.7 |
Please cite the repo if the model/code/conclusion in this repo are helpful to you.
@misc{abel,
author = {Chern, Ethan and Zou, Haoyang and Li, Xuefeng and Hu, Jiewen and Feng, Kehua and Li, Junlong and Liu, Pengfei},
title = {Generative AI for Math: Abel},
year = {2023},
publisher = {GitHub},
journal = {GitHub repository},
howpublished = {\url{https://github.com/GAIR-NLP/abel}},
}