Model Description
This is a model using the llama2 architecture and only 30 million parameters. It is based off of this model and was finetuned on approximately 85 million tokens of instruct data from the first 20000 rows of the openhermes 2.5 dataset with a low learning rate of 2e-6 and context length of 512.
Open LLM Leaderboard Evaluation Results
Detailed results can be found here
| Metric | Value |
|---|---|
| Avg. | 4.32 |
| IFEval (0-Shot) | 17.06 |
| BBH (3-Shot) | 2.48 |
| MATH Lvl 5 (4-Shot) | 0.00 |
| GPQA (0-shot) | 1.34 |
| MuSR (0-shot) | 3.18 |
| MMLU-PRO (5-shot) | 1.85 |
- Downloads last month
- 18
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social
visibility and check back later, or deploy to Inference Endpoints (dedicated)
instead.
Model tree for cpayne1303/cp2024-instruct
Base model
cpayne1303/cp2024Dataset used to train cpayne1303/cp2024-instruct
Evaluation results
- strict accuracy on IFEval (0-Shot)Open LLM Leaderboard17.060
- normalized accuracy on BBH (3-Shot)Open LLM Leaderboard2.480
- exact match on MATH Lvl 5 (4-Shot)Open LLM Leaderboard0.000
- acc_norm on GPQA (0-shot)Open LLM Leaderboard1.340
- acc_norm on MuSR (0-shot)Open LLM Leaderboard3.180
- accuracy on MMLU-PRO (5-shot)test set Open LLM Leaderboard1.850