---
license: llama2
datasets:
- gair-prox/open-web-math-pro
language:
- en
base_model:
- codellama/CodeLlama-7b-hf
---



# CodeLlama-7B-ProXMath

<p align="center">
  <img src="prox-teaser.png">
</p>

[ArXiv](http://arxiv.org/abs/2409.17115) | [Data: OpenWebMath-Pro](https://huggingface.co/datasets/gair-prox/open-web-math-pro) | [Code](https://github.com/GAIR-NLP/program-every-example)

**CodeLlama-7B-ProXMath** is a math-adapted language model that is continually pre-trained on [OpenWebMath-Pro](https://huggingface.co/datasets/gair-prox/open-web-math-pro) (a refined version by ProX) for **10**B tokens.

## Evaluations

ProX models are evaluated on 9 common math reasoning benchmarks.

| Model                 |   asdiv  |   gsm8k  |  mathqa  |   mawps  | minerva_math | mmlu_stem | sat_math |   svamp  |  tabmwp  |  average |
|-----------------------|:--------:|:--------:|:--------:|:--------:|:------------:|:---------:|:--------:|:--------:|:--------:|:--------:|
| CodeLlama-7B          |   50.7   |   11.8   |   14.3   |   62.6   |      5.0     |    20.4   |   21.9   |   44.2   |   30.6   |   29.1   |
| CodeLlama-7B-ProXMath | **67.9** | **35.6** | **38.9** | **82.7** |   **17.6**   |  **42.6** | **62.5** | **55.8** | **41.3** | **49.4** |

### Citation
```
@article{zhou2024programming,
  title={Programming Every Example: Lifting Pre-training Data Quality like Experts at Scale},
  author={Zhou, Fan and Wang, Zengzhi and Liu, Qian and Li, Junlong and Liu, Pengfei},
  journal={arXiv preprint arXiv:2409.17115},
  year={2024}
}
```