File size: 3,529 Bytes
90127cd 81533a4 90127cd 80cab8f 90127cd |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 |
---
base_model: Haleshot/Mathmate-7B-DELLA
tags:
- finetuned
- orpo
- math
- preference-learning
datasets:
- argilla/distilabel-math-preference-dpo
license: apache-2.0
---
# Mathmate-7B-DELLA-ORPO
Mathmate-7B-DELLA-ORPO is a finetuned version of [Haleshot/Mathmate-7B-DELLA](https://huggingface.co/Haleshot/Mathmate-7B-DELLA) using the ORPO (Odds Ratio Preference Optimization) technique. This model has been specifically tuned to improve its performance on mathematical reasoning tasks based on human preferences.
## Model Details
- **Base Model:** [Haleshot/Mathmate-7B-DELLA](https://huggingface.co/Haleshot/Mathmate-7B-DELLA)
- **Finetuning Method:** ORPO (Odds Ratio Preference Optimization)
- **Training Dataset:** [argilla/distilabel-math-preference-dpo](https://huggingface.co/datasets/argilla/distilabel-math-preference-dpo)
## Finetuning
This model was finetuned using the ORPO technique, which is an extension of DPO (Direct Preference Optimization) that can work with ranked preferences instead of just binary ones. The process was adapted from the tutorial ["Fine-tune Llama 3 with ORPO"](https://mlabonne.github.io/blog/posts/2024-04-19_Fine_tune_Llama_3_with_ORPO.html) by Maxime Labonne, with some custom modifications to the code.
## Dataset
The model was finetuned on the [argilla/distilabel-math-preference-dpo](https://huggingface.co/datasets/argilla/distilabel-math-preference-dpo) dataset. This dataset contains mathematical problems along with multiple solution attempts, ranked by human preference. This allowed the model to learn from human judgments about what constitutes a good mathematical explanation or solution.
## Usage
Here's an example of how to use the model:
```python
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
model_name = "Haleshot/Mathmate-7B-DELLA-ORPO"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.float16, device_map="auto")
def generate_response(prompt, max_length=512):
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_length=max_length, num_return_sequences=1, do_sample=True, temperature=0.7)
return tokenizer.decode(outputs[0], skip_special_tokens=True)
# Example usage
prompt = "Solve the following equation: 2x + 5 = 13"
response = generate_response(prompt)
print(response)
```
## Limitations
While this model has been finetuned on mathematical problems, it may still make mistakes or provide incorrect solutions. Always verify the model's output, especially for critical applications or complex mathematical problems.
## References
1. Maxime Labonne. (2024). [Fine-tune Llama 3 with ORPO](https://mlabonne.github.io/blog/posts/2024-04-19_Fine_tune_Llama_3_with_ORPO.html)
2. Argilla. [distilabel-math-preference-dpo dataset](https://huggingface.co/datasets/argilla/distilabel-math-preference-dpo)
3. Haleshot. [Mathmate-7B-DELLA](https://huggingface.co/Haleshot/Mathmate-7B-DELLA)
## Citation
If you use this model in your research, please cite:
```
@misc{mathmate-7b-della-orpo,
author = {Haleshot},
title = {Mathmate-7B-DELLA-ORPO},
year = {2024},
publisher = {HuggingFace},
journal = {HuggingFace Hub},
howpublished = {\url{https://huggingface.co/Haleshot/Mathmate-7B-DELLA-ORPO}},
}
```
## Acknowledgements
Special thanks to Maxime Labonne for the ORPO finetuning tutorial, and to the Argilla team for providing the dataset used in this finetuning process. |