Create README.md

README.md

@@ -0,0 +1,81 @@
+---
+base_model: Haleshot/Mathmate-7B-DELLA
+tags:
+- finetuned
+- orpo
+- math
+- preference-learning
+datasets:
+- argilla/distilabel-math-preference-dpo
+license: apache-2.0
+---
+
+# Mathmate-7B-DELLA-ORPO
+
+Mathmate-7B-DELLA-ORPO is a finetuned version of [Haleshot/Mathmate-7B-DELLA](https://huggingface.co/Haleshot/Mathmate-7B-DELLA) using the ORPO (Offline Ranked Preference Optimization) technique. This model has been specifically tuned to improve its performance on mathematical reasoning tasks based on human preferences.
+
+## Model Details
+
+- **Base Model:** [Haleshot/Mathmate-7B-DELLA](https://huggingface.co/Haleshot/Mathmate-7B-DELLA)
+- **Finetuning Method:** ORPO (Offline Ranked Preference Optimization)
+- **Training Dataset:** [argilla/distilabel-math-preference-dpo](https://huggingface.co/datasets/argilla/distilabel-math-preference-dpo)
+
+## Finetuning
+
+This model was finetuned using the ORPO technique, which is an extension of DPO (Direct Preference Optimization) that can work with ranked preferences instead of just binary ones. The process was adapted from the tutorial ["Fine-tune Llama 3 with ORPO"](https://mlabonne.github.io/blog/posts/2024-04-19_Fine_tune_Llama_3_with_ORPO.html) by Maxime Labonne, with some custom modifications to the code.
+
+## Dataset
+
+The model was finetuned on the [argilla/distilabel-math-preference-dpo](https://huggingface.co/datasets/argilla/distilabel-math-preference-dpo) dataset. This dataset contains mathematical problems along with multiple solution attempts, ranked by human preference. This allowed the model to learn from human judgments about what constitutes a good mathematical explanation or solution.
+
+## Usage
+
+Here's an example of how to use the model:
+
+```python
+from transformers import AutoTokenizer, AutoModelForCausalLM
+import torch
+
+model_name = "Haleshot/Mathmate-7B-DELLA-ORPO"
+tokenizer = AutoTokenizer.from_pretrained(model_name)
+model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.float16, device_map="auto")
+
+def generate_response(prompt, max_length=512):
+    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
+    outputs = model.generate(**inputs, max_length=max_length, num_return_sequences=1, do_sample=True, temperature=0.7)
+    return tokenizer.decode(outputs[0], skip_special_tokens=True)
+
+# Example usage
+prompt = "Solve the following equation: 2x + 5 = 13"
+response = generate_response(prompt)
+print(response)
+```
+
+## Limitations
+
+While this model has been finetuned on mathematical problems, it may still make mistakes or provide incorrect solutions. Always verify the model's output, especially for critical applications or complex mathematical problems.
+
+## References
+
+1. Maxime Labonne. (2024). [Fine-tune Llama 3 with ORPO](https://mlabonne.github.io/blog/posts/2024-04-19_Fine_tune_Llama_3_with_ORPO.html)
+2. Argilla. [distilabel-math-preference-dpo dataset](https://huggingface.co/datasets/argilla/distilabel-math-preference-dpo)
+3. Haleshot. [Mathmate-7B-DELLA](https://huggingface.co/Haleshot/Mathmate-7B-DELLA)
+
+## Citation
+
+If you use this model in your research, please cite:
+
+```
+@misc{mathmate-7b-della-orpo,
+  author = {Haleshot},
+  title = {Mathmate-7B-DELLA-ORPO},
+  year = {2024},
+  publisher = {HuggingFace},
+  journal = {HuggingFace Hub},
+  howpublished = {\url{https://huggingface.co/Haleshot/Mathmate-7B-DELLA-ORPO}},
+}
+```
+
+## Acknowledgements
+
+Special thanks to Maxime Labonne for the ORPO finetuning tutorial, and to the Argilla team for providing the dataset used in this finetuning process.