File size: 3,673 Bytes
b2c71a5 c9c3f0c b990572 c9c3f0c 2314e31 c9c3f0c b2c71a5 c9c3f0c fe1724b c9c3f0c fe1724b c9c3f0c fe1724b c9c3f0c fe1724b d2f8801 fe1724b d2f8801 fe1724b b990572 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 |
---
language:
- en
license: mit
base_model:
- mistralai/Mistral-7B-v0.1
datasets:
- HuggingFaceH4/ultrafeedback_binarized
pipeline_tag: text-generation
model-index:
- name: Mistral-ORPO-⍺
results:
- task:
type: text-generation
dataset:
name: AlpacaEval 1
type: AlpacaEval
metrics:
- type: AlpacaEval 1.0
value: 87.92%
name: Win Rate
source:
url: https://github.com/tatsu-lab/alpaca_eval
name: self-reported
- task:
type: text-generation
dataset:
name: AlpacaEval 2
type: AlpacaEval
metrics:
- type: AlpacaEval 2.0
value: 11.33%
name: Win Rate
source:
url: https://github.com/tatsu-lab/alpaca_eval
name: self-reported
- task:
type: text-generation
dataset:
name: MT-Bench
type: MT-Bench
metrics:
- type: MT-Bench
value: 7.23
name: Score
source:
url: https://github.com/lm-sys/FastChat/blob/main/fastchat/llm_judge/
name: self-reported
---
# **Mistral-ORPO-⍺ (7B)**
**Mistral-ORPO** is a fine-tuned version of [mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1) using the *odds ratio preference optimization (ORPO)*. With ORPO, the model directly learns the preference without the supervised fine-tuning warmup phase. **Mistral-ORPO-⍺** is fine-tuned exclusively on [HuggingFaceH4/ultrafeedback_binarized](https://huggingface.co/datasets/HuggingFaceH4/ultrafeedback_binarized).
- **Github Repository**: https://github.com/xfactlab/orpo
## 👍 **Model Performance**
### 1) AlpacaEval & MT-Bench
|Model Name|Size|Align|MT-Bench|AlpacaEval 1.0|AlpacaEval 2.0|
|:--------|:--------------:|:--------------:|:-------------------:|:------------:|:------------:|
|**Mistral-<tt>ORPO</tt>-⍺**|7B|<tt>ORPO</tt>|7.23|87.92|11.33|
|**Mistral-<tt>ORPO</tt>-β**|7B|<tt>ORPO</tt>|7.32|91.41|12.20|
|Zephyr β |7B|DPO|7.34|90.60|10.99|
|TULU-2-DPO |13B|DPO|7.00|89.5|10.12|
|Llama-2-Chat |7B|RLHF|6.27|71.37|4.96|
|Llama-2-Chat |13B|RLHF|6.65|81.09|7.70|
### 2) IFEval
| **Model Type** | **Prompt-Strict** | **Prompt-Loose** | **Inst-Strict** | **Inst-Loose** |
|--------------------|:-----------------:|:----------------:|:---------------:|:--------------:|
| **Mistral-ORPO-⍺** | 0.5009 | 0.5083 | 0.5995 | 0.6163 |
| **Mistral-ORPO-β** | 0.5287 | 0.5564 | 0.6355 | 0.6619 |
## 🗺️ **MT-Bench by Category**
## 🖥️ **Inference**
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("kaist-ai/mistral-orpo-alpha")
tokenizer = AutoTokenizer.from_pretrained("kaist-ai/mistral-orpo-alpha")
# Apply chat template
query = [{'role': 'user', 'content': 'Hi! How are you doing?'}]
prompt = tokenizer.apply_chat_template(query, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors='pt')
# Generation with specific configurations
output = model.generate(
**inputs,
max_new_tokens=128,
do_sample=True,
temperature=0.7
)
response = tokenizer.batch_decode(output)
#<|user|>
#Hi! How are you doing?</s>
#<|assistant|>
#I'm doing well, thank you! How are you?</s>
```
## 📎 **Citation**
```
@misc{hong2024orpo,
title={ORPO: Monolithic Preference Optimization without Reference Model},
author={Jiwoo Hong and Noah Lee and James Thorne},
year={2024},
eprint={2403.07691},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
``` |