zephyr-7b-dpo-lora-r16
This model is a fine-tuned version of alignment-handbook/zephyr-7b-sft-full on the HuggingFaceH4/ultrafeedback_binarized dataset. It achieves the following results on the evaluation set:
- Loss: 0.6410
- Rewards/chosen: -2.2125
- Rewards/rejected: -3.0591
- Rewards/accuracies: 0.6650
- Rewards/margins: 0.8466
- Logps/rejected: -554.3575
- Logps/chosen: -489.4880
- Logits/rejected: -2.1525
- Logits/chosen: -2.1542
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 5e-06
- train_batch_size: 4
- eval_batch_size: 8
- seed: 42
- distributed_type: multi-GPU
- gradient_accumulation_steps: 4
- total_train_batch_size: 16
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_ratio: 0.1
- num_epochs: 20
Training results
| Training Loss | Epoch | Step | Validation Loss | Rewards/chosen | Rewards/rejected | Rewards/accuracies | Rewards/margins | Logps/rejected | Logps/chosen | Logits/rejected | Logits/chosen |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 0.6933 | 0.992 | 62 | 0.6932 | 0.0001 | 0.0001 | 0.4950 | -0.0001 | -248.4330 | -268.2283 | -2.8603 | -2.8949 |
| 0.6898 | 2.0 | 125 | 0.6900 | -0.0019 | -0.0083 | 0.6300 | 0.0064 | -249.2751 | -268.4230 | -2.8539 | -2.8884 |
| 0.6767 | 2.992 | 187 | 0.6785 | -0.0026 | -0.0339 | 0.6300 | 0.0312 | -251.8334 | -268.4987 | -2.8254 | -2.8577 |
| 0.5861 | 4.0 | 250 | 0.6520 | -0.0414 | -0.1367 | 0.6350 | 0.0953 | -262.1136 | -272.3749 | -2.8027 | -2.8314 |
| 0.5654 | 4.992 | 312 | 0.6219 | -0.2603 | -0.4550 | 0.6500 | 0.1947 | -293.9497 | -294.2625 | -2.7777 | -2.8036 |
| 0.4986 | 6.0 | 375 | 0.6055 | -0.4927 | -0.7779 | 0.6800 | 0.2851 | -326.2355 | -317.5081 | -2.7652 | -2.7893 |
| 0.4719 | 6.992 | 437 | 0.6055 | -0.7077 | -1.0586 | 0.6900 | 0.3508 | -354.3046 | -339.0088 | -2.7391 | -2.7606 |
| 0.4512 | 8.0 | 500 | 0.6028 | -0.7213 | -1.1042 | 0.6750 | 0.3829 | -358.8697 | -340.3660 | -2.7246 | -2.7431 |
| 0.264 | 8.992 | 562 | 0.5955 | -1.0493 | -1.4939 | 0.7000 | 0.4446 | -397.8353 | -373.1655 | -2.6715 | -2.6867 |
| 0.3516 | 10.0 | 625 | 0.5927 | -1.1473 | -1.6948 | 0.6800 | 0.5474 | -417.9223 | -382.9673 | -2.5714 | -2.5856 |
| 0.3271 | 10.992 | 687 | 0.5922 | -1.4044 | -2.0377 | 0.6900 | 0.6332 | -452.2125 | -408.6782 | -2.4751 | -2.4864 |
| 0.336 | 12.0 | 750 | 0.6034 | -1.6164 | -2.3135 | 0.7100 | 0.6972 | -479.8002 | -429.8719 | -2.3841 | -2.3919 |
| 0.2157 | 12.992 | 812 | 0.6125 | -1.6968 | -2.4270 | 0.6800 | 0.7302 | -491.1445 | -437.9121 | -2.3161 | -2.3226 |
| 0.2436 | 14.0 | 875 | 0.6211 | -1.9546 | -2.7134 | 0.6800 | 0.7588 | -519.7897 | -463.6995 | -2.2583 | -2.2637 |
| 0.1747 | 14.992 | 937 | 0.6250 | -2.0090 | -2.8105 | 0.6750 | 0.8015 | -529.4984 | -469.1342 | -2.2179 | -2.2224 |
| 0.162 | 16.0 | 1000 | 0.6350 | -2.1464 | -2.9679 | 0.6750 | 0.8214 | -545.2337 | -482.8784 | -2.1872 | -2.1901 |
| 0.1898 | 16.992 | 1062 | 0.6415 | -2.2332 | -3.0695 | 0.6700 | 0.8363 | -555.3980 | -491.5554 | -2.1618 | -2.1639 |
| 0.1337 | 18.0 | 1125 | 0.6401 | -2.2070 | -3.0519 | 0.6700 | 0.8449 | -553.6342 | -488.9332 | -2.1605 | -2.1619 |
| 0.1233 | 18.992 | 1187 | 0.6414 | -2.2093 | -3.0569 | 0.6650 | 0.8476 | -554.1345 | -489.1610 | -2.1630 | -2.1636 |
| 0.1832 | 19.84 | 1240 | 0.6410 | -2.2125 | -3.0591 | 0.6650 | 0.8466 | -554.3575 | -489.4880 | -2.1525 | -2.1542 |
Framework versions
- PEFT 0.12.0
- Transformers 4.44.0
- Pytorch 2.4.0+cu121
- Datasets 2.20.0
- Tokenizers 0.19.1
- Downloads last month
- 1
Inference Providers
NEW
This model is not currently available via any of the supported Inference Providers.
The model cannot be deployed to the HF Inference API:
The model has no pipeline_tag.
Model tree for LaoRay/zephyr-7b-dpo-lora-r16
Base model
mistralai/Mistral-7B-v0.1
Finetuned
alignment-handbook/zephyr-7b-sft-full