End of training

README.md

@@ -0,0 +1,75 @@
+---
+license: apache-2.0
+library_name: peft
+tags:
+- trl
+- orpo
+- generated_from_trainer
+base_model: cognitivecomputations/dolphin-2.1-mistral-7b
+model-index:
+- name: doplhin-2.1-mistral-7b-orpo-ultrafeedback-binarized-preferences
+  results: []
+---
+
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment. -->
+
+# doplhin-2.1-mistral-7b-orpo-ultrafeedback-binarized-preferences
+
+This model is a fine-tuned version of [cognitivecomputations/dolphin-2.1-mistral-7b](https://huggingface.co/cognitivecomputations/dolphin-2.1-mistral-7b) on an unknown dataset.
+It achieves the following results on the evaluation set:
+- Loss: 0.8506
+- Rewards/chosen: -0.0852
+- Rewards/rejected: -0.1166
+- Rewards/accuracies: 0.6457
+- Rewards/margins: 0.0314
+- Logps/rejected: -1.1665
+- Logps/chosen: -0.8525
+- Logits/rejected: -2.6517
+- Logits/chosen: -2.7250
+- Nll Loss: 0.7896
+- Log Odds Ratio: -0.6110
+- Log Odds Chosen: 0.4581
+
+## Model description
+
+More information needed
+
+## Intended uses & limitations
+
+More information needed
+
+## Training and evaluation data
+
+More information needed
+
+## Training procedure
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- learning_rate: 5e-05
+- train_batch_size: 4
+- eval_batch_size: 4
+- seed: 42
+- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
+- lr_scheduler_type: cosine
+- lr_scheduler_warmup_ratio: 0.1
+- num_epochs: 1
+
+### Training results
+
+| Training Loss | Epoch | Step | Validation Loss | Rewards/chosen | Rewards/rejected | Rewards/accuracies | Rewards/margins | Logps/rejected | Logps/chosen | Logits/rejected | Logits/chosen | Nll Loss | Log Odds Ratio | Log Odds Chosen |
+|:-------------:|:-----:|:----:|:---------------:|:--------------:|:----------------:|:------------------:|:---------------:|:--------------:|:------------:|:---------------:|:-------------:|:--------:|:--------------:|:---------------:|
+| 0.9101        | 0.25  | 700  | 0.8845          | -0.0869        | -0.1106          | 0.6428             | 0.0237          | -1.1059        | -0.8694      | -2.6631         | -2.7431       | 0.8224   | -0.6225        | 0.3631          |
+| 0.8554        | 0.51  | 1400 | 0.8609          | -0.0877        | -0.1233          | 0.6555             | 0.0357          | -1.2332        | -0.8766      | -2.6169         | -2.6996       | 0.8007   | -0.6040        | 0.5048          |
+| 0.9011        | 0.76  | 2100 | 0.8506          | -0.0852        | -0.1166          | 0.6457             | 0.0314          | -1.1665        | -0.8525      | -2.6517         | -2.7250       | 0.7896   | -0.6110        | 0.4581          |
+
+
+### Framework versions
+
+- PEFT 0.10.1.dev0
+- Transformers 4.40.0.dev0
+- Pytorch 2.1.2+cu121
+- Datasets 2.18.1.dev0
+- Tokenizers 0.15.2

adapter_model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:15f76f2fc25970bb9e5b489c4c4ed9d12f197513bbfc5cf942067f098231ca0d
+oid sha256:48b85f457aba7a4762d77ff4d5d862b9e7a6d0255d52895574d1f9f3c8971752
 size 335605144

runs/Apr12_11-12-00_eace2362923f/events.out.tfevents.1712920638.eace2362923f.1330.0

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:3a46a92092b6909419c13a6d23885ed34d0af1af9820f061953e6f906d23cdd6
-size 94269
+oid sha256:5302358620912a03d31d3f8733ec94f4e710018d4e92eb093b6e4ff3789bb6c6
+size 103223