Hugging Face's logo

yakazimir
/
qwen_orpo_entropy

Text Generation
Transformers
Safetensors
qwen2
alignment-handbook
trl
simpo
Generated from Trainer
conversational
text-generation-inference
Inference Endpoints
Model card Files Community
yakazimir commited on
Commit
b027249
•
1 Parent(s): 310ef48

Model save

Browse files
Files changed (4) hide show
  1. README.md +26 -33
  2. all_results.json +4 -18
  3. train_results.json +4 -4
  4. trainer_state.json +0 -0
README.md CHANGED
@@ -3,15 +3,9 @@ library_name: transformers
3
  license: other
4
  base_model: trl-lib/qwen1.5-0.5b-sft
5
  tags:
6
- - alignment-handbook
7
  - trl
8
  - simpo
9
  - generated_from_trainer
10
- - trl
11
- - simpo
12
- - generated_from_trainer
13
- datasets:
14
- - yakazimir/ultrafeedback_binarized
15
  model-index:
16
  - name: qwen_orpo_entropy
17
  results: []
@@ -22,18 +16,17 @@ should probably proofread and complete it, then remove this comment. -->
22
 
23
  # qwen_orpo_entropy
24
 
25
- This model is a fine-tuned version of [trl-lib/qwen1.5-0.5b-sft](https://huggingface.co/trl-lib/qwen1.5-0.5b-sft) on the yakazimir/ultrafeedback_binarized dataset.
26
  It achieves the following results on the evaluation set:
27
- - Loss: 0.5245
28
- - Rewards/chosen: -9.9757
29
- - Rewards/rejected: -11.1054
30
- - Rewards/accuracies: 0.7240
31
- - Rewards/margins: 1.1296
32
- - Logps/rejected: -11.1054
33
- - Logps/chosen: -9.9757
34
- - Logits/rejected: 0.9577
35
- - Logits/chosen: 0.9163
36
- - Semantic Entropy: 0.0013
37
 
38
  ## Model description
39
 
@@ -66,22 +59,22 @@ The following hyperparameters were used during training:
66
 
67
  ### Training results
68
 
69
- | Training Loss | Epoch | Step | Validation Loss | Rewards/chosen | Rewards/rejected | Rewards/accuracies | Rewards/margins | Logps/rejected | Logps/chosen | Logits/rejected | Logits/chosen | Semantic Entropy |
70
- |:-------------:|:------:|:----:|:---------------:|:--------------:|:----------------:|:------------------:|:---------------:|:--------------:|:------------:|:---------------:|:-------------:|:----------------:|
71
- | 1.0119 | 0.2141 | 400 | 1.0132 | -1.7930 | -2.0280 | 0.5660 | 0.2350 | -2.0280 | -1.7930 | 0.3600 | 0.2738 | 0.6132 |
72
- | 0.5924 | 0.4282 | 800 | 0.5851 | -6.7409 | -7.2987 | 0.6810 | 0.5578 | -7.2987 | -6.7409 | 0.5386 | 0.4884 | 0.0131 |
73
- | 0.5951 | 0.6422 | 1200 | 0.5522 | -7.9883 | -8.6813 | 0.7062 | 0.6931 | -8.6813 | -7.9883 | 0.7969 | 0.7507 | 0.0050 |
74
- | 0.4796 | 0.8563 | 1600 | 0.5406 | -8.4790 | -9.1974 | 0.7047 | 0.7184 | -9.1974 | -8.4790 | 0.9158 | 0.8517 | 0.0035 |
75
- | 0.5834 | 1.0704 | 2000 | 0.5344 | -8.7256 | -9.5131 | 0.7159 | 0.7875 | -9.5131 | -8.7256 | 0.8620 | 0.7784 | 0.0027 |
76
- | 0.5261 | 1.2845 | 2400 | 0.5313 | -8.7103 | -9.6511 | 0.7136 | 0.9408 | -9.6511 | -8.7103 | 0.8723 | 0.8012 | 0.0029 |
77
- | 0.4879 | 1.4986 | 2800 | 0.5264 | -8.6267 | -9.5330 | 0.7218 | 0.9063 | -9.5330 | -8.6267 | 0.7496 | 0.6896 | 0.0033 |
78
- | 0.5524 | 1.7127 | 3200 | 0.5207 | -8.8757 | -9.8346 | 0.7166 | 0.9589 | -9.8346 | -8.8757 | 0.9052 | 0.8485 | 0.0030 |
79
- | 0.5311 | 1.9267 | 3600 | 0.5170 | -9.0983 | -10.0747 | 0.7233 | 0.9765 | -10.0747 | -9.0983 | 0.8342 | 0.7884 | 0.0024 |
80
- | 0.3953 | 2.1408 | 4000 | 0.5261 | -9.8407 | -10.9409 | 0.7196 | 1.1002 | -10.9409 | -9.8407 | 0.9782 | 0.9286 | 0.0015 |
81
- | 0.428 | 2.3549 | 4400 | 0.5250 | -9.9515 | -11.0890 | 0.7211 | 1.1375 | -11.0890 | -9.9515 | 0.9721 | 0.9215 | 0.0013 |
82
- | 0.4394 | 2.5690 | 4800 | 0.5238 | -9.8173 | -10.9421 | 0.7255 | 1.1248 | -10.9421 | -9.8173 | 0.8956 | 0.8550 | 0.0014 |
83
- | 0.4221 | 2.7831 | 5200 | 0.5239 | -9.9581 | -11.0861 | 0.7248 | 1.1280 | -11.0861 | -9.9581 | 0.9048 | 0.8672 | 0.0013 |
84
- | 0.4023 | 2.9972 | 5600 | 0.5245 | -9.9757 | -11.1054 | 0.7240 | 1.1296 | -11.1054 | -9.9757 | 0.9577 | 0.9163 | 0.0013 |
85
 
86
 
87
  ### Framework versions
 
3
  license: other
4
  base_model: trl-lib/qwen1.5-0.5b-sft
5
  tags:
 
6
  - trl
7
  - simpo
8
  - generated_from_trainer
 
 
 
 
 
9
  model-index:
10
  - name: qwen_orpo_entropy
11
  results: []
 
16
 
17
  # qwen_orpo_entropy
18
 
19
+ This model is a fine-tuned version of [trl-lib/qwen1.5-0.5b-sft](https://huggingface.co/trl-lib/qwen1.5-0.5b-sft) on an unknown dataset.
20
  It achieves the following results on the evaluation set:
21
+ - Loss: 0.5257
22
+ - Rewards/chosen: -5.2305
23
+ - Rewards/rejected: -6.3460
24
+ - Rewards/accuracies: 0.7285
25
+ - Rewards/margins: 1.1155
26
+ - Logps/rejected: -6.3460
27
+ - Logps/chosen: -5.2305
28
+ - Logits/rejected: 0.3311
29
+ - Logits/chosen: 0.2347
 
30
 
31
  ## Model description
32
 
 
59
 
60
  ### Training results
61
 
62
+ | Training Loss | Epoch | Step | Validation Loss | Rewards/chosen | Rewards/rejected | Rewards/accuracies | Rewards/margins | Logps/rejected | Logps/chosen | Logits/rejected | Logits/chosen |
63
+ |:-------------:|:------:|:----:|:---------------:|:--------------:|:----------------:|:------------------:|:---------------:|:--------------:|:------------:|:---------------:|:-------------:|
64
+ | 0.7057 | 0.2141 | 400 | 0.7062 | -1.5154 | -1.6776 | 0.5571 | 0.1622 | -1.6776 | -1.5154 | 0.3357 | 0.2498 |
65
+ | 0.597 | 0.4282 | 800 | 0.5956 | -2.3371 | -2.7822 | 0.6669 | 0.4451 | -2.7822 | -2.3371 | 0.4528 | 0.3584 |
66
+ | 0.5883 | 0.6422 | 1200 | 0.5486 | -3.5511 | -4.2123 | 0.7211 | 0.6612 | -4.2123 | -3.5511 | 0.3923 | 0.2876 |
67
+ | 0.4794 | 0.8563 | 1600 | 0.5320 | -3.5255 | -4.2178 | 0.7277 | 0.6924 | -4.2178 | -3.5255 | 0.3881 | 0.2849 |
68
+ | 0.5765 | 1.0704 | 2000 | 0.5305 | -3.6701 | -4.4352 | 0.7240 | 0.7651 | -4.4352 | -3.6701 | 0.3104 | 0.1978 |
69
+ | 0.5449 | 1.2845 | 2400 | 0.5198 | -4.3149 | -5.2348 | 0.7352 | 0.9199 | -5.2348 | -4.3149 | 0.2247 | 0.1184 |
70
+ | 0.518 | 1.4986 | 2800 | 0.5189 | -4.2439 | -5.1423 | 0.7352 | 0.8983 | -5.1423 | -4.2439 | 0.3318 | 0.2186 |
71
+ | 0.5602 | 1.7127 | 3200 | 0.5174 | -4.3315 | -5.2509 | 0.7381 | 0.9194 | -5.2509 | -4.3315 | 0.3472 | 0.2362 |
72
+ | 0.5482 | 1.9267 | 3600 | 0.5152 | -4.3680 | -5.3320 | 0.7329 | 0.9640 | -5.3320 | -4.3680 | 0.3330 | 0.2233 |
73
+ | 0.4259 | 2.1408 | 4000 | 0.5296 | -5.1372 | -6.2156 | 0.7270 | 1.0783 | -6.2156 | -5.1372 | 0.3103 | 0.2143 |
74
+ | 0.4141 | 2.3549 | 4400 | 0.5245 | -5.3001 | -6.3996 | 0.7277 | 1.0995 | -6.3996 | -5.3001 | 0.3776 | 0.2775 |
75
+ | 0.4481 | 2.5690 | 4800 | 0.5253 | -5.2343 | -6.3529 | 0.7307 | 1.1185 | -6.3529 | -5.2343 | 0.4139 | 0.3107 |
76
+ | 0.3925 | 2.7831 | 5200 | 0.5251 | -5.2099 | -6.3202 | 0.7285 | 1.1103 | -6.3202 | -5.2099 | 0.3386 | 0.2411 |
77
+ | 0.4044 | 2.9972 | 5600 | 0.5257 | -5.2305 | -6.3460 | 0.7285 | 1.1155 | -6.3460 | -5.2305 | 0.3311 | 0.2347 |
78
 
79
 
80
  ### Framework versions
all_results.json CHANGED
@@ -1,23 +1,9 @@
1
  {
2
  "epoch": 2.999297541394882,
3
- "eval_logits/chosen": 0.9163225889205933,
4
- "eval_logits/rejected": 0.9576992988586426,
5
- "eval_logps/chosen": -9.975740432739258,
6
- "eval_logps/rejected": -11.105355262756348,
7
- "eval_loss": 0.5245004892349243,
8
- "eval_rewards/accuracies": 0.7240356206893921,
9
- "eval_rewards/chosen": -9.975740432739258,
10
- "eval_rewards/margins": 1.1296145915985107,
11
- "eval_rewards/rejected": -11.105355262756348,
12
- "eval_runtime": 34.6187,
13
- "eval_samples": 1345,
14
- "eval_samples_per_second": 38.852,
15
- "eval_semantic_entropy": 0.0012647123076021671,
16
- "eval_steps_per_second": 9.735,
17
  "total_flos": 0.0,
18
- "train_loss": 0.5450739759491819,
19
- "train_runtime": 29046.9509,
20
  "train_samples": 59790,
21
- "train_samples_per_second": 6.175,
22
- "train_steps_per_second": 0.193
23
  }
 
1
  {
2
  "epoch": 2.999297541394882,
 
 
 
 
 
 
 
 
 
 
 
 
 
 
3
  "total_flos": 0.0,
4
+ "train_loss": 0.5174550147332267,
5
+ "train_runtime": 29971.1802,
6
  "train_samples": 59790,
7
+ "train_samples_per_second": 5.985,
8
+ "train_steps_per_second": 0.187
9
  }
train_results.json CHANGED
@@ -1,9 +1,9 @@
1
  {
2
  "epoch": 2.999297541394882,
3
  "total_flos": 0.0,
4
- "train_loss": 0.5450739759491819,
5
- "train_runtime": 29046.9509,
6
  "train_samples": 59790,
7
- "train_samples_per_second": 6.175,
8
- "train_steps_per_second": 0.193
9
  }
 
1
  {
2
  "epoch": 2.999297541394882,
3
  "total_flos": 0.0,
4
+ "train_loss": 0.5174550147332267,
5
+ "train_runtime": 29971.1802,
6
  "train_samples": 59790,
7
+ "train_samples_per_second": 5.985,
8
+ "train_steps_per_second": 0.187
9
  }
trainer_state.json CHANGED
The diff for this file is too large to render. See raw diff