Hugging Face's logo

DUAL-GPO
/
phi-2-dpo-chatml-lora-0k-20k-i2

PEFT
TensorBoard
Safetensors
phi
alignment-handbook
Generated from Trainer
trl
dpo
custom_code
Model card Files Metrics Community
BraylonDash commited on
Commit
a0c1f08
·
verified ·
1 Parent(s): 4151291

Model save

Browse files
Files changed (7) hide show
  1. README.md +61 -0
  2. adapter_model.safetensors +1 -1
  3. all_results.json +8 -0
  4. emissions.csv +2 -0
  5. runs/Sep13_11-32-21_gpu4-119-5/events.out.tfevents.1726191188.gpu4-119-5.2970637.0 +2 -2
  6. train_results.json +8 -0
  7. trainer_state.json +478 -0
README.md ADDED
@@ -0,0 +1,61 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ library_name: peft
3
+ tags:
4
+ - trl
5
+ - dpo
6
+ - generated_from_trainer
7
+ base_model: DUAL-GPO/phi-2-dpo-chatml-lora-40k-60k-i1-merged
8
+ model-index:
9
+ - name: phi-2-dpo-chatml-lora-0k-20k-i2
10
+ results: []
11
+ ---
12
+
13
+ <!-- This model card has been generated automatically according to the information the Trainer had access to. You
14
+ should probably proofread and complete it, then remove this comment. -->
15
+
16
+ # phi-2-dpo-chatml-lora-0k-20k-i2
17
+
18
+ This model is a fine-tuned version of [DUAL-GPO/phi-2-dpo-chatml-lora-40k-60k-i1-merged](https://huggingface.co/DUAL-GPO/phi-2-dpo-chatml-lora-40k-60k-i1-merged) on the None dataset.
19
+
20
+ ## Model description
21
+
22
+ More information needed
23
+
24
+ ## Intended uses & limitations
25
+
26
+ More information needed
27
+
28
+ ## Training and evaluation data
29
+
30
+ More information needed
31
+
32
+ ## Training procedure
33
+
34
+ ### Training hyperparameters
35
+
36
+ The following hyperparameters were used during training:
37
+ - learning_rate: 5e-06
38
+ - train_batch_size: 4
39
+ - eval_batch_size: 4
40
+ - seed: 42
41
+ - distributed_type: multi-GPU
42
+ - num_devices: 4
43
+ - gradient_accumulation_steps: 4
44
+ - total_train_batch_size: 64
45
+ - total_eval_batch_size: 16
46
+ - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
47
+ - lr_scheduler_type: cosine
48
+ - lr_scheduler_warmup_ratio: 0.1
49
+ - num_epochs: 1
50
+
51
+ ### Training results
52
+
53
+
54
+
55
+ ### Framework versions
56
+
57
+ - PEFT 0.7.1
58
+ - Transformers 4.36.2
59
+ - Pytorch 2.1.2
60
+ - Datasets 2.14.6
61
+ - Tokenizers 0.15.2
adapter_model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:5b48f1f0a951f8e38502d09a09df6f3d6072d63d42c52d6c68821e0c56c52358
3
  size 335579632
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:24e76b224ad1b597104577d5420d424308b6fe61430ca9d2dd763133e78ee8a6
3
  size 335579632
all_results.json ADDED
@@ -0,0 +1,8 @@
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "epoch": 1.0,
3
+ "train_loss": 0.6651641822014099,
4
+ "train_runtime": 4241.3739,
5
+ "train_samples": 20000,
6
+ "train_samples_per_second": 4.715,
7
+ "train_steps_per_second": 0.074
8
+ }
emissions.csv ADDED
@@ -0,0 +1,2 @@
 
 
 
1
+ timestamp,project_name,run_id,duration,emissions,emissions_rate,cpu_power,gpu_power,ram_power,cpu_energy,gpu_energy,ram_energy,energy_consumed,country_name,country_iso_code,region,cloud_provider,cloud_region,os,python_version,codecarbon_version,cpu_count,cpu_model,gpu_count,gpu_model,longitude,latitude,ram_total_size,tracking_mode,on_cloud,pue
2
+ 2024-09-13T12:43:49,codecarbon,d52a4edb-2daa-4548-87c5-e3a2fb49fe37,4241.388695478439,0.003876836939733367,9.140489632243078e-07,42.5,1268.194,188.74309015274048,0.050071360430287,1.3589831897167446,0.22177438062556462,1.630828930772596,Canada,CAN,quebec,,,Linux-5.15.0-84-generic-x86_64-with-glibc2.35,3.10.14,2.2.3,32,Intel(R) Xeon(R) W-3335 CPU @ 3.40GHz,4,4 x NVIDIA GeForce RTX 4090,-71.2,46.8,503.3149070739746,machine,N,1.0
runs/Sep13_11-32-21_gpu4-119-5/events.out.tfevents.1726191188.gpu4-119-5.2970637.0 CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:b7981334e4c60542e8ba1a830cefc20c391c4d72d8bfdbbe8b9796a8ac1e2ff1
3
- size 24456
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2dc8b4d8eab25c3a4173cd2cff531b862ef2c9d3d4bf8dd94ff92ed4455e7015
3
+ size 25444
train_results.json ADDED
@@ -0,0 +1,8 @@
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "epoch": 1.0,
3
+ "train_loss": 0.6651641822014099,
4
+ "train_runtime": 4241.3739,
5
+ "train_samples": 20000,
6
+ "train_samples_per_second": 4.715,
7
+ "train_steps_per_second": 0.074
8
+ }
trainer_state.json ADDED
@@ -0,0 +1,478 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "best_metric": null,
3
+ "best_model_checkpoint": null,
4
+ "epoch": 0.9984,
5
+ "eval_steps": 500,
6
+ "global_step": 312,
7
+ "is_hyper_param_search": false,
8
+ "is_local_process_zero": true,
9
+ "is_world_process_zero": true,
10
+ "log_history": [
11
+ {
12
+ "epoch": 0.0,
13
+ "learning_rate": 1.5625e-07,
14
+ "logits/chosen": 0.5981850028038025,
15
+ "logits/rejected": 0.7171921730041504,
16
+ "logps/chosen": -1114.7255859375,
17
+ "logps/rejected": -1146.7080078125,
18
+ "loss": 0.6931,
19
+ "rewards/accuracies": 0.0,
20
+ "rewards/chosen": 0.0,
21
+ "rewards/margins": 0.0,
22
+ "rewards/rejected": 0.0,
23
+ "step": 1
24
+ },
25
+ {
26
+ "epoch": 0.03,
27
+ "learning_rate": 1.5625e-06,
28
+ "logits/chosen": 0.4006383419036865,
29
+ "logits/rejected": 0.578837513923645,
30
+ "logps/chosen": -1050.73681640625,
31
+ "logps/rejected": -1196.98828125,
32
+ "loss": 0.693,
33
+ "rewards/accuracies": 0.4236111044883728,
34
+ "rewards/chosen": -0.0012154510477557778,
35
+ "rewards/margins": 0.0003596623719204217,
36
+ "rewards/rejected": -0.0015751136234030128,
37
+ "step": 10
38
+ },
39
+ {
40
+ "epoch": 0.06,
41
+ "learning_rate": 3.125e-06,
42
+ "logits/chosen": 0.43995967507362366,
43
+ "logits/rejected": 0.6202905774116516,
44
+ "logps/chosen": -1097.7890625,
45
+ "logps/rejected": -1239.5677490234375,
46
+ "loss": 0.6913,
47
+ "rewards/accuracies": 0.518750011920929,
48
+ "rewards/chosen": -0.02275204099714756,
49
+ "rewards/margins": 0.004399357363581657,
50
+ "rewards/rejected": -0.027151400223374367,
51
+ "step": 20
52
+ },
53
+ {
54
+ "epoch": 0.1,
55
+ "learning_rate": 4.6875000000000004e-06,
56
+ "logits/chosen": 0.5573526620864868,
57
+ "logits/rejected": 0.613277018070221,
58
+ "logps/chosen": -1259.811767578125,
59
+ "logps/rejected": -1435.0015869140625,
60
+ "loss": 0.6871,
61
+ "rewards/accuracies": 0.581250011920929,
62
+ "rewards/chosen": -0.11753509938716888,
63
+ "rewards/margins": 0.0262240469455719,
64
+ "rewards/rejected": -0.14375916123390198,
65
+ "step": 30
66
+ },
67
+ {
68
+ "epoch": 0.13,
69
+ "learning_rate": 4.989935734988098e-06,
70
+ "logits/chosen": 0.4959263801574707,
71
+ "logits/rejected": 0.6557127237319946,
72
+ "logps/chosen": -1347.3275146484375,
73
+ "logps/rejected": -1479.0496826171875,
74
+ "loss": 0.6814,
75
+ "rewards/accuracies": 0.4749999940395355,
76
+ "rewards/chosen": -0.24454805254936218,
77
+ "rewards/margins": 0.0380481593310833,
78
+ "rewards/rejected": -0.2825961709022522,
79
+ "step": 40
80
+ },
81
+ {
82
+ "epoch": 0.16,
83
+ "learning_rate": 4.949188496058089e-06,
84
+ "logits/chosen": 0.5846071243286133,
85
+ "logits/rejected": 0.6737171411514282,
86
+ "logps/chosen": -1493.431884765625,
87
+ "logps/rejected": -1723.5257568359375,
88
+ "loss": 0.6683,
89
+ "rewards/accuracies": 0.5249999761581421,
90
+ "rewards/chosen": -0.36657729744911194,
91
+ "rewards/margins": 0.11142341792583466,
92
+ "rewards/rejected": -0.4780006408691406,
93
+ "step": 50
94
+ },
95
+ {
96
+ "epoch": 0.19,
97
+ "learning_rate": 4.8776412907378845e-06,
98
+ "logits/chosen": 0.5173619985580444,
99
+ "logits/rejected": 0.784024715423584,
100
+ "logps/chosen": -1601.8973388671875,
101
+ "logps/rejected": -1787.753173828125,
102
+ "loss": 0.6729,
103
+ "rewards/accuracies": 0.5375000238418579,
104
+ "rewards/chosen": -0.46918588876724243,
105
+ "rewards/margins": 0.08332391828298569,
106
+ "rewards/rejected": -0.5525098443031311,
107
+ "step": 60
108
+ },
109
+ {
110
+ "epoch": 0.22,
111
+ "learning_rate": 4.7761938666470405e-06,
112
+ "logits/chosen": 0.5953202247619629,
113
+ "logits/rejected": 0.6078029274940491,
114
+ "logps/chosen": -1537.4744873046875,
115
+ "logps/rejected": -1700.5455322265625,
116
+ "loss": 0.6787,
117
+ "rewards/accuracies": 0.5,
118
+ "rewards/chosen": -0.41651010513305664,
119
+ "rewards/margins": 0.07196612656116486,
120
+ "rewards/rejected": -0.4884762167930603,
121
+ "step": 70
122
+ },
123
+ {
124
+ "epoch": 0.26,
125
+ "learning_rate": 4.646121984004666e-06,
126
+ "logits/chosen": 0.5242379307746887,
127
+ "logits/rejected": 0.600353479385376,
128
+ "logps/chosen": -1465.7352294921875,
129
+ "logps/rejected": -1685.4124755859375,
130
+ "loss": 0.6733,
131
+ "rewards/accuracies": 0.550000011920929,
132
+ "rewards/chosen": -0.37152013182640076,
133
+ "rewards/margins": 0.09310854226350784,
134
+ "rewards/rejected": -0.4646286964416504,
135
+ "step": 80
136
+ },
137
+ {
138
+ "epoch": 0.29,
139
+ "learning_rate": 4.4890613722044526e-06,
140
+ "logits/chosen": 0.5765184164047241,
141
+ "logits/rejected": 0.651719331741333,
142
+ "logps/chosen": -1519.32470703125,
143
+ "logps/rejected": -1666.23828125,
144
+ "loss": 0.6722,
145
+ "rewards/accuracies": 0.48750001192092896,
146
+ "rewards/chosen": -0.4189775884151459,
147
+ "rewards/margins": 0.048159025609493256,
148
+ "rewards/rejected": -0.4671366214752197,
149
+ "step": 90
150
+ },
151
+ {
152
+ "epoch": 0.32,
153
+ "learning_rate": 4.3069871595684795e-06,
154
+ "logits/chosen": 0.5278979539871216,
155
+ "logits/rejected": 0.7525848746299744,
156
+ "logps/chosen": -1449.8560791015625,
157
+ "logps/rejected": -1631.3978271484375,
158
+ "loss": 0.6676,
159
+ "rewards/accuracies": 0.5249999761581421,
160
+ "rewards/chosen": -0.3548738658428192,
161
+ "rewards/margins": 0.08127720654010773,
162
+ "rewards/rejected": -0.43615108728408813,
163
+ "step": 100
164
+ },
165
+ {
166
+ "epoch": 0.35,
167
+ "learning_rate": 4.102189034962561e-06,
168
+ "logits/chosen": 0.6309102773666382,
169
+ "logits/rejected": 0.6685208082199097,
170
+ "logps/chosen": -1543.900634765625,
171
+ "logps/rejected": -1706.1783447265625,
172
+ "loss": 0.6621,
173
+ "rewards/accuracies": 0.581250011920929,
174
+ "rewards/chosen": -0.3921688199043274,
175
+ "rewards/margins": 0.10156150162220001,
176
+ "rewards/rejected": -0.49373024702072144,
177
+ "step": 110
178
+ },
179
+ {
180
+ "epoch": 0.38,
181
+ "learning_rate": 3.8772424536302565e-06,
182
+ "logits/chosen": 0.6601932048797607,
183
+ "logits/rejected": 0.5677546262741089,
184
+ "logps/chosen": -1639.169677734375,
185
+ "logps/rejected": -1795.3013916015625,
186
+ "loss": 0.6627,
187
+ "rewards/accuracies": 0.5249999761581421,
188
+ "rewards/chosen": -0.5095900297164917,
189
+ "rewards/margins": 0.09199462831020355,
190
+ "rewards/rejected": -0.6015846133232117,
191
+ "step": 120
192
+ },
193
+ {
194
+ "epoch": 0.42,
195
+ "learning_rate": 3.634976249348867e-06,
196
+ "logits/chosen": 0.569564700126648,
197
+ "logits/rejected": 0.6693593859672546,
198
+ "logps/chosen": -1584.231689453125,
199
+ "logps/rejected": -1769.0833740234375,
200
+ "loss": 0.6718,
201
+ "rewards/accuracies": 0.48124998807907104,
202
+ "rewards/chosen": -0.49371838569641113,
203
+ "rewards/margins": 0.08162926137447357,
204
+ "rewards/rejected": -0.5753476023674011,
205
+ "step": 130
206
+ },
207
+ {
208
+ "epoch": 0.45,
209
+ "learning_rate": 3.3784370602033572e-06,
210
+ "logits/chosen": 0.7315655946731567,
211
+ "logits/rejected": 0.6511383652687073,
212
+ "logps/chosen": -1529.665771484375,
213
+ "logps/rejected": -1825.186767578125,
214
+ "loss": 0.6655,
215
+ "rewards/accuracies": 0.5249999761581421,
216
+ "rewards/chosen": -0.4015732705593109,
217
+ "rewards/margins": 0.1370248943567276,
218
+ "rewards/rejected": -0.5385981798171997,
219
+ "step": 140
220
+ },
221
+ {
222
+ "epoch": 0.48,
223
+ "learning_rate": 3.1108510153447352e-06,
224
+ "logits/chosen": 0.5098114013671875,
225
+ "logits/rejected": 0.7448014616966248,
226
+ "logps/chosen": -1524.7633056640625,
227
+ "logps/rejected": -1739.652099609375,
228
+ "loss": 0.6609,
229
+ "rewards/accuracies": 0.5249999761581421,
230
+ "rewards/chosen": -0.38486695289611816,
231
+ "rewards/margins": 0.09735463559627533,
232
+ "rewards/rejected": -0.4822216033935547,
233
+ "step": 150
234
+ },
235
+ {
236
+ "epoch": 0.51,
237
+ "learning_rate": 2.835583164544139e-06,
238
+ "logits/chosen": 0.523254930973053,
239
+ "logits/rejected": 0.7159970998764038,
240
+ "logps/chosen": -1473.3905029296875,
241
+ "logps/rejected": -1742.4814453125,
242
+ "loss": 0.6645,
243
+ "rewards/accuracies": 0.625,
244
+ "rewards/chosen": -0.34661874175071716,
245
+ "rewards/margins": 0.12415720522403717,
246
+ "rewards/rejected": -0.4707759916782379,
247
+ "step": 160
248
+ },
249
+ {
250
+ "epoch": 0.54,
251
+ "learning_rate": 2.556095160739513e-06,
252
+ "logits/chosen": 0.5594567060470581,
253
+ "logits/rejected": 0.7879417538642883,
254
+ "logps/chosen": -1441.941650390625,
255
+ "logps/rejected": -1688.218994140625,
256
+ "loss": 0.6584,
257
+ "rewards/accuracies": 0.6000000238418579,
258
+ "rewards/chosen": -0.3497202694416046,
259
+ "rewards/margins": 0.10849998146295547,
260
+ "rewards/rejected": -0.4582202434539795,
261
+ "step": 170
262
+ },
263
+ {
264
+ "epoch": 0.58,
265
+ "learning_rate": 2.2759017277414165e-06,
266
+ "logits/chosen": 0.6102200150489807,
267
+ "logits/rejected": 0.7705193758010864,
268
+ "logps/chosen": -1447.1510009765625,
269
+ "logps/rejected": -1748.801513671875,
270
+ "loss": 0.6584,
271
+ "rewards/accuracies": 0.53125,
272
+ "rewards/chosen": -0.40625113248825073,
273
+ "rewards/margins": 0.17269271612167358,
274
+ "rewards/rejected": -0.5789438486099243,
275
+ "step": 180
276
+ },
277
+ {
278
+ "epoch": 0.61,
279
+ "learning_rate": 1.9985264605418185e-06,
280
+ "logits/chosen": 0.7250276803970337,
281
+ "logits/rejected": 0.779090940952301,
282
+ "logps/chosen": -1608.396484375,
283
+ "logps/rejected": -1889.056640625,
284
+ "loss": 0.6531,
285
+ "rewards/accuracies": 0.574999988079071,
286
+ "rewards/chosen": -0.5311942100524902,
287
+ "rewards/margins": 0.14003470540046692,
288
+ "rewards/rejected": -0.6712288856506348,
289
+ "step": 190
290
+ },
291
+ {
292
+ "epoch": 0.64,
293
+ "learning_rate": 1.7274575140626318e-06,
294
+ "logits/chosen": 0.6869342923164368,
295
+ "logits/rejected": 0.7512118220329285,
296
+ "logps/chosen": -1676.6634521484375,
297
+ "logps/rejected": -1916.425048828125,
298
+ "loss": 0.656,
299
+ "rewards/accuracies": 0.543749988079071,
300
+ "rewards/chosen": -0.5880906581878662,
301
+ "rewards/margins": 0.13318488001823425,
302
+ "rewards/rejected": -0.7212755680084229,
303
+ "step": 200
304
+ },
305
+ {
306
+ "epoch": 0.67,
307
+ "learning_rate": 1.466103737583699e-06,
308
+ "logits/chosen": 0.6197313070297241,
309
+ "logits/rejected": 0.7475897073745728,
310
+ "logps/chosen": -1537.823974609375,
311
+ "logps/rejected": -1862.232421875,
312
+ "loss": 0.6564,
313
+ "rewards/accuracies": 0.5562499761581421,
314
+ "rewards/chosen": -0.4764162600040436,
315
+ "rewards/margins": 0.189828023314476,
316
+ "rewards/rejected": -0.6662443280220032,
317
+ "step": 210
318
+ },
319
+ {
320
+ "epoch": 0.7,
321
+ "learning_rate": 1.217751806485235e-06,
322
+ "logits/chosen": 0.7158914804458618,
323
+ "logits/rejected": 0.8779485821723938,
324
+ "logps/chosen": -1476.162841796875,
325
+ "logps/rejected": -1704.2161865234375,
326
+ "loss": 0.655,
327
+ "rewards/accuracies": 0.4937500059604645,
328
+ "rewards/chosen": -0.4569624066352844,
329
+ "rewards/margins": 0.09420967102050781,
330
+ "rewards/rejected": -0.5511720180511475,
331
+ "step": 220
332
+ },
333
+ {
334
+ "epoch": 0.74,
335
+ "learning_rate": 9.855248903979505e-07,
336
+ "logits/chosen": 0.6375827193260193,
337
+ "logits/rejected": 0.7420727014541626,
338
+ "logps/chosen": -1576.697509765625,
339
+ "logps/rejected": -1821.4849853515625,
340
+ "loss": 0.6593,
341
+ "rewards/accuracies": 0.606249988079071,
342
+ "rewards/chosen": -0.4542946219444275,
343
+ "rewards/margins": 0.15300381183624268,
344
+ "rewards/rejected": -0.6072984933853149,
345
+ "step": 230
346
+ },
347
+ {
348
+ "epoch": 0.77,
349
+ "learning_rate": 7.723433775328385e-07,
350
+ "logits/chosen": 0.6124081015586853,
351
+ "logits/rejected": 0.7668013572692871,
352
+ "logps/chosen": -1591.124267578125,
353
+ "logps/rejected": -1752.868408203125,
354
+ "loss": 0.6579,
355
+ "rewards/accuracies": 0.543749988079071,
356
+ "rewards/chosen": -0.45945438742637634,
357
+ "rewards/margins": 0.08051095902919769,
358
+ "rewards/rejected": -0.5399652719497681,
359
+ "step": 240
360
+ },
361
+ {
362
+ "epoch": 0.8,
363
+ "learning_rate": 5.808881491049723e-07,
364
+ "logits/chosen": 0.6094003915786743,
365
+ "logits/rejected": 0.7548435926437378,
366
+ "logps/chosen": -1563.33203125,
367
+ "logps/rejected": -1689.942626953125,
368
+ "loss": 0.6653,
369
+ "rewards/accuracies": 0.5,
370
+ "rewards/chosen": -0.48421138525009155,
371
+ "rewards/margins": 0.04744899272918701,
372
+ "rewards/rejected": -0.5316604375839233,
373
+ "step": 250
374
+ },
375
+ {
376
+ "epoch": 0.83,
377
+ "learning_rate": 4.1356686569674344e-07,
378
+ "logits/chosen": 0.6028648614883423,
379
+ "logits/rejected": 0.711211085319519,
380
+ "logps/chosen": -1524.207763671875,
381
+ "logps/rejected": -1821.697509765625,
382
+ "loss": 0.6525,
383
+ "rewards/accuracies": 0.612500011920929,
384
+ "rewards/chosen": -0.4109669625759125,
385
+ "rewards/margins": 0.14401301741600037,
386
+ "rewards/rejected": -0.5549799799919128,
387
+ "step": 260
388
+ },
389
+ {
390
+ "epoch": 0.86,
391
+ "learning_rate": 2.7248368952908055e-07,
392
+ "logits/chosen": 0.5701503753662109,
393
+ "logits/rejected": 0.771028459072113,
394
+ "logps/chosen": -1587.4259033203125,
395
+ "logps/rejected": -1885.8216552734375,
396
+ "loss": 0.6525,
397
+ "rewards/accuracies": 0.581250011920929,
398
+ "rewards/chosen": -0.4386199116706848,
399
+ "rewards/margins": 0.14451175928115845,
400
+ "rewards/rejected": -0.5831316709518433,
401
+ "step": 270
402
+ },
403
+ {
404
+ "epoch": 0.9,
405
+ "learning_rate": 1.59412823400657e-07,
406
+ "logits/chosen": 0.5914828181266785,
407
+ "logits/rejected": 0.6638094186782837,
408
+ "logps/chosen": -1495.4044189453125,
409
+ "logps/rejected": -1771.0677490234375,
410
+ "loss": 0.6518,
411
+ "rewards/accuracies": 0.574999988079071,
412
+ "rewards/chosen": -0.4058058261871338,
413
+ "rewards/margins": 0.1175435408949852,
414
+ "rewards/rejected": -0.5233493447303772,
415
+ "step": 280
416
+ },
417
+ {
418
+ "epoch": 0.93,
419
+ "learning_rate": 7.577619905828281e-08,
420
+ "logits/chosen": 0.6334878206253052,
421
+ "logits/rejected": 0.724925696849823,
422
+ "logps/chosen": -1534.744384765625,
423
+ "logps/rejected": -1730.7095947265625,
424
+ "loss": 0.6648,
425
+ "rewards/accuracies": 0.53125,
426
+ "rewards/chosen": -0.44486203789711,
427
+ "rewards/margins": 0.0799860954284668,
428
+ "rewards/rejected": -0.5248481035232544,
429
+ "step": 290
430
+ },
431
+ {
432
+ "epoch": 0.96,
433
+ "learning_rate": 2.262559558016325e-08,
434
+ "logits/chosen": 0.5741454362869263,
435
+ "logits/rejected": 0.7404450178146362,
436
+ "logps/chosen": -1426.1407470703125,
437
+ "logps/rejected": -1655.0103759765625,
438
+ "loss": 0.6569,
439
+ "rewards/accuracies": 0.512499988079071,
440
+ "rewards/chosen": -0.3900262713432312,
441
+ "rewards/margins": 0.10863729566335678,
442
+ "rewards/rejected": -0.4986635744571686,
443
+ "step": 300
444
+ },
445
+ {
446
+ "epoch": 0.99,
447
+ "learning_rate": 6.294126437336734e-10,
448
+ "logits/chosen": 0.603456437587738,
449
+ "logits/rejected": 0.8570040464401245,
450
+ "logps/chosen": -1530.1478271484375,
451
+ "logps/rejected": -1771.2337646484375,
452
+ "loss": 0.6513,
453
+ "rewards/accuracies": 0.5375000238418579,
454
+ "rewards/chosen": -0.4091600775718689,
455
+ "rewards/margins": 0.11743543297052383,
456
+ "rewards/rejected": -0.5265954732894897,
457
+ "step": 310
458
+ },
459
+ {
460
+ "epoch": 1.0,
461
+ "step": 312,
462
+ "total_flos": 0.0,
463
+ "train_loss": 0.6651641822014099,
464
+ "train_runtime": 4241.3739,
465
+ "train_samples_per_second": 4.715,
466
+ "train_steps_per_second": 0.074
467
+ }
468
+ ],
469
+ "logging_steps": 10,
470
+ "max_steps": 312,
471
+ "num_input_tokens_seen": 0,
472
+ "num_train_epochs": 1,
473
+ "save_steps": 20,
474
+ "total_flos": 0.0,
475
+ "train_batch_size": 4,
476
+ "trial_name": null,
477
+ "trial_params": null
478
+ }