Hugging Face's logo

AmberYifan
/
Llama-2-7b-Mistral-7B-v0.1-mix

Text Generation
Transformers
Safetensors
llama
Generated from Trainer
trl
dpo
conversational
text-generation-inference
Inference Endpoints
Model card Files Community
AmberYifan commited on
Commit
1b2e71c
·
verified ·
1 Parent(s): f65c8f2

Model save

Browse files
Files changed (5) hide show
  1. README.md +68 -0
  2. all_results.json +9 -0
  3. generation_config.json +10 -0
  4. train_results.json +9 -0
  5. trainer_state.json +375 -0
README.md ADDED
@@ -0,0 +1,68 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ base_model: AmberYifan/llama2-7b-sft-ultrachat-safeRLHF
3
+ library_name: transformers
4
+ model_name: Llama-2-7b-Mistral-7B-v0.1-mix
5
+ tags:
6
+ - generated_from_trainer
7
+ - trl
8
+ - dpo
9
+ licence: license
10
+ ---
11
+
12
+ # Model Card for Llama-2-7b-Mistral-7B-v0.1-mix
13
+
14
+ This model is a fine-tuned version of [AmberYifan/llama2-7b-sft-ultrachat-safeRLHF](https://huggingface.co/AmberYifan/llama2-7b-sft-ultrachat-safeRLHF).
15
+ It has been trained using [TRL](https://github.com/huggingface/trl).
16
+
17
+ ## Quick start
18
+
19
+ ```python
20
+ from transformers import pipeline
21
+
22
+ question = "If you had a time machine, but could only go to the past or the future once and never return, which would you choose and why?"
23
+ generator = pipeline("text-generation", model="AmberYifan/Llama-2-7b-Mistral-7B-v0.1-mix", device="cuda")
24
+ output = generator([{"role": "user", "content": question}], max_new_tokens=128, return_full_text=False)[0]
25
+ print(output["generated_text"])
26
+ ```
27
+
28
+ ## Training procedure
29
+
30
+ [<img src="https://raw.githubusercontent.com/wandb/assets/main/wandb-github-badge-28.svg" alt="Visualize in Weights & Biases" width="150" height="24"/>](https://wandb.ai/yifanwang/huggingface/runs/kknqnvw3)
31
+
32
+ This model was trained with DPO, a method introduced in [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://huggingface.co/papers/2305.18290).
33
+
34
+ ### Framework versions
35
+
36
+ - TRL: 0.12.2
37
+ - Transformers: 4.46.3
38
+ - Pytorch: 2.5.1+cu118
39
+ - Datasets: 3.2.0
40
+ - Tokenizers: 0.20.3
41
+
42
+ ## Citations
43
+
44
+ Cite DPO as:
45
+
46
+ ```bibtex
47
+ @inproceedings{rafailov2023direct,
48
+ title = {{Direct Preference Optimization: Your Language Model is Secretly a Reward Model}},
49
+ author = {Rafael Rafailov and Archit Sharma and Eric Mitchell and Christopher D. Manning and Stefano Ermon and Chelsea Finn},
50
+ year = 2023,
51
+ booktitle = {Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023},
52
+ url = {http://papers.nips.cc/paper_files/paper/2023/hash/a85b405ed65c6477a4fe8302b5e06ce7-Abstract-Conference.html},
53
+ editor = {Alice Oh and Tristan Naumann and Amir Globerson and Kate Saenko and Moritz Hardt and Sergey Levine},
54
+ }
55
+ ```
56
+
57
+ Cite TRL as:
58
+
59
+ ```bibtex
60
+ @misc{vonwerra2022trl,
61
+ title = {{TRL: Transformer Reinforcement Learning}},
62
+ author = {Leandro von Werra and Younes Belkada and Lewis Tunstall and Edward Beeching and Tristan Thrush and Nathan Lambert and Shengyi Huang and Kashif Rasul and Quentin Gallouédec},
63
+ year = 2020,
64
+ journal = {GitHub repository},
65
+ publisher = {GitHub},
66
+ howpublished = {\url{https://github.com/huggingface/trl}}
67
+ }
68
+ ```
all_results.json ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "epoch": 3.0,
3
+ "total_flos": 0.0,
4
+ "train_loss": 0.5234840029761905,
5
+ "train_runtime": 1680.5719,
6
+ "train_samples": 2000,
7
+ "train_samples_per_second": 3.57,
8
+ "train_steps_per_second": 0.112
9
+ }
generation_config.json ADDED
@@ -0,0 +1,10 @@
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token_id": 1,
3
+ "do_sample": true,
4
+ "eos_token_id": 2,
5
+ "max_length": 4096,
6
+ "pad_token_id": 0,
7
+ "temperature": 0.6,
8
+ "top_p": 0.9,
9
+ "transformers_version": "4.46.3"
10
+ }
train_results.json ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "epoch": 3.0,
3
+ "total_flos": 0.0,
4
+ "train_loss": 0.5234840029761905,
5
+ "train_runtime": 1680.5719,
6
+ "train_samples": 2000,
7
+ "train_samples_per_second": 3.57,
8
+ "train_steps_per_second": 0.112
9
+ }
trainer_state.json ADDED
@@ -0,0 +1,375 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "best_metric": null,
3
+ "best_model_checkpoint": null,
4
+ "epoch": 3.0,
5
+ "eval_steps": 500,
6
+ "global_step": 189,
7
+ "is_hyper_param_search": false,
8
+ "is_local_process_zero": true,
9
+ "is_world_process_zero": true,
10
+ "log_history": [
11
+ {
12
+ "epoch": 0.015873015873015872,
13
+ "grad_norm": 15.546492141333783,
14
+ "learning_rate": 2.6315789473684208e-08,
15
+ "logits/chosen": -1.4921875,
16
+ "logits/rejected": -1.3828125,
17
+ "logps/chosen": -83.0,
18
+ "logps/rejected": -108.5,
19
+ "loss": 0.6914,
20
+ "rewards/accuracies": 0.0,
21
+ "rewards/chosen": 0.0,
22
+ "rewards/margins": 0.0,
23
+ "rewards/rejected": 0.0,
24
+ "step": 1
25
+ },
26
+ {
27
+ "epoch": 0.15873015873015872,
28
+ "grad_norm": 14.774196452895438,
29
+ "learning_rate": 2.631578947368421e-07,
30
+ "logits/chosen": -1.2734375,
31
+ "logits/rejected": -1.1953125,
32
+ "logps/chosen": -107.5,
33
+ "logps/rejected": -90.0,
34
+ "loss": 0.6918,
35
+ "rewards/accuracies": 0.2222222238779068,
36
+ "rewards/chosen": -0.0020751953125,
37
+ "rewards/margins": -0.00139617919921875,
38
+ "rewards/rejected": -0.00069427490234375,
39
+ "step": 10
40
+ },
41
+ {
42
+ "epoch": 0.31746031746031744,
43
+ "grad_norm": 14.3389141291959,
44
+ "learning_rate": 4.970588235294118e-07,
45
+ "logits/chosen": -1.28125,
46
+ "logits/rejected": -1.2578125,
47
+ "logps/chosen": -87.0,
48
+ "logps/rejected": -102.5,
49
+ "loss": 0.6842,
50
+ "rewards/accuracies": 0.512499988079071,
51
+ "rewards/chosen": 0.0015411376953125,
52
+ "rewards/margins": 0.021240234375,
53
+ "rewards/rejected": -0.019775390625,
54
+ "step": 20
55
+ },
56
+ {
57
+ "epoch": 0.47619047619047616,
58
+ "grad_norm": 14.661136683478352,
59
+ "learning_rate": 4.676470588235294e-07,
60
+ "logits/chosen": -1.3125,
61
+ "logits/rejected": -1.21875,
62
+ "logps/chosen": -94.5,
63
+ "logps/rejected": -100.0,
64
+ "loss": 0.6687,
65
+ "rewards/accuracies": 0.5874999761581421,
66
+ "rewards/chosen": 0.036865234375,
67
+ "rewards/margins": 0.07470703125,
68
+ "rewards/rejected": -0.037841796875,
69
+ "step": 30
70
+ },
71
+ {
72
+ "epoch": 0.6349206349206349,
73
+ "grad_norm": 15.043075717803145,
74
+ "learning_rate": 4.38235294117647e-07,
75
+ "logits/chosen": -1.3359375,
76
+ "logits/rejected": -1.4140625,
77
+ "logps/chosen": -94.0,
78
+ "logps/rejected": -112.5,
79
+ "loss": 0.6159,
80
+ "rewards/accuracies": 0.612500011920929,
81
+ "rewards/chosen": 0.06005859375,
82
+ "rewards/margins": 0.1669921875,
83
+ "rewards/rejected": -0.10693359375,
84
+ "step": 40
85
+ },
86
+ {
87
+ "epoch": 0.7936507936507936,
88
+ "grad_norm": 20.286017694579517,
89
+ "learning_rate": 4.0882352941176465e-07,
90
+ "logits/chosen": -1.359375,
91
+ "logits/rejected": -1.3515625,
92
+ "logps/chosen": -103.0,
93
+ "logps/rejected": -115.0,
94
+ "loss": 0.6568,
95
+ "rewards/accuracies": 0.612500011920929,
96
+ "rewards/chosen": -0.00506591796875,
97
+ "rewards/margins": 0.2099609375,
98
+ "rewards/rejected": -0.2158203125,
99
+ "step": 50
100
+ },
101
+ {
102
+ "epoch": 0.9523809523809523,
103
+ "grad_norm": 15.604577240602536,
104
+ "learning_rate": 3.7941176470588235e-07,
105
+ "logits/chosen": -1.3984375,
106
+ "logits/rejected": -1.421875,
107
+ "logps/chosen": -98.0,
108
+ "logps/rejected": -117.0,
109
+ "loss": 0.6149,
110
+ "rewards/accuracies": 0.6499999761581421,
111
+ "rewards/chosen": 0.0322265625,
112
+ "rewards/margins": 0.1865234375,
113
+ "rewards/rejected": -0.154296875,
114
+ "step": 60
115
+ },
116
+ {
117
+ "epoch": 1.0,
118
+ "eval_logits/chosen": -1.3359375,
119
+ "eval_logits/rejected": -1.3046875,
120
+ "eval_logps/chosen": -91.0,
121
+ "eval_logps/rejected": -111.5,
122
+ "eval_loss": 0.6164844036102295,
123
+ "eval_rewards/accuracies": 0.7142857313156128,
124
+ "eval_rewards/chosen": 0.007598876953125,
125
+ "eval_rewards/margins": 0.224609375,
126
+ "eval_rewards/rejected": -0.2177734375,
127
+ "eval_runtime": 12.239,
128
+ "eval_samples_per_second": 16.341,
129
+ "eval_steps_per_second": 0.572,
130
+ "step": 63
131
+ },
132
+ {
133
+ "epoch": 1.1111111111111112,
134
+ "grad_norm": 13.062522185911641,
135
+ "learning_rate": 3.5e-07,
136
+ "logits/chosen": -1.375,
137
+ "logits/rejected": -1.375,
138
+ "logps/chosen": -93.0,
139
+ "logps/rejected": -108.0,
140
+ "loss": 0.5617,
141
+ "rewards/accuracies": 0.7250000238418579,
142
+ "rewards/chosen": 0.1533203125,
143
+ "rewards/margins": 0.375,
144
+ "rewards/rejected": -0.220703125,
145
+ "step": 70
146
+ },
147
+ {
148
+ "epoch": 1.2698412698412698,
149
+ "grad_norm": 12.793133064546527,
150
+ "learning_rate": 3.205882352941177e-07,
151
+ "logits/chosen": -1.2421875,
152
+ "logits/rejected": -1.359375,
153
+ "logps/chosen": -82.5,
154
+ "logps/rejected": -112.5,
155
+ "loss": 0.5127,
156
+ "rewards/accuracies": 0.7749999761581421,
157
+ "rewards/chosen": 0.1455078125,
158
+ "rewards/margins": 0.515625,
159
+ "rewards/rejected": -0.37109375,
160
+ "step": 80
161
+ },
162
+ {
163
+ "epoch": 1.4285714285714286,
164
+ "grad_norm": 13.067935081505256,
165
+ "learning_rate": 2.911764705882353e-07,
166
+ "logits/chosen": -1.484375,
167
+ "logits/rejected": -1.3203125,
168
+ "logps/chosen": -100.0,
169
+ "logps/rejected": -98.0,
170
+ "loss": 0.5069,
171
+ "rewards/accuracies": 0.7875000238418579,
172
+ "rewards/chosen": 0.283203125,
173
+ "rewards/margins": 0.5390625,
174
+ "rewards/rejected": -0.2578125,
175
+ "step": 90
176
+ },
177
+ {
178
+ "epoch": 1.5873015873015874,
179
+ "grad_norm": 15.317610443936815,
180
+ "learning_rate": 2.6176470588235295e-07,
181
+ "logits/chosen": -1.3671875,
182
+ "logits/rejected": -1.375,
183
+ "logps/chosen": -90.0,
184
+ "logps/rejected": -111.0,
185
+ "loss": 0.4966,
186
+ "rewards/accuracies": 0.762499988079071,
187
+ "rewards/chosen": 0.2353515625,
188
+ "rewards/margins": 0.5546875,
189
+ "rewards/rejected": -0.318359375,
190
+ "step": 100
191
+ },
192
+ {
193
+ "epoch": 1.746031746031746,
194
+ "grad_norm": 12.447441760307086,
195
+ "learning_rate": 2.323529411764706e-07,
196
+ "logits/chosen": -1.4296875,
197
+ "logits/rejected": -1.296875,
198
+ "logps/chosen": -97.0,
199
+ "logps/rejected": -115.0,
200
+ "loss": 0.4934,
201
+ "rewards/accuracies": 0.887499988079071,
202
+ "rewards/chosen": 0.1435546875,
203
+ "rewards/margins": 0.56640625,
204
+ "rewards/rejected": -0.423828125,
205
+ "step": 110
206
+ },
207
+ {
208
+ "epoch": 1.9047619047619047,
209
+ "grad_norm": 13.19848987425151,
210
+ "learning_rate": 2.0294117647058823e-07,
211
+ "logits/chosen": -1.34375,
212
+ "logits/rejected": -1.34375,
213
+ "logps/chosen": -94.0,
214
+ "logps/rejected": -117.5,
215
+ "loss": 0.4805,
216
+ "rewards/accuracies": 0.8500000238418579,
217
+ "rewards/chosen": 0.10205078125,
218
+ "rewards/margins": 0.60546875,
219
+ "rewards/rejected": -0.50390625,
220
+ "step": 120
221
+ },
222
+ {
223
+ "epoch": 2.0,
224
+ "eval_logits/chosen": -1.34375,
225
+ "eval_logits/rejected": -1.328125,
226
+ "eval_logps/chosen": -92.0,
227
+ "eval_logps/rejected": -113.0,
228
+ "eval_loss": 0.5889843702316284,
229
+ "eval_rewards/accuracies": 0.6428571343421936,
230
+ "eval_rewards/chosen": -0.0751953125,
231
+ "eval_rewards/margins": 0.31640625,
232
+ "eval_rewards/rejected": -0.392578125,
233
+ "eval_runtime": 14.7924,
234
+ "eval_samples_per_second": 13.52,
235
+ "eval_steps_per_second": 0.473,
236
+ "step": 126
237
+ },
238
+ {
239
+ "epoch": 2.0634920634920633,
240
+ "grad_norm": 10.98673276740895,
241
+ "learning_rate": 1.7352941176470587e-07,
242
+ "logits/chosen": -1.390625,
243
+ "logits/rejected": -1.3984375,
244
+ "logps/chosen": -101.0,
245
+ "logps/rejected": -116.0,
246
+ "loss": 0.4562,
247
+ "rewards/accuracies": 0.862500011920929,
248
+ "rewards/chosen": 0.193359375,
249
+ "rewards/margins": 0.67578125,
250
+ "rewards/rejected": -0.482421875,
251
+ "step": 130
252
+ },
253
+ {
254
+ "epoch": 2.2222222222222223,
255
+ "grad_norm": 9.416416295176981,
256
+ "learning_rate": 1.441176470588235e-07,
257
+ "logits/chosen": -1.4140625,
258
+ "logits/rejected": -1.3828125,
259
+ "logps/chosen": -100.5,
260
+ "logps/rejected": -107.5,
261
+ "loss": 0.4104,
262
+ "rewards/accuracies": 0.862500011920929,
263
+ "rewards/chosen": 0.2099609375,
264
+ "rewards/margins": 0.76171875,
265
+ "rewards/rejected": -0.55078125,
266
+ "step": 140
267
+ },
268
+ {
269
+ "epoch": 2.380952380952381,
270
+ "grad_norm": 11.858508121730686,
271
+ "learning_rate": 1.1470588235294116e-07,
272
+ "logits/chosen": -1.3671875,
273
+ "logits/rejected": -1.3515625,
274
+ "logps/chosen": -93.5,
275
+ "logps/rejected": -113.5,
276
+ "loss": 0.413,
277
+ "rewards/accuracies": 0.8999999761581421,
278
+ "rewards/chosen": 0.2177734375,
279
+ "rewards/margins": 0.7265625,
280
+ "rewards/rejected": -0.51171875,
281
+ "step": 150
282
+ },
283
+ {
284
+ "epoch": 2.5396825396825395,
285
+ "grad_norm": 9.556697117699082,
286
+ "learning_rate": 8.529411764705883e-08,
287
+ "logits/chosen": -1.296875,
288
+ "logits/rejected": -1.4296875,
289
+ "logps/chosen": -90.5,
290
+ "logps/rejected": -108.5,
291
+ "loss": 0.4203,
292
+ "rewards/accuracies": 0.800000011920929,
293
+ "rewards/chosen": 0.1337890625,
294
+ "rewards/margins": 0.82421875,
295
+ "rewards/rejected": -0.69140625,
296
+ "step": 160
297
+ },
298
+ {
299
+ "epoch": 2.6984126984126986,
300
+ "grad_norm": 10.065550572065934,
301
+ "learning_rate": 5.5882352941176474e-08,
302
+ "logits/chosen": -1.3671875,
303
+ "logits/rejected": -1.3671875,
304
+ "logps/chosen": -98.5,
305
+ "logps/rejected": -109.5,
306
+ "loss": 0.4442,
307
+ "rewards/accuracies": 0.7749999761581421,
308
+ "rewards/chosen": 0.09814453125,
309
+ "rewards/margins": 0.734375,
310
+ "rewards/rejected": -0.63671875,
311
+ "step": 170
312
+ },
313
+ {
314
+ "epoch": 2.857142857142857,
315
+ "grad_norm": 11.316830865980743,
316
+ "learning_rate": 2.6470588235294116e-08,
317
+ "logits/chosen": -1.3515625,
318
+ "logits/rejected": -1.34375,
319
+ "logps/chosen": -102.0,
320
+ "logps/rejected": -104.5,
321
+ "loss": 0.4264,
322
+ "rewards/accuracies": 0.762499988079071,
323
+ "rewards/chosen": 0.10498046875,
324
+ "rewards/margins": 0.6328125,
325
+ "rewards/rejected": -0.52734375,
326
+ "step": 180
327
+ },
328
+ {
329
+ "epoch": 3.0,
330
+ "eval_logits/chosen": -1.3515625,
331
+ "eval_logits/rejected": -1.3359375,
332
+ "eval_logps/chosen": -92.5,
333
+ "eval_logps/rejected": -114.0,
334
+ "eval_loss": 0.5811718702316284,
335
+ "eval_rewards/accuracies": 0.6428571343421936,
336
+ "eval_rewards/chosen": -0.1357421875,
337
+ "eval_rewards/margins": 0.34765625,
338
+ "eval_rewards/rejected": -0.484375,
339
+ "eval_runtime": 14.7749,
340
+ "eval_samples_per_second": 13.537,
341
+ "eval_steps_per_second": 0.474,
342
+ "step": 189
343
+ },
344
+ {
345
+ "epoch": 3.0,
346
+ "step": 189,
347
+ "total_flos": 0.0,
348
+ "train_loss": 0.5234840029761905,
349
+ "train_runtime": 1680.5719,
350
+ "train_samples_per_second": 3.57,
351
+ "train_steps_per_second": 0.112
352
+ }
353
+ ],
354
+ "logging_steps": 10,
355
+ "max_steps": 189,
356
+ "num_input_tokens_seen": 0,
357
+ "num_train_epochs": 3,
358
+ "save_steps": 500,
359
+ "stateful_callbacks": {
360
+ "TrainerControl": {
361
+ "args": {
362
+ "should_epoch_stop": false,
363
+ "should_evaluate": false,
364
+ "should_log": false,
365
+ "should_save": true,
366
+ "should_training_stop": true
367
+ },
368
+ "attributes": {}
369
+ }
370
+ },
371
+ "total_flos": 0.0,
372
+ "train_batch_size": 8,
373
+ "trial_name": null,
374
+ "trial_params": null
375
+ }