OpenELM-1_1B-DPO-full-max-min-reward

This model was trained from scratch on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 1.2624
• Rewards/chosen: -5.875
• Rewards/rejected: -6.0938
• Rewards/accuracies: 0.4707
• Rewards/margins: 0.2383
• Logps/rejected: -900.0
• Logps/chosen: -904.0
• Logits/rejected: -13.5625
• Logits/chosen: -14.0

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 8
• eval_batch_size: 16
• seed: 42
• distributed_type: multi-GPU
• num_devices: 4
• gradient_accumulation_steps: 2
• total_train_batch_size: 64
• total_eval_batch_size: 64
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 3

Training results

Training Loss	Epoch	Step	Validation Loss	Rewards/chosen	Rewards/rejected	Rewards/accuracies	Rewards/margins	Logps/rejected	Logps/chosen	Logits/rejected	Logits/chosen
0.054	0.1047	100	0.7392	-1.7969	-2.0781	0.5410	0.2754	-496.0	-498.0	-15.125	-15.1875
0.0654	0.2094	200	0.8200	-1.8828	-2.1094	0.5137	0.2285	-500.0	-506.0	-16.25	-16.375
0.0359	0.3141	300	0.9338	-2.6406	-2.9531	0.5020	0.3027	-584.0	-584.0	-13.875	-14.0
0.0593	0.4188	400	0.8586	-2.2812	-2.5156	0.5312	0.2324	-540.0	-548.0	-14.8125	-15.0
0.0262	0.5236	500	1.0656	-3.0	-3.2344	0.4844	0.2354	-612.0	-620.0	-16.0	-16.125
0.0525	0.6283	600	0.9800	-2.6406	-2.8281	0.4941	0.1777	-572.0	-584.0	-13.5625	-13.8125
0.064	0.7330	700	1.0007	-3.3906	-3.5156	0.4980	0.1211	-640.0	-660.0	-15.0625	-15.1875
0.0251	0.8377	800	1.0387	-3.875	-3.9375	0.4824	0.0598	-680.0	-704.0	-12.8125	-13.25
0.0443	0.9424	900	1.0605	-4.5312	-4.5938	0.4531	0.0466	-748.0	-772.0	-13.5625	-13.9375
0.0024	1.0471	1000	1.2371	-4.5	-4.75	0.4727	0.2373	-764.0	-768.0	-13.25	-13.625
0.0028	1.1518	1100	1.1591	-3.9219	-4.0312	0.4551	0.1089	-692.0	-708.0	-14.0625	-14.375
0.007	1.2565	1200	1.1814	-4.2188	-4.3438	0.4629	0.1328	-724.0	-740.0	-14.0625	-14.4375
0.0022	1.3613	1300	1.1827	-4.125	-4.2812	0.4785	0.1523	-716.0	-732.0	-13.875	-14.25
0.0036	1.4660	1400	1.2144	-4.75	-4.9688	0.4863	0.2314	-784.0	-792.0	-14.1875	-14.5625
0.0011	1.5707	1500	1.2473	-4.75	-4.9688	0.4863	0.2002	-784.0	-792.0	-13.9375	-14.375
0.0019	1.6754	1600	1.2159	-5.5	-5.75	0.4785	0.2539	-864.0	-868.0	-13.0	-13.5625
0.002	1.7801	1700	1.2082	-5.3438	-5.5625	0.4727	0.2275	-844.0	-852.0	-13.8125	-14.25
0.0025	1.8848	1800	1.1580	-4.7188	-4.9062	0.4746	0.1846	-780.0	-792.0	-13.375	-13.8125
0.007	1.9895	1900	1.1403	-4.8438	-4.9688	0.4766	0.1523	-788.0	-800.0	-13.0625	-13.4375
0.0002	2.0942	2000	1.1499	-5.0	-5.125	0.4805	0.1416	-804.0	-816.0	-13.125	-13.5625
0.0271	2.1990	2100	1.1933	-5.1562	-5.3438	0.4863	0.1641	-820.0	-836.0	-13.3125	-13.6875
0.0003	2.3037	2200	1.2642	-5.7188	-5.9688	0.4844	0.2441	-888.0	-892.0	-13.3125	-13.75
0.0004	2.4084	2300	1.2548	-5.7188	-5.9375	0.4805	0.2432	-884.0	-888.0	-13.4375	-13.8125
0.0003	2.5131	2400	1.2491	-5.7188	-5.9688	0.4746	0.2441	-888.0	-892.0	-13.5625	-14.0
0.0005	2.6178	2500	1.2546	-5.7812	-6.0312	0.4727	0.2432	-892.0	-896.0	-13.625	-14.0
0.0001	2.7225	2600	1.2598	-5.8438	-6.0938	0.4727	0.2383	-896.0	-904.0	-13.5625	-14.0
0.0002	2.8272	2700	1.2617	-5.875	-6.0938	0.4746	0.2354	-900.0	-904.0	-13.5625	-14.0
0.0002	2.9319	2800	1.2624	-5.875	-6.0938	0.4707	0.2383	-900.0	-904.0	-13.5625	-14.0

Framework versions

• Transformers 4.44.2
• Pytorch 2.3.0
• Datasets 2.21.0
• Tokenizers 0.19.1