qwen_cfUNL_entropy_0_01

This model is a fine-tuned version of trl-lib/qwen1.5-0.5b-sft on the yakazimir/ultrafeedback_binarized dataset. It achieves the following results on the evaluation set:

• Loss: 0.0501
• Sft Loss: 3.9427
• Rewards/chosen: -4.3435
• Rewards/rejected: -5.1114
• Rewards/accuracies: 0.6810
• Rewards/margins: 0.7679
• Logps/rejected: -5.1114
• Logps/chosen: -4.3435
• Logits/rejected: -0.0604
• Logits/chosen: -0.1374

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-06
• train_batch_size: 2
• eval_batch_size: 4
• seed: 42
• distributed_type: multi-GPU
• gradient_accumulation_steps: 16
• total_train_batch_size: 32
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 3.0

Training results

Training Loss	Epoch	Step	Validation Loss	Sft Loss	Rewards/chosen	Rewards/rejected	Rewards/accuracies	Rewards/margins	Logps/rejected	Logps/chosen	Logits/rejected	Logits/chosen
0.0548	0.2141	400	0.0564	4.2256	-4.8288	-5.0421	0.5378	0.2133	-5.0421	-4.8288	0.4440	0.3255
0.0531	0.4282	800	0.0526	4.0511	-4.5660	-4.9392	0.6157	0.3732	-4.9392	-4.5660	0.2541	0.1294
0.0534	0.6422	1200	0.0519	4.1663	-4.5650	-5.0390	0.6387	0.4740	-5.0390	-4.5650	0.2502	0.1373
0.0511	0.8563	1600	0.0513	3.9593	-4.4389	-4.9222	0.6358	0.4833	-4.9222	-4.4389	-0.0640	-0.1524
0.0533	1.0704	2000	0.0509	3.9533	-4.4316	-4.9577	0.6484	0.5261	-4.9577	-4.4316	-0.0257	-0.1111
0.0527	1.2845	2400	0.0508	4.2818	-4.7129	-5.3738	0.6610	0.6609	-5.3738	-4.7129	-0.0551	-0.1386
0.0513	1.4986	2800	0.0506	4.1502	-4.4933	-5.1357	0.6818	0.6424	-5.1357	-4.4933	-0.1729	-0.2577
0.0527	1.7127	3200	0.0505	4.1082	-4.4722	-5.1175	0.6743	0.6453	-5.1175	-4.4722	-0.0614	-0.1521
0.0538	1.9267	3600	0.0502	4.0026	-4.3928	-5.1056	0.6706	0.7129	-5.1056	-4.3928	-0.1185	-0.1939
0.0495	2.1408	4000	0.0502	4.0304	-4.4251	-5.1723	0.6825	0.7472	-5.1723	-4.4251	-0.0488	-0.1284
0.0522	2.3549	4400	0.0501	3.9711	-4.3751	-5.1111	0.6751	0.7360	-5.1111	-4.3751	-0.1449	-0.2170
0.0517	2.5690	4800	0.0501	4.0093	-4.3976	-5.1429	0.6832	0.7452	-5.1429	-4.3976	-0.0508	-0.1310
0.0496	2.7831	5200	0.0501	3.9605	-4.3494	-5.1084	0.6788	0.7590	-5.1084	-4.3494	-0.0700	-0.1475
0.0497	2.9972	5600	0.0501	3.9427	-4.3435	-5.1114	0.6810	0.7679	-5.1114	-4.3435	-0.0604	-0.1374

Framework versions

• Transformers 4.44.2
• Pytorch 2.2.2+cu121
• Datasets 2.18.0
• Tokenizers 0.19.1