SmolLM-360M-Instruct-dpo-16k

This model is a fine-tuned version of HuggingFaceTB/SmolLM-360M-Instruct on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.8873
• Rewards/chosen: 0.0047
• Rewards/rejected: 0.3539
• Rewards/accuracies: 0.0326
• Rewards/margins: -0.3493
• Logps/rejected: -470.7575
• Logps/chosen: -546.0133
• Logits/rejected: 0.3014
• Logits/chosen: 0.6045

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-06
• train_batch_size: 2
• eval_batch_size: 2
• seed: 42
• gradient_accumulation_steps: 2
• total_train_batch_size: 4
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 2
• num_epochs: 6

Training results

Training Loss	Epoch	Step	Validation Loss	Rewards/chosen	Rewards/rejected	Rewards/accuracies	Rewards/margins	Logps/rejected	Logps/chosen	Logits/rejected	Logits/chosen
0.5225	0.9999	3368	0.8679	0.0092	0.3258	0.0337	-0.3166	-471.0385	-545.9679	0.3212	0.6250
0.4511	2.0	6737	0.8863	0.0171	0.3649	0.0283	-0.3477	-470.6477	-545.8885	0.2889	0.5939
0.4453	2.9999	10105	0.8880	0.0006	0.3516	0.0304	-0.3510	-470.7807	-546.0537	0.3259	0.6291
0.4439	4.0	13474	0.8894	0.0067	0.3598	0.0228	-0.3531	-470.6990	-545.9932	0.2699	0.5815
0.4441	4.9999	16842	0.8881	0.0058	0.3569	0.0293	-0.3511	-470.7278	-546.0020	0.2999	0.6028
0.4442	5.9991	20208	0.8873	0.0047	0.3539	0.0326	-0.3493	-470.7575	-546.0133	0.3014	0.6045

Framework versions

• Transformers 4.41.0
• Pytorch 2.2.0
• Datasets 2.19.1
• Tokenizers 0.19.1