lmind_hotpot_train8000_eval7405_v1_docidx_Qwen_Qwen1.5-4B_3e-5_lora2

This model is a fine-tuned version of Qwen/Qwen1.5-4B on the tyzhu/lmind_hotpot_train8000_eval7405_v1_docidx dataset. It achieves the following results on the evaluation set:

• Loss: 1.3258
• Accuracy: 0.7513

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 3e-05
• train_batch_size: 1
• eval_batch_size: 2
• seed: 42
• distributed_type: multi-GPU
• num_devices: 4
• gradient_accumulation_steps: 8
• total_train_batch_size: 32
• total_eval_batch_size: 8
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: constant
• lr_scheduler_warmup_ratio: 0.05
• num_epochs: 20.0

Training results

Training Loss	Epoch	Step	Accuracy	Validation Loss
1.6137	0.9997	839	0.7177	1.8264
1.5978	1.9994	1678	0.7190	1.8264
1.5493	2.9991	2517	0.7211	1.7810
1.5101	4.0	3357	0.7231	1.7413
1.4678	4.9997	4196	0.7247	1.7400
1.4236	5.9994	5035	0.7267	1.7019
1.3843	6.9991	5874	0.7286	1.6725
1.3481	8.0	6714	0.7304	1.6381
1.2954	8.9997	7553	0.7324	1.6103
1.2426	9.9994	8392	0.7338	1.5785
1.2169	10.9991	9231	0.7355	1.5435
1.167	12.0	10071	0.7375	1.5216
1.1276	12.9997	10910	0.7392	1.4949
1.0819	13.9994	11749	0.7406	1.4819
1.032	14.9991	12588	0.7426	1.4468
0.9981	15.9997	13424	1.4092	0.7443
0.9523	16.9994	14263	1.3949	0.7463
0.9281	17.9991	15102	1.3853	0.7477
0.8664	19.0	15942	1.3669	0.7496
0.8537	19.9985	16780	1.3258	0.7513

Framework versions

• PEFT 0.5.0
• Transformers 4.40.2
• Pytorch 2.3.0
• Datasets 2.19.1
• Tokenizers 0.19.1