lmind_nq_train6000_eval6489_v1_docidx_v3_Qwen_Qwen1.5-4B_5e-5_lora2

This model is a fine-tuned version of Qwen/Qwen1.5-4B on the tyzhu/lmind_nq_train6000_eval6489_v1_docidx_v3 dataset. It achieves the following results on the evaluation set:

• Loss: 5.0355
• Accuracy: 0.4273

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 1
• eval_batch_size: 2
• seed: 42
• distributed_type: multi-GPU
• num_devices: 4
• gradient_accumulation_steps: 8
• total_train_batch_size: 32
• total_eval_batch_size: 8
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: constant
• lr_scheduler_warmup_ratio: 0.05
• num_epochs: 20.0

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
1.9626	0.9985	341	2.9919	0.4727
1.9158	2.0	683	2.9864	0.4737
1.8622	2.9985	1024	3.0420	0.4710
1.786	4.0	1366	3.1527	0.4662
1.7019	4.9985	1707	3.3819	0.4634
1.6036	6.0	2049	3.4969	0.4589
1.5175	6.9985	2390	3.6412	0.4577
1.4007	8.0	2732	3.8310	0.4537
1.326	8.9985	3073	3.9177	0.4487
1.231	10.0	3415	4.0665	0.4451
1.1298	10.9985	3756	4.1773	0.44
1.0276	12.0	4098	4.2875	0.4378
0.9525	12.9985	4439	4.4273	0.4352
0.8616	14.0	4781	4.4484	0.4324
0.7799	14.9985	5122	4.6228	0.4313
0.7084	16.0	5464	4.7239	0.4303
0.6478	16.9985	5805	4.8167	0.4310
0.5862	18.0	6147	4.8510	0.4303
0.5189	18.9985	6488	4.9265	0.4243
0.4767	19.9707	6820	5.0355	0.4273

Framework versions

• PEFT 0.5.0
• Transformers 4.40.2
• Pytorch 2.3.0
• Datasets 2.19.1
• Tokenizers 0.19.1