lmind_nq_train6000_eval6489_v1_docidx_v3_Qwen_Qwen1.5-4B_3e-5_lora2

This model is a fine-tuned version of Qwen/Qwen1.5-4B on the tyzhu/lmind_nq_train6000_eval6489_v1_docidx_v3 dataset. It achieves the following results on the evaluation set:

• Loss: 4.3717
• Accuracy: 0.4408

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 3e-05
• train_batch_size: 1
• eval_batch_size: 2
• seed: 42
• distributed_type: multi-GPU
• num_devices: 4
• gradient_accumulation_steps: 8
• total_train_batch_size: 32
• total_eval_batch_size: 8
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: constant
• lr_scheduler_warmup_ratio: 0.05
• num_epochs: 20.0

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
1.9685	0.9985	341	3.0537	0.4626
1.9337	2.0	683	2.9928	0.4709
1.9029	2.9985	1024	3.0243	0.4705
1.856	4.0	1366	3.0766	0.4697
1.8019	4.9985	1707	3.1923	0.4696
1.7406	6.0	2049	3.2573	0.4684
1.6974	6.9985	2390	3.3286	0.4672
1.6249	8.0	2732	3.4775	0.4647
1.5993	8.9985	3073	3.5378	0.4636
1.5449	10.0	3415	3.6347	0.4597
1.4855	10.9985	3756	3.6955	0.4553
1.4205	12.0	4098	3.8478	0.4479
1.3757	12.9985	4439	3.9185	0.4487
1.3098	14.0	4781	3.9575	0.4455
1.2574	14.9985	5122	4.1279	0.4457
1.2049	16.0	5464	4.1540	0.4448
1.1617	16.9985	5805	4.2049	0.4454
1.1046	18.0	6147	4.2909	0.4432
1.043	18.9985	6488	4.3535	0.4385
1.0044	19.9707	6820	4.3717	0.4408

Framework versions

• PEFT 0.5.0
• Transformers 4.40.2
• Pytorch 2.3.0
• Datasets 2.19.1
• Tokenizers 0.19.1