lmind_nq_train6000_eval6489_v1_qa_3e-4_lora2

This model is a fine-tuned version of meta-llama/Llama-2-7b-hf on the tyzhu/lmind_nq_train6000_eval6489_v1_qa dataset. It achieves the following results on the evaluation set:

• Loss: 3.0231
• Accuracy: 0.4910

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0003
• train_batch_size: 2
• eval_batch_size: 2
• seed: 42
• distributed_type: multi-GPU
• num_devices: 4
• gradient_accumulation_steps: 4
• total_train_batch_size: 32
• total_eval_batch_size: 8
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: constant
• lr_scheduler_warmup_ratio: 0.05
• num_epochs: 50.0

Training results

Training Loss	Epoch	Step	Accuracy	Validation Loss
1.4433	1.0	187	0.6166	1.2626
1.0761	2.0	375	0.6159	1.2960
0.7909	3.0	562	0.6094	1.4208
0.6117	4.0	750	0.6065	1.5099
0.535	5.0	937	0.6073	1.6384
0.4917	6.0	1125	0.6064	1.6918
0.4746	7.0	1312	0.6048	1.7547
0.463	8.0	1500	0.6057	1.7708
0.4307	9.0	1687	0.6051	1.7773
0.4323	10.0	1875	0.5997	1.7927
0.4301	11.0	2062	0.6046	1.8201
0.4331	12.0	2250	0.6051	1.8903
0.4338	13.0	2437	0.6035	1.8489
0.4331	14.0	2625	0.6024	1.9058
0.4362	15.0	2812	0.6004	1.8960
0.4371	16.0	3000	0.6012	1.9080
0.4153	17.0	3187	0.6019	1.9090
0.4151	18.0	3375	0.6004	1.9131
0.426	19.0	3562	0.5750	2.0016
0.4475	20.0	3750	0.6010	1.8873
0.4325	21.0	3937	0.6009	1.8351
0.4542	22.0	4125	0.5990	1.8659
0.5662	23.0	4312	0.5964	1.8427
0.4365	24.0	4500	0.5993	1.8093
0.4153	25.0	4687	0.6022	1.8418
0.4099	26.0	4875	0.6002	1.9742
0.4101	27.0	5062	0.5987	1.9682
0.4122	28.0	5250	0.5991	1.9906
0.4116	29.0	5437	0.5990	1.9428
0.4158	30.0	5625	0.5987	1.9262
0.4215	31.0	5812	0.5961	1.9735
0.4249	32.0	6000	0.5966	1.9393
0.4049	33.0	6187	0.5956	2.0083
0.4077	34.0	6375	0.5962	1.9472
0.4078	35.0	6562	0.5945	1.9796
2.7734	36.0	6750	0.3345	5.2338
4.2638	37.0	6937	0.4333	3.7370
2.5456	38.0	7125	0.3550	4.5389
5.3244	39.0	7312	0.4132	4.0601
3.9209	40.0	7500	0.4102	4.0942
3.4851	41.0	7687	0.5124	2.5265
1.3821	42.0	7875	0.5595	2.0728
0.7913	43.0	8062	0.5667	2.0104
0.6674	44.0	8250	0.5754	2.0220
0.6202	45.0	8437	0.5837	2.0235
0.4798	46.0	8625	0.5816	2.0670
0.4663	47.0	8812	0.5756	2.0748
4.502	48.0	9000	0.4479	3.4224
2.7893	49.0	9187	0.5287	2.3122
1.8211	49.87	9350	0.4910	3.0231

Framework versions

• Transformers 4.34.0
• Pytorch 2.1.0+cu121
• Datasets 2.18.0
• Tokenizers 0.14.1