MSc_llama2_finetuned_model_secondData2

This model is a fine-tuned version of meta-llama/Llama-2-7b-chat-hf on the None dataset. It achieves the following results on the evaluation set:

• Loss: 0.7297

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

The following bitsandbytes quantization config was used during training:

• quant_method: bitsandbytes
• _load_in_8bit: False
• _load_in_4bit: True
• llm_int8_threshold: 6.0
• llm_int8_skip_modules: None
• llm_int8_enable_fp32_cpu_offload: False
• llm_int8_has_fp16_weight: False
• bnb_4bit_quant_type: nf4
• bnb_4bit_use_double_quant: True
• bnb_4bit_compute_dtype: bfloat16
• load_in_4bit: True
• load_in_8bit: False

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 16
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 64
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.03
• training_steps: 250

Training results

Training Loss	Epoch	Step	Validation Loss
3.9191	1.33	10	3.4583
2.9769	2.67	20	2.3588
1.9753	4.0	30	1.6427
1.4615	5.33	40	1.1789
0.9839	6.67	50	0.8645
0.8058	8.0	60	0.7883
0.7209	9.33	70	0.7324
0.656	10.67	80	0.7005
0.5988	12.0	90	0.6821
0.5534	13.33	100	0.6761
0.5203	14.67	110	0.6735
0.4912	16.0	120	0.6733
0.4643	17.33	130	0.6757
0.4432	18.67	140	0.6820
0.4201	20.0	150	0.6891
0.4062	21.33	160	0.6964
0.3915	22.67	170	0.7051
0.3803	24.0	180	0.7066
0.371	25.33	190	0.7133
0.3612	26.67	200	0.7240
0.3587	28.0	210	0.7262
0.3558	29.33	220	0.7276
0.354	30.67	230	0.7279
0.3498	32.0	240	0.7291
0.3516	33.33	250	0.7297

Framework versions

• PEFT 0.4.0
• Transformers 4.38.2
• Pytorch 2.3.1+cu121
• Datasets 2.13.1
• Tokenizers 0.15.2