llama_3_sky_o1_5_full

This model is a fine-tuned version of meta-llama/Llama-3.1-8B-Instruct on the generator dataset. It achieves the following results on the evaluation set:

• Loss: 0.5866

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 4
• eval_batch_size: 4
• seed: 42
• distributed_type: multi-GPU
• num_devices: 4
• gradient_accumulation_steps: 2
• total_train_batch_size: 32
• total_eval_batch_size: 16
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss
0.8116	0.0487	100	0.6762
0.727	0.0975	200	0.6912
0.7451	0.1462	300	0.6847
0.7268	0.1950	400	0.6761
0.7208	0.2437	500	0.6729
0.7045	0.2925	600	0.6669
0.6849	0.3412	700	0.6605
0.67	0.3900	800	0.6498
0.658	0.4387	900	0.6423
0.6511	0.4874	1000	0.6345
0.6408	0.5362	1100	0.6282
0.6283	0.5849	1200	0.6191
0.6178	0.6337	1300	0.6100
0.5971	0.6824	1400	0.6036
0.5904	0.7312	1500	0.6008
0.5859	0.7799	1600	0.5947
0.5758	0.8287	1700	0.5912
0.5766	0.8774	1800	0.5888
0.578	0.9262	1900	0.5874
0.5716	0.9749	2000	0.5866

Framework versions

• Transformers 4.44.2
• Pytorch 2.4.1+cu121
• Datasets 3.0.0
• Tokenizers 0.19.1