code-llama-sparql-lcquad-large-dbpedia

This model is a fine-tuned version of codellama/CodeLlama-7b-hf on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.2040

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0003
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 2
• total_train_batch_size: 16
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 100
• training_steps: 400
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
1.3087	0.0184	20	1.3391
1.0131	0.0368	40	0.7246
0.5023	0.0553	60	0.4406
0.4287	0.0737	80	0.3738
0.3432	0.0921	100	0.3530
0.2725	0.1105	120	0.3329
0.2539	0.1290	140	0.2888
0.2847	0.1474	160	0.2762
0.253	0.1658	180	0.2598
0.2168	0.1842	200	0.2480
0.2194	0.2027	220	0.2516
0.2123	0.2211	240	0.2300
0.2366	0.2395	260	0.2180
0.2129	0.2579	280	0.2181
0.2073	0.2764	300	0.2154
0.1992	0.2948	320	0.2086
0.2086	0.3132	340	0.2069
0.2237	0.3316	360	0.2056
0.1982	0.3501	380	0.2043
0.2039	0.3685	400	0.2040

Framework versions

• Transformers 4.41.2
• Pytorch 2.3.0+cu121
• Datasets 2.10.1
• Tokenizers 0.19.1