Undi95/Llama2-13B-no_robots-alpaca-lora

no_robots-alpaca

This lora was trained with Doctor-Shotgun/no-robots-sharegpt dataset on TheBloke/Llama-2-13B-fp16. It achieves the following results on the evaluation set:

• Loss: 1.6087

Model description

The LoRA was trained on Doctor-Shotgun/no-robots-sharegpt, a ShareGPT converted dataset from the OG HuggingFaceH4/no_robots but with Alpaca prompting.

Prompt template: Alpaca

Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
{prompt}

### Response:

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.00065
• train_batch_size: 2
• eval_batch_size: 2
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: constant
• lr_scheduler_warmup_steps: 10
• num_epochs: 2

Training results

Training Loss	Epoch	Step	Validation Loss
1.5523	0.0	1	1.5476
1.2139	0.1	42	1.5008
1.6348	0.2	84	1.4968
1.6498	0.3	126	1.4962
1.5645	0.4	168	1.4983
1.6487	0.5	210	1.4981
1.6147	0.6	252	1.4965
1.3048	0.7	294	1.4973
1.6205	0.8	336	1.5007
1.6045	0.9	378	1.5003
1.5781	1.0	420	1.5013
1.4807	1.09	462	1.5492
1.0541	1.19	504	1.5596
1.2337	1.29	546	1.5789
0.9719	1.39	588	1.5859
1.2189	1.49	630	1.5959
1.2566	1.59	672	1.5968
0.7049	1.69	714	1.5987
1.2133	1.79	756	1.5907
1.0327	1.89	798	1.6087

Framework versions

• Transformers 4.34.1
• Pytorch 2.0.1+cu117
• Datasets 2.14.6
• Tokenizers 0.14.1

If you want to support me, you can here.

Open LLM Leaderboard Evaluation Results

Detailed results can be found here

Metric	Value
Avg.	46.55
ARC (25-shot)	58.87
HellaSwag (10-shot)	82.43
MMLU (5-shot)	53.11
TruthfulQA (0-shot)	40.46
Winogrande (5-shot)	75.3
GSM8K (5-shot)	6.44
DROP (3-shot)	9.26