roberta-base_PrefixTuning_lr5e-05_bs4_epoch20_wd0.01

This model is a fine-tuned version of roberta-base on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 14.4579

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 4
• eval_batch_size: 8
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 500
• num_epochs: 20

Training results

Training Loss	Epoch	Step	Validation Loss
15.4862	1.0	157	20.8478
14.9721	2.0	314	20.7345
15.0553	3.0	471	20.5177
14.4627	4.0	628	20.2344
14.0788	5.0	785	19.9480
13.2092	6.0	942	19.6334
13.0141	7.0	1099	19.2349
12.5052	8.0	1256	18.7395
11.8852	9.0	1413	18.2110
11.889	10.0	1570	17.7415
11.2937	11.0	1727	17.3058
11.0163	12.0	1884	16.8580
10.8939	13.0	2041	16.3684
10.8183	14.0	2198	15.9196
10.353	15.0	2355	15.5043
10.1386	16.0	2512	15.1272
9.9726	17.0	2669	14.8365
9.8421	18.0	2826	14.6183
9.9193	19.0	2983	14.4978
9.848	20.0	3140	14.4579

Framework versions

• PEFT 0.7.1
• Transformers 4.36.2
• Pytorch 2.0.1
• Datasets 2.16.1
• Tokenizers 0.15.0