roberta-base_PrefixTuning_lr0.0005_bs4_epoch20_wd0.01

This model is a fine-tuned version of roberta-base on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 3.9612

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0005
• train_batch_size: 4
• eval_batch_size: 8
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 500
• num_epochs: 20

Training results

Training Loss	Epoch	Step	Validation Loss
16.1454	1.0	157	20.9147
14.0712	2.0	314	19.1650
11.1417	3.0	471	9.2979
6.9892	4.0	628	6.3310
6.4327	5.0	785	5.8392
5.9068	6.0	942	5.5211
5.7042	7.0	1099	5.2767
5.399	8.0	1256	5.0556
5.117	9.0	1413	4.8469
5.0361	10.0	1570	4.6365
4.8558	11.0	1727	4.4569
4.6969	12.0	1884	4.3400
4.5535	13.0	2041	4.2330
4.522	14.0	2198	4.1549
4.3703	15.0	2355	4.1031
4.305	16.0	2512	4.0450
4.3117	17.0	2669	4.0057
4.2937	18.0	2826	3.9820
4.2778	19.0	2983	3.9654
4.2304	20.0	3140	3.9612

Framework versions

• PEFT 0.7.1
• Transformers 4.36.2
• Pytorch 2.0.1
• Datasets 2.16.1
• Tokenizers 0.15.0