roberta-base_IA3_lr0.0005_bs4_epoch20_wd0.01

This model is a fine-tuned version of roberta-base on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 1.3406

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0005
• train_batch_size: 4
• eval_batch_size: 8
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 500
• num_epochs: 20

Training results

Training Loss	Epoch	Step	Validation Loss
19.5038	1.0	157	18.4857
8.9314	2.0	314	5.8495
5.4493	3.0	471	4.2791
3.8453	4.0	628	3.4174
3.4001	5.0	785	2.8767
2.8518	6.0	942	2.5189
2.7181	7.0	1099	2.2672
2.3938	8.0	1256	2.0897
2.2025	9.0	1413	1.9660
2.1035	10.0	1570	1.8055
1.9748	11.0	1727	1.6968
1.8698	12.0	1884	1.6367
1.7843	13.0	2041	1.5600
1.7277	14.0	2198	1.5018
1.6915	15.0	2355	1.4518
1.5865	16.0	2512	1.4089
1.5934	17.0	2669	1.3896
1.5713	18.0	2826	1.3617
1.5521	19.0	2983	1.3453
1.5471	20.0	3140	1.3406

Framework versions

• PEFT 0.7.1
• Transformers 4.36.2
• Pytorch 2.0.1
• Datasets 2.16.1
• Tokenizers 0.15.0