DNADebertaK6f

This model is a fine-tuned version of on the None dataset. It achieves the following results on the evaluation set:

• Loss: 1.3707

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 64
• eval_batch_size: 64
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 150
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
2.1545	1.24	100000	1.5895
1.5727	2.49	200000	1.5383
1.5368	3.73	300000	1.5117
1.5143	4.97	400000	1.4926
1.4969	6.22	500000	1.4789
1.4841	7.46	600000	1.4677
1.475	8.7	700000	1.4599
1.4677	9.95	800000	1.4533
1.4623	11.19	900000	1.4491
1.4576	12.43	1000000	1.4461
1.4544	13.68	1100000	1.4420
1.451	14.92	1200000	1.4381
1.4482	16.16	1300000	1.4359
1.446	17.4	1400000	1.4345
1.443	18.65	1500000	1.4320
1.4412	19.89	1600000	1.4295
1.4385	21.13	1700000	1.4278
1.4368	22.38	1800000	1.4249
1.4346	23.62	1900000	1.4237
1.433	24.86	2000000	1.4219
1.4315	26.11	2100000	1.4201
1.4297	27.35	2200000	1.4198
1.4282	28.59	2300000	1.4180
1.4266	29.84	2400000	1.4142
1.4253	31.08	2500000	1.4146
1.4238	32.32	2600000	1.4130
1.4228	33.57	2700000	1.4113
1.4221	34.81	2800000	1.4100
1.42	36.05	2900000	1.4097
1.4188	37.3	3000000	1.4085
1.4174	38.54	3100000	1.4067
1.4161	39.78	3200000	1.4064
1.4149	41.03	3300000	1.4058
1.4139	42.27	3400000	1.4024
1.4134	43.51	3500000	1.4022
1.4126	44.76	3600000	1.4025
1.4117	46.0	3700000	1.4015
1.411	47.24	3800000	1.4001
1.4098	48.49	3900000	1.3968
1.4096	49.73	4000000	1.3997
1.4089	50.97	4100000	1.3974
1.4084	52.21	4200000	1.3972
1.4072	53.46	4300000	1.3965
1.4066	54.7	4400000	1.3974
1.4062	55.94	4500000	1.3960
1.4058	57.19	4600000	1.3958
1.4053	58.43	4700000	1.3950
1.4041	59.67	4800000	1.3936
1.4041	60.92	4900000	1.3963
1.4031	62.16	5000000	1.3915
1.4023	63.4	5100000	1.3917
1.4022	64.65	5200000	1.3930
1.4017	65.89	5300000	1.3904
1.4009	67.13	5400000	1.3899
1.4007	68.38	5500000	1.3892
1.3997	69.62	5600000	1.3910
1.3996	70.86	5700000	1.3892
1.3991	72.11	5800000	1.3890
1.3983	73.35	5900000	1.3870
1.3985	74.59	6000000	1.3889
1.3975	75.84	6100000	1.3865
1.3973	77.08	6200000	1.3852
1.3969	78.32	6300000	1.3869
1.3964	79.57	6400000	1.3843
1.396	80.81	6500000	1.3853
1.3955	82.05	6600000	1.3844
1.3952	83.3	6700000	1.3863
1.395	84.54	6800000	1.3835
1.3948	85.78	6900000	1.3841
1.394	87.02	7000000	1.3850
1.3934	88.27	7100000	1.3827
1.3932	89.51	7200000	1.3830
1.3929	90.75	7300000	1.3821
1.392	92.0	7400000	1.3820
1.392	93.24	7500000	1.3837
1.3913	94.48	7600000	1.3817
1.3909	95.73	7700000	1.3836
1.3906	96.97	7800000	1.3811
1.3903	98.21	7900000	1.3806
1.3902	99.46	8000000	1.3807
1.3896	100.7	8100000	1.3804
1.3895	101.94	8200000	1.3805
1.3891	103.19	8300000	1.3821
1.3889	104.43	8400000	1.3833
1.3881	105.67	8500000	1.3788
1.388	106.92	8600000	1.3818
1.3876	108.16	8700000	1.3806
1.387	109.4	8800000	1.3766
1.387	110.65	8900000	1.3765
1.3865	111.89	9000000	1.3800
1.3864	113.13	9100000	1.3830
1.386	114.38	9200000	1.3770
1.3853	115.62	9300000	1.3771
1.3852	116.86	9400000	1.3772
1.3848	118.1	9500000	1.3771
1.384	119.35	9600000	1.3749
1.3843	120.59	9700000	1.3764
1.3836	121.83	9800000	1.3802
1.3833	123.08	9900000	1.3756
1.3831	124.32	10000000	1.3748
1.3821	125.56	10100000	1.3755
1.3817	126.81	10200000	1.3744
1.3819	128.05	10300000	1.3763
1.381	129.29	10400000	1.3743
1.3805	130.54	10500000	1.3762
1.3804	131.78	10600000	1.3725
1.3795	133.02	10700000	1.3753
1.3791	134.27	10800000	1.3780
1.3785	135.51	10900000	1.3749
1.3781	136.75	11000000	1.3749
1.3779	138.0	11100000	1.3737
1.3772	139.24	11200000	1.3715
1.3763	140.48	11300000	1.3759
1.3761	141.73	11400000	1.3745
1.3752	142.97	11500000	1.3708
1.3744	144.21	11600000	1.3735
1.3736	145.46	11700000	1.3720
1.3728	146.7	11800000	1.3702
1.3714	147.94	11900000	1.3696
1.3706	149.19	12000000	1.3707

Framework versions

• Transformers 4.27.3
• Pytorch 2.0.0
• Datasets 2.10.1
• Tokenizers 0.13.2