Progen2_Kinase_PhosphositeGen_dkz_trainwithunlabeled

This model is a fine-tuned version of hugohrban/progen2-base on the None dataset. It achieves the following results on the evaluation set:

• Loss: 2.2967
• Perplexity: 9.9412

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0005
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 2
• total_train_batch_size: 16
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• training_steps: 5000

Training results

Training Loss	Epoch	Step	Validation Loss	Perplexity
4.763	0.0905	100	2.1876	8.9135
4.6193	0.1811	200	2.9409	18.9328
5.017	0.2716	300	2.4134	11.1717
4.7964	0.3622	400	2.3927	10.9434
4.7938	0.4527	500	2.3926	10.9421
4.7905	0.5432	600	2.3904	10.9178
4.7763	0.6338	700	2.3880	10.8919
4.7542	0.7243	800	2.3813	10.8193
4.763	0.8148	900	2.3772	10.7748
4.7519	0.9054	1000	2.3704	10.7016
4.7497	0.9959	1100	2.3618	10.6099
4.7085	1.0860	1200	2.3547	10.5355
4.7148	1.1766	1300	2.3564	10.5527
4.6749	1.2671	1400	2.3437	10.4194
4.6825	1.3576	1500	2.3395	10.3756
4.668	1.4482	1600	2.3389	10.3699
4.6826	1.5387	1700	2.3344	10.3232
4.6505	1.6292	1800	2.3321	10.3000
4.6549	1.7198	1900	2.3267	10.2445
4.6448	1.8103	2000	2.3268	10.2450
4.6368	1.9009	2100	2.3274	10.2516
4.6255	1.9914	2200	2.3308	10.2862
4.6042	2.0815	2300	2.3210	10.1855
4.6325	2.1720	2400	2.3211	10.1865
4.6238	2.2626	2500	2.3210	10.1856
4.6272	2.3531	2600	2.3153	10.1279
4.6027	2.4436	2700	2.3154	10.1287
4.6121	2.5342	2800	2.3122	10.0962
4.6061	2.6247	2900	2.3110	10.0841
4.6195	2.7153	3000	2.3131	10.1058
4.6046	2.8058	3100	2.3089	10.0634
4.6049	2.8963	3200	2.3133	10.1074
4.6221	2.9869	3300	2.3119	10.0932
4.5677	3.0770	3400	2.3087	10.0615
4.5952	3.1675	3500	2.3085	10.0590
4.5809	3.2580	3600	2.3084	10.0582
4.5803	3.3486	3700	2.3076	10.0507
4.5857	3.4391	3800	2.3077	10.0512
4.585	3.5297	3900	2.3073	10.0476
4.5868	3.6202	4000	2.3032	10.0060
4.5978	3.7107	4100	2.3047	10.0213
4.5732	3.8013	4200	2.3017	9.9909
4.5759	3.8918	4300	2.3021	9.9951
4.5808	3.9823	4400	2.3005	9.9793
4.5465	4.0724	4500	2.3014	9.9880
4.5563	4.1630	4600	2.3004	9.9784
4.5592	4.2535	4700	2.2992	9.9659
4.5596	4.3440	4800	2.2979	9.9532
4.5683	4.4346	4900	2.2969	9.9432
4.5703	4.5251	5000	2.2967	9.9412

Framework versions

• PEFT 0.13.2
• Transformers 4.47.1
• Pytorch 2.1.0.post301
• Datasets 3.0.2
• Tokenizers 0.21.0