---
library_name: peft
license: bsd-3-clause
base_model: hugohrban/progen2-base
tags:
- generated_from_trainer
model-index:
- name: Progen2_Kinase_PhosphositeGen_dkz_trainwithunlabeled
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# Progen2_Kinase_PhosphositeGen_dkz_trainwithunlabeled

This model is a fine-tuned version of [hugohrban/progen2-base](https://huggingface.co/hugohrban/progen2-base) on the None dataset.
It achieves the following results on the evaluation set:
- Loss: 2.2967
- Perplexity: 9.9412

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 0.0005
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- gradient_accumulation_steps: 2
- total_train_batch_size: 16
- optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
- lr_scheduler_type: linear
- training_steps: 5000

### Training results

| Training Loss | Epoch  | Step | Validation Loss | Perplexity |
|:-------------:|:------:|:----:|:---------------:|:----------:|
| 4.763         | 0.0905 | 100  | 2.1876          | 8.9135     |
| 4.6193        | 0.1811 | 200  | 2.9409          | 18.9328    |
| 5.017         | 0.2716 | 300  | 2.4134          | 11.1717    |
| 4.7964        | 0.3622 | 400  | 2.3927          | 10.9434    |
| 4.7938        | 0.4527 | 500  | 2.3926          | 10.9421    |
| 4.7905        | 0.5432 | 600  | 2.3904          | 10.9178    |
| 4.7763        | 0.6338 | 700  | 2.3880          | 10.8919    |
| 4.7542        | 0.7243 | 800  | 2.3813          | 10.8193    |
| 4.763         | 0.8148 | 900  | 2.3772          | 10.7748    |
| 4.7519        | 0.9054 | 1000 | 2.3704          | 10.7016    |
| 4.7497        | 0.9959 | 1100 | 2.3618          | 10.6099    |
| 4.7085        | 1.0860 | 1200 | 2.3547          | 10.5355    |
| 4.7148        | 1.1766 | 1300 | 2.3564          | 10.5527    |
| 4.6749        | 1.2671 | 1400 | 2.3437          | 10.4194    |
| 4.6825        | 1.3576 | 1500 | 2.3395          | 10.3756    |
| 4.668         | 1.4482 | 1600 | 2.3389          | 10.3699    |
| 4.6826        | 1.5387 | 1700 | 2.3344          | 10.3232    |
| 4.6505        | 1.6292 | 1800 | 2.3321          | 10.3000    |
| 4.6549        | 1.7198 | 1900 | 2.3267          | 10.2445    |
| 4.6448        | 1.8103 | 2000 | 2.3268          | 10.2450    |
| 4.6368        | 1.9009 | 2100 | 2.3274          | 10.2516    |
| 4.6255        | 1.9914 | 2200 | 2.3308          | 10.2862    |
| 4.6042        | 2.0815 | 2300 | 2.3210          | 10.1855    |
| 4.6325        | 2.1720 | 2400 | 2.3211          | 10.1865    |
| 4.6238        | 2.2626 | 2500 | 2.3210          | 10.1856    |
| 4.6272        | 2.3531 | 2600 | 2.3153          | 10.1279    |
| 4.6027        | 2.4436 | 2700 | 2.3154          | 10.1287    |
| 4.6121        | 2.5342 | 2800 | 2.3122          | 10.0962    |
| 4.6061        | 2.6247 | 2900 | 2.3110          | 10.0841    |
| 4.6195        | 2.7153 | 3000 | 2.3131          | 10.1058    |
| 4.6046        | 2.8058 | 3100 | 2.3089          | 10.0634    |
| 4.6049        | 2.8963 | 3200 | 2.3133          | 10.1074    |
| 4.6221        | 2.9869 | 3300 | 2.3119          | 10.0932    |
| 4.5677        | 3.0770 | 3400 | 2.3087          | 10.0615    |
| 4.5952        | 3.1675 | 3500 | 2.3085          | 10.0590    |
| 4.5809        | 3.2580 | 3600 | 2.3084          | 10.0582    |
| 4.5803        | 3.3486 | 3700 | 2.3076          | 10.0507    |
| 4.5857        | 3.4391 | 3800 | 2.3077          | 10.0512    |
| 4.585         | 3.5297 | 3900 | 2.3073          | 10.0476    |
| 4.5868        | 3.6202 | 4000 | 2.3032          | 10.0060    |
| 4.5978        | 3.7107 | 4100 | 2.3047          | 10.0213    |
| 4.5732        | 3.8013 | 4200 | 2.3017          | 9.9909     |
| 4.5759        | 3.8918 | 4300 | 2.3021          | 9.9951     |
| 4.5808        | 3.9823 | 4400 | 2.3005          | 9.9793     |
| 4.5465        | 4.0724 | 4500 | 2.3014          | 9.9880     |
| 4.5563        | 4.1630 | 4600 | 2.3004          | 9.9784     |
| 4.5592        | 4.2535 | 4700 | 2.2992          | 9.9659     |
| 4.5596        | 4.3440 | 4800 | 2.2979          | 9.9532     |
| 4.5683        | 4.4346 | 4900 | 2.2969          | 9.9432     |
| 4.5703        | 4.5251 | 5000 | 2.2967          | 9.9412     |


### Framework versions

- PEFT 0.13.2
- Transformers 4.47.1
- Pytorch 2.1.0.post301
- Datasets 3.0.2
- Tokenizers 0.21.0