---
library_name: peft
license: bsd-3-clause
base_model: hugohrban/progen2-base
tags:
- generated_from_trainer
model-index:
- name: Progen2_Kinase_PhosphositeGen
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# Progen2_Kinase_PhosphositeGen

This model is a fine-tuned version of [hugohrban/progen2-base](https://huggingface.co/hugohrban/progen2-base) on the None dataset.
It achieves the following results on the evaluation set:
- Loss: 2.0025
- Perplexity: 7.4078

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 0.0005
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- gradient_accumulation_steps: 2
- total_train_batch_size: 16
- optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
- lr_scheduler_type: linear
- training_steps: 5000

### Training results

| Training Loss | Epoch  | Step | Validation Loss | Perplexity |
|:-------------:|:------:|:----:|:---------------:|:----------:|
| 4.9303        | 0.1415 | 100  | 2.1811          | 8.8564     |
| 4.2874        | 0.2831 | 200  | 2.1198          | 8.3296     |
| 4.1941        | 0.4246 | 300  | 2.0694          | 7.9202     |
| 4.0803        | 0.5662 | 400  | 2.0362          | 7.6616     |
| 4.0613        | 0.7077 | 500  | 2.0053          | 7.4284     |
| 3.9511        | 0.8493 | 600  | 1.9922          | 7.3315     |
| 3.9216        | 0.9908 | 700  | 1.9477          | 7.0124     |
| 3.5053        | 1.1316 | 800  | 1.9200          | 6.8208     |
| 3.4311        | 1.2732 | 900  | 1.9035          | 6.7094     |
| 3.4238        | 1.4147 | 1000 | 1.8714          | 6.4974     |
| 3.392         | 1.5563 | 1100 | 1.8527          | 6.3772     |
| 3.3621        | 1.6978 | 1200 | 1.8317          | 6.2444     |
| 3.3577        | 1.8393 | 1300 | 1.8237          | 6.1945     |
| 3.3419        | 1.9809 | 1400 | 1.7889          | 5.9826     |
| 2.8256        | 2.1217 | 1500 | 1.7977          | 6.0356     |
| 2.8061        | 2.2633 | 1600 | 1.7860          | 5.9653     |
| 2.7837        | 2.4048 | 1700 | 1.7666          | 5.8507     |
| 2.7504        | 2.5464 | 1800 | 1.7428          | 5.7133     |
| 2.829         | 2.6879 | 1900 | 1.7288          | 5.6337     |
| 2.7567        | 2.8294 | 2000 | 1.7088          | 5.5225     |
| 2.7443        | 2.9710 | 2100 | 1.6986          | 5.4664     |
| 2.3409        | 3.1118 | 2200 | 1.7382          | 5.6869     |
| 2.2568        | 3.2534 | 2300 | 1.7487          | 5.7471     |
| 2.2481        | 3.3949 | 2400 | 1.7181          | 5.5740     |
| 2.2323        | 3.5364 | 2500 | 1.7058          | 5.5059     |
| 2.2654        | 3.6780 | 2600 | 1.7031          | 5.4912     |
| 2.2611        | 3.8195 | 2700 | 1.6707          | 5.3157     |
| 2.256         | 3.9611 | 2800 | 1.6719          | 5.3222     |
| 1.8849        | 4.1019 | 2900 | 1.7899          | 5.9886     |
| 1.771         | 4.2435 | 3000 | 1.7697          | 5.8694     |
| 1.7992        | 4.3850 | 3100 | 1.7880          | 5.9775     |
| 1.838         | 4.5265 | 3200 | 1.7871          | 5.9722     |
| 1.8285        | 4.6681 | 3300 | 1.7342          | 5.6644     |
| 1.8127        | 4.8096 | 3400 | 1.7196          | 5.5825     |
| 1.8353        | 4.9512 | 3500 | 1.7471          | 5.7377     |
| 1.5511        | 5.0920 | 3600 | 1.8285          | 6.2248     |
| 1.4449        | 5.2335 | 3700 | 1.8683          | 6.4770     |
| 1.4631        | 5.3751 | 3800 | 1.8880          | 6.6063     |
| 1.4525        | 5.5166 | 3900 | 1.8807          | 6.5581     |
| 1.4516        | 5.6582 | 4000 | 1.8723          | 6.5031     |
| 1.4423        | 5.7997 | 4100 | 1.8828          | 6.5716     |
| 1.4626        | 5.9413 | 4200 | 1.8535          | 6.3824     |
| 1.3065        | 6.0821 | 4300 | 1.9369          | 6.9369     |
| 1.1889        | 6.2236 | 4400 | 1.9767          | 7.2191     |
| 1.1865        | 6.3652 | 4500 | 1.9845          | 7.2752     |
| 1.1927        | 6.5067 | 4600 | 2.0029          | 7.4103     |
| 1.1937        | 6.6483 | 4700 | 1.9931          | 7.3380     |
| 1.1893        | 6.7898 | 4800 | 1.9814          | 7.2532     |
| 1.1654        | 6.9314 | 4900 | 1.9931          | 7.3383     |
| 1.1036        | 7.0722 | 5000 | 2.0025          | 7.4078     |


### Framework versions

- PEFT 0.13.2
- Transformers 4.47.1
- Pytorch 2.1.0.post301
- Datasets 3.0.2
- Tokenizers 0.21.0