|
--- |
|
library_name: peft |
|
license: bsd-3-clause |
|
base_model: hugohrban/progen2-base |
|
tags: |
|
- generated_from_trainer |
|
model-index: |
|
- name: Progen2_Kinase_PhosphositeGen |
|
results: [] |
|
--- |
|
|
|
<!-- This model card has been generated automatically according to the information the Trainer had access to. You |
|
should probably proofread and complete it, then remove this comment. --> |
|
|
|
# Progen2_Kinase_PhosphositeGen |
|
|
|
This model is a fine-tuned version of [hugohrban/progen2-base](https://huggingface.co/hugohrban/progen2-base) on the None dataset. |
|
It achieves the following results on the evaluation set: |
|
- Loss: 2.0025 |
|
- Perplexity: 7.4078 |
|
|
|
## Model description |
|
|
|
More information needed |
|
|
|
## Intended uses & limitations |
|
|
|
More information needed |
|
|
|
## Training and evaluation data |
|
|
|
More information needed |
|
|
|
## Training procedure |
|
|
|
### Training hyperparameters |
|
|
|
The following hyperparameters were used during training: |
|
- learning_rate: 0.0005 |
|
- train_batch_size: 8 |
|
- eval_batch_size: 8 |
|
- seed: 42 |
|
- gradient_accumulation_steps: 2 |
|
- total_train_batch_size: 16 |
|
- optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments |
|
- lr_scheduler_type: linear |
|
- training_steps: 5000 |
|
|
|
### Training results |
|
|
|
| Training Loss | Epoch | Step | Validation Loss | Perplexity | |
|
|:-------------:|:------:|:----:|:---------------:|:----------:| |
|
| 4.9303 | 0.1415 | 100 | 2.1811 | 8.8564 | |
|
| 4.2874 | 0.2831 | 200 | 2.1198 | 8.3296 | |
|
| 4.1941 | 0.4246 | 300 | 2.0694 | 7.9202 | |
|
| 4.0803 | 0.5662 | 400 | 2.0362 | 7.6616 | |
|
| 4.0613 | 0.7077 | 500 | 2.0053 | 7.4284 | |
|
| 3.9511 | 0.8493 | 600 | 1.9922 | 7.3315 | |
|
| 3.9216 | 0.9908 | 700 | 1.9477 | 7.0124 | |
|
| 3.5053 | 1.1316 | 800 | 1.9200 | 6.8208 | |
|
| 3.4311 | 1.2732 | 900 | 1.9035 | 6.7094 | |
|
| 3.4238 | 1.4147 | 1000 | 1.8714 | 6.4974 | |
|
| 3.392 | 1.5563 | 1100 | 1.8527 | 6.3772 | |
|
| 3.3621 | 1.6978 | 1200 | 1.8317 | 6.2444 | |
|
| 3.3577 | 1.8393 | 1300 | 1.8237 | 6.1945 | |
|
| 3.3419 | 1.9809 | 1400 | 1.7889 | 5.9826 | |
|
| 2.8256 | 2.1217 | 1500 | 1.7977 | 6.0356 | |
|
| 2.8061 | 2.2633 | 1600 | 1.7860 | 5.9653 | |
|
| 2.7837 | 2.4048 | 1700 | 1.7666 | 5.8507 | |
|
| 2.7504 | 2.5464 | 1800 | 1.7428 | 5.7133 | |
|
| 2.829 | 2.6879 | 1900 | 1.7288 | 5.6337 | |
|
| 2.7567 | 2.8294 | 2000 | 1.7088 | 5.5225 | |
|
| 2.7443 | 2.9710 | 2100 | 1.6986 | 5.4664 | |
|
| 2.3409 | 3.1118 | 2200 | 1.7382 | 5.6869 | |
|
| 2.2568 | 3.2534 | 2300 | 1.7487 | 5.7471 | |
|
| 2.2481 | 3.3949 | 2400 | 1.7181 | 5.5740 | |
|
| 2.2323 | 3.5364 | 2500 | 1.7058 | 5.5059 | |
|
| 2.2654 | 3.6780 | 2600 | 1.7031 | 5.4912 | |
|
| 2.2611 | 3.8195 | 2700 | 1.6707 | 5.3157 | |
|
| 2.256 | 3.9611 | 2800 | 1.6719 | 5.3222 | |
|
| 1.8849 | 4.1019 | 2900 | 1.7899 | 5.9886 | |
|
| 1.771 | 4.2435 | 3000 | 1.7697 | 5.8694 | |
|
| 1.7992 | 4.3850 | 3100 | 1.7880 | 5.9775 | |
|
| 1.838 | 4.5265 | 3200 | 1.7871 | 5.9722 | |
|
| 1.8285 | 4.6681 | 3300 | 1.7342 | 5.6644 | |
|
| 1.8127 | 4.8096 | 3400 | 1.7196 | 5.5825 | |
|
| 1.8353 | 4.9512 | 3500 | 1.7471 | 5.7377 | |
|
| 1.5511 | 5.0920 | 3600 | 1.8285 | 6.2248 | |
|
| 1.4449 | 5.2335 | 3700 | 1.8683 | 6.4770 | |
|
| 1.4631 | 5.3751 | 3800 | 1.8880 | 6.6063 | |
|
| 1.4525 | 5.5166 | 3900 | 1.8807 | 6.5581 | |
|
| 1.4516 | 5.6582 | 4000 | 1.8723 | 6.5031 | |
|
| 1.4423 | 5.7997 | 4100 | 1.8828 | 6.5716 | |
|
| 1.4626 | 5.9413 | 4200 | 1.8535 | 6.3824 | |
|
| 1.3065 | 6.0821 | 4300 | 1.9369 | 6.9369 | |
|
| 1.1889 | 6.2236 | 4400 | 1.9767 | 7.2191 | |
|
| 1.1865 | 6.3652 | 4500 | 1.9845 | 7.2752 | |
|
| 1.1927 | 6.5067 | 4600 | 2.0029 | 7.4103 | |
|
| 1.1937 | 6.6483 | 4700 | 1.9931 | 7.3380 | |
|
| 1.1893 | 6.7898 | 4800 | 1.9814 | 7.2532 | |
|
| 1.1654 | 6.9314 | 4900 | 1.9931 | 7.3383 | |
|
| 1.1036 | 7.0722 | 5000 | 2.0025 | 7.4078 | |
|
|
|
|
|
### Framework versions |
|
|
|
- PEFT 0.13.2 |
|
- Transformers 4.47.1 |
|
- Pytorch 2.1.0.post301 |
|
- Datasets 3.0.2 |
|
- Tokenizers 0.21.0 |
