---
license: apache-2.0
base_model: ctheodoris/Geneformer
tags:
- generated_from_trainer
metrics:
- precision
- recall
- accuracy
model-index:
- name: Geneformer_ft_BioS73_1kbpHG19_DHSs_H3K27AC
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# Geneformer_ft_BioS73_1kbpHG19_DHSs_H3K27AC

This model is a fine-tuned version of [ctheodoris/Geneformer](https://huggingface.co/ctheodoris/Geneformer) on the None dataset.
It achieves the following results on the evaluation set:
- Loss: 0.5484
- F1 Score: 0.7419
- Precision: 0.7859
- Recall: 0.7025
- Accuracy: 0.7391
- Auc: 0.8188
- Prc: 0.8263

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 1e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 20
- mixed_precision_training: Native AMP

### Training results

| Training Loss | Epoch  | Step | Validation Loss | F1 Score | Precision | Recall | Accuracy | Auc    | Prc    |
|:-------------:|:------:|:----:|:---------------:|:--------:|:---------:|:------:|:--------:|:------:|:------:|
| 0.6878        | 0.1864 | 500  | 0.6845          | 0.6851   | 0.5493    | 0.9099 | 0.5535   | 0.5520 | 0.5797 |
| 0.6585        | 0.3727 | 1000 | 0.6392          | 0.6249   | 0.7056    | 0.5608 | 0.6407   | 0.6920 | 0.7149 |
| 0.631         | 0.5591 | 1500 | 0.6069          | 0.6942   | 0.6901    | 0.6983 | 0.6716   | 0.7270 | 0.7461 |
| 0.6114        | 0.7454 | 2000 | 0.5890          | 0.7411   | 0.6639    | 0.8387 | 0.6873   | 0.7600 | 0.7657 |
| 0.5968        | 0.9318 | 2500 | 0.6026          | 0.7465   | 0.6453    | 0.8855 | 0.6791   | 0.7718 | 0.7718 |
| 0.5747        | 1.1182 | 3000 | 0.5604          | 0.7434   | 0.7094    | 0.7807 | 0.7123   | 0.7836 | 0.7867 |
| 0.5611        | 1.3045 | 3500 | 0.5515          | 0.7540   | 0.7296    | 0.7800 | 0.7283   | 0.7897 | 0.7912 |
| 0.5666        | 1.4909 | 4000 | 0.5482          | 0.7523   | 0.7424    | 0.7626 | 0.7320   | 0.7932 | 0.7975 |
| 0.5557        | 1.6772 | 4500 | 0.5394          | 0.7597   | 0.7313    | 0.7905 | 0.7331   | 0.8033 | 0.8049 |
| 0.5563        | 1.8636 | 5000 | 0.5729          | 0.7647   | 0.6624    | 0.9043 | 0.7029   | 0.8028 | 0.8063 |
| 0.5331        | 2.0499 | 5500 | 0.5457          | 0.7491   | 0.7719    | 0.7277 | 0.7398   | 0.8056 | 0.8123 |
| 0.5292        | 2.2363 | 6000 | 0.5751          | 0.7681   | 0.6794    | 0.8834 | 0.7152   | 0.8017 | 0.8063 |
| 0.5286        | 2.4227 | 6500 | 0.5377          | 0.7692   | 0.7213    | 0.8240 | 0.7361   | 0.8068 | 0.8115 |
| 0.5397        | 2.6090 | 7000 | 0.5351          | 0.7549   | 0.7717    | 0.7388 | 0.7439   | 0.8115 | 0.8160 |
| 0.5338        | 2.7954 | 7500 | 0.5346          | 0.7519   | 0.7860    | 0.7207 | 0.7462   | 0.8164 | 0.8190 |
| 0.5181        | 2.9817 | 8000 | 0.5458          | 0.7790   | 0.7028    | 0.8736 | 0.7354   | 0.8184 | 0.8211 |
| 0.5065        | 3.1681 | 8500 | 0.5379          | 0.7676   | 0.7393    | 0.7982 | 0.7421   | 0.8175 | 0.8242 |
| 0.5028        | 3.3545 | 9000 | 0.5292          | 0.7744   | 0.7417    | 0.8101 | 0.7480   | 0.8209 | 0.8270 |
| 0.5153        | 3.5408 | 9500 | 0.5484          | 0.7419   | 0.7859    | 0.7025 | 0.7391   | 0.8188 | 0.8263 |


### Framework versions

- Transformers 4.42.3
- Pytorch 2.3.0+cu121
- Datasets 2.18.0
- Tokenizers 0.19.0