Hugging Face's logo

gbyuvd
/
synthaccess-chemselfies

Text Classification
Transformers
PyTorch
Safetensors
bert
chemistry
drug-design
synthesis-accessibility
cheminformatics
drug-discovery
selfies
drugs
molecules
compounds
ranger21
madgrad
Inference Endpoints
Model card Files Community
synthaccess-chemselfies / README.md
gbyuvd's picture
gbyuvd
Update README.md
bfda42b verified
preview code
|
raw
history blame
11.1 kB
metadata
license: cc-by-nc-sa-4.0
tags:
  - chemistry
  - drug-design
  - synthesis-accessibility
  - cheminformatics
  - drug-discovery
  - selfies
  - drugs
  - molecules
  - compounds
  - ranger21
  - madgrad
pipeline_tag: text-classification

Model Card for ChemFIE-SA (Synthesis Accessibility)

This model is a BERT-like sequence classifier for predicting synthesis accessibility given a SELFIES string of a compound, fine-tuned from gbyuvd/chemselfies-base-bertmlm on a DeepSA expanded train dataset (Wang et al. 2023).

Disclaimer: For Academic Purposes Only

The information and model provided is for academic purposes only. It is intended for educational and research use, and should not be used for any commercial or legal purposes. The author do not guarantee the accuracy, completeness, or reliability of the information.

ko-fi

Model Details

Model Description

  • Model Type: Transformer (BertForSequenceClassification)
  • Base model: gbyuvd/chemselfies-base-bertmlm
  • Maximum Sequence Length: 512 tokens
  • Number of Labels: 2 classes (0 ES: easy synthesis; 1 HS: hard to synthesize)
  • Training Dataset: SELFIES with labels derived from DeepSA
  • Language: SELFIES
  • License: CC-BY-NC-SA 4.0

Uses

If you have Canonical SMILES instead of SELFIES, you can convert it first into a format readable by the model's tokenizer (using whitespace)

import selfies as sf

def smiles_to_selfies_sentence(smiles):
    try:
        selfies = sf.encoder(smiles)  # Encode SMILES into SELFIES
        selfies_tokens = list(sf.split_selfies(selfies))
        
        # Join dots with the nearest next tokens
        joined_tokens = []
        i = 0
        while i < len(selfies_tokens):
            if selfies_tokens[i] == '.' and i + 1 < len(selfies_tokens):
                joined_tokens.append(f".{selfies_tokens[i+1]}")
                i += 2
            else:
                joined_tokens.append(selfies_tokens[i])
                i += 1
        
        selfies_sentence = ' '.join(joined_tokens)
        return selfies_sentence
    except sf.EncoderError as e:
        print(f"Encoder Error: {e}")
        return None

# Example usage:
in_smi = "C1CCC(CC1)(CC(=O)O)CN" # Gabapentin (CID3446)
selfies_sentence = smiles_to_selfies_sentence(in_smi)
print(selfies_sentence)

"""
[C] [C] [C] [C] [Branch1] [Branch1] [C] [C] [Ring1] [=Branch1] [Branch1] [#Branch1] [C] [C] [=Branch1] [C] [=O] [O] [C] [N]

"""

Direct Use using Classifier Pipeline

You can also use pipeline:

from transformers import pipeline

classifier = pipeline("text-classification", model="gbyuvd/synthaccess-chemselfies")
classifier("[C] [C] [C] [C] [Branch1] [Branch1] [C] [C] [Ring1] [=Branch1] [Branch1] [#Branch1] [C] [C] [=Branch1] [C] [=O] [O] [C] [N]") # Gabapentin
# [{'label': 'Easy', 'score': 0.9187200665473938}]

Training Details

Training Data

Data Sources

Training data is fetched from DeepSA's repository.

Data Preparation
  • SMILES is converted into SELFIES
  • Chunked into three parts to accommodate Paperspace's Gradient 6hrs limit.
  • Then the data was split by 90:10 ratio of train:validation.
    • 1st chunk size: 1,197,683 (1,077,915 train : 119,768 validation)
  • The data contain labels for:
    • 0: Easy synthesis (requires less than 10 steps)
    • 1: Hard synthesis (requires more than 10 steps)

Training Procedure

Training Hyperparameters

  • Epoch = 1 for each chunk
  • Batch size = 128
  • Number of steps for each chunk: 8422 I am using Ranger21 with these configuration:
Ranger21 optimizer ready with following settings:

Core optimizer = [madgrad](https://arxiv.org/abs/2101.11075)
Learning rate of 5e-06

Important - num_epochs of training = ** 1 epochs **
using AdaBelief for variance computation
Warm-up: linear warmup, over 2000 iterations

Lookahead active, merging every 5 steps, with blend factor of 0.5
Norm Loss active, factor = 0.0001
Stable weight decay of 0.01
Gradient Centralization = On

Adaptive Gradient Clipping = True
    clipping value of 0.01
    steps for clipping = 0.001

1st Chunk:

Step Training Loss Validation Loss Accuracy Precision Recall F1 Roc Auc
8420 0.128700 0.128632 0.922860 0.975201 0.867836 0.918391 0.990007

Model Evaluation

Testing Data

The model (currently only trained on the 1st chunk) was evaluated using three distinct test sets provided by DeepSA's authors (Wang et al. 2023) to ensure comprehensive performance assessment across various scenarios:

  1. Main Expanded Test Set

  2. Independent Test Set 1 (TS1)

    • Characteristics: Contains ES and HS compounds with high intra-group fingerprint similarity, but significant inter-group pattern differences.

  3. Independent Test Set 2 (TS2)

    • Characteristics: Contains a small portion of ES and HS molecules showing similar fingerprint patterns.

  4. Independent Test Set 3 (TS3)

    • Characteristics: All compounds exhibit high fingerprint similarity, presenting the most challenging classification task.

Evaluation Metrics

We employed a comprehensive set of metrics to evaluate our model's performance:

  1. Accuracy (ACC): Overall correctness of predictions
  2. Recall: Ability to identify all relevant instances (sensitivity)
  3. Precision: Accuracy of positive predictions
  4. F1-score: Harmonic mean of precision and recall
  5. Area Under the Receiver Operating Characteristic curve (AUROC): Model's ability to distinguish between classes

All metrics were evaluated using a threshold of 0.50 for binary classification.

Results

Below are the detailed results of our model's performance across all test sets:

Expanded Test Set Results

Comparison data is sourced from Wang et al. (2023), used various models as encoding layer:

  • bert-mini (MinBert)
  • bert-tini (TinBert)
  • roberta-base (RoBERTa)
  • deberta-v3-base (DeBERTa)
  • Chem_GraphCodeBert (GraphCodeBert)
  • electra-small-discriminator (SmELECTRA)
  • ChemBERTa-77M-MTR (ChemMTR)
  • ChemBERTa-77M-MLM (ChemMLM)

which was trained/fine-tuned to predict based on SMILES - while ChemFIE-SA is SELFIES-based:

Model Recall Precision F–score AUROC
DeepSA_DeBERTa 0.873 0.920 0.896 0.959
DeepSA_GraphCodeBert 0.931 0.944 0.937 0.987
DeepSA_MinBert 0.933 0.945 0.939 0.988
DeepSA_RoBERTa 0.940 0.940 0.940 0.988
DeepSA_TinBert 0.937 0.947 0.942 0.990
DeepSA_SmELECTRA 0.938 0.949 0.943 0.990
ChemFIE-SA 0.952 0.942 0.947 0.990
DeepSA_ChemMLM 0.955 0.967 0.961 0.995
DeepSA_ChemMTR 0.968 0.974 0.971 0.997

TS1-3 Results

Comparison with DeepSA_SmELECTRA as described in Wang et al. (2023):

Datasets Model ACC Recall Precision F-score AUROC Threshold
TS1 DeepSA 0.995 1.000 0.990 0.995 1.000 0.500
ChemFIE-SA 0.996 1.000 0.992 0.996 1.000 0.500
TS2 DeepSA 0.838 0.730 0.871 0.795 0.913 0.500
ChemFIE-SA 0.805 0.775 0.770 0.773 0.886 0.500
TS3 DeepSA 0.817 0.753 0.864 0.805 0.896 0.500
ChemFIE-SA 0.731 0.642 0.781 0.705 0.797 0.500

Model Examination

You can visualize its attention heads using BertViz and attribution weights using Captum - as done in the base model in Interpretability section.

Compute Infrastructure

Hardware

  • Platform: Paperspace's Gradients
  • Compute: Free-P5000 (16 GB GPU, 30 GB RAM, 8 vCPU)

Software

  • Python: 3.9.13
  • Transformers: 4.42.4
  • PyTorch: 2.3.1+cu121
  • Accelerate: 0.32.0
  • Datasets: 2.20.0
  • Tokenizers: 0.19.1
  • Ranger21: 0.0.1
  • Selfies: 2.1.2
  • RDKit: 2024.3.3

Citation

If you find this project useful in your research and wish to cite it, please use the following BibTex entry:

@software{chemfie_basebertmlm,
  author = {GP Bayu},
  title = {{ChemFIE Base}: Pretraining A Lightweight BERT-like model on Molecular SELFIES},
  url = {https://huggingface.co/gbyuvd/chemselfies-base-bertmlm},
  version = {1.0},
  year = {2024},
}

References

DeepSA

@article{Wang2023DeepSA,
  title={DeepSA: a deep-learning driven predictor of compound synthesis accessibility},
  author={Wang, Shihang and Wang, Lin and Li, Fenglei and Bai, Fang},
  journal={Journal of Cheminformatics},
  volume={15},
  pages={103},
  year={2023},
  month={Nov},
  publisher={BioMed Central},
  doi={10.1186/s13321-023-00771-3},
}

SELFIES

@article{krenn2020selfies,
  title={Self-referencing embedded strings (SELFIES): A 100\% robust molecular string representation},
  author={Krenn, Mario and H{\"a}se, Florian and Nigam, AkshatKumar and Friederich, Pascal and Aspuru-Guzik, Alan},
  journal={Machine Learning: Science and Technology},
  volume={1},
  number={4},
  pages={045024},
  year={2020},
  doi={10.1088/2632-2153/aba947}
}

Ranger21

@article{wright2021ranger21,
      title={Ranger21: a synergistic deep learning optimizer}, 
      author={Wright, Less and Demeure, Nestor},
      year={2021},
      journal={arXiv preprint arXiv:2106.13731},
}

Contact & Support My Work

G Bayu ([email protected])

This project has been quiet a journey for me, I’ve dedicated hours on this and I would like to improve myself, this model, and future projects. However, financial and computational constraints can be challenging.

If you find my work valuable and would like to support my journey, please consider supporting me here. Your support will help me cover costs for computational resources, data acquisition, and further development of this project. Any amount, big or small, is greatly appreciated and will enable me to continue learning and explore more.

Thank you for checking out this model, I am more than happy to receive any feedback, so that I can improve myself and the future model/projects I will be working on.