Update README.md

19c210a verified 3 months ago

11.1 kB

	---
	license: cc-by-nc-sa-4.0
	tags:
	- chemistry
	- drug-design
	- synthesis-accessibility
	- cheminformatics
	- drug-discovery
	- selfies
	- drugs
	- molecules
	- compounds
	- ranger21
	- madgrad
	pipeline_tag: text-classification
	library_name: transformers
	---

	# Model Card for ChemFIE-SA (Synthesis Accessibility)

	This model is a BERT-like sequence classifier for predicting synthesis accessibility given a SELFIES string of a compound, fine-tuned from [gbyuvd/chemselfies-base-bertmlm](https://huggingface.co/gbyuvd/chemselfies-base-bertmlm) on a DeepSA expanded train dataset (Wang et al. 2023).


	### Disclaimer: For Academic Purposes Only
	The information and model provided is for academic purposes only. It is intended for educational and research use, and should not be used for any commercial or legal purposes. The author do not guarantee the accuracy, completeness, or reliability of the information.


	[![ko-fi](https://ko-fi.com/img/githubbutton_sm.svg)](https://ko-fi.com/O4O710GFBZ)

	## Model Details

	### Model Description

	- Model Type: Transformer (BertForSequenceClassification)
	- Base model: [gbyuvd/chemselfies-base-bertmlm](https://huggingface.co/gbyuvd/chemselfies-base-bertmlm)
	- Maximum Sequence Length: 512 tokens
	- Number of Labels: 2 classes (0 ES: easy synthesis; 1 HS: hard to synthesize)
	- Training Dataset: SELFIES with labels derived from DeepSA
	- Language: SELFIES
	- License: CC-BY-NC-SA 4.0

	## Uses

	If you have Canonical SMILES instead of SELFIES, you can convert it first into a format readable by the model's tokenizer (using whitespace)

	```python
	import selfies as sf

	def smiles_to_selfies_sentence(smiles):
	try:
	selfies = sf.encoder(smiles) # Encode SMILES into SELFIES
	selfies_tokens = list(sf.split_selfies(selfies))

	# Join dots with the nearest next tokens
	joined_tokens = []
	i = 0
	while i < len(selfies_tokens):
	if selfies_tokens[i] == '.' and i + 1 < len(selfies_tokens):
	joined_tokens.append(f".{selfies_tokens[i+1]}")
	i += 2
	else:
	joined_tokens.append(selfies_tokens[i])
	i += 1

	selfies_sentence = ' '.join(joined_tokens)
	return selfies_sentence
	except sf.EncoderError as e:
	print(f"Encoder Error: {e}")
	return None

	# Example usage:
	in_smi = "C1CCC(CC1)(CC(=O)O)CN" # Gabapentin (CID3446)
	selfies_sentence = smiles_to_selfies_sentence(in_smi)
	print(selfies_sentence)

	"""
	[C] [C] [C] [C] [Branch1] [Branch1] [C] [C] [Ring1] [=Branch1] [Branch1] [#Branch1] [C] [C] [=Branch1] [C] [=O] [O] [C] [N]

	"""

	```

	### Direct Use using Classifier Pipeline

	You can also use pipeline:

	```python
	from transformers import pipeline

	classifier = pipeline("text-classification", model="gbyuvd/synthaccess-chemselfies")
	classifier("[C] [C] [C] [C] [Branch1] [Branch1] [C] [C] [Ring1] [=Branch1] [Branch1] [#Branch1] [C] [C] [=Branch1] [C] [=O] [O] [C] [N]") # Gabapentin
	# [{'label': 'Easy', 'score': 0.9187200665473938}]

	```

	## Training Details

	### Training Data

	##### Data Sources

	Training data is fetched from [DeepSA's repository](https://github.com/Shihang-Wang-58/DeepSA).

	##### Data Preparation

	- SMILES is converted into SELFIES
	- Chunked into three parts to accommodate Paperspace's Gradient 6hrs limit.
	- Then the data was split by 90:10 ratio of train:validation.
	- 1st chunk size: 1,197,683 (1,077,915 train : 119,768 validation)
	- The data contain labels for:
	- 0: Easy synthesis (requires less than 10 steps)
	- 1: Hard synthesis (requires more than 10 steps)

	### Training Procedure

	#### Training Hyperparameters
	- Epoch = 1 for each chunk
	- Batch size = 128
	- Number of steps for each chunk: 8422
	I am using Ranger21 with these configuration:

	```
	Ranger21 optimizer ready with following settings:

	Core optimizer = [madgrad](https://arxiv.org/abs/2101.11075)
	Learning rate of 5e-06

	Important - num_epochs of training = 1 epochs
	using AdaBelief for variance computation
	Warm-up: linear warmup, over 2000 iterations

	Lookahead active, merging every 5 steps, with blend factor of 0.5
	Norm Loss active, factor = 0.0001
	Stable weight decay of 0.01
	Gradient Centralization = On

	Adaptive Gradient Clipping = True
	clipping value of 0.01
	steps for clipping = 0.001
	```

	1st Chunk:
	\| Step \| Training Loss \| Validation Loss \| Accuracy \| Precision \| Recall \| F1 \| Roc Auc \|
	\| :--: \| :-----------: \| :-------------: \| :------: \| :-------: \| :------: \| :------: \| :------: \|
	\| 8420 \| 0.128700 \| 0.128632 \| 0.922860 \| 0.975201 \| 0.867836 \| 0.918391 \| 0.990007 \|


	## Model Evaluation

	### Testing Data

	The model (currently only trained on the 1st chunk) was evaluated using three distinct test sets provided by DeepSA's authors (Wang et al. 2023) to ensure comprehensive performance assessment across various scenarios:
	1. Main Expanded Test Set

	2. Independent Test Set 1 (TS1)
	- Characteristics: Contains ES and HS compounds with high intra-group fingerprint similarity, but significant inter-group pattern differences.

	3. Independent Test Set 2 (TS2)
	- Characteristics: Contains a small portion of ES and HS molecules showing similar fingerprint patterns.

	4. Independent Test Set 3 (TS3)
	- Characteristics: All compounds exhibit high fingerprint similarity, presenting the most challenging classification task.

	### Evaluation Metrics

	We employed a comprehensive set of metrics to evaluate our model's performance:

	1. Accuracy (ACC): Overall correctness of predictions
	2. Recall: Ability to identify all relevant instances (sensitivity)
	3. Precision: Accuracy of positive predictions
	4. F1-score: Harmonic mean of precision and recall
	5. Area Under the Receiver Operating Characteristic curve (AUROC): Model's ability to distinguish between classes

	All metrics were evaluated using a threshold of 0.50 for binary classification.

	### Results

	Below are the detailed results of our model's performance across all test sets:

	#### Expanded Test Set Results
	Comparison data is sourced from Wang et al. (2023), used various models as encoding layer:
	- bert-mini (MinBert)
	- bert-tini (TinBert)
	- roberta-base (RoBERTa)
	- deberta-v3-base (DeBERTa)
	- Chem_GraphCodeBert (GraphCodeBert)
	- electra-small-discriminator (SmELECTRA)
	- ChemBERTa-77M-MTR (ChemMTR)
	- ChemBERTa-77M-MLM (ChemMLM)

	which was trained/fine-tuned to predict based on SMILES - while ChemFIE-SA is SELFIES-based:

	\| Model \| Recall \| Precision \| F–score \| AUROC \|
	\| -------------------- \| :--------: \| :-----------: \| :---------: \| :-------: \|
	\| DeepSA_DeBERTa \| 0.873 \| 0.920 \| 0.896 \| 0.959 \|
	\| DeepSA_GraphCodeBert \| 0.931 \| 0.944 \| 0.937 \| 0.987 \|
	\| DeepSA_MinBert \| 0.933 \| 0.945 \| 0.939 \| 0.988 \|
	\| DeepSA_RoBERTa \| 0.940 \| 0.940 \| 0.940 \| 0.988 \|
	\| DeepSA_TinBert \| 0.937 \| 0.947 \| 0.942 \| 0.990 \|
	\| DeepSA_SmELECTRA \| 0.938 \| 0.949 \| 0.943 \| 0.990 \|
	\| ChemFIE-SA \| 0.952 \| 0.942 \| 0.947 \| 0.990 \|
	\| DeepSA_ChemMLM \| 0.955 \| 0.967 \| 0.961 \| 0.995 \|
	\| DeepSA_ChemMTR \| 0.968 \| 0.974 \| 0.971 \| 0.997 \|

	#### TS1-3 Results

	Comparison with DeepSA_SmELECTRA as described in Wang et al. (2023):

	\| Datasets \| Model \| ACC \| Recall \| Precision \| F-score \| AUROC \| Threshold \|
	\| -------- \| ---------- \| :---: \| :----: \| :-------: \| :-----: \| :---: \| :-------: \|
	\| TS1 \| DeepSA \| 0.995 \| 1.000 \| 0.990 \| 0.995 \| 1.000 \| 0.500 \|
	\| \| ChemFIE-SA \| 0.996 \| 1.000 \| 0.992 \| 0.996 \| 1.000 \| 0.500 \|
	\| TS2 \| DeepSA \| 0.838 \| 0.730 \| 0.871 \| 0.795 \| 0.913 \| 0.500 \|
	\| \| ChemFIE-SA \| 0.805 \| 0.775 \| 0.770 \| 0.773 \| 0.886 \| 0.500 \|
	\| TS3 \| DeepSA \| 0.817 \| 0.753 \| 0.864 \| 0.805 \| 0.896 \| 0.500 \|
	\| \| ChemFIE-SA \| 0.731 \| 0.642 \| 0.781 \| 0.705 \| 0.797 \| 0.500 \|


	## Model Examination

	You can visualize its attention heads using [BertViz](https://github.com/jessevig/bertviz) and attribution weights using [Captum](https://captum.ai/) - as [done in the base model](gbyuvd/chemselfies-base-bertmlm) in Interpretability section.

	### Compute Infrastructure

	#### Hardware

	- Platform: Paperspace's Gradients
	- Compute: Free-P5000 (16 GB GPU, 30 GB RAM, 8 vCPU)

	#### Software

	- Python: 3.9.13
	- Transformers: 4.42.4
	- PyTorch: 2.3.1+cu121
	- Accelerate: 0.32.0
	- Datasets: 2.20.0
	- Tokenizers: 0.19.1
	- Ranger21: 0.0.1
	- Selfies: 2.1.2
	- RDKit: 2024.3.3


	## Citation

	If you find this project useful in your research and wish to cite it, please use the following BibTex entry:

	```bibtex
	@software{chemfie_basebertmlm,
	author = {GP Bayu},
	title = {{ChemFIE Base}: Pretraining A Lightweight BERT-like model on Molecular SELFIES},
	url = {https://huggingface.co/gbyuvd/chemselfies-base-bertmlm},
	version = {1.0},
	year = {2024},
	}
	```

	## References
	[DeepSA](https://doi.org/10.1186/s13321-023-00771-3)

	```bibtex
	@article{Wang2023DeepSA,
	title={DeepSA: a deep-learning driven predictor of compound synthesis accessibility},
	author={Wang, Shihang and Wang, Lin and Li, Fenglei and Bai, Fang},
	journal={Journal of Cheminformatics},
	volume={15},
	pages={103},
	year={2023},
	month={Nov},
	publisher={BioMed Central},
	doi={10.1186/s13321-023-00771-3},
	}

	```

	[SELFIES](https://doi.org/10.1088/2632-2153/aba947)
	```bibtex
	@article{krenn2020selfies,
	title={Self-referencing embedded strings (SELFIES): A 100\% robust molecular string representation},
	author={Krenn, Mario and H{\"a}se, Florian and Nigam, AkshatKumar and Friederich, Pascal and Aspuru-Guzik, Alan},
	journal={Machine Learning: Science and Technology},
	volume={1},
	number={4},
	pages={045024},
	year={2020},
	doi={10.1088/2632-2153/aba947}
	}
	```

	[Ranger21](https://arxiv.org/abs/2106.13731)
	```bibtex
	@article{wright2021ranger21,
	title={Ranger21: a synergistic deep learning optimizer},
	author={Wright, Less and Demeure, Nestor},
	year={2021},
	journal={arXiv preprint arXiv:2106.13731},
	}
	```

	## Contact & Support My Work

	G Bayu ([email protected])

	This project has been quiet a journey for me, I’ve dedicated hours on this and I would like to improve myself, this model, and future projects. However, financial and computational constraints can be challenging.

	If you find my work valuable and would like to support my journey, please consider supporting me [here](https://ko-fi.com/gbyuvd). Your support will help me cover costs for computational resources, data acquisition, and further development of this project. Any amount, big or small, is greatly appreciated and will enable me to continue learning and explore more.

	Thank you for checking out this model, I am more than happy to receive any feedback, so that I can improve myself and the future model/projects I will be working on.