Hugging Face's logo

gbyuvd
/
chemfie-gpt-experiment-1

Text Generation
Transformers
Safetensors
gpt2
chemistry
selfies
text-generation-inference
Inference Endpoints
Model card Files Community
chemfie-gpt-experiment-1 / README.md
gbyuvd's picture
gbyuvd
Update README.md
4f592de verified
preview code
|
raw
history blame
5.49 kB
metadata
license: cc-by-nc-sa-4.0
library_name: transformers
tags:
  - chemistry
  - selfies

chemfie-gpt-experiment-1

On-going training (3/4)

Model Details

  • Model Type: GPT-2
  • Architecture: L8, A6, H384
  • Task: Generation of SELFIES strings
  • Language: N/A (Chemical representation)

Personal Intended Use

  • Hands-on learning, research and experimentation in molecular generation
  • Baseline for ablation studies and comparisons with more advanced models

Use

Since this model doesn't use a proper GPT2 format tokenizer, special tokens still need to be set up manually (next experiment will use a proper one ofc):

from transformers import PreTrainedTokenizerFast, AutoModelForCausalLM
import torch

tokenizer = PreTrainedTokenizerFast(
    tokenizer_file="gpt2_tokenizer.json",
    model_max_length=512,
    unk_token="<unk>",
    pad_token="<pad>",
    eos_token="</s>",
    bos_token="<s>",
    mask_token="<mask>",
)

model = AutoModelForCausalLM.from_pretrained("gbyuvd/chemfie-gpt-experiment-1")

# Generate some sample outputs
def generate_molecules(model, tokenizer, num_samples=5, max_length=100):
    model.eval()
    generated = []
    for _ in range(num_samples):
        input_ids = torch.tensor([[tokenizer.bos_token_id]]).to(model.device)
        output = model.generate(input_ids, max_length=max_length, num_return_sequences=1, do_sample=True)
        generated.append(tokenizer.decode(output[0], skip_special_tokens=True))
    return generated

sample_molecules = generate_molecules(model, tokenizer)
print("Sample generated molecules:")
for i, mol in enumerate(sample_molecules, 1):
    print(f"{i}. {mol}")

Tokenized SELFIES to SMILES:

import selfies as sf

test = "[C] [Branch1] [O] [=C] [C] [C] [C] [C] [C] [C] [C] [=Branch1] [=O] [O] [=C] [C] [C] [C] [Ring1]"
test = test.replace(' ', '')
print(sf.decoder(test))

""""
C(CCCCCCCCO)=CCC=C

""""

Training Data

  • Source: Curated and merged from COCONUTDB (Sorokina et al., 2021), ChemBL34 (Zdrazil et al., 2023), and SuperNatural3 (Gallo et al. 2023) database
  • Total: 2,933,355 samples
  • Total Train: 2,346,680 samples
  • Validation: 293,336 samples
  • Per chunk: 586,670 train, 73,334 validation, 73,334 test
  • Random seed for split: 42

Training Procedure

  • Batch Size: 64
  • Num Epoch for Each Chunk: 1
  • Learning Rate: 1.5e-5
  • Optimizer: Ranger21 (MADGRAD-Lookahead-AdaBelief with gradient centralization, linear warm up (22%), gradient clipping, and L2 weight decay)

Training Logs

Chunk Chunk's Training Loss Chunk's Validation Loss Status
I 1.346400 1.065180 Done
II 1.123500 0.993118 Done
III 1.058300 0.948303 Done
IV Ongoing

Evaluation Results

[To be filled after model evaluation]

Limitations and Biases

  • May generate unrealistic or synthetically inaccessible molecules
  • Performance on complex, branched, and ringed molecules to be evaluated

Ethical Considerations

  • Potential misuse for generating harmful or illegal substances
  • May produce biased results based on training data composition
  • The information and model provided is for academic purposes only. It is intended for educational and research use, and should not be used for any commercial or legal purposes. The author do not guarantee the accuracy, completeness, or reliability of the information.

Additional Information

  • Part of experimental chemfie-gpt/T5 project
  • Serves as a baseline for future experiments with further curated datasets, training improvements, and architectural modifications

Citation

BibTeX

COCONUTDB 

@article{sorokina2021coconut,
  title={COCONUT online: Collection of Open Natural Products database},
  author={Sorokina, Maria and Merseburger, Peter and Rajan, Kohulan and Yirik, Mehmet Aziz and Steinbeck, Christoph},
  journal={Journal of Cheminformatics},
  volume={13},
  number={1},
  pages={2},
  year={2021},
  doi={10.1186/s13321-020-00478-9}
}

ChemBL34 

@article{zdrazil2023chembl,
  title={The ChEMBL Database in 2023: a drug discovery platform spanning multiple bioactivity data types and time periods},
  author={Zdrazil, Barbara and Felix, Eloy and Hunter, Fiona and Manners, Emma J and Blackshaw, James and Corbett, Sybilla and de Veij, Marleen and Ioannidis, Harris and Lopez, David Mendez and Mosquera, Juan F and Magarinos, Maria Paula and Bosc, Nicolas and Arcila, Ricardo and Kizil{\"o}ren, Tevfik and Gaulton, Anna and Bento, A Patr{\'i}cia and Adasme, Melissa F and Monecke, Peter and Landrum, Gregory A and Leach, Andrew R},
  journal={Nucleic Acids Research},
  year={2023},
  volume={gkad1004},
  doi={10.1093/nar/gkad1004}
}

@misc{chembl34,
  title={ChemBL34},
  year={2023},
  doi={10.6019/CHEMBL.database.34}
}

SuperNatural3 

@article{Gallo2023,
  author = {Gallo, K and Kemmler, E and Goede, A and Becker, F and Dunkel, M and Preissner, R and Banerjee, P},
  title = {{SuperNatural 3.0-a database of natural products and natural product-based derivatives}},
  journal = {Nucleic Acids Research},
  year = {2023},
  month = jan,
  day = {6},
  volume = {51},
  number = {D1},
  pages = {D654-D659},
  doi = {10.1093/nar/gkac1008}
}