|
--- |
|
license: mit |
|
metrics: |
|
- accuracy |
|
tags: |
|
- chemistry |
|
--- |
|
# Molecular BERT Pretrained Using ChEMBL Database |
|
|
|
This model has been pretrained based on the methodology outlined in the paper [Pushing the Boundaries of Molecular Property Prediction for Drug Discovery with Multitask Learning BERT Enhanced by SMILES Enumeration](https://spj.science.org/doi/10.34133/research.0004). While the original model was initially trained using custom code, it has been adapted for use within the Hugging Face Transformers framework in this project. |
|
|
|
## Model Details |
|
The model architecture utilized is based on BERT. Here are the key configuration details: |
|
|
|
``` |
|
BertConfig( |
|
vocab_size=70, |
|
hidden_size=256, |
|
num_hidden_layers=8, |
|
num_attention_heads=8, |
|
intermediate_size=1024, |
|
hidden_act="gelu", |
|
hidden_dropout_prob=0.1, |
|
attention_probs_dropout_prob=0.1, |
|
max_position_embeddings=max_seq_len, |
|
type_vocab_size=1, |
|
pad_token_id=tokenizer_pretrained.vocab["[PAD]"], |
|
position_embedding_type="absolute" |
|
) |
|
``` |
|
|
|
- Optimizer: AdamW |
|
- Learning rate: 1e-4 |
|
- Learning rate scheduler: False |
|
- Epochs: 50 |
|
- AMP: True |
|
- GPU: Single Nvidia RTX 3090 |
|
|
|
## Pretraining Database |
|
The model was pretrained using data from the ChEMBL database, specifically version 33. You can download the database from [ChEMBL](https://ftp.ebi.ac.uk/pub/databases/chembl/ChEMBLdb/latest/). |
|
Additionally, the dataset is available on the Hugging Face Datasets Hub and can be accessed at [Hugging Face Datasets - ChEMBL_v33_pretraining](https://huggingface.co/datasets/jonghyunlee/ChEMBL_v33_pretraining/viewer/default/train). |
|
|
|
## Performance |
|
The accuracy score achieved by the pretrained model is 0.9672. The testing dataset used for evaluation constitutes 10% of the ChEMBL dataset. |
|
|
|
|
