| | --- |
| | license: mit |
| | metrics: |
| | - accuracy |
| | tags: |
| | - chemistry |
| | --- |
| | # Molecular BERT Pretrained Using ChEMBL Database |
| |
|
| | This model has been pretrained based on the methodology outlined in the paper [Pushing the Boundaries of Molecular Property Prediction for Drug Discovery with Multitask Learning BERT Enhanced by SMILES Enumeration](https://spj.science.org/doi/10.34133/research.0004). While the original model was initially trained using custom code, it has been adapted for use within the Hugging Face Transformers framework in this project. |
| |
|
| | ## Model Details |
| | The model architecture utilized is based on BERT. Here are the key configuration details: |
| |
|
| | ``` |
| | BertConfig( |
| | vocab_size=70, |
| | hidden_size=256, |
| | num_hidden_layers=8, |
| | num_attention_heads=8, |
| | intermediate_size=1024, |
| | hidden_act="gelu", |
| | hidden_dropout_prob=0.1, |
| | attention_probs_dropout_prob=0.1, |
| | max_position_embeddings=max_seq_len, |
| | type_vocab_size=1, |
| | pad_token_id=tokenizer_pretrained.vocab["[PAD]"], |
| | position_embedding_type="absolute" |
| | ) |
| | ``` |
| |
|
| | - Optimizer: AdamW |
| | - Learning rate: 1e-4 |
| | - Learning rate scheduler: False |
| | - Epochs: 50 |
| | - AMP: True |
| | - GPU: Single Nvidia RTX 3090 |
| |
|
| | ## Pretraining Database |
| | The model was pretrained using data from the ChEMBL database, specifically version 33. You can download the database from [ChEMBL](https://ftp.ebi.ac.uk/pub/databases/chembl/ChEMBLdb/latest/). |
| | Additionally, the dataset is available on the Hugging Face Datasets Hub and can be accessed at [Hugging Face Datasets - ChEMBL_v33_pretraining](https://huggingface.co/datasets/jonghyunlee/ChEMBL_v33_pretraining/viewer/default/train). |
| |
|
| | ## Performance |
| | The accuracy score achieved by the pretrained model is 0.9672. The testing dataset used for evaluation constitutes 10% of the ChEMBL dataset. |
| |
|
| |
|
