---
language: en
license: apache-2.0
tags:
- biology
- chemistry
- smiles
- bert
- molecular-modeling
---

# chemical_structure_smiles_bert

## Overview
This model is a BERT-style encoder pre-trained on millions of SMILES (Simplified Molecular Input Line Entry System) strings. It learns the "grammar" of chemistry to provide high-dimensional embeddings of molecules, which can then be used for downstream tasks like property prediction, toxicity screening, and drug-target interaction modeling.



## Model Architecture
The model adapts the **BERT** (Bidirectional Encoder Representations from Transformers) architecture for molecular sequences:
- **Tokenizer**: Custom regex-based tokenizer that treats atoms (e.g., `[Fe+2]`, `Cl`) and structural markers (`=`, `#`, `(`, `)`) as individual tokens.
- **Encoder**: 12 layers of multi-head self-attention.
- **Pre-training Task**: Masked Language Modeling (MLM), where 15% of atoms/bonds are hidden and predicted based on context.

## Intended Use
- **Molecular Fingerprinting**: Generating dense vector representations for similarity searches in chemical databases.
- **Lead Optimization**: Serving as a feature extractor for models predicting LogP, solubility, or binding affinity.
- **Reaction Prediction**: Analyzing chemical transformations by comparing reactant and product embeddings.

## Limitations
- **3D Conformation**: This model only understands 1D string representations and does not account for 3D spatial stereochemistry or bond angles.
- **Sequence Length**: Extremely large polymers or proteins exceeding 512 SMILES tokens are truncated.
- **Chemical Diversity**: Predictions may be biased toward drug-like small molecules if the chemical space is outside the pre-training distribution.