YAML Metadata Warning:empty or missing yaml metadata in repo card

Check out the documentation for more information.

MolE-RTD-25pct

MolE molecular encoder trained with DeBERTa-v3-style Replaced Token Detection at 25% masking rate on 415M ZINC-Curated molecules. Higher mask probability than standard RTD-15% β€” motivated by ELECTRA paper findings that small-vocab domains benefit from more aggressive masking.

Based on MolE by Recursion Pharmaceuticals and DeBERTa-v3. For the 15% variant see caithmac/MolE-RTD-ZINC415M.


Model description

MolE represents molecules as sequences of Morgan fingerprint atom environments (radius-0, vocab ~211 tokens) and encodes them with a DeBERTa disentangled-attention transformer. Bond distances from the molecular graph are passed as relative position biases β€” no absolute positional embeddings.

RTD pre-training trains a generator (small, 3-layer) to corrupt 25% of input tokens, and a discriminator (full-size, 12-layer) to detect which tokens were replaced. GDES (Gradient-Disentangled Embedding Sharing) prevents RTD loss from corrupting shared embeddings.

Architecture

Component Config
Discriminator layers 12
Discriminator hidden size 768
Discriminator intermediate size 3072
Discriminator attention heads 12 Γ— 64
Generator layers 3
Generator hidden size 256
Shared embedding size 768
Vocabulary 211 atom environments (radius-0 Morgan)
Mask probability 25%
RTD Ξ» 50
Relative attention Yes (p2c + c2p)
Absolute position embeddings No

Pre-training details

Setting Value
Dataset SZU-ADDG/ZINC-Curated (~415M molecules)
Max atoms 96 heavy atoms
Steps 1,000,000
Effective batch size 512 (128 Γ— 4 GPUs)
Optimizer AdamW (lr=1e-4, weight_decay=0.01)
LR schedule Cosine with 10k warmup steps
Hardware 4 Γ— NVIDIA A100-SXM4-40GB
Wall time ~29 hours

Results (with Step 2 ChEMBL supervised pretraining)

Full TDC 22-task sweep. RTD-25% beats RTD-15% on hard pharmacokinetic tasks (vdss_lombardo +0.065, half_life_obach +0.183, cyp2d6_substrate +0.079). On easier tasks (herg, bioavailability), 15% is slightly better. Summary: 25% masking improves generalization on low-signal tasks at cost of slightly worse performance on saturated tasks.

Task RTD-25%+S2 RTD-15%+S2 Best
half_life_obach (Spearman) 0.473 0.290 25%
vdss_lombardo (Spearman) 0.623 0.558 25%
cyp2d6_substrate (AUPRC) 0.617 0.538 25%
cyp2c9_substrate (AUPRC) 0.372 0.321 25%
dili (AUROC) 0.903 0.868 25%
herg (AUROC) 0.757 0.818 15%
bioavailability_ma (AUROC) 0.644 0.680 15%

ASAP-Polaris Potency (pIC50): PARITY with #1 leaderboard (MolE author's entry). Aggregated MAE 0.500Β±0.022.

TYK2 DDG: RMSE 1.498, Pearson R 0.688 β€” beats Morgan ECFP4 (1.608/0.634).


How to use

from collections import OrderedDict
import torch
from huggingface_hub import hf_hub_download
from DeBERTa.deberta.config import ModelConfig
from mole.training.models.mole import AtomEnvEmbeddings

DISC_CFG = dict(
    embedding_size=768, hidden_size=768, intermediate_size=3072,
    num_hidden_layers=12, num_attention_heads=12, attention_head_size=64,
    attention_probs_dropout_prob=0.1, hidden_dropout_prob=0.1,
    hidden_act="gelu", layer_norm_eps=1e-7, max_position_embeddings=0,
    max_relative_positions=512, position_buckets=0, norm_rel_ebd="layer_norm",
    pos_att_type="p2c|c2p", position_biased_input=False, relative_attention=True,
    share_att_key=True, type_vocab_size=0, vocab_size=211,
)

ckpt = hf_hub_download("caithmac/MolE-RTD-25pct", "mole_rtd_25pct_final.ckpt")
raw = torch.load(ckpt, map_location="cpu", weights_only=False)
sd = raw.get("state_dict", raw)

gen_w = sd["model.generator.embeddings.word_embeddings.weight"]
bias  = sd["model.disc_word_bias"]
enc_sd = OrderedDict()
enc_sd["embeddings.word_embeddings.weight"] = gen_w + bias
for k, v in sd.items():
    if not k.startswith("model.discriminator."): continue
    if ".embeddings.word_embeddings." in k: continue
    enc_sd[k[len("model.discriminator."):]] = v

encoder = AtomEnvEmbeddings(ModelConfig.from_dict(DISC_CFG))
encoder.load_state_dict(enc_sd, strict=False)
encoder.eval()

For Step 2 version (recommended for downstream use), see caithmac/MolE-RTD-25pct-S2.


Citation

@misc{mole-rtd-25pct,
  author = {caithmac},
  title  = {MolE-RTD-25pct: RTD pre-training of MolE at 25% masking rate on 415M ZINC molecules},
  year   = {2026},
  url    = {https://huggingface.co/caithmac/MolE-RTD-25pct}
}
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Paper for caithmac/MolE-RTD-25pct