molcrawl-molecule-nat-lang-mol-instructions-bert-medium

Model Description

GPT-2 medium (345M parameters) fine-tuned on molecule-oriented instruction data from Mol-Instructions, starting from the molcrawl-molecule-nat-lang-gpt2-medium pre-trained model.

Datasets

• Mol-Instructions: https://huggingface.co/datasets/zjunlp/Mol-Instructions (Fine-tuning dataset)

• Model Type: bert

• Data Type: Molecule-NL

• Training Date: 2026-05-15

Usage

from transformers import AutoModelForMaskedLM, AutoTokenizer
import torch

model = AutoModelForMaskedLM.from_pretrained("kojima-lab/molcrawl-molecule-nat-lang-mol-instructions-bert-medium")
tokenizer = AutoTokenizer.from_pretrained("kojima-lab/molcrawl-molecule-nat-lang-mol-instructions-bert-medium")

# Predict masked token
# Use tokenizer.mask_token instead of hardcoded "[MASK]":
# BERT-style tokenizers vary ("[MASK]", "<mask>", etc.)
if tokenizer.mask_token is None:
    raise ValueError("This tokenizer has no mask_token; masked LM inference is not supported.")
prompt = "your input {MASK} sequence".replace("{MASK}", tokenizer.mask_token)
inputs = tokenizer(prompt, return_tensors="pt")
mask_index = (inputs["input_ids"] == tokenizer.mask_token_id).nonzero(as_tuple=True)[1]

with torch.no_grad():
    outputs = model(**inputs)
logits = outputs.logits

predicted_token_id = logits[0, mask_index].argmax(dim=-1)
predicted_token = tokenizer.decode(predicted_token_id)
result = prompt.replace(tokenizer.mask_token, predicted_token)
print(f"Predicted: {result}")

Source Code

Training pipeline, configuration files, and data preparation scripts are available in the MolCrawl GitHub repository: https://github.com/mmai-framework-lab/MolCrawl

License

This model is released under the APACHE-2.0 license.

Citation

If you use this model, please cite:

@misc{molcrawl_molecule_nat_lang_mol_instructions_bert_medium,
  title={molcrawl-molecule-nat-lang-mol-instructions-bert-medium},
  author={{RIKEN}},
  year={2026},
  publisher={{Hugging Face}},
  url={{https://huggingface.co/kojima-lab/molcrawl-molecule-nat-lang-mol-instructions-bert-medium}}
}

Example Output

End-to-end test downloaded from this repo on CPU. This BERT encoder produces context-aware embeddings; below shows that semantically related molecule descriptions sit closer in embedding space than unrelated ones.

import torch
from transformers import AutoTokenizer, AutoModelForMaskedLM

REPO_ID = "kojima-lab/molcrawl-molecule-nat-lang-mol-instructions-bert-medium"
tokenizer = AutoTokenizer.from_pretrained(REPO_ID)
model = AutoModelForMaskedLM.from_pretrained(REPO_ID)
model.eval()

def embed(text):
    inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=128)
    with torch.no_grad():
        out = model.bert(**inputs)  # encoder only; pooler is unused / random-init
    mask = inputs["attention_mask"][0].unsqueeze(-1).float()
    return ((out.last_hidden_state[0] * mask).sum(0) / mask.sum())

texts = [
    "Aspirin is an anti-inflammatory drug.",
    "Ibuprofen is an anti-inflammatory drug.",
    "Glucose is a simple sugar.",
    "DNA stores genetic information.",
]
embs = torch.stack([embed(t) for t in texts])
embs = embs / embs.norm(dim=-1, keepdim=True)

for i in range(len(texts)):
    for j in range(i + 1, len(texts)):
        print(f"sim('{texts[i][:24]}...', '{texts[j][:24]}...') = {(embs[i] @ embs[j]).item():.3f}")
# Expected (approximately):
#   sim('Aspirin is an anti-infl...', 'Ibuprofen is an anti-inf...') = 0.980
#   sim('Aspirin is an anti-infl...', 'Glucose is a simple suga...') = 0.964
#   sim('Aspirin is an anti-infl...', 'DNA stores genetic infor...') = 0.921
#   sim('Ibuprofen is an anti-in...', 'Glucose is a simple suga...') = 0.947
#   sim('Ibuprofen is an anti-in...', 'DNA stores genetic infor...') = 0.885
#   sim('Glucose is a simple sug...', 'DNA stores genetic infor...') = 0.957

Note on MLM head: The fine-tune was done in MLM mode but on a sequence distribution where the MLM head's per-position predictions are not meaningful out of context. For typical use, treat this model as an encoder that produces text embeddings; for downstream tasks, add a task-specific head on top.