README.md

---
license: apache-2.0
library_name: transformers
pipeline_tag: image-text-to-text
base_model: Qwen/Qwen2.5-VL-3B-Instruct
tags:
  - chemistry
  - multimodal
  - reasoning
  - vision-language
  - grpo
  - qwen2_5_vl
  - smiles
  - iupac
  - spectroscopy
language:
  - en
---

# CheMM-R1

CheMM-R1 is a chemistry-specific multimodal large language model (MLLM) for molecular structure recognition and spectral elucidation. It is built on top of **Qwen2.5-VL-3B-Instruct** and trained with **CheMMGRPO** — a domain-adapted Group Relative Policy Optimisation pipeline that combines a chemistry cold-start SFT stage with reinforcement learning driven by chemistry-specific reward functions.

The model is introduced in the paper *"CheMM-R1: Enhancing Chemical Structure Recognition and Elucidation with Reasoning Multimodal Large Language Models"*.

- Code & benchmark: https://github.com/liting980713/CheMM-R1

## Capabilities

CheMM-R1 is designed for the following tasks:

- **SmilesQA** — predict the SMILES string of a molecule from its 2D structure image.
- **IupacQA** — predict the IUPAC name of a molecule from its 2D structure image.
- **MwQA** — predict the molecular weight of a molecule from its 2D structure image.
- **SpectraQA** — predict the SMILES of a molecule from its spectral images (IR, ¹H-NMR, ¹³C-NMR, positive-ion MS, negative-ion MS).

The model produces explicit step-by-step reasoning in `<think>` tags, intermediate SMILES in `<smiles>` tags, and the final answer in `<answer>` tags.

## Method

### Base model

- Architecture: `Qwen2_5_VLForConditionalGeneration` (Qwen2.5-VL-3B-Instruct)
- Precision: bfloat16
- Context length: up to 128K tokens, trained with max sequence length 20,480 (cold-start) and output length 5,120 (RL)

### Training pipeline (CheMMGRPO)

1. **Cold start (SFT)** — the base model is fine-tuned on 40,000 chemistry QA instances from CheMM-Bench with multimodal Chain-of-Thought reasoning distilled from Gemini-2.5-Pro, to inject organic chemistry knowledge and reasoning patterns.
2. **Reinforcement learning (GRPO)** — the cold-started model is further optimised with GRPO using four chemistry-specific reward functions:
   - `smiles_acc` — chemical validity of generated SMILES
   - `atom_acc`, `func_group_acc` — structural accuracy
   - `output_format` — format compliance with `<think>/<smiles>/<answer>` tags
   - `answer_acc` — factual correctness via Levenshtein-based fuzzy match for textual answers (SMILES, IUPAC) and ±0.05 g/mol tolerance for molecular weights

### Key hyperparameters

- Framework: MS-SWIFT
- Cold start: lr 1e-4, Adam (β1=0.9, β2=0.95, ε=1e-8), batch size 64, max model length 20,480, max image pixels 262,144
- RL (GRPO): 4 rollouts per question, sampling temperature 1.0, lr 1e-4, KL coefficient β=0.001, batch size 48, max output length 5,120
- Hardware: 8× H200 (141 GB) GPUs, ~12 hours total training

## Dataset: CheMM-Bench

CheMM-R1 is trained and evaluated on **CheMM-Bench**, a multimodal chemistry reasoning benchmark with 48,500 long Chain-of-Thought reasoning steps across four tasks:

- 26,500 molecules for structure recognition (SmilesQA / IupacQA / MwQA)
- 22,000 molecules for structure elucidation (SpectraQA) with five spectral image types (IR, ¹H-NMR, ¹³C-NMR, +ion MS, −ion MS)

Molecules are derived from the Alberts spectroscopic dataset (USPTO reaction database). SMILES → 2D structure conversion is performed with RDKit; IUPAC names are sourced from PubChem.

## Results

Averaged accuracy / F1 on CheMM-Bench (CheMM-R1 vs. strongest baselines):

| Model              | SR Avg ACC | SR Avg F1 | SpectraQA ACC | SpectraQA F1 | Overall ACC | Overall F1 |
|--------------------|-----------:|----------:|--------------:|-------------:|------------:|-----------:|
| GPT-o3             |  5.78      | 10.94     | 1.50          | 2.96         | 3.34        | 6.46       |
| Gemini-2.5-Pro     | 16.13      | 27.78     | 1.80          | 3.54         | 7.95        | 14.72      |
| Claude-Sonnet-4    |  1.99      | 3.91      | 1.60          | 3.15         | 1.77        | 3.48       |
| Grok-4             |  2.79      | 5.43      | 4.05          | 7.78         | 3.51        | 6.78       |
| Gemini-2.5-Flash   |  8.58      | 15.80     | 1.10          | 2.18         | 4.31        | 8.26       |
| **CheMM-R1 (3B)**  | **23.73**  | **38.35** | **36.32**     | **53.28**    | **30.92**   | **47.23**  |

Tanimoto@1.0 structural match on SmilesQA / SpectraQA:

| Model              | SmilesQA | SpectraQA |
|--------------------|---------:|----------:|
| Gemini-2.5-Pro     | 44.52    | 11.46     |
| Grok-4             |  2.18    | 18.28     |
| ChemVLM-8B         | 31.99    |   —       |
| **CheMM-R1 (3B)**  | **60.00**| **56.57** |

See the paper for full tables including BLEU-1 and Levenshtein-distance similarity and ablations on cold-start vs. GRPO vs. CheMMGRPO.

## Usage

```python
from transformers import AutoProcessor, Qwen2_5_VLForConditionalGeneration
from PIL import Image
import torch

model_id = "zzha6204/CheMM-R1"

processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)

image = Image.open("molecule.png")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": image},
            {"type": "text", "text": "What is the SMILES representation of this molecule?"},
        ],
    }
]

text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = processor(text=[text], images=[image], return_tensors="pt").to(model.device)

output_ids = model.generate(**inputs, max_new_tokens=2048)
response = processor.batch_decode(
    output_ids[:, inputs.input_ids.shape[1]:], skip_special_tokens=True
)[0]
print(response)
```

For SpectraQA, pass all available spectral images (IR, ¹H-NMR, ¹³C-NMR, +ion MS, −ion MS) as a multi-image message.

## Output format

CheMM-R1 is trained to produce:

```
<think> step-by-step chemical reasoning </think>
<smiles> intermediate SMILES </smiles>
<answer> final answer </answer>
```

Downstream parsers should extract the content of `<answer>` as the final prediction.

## Intended use and limitations

- Intended for **research on multimodal chemistry reasoning**: molecular structure recognition and spectral elucidation of small organic molecules.
- Molecules are drawn from the USPTO-derived Alberts dataset; out-of-distribution performance on larger natural products, organometallics, or experimentally noisy real-world spectra is not guaranteed.
- Outputs — including SMILES, IUPAC names, molecular weights, and reasoning traces — may be incorrect and **must not be used for safety-critical decisions in chemistry or medicinal research** without expert verification.
- The model is derived from Qwen2.5-VL-3B-Instruct and inherits its license and any biases of the base model and distilled reasoning data (Gemini-2.5-Pro).

## Citation

If you use CheMM-R1 or CheMM-Bench in your research, please cite:

```bibtex
@inproceedings{huang2025chemmr1,
  title     = {CheMM-R1: Enhancing Chemical Structure Recognition and Elucidation with Reasoning Multimodal Large Language Models},
  author    = {Huang, Liting and Zhang, Zhihao and Wang, Shoujin},
  year      = {2025}
}
```

## Acknowledgements

- Base model: [Qwen2.5-VL-3B-Instruct](https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct)
- Training framework: [MS-SWIFT](https://github.com/modelscope/ms-swift)
- Spectroscopic data: Alberts et al. (USPTO-derived), PubChem, RDKit