| --- |
| license: apache-2.0 |
| library_name: transformers |
| pipeline_tag: image-text-to-text |
| base_model: Qwen/Qwen2.5-VL-3B-Instruct |
| tags: |
| - chemistry |
| - multimodal |
| - reasoning |
| - vision-language |
| - grpo |
| - qwen2_5_vl |
| - smiles |
| - iupac |
| - spectroscopy |
| language: |
| - en |
| --- |
| |
| # CheMM-R1 |
|
|
| CheMM-R1 is a chemistry-specific multimodal large language model (MLLM) for molecular structure recognition and spectral elucidation. It is built on top of **Qwen2.5-VL-3B-Instruct** and trained with **CheMMGRPO** — a domain-adapted Group Relative Policy Optimisation pipeline that combines a chemistry cold-start SFT stage with reinforcement learning driven by chemistry-specific reward functions. |
|
|
| The model is introduced in the paper *"CheMM-R1: Enhancing Chemical Structure Recognition and Elucidation with Reasoning Multimodal Large Language Models"*. |
|
|
| - Code & benchmark: https://github.com/liting980713/CheMM-R1 |
|
|
| ## Capabilities |
|
|
| CheMM-R1 is designed for the following tasks: |
|
|
| - **SmilesQA** — predict the SMILES string of a molecule from its 2D structure image. |
| - **IupacQA** — predict the IUPAC name of a molecule from its 2D structure image. |
| - **MwQA** — predict the molecular weight of a molecule from its 2D structure image. |
| - **SpectraQA** — predict the SMILES of a molecule from its spectral images (IR, ¹H-NMR, ¹³C-NMR, positive-ion MS, negative-ion MS). |
|
|
| The model produces explicit step-by-step reasoning in `<think>` tags, intermediate SMILES in `<smiles>` tags, and the final answer in `<answer>` tags. |
|
|
| ## Method |
|
|
| ### Base model |
|
|
| - Architecture: `Qwen2_5_VLForConditionalGeneration` (Qwen2.5-VL-3B-Instruct) |
| - Precision: bfloat16 |
| - Context length: up to 128K tokens, trained with max sequence length 20,480 (cold-start) and output length 5,120 (RL) |
|
|
| ### Training pipeline (CheMMGRPO) |
|
|
| 1. **Cold start (SFT)** — the base model is fine-tuned on 40,000 chemistry QA instances from CheMM-Bench with multimodal Chain-of-Thought reasoning distilled from Gemini-2.5-Pro, to inject organic chemistry knowledge and reasoning patterns. |
| 2. **Reinforcement learning (GRPO)** — the cold-started model is further optimised with GRPO using four chemistry-specific reward functions: |
| - `smiles_acc` — chemical validity of generated SMILES |
| - `atom_acc`, `func_group_acc` — structural accuracy |
| - `output_format` — format compliance with `<think>/<smiles>/<answer>` tags |
| - `answer_acc` — factual correctness via Levenshtein-based fuzzy match for textual answers (SMILES, IUPAC) and ±0.05 g/mol tolerance for molecular weights |
|
|
| ### Key hyperparameters |
|
|
| - Framework: MS-SWIFT |
| - Cold start: lr 1e-4, Adam (β1=0.9, β2=0.95, ε=1e-8), batch size 64, max model length 20,480, max image pixels 262,144 |
| - RL (GRPO): 4 rollouts per question, sampling temperature 1.0, lr 1e-4, KL coefficient β=0.001, batch size 48, max output length 5,120 |
| - Hardware: 8× H200 (141 GB) GPUs, ~12 hours total training |
|
|
| ## Dataset: CheMM-Bench |
|
|
| CheMM-R1 is trained and evaluated on **CheMM-Bench**, a multimodal chemistry reasoning benchmark with 48,500 long Chain-of-Thought reasoning steps across four tasks: |
|
|
| - 26,500 molecules for structure recognition (SmilesQA / IupacQA / MwQA) |
| - 22,000 molecules for structure elucidation (SpectraQA) with five spectral image types (IR, ¹H-NMR, ¹³C-NMR, +ion MS, −ion MS) |
|
|
| Molecules are derived from the Alberts spectroscopic dataset (USPTO reaction database). SMILES → 2D structure conversion is performed with RDKit; IUPAC names are sourced from PubChem. |
|
|
| ## Results |
|
|
| Averaged accuracy / F1 on CheMM-Bench (CheMM-R1 vs. strongest baselines): |
|
|
| | Model | SR Avg ACC | SR Avg F1 | SpectraQA ACC | SpectraQA F1 | Overall ACC | Overall F1 | |
| |--------------------|-----------:|----------:|--------------:|-------------:|------------:|-----------:| |
| | GPT-o3 | 5.78 | 10.94 | 1.50 | 2.96 | 3.34 | 6.46 | |
| | Gemini-2.5-Pro | 16.13 | 27.78 | 1.80 | 3.54 | 7.95 | 14.72 | |
| | Claude-Sonnet-4 | 1.99 | 3.91 | 1.60 | 3.15 | 1.77 | 3.48 | |
| | Grok-4 | 2.79 | 5.43 | 4.05 | 7.78 | 3.51 | 6.78 | |
| | Gemini-2.5-Flash | 8.58 | 15.80 | 1.10 | 2.18 | 4.31 | 8.26 | |
| | **CheMM-R1 (3B)** | **23.73** | **38.35** | **36.32** | **53.28** | **30.92** | **47.23** | |
|
|
| Tanimoto@1.0 structural match on SmilesQA / SpectraQA: |
|
|
| | Model | SmilesQA | SpectraQA | |
| |--------------------|---------:|----------:| |
| | Gemini-2.5-Pro | 44.52 | 11.46 | |
| | Grok-4 | 2.18 | 18.28 | |
| | ChemVLM-8B | 31.99 | — | |
| | **CheMM-R1 (3B)** | **60.00**| **56.57** | |
|
|
| See the paper for full tables including BLEU-1 and Levenshtein-distance similarity and ablations on cold-start vs. GRPO vs. CheMMGRPO. |
|
|
| ## Usage |
|
|
| ```python |
| from transformers import AutoProcessor, Qwen2_5_VLForConditionalGeneration |
| from PIL import Image |
| import torch |
| |
| model_id = "zzha6204/CheMM-R1" |
| |
| processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True) |
| model = Qwen2_5_VLForConditionalGeneration.from_pretrained( |
| model_id, |
| torch_dtype=torch.bfloat16, |
| device_map="auto", |
| trust_remote_code=True, |
| ) |
| |
| image = Image.open("molecule.png") |
| messages = [ |
| { |
| "role": "user", |
| "content": [ |
| {"type": "image", "image": image}, |
| {"type": "text", "text": "What is the SMILES representation of this molecule?"}, |
| ], |
| } |
| ] |
| |
| text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) |
| inputs = processor(text=[text], images=[image], return_tensors="pt").to(model.device) |
| |
| output_ids = model.generate(**inputs, max_new_tokens=2048) |
| response = processor.batch_decode( |
| output_ids[:, inputs.input_ids.shape[1]:], skip_special_tokens=True |
| )[0] |
| print(response) |
| ``` |
|
|
| For SpectraQA, pass all available spectral images (IR, ¹H-NMR, ¹³C-NMR, +ion MS, −ion MS) as a multi-image message. |
|
|
| ## Output format |
|
|
| CheMM-R1 is trained to produce: |
|
|
| ``` |
| <think> step-by-step chemical reasoning </think> |
| <smiles> intermediate SMILES </smiles> |
| <answer> final answer </answer> |
| ``` |
|
|
| Downstream parsers should extract the content of `<answer>` as the final prediction. |
|
|
| ## Intended use and limitations |
|
|
| - Intended for **research on multimodal chemistry reasoning**: molecular structure recognition and spectral elucidation of small organic molecules. |
| - Molecules are drawn from the USPTO-derived Alberts dataset; out-of-distribution performance on larger natural products, organometallics, or experimentally noisy real-world spectra is not guaranteed. |
| - Outputs — including SMILES, IUPAC names, molecular weights, and reasoning traces — may be incorrect and **must not be used for safety-critical decisions in chemistry or medicinal research** without expert verification. |
| - The model is derived from Qwen2.5-VL-3B-Instruct and inherits its license and any biases of the base model and distilled reasoning data (Gemini-2.5-Pro). |
|
|
| ## Citation |
|
|
| If you use CheMM-R1 or CheMM-Bench in your research, please cite: |
|
|
| ```bibtex |
| @inproceedings{huang2025chemmr1, |
| title = {CheMM-R1: Enhancing Chemical Structure Recognition and Elucidation with Reasoning Multimodal Large Language Models}, |
| author = {Huang, Liting and Zhang, Zhihao and Wang, Shoujin}, |
| year = {2025} |
| } |
| ``` |
|
|
| ## Acknowledgements |
|
|
| - Base model: [Qwen2.5-VL-3B-Instruct](https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct) |
| - Training framework: [MS-SWIFT](https://github.com/modelscope/ms-swift) |
| - Spectroscopic data: Alberts et al. (USPTO-derived), PubChem, RDKit |
|
|
