Add model card

README.md

@@ -0,0 +1,171 @@
+---
+license: apache-2.0
+library_name: transformers
+pipeline_tag: image-text-to-text
+base_model: Qwen/Qwen2.5-VL-3B-Instruct
+tags:
+  - chemistry
+  - multimodal
+  - reasoning
+  - vision-language
+  - grpo
+  - qwen2_5_vl
+  - smiles
+  - iupac
+  - spectroscopy
+language:
+  - en
+---
+
+# CheMM-R1
+
+CheMM-R1 is a chemistry-specific multimodal large language model (MLLM) for molecular structure recognition and spectral elucidation. It is built on top of **Qwen2.5-VL-3B-Instruct** and trained with **CheMMGRPO** — a domain-adapted Group Relative Policy Optimisation pipeline that combines a chemistry cold-start SFT stage with reinforcement learning driven by chemistry-specific reward functions.
+
+The model is introduced in the paper *"CheMM-R1: Enhancing Chemical Structure Recognition and Elucidation with Reasoning Multimodal Large Language Models"*.
+
+- Code & benchmark: https://github.com/ZhihaoZhang97/CheMM-R1
+
+## Capabilities
+
+CheMM-R1 is designed for the following tasks:
+
+- **SmilesQA** — predict the SMILES string of a molecule from its 2D structure image.
+- **IupacQA** — predict the IUPAC name of a molecule from its 2D structure image.
+- **MwQA** — predict the molecular weight of a molecule from its 2D structure image.
+- **SpectraQA** — predict the SMILES of a molecule from its spectral images (IR, ¹H-NMR, ¹³C-NMR, positive-ion MS, negative-ion MS).
+
+The model produces explicit step-by-step reasoning in `<think>` tags, intermediate SMILES in `<smiles>` tags, and the final answer in `<answer>` tags.
+
+## Method
+
+### Base model
+
+- Architecture: `Qwen2_5_VLForConditionalGeneration` (Qwen2.5-VL-3B-Instruct)
+- Precision: bfloat16
+- Context length: up to 128K tokens, trained with max sequence length 20,480 (cold-start) and output length 5,120 (RL)
+
+### Training pipeline (CheMMGRPO)
+
+1. **Cold start (SFT)** — the base model is fine-tuned on 40,000 chemistry QA instances from CheMM-Bench with multimodal Chain-of-Thought reasoning distilled from Gemini-2.5-Pro, to inject organic chemistry knowledge and reasoning patterns.
+2. **Reinforcement learning (GRPO)** — the cold-started model is further optimised with GRPO using four chemistry-specific reward functions:
+   - `smiles_acc` — chemical validity of generated SMILES
+   - `atom_acc`, `func_group_acc` — structural accuracy
+   - `output_format` — format compliance with `<think>/<smiles>/<answer>` tags
+   - `answer_acc` — factual correctness via Levenshtein-based fuzzy match for textual answers (SMILES, IUPAC) and ±0.05 g/mol tolerance for molecular weights
+
+### Key hyperparameters
+
+- Framework: MS-SWIFT
+- Cold start: lr 1e-4, Adam (β1=0.9, β2=0.95, ε=1e-8), batch size 64, max model length 20,480, max image pixels 262,144
+- RL (GRPO): 4 rollouts per question, sampling temperature 1.0, lr 1e-4, KL coefficient β=0.001, batch size 48, max output length 5,120
+- Hardware: 8× H200 (141 GB) GPUs, ~12 hours total training
+
+## Dataset: CheMM-Bench
+
+CheMM-R1 is trained and evaluated on **CheMM-Bench**, a multimodal chemistry reasoning benchmark with 48,500 long Chain-of-Thought reasoning steps across four tasks:
+
+- 26,500 molecules for structure recognition (SmilesQA / IupacQA / MwQA)
+- 22,000 molecules for structure elucidation (SpectraQA) with five spectral image types (IR, ¹H-NMR, ¹³C-NMR, +ion MS, −ion MS)
+
+Molecules are derived from the Alberts spectroscopic dataset (USPTO reaction database). SMILES → 2D structure conversion is performed with RDKit; IUPAC names are sourced from PubChem.
+
+## Results
+
+Averaged accuracy / F1 on CheMM-Bench (CheMM-R1 vs. strongest baselines):
+
+| Model              | SR Avg ACC | SR Avg F1 | SpectraQA ACC | SpectraQA F1 | Overall ACC | Overall F1 |
+|--------------------|-----------:|----------:|--------------:|-------------:|------------:|-----------:|
+| GPT-o3             |  5.78      | 10.94     | 1.50          | 2.96         | 3.34        | 6.46       |
+| Gemini-2.5-Pro     | 16.13      | 27.78     | 1.80          | 3.54         | 7.95        | 14.72      |
+| Claude-Sonnet-4    |  1.99      | 3.91      | 1.60          | 3.15         | 1.77        | 3.48       |
+| Grok-4             |  2.79      | 5.43      | 4.05          | 7.78         | 3.51        | 6.78       |
+| Gemini-2.5-Flash   |  8.58      | 15.80     | 1.10          | 2.18         | 4.31        | 8.26       |
+| **CheMM-R1 (3B)**  | **23.73**  | **38.35** | **36.32**     | **53.28**    | **30.92**   | **47.23**  |
+
+Tanimoto@1.0 structural match on SmilesQA / SpectraQA:
+
+| Model              | SmilesQA | SpectraQA |
+|--------------------|---------:|----------:|
+| Gemini-2.5-Pro     | 44.52    | 11.46     |
+| Grok-4             |  2.18    | 18.28     |
+| ChemVLM-8B         | 31.99    |   —       |
+| **CheMM-R1 (3B)**  | **60.00**| **56.57** |
+
+See the paper for full tables including BLEU-1 and Levenshtein-distance similarity and ablations on cold-start vs. GRPO vs. CheMMGRPO.
+
+## Usage
+
+```python
+from transformers import AutoProcessor, Qwen2_5_VLForConditionalGeneration
+from PIL import Image
+import torch
+
+model_id = "zzha6204/CheMM-R1"
+
+processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
+model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
+    model_id,
+    torch_dtype=torch.bfloat16,
+    device_map="auto",
+    trust_remote_code=True,
+)
+
+image = Image.open("molecule.png")
+messages = [
+    {
+        "role": "user",
+        "content": [
+            {"type": "image", "image": image},
+            {"type": "text", "text": "What is the SMILES representation of this molecule?"},
+        ],
+    }
+]
+
+text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
+inputs = processor(text=[text], images=[image], return_tensors="pt").to(model.device)
+
+output_ids = model.generate(**inputs, max_new_tokens=2048)
+response = processor.batch_decode(
+    output_ids[:, inputs.input_ids.shape[1]:], skip_special_tokens=True
+)[0]
+print(response)
+```
+
+For SpectraQA, pass all available spectral images (IR, ¹H-NMR, ¹³C-NMR, +ion MS, −ion MS) as a multi-image message.
+
+## Output format
+
+CheMM-R1 is trained to produce:
+
+```
+<think> step-by-step chemical reasoning </think>
+<smiles> intermediate SMILES </smiles>
+<answer> final answer </answer>
+```
+
+Downstream parsers should extract the content of `<answer>` as the final prediction.
+
+## Intended use and limitations
+
+- Intended for **research on multimodal chemistry reasoning**: molecular structure recognition and spectral elucidation of small organic molecules.
+- Molecules are drawn from the USPTO-derived Alberts dataset; out-of-distribution performance on larger natural products, organometallics, or experimentally noisy real-world spectra is not guaranteed.
+- Outputs — including SMILES, IUPAC names, molecular weights, and reasoning traces — may be incorrect and **must not be used for safety-critical decisions in chemistry or medicinal research** without expert verification.
+- The model is derived from Qwen2.5-VL-3B-Instruct and inherits its license and any biases of the base model and distilled reasoning data (Gemini-2.5-Pro).
+
+## Citation
+
+If you use CheMM-R1 or CheMM-Bench in your research, please cite:
+
+```bibtex
+@inproceedings{huang2025chemmr1,
+  title     = {CheMM-R1: Enhancing Chemical Structure Recognition and Elucidation with Reasoning Multimodal Large Language Models},
+  author    = {Huang, Liting and Zhang, Zhihao and Wang, Shoujin},
+  year      = {2025}
+}
+```
+
+## Acknowledgements
+
+- Base model: [Qwen2.5-VL-3B-Instruct](https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct)
+- Training framework: [MS-SWIFT](https://github.com/modelscope/ms-swift)
+- Spectroscopic data: Alberts et al. (USPTO-derived), PubChem, RDKit