Hiro-Chemical-Insights

Hiro-Chemical-Insights is a multimodal model for chemical structure-text coreference in intellectual property documents. Given a patent page image with a boxed chemical structure, the model identifies the textual reference name for the structure and classifies the structure type.

The model is associated with the ACL 2026 paper:

Multimodal Chemical Structure-Text Coreference in Intellectual Property via Rule-guided Reinforcement Learning

• Paper: ACL 2026
• GitHub repository: Hiro-Chemical-Insights

Task

CheST focuses on multimodal chemical structure-text coreference in patent documents. The model receives a patent page image containing a highlighted chemical structure and predicts:

• the reference name or names associated with the highlighted structure;
• the structure type, such as specific compound, substituent, Markush structure, or Markush structure & substituent.

The expected answer format is:

\boxed{[reference name]: structure type}

For examples with multiple reference names, the names are returned as a list in the boxed answer.

Main Results

Main results are shown below.

Model	RefMatch Pass@1	StruCls Pass@1	All Pass@1	RefMatch Pass@all	StruCls Pass@all	All Pass@all
GPT-5	63.13	87.88	59.60	53.54	86.36	50.51
Claude-4.5-sonnet	57.07	78.79	48.99	45.96	73.74	36.87
Gemini-2.5-pro	75.25	83.84	73.23	66.16	82.32	63.13
GLM-4.5V	56.06	81.31	49.49	53.54	76.77	44.95
Qwen-vl-max	47.98	76.77	43.43	44.44	66.16	33.84
Qwen3-vl-8B	44.95	34.34	17.17	40.40	29.29	10.61
Hiro-Chemical-Insights	93.43	98.48	91.92	90.40	97.98	88.38

Compared with the strongest general MLLM baseline, Gemini-2.5-Pro, Hiro-Chemical-Insights improves All Pass@1 from 73.23 to 91.92 and All Pass@all from 63.13 to 88.38.

Usage

Install recent transformers, accelerate, torch, and Qwen-VL compatible image/video utilities in a GPU environment.

from PIL import Image
from transformers import AutoProcessor, Qwen3VLForConditionalGeneration

model_id = "PatSnap/Hiro-Chemical-Insights"

model = Qwen3VLForConditionalGeneration.from_pretrained(
    model_id,
    torch_dtype="auto",
    device_map="auto",
)
processor = AutoProcessor.from_pretrained(model_id)

image = Image.open("images/US20230002396A1/page_064_4_mol_with_box.jpeg")
prompt = """Please identify the reference name(s) of the chemical molecule within the blue rectangle in the image. Determine which category it belongs to: [Markush structure], [substituent], [specific compound], or [Markush structure, substituent].

- A Markush structure contains a fixed parent nucleus and at least one variable substituent with a defined range (for example, R1).
- A substituent is a component of a complete molecule and cannot form a complete molecule independently. Image context matters: a structure listed in table column "R" is a substituent.
- A Markush structure can itself act as a substituent; use both structure types in that case.
- In tables, a Markush structure can correspond to multiple names, starting at the same vertical height and proceeding downwards until the next structure.
- Return complete reference names. Join a table title and sequence number when needed (for example, "Formula IV"). Return all names if there are multiple. Return "None" if no name is visible.

End with exactly one answer in this format:
\\boxed{[reference name 1, reference name 2]: [structure type 1, structure type 2]}

Examples:
\\boxed{[Compound No. 55]: [substituent, Markush structure]}
\\boxed{[Compound 1, Compound 2]: [Markush structure]}
\\boxed{[3,6-Dichloropyridazine]: [specific compound]}
\\boxed{[None]: [specific compound]}"""

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": image},
            {"type": "text", "text": prompt},
        ],
    }
]

text = processor.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)
inputs = processor(text=[text], images=[image], return_tensors="pt").to(model.device)
generated_ids = model.generate(**inputs, max_new_tokens=4096)
output = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]
print(output)

The GitHub repository contains the released data files, image assets, inference script, and evaluation script.

Limitations

The model is intended for research on chemical structure-text coreference in patent documents. It should not be used as a standalone source of chemical, medical, legal, or regulatory decisions.

Citation

Please cite the ACL 2026 paper when using this model or the associated CheST benchmark:

@inproceedings{zhong-etal-2026-multimodal,
    title = "Multimodal Chemical Structure-Text Coreference in Intellectual Property via Rule-guided Reinforcement Learning",
    author = "Zhong, Hanmeng  and
      Wu, Wentao  and
      Chen, Linqing  and
      Zhou, Peng",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {ACL} 2026",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.findings-acl.1489/",
    pages = "29784--29796",
    ISBN = "979-8-89176-395-1",
}