Post
24
📐 MechVQA: a 4B open VLM that beats GPT-5 on mechanical engineering drawings (ICML 2026)
We release MechVQA — the first comprehensive benchmark and training resource for multimodal LLMs on mechanical engineering drawings — plus MechVL, a 4B domain-specialized VLM trained with SFT → DAPO RL self-play.
📊 The benchmark: 13,515 bilingual (中/EN) QA pairs over 3,371 unique drawings, covering 10 subcategories — dimension & annotation, assembly relationship, geometric calculation, anomaly detection, consistency judgment, projection & multi-view, structure understanding, identification & counting, item localization, text & table. Train/validation splits are image-hash disjoint and decontaminated against the public eval benchmark.
🚀 The model: MechVL-4B-RL (Qwen3-VL-4B-Instruct → full-param SFT → two-stage DAPO self-play) reaches 84.85 total on MechVQA — ahead of every closed-source model we evaluated: GLM-4.6V 78.91 · Gemini-3-Pro-Preview 77.28 · MechVL-4B-SFT 76.36 · GPT-5 75.44. Reward = accuracy (LLM-as-a-judge) + format + quality; ablations show DAPO > GRPO > GSPO.
🎁 All open (Apache-2.0), with full curation lineage, SHA-256 manifests and a LLaMA-Factory-ready dataset_info.json:
• Dataset: XiaofengAlg/MechVQA
• RL checkpoint: XiaofengAlg/MechVL-4B-RL
• SFT checkpoint: XiaofengAlg/MechVL-4B-SFT
• Demo: XiaofengAlg/mechvqa-project
• Paper: https://arxiv.org/abs/2605.30794
We hope MechVQA helps the community move beyond natural-image VQA into real industrial document understanding — feedback and contributions welcome!
We release MechVQA — the first comprehensive benchmark and training resource for multimodal LLMs on mechanical engineering drawings — plus MechVL, a 4B domain-specialized VLM trained with SFT → DAPO RL self-play.
📊 The benchmark: 13,515 bilingual (中/EN) QA pairs over 3,371 unique drawings, covering 10 subcategories — dimension & annotation, assembly relationship, geometric calculation, anomaly detection, consistency judgment, projection & multi-view, structure understanding, identification & counting, item localization, text & table. Train/validation splits are image-hash disjoint and decontaminated against the public eval benchmark.
🚀 The model: MechVL-4B-RL (Qwen3-VL-4B-Instruct → full-param SFT → two-stage DAPO self-play) reaches 84.85 total on MechVQA — ahead of every closed-source model we evaluated: GLM-4.6V 78.91 · Gemini-3-Pro-Preview 77.28 · MechVL-4B-SFT 76.36 · GPT-5 75.44. Reward = accuracy (LLM-as-a-judge) + format + quality; ablations show DAPO > GRPO > GSPO.
🎁 All open (Apache-2.0), with full curation lineage, SHA-256 manifests and a LLaMA-Factory-ready dataset_info.json:
• Dataset: XiaofengAlg/MechVQA
• RL checkpoint: XiaofengAlg/MechVL-4B-RL
• SFT checkpoint: XiaofengAlg/MechVL-4B-SFT
• Demo: XiaofengAlg/mechvqa-project
• Paper: https://arxiv.org/abs/2605.30794
We hope MechVQA helps the community move beyond natural-image VQA into real industrial document understanding — feedback and contributions welcome!
