Hugging Face's logo

Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
Back to feed
XiaofengAlgΒ 
posted an update 7 days ago
Post
2,738
πŸ“ MechVQA: a 4B open VLM that beats GPT-5 on mechanical engineering drawings (ICML 2026)

We release MechVQA β€” the first comprehensive benchmark and training resource for multimodal LLMs on mechanical engineering drawings β€” plus MechVL, a 4B domain-specialized VLM trained with SFT β†’ DAPO RL self-play.

πŸ“Š The benchmark: 13,515 bilingual (δΈ­/EN) QA pairs over 3,371 unique drawings, covering 10 subcategories β€” dimension & annotation, assembly relationship, geometric calculation, anomaly detection, consistency judgment, projection & multi-view, structure understanding, identification & counting, item localization, text & table. Train/validation splits are image-hash disjoint and decontaminated against the public eval benchmark.

πŸš€ The model: MechVL-4B-RL (Qwen3-VL-4B-Instruct β†’ full-param SFT β†’ two-stage DAPO self-play) reaches 84.85 total on MechVQA β€” ahead of every closed-source model we evaluated: GLM-4.6V 78.91 Β· Gemini-3-Pro-Preview 77.28 Β· MechVL-4B-SFT 76.36 Β· GPT-5 75.44. Reward = accuracy (LLM-as-a-judge) + format + quality; ablations show DAPO > GRPO > GSPO.

🎁 All open (Apache-2.0), with full curation lineage, SHA-256 manifests and a LLaMA-Factory-ready dataset_info.json:
β€’ Dataset: XiaofengAlg/MechVQA
β€’ RL checkpoint: XiaofengAlg/MechVL-4B-RL
β€’ SFT checkpoint: XiaofengAlg/MechVL-4B-SFT
β€’ Demo: XiaofengAlg/mechvqa-project
β€’ Paper: https://arxiv.org/abs/2605.30794

We hope MechVQA helps the community move beyond natural-image VQA into real industrial document understanding β€” feedback and contributions welcome!
dipankarsarkar

The most useful file in this release is your own exclusions log.

audit/exclusions.jsonl drops 2,555 records for exact image SHA-256 overlap with the public benchmark. 1,576 of them are on the train side, covering 122 distinct benchmark images. Across train and val it is 287 of the benchmark's 562 images.

The paper says the 8:1:1 split keeps every QA record from a drawing in one split. 122 benchmark drawings in the train pool does not square with that, unless the reviewed input built on 07-31 is a different pool from the paper's train split.

The dates are why it matters. MechVL-4B-RL's weights were uploaded on 2026-06-26. The decontaminated build is 2026-08-03.

Which pool trained the 84.85 checkpoint? If it was the earlier one, the clean number is the score on the 440 benchmark images that never appear on the train side. Have you split the total that way?

Β·
RZ-Q

@dipankarsarkar Thank you for checking this carefully. We confirmed that 122 of the 562 public benchmark drawings also appeared in the training data used for the checkpoint reporting 84.85%. Excluding these drawings and their associated QA pairs leaves 440 drawings and 732 QA pairs, on which MechVL-4B-RL scores 88.66% in the historical results. The August 3 decontaminated release was prepared later and was not used to train that checkpoint.

We traced the overlap to a missing global check when merging the QA-generation branches. The released splitting code checks image separation within a single run. Each branch was split independently by drawing, but we did not recheck image separation across the combined outputs. This allowed some drawings to appear in both training and evaluation. This was our oversight in enforcing the drawing-level split described in the paper, and we apologize for it.

Our QA-level check found no duplicate image-question pairs between training and evaluation after normalizing prompt formatting. The overlapping drawings were associated with different question texts, rather than duplicated evaluation QA examples.

The original evaluation contained 619 drawings and 1,413 QA pairs. In preparation for the open-source release, we conducted an additional round of quality screening, retaining 562 drawings and 1,185 QA pairs in the public benchmark. Below is the comparison with the requested 440-drawing subset, including all models reported in the paper. Scores are percentages.

Model Original: 619 drawings / 1,413 QA Retained: 562 drawings / 1,185 QA Excluding train-image overlap: 440 drawings / 732 QA
MechVL-4B-RL 84.85 86.67 88.66
MechVL-4B-SFT 76.36 80.25 82.38
GLM-4.6V 78.91 82.03 86.20
Gemini-3-Pro-Preview 77.28 81.18 85.38
GPT-5 75.44 78.99 85.79
Claude-Sonnet-4.5 71.20 74.77 81.15
Qwen3-VL-Plus 76.33 79.11 84.84
Qwen3-VL-32B-Instruct 76.50 80.00 84.02
Qwen3-VL-30B-A3B-Instruct 64.47 67.76 74.18
Qwen3-VL-4B-Instruct 60.23 64.39 69.81
MiniCPM-V-4.5 60.51 64.14 68.99
Mimo-VL-7B-SFT 59.66 64.81 70.08
InternVL3.5-8B 49.82 53.08 61.89
Gemma-3-27B-it 42.68 46.67 58.20
Llama-3.2-11B-Vision-Instruct 25.48 27.85 35.25
GPT-4o 63.06 66.58 74.32
GPT-4o-mini 45.65 48.69 57.38

MechVL-4B-RL remains the highest-scoring model among those reported in the paper in all three columns.
dipankarsarkar

@RZ-Q Thank you for running it and posting the full table. Owning it in public makes the benchmark more useful, not less.

Your table also lets anyone back out the 453 QA pairs you removed (1,185 minus 732).

Every outside model finds those 453 much harder than the clean 732.
GPT-5: 68.0 vs 85.8. GLM-4.6V: 75.3 vs 86.2.
Your own base, Qwen3-VL-4B: 55.6 vs 69.8, a 14.2 point gap.

The two MechVL checkpoints, the only models trained on those drawings, show gaps of 5.2 (RL) and 5.6 (SFT). No other model in the table is under 10.5.

So different question text did not stop the drawing itself from transferring.
On the overlap subset RL leads GLM-4.6V by 8.2 points. On the clean 440 the lead is 2.5, about 18 of 732 questions.

One small thing in the linked check: it keys on (data_source, image path), so the same drawing under two paths passes even within one run. Your audit/exclusions.jsonl already hashes with SHA-256, which is the key that caught it.

Is the per-subcategory breakdown on the 440 available? If the remaining lead sits in one or two categories, that says where domain training really pays.

Β·
RZ-Q

@dipankarsarkar Thanks for the follow-up. Here is the breakdown for the 440 drawings and 732 QA pairs with no training-image overlap, calculated from the historical results.

Subcategory QA pairs Qwen3-VL-4B GLM-4.6V MechVL-4B-RL
Identification & Counting 36 63.89 91.67 88.89
Dimension & Annotation 327 83.18 88.99 92.35
Text & Table 34 91.18 97.06 97.06
Item Localization 20 55.00 80.00 85.00
Structure Understanding 26 42.31 84.62 88.46
Geometric Calculation 151 67.55 90.07 88.74
Assembly Relationship 20 45.00 80.00 85.00
Projection & Multi-view 30 20.00 56.67 56.67
Anomaly Detection 29 65.52 86.21 96.55
Consistency Judgment 59 45.76 71.19 77.97
Overall 732 69.81 86.20 88.66

Against GLM-4.6V, MechVL-4B-RL leads in six subcategories and ties in two. The main contributions to the overall lead are Dimension & Annotation (+11 correct answers), Consistency Judgment (+4), and Anomaly Detection (+3).

Relative to the Qwen3-VL-4B base model, MechVL-4B-RL improves in all ten subcategories, gaining 18.85 percentage points overall. The largest gains are in Structure Understanding (+46.15 points), Assembly Relationship (+40.00), and Projection & Multi-view (+36.67).
dipankarsarkar

@RZ-Q Thank you. This is the table I hoped for, and it reconciles to the answer: 649 vs 631 of 732, an 18-answer lead.

In counts, most of that lead is one category.
Dimension & Annotation is 302 vs 291, so 11 of the 18.
Outside it, MechVL-4B-RL leads 347 to 340 on 405 questions, about 1.7 points.

Three of the six subcategory leads are a single question each: Item Localization, Structure Understanding, Assembly Relationship (n of 20 to 26).
Geometric Calculation, the second largest bucket at 151, goes to GLM-4.6V by 2.

Projection & Multi-view is the tie I find most telling.
RL lifts your 4B base from 6 to 17 of 30, exactly where GLM-4.6V sits.
On the hardest category, domain RL caught a general model but did not pass it.

So on the clean 440, the domain-training story is mostly a dimension-reading story.

How many of the 732 do MechVL-4B-RL and GLM-4.6V answer differently? With that discordant count, the 18 becomes a McNemar test, and you would know whether the clean lead is signal or inside the noise.
In this post