qian kou

RZ-Q

AI & ML interests

None yet

Recent Activity

repliedto XiaofengAlg's post 1 day ago

📐 MechVQA: a 4B open VLM that beats GPT-5 on mechanical engineering drawings (ICML 2026) We release MechVQA — the first comprehensive benchmark and training resource for multimodal LLMs on mechanical engineering drawings — plus MechVL, a 4B domain-specialized VLM trained with SFT → DAPO RL self-play. 📊 The benchmark: 13,515 bilingual (中/EN) QA pairs over 3,371 unique drawings, covering 10 subcategories — dimension & annotation, assembly relationship, geometric calculation, anomaly detection, consistency judgment, projection & multi-view, structure understanding, identification & counting, item localization, text & table. Train/validation splits are image-hash disjoint and decontaminated against the public eval benchmark. 🚀 The model: MechVL-4B-RL (Qwen3-VL-4B-Instruct → full-param SFT → two-stage DAPO self-play) reaches 84.85 total on MechVQA — ahead of every closed-source model we evaluated: GLM-4.6V 78.91 · Gemini-3-Pro-Preview 77.28 · MechVL-4B-SFT 76.36 · GPT-5 75.44. Reward = accuracy (LLM-as-a-judge) + format + quality; ablations show DAPO > GRPO > GSPO. 🎁 All open (Apache-2.0), with full curation lineage, SHA-256 manifests and a LLaMA-Factory-ready dataset_info.json: • Dataset: https://huggingface.co/datasets/XiaofengAlg/MechVQA • RL checkpoint: https://huggingface.co/XiaofengAlg/MechVL-4B-RL • SFT checkpoint: https://huggingface.co/XiaofengAlg/MechVL-4B-SFT • Demo: https://huggingface.co/spaces/XiaofengAlg/mechvqa-project • Paper: https://arxiv.org/abs/2605.30794 We hope MechVQA helps the community move beyond natural-image VQA into real industrial document understanding — feedback and contributions welcome!

repliedto XiaofengAlg's post 3 days ago

upvoted a paper about 1 month ago

DriveZero: End-to-End Driving Beyond Human Demonstrations

View all activity

Organizations

None yet

replied to XiaofengAlg's post 1 day ago

@dipankarsarkar Thanks for the follow-up. Here is the breakdown for the 440 drawings and 732 QA pairs with no training-image overlap, calculated from the historical results.

Subcategory	QA pairs	Qwen3-VL-4B	GLM-4.6V	MechVL-4B-RL
Identification & Counting	36	63.89	91.67	88.89
Dimension & Annotation	327	83.18	88.99	92.35
Text & Table	34	91.18	97.06	97.06
Item Localization	20	55.00	80.00	85.00
Structure Understanding	26	42.31	84.62	88.46
Geometric Calculation	151	67.55	90.07	88.74
Assembly Relationship	20	45.00	80.00	85.00
Projection & Multi-view	30	20.00	56.67	56.67
Anomaly Detection	29	65.52	86.21	96.55
Consistency Judgment	59	45.76	71.19	77.97
Overall	732	69.81	86.20	88.66

Against GLM-4.6V, MechVL-4B-RL leads in six subcategories and ties in two. The main contributions to the overall lead are Dimension & Annotation (+11 correct answers), Consistency Judgment (+4), and Anomaly Detection (+3).

Relative to the Qwen3-VL-4B base model, MechVL-4B-RL improves in all ten subcategories, gaining 18.85 percentage points overall. The largest gains are in Structure Understanding (+46.15 points), Assembly Relationship (+40.00), and Projection & Multi-view (+36.67).

replied to XiaofengAlg's post 3 days ago

@dipankarsarkar Thank you for checking this carefully. We confirmed that 122 of the 562 public benchmark drawings also appeared in the training data used for the checkpoint reporting 84.85%. Excluding these drawings and their associated QA pairs leaves 440 drawings and 732 QA pairs, on which MechVL-4B-RL scores 88.66% in the historical results. The August 3 decontaminated release was prepared later and was not used to train that checkpoint.

We traced the overlap to a missing global check when merging the QA-generation branches. The released splitting code checks image separation within a single run. Each branch was split independently by drawing, but we did not recheck image separation across the combined outputs. This allowed some drawings to appear in both training and evaluation. This was our oversight in enforcing the drawing-level split described in the paper, and we apologize for it.

Our QA-level check found no duplicate image-question pairs between training and evaluation after normalizing prompt formatting. The overlapping drawings were associated with different question texts, rather than duplicated evaluation QA examples.

The original evaluation contained 619 drawings and 1,413 QA pairs. In preparation for the open-source release, we conducted an additional round of quality screening, retaining 562 drawings and 1,185 QA pairs in the public benchmark. Below is the comparison with the requested 440-drawing subset, including all models reported in the paper. Scores are percentages.

Model	Original: 619 drawings / 1,413 QA	Retained: 562 drawings / 1,185 QA	Excluding train-image overlap: 440 drawings / 732 QA
MechVL-4B-RL	84.85	86.67	88.66
MechVL-4B-SFT	76.36	80.25	82.38
GLM-4.6V	78.91	82.03	86.20
Gemini-3-Pro-Preview	77.28	81.18	85.38
GPT-5	75.44	78.99	85.79
Claude-Sonnet-4.5	71.20	74.77	81.15
Qwen3-VL-Plus	76.33	79.11	84.84
Qwen3-VL-32B-Instruct	76.50	80.00	84.02
Qwen3-VL-30B-A3B-Instruct	64.47	67.76	74.18
Qwen3-VL-4B-Instruct	60.23	64.39	69.81
MiniCPM-V-4.5	60.51	64.14	68.99
Mimo-VL-7B-SFT	59.66	64.81	70.08
InternVL3.5-8B	49.82	53.08	61.89
Gemma-3-27B-it	42.68	46.67	58.20
Llama-3.2-11B-Vision-Instruct	25.48	27.85	35.25
GPT-4o	63.06	66.58	74.32
GPT-4o-mini	45.65	48.69	57.38