Benchmarking and Mechanistic Analysis of Vision-Language Models for Cross-Depiction Assembly Instruction Alignment

We introduce IKEA-Bench, the first cross-depiction alignment benchmark for 2D-manual-based assembly guidance. Our benchmark comprises 1,623 questions across 6 task types on 29 IKEA furniture products, designed to systematically evaluate whether VLMs can align abstract schematic assembly diagrams with photorealistic assembly video — a fundamental capability for mixed-reality assembly assistants.

We evaluate 19 VLMs (2B–38B), including Gemini 3.1 Pro and Gemini 3 Flash, under three alignment strategies (Visual, Visual+Text, Text Only). Our key findings are: (1) text descriptions can recover assembly instruction understanding (+23.6 pp on diagram ordering), but counter-intuitively hurt diagram-to-video alignment (−3.1 pp on step recognition); (2) architecture family predicts alignment accuracy more reliably than parameter count — at ~8B scale, each Qwen generation yields larger gains than tripling parameters within a generation; (3) video understanding remains a hard bottleneck that no alignment strategy overcomes, with even the best proprietary model reaching only 71.1%.

To understand why models fail, we conduct a three-layer mechanistic analysis probing ViT representations, LLM hidden states, and attention routing. We find that diagrams and video occupy completely disjoint subspaces at the ViT level (CKA ≈ 0), and that adding text actively redirects both hidden-state reliance and attention away from visual modalities rather than supplementing them. Our results localize the bottleneck to the visual encoder and motivate cross-depiction contrastive pre-training as a direct path to closing this gap.