GeoToken — Implicit Geometric Grounding for Ctrl-World (results bucket)

Inject the robot's exact 3D shape (CAD + differentiable FK, canonical robot-base frame) into a frozen-ish Ctrl-World video world model via zero-init gated cross-attention geometry tokens — no camera extrinsics, no rendering — and let multi-view attention learn the 3D→2D projection implicitly.

Systems compared

All on DROID droid_full (10k train eps), 60 val eps, GT-action replay, seed 0.

Headline result (see RESULTS.md)

Honest verdict: on in-distribution GT-action replay, geometry grounding does not improve the world model (slightly worse across the board). All mechanisms were verified active (adapter fk_to_out‖W‖≈0.9; aux q-decode reached 0.17 rad, vs 1.08 rad for a probe on the frozen base). The FID/FVD drop is consistent with finetune-on-subset drift (the aux partially recovers it), not geometry per se. psnr_arm is flat because GT actions already determine the arm — the zero-shot-new-camera thesis (where geometry should help) was NOT tested here (the DROID annotations carry no camera extrinsics).

Contents

• RESULTS.md — final table + verdict.
• metrics/*.json — per-system full eval reports (aggregate + per-episode).
• videos/compare_ep*.mp4 — real decoded rollouts, layout [GT | base | Stage B (geo) | Stage C (geo+aux)], each panel = 3 cameras stacked vertically, ~41-frame autoregressive horizon.
• figures/drift_curve.png — per-frame PSNR vs autoregressive time (drift): all three drift ~16 dB over the horizon; geometry does not slow drift here.
• figures/stageA_probe_table.md — Stage-A probe study on the frozen base: which UNet layer decodes robot 3D best (down3), and the shape-vs-scale finding (features encode arm shape ~25 mm but are blind to absolute scale/joints).
• figures/attn_projection_ep0.png — visual→geometry-token attention at the finest UNet layer (72×40), overlaid on the 3-camera generated frame for the hand / tcp / link4 / leftfinger tokens. Honest reading: the attention is diffuse, not a clean per-link soft projection — the tokens are attended to broadly across the scene rather than localized at each link's pixel position in each view. This directly corroborates the negative result: the model did not learn to use the geometry tokens as a precise 3D→2D projection, which is why they did not improve generation.

What could NOT be run (asset-blocked)

The DROID droid_full annotations have no camera extrinsics and no pretrained Panda pose detector is available, which blocks: 2D keypoint reprojection error, silhouette IoU, cm-level controllability via reprojection, joint-config recovery, and the extrinsics-probe. The runnable, extrinsics-free evidence is the comparison videos, the drift curve, and the attention-projection figure.