README.md

# GeoToken — Implicit Geometric Grounding for Ctrl-World (results bucket)

Inject the robot's exact 3D **shape** (CAD + differentiable FK, canonical
robot-base frame) into a frozen-ish Ctrl-World video world model via zero-init
gated cross-attention geometry tokens — **no camera extrinsics, no rendering** —
and let multi-view attention learn the 3D→2D projection implicitly.

## Systems compared

| tag | what | init |
|---|---|---|
| `base_10k` | pretrained Ctrl-World (zero-shot) | — |
| `stageB_geo_40k` | + geometry tokens, 40k finetune | base_10k |
| `stageC_aux_40k` | + geometry tokens + aux FK-space 3D-decode loss @down3 | base_10k |

All on DROID `droid_full` (10k train eps), 60 val eps, GT-action replay, seed 0.

## Headline result (see `RESULTS.md`)

| system | psnr↑ | ssim↑ | lpips↓ | fid↓ | fvd↓ | psnr_arm↑ |
|---|---|---|---|---|---|---|
| base_10k | **19.35** | **0.706** | **0.262** | **35.5** | **33.0** | **13.88** |
| stageB_geo_40k | 19.13 | 0.705 | 0.265 | 42.9 | 41.1 | 13.66 |
| stageC_aux_40k | 19.01 | 0.703 | 0.266 | 40.2 | 36.6 | 13.56 |

**Honest verdict:** on in-distribution GT-action replay, geometry grounding does
**not** improve the world model (slightly worse across the board). All mechanisms
were verified active (adapter `fk_to_out‖W‖`≈0.9; aux q-decode reached 0.17 rad,
vs 1.08 rad for a probe on the frozen base). The FID/FVD drop is consistent with
finetune-on-subset drift (the aux partially recovers it), not geometry per se.
`psnr_arm` is flat because GT actions already determine the arm — the
zero-shot-**new-camera** thesis (where geometry should help) was NOT tested here
(the DROID annotations carry no camera extrinsics).

## Contents

- `RESULTS.md` — final table + verdict.
- `metrics/*.json` — per-system full eval reports (aggregate + per-episode).
- `videos/compare_ep*.mp4` — **real decoded rollouts**, layout
  `[GT | base | Stage B (geo) | Stage C (geo+aux)]`, each panel = **3 cameras
  stacked vertically**, ~41-frame autoregressive horizon.
- `figures/drift_curve.png` — per-frame PSNR vs autoregressive time (drift): all
  three drift ~16 dB over the horizon; geometry does not slow drift here.
- `figures/stageA_probe_table.md` — Stage-A probe study on the frozen base: which
  UNet layer decodes robot 3D best (`down3`), and the shape-vs-scale finding
  (features encode arm *shape* ~25 mm but are blind to absolute *scale*/joints).
- `figures/attn_projection_ep0.png` — visual→geometry-token attention at the
  finest UNet layer (72×40), overlaid on the 3-camera generated frame for the
  hand / tcp / link4 / leftfinger tokens. **Honest reading:** the attention is
  **diffuse**, not a clean per-link soft projection — the tokens are attended to
  broadly across the scene rather than localized at each link's pixel position in
  each view. This directly corroborates the negative result: the model did not
  learn to use the geometry tokens as a precise 3D→2D projection, which is why
  they did not improve generation.

## What could NOT be run (asset-blocked)

The DROID `droid_full` annotations have **no camera extrinsics** and no pretrained
Panda pose detector is available, which blocks: 2D keypoint reprojection error,
silhouette IoU, cm-level controllability via reprojection, joint-config recovery,
and the extrinsics-probe. The runnable, extrinsics-free evidence is the
comparison videos, the drift curve, and the attention-projection figure.