FlowWM/causal_droidstd/phi_check/README.md

# Is the predicted object flow (Phi) correct? — causal ck-25000, val/199

Answer: **no, and two separate things are wrong.** Also note this run predicts
**RGB + object flow only — there is no depth**. `ABLATION=+flow` sets `gamma_sdf=0`, and the
live training log confirms `loss_sdf=0.000000`, `sdf_valid_frac=0.000000`. The SDF/depth term
has never been on in this run (the gate-4 depth anchor failed earlier and was left out).

## Finding 1 — the `_flow.mp4` rollout videos contain NO ground truth

`scripts/rollout_replay_traj.py:646`:

    videos_cat_flow = np.concatenate([videos_flow, videos_flow], axis=-3)

Both halves of the flow panel are the *same* predicted Phi, duplicated to match the RGB
video's height. Measured difference between the halves: **0.85/255 = pure mp4 codec noise**.
Any "GT vs pred" read off those videos is comparing the prediction against itself.

## Finding 2 — 73% of every rollout is a frozen action against a frozen target

`val/199` is a **33-frame** episode (`video_length=33`, latents `(33,4,24,40)`). The replay
runs **22 chunks** covering GT frame ids 0..88. `get_traj_info` clamps out-of-range ids to the
last frame, so:

| chunk | gt_frame_ids | action spread |
|---|---|---|
| 0 | 0–4 | 6.267 |
| 1–5 | 4–24 | 0.05–0.12 |
| **6–21** | 24–88 | **0.00000** |

Chunks 8–21 contain **1 unique action row out of 70**. From chunk 6 on — 16 of 22 chunks —
the model is fed a constant frozen action and scored against a frozen GT frame.

**This invalidates the interpretation of every PSNR number gathered so far**: causal 16.9,
base 20.0, and the whole guidance sweep are ~73% measured over a dead segment where nothing
is asked to change. It also fully explains the earlier "stable, no divergence for 85 frames"
result — nothing was moving. Rollouts should use ~7 chunks for a 33-frame episode.

## Finding 3 — predicted Phi lands where GT Phi is not

Comparing predicted Phi against **genuinely decoded GT Phi latents**, over the valid chunks
only (frames 0–32), active pixel = |Phi − grey| > 12:

| view | GT active | GT mean\|Phi\| | PRED mean\|Phi\| | IoU(pred, GT) |
|---|---|---|---|---|
| ext1 | 0.72% | 0.916 | 1.599 | **0.008** |
| ext2 | 1.31% | 1.190 | 2.070 | **0.003** |
| wrist | 7.11% | 2.474 | 1.427 | **0.004** |

The control that makes this readable — GT Phi against *itself*:

| view | IoU(t, t+1) | IoU(t, t+2) | IoU(t, random t) |
|---|---|---|---|
| ext1 | 0.148 | 0.065 | 0.036 |
| ext2 | 0.169 | 0.052 | 0.038 |
| wrist | 0.110 | 0.070 | 0.052 |

So prediction-vs-GT (0.003–0.008) sits **5–10x below the random-frame baseline**
(0.036–0.052). The predicted flow is not merely imprecise; it occupies regions where the GT
flow essentially never is.

The magnitudes point the same way: the model over-produces flow in the two external views
(1.6–2.1 vs GT 0.9–1.2) and **under-produces it in the wrist view** (1.4 vs GT 2.5) — which is
the view with by far the most real motion (7.11% active vs 0.72%/1.31%).

## The caveat that matters

GT Phi is itself only **0.11–0.17 IoU between adjacent frames**. The supervision target is so
sparse and temporally incoherent that it is weak to learn from at all. Finding 3 should not be
read purely as "the Phi head is broken" — the target it is being trained against is close to
noise from one frame to the next. Fixing the flow-extraction stability is likely a
prerequisite to any Phi head working.

---

# Follow-up: WAFT is fine. The post-processing destroys it.

`latent_videos_flow` (raw WAFT) and `latent_videos_phi` (the Phi actually trained on) both
exist for val/99, so they can be compared directly. **Each stream has a different zero-motion
reference** — measured medians: raw WAFT = 254 (WHITE, colour-wheel centre), Phi = 128 (GREY).
Scoring WAFT against grey makes a static background look 100% "active"; that is a measurement
artifact, not saturation.

Measured against the correct reference for each:

| stream | view | active% | mean dist | IoU(t,t+1) |
|---|---|---|---|---|
| WAFT raw | ext1 | 5.62% | 3.53 | **0.359** |
| WAFT raw | ext2 | 6.07% | 4.08 | **0.425** |
| WAFT raw | wrist | **62.62%** | 39.28 | **0.666** |
| Phi | ext1 | 3.44% | 1.85 | 0.271 |
| Phi | ext2 | **1.56%** | 1.38 | 0.204 |
| Phi | wrist | **6.88%** | 2.32 | **0.128** |

Raw WAFT is 2–5x more temporally coherent. The construction on top of it discards **90% of the
wrist signal and 74% of ext2**, and drops coherence from 0.67 to 0.13.

`WAFT_raw_vs_Phi_3views_val99.mp4` shows why. At frame 18 WAFT gives a crisp, clean flow field
— the arm silhouette in both exterior views, a smooth ego-motion gradient in the wrist — while
the Phi built from it reads **0.3% / 0.1% / 0.0% active**. The wrist Phi is completely empty.

## Where the signal goes

`extract_causal_latent.py:1034`

    keep = (~rob) & (lat > args.phi_noise_m) & np.isfinite(mag)

1. `~rob` removes the robot. In the exterior views the robot arm *is* the dominant moving
   thing, so the strongest, cleanest part of WAFT's output is deleted by design.
2. `lat > phi_noise_m` with `phi_noise_m = 0.01` m/step. At the 5 Hz video rate that is a
   5 cm/s floor — most genuine object motion in these trajectories is slower than that.

What survives is a flickering ~1% residue (IoU 0.13–0.27 between adjacent frames), which is
the target the Phi head is being trained against.

Corroborating signal from `causal_aux/val/199.npz`: `gripper_contact` is **0 for all 33
frames**, and `contact` never exceeds 0.282, on a trajectory whose instruction is "Move the
green towel to the right". `motion_evidence` peaks at 0.015 against `m_ref = 0.0518`. The
contact gate — the core switch of the causal factorization — never fires, because the Phi it
is computed from has almost nothing in it.

**Conclusion: the flow estimator is not the problem. `phi_noise_m` and the `~rob` mask are.**

---

# The fix: Phi from 2-D WAFT flow, no depth at all

## Why the metric anchoring existed, and why it goes

The contract wanted Phi's magnitude to mean "how fast is the object moving **relative to the
end-effector**": `m = |v_obj| / (kappa * |v_ee|)`. EE speed comes from kinematics in m/s, so
the object's speed had to be in m/s too -- pixel flow is not comparable across depths. Hence
Depth-Anything-3 + RANSAC anchoring to convert pixels to metres.

Legitimate goal, but the price is a monocular-depth chain that fails on 43% of trajectories,
and the payoff is only a normalisation. The model consumes Phi as a latent image; pixels vs
metres is a choice of scale, not of information.

    Phi[...,0] = clip(u / flow_max[view], -1, 1)
    Phi[...,1] = clip(v / flow_max[view], -1, 1)
    Phi[...,2] = 0
    Phi = 0 inside the (dilated) robot silhouette

Same 3-channel [-1,1] layout as before, so "no motion = grey" and `zero_phi_latent.pt` stay
valid and nothing on the model side changes.

## flow_max is per-view, because the cameras are not alike

Non-robot flow magnitude, 40 sampled trajectories:

| view | p50 | p90 | p95 | p99 | p99.9 |
|---|---|---|---|---|---|
| ext1 | 0.03 px | 0.19 | 0.40 | 6.09 | 22.1 |
| ext2 | 0.03 px | 0.19 | 0.46 | 6.42 | 21.9 |
| wrist | **3.66 px** | **23.0** | 31.0 | 49.3 | 88.0 |

The exterior cameras are genuinely static (p90 = 0.19 px). The wrist camera moves with the
arm. One shared flow_max would either saturate the wrist or push exterior object motion below
what the VAE can resolve, so: **flow_max = 20 / 20 / 50**.

## Validation before committing to a re-extract

**Boundary leakage** -- is the surviving Phi real motion, or flow bleeding off the mask edge?
Active Phi pixels bucketed by distance to the robot silhouette, 25 sampled trajectories:

| view | 0-3 px | 3-8 px | 8-20 px | >20 px |
|---|---|---|---|---|
| ext1 | 7.1% | 17.0% | 23.6% | **52.2%** |
| ext2 | 6.9% | 13.0% | 19.4% | **60.8%** |
| wrist | 0.4% | 1.4% | 3.8% | **94.4%** |

Only ~7% hugs the silhouette; the majority sits >20 px away. Not leakage.

**Result on val/99** (`NEW_phi2d_vs_OLD_val99.mp4`):

| view | OLD active | NEW active | NEW IoU(t,t+1) |
|---|---|---|---|
| ext1 | 3.54% | 4.90% | 0.298 |
| ext2 | 1.61% | 2.50% | 0.311 |
| wrist | 7.07% | 42.33% | 0.474 |

Honest reading: in the exterior views this is a **moderate** gain (~1.4x the active pixels,
comparable coherence). The point is not density -- it is that the new Phi does not ride on a
depth estimate that fails 43% of the time.

## Wrist: keeping ego-motion, deliberately

The wrist camera moves, so its non-robot flow is the scene sliding past (77.5% of pixels per
frame), not object motion -- which is why the original code excluded the wrist from
`motion_evidence`. The options were to zero it, to compensate ego-motion (needs depth again),
or to keep it. **Keeping it was chosen deliberately by the user**, accepting that the wrist
Phi channel is ego-motion rather than object motion. Recording it here so a later reader does
not mistake it for an oversight.