FlowWM/causal_droidstd/phi_check/README.md

Is the predicted object flow (Phi) correct? — causal ck-25000, val/199

Answer: no, and two separate things are wrong. Also note this run predicts RGB + object flow only — there is no depth. ABLATION=+flow sets gamma_sdf=0, and the live training log confirms loss_sdf=0.000000, sdf_valid_frac=0.000000. The SDF/depth term has never been on in this run (the gate-4 depth anchor failed earlier and was left out).

Finding 1 — the _flow.mp4 rollout videos contain NO ground truth

scripts/rollout_replay_traj.py:646:

videos_cat_flow = np.concatenate([videos_flow, videos_flow], axis=-3)

Both halves of the flow panel are the same predicted Phi, duplicated to match the RGB video's height. Measured difference between the halves: 0.85/255 = pure mp4 codec noise. Any "GT vs pred" read off those videos is comparing the prediction against itself.

Finding 2 — 73% of every rollout is a frozen action against a frozen target

val/199 is a 33-frame episode (video_length=33, latents (33,4,24,40)). The replay runs 22 chunks covering GT frame ids 0..88. get_traj_info clamps out-of-range ids to the last frame, so:

chunk	gt_frame_ids	action spread
0	0–4	6.267
1–5	4–24	0.05–0.12
6–21	24–88	0.00000

Chunks 8–21 contain 1 unique action row out of 70. From chunk 6 on — 16 of 22 chunks — the model is fed a constant frozen action and scored against a frozen GT frame.

This invalidates the interpretation of every PSNR number gathered so far: causal 16.9, base 20.0, and the whole guidance sweep are ~73% measured over a dead segment where nothing is asked to change. It also fully explains the earlier "stable, no divergence for 85 frames" result — nothing was moving. Rollouts should use ~7 chunks for a 33-frame episode.

Finding 3 — predicted Phi lands where GT Phi is not

Comparing predicted Phi against genuinely decoded GT Phi latents, over the valid chunks only (frames 0–32), active pixel = |Phi − grey| > 12:

view	GT active	GT mean|Phi|	PRED mean|Phi|	IoU(pred, GT)
ext1	0.72%	0.916	1.599	0.008
ext2	1.31%	1.190	2.070	0.003
wrist	7.11%	2.474	1.427	0.004

The control that makes this readable — GT Phi against itself:

view	IoU(t, t+1)	IoU(t, t+2)	IoU(t, random t)
ext1	0.148	0.065	0.036
ext2	0.169	0.052	0.038
wrist	0.110	0.070	0.052

So prediction-vs-GT (0.003–0.008) sits 5–10x below the random-frame baseline (0.036–0.052). The predicted flow is not merely imprecise; it occupies regions where the GT flow essentially never is.

The magnitudes point the same way: the model over-produces flow in the two external views (1.6–2.1 vs GT 0.9–1.2) and under-produces it in the wrist view (1.4 vs GT 2.5) — which is the view with by far the most real motion (7.11% active vs 0.72%/1.31%).

The caveat that matters

GT Phi is itself only 0.11–0.17 IoU between adjacent frames. The supervision target is so sparse and temporally incoherent that it is weak to learn from at all. Finding 3 should not be read purely as "the Phi head is broken" — the target it is being trained against is close to noise from one frame to the next. Fixing the flow-extraction stability is likely a prerequisite to any Phi head working.

---

Follow-up: WAFT is fine. The post-processing destroys it.

latent_videos_flow (raw WAFT) and latent_videos_phi (the Phi actually trained on) both exist for val/99, so they can be compared directly. Each stream has a different zero-motion reference — measured medians: raw WAFT = 254 (WHITE, colour-wheel centre), Phi = 128 (GREY). Scoring WAFT against grey makes a static background look 100% "active"; that is a measurement artifact, not saturation.

Measured against the correct reference for each:

stream	view	active%	mean dist	IoU(t,t+1)
WAFT raw	ext1	5.62%	3.53	0.359
WAFT raw	ext2	6.07%	4.08	0.425
WAFT raw	wrist	62.62%	39.28	0.666
Phi	ext1	3.44%	1.85	0.271
Phi	ext2	1.56%	1.38	0.204
Phi	wrist	6.88%	2.32	0.128

Raw WAFT is 2–5x more temporally coherent. The construction on top of it discards 90% of the wrist signal and 74% of ext2, and drops coherence from 0.67 to 0.13.

WAFT_raw_vs_Phi_3views_val99.mp4 shows why. At frame 18 WAFT gives a crisp, clean flow field — the arm silhouette in both exterior views, a smooth ego-motion gradient in the wrist — while the Phi built from it reads 0.3% / 0.1% / 0.0% active. The wrist Phi is completely empty.

Where the signal goes

extract_causal_latent.py:1034

keep = (~rob) & (lat > args.phi_noise_m) & np.isfinite(mag)

1. ~rob removes the robot. In the exterior views the robot arm is the dominant moving thing, so the strongest, cleanest part of WAFT's output is deleted by design.
2. lat > phi_noise_m with phi_noise_m = 0.01 m/step. At the 5 Hz video rate that is a 5 cm/s floor — most genuine object motion in these trajectories is slower than that.

What survives is a flickering ~1% residue (IoU 0.13–0.27 between adjacent frames), which is the target the Phi head is being trained against.

Corroborating signal from causal_aux/val/199.npz: gripper_contact is 0 for all 33 frames, and contact never exceeds 0.282, on a trajectory whose instruction is "Move the green towel to the right". motion_evidence peaks at 0.015 against m_ref = 0.0518. The contact gate — the core switch of the causal factorization — never fires, because the Phi it is computed from has almost nothing in it.

Conclusion: the flow estimator is not the problem. phi_noise_m and the ~rob mask are.

---

The fix: Phi from 2-D WAFT flow, no depth at all

Why the metric anchoring existed, and why it goes

The contract wanted Phi's magnitude to mean "how fast is the object moving relative to the end-effector": m = |v_obj| / (kappa * |v_ee|). EE speed comes from kinematics in m/s, so the object's speed had to be in m/s too -- pixel flow is not comparable across depths. Hence Depth-Anything-3 + RANSAC anchoring to convert pixels to metres.

Legitimate goal, but the price is a monocular-depth chain that fails on 43% of trajectories, and the payoff is only a normalisation. The model consumes Phi as a latent image; pixels vs metres is a choice of scale, not of information.

Phi[...,0] = clip(u / flow_max[view], -1, 1)
Phi[...,1] = clip(v / flow_max[view], -1, 1)
Phi[...,2] = 0
Phi = 0 inside the (dilated) robot silhouette

Same 3-channel [-1,1] layout as before, so "no motion = grey" and zero_phi_latent.pt stay valid and nothing on the model side changes.

flow_max is per-view, because the cameras are not alike

Non-robot flow magnitude, 40 sampled trajectories:

view	p50	p90	p95	p99	p99.9
ext1	0.03 px	0.19	0.40	6.09	22.1
ext2	0.03 px	0.19	0.46	6.42	21.9
wrist	3.66 px	23.0	31.0	49.3	88.0

The exterior cameras are genuinely static (p90 = 0.19 px). The wrist camera moves with the arm. One shared flow_max would either saturate the wrist or push exterior object motion below what the VAE can resolve, so: flow_max = 20 / 20 / 50.

Validation before committing to a re-extract

Boundary leakage -- is the surviving Phi real motion, or flow bleeding off the mask edge? Active Phi pixels bucketed by distance to the robot silhouette, 25 sampled trajectories:

view	0-3 px	3-8 px	8-20 px	>20 px
ext1	7.1%	17.0%	23.6%	52.2%
ext2	6.9%	13.0%	19.4%	60.8%
wrist	0.4%	1.4%	3.8%	94.4%

Only ~7% hugs the silhouette; the majority sits >20 px away. Not leakage.

Result on val/99 (NEW_phi2d_vs_OLD_val99.mp4):

view	OLD active	NEW active	NEW IoU(t,t+1)
ext1	3.54%	4.90%	0.298
ext2	1.61%	2.50%	0.311
wrist	7.07%	42.33%	0.474

Honest reading: in the exterior views this is a moderate gain (~1.4x the active pixels, comparable coherence). The point is not density -- it is that the new Phi does not ride on a depth estimate that fails 43% of the time.

Wrist: keeping ego-motion, deliberately

The wrist camera moves, so its non-robot flow is the scene sliding past (77.5% of pixels per frame), not object motion -- which is why the original code excluded the wrist from motion_evidence. The options were to zero it, to compensate ego-motion (needs depth again), or to keep it. Keeping it was chosen deliberately by the user, accepting that the wrist Phi channel is ego-motion rather than object motion. Recording it here so a later reader does not mistake it for an oversight.