FlowWM/causal_droidstd/RESUME.md

# Resume on a new GPU

Stopped cleanly at **step 25280 / 60000** on 2026-07-27, right after `checkpoint-25000.pt`
was written — so nothing meaningful was lost. Nothing is running.

## Restart training exactly where it stopped

```bash
cd /mnt/data/sftp/data/hunght23/quangpt3_data/masked-visual-actions/Ctrl-World

PYTHONNOUSERSITE=1 ABLATION=+flow FLOW_DROPOUT_PROB=0.2 \
  DATASET_NAMES=droid_std \
  DATASET_META_INFO_PATH=dataset_meta_info/droid_std_causalready \
  CKPT_PATH=model_ckpt/causal_droidstd/checkpoint-25000.pt \
  TRAIN_BATCH_SIZE=1 GRADIENT_ACCUMULATION_STEPS=4 \
  MAX_TRAIN_STEPS=35000 WARMUP_NEW_PARAMS_STEPS=0 \
  LOG_EVERY_STEPS=50 VALIDATION_STEPS=-1 CHECKPOINTING_STEPS=5000 \
  NUM_WORKERS=4 TAG=causal_droidstd OUTPUT_DIR=model_ckpt/causal_droidstd \
  LOG_FILE=model_ckpt/causal_droidstd/train.log \
  bash scripts/train_causal.sh
```

`WARMUP_NEW_PARAMS_STEPS=0` because the conv_in/conv_out warmup already finished at step
1000. `MAX_TRAIN_STEPS` is the number of **remaining** steps. The checkpoint is already
24-channel so `train_wm.py` loads it directly without re-widening.

Auto eval + upload (memory-gated; waits when the GPU is busy):

```bash
OUTDIR=model_ckpt/causal_droidstd VAL_ID=199 \
  VAL_DATASET_DIR=dataset_example/droid_std TAG=causal_droidstd \
  MAX_STEPS=60000 MIN_FREE_MIB=8000 POLL=120 \
  nohup bash scripts/eval_upload_watcher.sh >> model_ckpt/causal_droidstd/watcher.log 2>&1 &
```

If the new GPU is free, `TRAIN_BATCH_SIZE=2 GRADIENT_ACCUMULATION_STEPS=2` roughly halves
wall-clock (measured ~44 GiB peak; batch 1 was chosen only to leave room for evals on a
shared card).

## Data is done — do not re-extract

`droid_std` causal artefacts are complete: **387 train + 4 val** trajectories with
`latent_videos_{phi,ra,rb}` and `causal_aux`, filtered meta at
`dataset_meta_info/droid_std_causalready/`. 9 trajectories were dropped on purpose (all three
views had `cameraMatrix: null`). The 5-stage extraction (`scripts/extract_droid_std.sh`) is
resumable and would just skip everything.

## State of the evidence

| | |
|---|---|
| PSNR, causal ck20000, 4 val | 16.92 / 16.86 / 18.42 / 15.22 — mean **16.9** |
| PSNR, base Ctrl-World, same 4 | 19.28 / 20.68 / 23.05 / 16.92 — mean **20.0** |
| VAE round-trip ceiling | **33.9 dB** |
| Rollout stability | drops over ~5 frames then **flat for 85** — no divergence |
| 3-branch guidance | runs (2.35x cost), changes 58% of pixels **on edges**, PSNR moves 0.05 dB |

Everything is mirrored to `hf://buckets/twanghcmut/intract-wm/FlowWM/causal_droidstd/`
(`step_*`, `ck20000_4val`, `base_vs_causal`, `guidance_sweep`), each with a README.

## The two things worth doing next, in order

1. **A metric that measures what this project is for.** PSNR rewards the blurry conditional
   mean, is blind to the sharpening that guidance demonstrably does, and barely penalises an
   object ending up in the wrong place — which is the entire thesis. Score instead whether
   the object is displaced the way the action implies (e.g. flow/position of the object
   region after N chunks). Until that exists, the base-vs-causal gap cannot be interpreted.

2. **Run the `baseline` ablation row — it has never been run.** Fine-tune stock Ctrl-World on
   these same 387 trajectories with `lambda_flow=0`. That is the only way to separate "the Phi
   branch costs appearance" from "fine-tuning on a 200x narrower corpus costs appearance"
   (catastrophic forgetting). Right now those two explanations are indistinguishable.
   `ABLATION=baseline bash scripts/train_causal.sh` with the same data flags.

## Bugs already found and fixed (do not re-hit)

* **Phi bootstrap latent** — Phi's "no motion" is the zeros/**grey** image, not the colour
  wheel's white. Using `zero_flow_latent.pt` for a causal model costs ~2 dB and drifts the
  rollout. Fixed; `config.py:zero_phi_latent_path` now points at `zero_phi_latent.pt`.
* **depth stage env** — `oscar-droid/.venv` gets numpy from `~/.local`, so it needs
  `env -u PYTHONNOUSERSITE` while the conda stages need `PYTHONNOUSERSITE=1`.
* **rollout on droid_std** — that dataset has no pre-decimated `joints` key; it is rebuilt
  from `joint_position` + `gripper_position` (verified bit-identical on droid_subset).
* **`flow_max`** — shipped latents were built with **20.0**, but
  `extract_flow_latent.py::resolve_flow_max` prefers `flow_max_calibration.json` (121.70).
  Regenerating flow latents without `--flow_max 20.0` silently yields an incompatible dataset.
* The `COLLAPSE RISK` line in the train log is unreliable whenever `zero_phi_latent_path` is
  unset — it then compares raw latent norms, which separate moving from static by only 1.03x
  even on ground truth.