FlowWM/causal_droidstd/RESUME.md

Resume on a new GPU

Stopped cleanly at step 25280 / 60000 on 2026-07-27, right after checkpoint-25000.pt was written — so nothing meaningful was lost. Nothing is running.

Restart training exactly where it stopped

cd /mnt/data/sftp/data/hunght23/quangpt3_data/masked-visual-actions/Ctrl-World

PYTHONNOUSERSITE=1 ABLATION=+flow FLOW_DROPOUT_PROB=0.2 \
  DATASET_NAMES=droid_std \
  DATASET_META_INFO_PATH=dataset_meta_info/droid_std_causalready \
  CKPT_PATH=model_ckpt/causal_droidstd/checkpoint-25000.pt \
  TRAIN_BATCH_SIZE=1 GRADIENT_ACCUMULATION_STEPS=4 \
  MAX_TRAIN_STEPS=35000 WARMUP_NEW_PARAMS_STEPS=0 \
  LOG_EVERY_STEPS=50 VALIDATION_STEPS=-1 CHECKPOINTING_STEPS=5000 \
  NUM_WORKERS=4 TAG=causal_droidstd OUTPUT_DIR=model_ckpt/causal_droidstd \
  LOG_FILE=model_ckpt/causal_droidstd/train.log \
  bash scripts/train_causal.sh

WARMUP_NEW_PARAMS_STEPS=0 because the conv_in/conv_out warmup already finished at step 1000. MAX_TRAIN_STEPS is the number of remaining steps. The checkpoint is already 24-channel so train_wm.py loads it directly without re-widening.

Auto eval + upload (memory-gated; waits when the GPU is busy):

OUTDIR=model_ckpt/causal_droidstd VAL_ID=199 \
  VAL_DATASET_DIR=dataset_example/droid_std TAG=causal_droidstd \
  MAX_STEPS=60000 MIN_FREE_MIB=8000 POLL=120 \
  nohup bash scripts/eval_upload_watcher.sh >> model_ckpt/causal_droidstd/watcher.log 2>&1 &

If the new GPU is free, TRAIN_BATCH_SIZE=2 GRADIENT_ACCUMULATION_STEPS=2 roughly halves wall-clock (measured ~44 GiB peak; batch 1 was chosen only to leave room for evals on a shared card).

Data is done — do not re-extract

droid_std causal artefacts are complete: 387 train + 4 val trajectories with latent_videos_{phi,ra,rb} and causal_aux, filtered meta at dataset_meta_info/droid_std_causalready/. 9 trajectories were dropped on purpose (all three views had cameraMatrix: null). The 5-stage extraction (scripts/extract_droid_std.sh) is resumable and would just skip everything.

State of the evidence

PSNR, causal ck20000, 4 val	16.92 / 16.86 / 18.42 / 15.22 — mean 16.9
PSNR, base Ctrl-World, same 4	19.28 / 20.68 / 23.05 / 16.92 — mean 20.0
VAE round-trip ceiling	33.9 dB
Rollout stability	drops over ~5 frames then flat for 85 — no divergence
3-branch guidance	runs (2.35x cost), changes 58% of pixels on edges, PSNR moves 0.05 dB

Everything is mirrored to hf://buckets/twanghcmut/intract-wm/FlowWM/causal_droidstd/ (step_*, ck20000_4val, base_vs_causal, guidance_sweep), each with a README.

The two things worth doing next, in order

1. A metric that measures what this project is for. PSNR rewards the blurry conditional mean, is blind to the sharpening that guidance demonstrably does, and barely penalises an object ending up in the wrong place — which is the entire thesis. Score instead whether the object is displaced the way the action implies (e.g. flow/position of the object region after N chunks). Until that exists, the base-vs-causal gap cannot be interpreted.

2. Run the baseline ablation row — it has never been run. Fine-tune stock Ctrl-World on these same 387 trajectories with lambda_flow=0. That is the only way to separate "the Phi branch costs appearance" from "fine-tuning on a 200x narrower corpus costs appearance" (catastrophic forgetting). Right now those two explanations are indistinguishable. ABLATION=baseline bash scripts/train_causal.sh with the same data flags.

Bugs already found and fixed (do not re-hit)

• Phi bootstrap latent — Phi's "no motion" is the zeros/grey image, not the colour wheel's white. Using zero_flow_latent.pt for a causal model costs ~2 dB and drifts the rollout. Fixed; config.py:zero_phi_latent_path now points at zero_phi_latent.pt.
• depth stage env — oscar-droid/.venv gets numpy from ~/.local, so it needs env -u PYTHONNOUSERSITE while the conda stages need PYTHONNOUSERSITE=1.
• rollout on droid_std — that dataset has no pre-decimated joints key; it is rebuilt from joint_position + gripper_position (verified bit-identical on droid_subset).
• flow_max — shipped latents were built with 20.0, but extract_flow_latent.py::resolve_flow_max prefers flow_max_calibration.json (121.70). Regenerating flow latents without --flow_max 20.0 silently yields an incompatible dataset.
• The COLLAPSE RISK line in the train log is unreliable whenever zero_phi_latent_path is unset — it then compares raw latent norms, which separate moving from static by only 1.03x even on ground truth.