# WidowX / full-Bridge adaptation (updated 2026-09-24)

This trains the released Zeva Stage-2 Cosmos model, selecting its action heads,
behavior injection modules and proprio encoder for fine-tuning. The Qwen/MoT
backbone, VAE, CTE and static retrieval head remain frozen. This is an initial
cross-robot adaptation, not a reproduction of a WidowX-trained release.

The smoke root is `/opt/zhangchenyu/datasets/bridge_orig_lerobot_smoke`.
The old sampler used only 397 of its 654 metadata episodes (1,255 windows).
The new `full` sampler includes short episodes and all stride-4 start positions.
Camera `image_0`,
5 FPS, 256-pixel resolution, batch size 1. Per-window task IDs are read from
parquet. Six motion dimensions use the documented Bridge quantile constants;
gripper is -1 closed / +1 open. Proprio is Bridge's eight raw state fields plus
the repeated gripper value, matching the inference adapter's nine fields.

`prepare_widowx_features.py` caches frozen release CTE features for every window.
The phase/effect features use only observations and actions up to that window's
start, including late-trajectory windows. Global context is retrieved from the episode's
first observation and instruction; this bank is still RoboCasa-trained.

Use the full-IPEC command below after reserving GPUs; do not invoke the generic
smoke launcher with its historical GPU/cache defaults. Both launchers use the
existing Cosmos venv. The practical step limit is 2,147,483,647;
training continues across data epochs and does not stop after an evaluation.
The installed Cosmos source/venv are reused, with compatibility fixes scoped to
the launcher. No new model download is required.

Run `python scripts/watch_widowx_eval.py` to watch atomic delta checkpoints.
For saved multiples of 2,000 steps, the selected GPU serves the release plus that trained delta and runs ten
complete `widowx_carrot_on_plate` episodes, seeds 0–9, in the existing SimplerEnv
environment. `--exec-horizon` controls how many predictions are executed before
replanning (1–32, default 4). It is independent of the number of denoising steps.
The causal history records every executed control and makes a CTE boundary
every four controls, including when execution horizons differ from four. XYZ commands
are denormalized once; Euler rotation increments are converted to axis-angle.
Only complete evaluations get success-rate JSON files; errors remain errors.

Checkpoint output is `$WIDOWX_RUN_DIR/checkpoints/`. Checkpoints contain selected
parameters, optimizer, scheduler, scaler, RNG, a data contract and per-rank
CONSUMED data cursors. They require the original Stage-2 weights. All checkpoints
are retained; none are automatically deleted, even after evaluation. Writes are
atomic, existing checkpoint files cannot be overwritten, and saving fails loudly
below 2 GiB free space. This does not reserve disk space against other jobs.

## Full IPEC run

`bash scripts/run_ipec_full_training.sh` uses the full local IPEC Bridge export
and **`.artifacts/ipec_full_v2`** for new features and training state. The old
`.artifacts/ipec_full` run is preserved as a historical baseline. As of 2026-09-24,
the WidowX recipe and TOML both save every **10,000 optimizer steps**. Existing
checkpoints are retained; the launcher resumes the latest complete checkpoint
when explicitly started. It does not start an evaluator automatically.

The default recipe records W&B metrics in **offline** mode. Records are stored
under `$WIDOWX_RUN_DIR/train/<project>/widowx/<name>/wandb/`. Logging begins
at the resumed checkpoint's step; previously disabled W&B history is not
reconstructed automatically. Training state remains in `checkpoints/`; W&B
logging does not replace model checkpoints.

To send a future full-IPEC run to your own W&B account, set `WANDB_API_KEY` and
`WANDB_ENTITY` in the launching shell. Do not put the API key in this repository
or replace the shared machine's W&B login. Pass your project and a **new** run
name as Hydra overrides; the full-IPEC launcher forwards these to the trainer:

```bash
# Example after WANDB_API_KEY and WANDB_ENTITY are exported in this shell:
CUDA_VISIBLE_DEVICES=2,4 NPROC_PER_NODE=2 WANDB_MODE=online \
  bash scripts/run_ipec_full_training.sh -- \
  job.project=YOUR_PROJECT job.name=ipec_full_v2 job.wandb_mode=online
```

The new run name gives W&B a fresh ID and local directory. The model still
resumes the latest delta under `$WIDOWX_RUN_DIR/checkpoints/` because that
directory is controlled by `WIDOWX_RUN_DIR`, independently of the W&B run name.
The offline history from earlier training is separate; after authenticating,
it can be previewed with `wandb sync --dry-run` and uploaded with `wandb sync`
using `--entity` and `--project` on its `offline-run-*` directory.

The launcher defaults `WANDB_BASE_URL` to `https://api.wandb.ai`, overriding
unrelated global W&B settings on this shared account. Set `WANDB_BASE_URL`
explicitly before launching if using a self-hosted W&B server.
If online W&B initialization raises a W&B error, the local training entrypoint
retries once in offline mode with the same project, name and persisted run ID.
An offline initialization failure or an unrelated training error is not hidden.

The full-IPEC launcher defaults to GPU 1 and one process. To use a different
reserved GPU set, supply both `CUDA_VISIBLE_DEVICES` and `NPROC_PER_NODE` in
the environment. The TOML uses `data_parallel_shard_degree = -1`, so FSDP
infers the shard count from the launched world size. Eight GPUs with one
sample per rank yield an effective batch size of eight (two GPUs: two). Check GPU processes
and availability before launching; a momentary zero utilization reading
does not mean a card is unoccupied.

To evaluate one checkpoint on 20 carrot-on-plate episodes (seeds 0–19), with
videos and per-control action/state traces:

```bash
WIDOWX_RUN_DIR=/home/gpu4/tianyi/Zeva/.artifacts/ipec_full WIDOWX_EVAL_GPU=1 \
  /home/gpu4/tianyi/cosmos-framework/.venv/bin/python scripts/watch_widowx_eval.py \
  --once --checkpoint /home/gpu4/tianyi/Zeva/.artifacts/ipec_full/checkpoints/step_000290000.pt \
  --episodes 20 --port 18785 --record-traces \
  --num-steps 30 --max-steps 150 --exec-horizon 8
```

Choose a free GPU and port before launching. This mode exits after the requested
evaluation and never resumes training or backfills other checkpoints. Results
are written under `eval/`, MP4s under `eval/videos/step_000290000/`, and traces
under `eval/traces/step_000290000/`. It refuses to overwrite an existing completed
result for the checkpoint; use a separate `WIDOWX_RUN_DIR` for repeat evaluations.
The watcher resolves the GPU to its UUID and checks the renderer's actual GPU
before loading the policy. Rendering requests `cuda:0` within that visible set.
This installed SAPIEN version can confuse identical PCI bus numbers in different
domains: GPU 4 incorrectly renders on GPU 0 even with `cuda:0`. The preflight
rejects that mismatch instead of starting a rollout on someone else's card.
GPU 2 passed the physical-device probe; check its workload before evaluation.
`WIDOWX_RENDER_GPU` can select a separate, verified renderer GPU (defaults to
`WIDOWX_EVAL_GPU`). Both physical UUIDs are recorded in the evaluation report.
GPU 6 also collides with GPU 2 in this SAPIEN version, so this experiment uses
policy GPU 6 and renderer GPU 2 explicitly, not an unverified renderer on GPU 6.

For a controlled sampling comparison, pass `--num-steps 30 --max-steps 150`
to the watcher. These set the policy's UniPC denoising count and the simulator's
control-step limit, respectively. `--seed-start` defaults to 0; disjoint ranges
can run in separate artifact directories and use the same per-step inference
seeds as a single sequential run. Completed reports record `inference_num_steps`.

## Full sampling and cache provenance

The local IPEC export has 53,192 episodes and 1,893,026 recorded steps. The old
sampler used 30,770 episodes and 99,075 windows. The new sampler uses **53,192
episodes / 492,160 windows**, starts at 0, 4, 8, ... through the last recorded
frame, and keeps the model's 32-action / 33-image shape. Tail images repeat the
last observation and actions are padded, but separate validity masks exclude
padded action, vision and Gaussian-prior targets from their losses. The terminal
recorded action remains supervised even if its next image was not exported.
This changes loss normalization; old and new scalar losses are not directly
comparable. Full sampling does not synthesize missing language annotations or
balance task frequencies.

The current v2 feature cache is stored beside the raw dataset at
`/home/gpu4/tianyi/TTTVLA/datasets/bridge_orig_1.0.0_lerobot_zeva_features_v2`.
Its `manifest.json` binds metadata SHA256, model-file identities and versioned
preprocessing contracts to the cache. Small model files are hashed; large shards
use path, size and modification time, so this is provenance validation, NOT a
full dataset-content integrity scan. Merely adding a fingerprint does not alter
features. This migration prepares a separate cache because old caches lack
provenance and omit late prefixes/short episodes. The old cache is not deleted.
The full launcher runs one preparation worker per selected GPU, validates every
episode's feature tensors, and starts training only after all are present.
Interrupting preparation is safe; rerun to validate and skip completed episodes.

Do not set `WIDOWX_MAX_EPISODES` for production; it is an optional smoke-test
limit included in the checkpoint's data contract. `WIDOWX_WINDOW_MODE=legacy`
exists only for explicit historical diagnostics, not the full-IPEC launcher.

## Resume semantics

`WIDOWX_RESUME=auto` (default) selects the highest **saved optimizer step** in the
current checkpoint directory. This is unrelated to `trainer.max_iter`, the total
optimizer-step stopping limit, and to `--max-steps`, the evaluation episode limit.
`WIDOWX_RESUME=none` requires a new/empty checkpoint directory. An explicit
checkpoint path is also accepted; use a new output directory when branching.

New v3 checkpoints restore the next actually consumed data window, discarding
stale prefetched batches. Exact sample-order continuity currently requires the
same data/cache contract, GPU count, seed, one worker/rank and one packed
sample/rank. It is not a promise of bitwise-identical GPU arithmetic. Old v2
checkpoints do not contain this cursor and cannot retroactively recover it.

Changing GPU count is rejected by default. To deliberately re-shard data, set
`WIDOWX_ALLOW_DATA_RESHARD=1`; this begins the next shuffled data epoch instead
of pretending to preserve exact sample order. `WIDOWX_ALLOW_LEGACY_RESUME=1`
is an explicit compatibility escape hatch for old/data-changed checkpoints;
it is NOT the recommended migration for the corrected full-data recipe.
Restart this corrected experiment in a new directory from released Stage-2.

## Manual checkpoint deletion

Nothing is pruned automatically. Check sizes/free space first:

```bash
df -h /home/gpu4/tianyi/Zeva
ls -lh /home/gpu4/tianyi/Zeva/.artifacts/ipec_full/checkpoints/
ls -lh /home/gpu4/tianyi/Zeva/.artifacts/ipec_full_v2/checkpoints/
```

Example: explicitly select and interactively confirm deletion of ONLY an old
2,000-step checkpoint (do not run this unless you want to delete that file):

```bash
rm -i -- /home/gpu4/tianyi/Zeva/.artifacts/ipec_full/checkpoints/step_000002000.pt
```

Keep the latest checkpoint, any checkpoint currently being loaded/evaluated,
and your comparison baselines. Copy to another disk before deletion if recovery
may be needed; `rm` does not use a trash bin. These deltas also depend on the
released base model: do not delete `/home/gpu4/tianyi/zeva-release/weights/stage2`.
Stopping a trainer does not force a final save; unsaved progress may be lost.

## Frozen behavior memory

This change does not claim to adapt the release's five-task RoboCasa memory
bank to Bridge. CTE/retrieval are frozen and precomputed; changing
`requires_grad` alone cannot train them. A proper adaptation trains the Stage-1
CTE on Bridge, rebuilds a Bridge effect bank, adapts Stage-3 retrieval, then
rebuilds features and trains Stage-2. An ablation can measure whether the current
bank helps or harms. Making horizon 32 smaller does not fix memory-domain
mismatch, and changes to that horizon must also update behavior-PBD heads and
checkpoint shapes. Keep horizon 32 and use masked tail windows for now.

## External-bank ablation

`--memory-mode bank_off` does not load the external task-context bank or retrieval
head, and supplies zero global context. It keeps the CTE, within-episode phase
and effect history, and all policy weights (including learned injection biases).
The default `on` is unchanged. The client checks the server's memory-mode
metadata before running any episode. This is an inference-only ablation, not
training without memory or disabling every Zeva component.

Run a matched bank-off/on comparison, each with 20 episodes, 30 denoising steps,
150 control steps and guidance 3, after checking GPU availability:

```bash
WIDOWX_EVAL_GPU=6 WIDOWX_RENDER_GPU=2 OMP_NUM_THREADS=2 \
  /home/gpu4/tianyi/cosmos-framework/.venv/bin/python scripts/run_widowx_memory_ablation.py \
  --checkpoint /home/gpu4/tianyi/Zeva/.artifacts/ipec_full/checkpoints/step_000290000.pt \
  --output /home/gpu4/tianyi/Zeva/.artifacts/CHOOSE_NEW_ABLATION_DIRECTORY \
  --episodes 20 --exec-horizon 16 --port 18825
```

The execution horizon is independent of the 32-action prediction horizon and
the 30-step denoising count. At 5 Hz, executing 16 actions means replanning every
3.2 seconds (four actions: 0.8 seconds). History still records all executed
controls at four-control CTE boundaries. Each mode saves videos and JSON traces
under its own `eval/`; `comparison.json` appears only after BOTH modes finish.
The controller validates checkpoint, seeds and protocol before accepting reports.
`--after /absolute/path/to/previous/comparison.json` queues a comparison without
loading a model until the preceding comparison has completed successfully.

The 2026-09-24 runs are `.artifacts/ipec_memory_ablation_290000_20260924` (four
actions) and `.artifacts/ipec_memory_ablation_290000_h16_20260924` (16 actions,
queued after the first). The latter runs in tmux `Zeva:memory_eval16`, with
`launcher.log` in its output directory. These evaluate the old 290k baseline;
the corrected full-data pipeline is still preparing its feature cache.

See [Bridge memory audit and implementation plan](../docs/bridge_memory_plan.md)
for the actual released bank contents, missing training/export components,
data-label requirements, and disk-space estimates. No Bridge bank has been built
or substituted as part of this diagnostic evaluation.

CPU regressions:

```bash
PYTHONPATH=$PWD:/home/gpu4/tianyi/cosmos-framework CUDA_VISIBLE_DEVICES='' \
  /home/gpu4/tianyi/cosmos-framework/.venv/bin/python -m unittest \
  scripts.test_widowx_full scripts.test_fp32_master_adamw scripts.test_wandb_fallback \
  scripts.test_widowx_memory -v
```

The actual rollout loop also has CPU-only fake-environment tests for horizons
3/4/16/32, all 150 control steps, partial final chunks, history and mode checks.
Run these with the installed SimplerEnv dependencies (no GPU context is created):

```bash
PYTHONPATH=$PWD:/opt/benchmarks/SimplerEnv CUDA_VISIBLE_DEVICES='' \
  /home/gpu4/miniconda3/envs/simpler_env/bin/python -m unittest \
  scripts.test_widowx_execution -v
```