Ctrl-World Robo Multiview Finetuning

This package keeps the new finetuning logic outside models/Ctrl-World.

The first target is Airbot/MMK2 data under:

datasets/humanoid/multiview/non_makovian

The default action interface is eef_sim_pose_action with 12 dimensions. This is intentional: Airbot raw action is 36D and Leju raw action is 54D, while the EEF pose fields provide a shared multi-embodiment interface.

Prepare Airbot Latents

bash finetuning/scripts/prepare_airbot.sh

Useful smoke-test override:

MAX_EPISODES_PER_DATASET=2 LIMIT_RECORDS=4 bash finetuning/scripts/prepare_airbot.sh

Outputs are written to:

datasets/processed/ctrl_world_robo_multiview

Important files:

manifest.json
meta/train_sample.json
meta/val_sample.json
meta/stat.json
datasets/<dataset>/annotation/<split>/episode_*.json
datasets/<dataset>/latent_videos/<split>/episode_*/<view>.pt

Dataset Smoke Test

python -m finetuning.data.dataset_robo_ctrl_world \
  --processed-root datasets/processed/ctrl_world_robo_multiview \
  --mode train \
  --batch-size 2

Expected shapes:

latent: [B, 11, 4, 96, 40] for 4 views at 192x320
action: [B, 11, 12]

Train

Single-node or debug:

GPUS_PER_NODE=1 bash finetuning/scripts/train_airbot_multinode.sh \
  --max-train-steps 100 \
  --checkpointing-steps 50 \
  --validation-steps 50 \
  --train-batch-size 1

Multi-node:

NNODES=2 GPUS_PER_NODE=8 NODE_RANK=$NODE_RANK MASTER_ADDR=$MASTER_ADDR \
  bash finetuning/scripts/train_airbot_multinode.sh

The trainer saves:

datasets/processed/ctrl_world_robo_multiview/checkpoints/latest.pt
datasets/processed/ctrl_world_robo_multiview/checkpoints/checkpoint-<step>.pt

Replay Eval

bash finetuning/scripts/eval_airbot.sh \
  --ckpt datasets/processed/ctrl_world_robo_multiview/checkpoints/latest.pt

Replay eval uses ground-truth actions. It does not use pi0.5.

Notes

pi0.5 finetuning is intentionally separate. The world model can be trained and validated with replay actions first.
Leju can be added by changing --include-pattern after Airbot is stable; missing fourth view is represented by a zero latent slot.
Existing models/Ctrl-World files are not modified.