ASethi04/pi05-BimanualYAM-oranges

π0.5 · Bimanual YAM · "Put all oranges in the bowl"

Full fine-tune of lerobot/pi05_base (π0.5, PaliGemma-2B VLM + 300M action expert, flow-matching action head) on brandonyang/yam-vive-teleop — 80 episodes / 74,927 frames of bimanual YAM VR teleoperation (3 RealSense cameras: top + left/right wrist, 480×270 @ 30 fps) for the task Put all oranges in the bowl (use this exact string at inference).

Directly comparable to the MolmoAct2 fine-tunes (ASethi04/MolmoAct2-BimanualYAM-oranges-12k): same dataset, 12,000 optimizer steps, global batch 64, seed 1000, same augmentation.

Action / observation interface

• Action: 30-step chunks of 14-D absolute joint positions [left_joint_0..5, left_gripper, right_joint_0..5, right_gripper] at 30 Hz; grippers in [0,1]. chunk_size = n_action_steps = 30.
• Observation: 3 RGB cameras + 14-D state (quantile-normalized, discretized into the π0.5 text prompt), plus the task string.
• Camera mapping ships inside the saved processor pipelines: observation.images.{top,left,right} → π0.5 slots {base_0_rgb, left_wrist_0_rgb, right_wrist_0_rgb} — feed raw dataset-style keys.

Training configuration

Field	Value
Steps / global batch / epochs	12,000 / 64 (8×RTX PRO 6000, bs 8/GPU) / ≈10.3
Precision	fp32 weights + fp32 AdamW (TF32 matmul). Deliberate: LeRobot's bf16 path keeps AdamW moments in bf16 with no fp32 master weights, and updates at lr 2.5e-5 round to zero
LR schedule	AdamW (0.9, 0.95), wd 0.01, clip 1.0; cosine 2.5e-5 → 2.5e-6, warmup 600
Normalization	STATE/ACTION quantiles (q01/q99 → [−1,1]), VISUAL identity
Augmentation	ColorJitter (brightness/contrast/saturation 0.8–1.2, hue ±0.02), sharpness 0.5–1.5, affine ±5° / 5% translate (≤3 sampled per frame)
Gradient checkpointing	on; torch.compile off (upstream #4178)
EMA	none (not implemented in LeRobot; openpi uses 0.99)
Code	AfterQuery-Research/lerobot @ molmoact2-pr4249-fixes (ccb4e14b; pi05 paths byte-identical to upstream main @ 2026-08-04)

Corrected normalization stats: the dataset's stock q01/q99 carry LeRobot's per-episode weighted-mean aggregation bug (lerobot#4156), which left 18.6% of action/state values outside [−1,1] after normalization. This model was trained with exact global quantiles recomputed over all 74,927 frames (1.8% outside, the expected tail) — the corrected stats are embedded in this repo's processor safetensors, so inference is self-consistent. Do not re-normalize with the hub dataset's stock stats.

Offline eval (final checkpoint, step 12000)

Open-loop 30-step action-chunk error on training episodes [3, 17, 41, 62, 78], stride 60, no augmentation (104 samples). All-in-sample (100/0 split): these gate brokenness and memorization — they do not predict task success. Hardware rollouts pending.

Metric	π0.5 12k	MolmoAct2 run 2 (same protocol)
Joint L1 (rad)	0.0071	0.0515
Gripper L1	0.0029	0.0036
Shuffled-camera L1 ratio	5.9×	1.53×
Jerk (mean abs 2nd diff)	0.0029	0.002
Non-finite predictions	0	0

L1 is nearly flat across the 30-step horizon (0.0065 → 0.0079). Prediction spread is healthy (σ across samples 0.38; gripper σ 0.186 — no constant-output degeneracy). ~4.7% of predicted scalars overshoot joint/gripper bounds by tiny margins (worst ≈ 0.02 rad) where the demonstrations sit exactly on the boundary (j1/j2 at 0.0, grippers at 1.0) — flow-matching regression noise; rely on client-side clamping.

Deployment notes (YAM / AfterQuery stack)

• Task string must be exactly Put all oranges in the bowl.
• Set --robot.max_gripper_delta=0.05 (data contains 0.05/tick gripper moves; the 0.03 default throttles grasps). Cameras must be named top, left, right.
• fp32 inference: ~16 GB VRAM + activations → use a ≥24 GB GPU (bf16 autocast at inference is safe if needed; the bf16 hazard is training-only). 10 denoising steps per 1 s chunk.
• Local lerobot-rollout: pass --rename_map='{"observation.images.top":"observation.images.base_0_rgb","observation.images.left":"observation.images.left_wrist_0_rgb","observation.images.right":"observation.images.right_wrist_0_rgb"}' — required both to skip the visual-feature check and because an empty CLI map would overwrite the checkpoint's saved rename step.
• Remote policy server (remote_inference/backend.py): needs three small fixes for pi05 checkpoints (generic branch loads weights from config pretrained_path; manifest derives 32-D padded state and renamed camera keys). config.json here ships pretrained_path: null so a stock server fails loudly instead of silently serving base weights. Patch staged in the training workspace (molmoact-ft/proposed/).
• Known train/pretrain deviations: state prompt carries 14 values (this port doesn't pad to openpi's 32); no EMA; LeRobot pi05 has open image-padding misalignment vs openpi (#2891) — consistent between this training and LeRobot inference, may differ from openpi-native runs.

Training curves: wandb aq-robotics/molmoact2-yam-oranges (run pi05-12k, resumed at step 7000 after a storage-quota crash — two wandb entries, one training trajectory; checkpoint lineage verified).