| --- |
| license: apache-2.0 |
| language: |
| - en |
| library_name: openpi |
| pipeline_tag: robotics |
| tags: |
| - openpi |
| - pi05 |
| - robotics |
| - manipulation |
| - bimanual |
| - tactile |
| - lora |
| --- |
| |
| # pi05_bi — two_tubes_all, step 8000 |
| |
| openpi `pi05_bi` checkpoint fine-tuned on a bimanual two-tube pick-and-place task |
| with paired RGB + tactile input. **Step 8000 is the checkpoint with the best |
| held-out validation loss** for this run (see the curve below). |
|
|
| ## Task |
|
|
| > Firstly, use the left hand to pick up the blue tube, and then use the right hand |
| > to pick up the green tube. Next, use the left hand to place the blue tube back |
| > firstly, and then use the right hand to place the green tube back. |
|
|
| ## Data |
|
|
| Derived from [`KaiyueChen/two_tubes_01`](https://huggingface.co/datasets/KaiyueChen/two_tubes_01) |
| (LeRobot v2.1, 30 fps, `robot_type: bimanual`). |
|
|
| | | | |
| |---|---| |
| | Episodes | 519 (train 467 / val_seen 52 / val_unseen 52) | |
| | Frames | 425,115 | |
| | 1 epoch | 2,983 steps → step 8000 ≈ **2.68 epochs** | |
| | Observation | 6 × 224×224×3 images: `camera0`, `camera1`, `tactile_{left,right}_{0,1}` | |
| | State / action | 20-D each | |
|
|
| The split is per-source stratified with 10% held out (`seed=42`). `val_unseen` |
| episodes are never trained on; `val_seen` is a same-sized sample drawn from the |
| training episodes, so the two are directly comparable and their difference |
| isolates memorisation from generalisation. |
|
|
| **Normalisation statistics were computed on the training split only** (quantile |
| q01/q99, as PI05 expects) and ship with the checkpoint at |
| `checkpoint/assets/two_tubes_all/norm_stats.json`. Computing them over the full |
| dataset would leak held-out episodes into the input scaling. |
|
|
| ## Training |
|
|
| | | | |
| |---|---| |
| | Base | `pi05_bi` (PaliGemma + SigLIP vision tower, flow-matching action expert) | |
| | Adaptation | LoRA rank 16 on the LLM, rank 32 on the action expert; **the vision tower is fully fine-tuned** (the freeze filter only matches `.*llm.*`, so ~413M of 463M trainable parameters are ViT weights) | |
| | Hardware | 2 × A100-80GB, FSDP | |
| | Batch size | 128 | |
| | Total steps | 12,000 (this checkpoint: 8,000) | |
|
|
| ## Validation |
|
|
| | step | train | val_seen | val_unseen | gap | |
| |---:|---:|---:|---:|---:| |
| | 2000 | 0.0547 | 0.0549 | 0.0614 | +0.0065 | |
| | 4000 | 0.0475 | 0.0486 | 0.0548 | +0.0062 | |
| | 6000 | 0.0443 | 0.0463 | 0.0551 | +0.0088 | |
| | **8000** | **0.0419** | **0.0436** | **0.0542** | +0.0106 | |
| | 10000 | 0.0403 | 0.0416 | 0.0553 | +0.0137 | |
|
|
| `val_unseen` bottoms out at step 8000 and does not improve afterwards: from step |
| 4000 to 10000 it moves by only +0.9% while `val_seen` drops 14.4% and the gap |
| more than doubles. Everything learned past ~step 4000 is memorisation rather |
| than generalisation, which is why this checkpoint — not the final one — is the |
| one to use. |
|
|
| Caveat on the numbers: each validation pass covers 20 batches × 128 frames |
| ≈ 2,560 frames, which for ~800-frame episodes is only 3–4 episodes per split. |
| The absolute `val_unseen` trend is meaningful; the exact value of `gap` is not. |
|
|
| ## Layout |
|
|
| ``` |
| checkpoint/ |
| ├── _CHECKPOINT_METADATA |
| ├── assets/two_tubes_all/norm_stats.json |
| ├── params/ # orbax OCDBT, inference weights |
| └── train_state/ # optimiser state, for resuming |
| ``` |
|
|
| Load with openpi by pointing the config's checkpoint directory at `checkpoint/`. |
| Inference only needs `params/` and `assets/`; `train_state/` is for resuming. |
|
|
