--- license: other language: - en library_name: openpi tags: - robotics - vision-language-action - openpi - pi0.5 - so101 - lerobot - orbax --- # pi0.5 SO-101 Erythromycin-on-Tea OpenPI pi0.5 fully fine-tuned for a dual-camera SO-101 follower on one tabletop manipulation task: > Pick up the red erythromycin ointment box and place it on top of the green Rizhao tea tin. Author: **CodeChild** This repository contains inference-only OpenPI/Orbax artifacts. It is not a standard Transformers checkpoint and cannot be loaded with `transformers.AutoModel.from_pretrained()`. ## Model details | Field | Value | | --- | --- | | Base checkpoint | `gs://openpi-assets/checkpoints/pi05_base/params` | | Fine-tuning | Full parameters, 8000 optimizer updates | | Inference weights | EMA parameters, decay 0.99 | | Cameras | Fixed RGB + wrist RGB, 640x480 at collection time | | State/action | 6-D calibrated SO-101 position space | | Action horizon | 50 steps at 30 Hz | | Output | `(50, 6)` absolute SO-101 position targets | | Code release tag | `so101-pi05-erythromycin-v1` | | W&B | | The full deployment and safety handoff is in [`SO101_PI05_HANDOFF.md`](./SO101_PI05_HANDOFF.md). Read it before connecting this policy to motors. ## Repository contents ```text params/ # EMA inference parameters, about 12 GiB assets/ # clean-train normalization statistics _CHECKPOINT_METADATA # original Orbax checkpoint metadata code/so101-pi05-erythromycin-v1.patch SO101_PI05_HANDOFF.md LICENSE_OPENPI.txt LICENSE_GEMMA.txt NOTICE ``` The optimizer state is intentionally not published. This repository is suitable for inference, not direct training resume. ## Code setup The SO-101 adapter was developed from OpenPI commit: ```text 15a9616a00943ada6c20a0f158e3adb39df2ccac ``` The release commit is tagged locally as `so101-pi05-erythromycin-v1`. Because the source checkout only has the upstream Physical Intelligence remote, the exact code delta is also included in this model repository: ```bash git clone https://github.com/Physical-Intelligence/openpi.git cd openpi git checkout 15a9616a00943ada6c20a0f158e3adb39df2ccac git apply /path/to/so101-pi05-erythromycin-v1.patch GIT_LFS_SKIP_SMUDGE=1 UV_LINK_MODE=copy uv sync ``` The data config expects the portable dataset package next to the OpenPI checkout when running data-dependent scripts. Policy inference only needs this repository's `params/` and `assets/`. ## Download and serve ```python from huggingface_hub import snapshot_download checkpoint_dir = snapshot_download("CodeChild/pi05-so101-erythromycin-tea") print(checkpoint_dir) ``` From the patched OpenPI checkout: ```bash CUDA_VISIBLE_DEVICES= .venv/bin/python scripts/serve_policy.py \ policy:checkpoint \ --policy.config pi05_so101_erythromycin \ --policy.dir ``` Policy input: ```python observation = { "observation/state": state_float32_6, "observation/fixed_image": fixed_rgb_uint8_hwc, "observation/wrist_image": wrist_rgb_uint8_hwc, "prompt": "Pick up the red erythromycin ointment box and place it on top of the green Rizhao tea tin.", } ``` The server returns `result["actions"]` with shape `(50, 6)`. ## Critical action semantics During training, dimensions 0-4 are represented relative to the same current state and dimension 5 remains absolute: ```text delta[t, 0:5] = absolute_target[t, 0:5] - current_state[0:5] delta[t, 5] = absolute_target[t, 5] ``` This is not a step-to-step increment. OpenPI applies the inverse transform before returning actions, so the policy server output is already absolute. A robot client must **not** take a cumulative sum and must **not** add the current state again. The trajectory was collected at 30 Hz. Fifty predicted steps correspond to approximately 1.67 seconds of control ticks. For an initial supervised robot test, execute a short prefix and replan; the accompanying handoff recommends starting with 5 steps at 30 Hz. This is a deployment recommendation, not a robot-validated hyperparameter. ## Training data The source dataset has 90 episodes and two synchronized camera streams. It is not redistributed in this model repository. The authoritative split was `splits/split_manifest.json`: | Split | Episodes | Frames | Use | | --- | ---: | ---: | --- | | `clean_train` | 67 | 15070 | Training and normalization statistics | | `clean_val` | 18 | 3972 | Offline validation only | | `recovery` | 5 | 1515 | Excluded | The default `train: 0:90` field in the source LeRobot metadata was not used because it would leak validation and recovery episodes into training. OpenPI's standard training augmentation was active: crop/rotation/color augmentation for the fixed camera and color augmentation for the wrist camera. Evaluation and inference use no random augmentation. ## Offline evaluation | Model | Split | Samples | Flow-matching loss | | --- | --- | ---: | ---: | | Original pi0.5 base | `clean_val` | 3972 | 0.04753249 | | Fine-tuned checkpoint | `clean_train` | 15068 | 0.00407172 | | Fine-tuned checkpoint | `clean_val` | 3972 | 0.01467515 | The held-out validation loss is 69.126% lower than the base checkpoint under this evaluation. The validation/train ratio is 3.604, indicating a generalization gap. There is no independent test split. Flow-matching loss is not a robot task-success metric. No closed-loop real-robot success rate has been measured for this checkpoint yet. ## Intended use and limitations - Intended for research and supervised evaluation on the stated SO-101 task. - Requires the same joint order, direction, zero points, gripper calibration, camera assignment, RGB convention and 30 Hz timing used during collection. - Before motor execution, validate finite values and shape, enforce hardware joint/gripper limits, maximum target deltas, velocity/workspace limits, timeouts and an emergency stop. - Start with motors-off shadow inference, then low-speed supervised closed-loop tests. - The model was trained on one task with a small dataset and may fail under new layouts, lighting, camera movement, object appearance or calibration drift. - Do not infer safety or reliability from the offline flow-matching loss. ## Licenses OpenPI code is provided under Apache-2.0; see `LICENSE_OPENPI.txt`. The model is derived from pi0.5, which includes Gemma components. Gemma use and redistribution are subject to the Gemma Terms of Use in `LICENSE_GEMMA.txt`, and the required notice is provided in `NOTICE`. For this reason the Hugging Face metadata uses `license: other` rather than describing the complete artifact as Apache-2.0 only.