README.md

---
license: other
language:
  - en
library_name: openpi
tags:
  - robotics
  - vision-language-action
  - openpi
  - pi0.5
  - so101
  - lerobot
  - orbax
---

# pi0.5 SO-101 Erythromycin-on-Tea

OpenPI pi0.5 fully fine-tuned for a dual-camera SO-101 follower on one tabletop manipulation task:

> Pick up the red erythromycin ointment box and place it on top of the green Rizhao tea tin.

Author: **CodeChild**

This repository contains inference-only OpenPI/Orbax artifacts. It is not a standard Transformers checkpoint and cannot be loaded with `transformers.AutoModel.from_pretrained()`.

## Model details

| Field | Value |
| --- | --- |
| Base checkpoint | `gs://openpi-assets/checkpoints/pi05_base/params` |
| Fine-tuning | Full parameters, 8000 optimizer updates |
| Inference weights | EMA parameters, decay 0.99 |
| Cameras | Fixed RGB + wrist RGB, 640x480 at collection time |
| State/action | 6-D calibrated SO-101 position space |
| Action horizon | 50 steps at 30 Hz |
| Output | `(50, 6)` absolute SO-101 position targets |
| Code release tag | `so101-pi05-erythromycin-v1` |
| W&B | <https://wandb.ai/99087192-zhejiang-university/openpi/runs/xgk0h74f> |

The full deployment and safety handoff is in [`SO101_PI05_HANDOFF.md`](./SO101_PI05_HANDOFF.md). Read it before connecting this policy to motors.

## Repository contents

```text
params/                         # EMA inference parameters, about 12 GiB
assets/                         # clean-train normalization statistics
_CHECKPOINT_METADATA            # original Orbax checkpoint metadata
code/so101-pi05-erythromycin-v1.patch
SO101_PI05_HANDOFF.md
LICENSE_OPENPI.txt
LICENSE_GEMMA.txt
NOTICE
```

The optimizer state is intentionally not published. This repository is suitable for inference, not direct training resume.

## Code setup

The SO-101 adapter was developed from OpenPI commit:

```text
15a9616a00943ada6c20a0f158e3adb39df2ccac
```

The release commit is tagged locally as `so101-pi05-erythromycin-v1`. Because the source checkout only has the upstream Physical Intelligence remote, the exact code delta is also included in this model repository:

```bash
git clone https://github.com/Physical-Intelligence/openpi.git
cd openpi
git checkout 15a9616a00943ada6c20a0f158e3adb39df2ccac
git apply /path/to/so101-pi05-erythromycin-v1.patch
GIT_LFS_SKIP_SMUDGE=1 UV_LINK_MODE=copy uv sync
```

The data config expects the portable dataset package next to the OpenPI checkout when running data-dependent scripts. Policy inference only needs this repository's `params/` and `assets/`.

## Download and serve

```python
from huggingface_hub import snapshot_download

checkpoint_dir = snapshot_download("CodeChild/pi05-so101-erythromycin-tea")
print(checkpoint_dir)
```

From the patched OpenPI checkout:

```bash
CUDA_VISIBLE_DEVICES=<GPU_ID> .venv/bin/python scripts/serve_policy.py \
  policy:checkpoint \
  --policy.config pi05_so101_erythromycin \
  --policy.dir <HF_SNAPSHOT_DIR>
```

Policy input:

```python
observation = {
    "observation/state": state_float32_6,
    "observation/fixed_image": fixed_rgb_uint8_hwc,
    "observation/wrist_image": wrist_rgb_uint8_hwc,
    "prompt": "Pick up the red erythromycin ointment box and place it on top of the green Rizhao tea tin.",
}
```

The server returns `result["actions"]` with shape `(50, 6)`.

## Critical action semantics

During training, dimensions 0-4 are represented relative to the same current state and dimension 5 remains absolute:

```text
delta[t, 0:5] = absolute_target[t, 0:5] - current_state[0:5]
delta[t, 5]   = absolute_target[t, 5]
```

This is not a step-to-step increment. OpenPI applies the inverse transform before returning actions, so the policy server output is already absolute. A robot client must **not** take a cumulative sum and must **not** add the current state again.

The trajectory was collected at 30 Hz. Fifty predicted steps correspond to approximately 1.67 seconds of control ticks. For an initial supervised robot test, execute a short prefix and replan; the accompanying handoff recommends starting with 5 steps at 30 Hz. This is a deployment recommendation, not a robot-validated hyperparameter.

## Training data

The source dataset has 90 episodes and two synchronized camera streams. It is not redistributed in this model repository. The authoritative split was `splits/split_manifest.json`:

| Split | Episodes | Frames | Use |
| --- | ---: | ---: | --- |
| `clean_train` | 67 | 15070 | Training and normalization statistics |
| `clean_val` | 18 | 3972 | Offline validation only |
| `recovery` | 5 | 1515 | Excluded |

The default `train: 0:90` field in the source LeRobot metadata was not used because it would leak validation and recovery episodes into training.

OpenPI's standard training augmentation was active: crop/rotation/color augmentation for the fixed camera and color augmentation for the wrist camera. Evaluation and inference use no random augmentation.

## Offline evaluation

| Model | Split | Samples | Flow-matching loss |
| --- | --- | ---: | ---: |
| Original pi0.5 base | `clean_val` | 3972 | 0.04753249 |
| Fine-tuned checkpoint | `clean_train` | 15068 | 0.00407172 |
| Fine-tuned checkpoint | `clean_val` | 3972 | 0.01467515 |

The held-out validation loss is 69.126% lower than the base checkpoint under this evaluation. The validation/train ratio is 3.604, indicating a generalization gap. There is no independent test split.

Flow-matching loss is not a robot task-success metric. No closed-loop real-robot success rate has been measured for this checkpoint yet.

## Intended use and limitations

- Intended for research and supervised evaluation on the stated SO-101 task.
- Requires the same joint order, direction, zero points, gripper calibration, camera assignment, RGB convention and 30 Hz timing used during collection.
- Before motor execution, validate finite values and shape, enforce hardware joint/gripper limits, maximum target deltas, velocity/workspace limits, timeouts and an emergency stop.
- Start with motors-off shadow inference, then low-speed supervised closed-loop tests.
- The model was trained on one task with a small dataset and may fail under new layouts, lighting, camera movement, object appearance or calibration drift.
- Do not infer safety or reliability from the offline flow-matching loss.

## Licenses

OpenPI code is provided under Apache-2.0; see `LICENSE_OPENPI.txt`.

The model is derived from pi0.5, which includes Gemma components. Gemma use and redistribution are subject to the Gemma Terms of Use in `LICENSE_GEMMA.txt`, and the required notice is provided in `NOTICE`. For this reason the Hugging Face metadata uses `license: other` rather than describing the complete artifact as Apache-2.0 only.