---
library_name: openpi
license: apache-2.0
tags: [robotics, pi0, pi05, openpi, so101, lerobot, tactile]
datasets: [endoard/grab_ball_3cam_skin, endoard/grab_cup_3cam_skin, endoard/grab_red_cube_3cam_skin, endoard/grab_green_cube_3cam_skin, endoard/grab_green_cylinder_3cam_skin, endoard/grab_red_cube_distractor_green_cube_3cam_skin, endoard/grab_green_cube_distractor_red_cube_3cam_skin]
---

# pi0.5 SO-101 with a tactile skin

pi0.5 policies for a single SO-101 arm with a 5-chip magnetic tactile skin, fine-tuned from
`pi05_base`. One folder per training run, one sub-folder per checkpoint step.
None has been evaluated on the robot yet.

## What is here

| folder | config | cameras | prompt(s) | data | steps |
|---|---|---|---|---|---|
| `grab_ball_skin/` | `pi05_so101_grab_ball_skin` | 2: base = `fixed` (train: `fixed`/`side` at random), wrist = `handeye` | "Grab the ball" | `endoard/grab_ball_3cam_skin`, first 100 episodes / 46,076 frames | 10000, 20000, 29999 |
| `grab_ball_skin_3cam/` | `pi05_so101_grab_ball_skin_3cam` | 3: `fixed`, `handeye`, `side` | "Grab the ball" | `endoard/grab_ball_3cam_skin`, 150 episodes / 65,989 frames | 10000, 20000, 29999 |
| `grab_cup_skin_3cam/` | `pi05_so101_grab_cup_skin_3cam` | 3: `fixed`, `handeye`, `side` | "Grab the blue cup" | `endoard/grab_cup_3cam_skin`, 166 episodes / 64,812 frames | 10000, 20000, 29999 |
| `grab_3obj_skin_3cam/` | `pi05_so101_grab_3obj_skin_3cam` | 3: `fixed`, `handeye`, `side` | "Grab the red cube" / "Grab the green cube" / "Grab the green cylinder" | `endoard/grab_{red_cube,green_cube,green_cylinder}_3cam_skin` merged, 450 episodes / 161,918 frames, one task per source | 10000, 20000, 28000 |
| `grab_cubes_skin_3cam/` | `pi05_so101_grab_cubes_skin_3cam` | 3: `fixed`, `handeye`, `side` | "Grab the red cube" / "Grab the green cube" (see [Prompts](#prompts)) | `endoard/grab_{red_cube,green_cube,red_cube_distractor_green_cube,green_cube_distractor_red_cube}_3cam_skin` merged, 550 episodes / 191,324 frames | 10000, 20000, 29999 |
| `lift_leave_cubes_skin_3cam/` | `pi05_so101_lift_leave_cubes_skin_3cam` | 3: `fixed`, `handeye`, `side` | "Lift the red cube" / "Lift the green cube" / "Lift the red cube, leave the green cube on the table" / "Lift the green cube, leave the red cube on the table" (see [Prompts](#prompts)) | same 4 cube datasets merged, 550 episodes / 191,324 frames | 10000, 20000, 29999 |

`grab_3obj_skin_3cam` is multi-task: the prompt selects the object. Its run was stopped at
step 28,500 of 30k, so 28000 is its last checkpoint. Its held-out split is stratified per task
(15 episodes each).

`grab_cubes_skin_3cam` and `lift_leave_cubes_skin_3cam` are trained on the same 550 cube episodes
with different prompts; their held-out split is stratified per source (15/15/15/10 episodes).

## Prompts

Every policy here was trained on a fixed set of text prompts. At inference send one of them
**verbatim** as the observation `prompt` (same wording, case and punctuation); other phrasings
are untested.

| folder | training prompts (exact strings) |
|---|---|
| `grab_ball_skin/`, `grab_ball_skin_3cam/` | `Grab the ball` |
| `grab_cup_skin_3cam/` | `Grab the blue cup` |
| `grab_3obj_skin_3cam/` | `Grab the red cube`, `Grab the green cube`, `Grab the green cylinder` |
| `grab_cubes_skin_3cam/` | `Grab the red cube`, `Grab the green cube` |
| `lift_leave_cubes_skin_3cam/` | `Lift the red cube`, `Lift the green cube`, `Lift the red cube, leave the green cube on the table`, `Lift the green cube, leave the red cube on the table` |

Which prompt each source dataset was trained with in the two cube runs:

| source dataset (episodes) | scene | `grab_cubes_skin_3cam` | `lift_leave_cubes_skin_3cam` |
|---|---|---|---|
| `endoard/grab_red_cube_3cam_skin` (150) | red cube alone | `Grab the red cube` | `Lift the red cube` |
| `endoard/grab_green_cube_3cam_skin` (150) | green cube alone | `Grab the green cube` | `Lift the green cube` |
| `endoard/grab_red_cube_distractor_green_cube_3cam_skin` (150) | red cube + green cube | `Grab the red cube` | `Lift the red cube, leave the green cube on the table` |
| `endoard/grab_green_cube_distractor_red_cube_3cam_skin` (100) | green cube + red cube | `Grab the green cube` | `Lift the green cube, leave the red cube on the table` |

So `grab_cubes_skin_3cam` uses one prompt per target colour whether or not the other cube is
on the table, and `lift_leave_cubes_skin_3cam` also names the distractor when there is one.

Each step folder is an openpi checkpoint without optimizer state: `params/` (EMA weights),
`assets/<asset_id>/norm_stats.json` (quantile stats), `_CHECKPOINT_METADATA`, and a `README.md`
with that checkpoint's details.

## Download one checkpoint

```python
from huggingface_hub import snapshot_download
d = snapshot_download("PINlab/pi05-so101", allow_patterns="grab_cup_skin_3cam/29999/*")
# serve with --policy.dir={d}/grab_cup_skin_3cam/29999
```

Or with the CLI, several folders at once (here both cube policies at step 20000, ~12 GB each).
Needs read access to PINlab and `hf auth login` (or `HF_TOKEN`); on an older `huggingface_hub`
use `huggingface-cli download` with the same arguments:

```bash
hf download PINlab/pi05-so101 \
  --include "grab_cubes_skin_3cam/20000/*" "lift_leave_cubes_skin_3cam/20000/*" \
  --local-dir pi05-so101
# -> pi05-so101/grab_cubes_skin_3cam/20000/ and pi05-so101/lift_leave_cubes_skin_3cam/20000/
```

Swap the folder and step in `--include` for any other checkpoint in the table.

## Serving

The checkpoints carry no config. Serve each one under the config in the table, which pins
its `asset_id`:

```bash
python scripts/serve_policy.py policy:checkpoint \
    --policy.config=pi05_so101_grab_cup_skin_3cam \
    --policy.dir=<download>/grab_cup_skin_3cam/29999
```

The policy returns 16 actions **spaced 2 frames apart (15 Hz)**, absolute joints (the delta is
undone server-side). Replay at 15 Hz or interpolate; replaying at 30 Hz halves the speed.
The 3-camera policies need all three cameras every step.

## Recipe (all runs)

- State 21-D (padded to 32): 5 joints (deg), gripper (0-100), 5 skin chips x xyz (uT), raw.
- Action 6-D (padded to 32): **delta on the 5 joints, absolute gripper** (`make_bool_mask(5, -1)`).
- Images: `base_0_rgb` = `fixed`, `left_wrist_0_rgb` = `handeye`; `right_wrist_0_rgb` = `side`
  (mask on) for 3-camera runs, zeros with mask off for `grab_ball_skin`.
- `action_horizon=16`, `action_stride=2`, prompt from the dataset task string (remapped per
  source for the two cube runs, see [Prompts](#prompts)).
- From `pi05_base`, 4 nodes x 1 A100-64GB, `--fsdp-devices=4`, batch 32, AdamW with cosine
  decay, EMA 0.99, seed 42, 30k steps, 10% of episodes held out.