README.md

---
license: apache-2.0
base_model: lerobot/pi0
tags:
  - robotics
  - lerobot
  - pi0
  - vla
  - so101
  - so-arm101
  - manipulation
  - pick-and-place
pipeline_tag: robotics
library_name: lerobot
---

# SO-ARM101 pick-and-place — π0 full fine-tune (9 tasks)

A [π0](https://www.physicalintelligence.company/blog/pi0) flow-matching VLA policy fine-tuned on a
9-task SO-ARM101 tabletop pick-and-place dataset. Language-conditioned: one checkpoint serves all
nine tasks, selected by the instruction string you pass at inference.

Trained with [LeRobot](https://github.com/huggingface/lerobot) (`lerobot-train`), stock recipe,
full-parameter fine-tuning (vision encoder **not** frozen, action expert **not** trained in isolation).

> The base-model tag may render as `lerobot/pi0_old`: upstream renamed that repo and `lerobot/pi0`
> now redirects to it. It is the same weights this was fine-tuned from.

## Tasks

Nine tabletop tasks. The instruction must be passed **verbatim** — the strings below are exactly the
ones present in the training data, including the article oddity in `a eraser` and the missing `it`
in `place in the bin`.

| Instruction | Episodes | Frames |
|---|---|---|
| `Pick up a battery and place in the bin` | 200 | 56,295 |
| `Pick up a can and place in the bin` | 250 | 82,792 |
| `Pick up a cube and place in the bin` | 300 | 106,085 |
| `Pick up a eraser and place in the bin` | 300 | 109,899 |
| `Pick up a golf and place in the bin` | 250 | 77,409 |
| `Pick up a medicine bottle and place in the bin` | 300 | 116,342 |
| `Pick up a plush toy and place in the bin` | 200 | 80,811 |
| `Stack the cube on the can` | 200 | 74,425 |
| `Stack the smaller cube on the larger one` | 200 | 80,905 |
| **Total** | **2,200** | **784,963** |

All nine were merged into a single training set with **no re-balancing / no up-sampling** — episode
counts are already roughly uniform, so the merge is balanced as-is.

## Data

- Source: [`zhuzhuangtian/so101-pick-place-tasks`](https://www.modelscope.cn/datasets/zhuzhuangtian/so101-pick-place-tasks) on ModelScope (Apache-2.0), LeRobot v3.0 format.
- Robot: SO-ARM101 follower arm, 6 DoF.
- Cameras: two RGB streams, `observation.images.top` and `observation.images.wrist`, 480×640 each, 30 fps.
- `observation.state` and `action` are both `float32(6)`: `shoulder_pan`, `shoulder_lift`,
  `elbow_flex`, `wrist_flex`, `wrist_roll`, `gripper` (joint positions, **absolute** targets — not deltas).

## Training

| | |
|---|---|
| Base | `lerobot/pi0` (3.3 B params), full fine-tune |
| Steps | 30,000 (≈ 2.4 epochs at effective batch 64) |
| Batch | 8 per device × 8 devices = 64 effective |
| Precision | bfloat16 |
| Optimizer | AdamW, lr 2.5e-5, wd 0.01, betas (0.9, 0.95) |
| Schedule | cosine decay, 2,000 warmup steps, decay to 2.5e-6 |
| Action chunk | 50 (`chunk_size` = `n_action_steps` = 50) |
| Augmentation | LeRobot stock image transforms, strengthened photometric range: brightness/contrast 0.7–1.3, hue ±0.05, saturation 0.5–1.5, sharpness 0.5–1.5, affine ±5° / translate 0.05; up to 5 of 6 sampled per frame |
| Normalization | state/action MEAN_STD from real dataset statistics; visual IDENTITY |

**Published checkpoint = step 25,000**, not the final step. All 12 saved checkpoints were evaluated
offline and step 25,000 was best on both metrics below; the curve is U-shaped (degrades after 25k),
so training longer would have been worse.

## Offline evaluation

There is no simulator for this rig, so evaluation is offline only, against a
**copy-the-current-joint-angles baseline**:

- `ratio = MAE(prediction, action) / MAE(current_state, action)` — must be **< 1**, i.e. the policy
  must beat "just hold still". This is a much harder bar than raw correlation: the naive baseline
  already scores 0.93–0.997 correlation on every joint, so `corr > 0.9` has no discriminative power
  on this data and is not used here.
- `delta_corr = corr(prediction − state, action − state)` — must be **> 0** (direction of motion).
- `delta_std` — guards against a policy that collapses to shrinking outputs.

Published checkpoint (200 frames sampled uniformly across the whole dataset):

| Joint | ratio ↓ | delta_corr ↑ |
|---|---|---|
| shoulder_pan | 0.548 | 0.880 |
| shoulder_lift | 0.497 | 0.898 |
| elbow_flex | 0.394 | 0.923 |
| wrist_flex | 0.775 | 0.627 |
| wrist_roll | 0.906 | 0.796 |
| gripper | 0.368 | 0.864 |
| **worst** | **0.906** | **0.627** |

All six joints beat the baseline. **Weakest axes are the two wrist DoF** (`wrist_roll`, `wrist_flex`)
— those are the most likely to misbehave first on hardware.

## Real-robot status

One closed-loop run has been done: the policy drove a physical SO-ARM101 through
`Pick up a plush toy and place in the bin` with both cameras live and inference served remotely from
a GPU host, visually confirmed correct by a human observer. **No quantified success rate exists.**
Treat this as "the checkpoint loads, the scale is right, and it moves sensibly", not as a validated
success rate.

## Usage

```python
from lerobot.policies.pi0.modeling_pi0 import PI0Policy

policy = PI0Policy.from_pretrained("Harrysunshine/so101-pi0-9task")
# batch: observation.state (6,), observation.images.top (3,480,640),
#        observation.images.wrist (3,480,640), task = one instruction string from the table above
action = policy.select_action(batch)
```

Or as a LeRobot policy server for a low-power robot host:

```bash
# on the GPU host
lerobot-policy-server --host=0.0.0.0 --port=8080
# on the robot host
lerobot-robot-client \
  --policy_type=pi0 \
  --pretrained_name_or_path=Harrysunshine/so101-pi0-9task \
  --task="Pick up a plush toy and place in the bin" \
  --actions_per_chunk=50
```

The two `*normalizer_processor.safetensors` files are **required**. Without them LeRobot silently
falls back to identity normalization: loss-free, but every action comes out at the wrong scale and
the arm will fly off. Load the repo as a whole, do not cherry-pick `model.safetensors`.

Only inference assets are published here (weights, config, pre/post-processors, training config).
Optimizer/scheduler/RNG state is not included, so this repo cannot be used to resume training.

## Limitations

- **Rig-specific top camera.** The training data comes from a different physical setup than ours:
  the wrist view matches pixel-for-pixel (same SO-ARM101 follower hardware), but the **top camera
  extrinsics differ** — the same joint configuration images the arm at a different position in frame.
  Dropping this checkpoint onto another SO-ARM101 will not work without camera adaptation or a short
  fine-tune on data from the target rig.
- Offline metrics do not predict real-world success. Related work on other arms has produced
  checkpoints with 0.995 offline correlation that still failed to grasp on hardware.
- Absolute joint-position action space. Sending these as deltas, or recording target data with a
  per-session origin offset baked in, will diverge in closed loop.
- Fixed 480×640 dual-camera observation. Single-camera or differently-cropped inputs are out of
  distribution.
- Tabletop objects and bin from the training scene only; no generalization claims beyond them.

## Related

- [`Harrysunshine/so101-smolvla-9task`](https://huggingface.co/Harrysunshine/so101-smolvla-9task) — SmolVLA on the same merged dataset, same evaluation harness (cost/accuracy comparison point).
- [`Harrysunshine/so101-act-cube`](https://huggingface.co/Harrysunshine/so101-act-cube) — ACT on the single cube task.

## License

Apache-2.0, following the `lerobot/pi0` base model and the source dataset.

---

## 中文说明

SO-ARM101 桌面抓放 **π0 全参微调**权重，9 个任务合并训练，语言条件——一个 checkpoint 覆盖全部九个任务，
靠推理时传入的指令字符串区分。指令必须与上表**逐字一致**（注意 `a eraser` 和 `place in the bin` 里没有 `it`）。

- 数据：ModelScope `zhuzhuangtian/so101-pick-place-tasks`（Apache-2.0，LeRobot v3.0），
  2200 集 / 784,963 帧 / 双相机 480×640 / 6 维绝对关节位置。九任务全量合并，不做偏斜上采样。
- 训练：`lerobot-train` 官方入口，全参微调，30k 步，有效 batch 64，bf16，余弦退火 + 2000 步 warmup。
- **发布的是第 25,000 步，不是末点**：12 个存点逐点离线验收，25k 双指标同时最优，曲线 U 形（25k 后退化）。
- 验收判据不用 `corr > 0.9`：「照抄当前关节角」这个恒等基线的逐关节 corr 就有 0.93–0.997，全部过门、零筛选力。
  改用 `ratio = MAE(预测) / MAE(照抄)` 必须 < 1，配 `delta_corr` 与防坍缩的 `delta_std`。
- 六关节 ratio 全部 < 1；**短板是腕部两个自由度**（`wrist_roll` 0.906、`wrist_flex` 0.775）。
- 真机只做过一次闭环目检通过（抓毛绒玩具放进筐，双相机 + 远程 GPU 推理），**没有量化成功率**。
- 两个 `*normalizer_processor.safetensors` 必须一起加载，缺了会静默退回恒等归一化，动作尺度整体错、机械臂跑飞。
- **边界**：这批数据的 top 相机外参与我们自己的臂不同（wrist 视图逐像素同款，top 不同），
  直接换机器用不了，需要相机适配或用目标机器的少量数据再微调。