README.md

---
license: apache-2.0
library_name: openpi
pipeline_tag: robotics
tags:
  - robotics
  - manipulation
  - pi0
  - pi05
  - vla
  - embodichain
  - robosynchallenge
  - bimanual
---

# click_bell — π0.5 policy for RoboSynChallenge

A π0.5 (pi05) policy finetuned on the **RoboSynChallenge `click_bell`** task: a bimanual
CobotMagic robot must press a desk bell placed at a random position on the table.

Trained from `pi05_base` on 1,000 synthetic demonstrations
(`RoboSynChallenge/cobotmagic_Sim_click_bell`) for 20,000 steps.

- **Checkpoint**: step 19999 (final)
- **Framework**: [openpi](https://github.com/Physical-Intelligence/openpi) (JAX / Orbax)
- **Simulator**: [EmbodiChain](https://dexforce.github.io/EmbodiChain/) 0.2.3 + dexsim 0.4.3
- **Benchmark**: [RoboSynChallenge](https://github.com/EDEM-AI/RoboSynChallenge)

---

## Results

Evaluated in simulation on an RTX 4090. Every number below is from a run of this exact
checkpoint; nothing is copied from a paper or leaderboard.

### Headline

| Button position range | `clear` (no randomization) | `random` (full domain randomization) |
| --- | --- | --- |
| Repo default — x∈[0.40, 0.85], y∈[−0.30, +0.30] | 13/20 = **65%** | 14/20 = **70%** |
| Right-arm reachable — x∈[0.40, 0.72], y∈[−0.30, +0.06] | 30/30 = **100%** | 28/30 = **93.3%** |

**The gap between those two rows is not a model limitation.** The default sampling range
contains a large region that the right arm physically cannot reach, and this policy only
ever uses the right arm (see [Known limitations](#known-limitations)). Restricted to
positions the arm can actually reach, the policy solves the task essentially perfectly
without domain randomization, and loses only ~7 points with it.

### Efficiency

| Metric | `clear` | `random` |
| --- | --- | --- |
| Steps to success (median) | 60 | 60 |
| Inference calls per episode (median) | 6 | 6 |
| Inference latency (median) | 0.387 s | 0.397 s |

Successful episodes are tightly clustered at 60 steps / 6 inference calls — the policy
either solves it in one smooth approach or not at all. There is no "retry until it works"
behaviour.

### Training-step comparison

Same seeds, same configuration, three wrist+head camera streams recorded:

| | step 10000 | step 19999 |
| --- | --- | --- |
| `clear` | 13/20 = 65% | 13/20 = 65% |
| `random` | 10/20 = 50% | 14/20 = 70% |

Doubling training steps did **not** change `clear` at all, and improved `random`. A
per-episode breakdown shows exactly where the gain came from: failures of the type
"end-effector hovers 2–10 mm above the bell without pressing" dropped from 4 to 2, and
"pressed but not deep enough" dropped from 2 to 0. In other words the extra training
bought **press precision under visual perturbation** and nothing else.

Caveat: at n=20 these differences are not statistically separable — see
[Reproducibility](#reproducibility).

---

## Usage

This is an Orbax checkpoint for openpi. It is not a `transformers` model and will not
load with `AutoModel`.

```
click_bell/
├── params/                 # Orbax PyTree — model weights
├── assets/
│   └── RoboSynChallenge/
│       └── cobotmagic_Sim_click_bell/
│           └── norm_stats.json    # normalization stats (required)
└── _CHECKPOINT_METADATA
```

### Download

```bash
hf download puheliang/click_bell --local-dir ./click_bell_19999
```

### Load

```python
from openpi.training import config as _config
from openpi.policies import policy_config as _policy_config

train_config = _config.get_config("pi05_base_robosynchallenge_full")
policy = _policy_config.create_trained_policy(train_config, "./click_bell_19999")

action_chunk = policy.infer({
    "observation/image":              img_head,        # (H, W, 3) uint8
    "observation/left_wrist_image":   img_left_wrist,  # (H, W, 3) uint8
    "observation/right_wrist_image":  img_right_wrist, # (H, W, 3) uint8
    "observation/state":              qpos,            # joint positions
    "prompt":                         "Click the bell",
})["actions"]
```

Images must be **HWC uint8**, not CHW. Norm stats are read automatically from
`assets/<asset_id>/` inside the checkpoint directory, where `asset_id` comes from the
train config (`RoboSynChallenge/cobotmagic_Sim_click_bell`).

`train_state/` (optimizer state, 31 GB) is **not** included — this checkpoint is for
inference and evaluation, not for resuming training.

### Evaluate in RoboSynChallenge

```bash
# place under policy/pi05/checkpoints/<train_config>/<model_name>/19999/
bash policy/pi05/eval.sh click_bell random \
     pi05_base_robosynchallenge_full pi05_click_bell_baseline 0 \
     --checkpoint_id 19999 --max_episodes 30 --headless true \
     --eval_video_obs_keys cam_high,cam_left_wrist,cam_right_wrist
```

The checkpoint directory **must be named with the bare step number** (`19999`) — the
adapter does `int(checkpoint_id)`.

---

## Task definition

Success is decided purely by the physical displacement of the button
(`robosynchallenge/tasks/click_bell/click_bell.py`):

```python
press_depth = -button_qpos[:, 0]
success = press_depth >= 0.0048     # button joint travel is [-0.005, 0]
self._button_pressed |= success     # latched for the episode
```

The threshold is **4.8 mm out of 5.0 mm of total travel (96%)** — a light touch does not
count. Success is latched: one qualifying frame marks the episode successful.

### `clear` vs `random`

Both settings randomize the **bell position identically** over
x∈[0.40, 0.85], y∈[−0.30, +0.30]. The difference is 10 additional perturbations present
only in `random`:

| Perturbation | Applied |
| --- | --- |
| Light position / colour / intensity (10–30, a 3× range) | every 10 steps |
| Material of table, robot, floor, button (50% chance of random texture) | every 10 steps |
| Head camera intrinsics (±50 px focal) and extrinsics (±2 cm, ±10°) | each reset |
| Robot initial EEF pose (±1 cm) and joint angles (±0.05 rad) | each reset |
| Two distractor objects (cup / fork / spoon), random pose, ≥8 cm from the bell | each reset |

Note that lighting and materials are re-randomized **within** an episode, so the policy
must cope with appearance changing mid-trajectory.

---

## Known limitations

These were found by instrumenting all 110 evaluation episodes and are, in our view, more
useful than the headline number.

### 1. The policy only ever uses the right arm

Across **110/110 episodes it never once used the left arm** — at step 10000 and at step
19999 alike. Success is therefore strongly asymmetric:

| Bell position | Success rate |
| --- | --- |
| y < −0.10 (right side) | **90%** |
| y ∈ [−0.10, +0.18) | 65% |
| y > +0.18 (left side) | **20%** |

The root cause is upstream, in the expert data generator
(`robosynchallenge/tasks/click_bell/action_bank.py:57`), flagged by its own authors:

```python
# FIXME FIXME FIXME FIXME
logger.log_warning("CAUTION====THIS FUNC generate_left_arm_aim_qpos IS WRONG!!!! PLEASE FIX IT!!!!")
```

The right-arm equivalent is marked `# DONE`. Worse, the collection loop
(`scripts/run_env.py:_generate_function`) silently **resets and re-randomizes** whenever
action generation fails, leaving no record:

```python
valid = generate_and_execute_action_list(...)
if not valid:
    _, _ = env.reset(options={"save_data": False})   # try a different scene
    break
```

So left-side scenes are systematically dropped from the dataset rather than sampled and
failed. The policy never sees a left-arm demonstration and cannot invent one. **More
training will not fix this — the data has to be regenerated after the function is fixed.**

### 2. Part of the default evaluation range is unreachable

An IK sweep of the right arm's press pose over a grid of bell positions gives a clean
diagonal boundary (`O` reachable, `.` not):

```
       y=-0.30              -0.00              +0.30
x=0.40  O O O O O O O O O O O O O O O O O O O O .
x=0.50  O O O O O O O O O O O O O O O O O O O . .
x=0.60  O O O O O O O O O O O O O O O O . . . . .
x=0.70  O O O O O O O O O O O O O . . . . . . . .
x=0.80  O O O O O O O . . . . . . . . . . . . . .
x=0.84  . . . . . . . . . . . . . . . . . . . . .
```

Roughly **35% of the default sampling area is out of the right arm's workspace**. Note
this is a property of the *config shipped for data collection*, not an official benchmark
specification — RoboSynChallenge's official ranking is on held-out physical robots, and
the repository ships no evaluation config of its own.

### 3. Failure breakdown

Of 16 failures over 40 instrumented episodes at step 10000:

| Cause | Count | Whose problem |
| --- | --- | --- |
| Hovers 2–10 mm above the bell, never contacts | 9 | the policy |
| Right arm cannot reach (all had y > +0.18) | 5 | task configuration |
| Pressed 4.3–4.7 mm, threshold is 4.8 mm | 2 | success criterion |

Successes and failures separate almost perfectly by a single number — the closest
approach of the right end-effector to the bell:

| | Closest approach |
| --- | --- |
| 24 successes | 2.0 – **2.8** cm |
| 16 failures | **2.7** – 24.4 cm |

Press depth is bimodal, not continuous: successes all bottom out at 5.00 mm, and 14 of 16
failures never leave the 0.95 mm resting value. The task is all-or-nothing.

### 4. Reproducibility

Because the threshold sits at 96% of total travel, marginal episodes flip between runs
from GPU floating-point non-determinism alone. With the same seed we observed episodes
going `fail → success` and `success → fail` across repeats, and `random` scored 60% / 45%
/ 50% / 70% across four 20-episode runs of the same configuration.

`clear` reproduced exactly (13/20 three times). **Use ≥50 episodes when comparing
checkpoints**, or the noise will exceed the effect.

---

## Reproducing the evaluation

The upstream `policy/pi05/` adapter is out of date relative to its own `openpi` copy and
to `scripts/eval_policy.py`; evaluation cannot run without these four fixes (all present
in `policy/pi0/`, which is current):

| File | Problem | Fix |
| --- | --- | --- |
| `pi_model.py` | passes `robotwin_repo_id`, which this openpi does not accept | drop it — openpi resolves norm stats from `data_config.asset_id` |
| `pi_model.py` | `__init__` does not accept `pytorch_device`, but `deploy_policy.py` passes it | add the parameter and forward it |
| `pi_model.py` | builds an aloha-style observation (`{"state", "images"}`, CHW) | build flat `observation/*` keys in HWC, as `EmbodiChainInputs` expects |
| `deploy_policy.py` | `eval()` returns 2 values, `eval_policy.py` unpacks 3 | also return `truncated` |

One further trap: in `scripts/eval_policy.py` the episode loop's `finally` calls
`env.close()`, which terminates the process **before Python prints the traceback**, and
the exit code is 0. Every one of the bugs above therefore presented as "the run finished
normally but the robot never moved." Printing the exception inside the loop before
`env.close()` runs is what made them findable.

---

## Citation

```bibtex
@misc{click_bell_pi05,
  title  = {click_bell: a pi0.5 policy for the RoboSynChallenge bell-pressing task},
  author = {puheliang},
  year   = {2026},
  url    = {https://huggingface.co/puheliang/click_bell}
}
```

Base model: `pi05_base` (Physical Intelligence).
Benchmark and training data: [RoboSynChallenge](https://github.com/EDEM-AI/RoboSynChallenge) (EDEM-AI).