Upload README.md with huggingface_hub

b71d051 verified 5 days ago

11.5 kB

license: apache-2.0
library_name: openpi
pipeline_tag: robotics
tags:
  - robotics
  - manipulation
  - pi0
  - pi05
  - vla
  - embodichain
  - robosynchallenge
  - bimanual

click_bell — π0.5 policy for RoboSynChallenge

A π0.5 (pi05) policy finetuned on the RoboSynChallenge click_bell task: a bimanual CobotMagic robot must press a desk bell placed at a random position on the table.

Trained from pi05_base on 1,000 synthetic demonstrations (RoboSynChallenge/cobotmagic_Sim_click_bell) for 20,000 steps.

Checkpoint: step 19999 (final)
Framework: openpi (JAX / Orbax)
Simulator: EmbodiChain 0.2.3 + dexsim 0.4.3
Benchmark: RoboSynChallenge

Results

Evaluated in simulation on an RTX 4090. Every number below is from a run of this exact checkpoint; nothing is copied from a paper or leaderboard.

Headline

Button position range	`clear` (no randomization)	`random` (full domain randomization)
Repo default — x∈[0.40, 0.85], y∈[−0.30, +0.30]	13/20 = 65%	14/20 = 70%
Right-arm reachable — x∈[0.40, 0.72], y∈[−0.30, +0.06]	30/30 = 100%	28/30 = 93.3%

The gap between those two rows is not a model limitation. The default sampling range contains a large region that the right arm physically cannot reach, and this policy only ever uses the right arm (see Known limitations). Restricted to positions the arm can actually reach, the policy solves the task essentially perfectly without domain randomization, and loses only ~7 points with it.

Efficiency

Metric	`clear`	`random`
Steps to success (median)	60	60
Inference calls per episode (median)	6	6
Inference latency (median)	0.387 s	0.397 s

Successful episodes are tightly clustered at 60 steps / 6 inference calls — the policy either solves it in one smooth approach or not at all. There is no "retry until it works" behaviour.

Training-step comparison

Same seeds, same configuration, three wrist+head camera streams recorded:

	step 10000	step 19999
`clear`	13/20 = 65%	13/20 = 65%
`random`	10/20 = 50%	14/20 = 70%

Doubling training steps did not change clear at all, and improved random. A per-episode breakdown shows exactly where the gain came from: failures of the type "end-effector hovers 2–10 mm above the bell without pressing" dropped from 4 to 2, and "pressed but not deep enough" dropped from 2 to 0. In other words the extra training bought press precision under visual perturbation and nothing else.

Caveat: at n=20 these differences are not statistically separable — see Reproducibility.

Usage

This is an Orbax checkpoint for openpi. It is not a transformers model and will not load with AutoModel.

click_bell/
├── params/                 # Orbax PyTree — model weights
├── assets/
│   └── RoboSynChallenge/
│       └── cobotmagic_Sim_click_bell/
│           └── norm_stats.json    # normalization stats (required)
└── _CHECKPOINT_METADATA

Download

hf download puheliang/click_bell --local-dir ./click_bell_19999

Load

from openpi.training import config as _config
from openpi.policies import policy_config as _policy_config

train_config = _config.get_config("pi05_base_robosynchallenge_full")
policy = _policy_config.create_trained_policy(train_config, "./click_bell_19999")

action_chunk = policy.infer({
    "observation/image":              img_head,        # (H, W, 3) uint8
    "observation/left_wrist_image":   img_left_wrist,  # (H, W, 3) uint8
    "observation/right_wrist_image":  img_right_wrist, # (H, W, 3) uint8
    "observation/state":              qpos,            # joint positions
    "prompt":                         "Click the bell",
})["actions"]

Images must be HWC uint8, not CHW. Norm stats are read automatically from assets/<asset_id>/ inside the checkpoint directory, where asset_id comes from the train config (RoboSynChallenge/cobotmagic_Sim_click_bell).

train_state/ (optimizer state, 31 GB) is not included — this checkpoint is for inference and evaluation, not for resuming training.

Evaluate in RoboSynChallenge

# place under policy/pi05/checkpoints/<train_config>/<model_name>/19999/
bash policy/pi05/eval.sh click_bell random \
     pi05_base_robosynchallenge_full pi05_click_bell_baseline 0 \
     --checkpoint_id 19999 --max_episodes 30 --headless true \
     --eval_video_obs_keys cam_high,cam_left_wrist,cam_right_wrist

The checkpoint directory must be named with the bare step number (19999) — the adapter does int(checkpoint_id).

Task definition

Success is decided purely by the physical displacement of the button (robosynchallenge/tasks/click_bell/click_bell.py):

press_depth = -button_qpos[:, 0]
success = press_depth >= 0.0048     # button joint travel is [-0.005, 0]
self._button_pressed |= success     # latched for the episode

The threshold is 4.8 mm out of 5.0 mm of total travel (96%) — a light touch does not count. Success is latched: one qualifying frame marks the episode successful.

`clear` vs `random`

Both settings randomize the bell position identically over x∈[0.40, 0.85], y∈[−0.30, +0.30]. The difference is 10 additional perturbations present only in random:

Perturbation	Applied
Light position / colour / intensity (10–30, a 3× range)	every 10 steps
Material of table, robot, floor, button (50% chance of random texture)	every 10 steps
Head camera intrinsics (±50 px focal) and extrinsics (±2 cm, ±10°)	each reset
Robot initial EEF pose (±1 cm) and joint angles (±0.05 rad)	each reset
Two distractor objects (cup / fork / spoon), random pose, ≥8 cm from the bell	each reset

Note that lighting and materials are re-randomized within an episode, so the policy must cope with appearance changing mid-trajectory.

Known limitations

These were found by instrumenting all 110 evaluation episodes and are, in our view, more useful than the headline number.

1. The policy only ever uses the right arm

Across 110/110 episodes it never once used the left arm — at step 10000 and at step 19999 alike. Success is therefore strongly asymmetric:

Bell position	Success rate
y < −0.10 (right side)	90%
y ∈ [−0.10, +0.18)	65%
y > +0.18 (left side)	20%

The root cause is upstream, in the expert data generator (robosynchallenge/tasks/click_bell/action_bank.py:57), flagged by its own authors:

# FIXME FIXME FIXME FIXME
logger.log_warning("CAUTION====THIS FUNC generate_left_arm_aim_qpos IS WRONG!!!! PLEASE FIX IT!!!!")

The right-arm equivalent is marked # DONE. Worse, the collection loop (scripts/run_env.py:_generate_function) silently resets and re-randomizes whenever action generation fails, leaving no record:

valid = generate_and_execute_action_list(...)
if not valid:
    _, _ = env.reset(options={"save_data": False})   # try a different scene
    break

So left-side scenes are systematically dropped from the dataset rather than sampled and failed. The policy never sees a left-arm demonstration and cannot invent one. More training will not fix this — the data has to be regenerated after the function is fixed.

2. Part of the default evaluation range is unreachable

An IK sweep of the right arm's press pose over a grid of bell positions gives a clean diagonal boundary (O reachable, . not):

       y=-0.30              -0.00              +0.30
x=0.40  O O O O O O O O O O O O O O O O O O O O .
x=0.50  O O O O O O O O O O O O O O O O O O O . .
x=0.60  O O O O O O O O O O O O O O O O . . . . .
x=0.70  O O O O O O O O O O O O O . . . . . . . .
x=0.80  O O O O O O O . . . . . . . . . . . . . .
x=0.84  . . . . . . . . . . . . . . . . . . . . .

Roughly 35% of the default sampling area is out of the right arm's workspace. Note this is a property of the config shipped for data collection, not an official benchmark specification — RoboSynChallenge's official ranking is on held-out physical robots, and the repository ships no evaluation config of its own.

3. Failure breakdown

Of 16 failures over 40 instrumented episodes at step 10000:

Cause	Count	Whose problem
Hovers 2–10 mm above the bell, never contacts	9	the policy
Right arm cannot reach (all had y > +0.18)	5	task configuration
Pressed 4.3–4.7 mm, threshold is 4.8 mm	2	success criterion

Successes and failures separate almost perfectly by a single number — the closest approach of the right end-effector to the bell:

	Closest approach
24 successes	2.0 – 2.8 cm
16 failures	2.7 – 24.4 cm

Press depth is bimodal, not continuous: successes all bottom out at 5.00 mm, and 14 of 16 failures never leave the 0.95 mm resting value. The task is all-or-nothing.

4. Reproducibility

Because the threshold sits at 96% of total travel, marginal episodes flip between runs from GPU floating-point non-determinism alone. With the same seed we observed episodes going fail → success and success → fail across repeats, and random scored 60% / 45% / 50% / 70% across four 20-episode runs of the same configuration.

clear reproduced exactly (13/20 three times). Use ≥50 episodes when comparing checkpoints, or the noise will exceed the effect.

Reproducing the evaluation

The upstream policy/pi05/ adapter is out of date relative to its own openpi copy and to scripts/eval_policy.py; evaluation cannot run without these four fixes (all present in policy/pi0/, which is current):

File	Problem	Fix
`pi_model.py`	passes `robotwin_repo_id`, which this openpi does not accept	drop it — openpi resolves norm stats from `data_config.asset_id`
`pi_model.py`	`__init__` does not accept `pytorch_device`, but `deploy_policy.py` passes it	add the parameter and forward it
`pi_model.py`	builds an aloha-style observation (`{"state", "images"}`, CHW)	build flat `observation/*` keys in HWC, as `EmbodiChainInputs` expects
`deploy_policy.py`	`eval()` returns 2 values, `eval_policy.py` unpacks 3	also return `truncated`

One further trap: in scripts/eval_policy.py the episode loop's finally calls env.close(), which terminates the process before Python prints the traceback, and the exit code is 0. Every one of the bugs above therefore presented as "the run finished normally but the robot never moved." Printing the exception inside the loop before env.close() runs is what made them findable.

Citation

@misc{click_bell_pi05,
  title  = {click_bell: a pi0.5 policy for the RoboSynChallenge bell-pressing task},
  author = {puheliang},
  year   = {2026},
  url    = {https://huggingface.co/puheliang/click_bell}
}

Base model: pi05_base (Physical Intelligence). Benchmark and training data: RoboSynChallenge (EDEM-AI).