Add model card

a2565ec verified 5 days ago

5.59 kB

license: gemma
library_name: openpi
pipeline_tag: robotics
tags:
  - robotics
  - vla
  - pi0
  - pi05
  - openpi
  - franka
  - manipulation
  - imitation-learning

π₀.₅ — place the cube in the bowl (real Franka)

A π₀.₅ vision-language-action policy fine-tuned on 100 real Franka Emika teleoperation demonstrations of "place the cube in the bowl".

Serving code, robot client and the full I/O contract: github.com/sleepmastergx/deploy, branch pi0.5

Checkpoints

Four training steps are published so you can evaluate across them. Only params/ and assets/ are here — train_state/ (optimizer state, 32 GB per step) is not needed for inference and is not published.

folder	step	≈ epochs	size
`5000/`	5,000	10	12.4 GB
`10000/`	10,000	20	12.4 GB
`15000/`	15,000	31	12.4 GB
`19999/`	20,000 (final)	41	12.4 GB

openpi numbers steps 0-indexed, so 19999 is the completed 20,000-step model.

Start with 19999, but do not assume it is best. 100 demonstrations over ~41 epochs is a deep-overfit regime; training loss reached 0.0016, which says the model fit the data, not that it will place cubes. Only real-robot rollouts settle which checkpoint to use — if 19999 looks memorised or brittle, work backwards.

assets/george/place_cube_new_first100/norm_stats.json inside each folder holds that checkpoint's normalization statistics. They travel with the checkpoint and must not be mixed between runs — the policy cannot be served without them, and the wrong ones produce mis-scaled actions rather than an error.

Training


Base	`pi05_base` (openpi's robot-pretrained π₀.₅)
Data	100 episodes / 15,624 frames @ 10 Hz, single Franka + 2 RealSense cameras
Steps	20,000, batch 32, AdamW, cosine 5e-5 → 5e-6 after 1k warmup, EMA 0.999
Model	`action_dim=32`, `action_horizon=16`, `discrete_state_input=True`
Hardware	2 × H200, ~4 h 25 m
Final loss	0.0016 (from 0.163)

Usage

hf download SleepMastger/pi05_place_cube --local-dir ./pi05_place_cube

git clone -b pi0.5 https://github.com/sleepmastergx/deploy.git
cd deploy/pi05_place_cube
PYTHONPATH=$PWD/src python serve_pi05.py --ckpt ../../pi05_place_cube/19999

Requires openpi installed (JAX + CUDA, Python ≥ 3.11) and a GPU with > 8 GB VRAM.

Interface

The policy server expects exactly these keys — no batch dimension, no client-side resizing, no client-side normalization:

{
  "observation/image":       uint8 (256, 256, 3),   # third-person / agentview
  "observation/wrist_image": uint8 (256, 256, 3),   # wrist
  "observation/state":       float (8,),            # RAW, unnormalized
  "prompt":                  "place the cube in the bowl",
}

with

state[0:3] = eef_pos                    # metres
state[3:6] = quat2axisangle(eef_quat)   # eef_quat is (x, y, z, w), scalar-last
state[6]   = +gripper_width / 2         # metres (~0.0796 open, ~0.029 closed)
state[7]   = -gripper_width / 2

Returns (16, 7) actions, already un-normalized:

dims 0:3 — dx, dy, dz delta end-effector position in metres per 100 ms step
dims 3:6 — rotation, always ≈ 0; this policy cannot rotate (training saw only zero rotations)
dim 6 — gripper, 1.0 = OPEN, 0.0 = CLOSE, threshold at 0.5

Control rate is 10 Hz. Predict 16, execute ~8, re-plan.

The gripper polarity is the easiest thing to get backwards, so to be explicit: 1 is open. It was verified against the training data — action[6] steps 1 → 0 at exactly the frame state[6] falls 0.0398 → 0.0146 (width 0.0796 → 0.029, closing on the cube).

State is not a side channel: with discrete_state_input=True it is normalized, binned into 256 buckets and written into the text prompt, so a wrong state changes what the model reads. Full spec in CONTRACT.md.

Verification

Replaying held-out recorded episodes through the served policy and scoring against logged actions:

	episode_0	episode_1
xyz RMSE	0.00019 m	0.00020 m
ground-truth per-step RMS	0.00106 m (error/signal 0.18)	0.00121 m (0.16)
max \|rpy\|	3.3e-09	3.6e-09
gripper match	100 % (96/96)	100 % (96/96)
inference	~51 ms	~51 ms

Reproduce with tools/replay_test.py in the deploy repo. This measures whether a deployment reproduces the training contract — not whether the policy succeeds at the task, which only real rollouts can tell you.

Limitations

No rotation control. Rotation was constant zero in every training episode.
One task, one scene, one camera setup. 100 demonstrations from a single rig. Expect nothing outside that distribution.
Deep-overfit regime. See the checkpoint note above.
Camera framing matters. The policy is sensitive to it, and a mismatch degrades behaviour silently — it keeps acting confidently and does the wrong thing. Reproduce the collection framing, and run the replay test before trusting a new setup.

License

Released under the Gemma Terms of Use. This model is fine-tuned from openpi's pi05_base, whose vision-language backbone is PaliGemma/Gemma, so Gemma's terms carry through to this derivative. The openpi code is Apache-2.0 separately.