README.md

---
license: gemma
library_name: openpi
pipeline_tag: robotics
tags:
  - robotics
  - vla
  - pi0
  - pi05
  - openpi
  - franka
  - manipulation
  - imitation-learning
---

# π₀.₅ — place the cube in the bowl (real Franka)

A [π₀.₅](https://github.com/Physical-Intelligence/openpi) vision-language-action policy
fine-tuned on **100 real Franka Emika teleoperation demonstrations** of *"place the cube
in the bowl"*.

Serving code, robot client and the full I/O contract:
**[github.com/sleepmastergx/deploy, branch `pi0.5`](https://github.com/sleepmastergx/deploy/tree/pi0.5/pi05_place_cube)**

## Checkpoints

Four training steps are published so you can evaluate across them. Only `params/` and
`assets/` are here — `train_state/` (optimizer state, 32 GB per step) is not needed for
inference and is not published.

| folder | step | ≈ epochs | size |
|---|---|---|---|
| `5000/` | 5,000 | 10 | 12.4 GB |
| `10000/` | 10,000 | 20 | 12.4 GB |
| `15000/` | 15,000 | 31 | 12.4 GB |
| `19999/` | 20,000 (final) | 41 | 12.4 GB |

openpi numbers steps 0-indexed, so `19999` is the completed 20,000-step model.

**Start with `19999`, but do not assume it is best.** 100 demonstrations over ~41 epochs
is a deep-overfit regime; training loss reached 0.0016, which says the model fit the data,
not that it will place cubes. Only real-robot rollouts settle which checkpoint to use — if
`19999` looks memorised or brittle, work backwards.

`assets/george/place_cube_new_first100/norm_stats.json` inside each folder holds that
checkpoint's normalization statistics. **They travel with the checkpoint and must not be
mixed between runs** — the policy cannot be served without them, and the wrong ones
produce mis-scaled actions rather than an error.

## Training

| | |
|---|---|
| Base | `pi05_base` (openpi's robot-pretrained π₀.₅) |
| Data | 100 episodes / 15,624 frames @ 10 Hz, single Franka + 2 RealSense cameras |
| Steps | 20,000, batch 32, AdamW, cosine 5e-5 → 5e-6 after 1k warmup, EMA 0.999 |
| Model | `action_dim=32`, `action_horizon=16`, `discrete_state_input=True` |
| Hardware | 2 × H200, ~4 h 25 m |
| Final loss | 0.0016 (from 0.163) |

## Usage

```bash
hf download SleepMastger/pi05_place_cube --local-dir ./pi05_place_cube

git clone -b pi0.5 https://github.com/sleepmastergx/deploy.git
cd deploy/pi05_place_cube
PYTHONPATH=$PWD/src python serve_pi05.py --ckpt ../../pi05_place_cube/19999
```

Requires openpi installed (JAX + CUDA, Python ≥ 3.11) and a GPU with > 8 GB VRAM.

## Interface

The policy server expects exactly these keys — no batch dimension, no client-side
resizing, no client-side normalization:

```python
{
  "observation/image":       uint8 (256, 256, 3),   # third-person / agentview
  "observation/wrist_image": uint8 (256, 256, 3),   # wrist
  "observation/state":       float (8,),            # RAW, unnormalized
  "prompt":                  "place the cube in the bowl",
}
```

with

```python
state[0:3] = eef_pos                    # metres
state[3:6] = quat2axisangle(eef_quat)   # eef_quat is (x, y, z, w), scalar-last
state[6]   = +gripper_width / 2         # metres (~0.0796 open, ~0.029 closed)
state[7]   = -gripper_width / 2
```

Returns `(16, 7)` actions, already un-normalized:

- dims `0:3` — `dx, dy, dz` delta end-effector position in **metres per 100 ms step**
- dims `3:6` — rotation, **always ≈ 0**; this policy cannot rotate (training saw only
  zero rotations)
- dim `6` — gripper, **1.0 = OPEN, 0.0 = CLOSE**, threshold at 0.5

Control rate is **10 Hz**. Predict 16, execute ~8, re-plan.

> The gripper polarity is the easiest thing to get backwards, so to be explicit: **1 is
> open.** It was verified against the training data — `action[6]` steps 1 → 0 at exactly
> the frame `state[6]` falls 0.0398 → 0.0146 (width 0.0796 → 0.029, closing on the cube).

State is not a side channel: with `discrete_state_input=True` it is normalized, binned
into 256 buckets and written into the *text prompt*, so a wrong state changes what the
model reads. Full spec in
[CONTRACT.md](https://github.com/sleepmastergx/deploy/blob/pi0.5/pi05_place_cube/CONTRACT.md).

## Verification

Replaying held-out recorded episodes through the served policy and scoring against logged
actions:

| | episode_0 | episode_1 |
|---|---|---|
| xyz RMSE | 0.00019 m | 0.00020 m |
| ground-truth per-step RMS | 0.00106 m (error/signal 0.18) | 0.00121 m (0.16) |
| max \|rpy\| | 3.3e-09 | 3.6e-09 |
| gripper match | 100 % (96/96) | 100 % (96/96) |
| inference | ~51 ms | ~51 ms |

Reproduce with `tools/replay_test.py` in the deploy repo. This measures whether a
deployment reproduces the training contract — **not** whether the policy succeeds at the
task, which only real rollouts can tell you.

## Limitations

- **No rotation control.** Rotation was constant zero in every training episode.
- **One task, one scene, one camera setup.** 100 demonstrations from a single rig. Expect
  nothing outside that distribution.
- **Deep-overfit regime.** See the checkpoint note above.
- **Camera framing matters.** The policy is sensitive to it, and a mismatch degrades
  behaviour silently — it keeps acting confidently and does the wrong thing. Reproduce the
  collection framing, and run the replay test before trusting a new setup.

## License

Released under the [Gemma Terms of Use](https://ai.google.dev/gemma/terms). This model is
fine-tuned from openpi's `pi05_base`, whose vision-language backbone is PaliGemma/Gemma,
so Gemma's terms carry through to this derivative. The openpi *code* is Apache-2.0
separately.