README.md

---
license: mit
tags:
  - robotics
  - imitation-learning
  - manipulation
  - pi0
  - vla
---

# cup_pnp — yellow-cup pick-and-place policies

Four behavior-cloning policies for the task **"pick up a yellow cup and place it
on a baking tray,"** trained from egocentric human video reconstructed into
robot demonstrations. This is a 2×2 ablation:

| architecture × observation | **current (raw tray)** | **sim-tray** |
|---|---|---|
| **π0.5** (LoRA on `pi05_base`) | `pi05_current_tray/` | `pi05_sim_tray/` |
| **ABC-DiT** (from scratch) | `abc_dit_current_tray.pt` | `abc_dit_sim_tray.pt` |

- **Action / state space:** 28-D — right arm only (3-slide + 3-hinge wrist + 22
  finger joints), absolute joint targets.
- **Observation:** single rendered robot-view RGB (224×224).
- **sim-tray** variant re-renders the tray from its simulation mesh (inpainted
  over the original footage); the **current** variant keeps the tray as raw
  footage. Actions are identical across the two variants — only the observation
  pixels differ.

## Results — held-out action-prediction MSE (normalized, 10-step horizon; lower is better)

| policy | current (raw tray) | sim-tray |
|---|---|---|
| **π0.5** | **0.021** | **0.021** |
| **ABC-DiT** | **0.345** | **0.318** |

π0.5 (pretrained VLA + LoRA) is ~16× more accurate than ABC-DiT (trained from
scratch with a randomly-initialized visual backbone). The sim-tray processing is
neutral for π0.5 and a small improvement for ABC-DiT.

## Contents

- `pi05_current_tray/`, `pi05_sim_tray/` — π0.5 checkpoints (orbax `params/` +
  `assets/` norm-stats). Load with [openpi](https://github.com/Physical-Intelligence/openpi)
  `policy_config.create_trained_policy`. Config: `pi05_cup_v1` / `pi05_cup_v2`.
- `abc_dit_current_tray.pt`, `abc_dit_sim_tray.pt` — ABC-DiT checkpoints
  (`{model, step, norm_stats}` + optimizer). Load with
  [ABC](https://github.com/amazon-far/abc). state/action dim = 28, camera-keys = `top`.

Trained on 80 verified reconstructed demonstrations. See the project report for
data provenance and method details.