Blame for README.md

---
license: other
license_name: see-model-card
license_link: https://huggingface.co/LogosRoboticsLab/roboreel_models#licensing
pipeline_tag: robotics
tags:
- robotics
- imitation-learning
- learning-from-observation
- manipulation
- benchmark
---

# RoboReel — evaluated policy checkpoints

Policy checkpoints for **"Monkey See, Can Monkey Do? A Benchmark for Evaluating
Robot Skill Learning by Observation"** ([arXiv:2609.08209](https://arxiv.org/abs/2609.08209)),
benchmark **RoboReel** · [project page](https://roboreel.github.io).

Each folder holds **exactly the checkpoint the paper's numbers were computed
from** — not the last checkpoint a run happened to write. Training code,
evaluation harness and configs live in the `roboreel_models` repository; the
simulation environments live in the benchmark environment repository.

## Layout

```
<condition>/<method>/
├── train_config.json      # the run's own config (see "Paths" below)
├── <sidecar>              # norm_stats.json | action_codec.json | demonstration_config.txt
└── ckpt/ | ckpt.pt | ckpt.pth    # the evaluated checkpoint
```

Every method's weights are called **`ckpt`** whichever trainer wrote them — a
directory for the JAX/orbax and LeRobot policies, a single file for Vid2Robot
(`ckpt.pt`) and UniSkill (`ckpt.pth`).

`<condition>` is the training suite, and it is the **top** level: **`nd/`** = no
distraction (clean scenes), **`wd/`** = with distraction (distractors in both
train and eval). So the keyframe policy trained with distractors is
`wd/vid_cond_pi05_keyframes/ckpt/`.

The paper's third suite, **ED** (clean training, distracted evaluation), needs
no extra weights — it evaluates everything under `nd/` against the WD scene
bank.

## Checkpoints

| Paper label | Method dir | Weights | Size each |
|---|---|---|---|
| VLM + π0.5 | `vlm_pi05` | `ckpt/` | 9.35 GB |
| Vid. Cond. π0.5 (Uniform) | `vid_cond_pi05_uniform` | `ckpt/` | 6.20 GB |
| Vid. Cond. π0.5 (Keyframes) | `vid_cond_pi05_keyframes` | `ckpt/` | 6.20 GB |
| Vid2Robot | `vid2robot` | `ckpt.pt` | 2.37 GB |
| UniSkill | `uniskill_policy` | `ckpt.pth` | 0.74 GB |

UniSkill is a Robomimic diffusion policy over skill representations produced by
an **off-the-shelf, un-finetuned UniSkill inverse-skill-dynamics model**. That
ISD is deliberately *not* redistributed here — take it from the UniSkill
authors' own release and use it unmodified.

## Training setup

Common to all runs: 10 tasks (`bowl_in_plate`, `close_drawer`, `empty_basket`,
`open_lid`, `pick_cube`, `press_toaster`, `push_cube`, `stack_cups`,
`toys_in_basket`, `toys_in_drawer`), 80 demonstrations per task
(`max_demo_index: 79`), 8-D state and action.

| Method | Setup |
|---|---|
| VLM + π0.5 | π0.5 full fine-tune from `lerobot/pi05_base`, expert-only; AdamW, peak LR 2.5e-5 cosine→0.1×, batch 32, bf16, gradient checkpointing, 50k steps, action chunk 20, seed 42. Instructions generated by Qwen3-VL-4B from the human video. |
| Vid. Cond. π0.5 (Uniform) | openpi π0.5 from `pi05_base`, JAX FSDP; AdamW (β 0.9/0.95, ε 1e-8, wd 1e-10), grad clip 1.0, peak LR 5e-5, 1k warmup, EMA 0.999, batch 64, 50k steps, seed 42. Demonstration tokens: 512 tokens / 32 frames / 16 tokens per frame, uniform frame sampling. Demo camera `front`, robot cameras `base_camera` + `hand_camera`. |
| Vid. Cond. π0.5 (Keyframes) | Identical to Uniform — **only frame selection differs**: GPT-5.4-mini best-point-prompt keyframes over the human video downsampled to 2 fps. |
| Vid2Robot | ViT-B/16 encoders, 2-layer Perceiver Resamplers (64 latents), 4-layer decoders d=768, 16 prompt / 8 state frames, action chunk 4 over 256 bins; AdamW peak 3e-4 → 1e-6, wd 0.05, 2k warmup, batch 32, 50k steps, seed 42. |
| UniSkill | Robomimic diffusion policy only; Adam 1e-4, batch 256, 1000 epochs × 100 iters = 100k steps, EMA power 0.75, 100 DDPM train / 10 DDIM inference steps, `skill_aug` aug_num 5, seed 0. |

Every value above is also recorded verbatim in each folder's
`train_config.json`.

## Paths

`train_config.json` and the UniSkill `config.json` are the configs the runs
were actually launched with, with one edit: **host-specific absolute paths were
rewritten to placeholders**, since they pointed at cluster mounts that nobody
else has. Everything else is untouched.

| Placeholder | Set it to |
|---|---|
| `${ROBOREEL_DATA}` | your RoboReel dataset root |
| `${ROBOREEL_CKPT}` | where you downloaded this repo |
| `${OPENPI_DATA_HOME}` | your openpi asset cache (holds `openpi-assets/checkpoints/pi05_base`) |

The evaluation harness reads `data_root` / `data_dir` out of
`train_config.json` to locate the human demonstration videos, so these must be
pointed at a real dataset root before evaluating.

## Download

```bash
pip install -U huggingface_hub

# one method + condition
hf download LogosRoboticsLab/roboreel_models \
  --include "nd/vid_cond_pi05_keyframes/*" --local-dir ./roboreel_ckpt

# one condition, all five methods
hf download LogosRoboticsLab/roboreel_models \
  --include "nd/*" --local-dir ./roboreel_ckpt

# everything (≈50 GB)
hf download LogosRoboticsLab/roboreel_models --local-dir ./roboreel_ckpt
```

## Not in this repo

- **The RoboReel dataset** (human videos, robot trajectories, keyframe
  annotations, skill representations) — released separately.
- **SeeDo** and **PointPolicy** checkpoints — two of the paper's seven methods,
  to be added.
- **BC-Z** — discussed in related work only, never evaluated.
- Base/pretrained weights that are not ours to redistribute: `pi05_base`,
  PaliGemma, CLIP, Depth-Anything-V2, and the UniSkill ISD.

## Licensing

Mixed, because the checkpoints are derived from different base models:

- `vlm_pi05`, `vid_cond_pi05_uniform` and `vid_cond_pi05_keyframes` are
  fine-tunes of **π0.5** (openpi, Apache-2.0), whose backbone is **PaliGemma /
  Gemma** — these weights are therefore also subject to the
  [Gemma Terms of Use](https://ai.google.dev/gemma/terms).
- `uniskill_policy` is a Robomimic diffusion policy trained on
  representations from **UniSkill**; see the UniSkill release for its terms.
- `vid2robot` is trained from scratch on the RoboReel dataset apart from its
  ImageNet-pretrained ViT-B/16 and SigLIP text encoder.

## Citation

```bibtex
@article{roboreel2026,
  title   = {Monkey See, Can Monkey Do? A Benchmark for Evaluating Robot Skill Learning by Observation},
  journal = {arXiv preprint arXiv:2609.08209},
  year    = {2026}
}
```