Frank-Insert-Wipe / README.md
cognaclee's picture
bread_gripstate_expert: bread retrain (gripper width only, grasp oversampling) + code, README, SHA256SUMS
4f263f6 verified
|
Raw History Blame Contribute Delete
28.3 kB
---
tags:
- robotics
- openpi
- pi0.5
- franka
library_name: openpi
---
# Frank-Insert-Wipe: AdaSpeVLA checkpoints for Franka insert_tube, wipe_whiteboard, bread, test_tube, tennis_balls and a 5-task co-trained model (H=16)
AdaSpeVLA is a pi0.5 (OpenPI, JAX) variant. It predicts cubic B-spline action chunks with adaptive knots, plus a
timing head that decides when to query the policy again. This repo holds two variants that differ only in the
generator that produces the 11 control points:
- **flow_head** (`<task>/`, 2026-11-12): a 20.4 M FlowPolicy trained from scratch.
- **pi05_expert** (`<task>_expert/`, 2026-11-14): pi0.5's pretrained Gemma-300M action expert (427.9 M with adaRMS,
from `pi05_base`) runs on the 11 control-point tokens, as pi0.5 does on its action tokens.
Everything else (data, norm stats, H=16, K=11, labels, knot head, timing head, 40k + 10k steps) is the same.
The training code for both is in `code/`.
These are params-only Orbax checkpoints in the OpenPI layout. The optimizer `train_state` is not included.
| Folder | Training stage | Config name (with `TASK=<task>`) | Use |
|---|---|---|---|
| `<task>/timing-from40k-step-9999/` | Timing head, 10k steps, from Action step 39999 (everything except `freq_head` frozen) | `<task>_adaspe_timing_from40k` | **Deployment**: full model with the trained timing head |
| `<task>/action-step-39999/` | Action stage, 40k steps (EMA params) | `<task>_adaspe_action` | Action generator / knot head only. Its timing head is untrained. |
| `<task>_expert/timing-from40k-step-9999/` | Same, pi05_expert generator | `<task>_adaspe_expert_timing_from40k` | **Deployment** (expert variant) |
| `<task>_expert/action-step-39999/` | Same, pi05_expert generator | `<task>_adaspe_expert_action` | Action generator / knot head only (expert variant) |
| `<task>/timing-from40k-lr3e-4-step-9999/` | Timing head only, retrained 2026-11-14 with peak LR 3e-4 (same 10k steps; see below) | `<task>_adaspe_timing_from40k_lr3e-4` | Deployment candidate |
| `<task>/timing-from40k-lr1e-3-step-9999/` | Timing head only, retrained 2026-11-14 with peak LR 1e-3 (same 10k steps; see below) | `<task>_adaspe_timing_from40k_lr1e-3` | Deployment candidate |
| `<task>_expert/timing-from40k-lr3e-4-step-9999/` | Timing head only, retrained 2026-11-14 with peak LR 3e-4 (same 10k steps; see below) | `<task>_adaspe_expert_timing_from40k_lr3e-4` | Deployment candidate (expert variant) |
| `<task>_expert/timing-from40k-lr1e-3-step-9999/` | Timing head only, retrained 2026-11-14 with peak LR 1e-3 (same 10k steps; see below) | `<task>_adaspe_expert_timing_from40k_lr1e-3` | Deployment candidate (expert variant) |
`<task>` is `insert_tube` or `wipe_whiteboard`. Each folder has `params/`, `assets/dddddsds31341/<dataset>/norm_stats.json`
(the normalization stats used in training) and `_CHECKPOINT_METADATA`. `SHA256SUMS` lists every file.
## Loading
You need the AdaSpeVLA OpenPI plugin, included here as `code/adaspe_train` (run with `TASK=<task>`; it registers
these config names). It runs on OpenPI at the commit in `code/OPENPI_COMMIT`. Setup is in `code/README.md`, and
`code/adaspe_train/EXPERT_VARIANT.md` documents the expert variant.
The folders use the standard OpenPI checkpoint layout (`params/` + `assets/`).
```python
from huggingface_hub import snapshot_download
root = snapshot_download("cognaclee/Frank-Insert-Wipe", allow_patterns=["insert_tube/timing-from40k-step-9999/*"])
# higher-LR timing head: "insert_tube/timing-from40k-lr3e-4-step-9999/*" with config insert_tube_adaspe_timing_from40k_lr3e-4
ckpt_dir = f"{root}/insert_tube/timing-from40k-step-9999" # with config insert_tube_adaspe_timing_from40k
```
## Data
| Task | Dataset (revision) | Size | pi0.5 baseline (same norm stats) |
|---|---|---|---|
| insert_tube | `dddddsds31341/insert-tube-lerobot` (`70bcd1f86ed35fc93a0422a564472355242a51ab`) | 102 episodes / 16,403 frames | `dddddsds31341/Franka-Insert-Tube-pi05-step-20000` |
| wipe_whiteboard | `dddddsds31341/wipe-whiteboard-lerobot` (`2c8cfa8ddef1f83cb09ca7881a0944cd637cd6d0`) | 112 episodes / 29,795 frames | `dddddsds31341/Franka-Wipe-Whiteboard-pi05-step-20000` |
Two 128Γ—128 cameras, 19-D state, 7-D action, 10 Hz.
## Recipe
- Action horizon H=16 (1.5 s), 11 cubic B-spline control points, query interval 0.2–1.5 s.
- Gripper events for the timing and knot labels come from the gripper command (the measured width lags by about 1.9 s).
- Knot density is tanh(motion_variation / e_crit), with e_crit calibrated per task at H=16: insert_tube 11.98, wipe_whiteboard 8.96.
- Initialized from `pi05_base`. Batch 32, Action 40k steps (LR decay over 30k), then timing 10k steps.
## Training loss
Mean training loss over the last 1k–2k steps of each stage.
| Task | Variant | Action loss, steps 38k–40k | Timing loss (log-interval MSE), steps 9k–10k |
|---|---|---|---|
| insert_tube | flow_head | 0.086 | 0.206 |
| insert_tube | pi05_expert | **0.030** | 0.255 |
| wipe_whiteboard | flow_head | 0.118 | 0.196 |
| wipe_whiteboard | pi05_expert | **0.040** | 0.271 |
Action loss over training (mean per 2k-step window):
| Steps | insert_tube flow_head | insert_tube expert | wipe flow_head | wipe expert |
|---|---|---|---|---|
| 0–2k | 0.624 | 0.334 | 0.597 | 0.295 |
| 8–10k | 0.238 | 0.121 | 0.238 | 0.121 |
| 18–20k | 0.156 | 0.067 | 0.169 | 0.083 |
| 28–30k | 0.105 | 0.038 | 0.130 | 0.049 |
| 38–40k | 0.086 | 0.030 | 0.118 | 0.040 |
Notes:
- The two variants use different flow-matching conventions (flow_head: t ~ U(0,1); pi05_expert: pi0.5's
Beta(1.5,1)), so the action losses are not exactly the same metric.
- The expert's timing head has a **higher** loss. It reads a pooled observation feature that, in the expert
variant, is trained only by the knot loss (see `code/adaspe_train/EXPERT_VARIANT.md`).
- These are training losses. No held-out or real-robot evaluation results are included here.
## Timing head retrained with a higher learning rate (2026-11-14)
The original timing heads (`timing-from40k-step-9999`) used OpenPI's default schedule: peak LR 2.5e-5 with a cosine
decay over 30k steps. The timing stage only runs 10k steps, so the LR stayed at about 2.0–2.5e-5, which is low for
`freq_head`, a freshly initialized 2-layer MLP (633 β†’ 128 β†’ 1, ~81k params) and the only trainable part. The loss was
still falling at 10k.
The `timing-from40k-lr<LR>-step-9999` folders retrain **only `freq_head`**, with the same data, labels, 10k steps,
batch 32 and 1k warmup, a cosine decay to LR/10 over the 10k steps, and peak LR 3e-4 or 1e-3. They start from the
same Action step-39999 params, and everything except `freq_head` is frozen. The predicted actions, knots and
B-spline are therefore identical to `timing-from40k-step-9999`. Only the query interval (and so `execute_steps`)
changes.
Timing loss (log-interval MSE, training loss, mean over steps 9k–10k):
| Task | Variant | LR 2.5e-5 (original) | LR 3e-4 | LR 1e-3 |
|---|---|---|---|---|
| insert_tube | flow_head | 0.206 | 0.157 | 0.091 |
| insert_tube | pi05_expert | 0.255 | 0.182 | 0.100 |
| wipe_whiteboard | flow_head | 0.196 | 0.158 | 0.112 |
| wipe_whiteboard | pi05_expert | 0.271 | 0.214 | 0.153 |
These are **training** losses. With the higher LR, part of the drop may come from fitting the training frames; a
held-out comparison has not been done yet. The configs are `configs.TIMINGS_LR` in `code/adaspe_train/configs.py`,
and `code/adaspe_train/slurm/timing_lr.slurm` runs one of them.
## Bread (2026-11-15, pi05_expert only)
`bread_expert/` holds the bread task trained with the same H=16 recipe and the **pi05_expert** generator. There
is no flow_head version of bread. Unlike the other tasks, every saved checkpoint is included (params only):
| Folder | Training stage | Config name (with `TASK=bread`) | Use |
|---|---|---|---|
| `bread_expert/timing-from40k-lr3e-4-step-9999/` | Timing head only, peak LR 3e-4, 10k steps, from Action step 39999 | `bread_adaspe_expert_timing_from40k_lr3e-4` | **Deployment (recommended)** |
| `bread_expert/timing-from40k-step-9999/` | Timing head, default LR (peak 2.5e-5), from Action step 39999 | `bread_adaspe_expert_timing_from40k` | Deployment, original timing LR |
| `bread_expert/timing-from30k-step-9999/` | Timing head, default LR, from Action step 30000 | `bread_adaspe_expert_timing_from30k` | Ablation |
| `bread_expert/timing-from20k-step-9999/` | Timing head, default LR, from Action step 20000 | `bread_adaspe_expert_timing_from20k` | Ablation |
| `bread_expert/action-step-{10000,20000,30000,39999}/` | Action stage (EMA params) at that step | `bread_adaspe_expert_action` | Action generator / knot head only. The timing head is untrained. |
Every timing folder is a full model. Only `freq_head` was trained in the timing stage, so actions and knots equal
those of the Action checkpoint it starts from.
Data: `dddddsds31341/bread-lerobot` (revision `a45b8ba49fc164fb05d8a9c8bdf871ff97004357`), 100 episodes /
28,380 frames, 10 Hz. e_crit = 10.01 (calibrated at H=16). Training: batch 32, Action 40k steps (OpenPI default LR:
1k warmup, peak 2.5e-5, cosine decay to 2.5e-6 at 30k), then the timing stage (10k steps).
Action loss (training, mean per 2k-step window):
| Steps | bread expert |
|---|---|
| 0–2k | 0.357 |
| 8–10k | 0.161 |
| 18–20k | 0.114 |
| 28–30k | 0.093 |
| 38–40k | 0.083 |
Timing loss (log-interval MSE, training, steps 9k–10k): from20k 0.217, from30k 0.217,
from40k 0.212, from40k lr3e-4 0.159.
These are training losses. No held-out or real-robot evaluation results are included. `code/adaspe_train` has the
bread settings (`tasks.py`, with checkpoints saved every 10k steps).
```python
root = snapshot_download("cognaclee/Frank-Insert-Wipe", allow_patterns=["bread_expert/timing-from40k-lr3e-4-step-9999/*"])
ckpt_dir = f"{root}/bread_expert/timing-from40k-lr3e-4-step-9999" # TASK=bread, config bread_adaspe_expert_timing_from40k_lr3e-4
```
## Five-task co-training (2026-11-15, pi05_expert only)
`cotrain5_expert/` is **one model for all five tasks** of the `dddddsds31341/franka-data` collection, trained with the
same H=16 recipe and the **pi05_expert** generator. The task is chosen by the prompt. Use the dataset's task string
exactly:
| Task | Dataset | Revision | Episodes | Frames | Share of training | Prompt |
|---|---|---|---|---|---|---|
| `bread` | `dddddsds31341/bread-lerobot` | `a45b8ba49fc1` | 100 | 28,380 | 19.3% | `Pick up both slices of bread from the toaster and place them on the plate.` |
| `insert_tube` | `dddddsds31341/insert-tube-lerobot` | `70bcd1f86ed3` | 102 | 16,403 | 11.2% | `Insert the gray tube into the empty black holder.` |
| `wipe_whiteboard` | `dddddsds31341/wipe-whiteboard-lerobot` | `2c8cfa8ddef1` | 112 | 29,795 | 20.3% | `Erase the writing from the whiteboard.` |
| `tennis_balls` | `dddddsds31341/tennis-balls-lerobot` | `04a27cd5ffe2` | 100 | 36,950 | 25.1% | `Place all tennis balls into the green basket.` |
| `test_tube` | `dddddsds31341/test-tube-lerobot` | `29d821a1a711` | 99 | 35,425 | 24.1% | `Place the test tube into the test tube rack.` |
The five datasets are merged into one local LeRobot v2.1 dataset (`code/adaspe_train/merge_datasets.py`; record in
`cotrain5_expert/merge_provenance.json`): 513 episodes, 146,953 frames, 10 Hz. Batches are sampled
uniformly over frames (no per-task reweighting), so each task's share equals its share of frames.
Differences from the single-task models:
- **Norm stats** were computed on the merged data. They are in each folder at
`assets/local/franka-cotrain5/norm_stats.json`. The per-task stats do not apply to this model.
- **One set of knot/timing constants** for all tasks, calibrated on the merged data (`cotrain5_expert/calibration_cotrain5_H16.json`):
e_crit = 9.216; timing labels 0.2 s 10.9%, 1.5 s 57.9%.
- Same step budget as one single-task run (Action 40k steps, batch 32), so each frame is seen about
8.7 times (single-task runs see each frame about 45 times).
| Folder | Training stage | Config name (with `TASK=cotrain5`) | Use |
|---|---|---|---|
| `cotrain5_expert/timing-from40k-lr3e-4-step-9999/` | Timing head only, peak LR 3e-4, 10k steps, from Action step 39999 | `cotrain5_adaspe_expert_timing_from40k_lr3e-4` | **Deployment (recommended)** |
| `cotrain5_expert/timing-from40k-step-9999/` | Timing head, default LR (peak 2.5e-5), from Action step 39999 | `cotrain5_adaspe_expert_timing_from40k` | Deployment, original timing LR |
| `cotrain5_expert/timing-from30k-step-9999/` | Timing head, default LR, from Action step 30000 | `cotrain5_adaspe_expert_timing_from30k` | Ablation |
| `cotrain5_expert/timing-from20k-step-9999/` | Timing head, default LR, from Action step 20000 | `cotrain5_adaspe_expert_timing_from20k` | Ablation |
| `cotrain5_expert/action-step-{10000,20000,30000,39999}/` | Action stage (EMA params) at that step | `cotrain5_adaspe_expert_action` | Action generator / knot head only. The timing head is untrained. |
Every timing folder is a full model. Only `freq_head` was trained in the timing stage, so actions and knots equal
those of the Action checkpoint it starts from.
Action loss (training, all five tasks mixed, mean per 2k-step window):
| Steps | cotrain5 expert |
|---|---|
| 0–2k | 0.345 |
| 8–10k | 0.192 |
| 18–20k | 0.146 |
| 28–30k | 0.123 |
| 38–40k | 0.110 |
Timing loss (log-interval MSE, training, steps 9k–10k): from20k 0.287, from30k 0.269,
from40k 0.267, from40k lr3e-4 0.232.
These are training losses. No held-out or real-robot evaluation results are included. `code/adaspe_train` now has the
`cotrain5` task (`tasks.py`) and `merge_datasets.py`. To retrain, download the five datasets, run
`TASK=cotrain5 python adaspe_train/merge_datasets.py`, then train with `TASK=cotrain5 VARIANT=expert`.
```python
root = snapshot_download("cognaclee/Frank-Insert-Wipe", allow_patterns=["cotrain5_expert/timing-from40k-lr3e-4-step-9999/*"])
ckpt_dir = f"{root}/cotrain5_expert/timing-from40k-lr3e-4-step-9999" # TASK=cotrain5, config cotrain5_adaspe_expert_timing_from40k_lr3e-4
prompt = "Place all tennis balls into the green basket." # one of the five task strings above
```
## Test tube (2026-11-16, pi05_expert only)
`test_tube_expert/` holds a **single-task** model for the test-tube task, trained with the same H=16 recipe and the
**pi05_expert** generator. (`cotrain5_expert/` also covers this task, but it is shared with four other tasks.) There
is no flow_head version. Every saved checkpoint is included (params only):
| Folder | Training stage | Config name (with `TASK=test_tube`) | Use |
|---|---|---|---|
| `test_tube_expert/timing-from40k-lr3e-4-step-9999/` | Timing head only, peak LR 3e-4, 10k steps, from Action step 39999 | `test_tube_adaspe_expert_timing_from40k_lr3e-4` | **Deployment (recommended)** |
| `test_tube_expert/timing-from40k-step-9999/` | Timing head, default LR (peak 2.5e-5), from Action step 39999 | `test_tube_adaspe_expert_timing_from40k` | Deployment, original timing LR |
| `test_tube_expert/timing-from30k-step-9999/` | Timing head, default LR, from Action step 30000 | `test_tube_adaspe_expert_timing_from30k` | Ablation |
| `test_tube_expert/timing-from20k-step-9999/` | Timing head, default LR, from Action step 20000 | `test_tube_adaspe_expert_timing_from20k` | Ablation |
| `test_tube_expert/action-step-{10000,20000,30000,39999}/` | Action stage (EMA params) at that step | `test_tube_adaspe_expert_action` | Action generator / knot head only. The timing head is untrained. |
Every timing folder is a full model. Only `freq_head` was trained in the timing stage, so actions and knots equal
those of the Action checkpoint it starts from.
Data: `dddddsds31341/test-tube-lerobot` (revision `29d821a1a71180948773e29c791f906306ac8018`), 99 episodes /
35,425 frames, 10 Hz. Prompt (the dataset's task string): `Place the test tube into the test tube rack.`
- **Norm stats** were computed on this dataset (there is no baseline-repo copy). They are in each folder at
`assets/dddddsds31341/test-tube-lerobot/norm_stats.json`, and also in
`code/adaspe_train/assets/pi05_test_tube_franka_ref/`.
- **Knot/timing constants** were calibrated on this dataset at H=16 (`test_tube_expert/calibration_test_tube_H16.json`):
e_crit = 9.930; timing labels 0.2 s 10.6%,
1.5 s 58.0%.
- Training: batch 32, Action 40k steps (OpenPI default LR: 1k warmup, peak 2.5e-5, cosine decay to 2.5e-6 at 30k),
then the timing stage (10k steps).
Action loss (training, mean per 2k-step window):
| Steps | test_tube expert |
|---|---|
| 0–2k | 0.328 |
| 8–10k | 0.151 |
| 18–20k | 0.102 |
| 28–30k | 0.065 |
| 38–40k | 0.056 |
Timing loss (log-interval MSE, training, steps 9k–10k): from20k 0.223, from30k 0.217,
from40k 0.216, from40k lr3e-4 0.171.
These are training losses. No held-out or real-robot evaluation results are included. `code/adaspe_train` has the
test_tube settings (`tasks.py`) and its calibration, timing-label and norm-stats files.
```python
root = snapshot_download("cognaclee/Frank-Insert-Wipe", allow_patterns=["test_tube_expert/timing-from40k-lr3e-4-step-9999/*"])
ckpt_dir = f"{root}/test_tube_expert/timing-from40k-lr3e-4-step-9999" # TASK=test_tube, config test_tube_adaspe_expert_timing_from40k_lr3e-4
prompt = "Place the test tube into the test tube rack."
```
## Tennis balls (2026-11-17, pi05_expert only)
`tennis_balls_expert/` holds a **single-task** model for the tennis-balls task, trained with the same H=16 recipe and the
**pi05_expert** generator. (`cotrain5_expert/` also covers this task, but it is shared with four other tasks.) There
is no flow_head version. Every saved checkpoint is included (params only):
| Folder | Training stage | Config name (with `TASK=tennis_balls`) | Use |
|---|---|---|---|
| `tennis_balls_expert/timing-from40k-lr3e-4-step-9999/` | Timing head only, peak LR 3e-4, 10k steps, from Action step 39999 | `tennis_balls_adaspe_expert_timing_from40k_lr3e-4` | **Deployment (recommended)** |
| `tennis_balls_expert/timing-from40k-step-9999/` | Timing head, default LR (peak 2.5e-5), from Action step 39999 | `tennis_balls_adaspe_expert_timing_from40k` | Deployment, original timing LR |
| `tennis_balls_expert/timing-from30k-step-9999/` | Timing head, default LR, from Action step 30000 | `tennis_balls_adaspe_expert_timing_from30k` | Ablation |
| `tennis_balls_expert/timing-from20k-step-9999/` | Timing head, default LR, from Action step 20000 | `tennis_balls_adaspe_expert_timing_from20k` | Ablation |
| `tennis_balls_expert/action-step-{10000,20000,30000,39999}/` | Action stage (EMA params) at that step | `tennis_balls_adaspe_expert_action` | Action generator / knot head only. The timing head is untrained. |
Every timing folder is a full model. Only `freq_head` was trained in the timing stage, so actions and knots equal
those of the Action checkpoint it starts from.
Data: `dddddsds31341/tennis-balls-lerobot` (revision `04a27cd5ffe22c250f0a23966256dc646b724a98`), 100 episodes /
36,950 frames, 10 Hz. Prompt (the dataset's task string): `Place all tennis balls into the green basket.`
- **Norm stats** were computed on this dataset (there is no baseline-repo copy). They are in each folder at
`assets/dddddsds31341/tennis-balls-lerobot/norm_stats.json`, and also in
`code/adaspe_train/assets/pi05_tennis_balls_franka_ref/`.
- **Knot/timing constants** were calibrated on this dataset at H=16 (`tennis_balls_expert/calibration_tennis_balls_H16.json`):
e_crit = 8.338; timing labels 0.2 s 12.0%,
1.5 s 49.2%.
- Training: batch 32, Action 40k steps (OpenPI default LR: 1k warmup, peak 2.5e-5, cosine decay to 2.5e-6 at 30k),
then the timing stage (10k steps).
Action loss (training, mean per 2k-step window):
| Steps | tennis_balls expert |
|---|---|
| 0–2k | 0.404 |
| 8–10k | 0.182 |
| 18–20k | 0.126 |
| 28–30k | 0.084 |
| 38–40k | 0.071 |
Timing loss (log-interval MSE, training, steps 9k–10k): from20k 0.295, from30k 0.290,
from40k 0.288, from40k lr3e-4 0.231.
These are training losses. No held-out or real-robot evaluation results are included. `code/adaspe_train` has the
tennis_balls settings (`tasks.py`) and its calibration, timing-label and norm-stats files.
```python
root = snapshot_download("cognaclee/Frank-Insert-Wipe", allow_patterns=["tennis_balls_expert/timing-from40k-lr3e-4-step-9999/*"])
ckpt_dir = f"{root}/tennis_balls_expert/timing-from40k-lr3e-4-step-9999" # TASK=tennis_balls, config tennis_balls_adaspe_expert_timing_from40k_lr3e-4
prompt = "Place all tennis balls into the green basket."
```
## Bread retrain: no state (2026-11-19, pi05_expert only)
`bread_nostate_expert/` is a retrain of the bread task on the **same data** as `bread_expert/`, with two recipe changes made after a
real-robot test of `bread_expert`. In that test the first close landed on air about one slice-height too high, the
policy still carried the empty gripper to the plate and opened it, and then closed around both slices (still in the
toaster) at the slice-2 position. pi0.5 has no memory, and the 19-D state in its prompt (gripper width, force, end-effector
pose, velocity) lets it tell the task phase from the arm pose rather than from the images.
1. **State input (no state)**: `state_keep=()`: the whole 19-D state is zeroed after normalization, so the prompt carries no proprioception and the policy must read the task phase from the two cameras. The mask (`configs.MaskState`) is the first model transform, after
normalization, so a policy built from this config (`TASK=bread_nostate`) applies it at inference as well. If your client
builds the model input another way, apply the same mask.
2. **Grasp-window oversampling (Action stage only)**: frames from 1.5 s before to 2 s after each gripper close command
(6,700 of 28,380 frames, 200 closes) are sampled 3Γ— as often, so they make up
48% of the samples
(`code/adaspe_train/adaspe/oversample.py`, index `code/adaspe_train/assets/sample_index/bread_grasp_x3.npy`). The
timing stage samples uniformly.
Everything else is as in `bread_expert/`: data `dddddsds31341/bread-lerobot` (revision `a45b8ba49fc164fb05d8a9c8bdf871ff97004357`),
100 episodes / 28,380 frames, same norm stats (`assets/dddddsds31341/bread-lerobot/norm_stats.json` in each folder), same calibration
(e_crit = 10.008, `bread_nostate_expert/calibration_bread_nostate_H16.json`), H=16, K=11, batch 32, Action 40k steps, timing 10k steps.
Prompt: `Pick up both slices of bread from the toaster and place them on the plate.`
| Folder | Training stage | Config name (with `TASK=bread_nostate`) | Use |
|---|---|---|---|
| `bread_nostate_expert/timing-from40k-lr3e-4-step-9999/` | Timing head only, peak LR 3e-4, 10k steps, from Action step 39999 | `bread_nostate_adaspe_expert_timing_from40k_lr3e-4` | **Deployment (recommended)** |
| `bread_nostate_expert/timing-from40k-step-9999/` | Timing head, default LR, from Action step 39999 | `bread_nostate_adaspe_expert_timing_from40k` | Deployment, original timing LR |
| `bread_nostate_expert/timing-from30k-step-9999/` | Timing head, default LR, from Action step 30000 | `bread_nostate_adaspe_expert_timing_from30k` | Ablation |
| `bread_nostate_expert/timing-from20k-step-9999/` | Timing head, default LR, from Action step 20000 | `bread_nostate_adaspe_expert_timing_from20k` | Ablation |
| `bread_nostate_expert/action-step-{10000,20000,30000,39999}/` | Action stage (EMA params) at that step | `bread_nostate_adaspe_expert_action` | Action generator / knot head only. The timing head is untrained. |
Action loss (training, mean per 2k-step window; grasp windows are oversampled, so not directly comparable with
`bread_expert`):
| Steps | bread_nostate expert |
|---|---|
| 0–2k | 0.329 |
| 8–10k | 0.191 |
| 18–20k | 0.138 |
| 28–30k | 0.097 |
| 38–40k | 0.090 |
Timing loss (log-interval MSE, training, steps 9k–10k): from20k 0.224, from30k 0.213, from40k 0.206, from40k lr3e-4 0.158.
These are training losses. No held-out or real-robot evaluation results are included.
```python
root = snapshot_download("cognaclee/Frank-Insert-Wipe", allow_patterns=["bread_nostate_expert/timing-from40k-lr3e-4-step-9999/*"])
ckpt_dir = f"{root}/bread_nostate_expert/timing-from40k-lr3e-4-step-9999" # TASK=bread_nostate, config bread_nostate_adaspe_expert_timing_from40k_lr3e-4
prompt = "Pick up both slices of bread from the toaster and place them on the plate."
```
## Bread retrain: gripper width only (2026-11-19, pi05_expert only)
`bread_gripstate_expert/` is a retrain of the bread task on the **same data** as `bread_expert/`, with two recipe changes made after a
real-robot test of `bread_expert`. In that test the first close landed on air about one slice-height too high, the
policy still carried the empty gripper to the plate and opened it, and then closed around both slices (still in the
toaster) at the slice-2 position. pi0.5 has no memory, and the 19-D state in its prompt (gripper width, force, end-effector
pose, velocity) lets it tell the task phase from the arm pose rather than from the images.
1. **State input (gripper width only)**: `state_keep=(0,)`: only `state[0]` (gripper width) is kept; the other 18 dims are zeroed after normalization. The mask (`configs.MaskState`) is the first model transform, after
normalization, so a policy built from this config (`TASK=bread_gripstate`) applies it at inference as well. If your client
builds the model input another way, apply the same mask.
2. **Grasp-window oversampling (Action stage only)**: frames from 1.5 s before to 2 s after each gripper close command
(6,700 of 28,380 frames, 200 closes) are sampled 3Γ— as often, so they make up
48% of the samples
(`code/adaspe_train/adaspe/oversample.py`, index `code/adaspe_train/assets/sample_index/bread_grasp_x3.npy`). The
timing stage samples uniformly.
Everything else is as in `bread_expert/`: data `dddddsds31341/bread-lerobot` (revision `a45b8ba49fc164fb05d8a9c8bdf871ff97004357`),
100 episodes / 28,380 frames, same norm stats (`assets/dddddsds31341/bread-lerobot/norm_stats.json` in each folder), same calibration
(e_crit = 10.008, `bread_gripstate_expert/calibration_bread_gripstate_H16.json`), H=16, K=11, batch 32, Action 40k steps, timing 10k steps.
Prompt: `Pick up both slices of bread from the toaster and place them on the plate.`
| Folder | Training stage | Config name (with `TASK=bread_gripstate`) | Use |
|---|---|---|---|
| `bread_gripstate_expert/timing-from40k-lr3e-4-step-9999/` | Timing head only, peak LR 3e-4, 10k steps, from Action step 39999 | `bread_gripstate_adaspe_expert_timing_from40k_lr3e-4` | **Deployment (recommended)** |
| `bread_gripstate_expert/timing-from40k-step-9999/` | Timing head, default LR, from Action step 39999 | `bread_gripstate_adaspe_expert_timing_from40k` | Deployment, original timing LR |
| `bread_gripstate_expert/timing-from30k-step-9999/` | Timing head, default LR, from Action step 30000 | `bread_gripstate_adaspe_expert_timing_from30k` | Ablation |
| `bread_gripstate_expert/timing-from20k-step-9999/` | Timing head, default LR, from Action step 20000 | `bread_gripstate_adaspe_expert_timing_from20k` | Ablation |
| `bread_gripstate_expert/action-step-{10000,20000,30000,39999}/` | Action stage (EMA params) at that step | `bread_gripstate_adaspe_expert_action` | Action generator / knot head only. The timing head is untrained. |
Action loss (training, mean per 2k-step window; grasp windows are oversampled, so not directly comparable with
`bread_expert`):
| Steps | bread_gripstate expert |
|---|---|
| 0–2k | 0.348 |
| 8–10k | 0.190 |
| 18–20k | 0.136 |
| 28–30k | 0.085 |
| 38–40k | 0.076 |
Timing loss (log-interval MSE, training, steps 9k–10k): from20k 0.224, from30k 0.212, from40k 0.208, from40k lr3e-4 0.156.
These are training losses. No held-out or real-robot evaluation results are included.
```python
root = snapshot_download("cognaclee/Frank-Insert-Wipe", allow_patterns=["bread_gripstate_expert/timing-from40k-lr3e-4-step-9999/*"])
ckpt_dir = f"{root}/bread_gripstate_expert/timing-from40k-lr3e-4-step-9999" # TASK=bread_gripstate, config bread_gripstate_adaspe_expert_timing_from40k_lr3e-4
prompt = "Pick up both slices of bread from the toaster and place them on the plate."
```