|
Download README.md from cognaclee/Frank-Insert-Wipe: direct link, hf CLI and curl.
- Browser
- Download file 28.3 kB
-
https://huggingface.co/cognaclee/Frank-Insert-Wipe/resolve/main/README.md
- Command line
-
hf download hf://cognaclee/Frank-Insert-Wipe/README.md
-
curl -L -o README.md https://huggingface.co/cognaclee/Frank-Insert-Wipe/resolve/main/README.md
28.3 kB
| tags: | |
| - robotics | |
| - openpi | |
| - pi0.5 | |
| - franka | |
| library_name: openpi | |
| # Frank-Insert-Wipe: AdaSpeVLA checkpoints for Franka insert_tube, wipe_whiteboard, bread, test_tube, tennis_balls and a 5-task co-trained model (H=16) | |
| AdaSpeVLA is a pi0.5 (OpenPI, JAX) variant. It predicts cubic B-spline action chunks with adaptive knots, plus a | |
| timing head that decides when to query the policy again. This repo holds two variants that differ only in the | |
| generator that produces the 11 control points: | |
| - **flow_head** (`<task>/`, 2026-11-12): a 20.4 M FlowPolicy trained from scratch. | |
| - **pi05_expert** (`<task>_expert/`, 2026-11-14): pi0.5's pretrained Gemma-300M action expert (427.9 M with adaRMS, | |
| from `pi05_base`) runs on the 11 control-point tokens, as pi0.5 does on its action tokens. | |
| Everything else (data, norm stats, H=16, K=11, labels, knot head, timing head, 40k + 10k steps) is the same. | |
| The training code for both is in `code/`. | |
| These are params-only Orbax checkpoints in the OpenPI layout. The optimizer `train_state` is not included. | |
| | Folder | Training stage | Config name (with `TASK=<task>`) | Use | | |
| |---|---|---|---| | |
| | `<task>/timing-from40k-step-9999/` | Timing head, 10k steps, from Action step 39999 (everything except `freq_head` frozen) | `<task>_adaspe_timing_from40k` | **Deployment**: full model with the trained timing head | | |
| | `<task>/action-step-39999/` | Action stage, 40k steps (EMA params) | `<task>_adaspe_action` | Action generator / knot head only. Its timing head is untrained. | | |
| | `<task>_expert/timing-from40k-step-9999/` | Same, pi05_expert generator | `<task>_adaspe_expert_timing_from40k` | **Deployment** (expert variant) | | |
| | `<task>_expert/action-step-39999/` | Same, pi05_expert generator | `<task>_adaspe_expert_action` | Action generator / knot head only (expert variant) | | |
| | `<task>/timing-from40k-lr3e-4-step-9999/` | Timing head only, retrained 2026-11-14 with peak LR 3e-4 (same 10k steps; see below) | `<task>_adaspe_timing_from40k_lr3e-4` | Deployment candidate | | |
| | `<task>/timing-from40k-lr1e-3-step-9999/` | Timing head only, retrained 2026-11-14 with peak LR 1e-3 (same 10k steps; see below) | `<task>_adaspe_timing_from40k_lr1e-3` | Deployment candidate | | |
| | `<task>_expert/timing-from40k-lr3e-4-step-9999/` | Timing head only, retrained 2026-11-14 with peak LR 3e-4 (same 10k steps; see below) | `<task>_adaspe_expert_timing_from40k_lr3e-4` | Deployment candidate (expert variant) | | |
| | `<task>_expert/timing-from40k-lr1e-3-step-9999/` | Timing head only, retrained 2026-11-14 with peak LR 1e-3 (same 10k steps; see below) | `<task>_adaspe_expert_timing_from40k_lr1e-3` | Deployment candidate (expert variant) | | |
| `<task>` is `insert_tube` or `wipe_whiteboard`. Each folder has `params/`, `assets/dddddsds31341/<dataset>/norm_stats.json` | |
| (the normalization stats used in training) and `_CHECKPOINT_METADATA`. `SHA256SUMS` lists every file. | |
| ## Loading | |
| You need the AdaSpeVLA OpenPI plugin, included here as `code/adaspe_train` (run with `TASK=<task>`; it registers | |
| these config names). It runs on OpenPI at the commit in `code/OPENPI_COMMIT`. Setup is in `code/README.md`, and | |
| `code/adaspe_train/EXPERT_VARIANT.md` documents the expert variant. | |
| The folders use the standard OpenPI checkpoint layout (`params/` + `assets/`). | |
| ```python | |
| from huggingface_hub import snapshot_download | |
| root = snapshot_download("cognaclee/Frank-Insert-Wipe", allow_patterns=["insert_tube/timing-from40k-step-9999/*"]) | |
| # higher-LR timing head: "insert_tube/timing-from40k-lr3e-4-step-9999/*" with config insert_tube_adaspe_timing_from40k_lr3e-4 | |
| ckpt_dir = f"{root}/insert_tube/timing-from40k-step-9999" # with config insert_tube_adaspe_timing_from40k | |
| ``` | |
| ## Data | |
| | Task | Dataset (revision) | Size | pi0.5 baseline (same norm stats) | | |
| |---|---|---|---| | |
| | insert_tube | `dddddsds31341/insert-tube-lerobot` (`70bcd1f86ed35fc93a0422a564472355242a51ab`) | 102 episodes / 16,403 frames | `dddddsds31341/Franka-Insert-Tube-pi05-step-20000` | | |
| | wipe_whiteboard | `dddddsds31341/wipe-whiteboard-lerobot` (`2c8cfa8ddef1f83cb09ca7881a0944cd637cd6d0`) | 112 episodes / 29,795 frames | `dddddsds31341/Franka-Wipe-Whiteboard-pi05-step-20000` | | |
| Two 128Γ128 cameras, 19-D state, 7-D action, 10 Hz. | |
| ## Recipe | |
| - Action horizon H=16 (1.5 s), 11 cubic B-spline control points, query interval 0.2β1.5 s. | |
| - Gripper events for the timing and knot labels come from the gripper command (the measured width lags by about 1.9 s). | |
| - Knot density is tanh(motion_variation / e_crit), with e_crit calibrated per task at H=16: insert_tube 11.98, wipe_whiteboard 8.96. | |
| - Initialized from `pi05_base`. Batch 32, Action 40k steps (LR decay over 30k), then timing 10k steps. | |
| ## Training loss | |
| Mean training loss over the last 1kβ2k steps of each stage. | |
| | Task | Variant | Action loss, steps 38kβ40k | Timing loss (log-interval MSE), steps 9kβ10k | | |
| |---|---|---|---| | |
| | insert_tube | flow_head | 0.086 | 0.206 | | |
| | insert_tube | pi05_expert | **0.030** | 0.255 | | |
| | wipe_whiteboard | flow_head | 0.118 | 0.196 | | |
| | wipe_whiteboard | pi05_expert | **0.040** | 0.271 | | |
| Action loss over training (mean per 2k-step window): | |
| | Steps | insert_tube flow_head | insert_tube expert | wipe flow_head | wipe expert | | |
| |---|---|---|---|---| | |
| | 0β2k | 0.624 | 0.334 | 0.597 | 0.295 | | |
| | 8β10k | 0.238 | 0.121 | 0.238 | 0.121 | | |
| | 18β20k | 0.156 | 0.067 | 0.169 | 0.083 | | |
| | 28β30k | 0.105 | 0.038 | 0.130 | 0.049 | | |
| | 38β40k | 0.086 | 0.030 | 0.118 | 0.040 | | |
| Notes: | |
| - The two variants use different flow-matching conventions (flow_head: t ~ U(0,1); pi05_expert: pi0.5's | |
| Beta(1.5,1)), so the action losses are not exactly the same metric. | |
| - The expert's timing head has a **higher** loss. It reads a pooled observation feature that, in the expert | |
| variant, is trained only by the knot loss (see `code/adaspe_train/EXPERT_VARIANT.md`). | |
| - These are training losses. No held-out or real-robot evaluation results are included here. | |
| ## Timing head retrained with a higher learning rate (2026-11-14) | |
| The original timing heads (`timing-from40k-step-9999`) used OpenPI's default schedule: peak LR 2.5e-5 with a cosine | |
| decay over 30k steps. The timing stage only runs 10k steps, so the LR stayed at about 2.0β2.5e-5, which is low for | |
| `freq_head`, a freshly initialized 2-layer MLP (633 β 128 β 1, ~81k params) and the only trainable part. The loss was | |
| still falling at 10k. | |
| The `timing-from40k-lr<LR>-step-9999` folders retrain **only `freq_head`**, with the same data, labels, 10k steps, | |
| batch 32 and 1k warmup, a cosine decay to LR/10 over the 10k steps, and peak LR 3e-4 or 1e-3. They start from the | |
| same Action step-39999 params, and everything except `freq_head` is frozen. The predicted actions, knots and | |
| B-spline are therefore identical to `timing-from40k-step-9999`. Only the query interval (and so `execute_steps`) | |
| changes. | |
| Timing loss (log-interval MSE, training loss, mean over steps 9kβ10k): | |
| | Task | Variant | LR 2.5e-5 (original) | LR 3e-4 | LR 1e-3 | | |
| |---|---|---|---|---| | |
| | insert_tube | flow_head | 0.206 | 0.157 | 0.091 | | |
| | insert_tube | pi05_expert | 0.255 | 0.182 | 0.100 | | |
| | wipe_whiteboard | flow_head | 0.196 | 0.158 | 0.112 | | |
| | wipe_whiteboard | pi05_expert | 0.271 | 0.214 | 0.153 | | |
| These are **training** losses. With the higher LR, part of the drop may come from fitting the training frames; a | |
| held-out comparison has not been done yet. The configs are `configs.TIMINGS_LR` in `code/adaspe_train/configs.py`, | |
| and `code/adaspe_train/slurm/timing_lr.slurm` runs one of them. | |
| ## Bread (2026-11-15, pi05_expert only) | |
| `bread_expert/` holds the bread task trained with the same H=16 recipe and the **pi05_expert** generator. There | |
| is no flow_head version of bread. Unlike the other tasks, every saved checkpoint is included (params only): | |
| | Folder | Training stage | Config name (with `TASK=bread`) | Use | | |
| |---|---|---|---| | |
| | `bread_expert/timing-from40k-lr3e-4-step-9999/` | Timing head only, peak LR 3e-4, 10k steps, from Action step 39999 | `bread_adaspe_expert_timing_from40k_lr3e-4` | **Deployment (recommended)** | | |
| | `bread_expert/timing-from40k-step-9999/` | Timing head, default LR (peak 2.5e-5), from Action step 39999 | `bread_adaspe_expert_timing_from40k` | Deployment, original timing LR | | |
| | `bread_expert/timing-from30k-step-9999/` | Timing head, default LR, from Action step 30000 | `bread_adaspe_expert_timing_from30k` | Ablation | | |
| | `bread_expert/timing-from20k-step-9999/` | Timing head, default LR, from Action step 20000 | `bread_adaspe_expert_timing_from20k` | Ablation | | |
| | `bread_expert/action-step-{10000,20000,30000,39999}/` | Action stage (EMA params) at that step | `bread_adaspe_expert_action` | Action generator / knot head only. The timing head is untrained. | | |
| Every timing folder is a full model. Only `freq_head` was trained in the timing stage, so actions and knots equal | |
| those of the Action checkpoint it starts from. | |
| Data: `dddddsds31341/bread-lerobot` (revision `a45b8ba49fc164fb05d8a9c8bdf871ff97004357`), 100 episodes / | |
| 28,380 frames, 10 Hz. e_crit = 10.01 (calibrated at H=16). Training: batch 32, Action 40k steps (OpenPI default LR: | |
| 1k warmup, peak 2.5e-5, cosine decay to 2.5e-6 at 30k), then the timing stage (10k steps). | |
| Action loss (training, mean per 2k-step window): | |
| | Steps | bread expert | | |
| |---|---| | |
| | 0β2k | 0.357 | | |
| | 8β10k | 0.161 | | |
| | 18β20k | 0.114 | | |
| | 28β30k | 0.093 | | |
| | 38β40k | 0.083 | | |
| Timing loss (log-interval MSE, training, steps 9kβ10k): from20k 0.217, from30k 0.217, | |
| from40k 0.212, from40k lr3e-4 0.159. | |
| These are training losses. No held-out or real-robot evaluation results are included. `code/adaspe_train` has the | |
| bread settings (`tasks.py`, with checkpoints saved every 10k steps). | |
| ```python | |
| root = snapshot_download("cognaclee/Frank-Insert-Wipe", allow_patterns=["bread_expert/timing-from40k-lr3e-4-step-9999/*"]) | |
| ckpt_dir = f"{root}/bread_expert/timing-from40k-lr3e-4-step-9999" # TASK=bread, config bread_adaspe_expert_timing_from40k_lr3e-4 | |
| ``` | |
| ## Five-task co-training (2026-11-15, pi05_expert only) | |
| `cotrain5_expert/` is **one model for all five tasks** of the `dddddsds31341/franka-data` collection, trained with the | |
| same H=16 recipe and the **pi05_expert** generator. The task is chosen by the prompt. Use the dataset's task string | |
| exactly: | |
| | Task | Dataset | Revision | Episodes | Frames | Share of training | Prompt | | |
| |---|---|---|---|---|---|---| | |
| | `bread` | `dddddsds31341/bread-lerobot` | `a45b8ba49fc1` | 100 | 28,380 | 19.3% | `Pick up both slices of bread from the toaster and place them on the plate.` | | |
| | `insert_tube` | `dddddsds31341/insert-tube-lerobot` | `70bcd1f86ed3` | 102 | 16,403 | 11.2% | `Insert the gray tube into the empty black holder.` | | |
| | `wipe_whiteboard` | `dddddsds31341/wipe-whiteboard-lerobot` | `2c8cfa8ddef1` | 112 | 29,795 | 20.3% | `Erase the writing from the whiteboard.` | | |
| | `tennis_balls` | `dddddsds31341/tennis-balls-lerobot` | `04a27cd5ffe2` | 100 | 36,950 | 25.1% | `Place all tennis balls into the green basket.` | | |
| | `test_tube` | `dddddsds31341/test-tube-lerobot` | `29d821a1a711` | 99 | 35,425 | 24.1% | `Place the test tube into the test tube rack.` | | |
| The five datasets are merged into one local LeRobot v2.1 dataset (`code/adaspe_train/merge_datasets.py`; record in | |
| `cotrain5_expert/merge_provenance.json`): 513 episodes, 146,953 frames, 10 Hz. Batches are sampled | |
| uniformly over frames (no per-task reweighting), so each task's share equals its share of frames. | |
| Differences from the single-task models: | |
| - **Norm stats** were computed on the merged data. They are in each folder at | |
| `assets/local/franka-cotrain5/norm_stats.json`. The per-task stats do not apply to this model. | |
| - **One set of knot/timing constants** for all tasks, calibrated on the merged data (`cotrain5_expert/calibration_cotrain5_H16.json`): | |
| e_crit = 9.216; timing labels 0.2 s 10.9%, 1.5 s 57.9%. | |
| - Same step budget as one single-task run (Action 40k steps, batch 32), so each frame is seen about | |
| 8.7 times (single-task runs see each frame about 45 times). | |
| | Folder | Training stage | Config name (with `TASK=cotrain5`) | Use | | |
| |---|---|---|---| | |
| | `cotrain5_expert/timing-from40k-lr3e-4-step-9999/` | Timing head only, peak LR 3e-4, 10k steps, from Action step 39999 | `cotrain5_adaspe_expert_timing_from40k_lr3e-4` | **Deployment (recommended)** | | |
| | `cotrain5_expert/timing-from40k-step-9999/` | Timing head, default LR (peak 2.5e-5), from Action step 39999 | `cotrain5_adaspe_expert_timing_from40k` | Deployment, original timing LR | | |
| | `cotrain5_expert/timing-from30k-step-9999/` | Timing head, default LR, from Action step 30000 | `cotrain5_adaspe_expert_timing_from30k` | Ablation | | |
| | `cotrain5_expert/timing-from20k-step-9999/` | Timing head, default LR, from Action step 20000 | `cotrain5_adaspe_expert_timing_from20k` | Ablation | | |
| | `cotrain5_expert/action-step-{10000,20000,30000,39999}/` | Action stage (EMA params) at that step | `cotrain5_adaspe_expert_action` | Action generator / knot head only. The timing head is untrained. | | |
| Every timing folder is a full model. Only `freq_head` was trained in the timing stage, so actions and knots equal | |
| those of the Action checkpoint it starts from. | |
| Action loss (training, all five tasks mixed, mean per 2k-step window): | |
| | Steps | cotrain5 expert | | |
| |---|---| | |
| | 0β2k | 0.345 | | |
| | 8β10k | 0.192 | | |
| | 18β20k | 0.146 | | |
| | 28β30k | 0.123 | | |
| | 38β40k | 0.110 | | |
| Timing loss (log-interval MSE, training, steps 9kβ10k): from20k 0.287, from30k 0.269, | |
| from40k 0.267, from40k lr3e-4 0.232. | |
| These are training losses. No held-out or real-robot evaluation results are included. `code/adaspe_train` now has the | |
| `cotrain5` task (`tasks.py`) and `merge_datasets.py`. To retrain, download the five datasets, run | |
| `TASK=cotrain5 python adaspe_train/merge_datasets.py`, then train with `TASK=cotrain5 VARIANT=expert`. | |
| ```python | |
| root = snapshot_download("cognaclee/Frank-Insert-Wipe", allow_patterns=["cotrain5_expert/timing-from40k-lr3e-4-step-9999/*"]) | |
| ckpt_dir = f"{root}/cotrain5_expert/timing-from40k-lr3e-4-step-9999" # TASK=cotrain5, config cotrain5_adaspe_expert_timing_from40k_lr3e-4 | |
| prompt = "Place all tennis balls into the green basket." # one of the five task strings above | |
| ``` | |
| ## Test tube (2026-11-16, pi05_expert only) | |
| `test_tube_expert/` holds a **single-task** model for the test-tube task, trained with the same H=16 recipe and the | |
| **pi05_expert** generator. (`cotrain5_expert/` also covers this task, but it is shared with four other tasks.) There | |
| is no flow_head version. Every saved checkpoint is included (params only): | |
| | Folder | Training stage | Config name (with `TASK=test_tube`) | Use | | |
| |---|---|---|---| | |
| | `test_tube_expert/timing-from40k-lr3e-4-step-9999/` | Timing head only, peak LR 3e-4, 10k steps, from Action step 39999 | `test_tube_adaspe_expert_timing_from40k_lr3e-4` | **Deployment (recommended)** | | |
| | `test_tube_expert/timing-from40k-step-9999/` | Timing head, default LR (peak 2.5e-5), from Action step 39999 | `test_tube_adaspe_expert_timing_from40k` | Deployment, original timing LR | | |
| | `test_tube_expert/timing-from30k-step-9999/` | Timing head, default LR, from Action step 30000 | `test_tube_adaspe_expert_timing_from30k` | Ablation | | |
| | `test_tube_expert/timing-from20k-step-9999/` | Timing head, default LR, from Action step 20000 | `test_tube_adaspe_expert_timing_from20k` | Ablation | | |
| | `test_tube_expert/action-step-{10000,20000,30000,39999}/` | Action stage (EMA params) at that step | `test_tube_adaspe_expert_action` | Action generator / knot head only. The timing head is untrained. | | |
| Every timing folder is a full model. Only `freq_head` was trained in the timing stage, so actions and knots equal | |
| those of the Action checkpoint it starts from. | |
| Data: `dddddsds31341/test-tube-lerobot` (revision `29d821a1a71180948773e29c791f906306ac8018`), 99 episodes / | |
| 35,425 frames, 10 Hz. Prompt (the dataset's task string): `Place the test tube into the test tube rack.` | |
| - **Norm stats** were computed on this dataset (there is no baseline-repo copy). They are in each folder at | |
| `assets/dddddsds31341/test-tube-lerobot/norm_stats.json`, and also in | |
| `code/adaspe_train/assets/pi05_test_tube_franka_ref/`. | |
| - **Knot/timing constants** were calibrated on this dataset at H=16 (`test_tube_expert/calibration_test_tube_H16.json`): | |
| e_crit = 9.930; timing labels 0.2 s 10.6%, | |
| 1.5 s 58.0%. | |
| - Training: batch 32, Action 40k steps (OpenPI default LR: 1k warmup, peak 2.5e-5, cosine decay to 2.5e-6 at 30k), | |
| then the timing stage (10k steps). | |
| Action loss (training, mean per 2k-step window): | |
| | Steps | test_tube expert | | |
| |---|---| | |
| | 0β2k | 0.328 | | |
| | 8β10k | 0.151 | | |
| | 18β20k | 0.102 | | |
| | 28β30k | 0.065 | | |
| | 38β40k | 0.056 | | |
| Timing loss (log-interval MSE, training, steps 9kβ10k): from20k 0.223, from30k 0.217, | |
| from40k 0.216, from40k lr3e-4 0.171. | |
| These are training losses. No held-out or real-robot evaluation results are included. `code/adaspe_train` has the | |
| test_tube settings (`tasks.py`) and its calibration, timing-label and norm-stats files. | |
| ```python | |
| root = snapshot_download("cognaclee/Frank-Insert-Wipe", allow_patterns=["test_tube_expert/timing-from40k-lr3e-4-step-9999/*"]) | |
| ckpt_dir = f"{root}/test_tube_expert/timing-from40k-lr3e-4-step-9999" # TASK=test_tube, config test_tube_adaspe_expert_timing_from40k_lr3e-4 | |
| prompt = "Place the test tube into the test tube rack." | |
| ``` | |
| ## Tennis balls (2026-11-17, pi05_expert only) | |
| `tennis_balls_expert/` holds a **single-task** model for the tennis-balls task, trained with the same H=16 recipe and the | |
| **pi05_expert** generator. (`cotrain5_expert/` also covers this task, but it is shared with four other tasks.) There | |
| is no flow_head version. Every saved checkpoint is included (params only): | |
| | Folder | Training stage | Config name (with `TASK=tennis_balls`) | Use | | |
| |---|---|---|---| | |
| | `tennis_balls_expert/timing-from40k-lr3e-4-step-9999/` | Timing head only, peak LR 3e-4, 10k steps, from Action step 39999 | `tennis_balls_adaspe_expert_timing_from40k_lr3e-4` | **Deployment (recommended)** | | |
| | `tennis_balls_expert/timing-from40k-step-9999/` | Timing head, default LR (peak 2.5e-5), from Action step 39999 | `tennis_balls_adaspe_expert_timing_from40k` | Deployment, original timing LR | | |
| | `tennis_balls_expert/timing-from30k-step-9999/` | Timing head, default LR, from Action step 30000 | `tennis_balls_adaspe_expert_timing_from30k` | Ablation | | |
| | `tennis_balls_expert/timing-from20k-step-9999/` | Timing head, default LR, from Action step 20000 | `tennis_balls_adaspe_expert_timing_from20k` | Ablation | | |
| | `tennis_balls_expert/action-step-{10000,20000,30000,39999}/` | Action stage (EMA params) at that step | `tennis_balls_adaspe_expert_action` | Action generator / knot head only. The timing head is untrained. | | |
| Every timing folder is a full model. Only `freq_head` was trained in the timing stage, so actions and knots equal | |
| those of the Action checkpoint it starts from. | |
| Data: `dddddsds31341/tennis-balls-lerobot` (revision `04a27cd5ffe22c250f0a23966256dc646b724a98`), 100 episodes / | |
| 36,950 frames, 10 Hz. Prompt (the dataset's task string): `Place all tennis balls into the green basket.` | |
| - **Norm stats** were computed on this dataset (there is no baseline-repo copy). They are in each folder at | |
| `assets/dddddsds31341/tennis-balls-lerobot/norm_stats.json`, and also in | |
| `code/adaspe_train/assets/pi05_tennis_balls_franka_ref/`. | |
| - **Knot/timing constants** were calibrated on this dataset at H=16 (`tennis_balls_expert/calibration_tennis_balls_H16.json`): | |
| e_crit = 8.338; timing labels 0.2 s 12.0%, | |
| 1.5 s 49.2%. | |
| - Training: batch 32, Action 40k steps (OpenPI default LR: 1k warmup, peak 2.5e-5, cosine decay to 2.5e-6 at 30k), | |
| then the timing stage (10k steps). | |
| Action loss (training, mean per 2k-step window): | |
| | Steps | tennis_balls expert | | |
| |---|---| | |
| | 0β2k | 0.404 | | |
| | 8β10k | 0.182 | | |
| | 18β20k | 0.126 | | |
| | 28β30k | 0.084 | | |
| | 38β40k | 0.071 | | |
| Timing loss (log-interval MSE, training, steps 9kβ10k): from20k 0.295, from30k 0.290, | |
| from40k 0.288, from40k lr3e-4 0.231. | |
| These are training losses. No held-out or real-robot evaluation results are included. `code/adaspe_train` has the | |
| tennis_balls settings (`tasks.py`) and its calibration, timing-label and norm-stats files. | |
| ```python | |
| root = snapshot_download("cognaclee/Frank-Insert-Wipe", allow_patterns=["tennis_balls_expert/timing-from40k-lr3e-4-step-9999/*"]) | |
| ckpt_dir = f"{root}/tennis_balls_expert/timing-from40k-lr3e-4-step-9999" # TASK=tennis_balls, config tennis_balls_adaspe_expert_timing_from40k_lr3e-4 | |
| prompt = "Place all tennis balls into the green basket." | |
| ``` | |
| ## Bread retrain: no state (2026-11-19, pi05_expert only) | |
| `bread_nostate_expert/` is a retrain of the bread task on the **same data** as `bread_expert/`, with two recipe changes made after a | |
| real-robot test of `bread_expert`. In that test the first close landed on air about one slice-height too high, the | |
| policy still carried the empty gripper to the plate and opened it, and then closed around both slices (still in the | |
| toaster) at the slice-2 position. pi0.5 has no memory, and the 19-D state in its prompt (gripper width, force, end-effector | |
| pose, velocity) lets it tell the task phase from the arm pose rather than from the images. | |
| 1. **State input (no state)**: `state_keep=()`: the whole 19-D state is zeroed after normalization, so the prompt carries no proprioception and the policy must read the task phase from the two cameras. The mask (`configs.MaskState`) is the first model transform, after | |
| normalization, so a policy built from this config (`TASK=bread_nostate`) applies it at inference as well. If your client | |
| builds the model input another way, apply the same mask. | |
| 2. **Grasp-window oversampling (Action stage only)**: frames from 1.5 s before to 2 s after each gripper close command | |
| (6,700 of 28,380 frames, 200 closes) are sampled 3Γ as often, so they make up | |
| 48% of the samples | |
| (`code/adaspe_train/adaspe/oversample.py`, index `code/adaspe_train/assets/sample_index/bread_grasp_x3.npy`). The | |
| timing stage samples uniformly. | |
| Everything else is as in `bread_expert/`: data `dddddsds31341/bread-lerobot` (revision `a45b8ba49fc164fb05d8a9c8bdf871ff97004357`), | |
| 100 episodes / 28,380 frames, same norm stats (`assets/dddddsds31341/bread-lerobot/norm_stats.json` in each folder), same calibration | |
| (e_crit = 10.008, `bread_nostate_expert/calibration_bread_nostate_H16.json`), H=16, K=11, batch 32, Action 40k steps, timing 10k steps. | |
| Prompt: `Pick up both slices of bread from the toaster and place them on the plate.` | |
| | Folder | Training stage | Config name (with `TASK=bread_nostate`) | Use | | |
| |---|---|---|---| | |
| | `bread_nostate_expert/timing-from40k-lr3e-4-step-9999/` | Timing head only, peak LR 3e-4, 10k steps, from Action step 39999 | `bread_nostate_adaspe_expert_timing_from40k_lr3e-4` | **Deployment (recommended)** | | |
| | `bread_nostate_expert/timing-from40k-step-9999/` | Timing head, default LR, from Action step 39999 | `bread_nostate_adaspe_expert_timing_from40k` | Deployment, original timing LR | | |
| | `bread_nostate_expert/timing-from30k-step-9999/` | Timing head, default LR, from Action step 30000 | `bread_nostate_adaspe_expert_timing_from30k` | Ablation | | |
| | `bread_nostate_expert/timing-from20k-step-9999/` | Timing head, default LR, from Action step 20000 | `bread_nostate_adaspe_expert_timing_from20k` | Ablation | | |
| | `bread_nostate_expert/action-step-{10000,20000,30000,39999}/` | Action stage (EMA params) at that step | `bread_nostate_adaspe_expert_action` | Action generator / knot head only. The timing head is untrained. | | |
| Action loss (training, mean per 2k-step window; grasp windows are oversampled, so not directly comparable with | |
| `bread_expert`): | |
| | Steps | bread_nostate expert | | |
| |---|---| | |
| | 0β2k | 0.329 | | |
| | 8β10k | 0.191 | | |
| | 18β20k | 0.138 | | |
| | 28β30k | 0.097 | | |
| | 38β40k | 0.090 | | |
| Timing loss (log-interval MSE, training, steps 9kβ10k): from20k 0.224, from30k 0.213, from40k 0.206, from40k lr3e-4 0.158. | |
| These are training losses. No held-out or real-robot evaluation results are included. | |
| ```python | |
| root = snapshot_download("cognaclee/Frank-Insert-Wipe", allow_patterns=["bread_nostate_expert/timing-from40k-lr3e-4-step-9999/*"]) | |
| ckpt_dir = f"{root}/bread_nostate_expert/timing-from40k-lr3e-4-step-9999" # TASK=bread_nostate, config bread_nostate_adaspe_expert_timing_from40k_lr3e-4 | |
| prompt = "Pick up both slices of bread from the toaster and place them on the plate." | |
| ``` | |
| ## Bread retrain: gripper width only (2026-11-19, pi05_expert only) | |
| `bread_gripstate_expert/` is a retrain of the bread task on the **same data** as `bread_expert/`, with two recipe changes made after a | |
| real-robot test of `bread_expert`. In that test the first close landed on air about one slice-height too high, the | |
| policy still carried the empty gripper to the plate and opened it, and then closed around both slices (still in the | |
| toaster) at the slice-2 position. pi0.5 has no memory, and the 19-D state in its prompt (gripper width, force, end-effector | |
| pose, velocity) lets it tell the task phase from the arm pose rather than from the images. | |
| 1. **State input (gripper width only)**: `state_keep=(0,)`: only `state[0]` (gripper width) is kept; the other 18 dims are zeroed after normalization. The mask (`configs.MaskState`) is the first model transform, after | |
| normalization, so a policy built from this config (`TASK=bread_gripstate`) applies it at inference as well. If your client | |
| builds the model input another way, apply the same mask. | |
| 2. **Grasp-window oversampling (Action stage only)**: frames from 1.5 s before to 2 s after each gripper close command | |
| (6,700 of 28,380 frames, 200 closes) are sampled 3Γ as often, so they make up | |
| 48% of the samples | |
| (`code/adaspe_train/adaspe/oversample.py`, index `code/adaspe_train/assets/sample_index/bread_grasp_x3.npy`). The | |
| timing stage samples uniformly. | |
| Everything else is as in `bread_expert/`: data `dddddsds31341/bread-lerobot` (revision `a45b8ba49fc164fb05d8a9c8bdf871ff97004357`), | |
| 100 episodes / 28,380 frames, same norm stats (`assets/dddddsds31341/bread-lerobot/norm_stats.json` in each folder), same calibration | |
| (e_crit = 10.008, `bread_gripstate_expert/calibration_bread_gripstate_H16.json`), H=16, K=11, batch 32, Action 40k steps, timing 10k steps. | |
| Prompt: `Pick up both slices of bread from the toaster and place them on the plate.` | |
| | Folder | Training stage | Config name (with `TASK=bread_gripstate`) | Use | | |
| |---|---|---|---| | |
| | `bread_gripstate_expert/timing-from40k-lr3e-4-step-9999/` | Timing head only, peak LR 3e-4, 10k steps, from Action step 39999 | `bread_gripstate_adaspe_expert_timing_from40k_lr3e-4` | **Deployment (recommended)** | | |
| | `bread_gripstate_expert/timing-from40k-step-9999/` | Timing head, default LR, from Action step 39999 | `bread_gripstate_adaspe_expert_timing_from40k` | Deployment, original timing LR | | |
| | `bread_gripstate_expert/timing-from30k-step-9999/` | Timing head, default LR, from Action step 30000 | `bread_gripstate_adaspe_expert_timing_from30k` | Ablation | | |
| | `bread_gripstate_expert/timing-from20k-step-9999/` | Timing head, default LR, from Action step 20000 | `bread_gripstate_adaspe_expert_timing_from20k` | Ablation | | |
| | `bread_gripstate_expert/action-step-{10000,20000,30000,39999}/` | Action stage (EMA params) at that step | `bread_gripstate_adaspe_expert_action` | Action generator / knot head only. The timing head is untrained. | | |
| Action loss (training, mean per 2k-step window; grasp windows are oversampled, so not directly comparable with | |
| `bread_expert`): | |
| | Steps | bread_gripstate expert | | |
| |---|---| | |
| | 0β2k | 0.348 | | |
| | 8β10k | 0.190 | | |
| | 18β20k | 0.136 | | |
| | 28β30k | 0.085 | | |
| | 38β40k | 0.076 | | |
| Timing loss (log-interval MSE, training, steps 9kβ10k): from20k 0.224, from30k 0.212, from40k 0.208, from40k lr3e-4 0.156. | |
| These are training losses. No held-out or real-robot evaluation results are included. | |
| ```python | |
| root = snapshot_download("cognaclee/Frank-Insert-Wipe", allow_patterns=["bread_gripstate_expert/timing-from40k-lr3e-4-step-9999/*"]) | |
| ckpt_dir = f"{root}/bread_gripstate_expert/timing-from40k-lr3e-4-step-9999" # TASK=bread_gripstate, config bread_gripstate_adaspe_expert_timing_from40k_lr3e-4 | |
| prompt = "Pick up both slices of bread from the toaster and place them on the plate." | |
| ``` | |