Frank-Insert-Wipe / README.md
cognaclee's picture
bread_gripstate_expert: bread retrain (gripper width only, grasp oversampling) + code, README, SHA256SUMS
4f263f6 verified
|
Raw History Blame Contribute Delete
28.3 kB
metadata
tags:
  - robotics
  - openpi
  - pi0.5
  - franka
library_name: openpi

Frank-Insert-Wipe: AdaSpeVLA checkpoints for Franka insert_tube, wipe_whiteboard, bread, test_tube, tennis_balls and a 5-task co-trained model (H=16)

AdaSpeVLA is a pi0.5 (OpenPI, JAX) variant. It predicts cubic B-spline action chunks with adaptive knots, plus a timing head that decides when to query the policy again. This repo holds two variants that differ only in the generator that produces the 11 control points:

  • flow_head (<task>/, 2026-11-12): a 20.4 M FlowPolicy trained from scratch.
  • pi05_expert (<task>_expert/, 2026-11-14): pi0.5's pretrained Gemma-300M action expert (427.9 M with adaRMS, from pi05_base) runs on the 11 control-point tokens, as pi0.5 does on its action tokens.

Everything else (data, norm stats, H=16, K=11, labels, knot head, timing head, 40k + 10k steps) is the same. The training code for both is in code/.

These are params-only Orbax checkpoints in the OpenPI layout. The optimizer train_state is not included.

Folder Training stage Config name (with TASK=<task>) Use
<task>/timing-from40k-step-9999/ Timing head, 10k steps, from Action step 39999 (everything except freq_head frozen) <task>_adaspe_timing_from40k Deployment: full model with the trained timing head
<task>/action-step-39999/ Action stage, 40k steps (EMA params) <task>_adaspe_action Action generator / knot head only. Its timing head is untrained.
<task>_expert/timing-from40k-step-9999/ Same, pi05_expert generator <task>_adaspe_expert_timing_from40k Deployment (expert variant)
<task>_expert/action-step-39999/ Same, pi05_expert generator <task>_adaspe_expert_action Action generator / knot head only (expert variant)
<task>/timing-from40k-lr3e-4-step-9999/ Timing head only, retrained 2026-11-14 with peak LR 3e-4 (same 10k steps; see below) <task>_adaspe_timing_from40k_lr3e-4 Deployment candidate
<task>/timing-from40k-lr1e-3-step-9999/ Timing head only, retrained 2026-11-14 with peak LR 1e-3 (same 10k steps; see below) <task>_adaspe_timing_from40k_lr1e-3 Deployment candidate
<task>_expert/timing-from40k-lr3e-4-step-9999/ Timing head only, retrained 2026-11-14 with peak LR 3e-4 (same 10k steps; see below) <task>_adaspe_expert_timing_from40k_lr3e-4 Deployment candidate (expert variant)
<task>_expert/timing-from40k-lr1e-3-step-9999/ Timing head only, retrained 2026-11-14 with peak LR 1e-3 (same 10k steps; see below) <task>_adaspe_expert_timing_from40k_lr1e-3 Deployment candidate (expert variant)

<task> is insert_tube or wipe_whiteboard. Each folder has params/, assets/dddddsds31341/<dataset>/norm_stats.json (the normalization stats used in training) and _CHECKPOINT_METADATA. SHA256SUMS lists every file.

Loading

You need the AdaSpeVLA OpenPI plugin, included here as code/adaspe_train (run with TASK=<task>; it registers these config names). It runs on OpenPI at the commit in code/OPENPI_COMMIT. Setup is in code/README.md, and code/adaspe_train/EXPERT_VARIANT.md documents the expert variant. The folders use the standard OpenPI checkpoint layout (params/ + assets/).

from huggingface_hub import snapshot_download
root = snapshot_download("cognaclee/Frank-Insert-Wipe", allow_patterns=["insert_tube/timing-from40k-step-9999/*"])
# higher-LR timing head: "insert_tube/timing-from40k-lr3e-4-step-9999/*" with config insert_tube_adaspe_timing_from40k_lr3e-4
ckpt_dir = f"{root}/insert_tube/timing-from40k-step-9999"   # with config insert_tube_adaspe_timing_from40k

Data

Task Dataset (revision) Size pi0.5 baseline (same norm stats)
insert_tube dddddsds31341/insert-tube-lerobot (70bcd1f86ed35fc93a0422a564472355242a51ab) 102 episodes / 16,403 frames dddddsds31341/Franka-Insert-Tube-pi05-step-20000
wipe_whiteboard dddddsds31341/wipe-whiteboard-lerobot (2c8cfa8ddef1f83cb09ca7881a0944cd637cd6d0) 112 episodes / 29,795 frames dddddsds31341/Franka-Wipe-Whiteboard-pi05-step-20000

Two 128Γ—128 cameras, 19-D state, 7-D action, 10 Hz.

Recipe

  • Action horizon H=16 (1.5 s), 11 cubic B-spline control points, query interval 0.2–1.5 s.
  • Gripper events for the timing and knot labels come from the gripper command (the measured width lags by about 1.9 s).
  • Knot density is tanh(motion_variation / e_crit), with e_crit calibrated per task at H=16: insert_tube 11.98, wipe_whiteboard 8.96.
  • Initialized from pi05_base. Batch 32, Action 40k steps (LR decay over 30k), then timing 10k steps.

Training loss

Mean training loss over the last 1k–2k steps of each stage.

Task Variant Action loss, steps 38k–40k Timing loss (log-interval MSE), steps 9k–10k
insert_tube flow_head 0.086 0.206
insert_tube pi05_expert 0.030 0.255
wipe_whiteboard flow_head 0.118 0.196
wipe_whiteboard pi05_expert 0.040 0.271

Action loss over training (mean per 2k-step window):

Steps insert_tube flow_head insert_tube expert wipe flow_head wipe expert
0–2k 0.624 0.334 0.597 0.295
8–10k 0.238 0.121 0.238 0.121
18–20k 0.156 0.067 0.169 0.083
28–30k 0.105 0.038 0.130 0.049
38–40k 0.086 0.030 0.118 0.040

Notes:

  • The two variants use different flow-matching conventions (flow_head: t ~ U(0,1); pi05_expert: pi0.5's Beta(1.5,1)), so the action losses are not exactly the same metric.
  • The expert's timing head has a higher loss. It reads a pooled observation feature that, in the expert variant, is trained only by the knot loss (see code/adaspe_train/EXPERT_VARIANT.md).
  • These are training losses. No held-out or real-robot evaluation results are included here.

Timing head retrained with a higher learning rate (2026-11-14)

The original timing heads (timing-from40k-step-9999) used OpenPI's default schedule: peak LR 2.5e-5 with a cosine decay over 30k steps. The timing stage only runs 10k steps, so the LR stayed at about 2.0–2.5e-5, which is low for freq_head, a freshly initialized 2-layer MLP (633 β†’ 128 β†’ 1, ~81k params) and the only trainable part. The loss was still falling at 10k.

The timing-from40k-lr<LR>-step-9999 folders retrain only freq_head, with the same data, labels, 10k steps, batch 32 and 1k warmup, a cosine decay to LR/10 over the 10k steps, and peak LR 3e-4 or 1e-3. They start from the same Action step-39999 params, and everything except freq_head is frozen. The predicted actions, knots and B-spline are therefore identical to timing-from40k-step-9999. Only the query interval (and so execute_steps) changes.

Timing loss (log-interval MSE, training loss, mean over steps 9k–10k):

Task Variant LR 2.5e-5 (original) LR 3e-4 LR 1e-3
insert_tube flow_head 0.206 0.157 0.091
insert_tube pi05_expert 0.255 0.182 0.100
wipe_whiteboard flow_head 0.196 0.158 0.112
wipe_whiteboard pi05_expert 0.271 0.214 0.153

These are training losses. With the higher LR, part of the drop may come from fitting the training frames; a held-out comparison has not been done yet. The configs are configs.TIMINGS_LR in code/adaspe_train/configs.py, and code/adaspe_train/slurm/timing_lr.slurm runs one of them.

Bread (2026-11-15, pi05_expert only)

bread_expert/ holds the bread task trained with the same H=16 recipe and the pi05_expert generator. There is no flow_head version of bread. Unlike the other tasks, every saved checkpoint is included (params only):

Folder Training stage Config name (with TASK=bread) Use
bread_expert/timing-from40k-lr3e-4-step-9999/ Timing head only, peak LR 3e-4, 10k steps, from Action step 39999 bread_adaspe_expert_timing_from40k_lr3e-4 Deployment (recommended)
bread_expert/timing-from40k-step-9999/ Timing head, default LR (peak 2.5e-5), from Action step 39999 bread_adaspe_expert_timing_from40k Deployment, original timing LR
bread_expert/timing-from30k-step-9999/ Timing head, default LR, from Action step 30000 bread_adaspe_expert_timing_from30k Ablation
bread_expert/timing-from20k-step-9999/ Timing head, default LR, from Action step 20000 bread_adaspe_expert_timing_from20k Ablation
bread_expert/action-step-{10000,20000,30000,39999}/ Action stage (EMA params) at that step bread_adaspe_expert_action Action generator / knot head only. The timing head is untrained.

Every timing folder is a full model. Only freq_head was trained in the timing stage, so actions and knots equal those of the Action checkpoint it starts from.

Data: dddddsds31341/bread-lerobot (revision a45b8ba49fc164fb05d8a9c8bdf871ff97004357), 100 episodes / 28,380 frames, 10 Hz. e_crit = 10.01 (calibrated at H=16). Training: batch 32, Action 40k steps (OpenPI default LR: 1k warmup, peak 2.5e-5, cosine decay to 2.5e-6 at 30k), then the timing stage (10k steps).

Action loss (training, mean per 2k-step window):

Steps bread expert
0–2k 0.357
8–10k 0.161
18–20k 0.114
28–30k 0.093
38–40k 0.083

Timing loss (log-interval MSE, training, steps 9k–10k): from20k 0.217, from30k 0.217, from40k 0.212, from40k lr3e-4 0.159.

These are training losses. No held-out or real-robot evaluation results are included. code/adaspe_train has the bread settings (tasks.py, with checkpoints saved every 10k steps).

root = snapshot_download("cognaclee/Frank-Insert-Wipe", allow_patterns=["bread_expert/timing-from40k-lr3e-4-step-9999/*"])
ckpt_dir = f"{root}/bread_expert/timing-from40k-lr3e-4-step-9999"   # TASK=bread, config bread_adaspe_expert_timing_from40k_lr3e-4

Five-task co-training (2026-11-15, pi05_expert only)

cotrain5_expert/ is one model for all five tasks of the dddddsds31341/franka-data collection, trained with the same H=16 recipe and the pi05_expert generator. The task is chosen by the prompt. Use the dataset's task string exactly:

Task Dataset Revision Episodes Frames Share of training Prompt
bread dddddsds31341/bread-lerobot a45b8ba49fc1 100 28,380 19.3% Pick up both slices of bread from the toaster and place them on the plate.
insert_tube dddddsds31341/insert-tube-lerobot 70bcd1f86ed3 102 16,403 11.2% Insert the gray tube into the empty black holder.
wipe_whiteboard dddddsds31341/wipe-whiteboard-lerobot 2c8cfa8ddef1 112 29,795 20.3% Erase the writing from the whiteboard.
tennis_balls dddddsds31341/tennis-balls-lerobot 04a27cd5ffe2 100 36,950 25.1% Place all tennis balls into the green basket.
test_tube dddddsds31341/test-tube-lerobot 29d821a1a711 99 35,425 24.1% Place the test tube into the test tube rack.

The five datasets are merged into one local LeRobot v2.1 dataset (code/adaspe_train/merge_datasets.py; record in cotrain5_expert/merge_provenance.json): 513 episodes, 146,953 frames, 10 Hz. Batches are sampled uniformly over frames (no per-task reweighting), so each task's share equals its share of frames.

Differences from the single-task models:

  • Norm stats were computed on the merged data. They are in each folder at assets/local/franka-cotrain5/norm_stats.json. The per-task stats do not apply to this model.
  • One set of knot/timing constants for all tasks, calibrated on the merged data (cotrain5_expert/calibration_cotrain5_H16.json): e_crit = 9.216; timing labels 0.2 s 10.9%, 1.5 s 57.9%.
  • Same step budget as one single-task run (Action 40k steps, batch 32), so each frame is seen about 8.7 times (single-task runs see each frame about 45 times).
Folder Training stage Config name (with TASK=cotrain5) Use
cotrain5_expert/timing-from40k-lr3e-4-step-9999/ Timing head only, peak LR 3e-4, 10k steps, from Action step 39999 cotrain5_adaspe_expert_timing_from40k_lr3e-4 Deployment (recommended)
cotrain5_expert/timing-from40k-step-9999/ Timing head, default LR (peak 2.5e-5), from Action step 39999 cotrain5_adaspe_expert_timing_from40k Deployment, original timing LR
cotrain5_expert/timing-from30k-step-9999/ Timing head, default LR, from Action step 30000 cotrain5_adaspe_expert_timing_from30k Ablation
cotrain5_expert/timing-from20k-step-9999/ Timing head, default LR, from Action step 20000 cotrain5_adaspe_expert_timing_from20k Ablation
cotrain5_expert/action-step-{10000,20000,30000,39999}/ Action stage (EMA params) at that step cotrain5_adaspe_expert_action Action generator / knot head only. The timing head is untrained.

Every timing folder is a full model. Only freq_head was trained in the timing stage, so actions and knots equal those of the Action checkpoint it starts from.

Action loss (training, all five tasks mixed, mean per 2k-step window):

Steps cotrain5 expert
0–2k 0.345
8–10k 0.192
18–20k 0.146
28–30k 0.123
38–40k 0.110

Timing loss (log-interval MSE, training, steps 9k–10k): from20k 0.287, from30k 0.269, from40k 0.267, from40k lr3e-4 0.232.

These are training losses. No held-out or real-robot evaluation results are included. code/adaspe_train now has the cotrain5 task (tasks.py) and merge_datasets.py. To retrain, download the five datasets, run TASK=cotrain5 python adaspe_train/merge_datasets.py, then train with TASK=cotrain5 VARIANT=expert.

root = snapshot_download("cognaclee/Frank-Insert-Wipe", allow_patterns=["cotrain5_expert/timing-from40k-lr3e-4-step-9999/*"])
ckpt_dir = f"{root}/cotrain5_expert/timing-from40k-lr3e-4-step-9999"   # TASK=cotrain5, config cotrain5_adaspe_expert_timing_from40k_lr3e-4
prompt = "Place all tennis balls into the green basket."                 # one of the five task strings above

Test tube (2026-11-16, pi05_expert only)

test_tube_expert/ holds a single-task model for the test-tube task, trained with the same H=16 recipe and the pi05_expert generator. (cotrain5_expert/ also covers this task, but it is shared with four other tasks.) There is no flow_head version. Every saved checkpoint is included (params only):

Folder Training stage Config name (with TASK=test_tube) Use
test_tube_expert/timing-from40k-lr3e-4-step-9999/ Timing head only, peak LR 3e-4, 10k steps, from Action step 39999 test_tube_adaspe_expert_timing_from40k_lr3e-4 Deployment (recommended)
test_tube_expert/timing-from40k-step-9999/ Timing head, default LR (peak 2.5e-5), from Action step 39999 test_tube_adaspe_expert_timing_from40k Deployment, original timing LR
test_tube_expert/timing-from30k-step-9999/ Timing head, default LR, from Action step 30000 test_tube_adaspe_expert_timing_from30k Ablation
test_tube_expert/timing-from20k-step-9999/ Timing head, default LR, from Action step 20000 test_tube_adaspe_expert_timing_from20k Ablation
test_tube_expert/action-step-{10000,20000,30000,39999}/ Action stage (EMA params) at that step test_tube_adaspe_expert_action Action generator / knot head only. The timing head is untrained.

Every timing folder is a full model. Only freq_head was trained in the timing stage, so actions and knots equal those of the Action checkpoint it starts from.

Data: dddddsds31341/test-tube-lerobot (revision 29d821a1a71180948773e29c791f906306ac8018), 99 episodes / 35,425 frames, 10 Hz. Prompt (the dataset's task string): Place the test tube into the test tube rack.

  • Norm stats were computed on this dataset (there is no baseline-repo copy). They are in each folder at assets/dddddsds31341/test-tube-lerobot/norm_stats.json, and also in code/adaspe_train/assets/pi05_test_tube_franka_ref/.
  • Knot/timing constants were calibrated on this dataset at H=16 (test_tube_expert/calibration_test_tube_H16.json): e_crit = 9.930; timing labels 0.2 s 10.6%, 1.5 s 58.0%.
  • Training: batch 32, Action 40k steps (OpenPI default LR: 1k warmup, peak 2.5e-5, cosine decay to 2.5e-6 at 30k), then the timing stage (10k steps).

Action loss (training, mean per 2k-step window):

Steps test_tube expert
0–2k 0.328
8–10k 0.151
18–20k 0.102
28–30k 0.065
38–40k 0.056

Timing loss (log-interval MSE, training, steps 9k–10k): from20k 0.223, from30k 0.217, from40k 0.216, from40k lr3e-4 0.171.

These are training losses. No held-out or real-robot evaluation results are included. code/adaspe_train has the test_tube settings (tasks.py) and its calibration, timing-label and norm-stats files.

root = snapshot_download("cognaclee/Frank-Insert-Wipe", allow_patterns=["test_tube_expert/timing-from40k-lr3e-4-step-9999/*"])
ckpt_dir = f"{root}/test_tube_expert/timing-from40k-lr3e-4-step-9999"   # TASK=test_tube, config test_tube_adaspe_expert_timing_from40k_lr3e-4
prompt = "Place the test tube into the test tube rack."

Tennis balls (2026-11-17, pi05_expert only)

tennis_balls_expert/ holds a single-task model for the tennis-balls task, trained with the same H=16 recipe and the pi05_expert generator. (cotrain5_expert/ also covers this task, but it is shared with four other tasks.) There is no flow_head version. Every saved checkpoint is included (params only):

Folder Training stage Config name (with TASK=tennis_balls) Use
tennis_balls_expert/timing-from40k-lr3e-4-step-9999/ Timing head only, peak LR 3e-4, 10k steps, from Action step 39999 tennis_balls_adaspe_expert_timing_from40k_lr3e-4 Deployment (recommended)
tennis_balls_expert/timing-from40k-step-9999/ Timing head, default LR (peak 2.5e-5), from Action step 39999 tennis_balls_adaspe_expert_timing_from40k Deployment, original timing LR
tennis_balls_expert/timing-from30k-step-9999/ Timing head, default LR, from Action step 30000 tennis_balls_adaspe_expert_timing_from30k Ablation
tennis_balls_expert/timing-from20k-step-9999/ Timing head, default LR, from Action step 20000 tennis_balls_adaspe_expert_timing_from20k Ablation
tennis_balls_expert/action-step-{10000,20000,30000,39999}/ Action stage (EMA params) at that step tennis_balls_adaspe_expert_action Action generator / knot head only. The timing head is untrained.

Every timing folder is a full model. Only freq_head was trained in the timing stage, so actions and knots equal those of the Action checkpoint it starts from.

Data: dddddsds31341/tennis-balls-lerobot (revision 04a27cd5ffe22c250f0a23966256dc646b724a98), 100 episodes / 36,950 frames, 10 Hz. Prompt (the dataset's task string): Place all tennis balls into the green basket.

  • Norm stats were computed on this dataset (there is no baseline-repo copy). They are in each folder at assets/dddddsds31341/tennis-balls-lerobot/norm_stats.json, and also in code/adaspe_train/assets/pi05_tennis_balls_franka_ref/.
  • Knot/timing constants were calibrated on this dataset at H=16 (tennis_balls_expert/calibration_tennis_balls_H16.json): e_crit = 8.338; timing labels 0.2 s 12.0%, 1.5 s 49.2%.
  • Training: batch 32, Action 40k steps (OpenPI default LR: 1k warmup, peak 2.5e-5, cosine decay to 2.5e-6 at 30k), then the timing stage (10k steps).

Action loss (training, mean per 2k-step window):

Steps tennis_balls expert
0–2k 0.404
8–10k 0.182
18–20k 0.126
28–30k 0.084
38–40k 0.071

Timing loss (log-interval MSE, training, steps 9k–10k): from20k 0.295, from30k 0.290, from40k 0.288, from40k lr3e-4 0.231.

These are training losses. No held-out or real-robot evaluation results are included. code/adaspe_train has the tennis_balls settings (tasks.py) and its calibration, timing-label and norm-stats files.

root = snapshot_download("cognaclee/Frank-Insert-Wipe", allow_patterns=["tennis_balls_expert/timing-from40k-lr3e-4-step-9999/*"])
ckpt_dir = f"{root}/tennis_balls_expert/timing-from40k-lr3e-4-step-9999"   # TASK=tennis_balls, config tennis_balls_adaspe_expert_timing_from40k_lr3e-4
prompt = "Place all tennis balls into the green basket."

Bread retrain: no state (2026-11-19, pi05_expert only)

bread_nostate_expert/ is a retrain of the bread task on the same data as bread_expert/, with two recipe changes made after a real-robot test of bread_expert. In that test the first close landed on air about one slice-height too high, the policy still carried the empty gripper to the plate and opened it, and then closed around both slices (still in the toaster) at the slice-2 position. pi0.5 has no memory, and the 19-D state in its prompt (gripper width, force, end-effector pose, velocity) lets it tell the task phase from the arm pose rather than from the images.

  1. State input (no state): state_keep=(): the whole 19-D state is zeroed after normalization, so the prompt carries no proprioception and the policy must read the task phase from the two cameras. The mask (configs.MaskState) is the first model transform, after normalization, so a policy built from this config (TASK=bread_nostate) applies it at inference as well. If your client builds the model input another way, apply the same mask.
  2. Grasp-window oversampling (Action stage only): frames from 1.5 s before to 2 s after each gripper close command (6,700 of 28,380 frames, 200 closes) are sampled 3Γ— as often, so they make up 48% of the samples (code/adaspe_train/adaspe/oversample.py, index code/adaspe_train/assets/sample_index/bread_grasp_x3.npy). The timing stage samples uniformly.

Everything else is as in bread_expert/: data dddddsds31341/bread-lerobot (revision a45b8ba49fc164fb05d8a9c8bdf871ff97004357), 100 episodes / 28,380 frames, same norm stats (assets/dddddsds31341/bread-lerobot/norm_stats.json in each folder), same calibration (e_crit = 10.008, bread_nostate_expert/calibration_bread_nostate_H16.json), H=16, K=11, batch 32, Action 40k steps, timing 10k steps. Prompt: Pick up both slices of bread from the toaster and place them on the plate.

Folder Training stage Config name (with TASK=bread_nostate) Use
bread_nostate_expert/timing-from40k-lr3e-4-step-9999/ Timing head only, peak LR 3e-4, 10k steps, from Action step 39999 bread_nostate_adaspe_expert_timing_from40k_lr3e-4 Deployment (recommended)
bread_nostate_expert/timing-from40k-step-9999/ Timing head, default LR, from Action step 39999 bread_nostate_adaspe_expert_timing_from40k Deployment, original timing LR
bread_nostate_expert/timing-from30k-step-9999/ Timing head, default LR, from Action step 30000 bread_nostate_adaspe_expert_timing_from30k Ablation
bread_nostate_expert/timing-from20k-step-9999/ Timing head, default LR, from Action step 20000 bread_nostate_adaspe_expert_timing_from20k Ablation
bread_nostate_expert/action-step-{10000,20000,30000,39999}/ Action stage (EMA params) at that step bread_nostate_adaspe_expert_action Action generator / knot head only. The timing head is untrained.

Action loss (training, mean per 2k-step window; grasp windows are oversampled, so not directly comparable with bread_expert):

Steps bread_nostate expert
0–2k 0.329
8–10k 0.191
18–20k 0.138
28–30k 0.097
38–40k 0.090

Timing loss (log-interval MSE, training, steps 9k–10k): from20k 0.224, from30k 0.213, from40k 0.206, from40k lr3e-4 0.158.

These are training losses. No held-out or real-robot evaluation results are included.

root = snapshot_download("cognaclee/Frank-Insert-Wipe", allow_patterns=["bread_nostate_expert/timing-from40k-lr3e-4-step-9999/*"])
ckpt_dir = f"{root}/bread_nostate_expert/timing-from40k-lr3e-4-step-9999"   # TASK=bread_nostate, config bread_nostate_adaspe_expert_timing_from40k_lr3e-4
prompt = "Pick up both slices of bread from the toaster and place them on the plate."

Bread retrain: gripper width only (2026-11-19, pi05_expert only)

bread_gripstate_expert/ is a retrain of the bread task on the same data as bread_expert/, with two recipe changes made after a real-robot test of bread_expert. In that test the first close landed on air about one slice-height too high, the policy still carried the empty gripper to the plate and opened it, and then closed around both slices (still in the toaster) at the slice-2 position. pi0.5 has no memory, and the 19-D state in its prompt (gripper width, force, end-effector pose, velocity) lets it tell the task phase from the arm pose rather than from the images.

  1. State input (gripper width only): state_keep=(0,): only state[0] (gripper width) is kept; the other 18 dims are zeroed after normalization. The mask (configs.MaskState) is the first model transform, after normalization, so a policy built from this config (TASK=bread_gripstate) applies it at inference as well. If your client builds the model input another way, apply the same mask.
  2. Grasp-window oversampling (Action stage only): frames from 1.5 s before to 2 s after each gripper close command (6,700 of 28,380 frames, 200 closes) are sampled 3Γ— as often, so they make up 48% of the samples (code/adaspe_train/adaspe/oversample.py, index code/adaspe_train/assets/sample_index/bread_grasp_x3.npy). The timing stage samples uniformly.

Everything else is as in bread_expert/: data dddddsds31341/bread-lerobot (revision a45b8ba49fc164fb05d8a9c8bdf871ff97004357), 100 episodes / 28,380 frames, same norm stats (assets/dddddsds31341/bread-lerobot/norm_stats.json in each folder), same calibration (e_crit = 10.008, bread_gripstate_expert/calibration_bread_gripstate_H16.json), H=16, K=11, batch 32, Action 40k steps, timing 10k steps. Prompt: Pick up both slices of bread from the toaster and place them on the plate.

Folder Training stage Config name (with TASK=bread_gripstate) Use
bread_gripstate_expert/timing-from40k-lr3e-4-step-9999/ Timing head only, peak LR 3e-4, 10k steps, from Action step 39999 bread_gripstate_adaspe_expert_timing_from40k_lr3e-4 Deployment (recommended)
bread_gripstate_expert/timing-from40k-step-9999/ Timing head, default LR, from Action step 39999 bread_gripstate_adaspe_expert_timing_from40k Deployment, original timing LR
bread_gripstate_expert/timing-from30k-step-9999/ Timing head, default LR, from Action step 30000 bread_gripstate_adaspe_expert_timing_from30k Ablation
bread_gripstate_expert/timing-from20k-step-9999/ Timing head, default LR, from Action step 20000 bread_gripstate_adaspe_expert_timing_from20k Ablation
bread_gripstate_expert/action-step-{10000,20000,30000,39999}/ Action stage (EMA params) at that step bread_gripstate_adaspe_expert_action Action generator / knot head only. The timing head is untrained.

Action loss (training, mean per 2k-step window; grasp windows are oversampled, so not directly comparable with bread_expert):

Steps bread_gripstate expert
0–2k 0.348
8–10k 0.190
18–20k 0.136
28–30k 0.085
38–40k 0.076

Timing loss (log-interval MSE, training, steps 9k–10k): from20k 0.224, from30k 0.212, from40k 0.208, from40k lr3e-4 0.156.

These are training losses. No held-out or real-robot evaluation results are included.

root = snapshot_download("cognaclee/Frank-Insert-Wipe", allow_patterns=["bread_gripstate_expert/timing-from40k-lr3e-4-step-9999/*"])
ckpt_dir = f"{root}/bread_gripstate_expert/timing-from40k-lr3e-4-step-9999"   # TASK=bread_gripstate, config bread_gripstate_adaspe_expert_timing_from40k_lr3e-4
prompt = "Pick up both slices of bread from the toaster and place them on the plate."