Pilot complete: 5 clean seeds, 3 seeds for desync and idle start, wrist-absent

README.md

@@ -30,30 +30,33 @@ the same VLA on every copy with the same recipe, and compare success in simulati
 
 ## Results (2026-10-11, 10,000 training steps)
 
-**Clean reference: 42.4%** success, mean of 3 seeds (43.8%, 42.8%, 40.6%; spread ±1.6 points).
-
-| Variant | Defect, applied to 50% of episodes | Success | Δ vs clean | Verdict |
-|---|---|---|---|---|
-| `stuck-actions-r50` | every action in the episode equals the first one | **5.2%** | **−37.2** | **breaks the policy** |
-| `desync-k3` | video 3 frames behind the actions (all episodes) | 38.0% | −4.4 | within noise; being re-tested |
-| `idle-start-r50` | 3 s of a still robot added before the episode | 39.2% | −3.2 | within noise; being re-tested |
-| `action-noise-s30` | Gaussian noise, 30% of each action dimension's std | 40.8% | −1.6 | no measurable effect |
-| `freeze-r50` | 2 s of frozen video mid-episode | 42.8% | +0.4 | no measurable effect |
-| `blur-r50` | two 2 s blurred windows | 46.2% | +3.8 | no measurable effect |
-| `wrist-absent-r50` | wrist camera black for the whole episode | pending | | |
-
-**How to read Δ.** Each defect has one training seed, so a difference smaller than about
-±5 points cannot be told apart from noise: the spread between clean seeds (±1.6) plus the
-sampling error of 500 episodes (±2.2).
+**Clean reference: 42.2%** success, mean of 5 seeds (43.8, 42.8, 40.6, 41.0, 42.6; spread ±1.3 points).
+
+| Variant | Defect, applied to 50% of episodes | Seeds | Success | Δ vs clean | Relative | Verdict |
+|---|---|---|---|---|---|---|
+| `stuck-actions-r50` | every action in the episode equals the first one | 1 | **5.2%** | **−37.0** | **−88%** | **breaks the policy** |
+| `desync-k3` | video 3 frames behind the actions (all episodes) | 3 | 38.3% | −3.8 ± 2.0 | −9% | probably a small effect, not conclusive |
+| `action-noise-s30` | Gaussian noise, 30% of each action dimension's std | 1 | 40.8% | −1.4 | −3% | no measurable effect |
+| `idle-start-r50` | 3 s of a still robot added before the episode | 3 | 41.9% | −0.3 ± 1.5 | −1% | no effect |
+| `freeze-r50` | 2 s of frozen video mid-episode | 1 | 42.8% | +0.6 | +1% | no measurable effect |
+| `wrist-absent-r50` | wrist camera black for the whole episode | 1 | 42.8% | +0.6 | +1% | no measurable effect at 50% |
+| `blur-r50` | two 2 s blurred windows | 1 | 46.2% | +4.0 | +9% | no measurable effect |
+
+**How to read Δ.** With one seed, a difference within about ±5 points is noise (spread between
+clean seeds plus the sampling error of 500 episodes). Desync and idle start were re-run with
+3 seeds each; the ± is the standard error of the difference.
 
 ## What this says so far
 
 1. **Stuck actions are severe.** When half the episodes record an action that never changes,
    SmolVLA's success drops from 42% to 5%. LeBoard already flags this (`static_actions`).
-2. **Frozen video, blur and moderate action noise did not measurably hurt** SmolVLA at these
-   levels, so a flag for them alone should not count against a dataset.
-3. **Desynchronised video and an idle start may cost a few points.** That is within noise with
-   one seed; more seeds are running to settle it.
+2. **An idle start does not hurt SmolVLA** (3 seeds: −0.3 points). The literature's large idle
+   effects are for single-step policies; SmolVLA predicts 50-action chunks.
+3. **Desynchronised video probably costs a little** (3 seeds: −3.8 ± 2.0 points, about −9%),
+   not yet conclusive.
+4. **Frozen video, blur, moderate action noise, and a wrist camera missing in half the episodes
+   did not measurably hurt** at these doses. The literature's large wrist-camera effect is for a
+   camera missing from all training data; that dose is next.
 
 ## Limits
 
@@ -84,13 +87,13 @@ replacement. Status: ✅ implemented and validated on injected defects · 🟡 p
 
 | Defect | What the literature says hurts | References | How LeBoard v2 measures it | v2 status | Our calibration (Δ vs clean) |
 |---|---|---|---|---|---|
-| **Static / no-op actions** | Wrong action labels are the most damaging training defect: an all-zero first action in Bridge made OpenVLA predict zeros and freeze; filtering no-ops was "crucial" | [OpenVLA](https://arxiv.org/abs/2406.09246) | `static_actions` (every action dimension constant for the episode), `static_action_dims` (single frozen dimensions), `zero_first_action` | ✅ recall 1.00, precision 1.00 | **−37.2** (stuck-actions-r50) |
-| **Idle starts and pauses** | Idle actions make policies get stuck, worst for single-step policies; teams filter idle frames before training | [Diffusion Policy](https://arxiv.org/abs/2303.04137), [openpi DROID recipe](https://github.com/Physical-Intelligence/openpi/blob/main/examples/droid/README_train.md), [DemInf](https://arxiv.org/abs/2502.08623), [FrameSkip](https://arxiv.org/abs/2605.13757) | `idle_start_s`, `idle_mid_s`, `idle_end_s`, `longest_idle_s` (robot state moves < 1% of its typical range over a 1 s window); v1 `idle_episodes` | ✅ idle start (recall 1.00); 🟡 mid-episode pauses measured but not yet validated | −3.2 (idle-start-r50, within noise; re-testing) |
-| **Video / action desync** | Timing errors cut success sharply at deployment: −30 points without latency matching; stale camera streams −65 to −77 points | [UMI](https://arxiv.org/abs/2402.10329), [LIBERO-VPro](https://arxiv.org/abs/2609.24350), [Diffusion Policy](https://arxiv.org/abs/2303.04137) | `per_camera.sync.lag_frames`: correlation of video motion with the best-tracking state dimension, pooled over episodes | ✅ dataset lag exact (1 and 3 frames); 🟡 per-episode lag noisy | −4.4 (desync-k3, within noise; re-testing) |
-| **Missing camera / dropouts** | Losing a view, the wrist view above all, is among the largest effects: −30 points without the wrist camera in training; −64 to −75 at test time | [robomimic study](https://robomimic.github.io/study/), [MINT](https://arxiv.org/abs/2609.35003), [LIBERO-VPro](https://arxiv.org/abs/2609.24350) | camera `absent` (every frame near-blank), `near_blank_frames` (dropouts) | ✅ absent camera (recall 1.00); 🟡 mid-episode dropouts not yet validated | pending (wrist-absent-r50) |
-| **Operator quality / inconsistent actions** | Demonstrations from less skilled or inconsistent operators lower success (robomimic Square BC 78.7% → 52.7%); action noise costs most on small datasets; smoothness and consistency predict success | [robomimic](https://arxiv.org/abs/2108.03298), [Belkhale et al.](https://arxiv.org/abs/2306.02437), [Sakr et al.](https://arxiv.org/abs/2412.14309), [RINSE](https://arxiv.org/abs/2604.23000), [PSD](https://arxiv.org/abs/2605.01544) | `jerk_rms`, `sparc`, `path_length` | 🟡 detects random action noise (AUROC 0.95–1.00); ❌ inconsistency between demonstrators (different strategies for the same state) not measured | −1.6 (random noise only, action-noise-s30) |
-| **Frozen frames** | A frozen frame paired with moving actions is a stale observation baked into training (same mechanism as test-time staleness above) | [LIBERO-VPro](https://arxiv.org/abs/2609.24350) (test-time staleness) | `freeze_frames`, `longest_freeze_s` (pixel-exact repeats ≥ 1 s) | ✅ recall 1.00, precision 1.00 | +0.4 (freeze-r50: 2 s in half the episodes) |
-| **Blur / darkness / lighting** | Mostly a robustness issue: lighting changes cost ~15 points at test time, while varied images in training act as augmentation and help generalization | [LIBERO-Plus](https://arxiv.org/abs/2510.13626), [SEVO](https://arxiv.org/abs/2605.11114) | `blur_frames`, `dark_frames` per camera (v1 rules) | ✅ blur (AUROC 1.00); 🟡 darkness not yet validated | +3.8 (blur-r50) |
+| **Static / no-op actions** | Wrong action labels are the most damaging training defect: an all-zero first action in Bridge made OpenVLA predict zeros and freeze; filtering no-ops was "crucial" | [OpenVLA](https://arxiv.org/abs/2406.09246) | `static_actions` (every action dimension constant for the episode), `static_action_dims` (single frozen dimensions), `zero_first_action` | ✅ recall 1.00, precision 1.00 | **−37.0** (stuck-actions-r50) |
+| **Idle starts and pauses** | Idle actions make policies get stuck, worst for single-step policies; teams filter idle frames before training | [Diffusion Policy](https://arxiv.org/abs/2303.04137), [openpi DROID recipe](https://github.com/Physical-Intelligence/openpi/blob/main/examples/droid/README_train.md), [DemInf](https://arxiv.org/abs/2502.08623), [FrameSkip](https://arxiv.org/abs/2605.13757) | `idle_start_s`, `idle_mid_s`, `idle_end_s`, `longest_idle_s` (robot state moves < 1% of its typical range over a 1 s window); v1 `idle_episodes` | ✅ idle start (recall 1.00); 🟡 mid-episode pauses measured but not yet validated | −0.3 ± 1.5 (idle-start-r50, 3 seeds: no effect) |
+| **Video / action desync** | Timing errors cut success sharply at deployment: −30 points without latency matching; stale camera streams −65 to −77 points | [UMI](https://arxiv.org/abs/2402.10329), [LIBERO-VPro](https://arxiv.org/abs/2609.24350), [Diffusion Policy](https://arxiv.org/abs/2303.04137) | `per_camera.sync.lag_frames`: correlation of video motion with the best-tracking state dimension, pooled over episodes | ✅ dataset lag exact (1 and 3 frames); 🟡 per-episode lag noisy | −3.8 ± 2.0 (desync-k3, 3 seeds: probably small, not conclusive) |
+| **Missing camera / dropouts** | Losing a view, the wrist view above all, is among the largest effects: −30 points without the wrist camera in training; −64 to −75 at test time | [robomimic study](https://robomimic.github.io/study/), [MINT](https://arxiv.org/abs/2609.35003), [LIBERO-VPro](https://arxiv.org/abs/2609.24350) | camera `absent` (every frame near-blank), `near_blank_frames` (dropouts) | ✅ absent camera (recall 1.00); 🟡 mid-episode dropouts not yet validated | +0.6 (wrist missing in 50% of episodes; literature: missing in all) |
+| **Operator quality / inconsistent actions** | Demonstrations from less skilled or inconsistent operators lower success (robomimic Square BC 78.7% → 52.7%); action noise costs most on small datasets; smoothness and consistency predict success | [robomimic](https://arxiv.org/abs/2108.03298), [Belkhale et al.](https://arxiv.org/abs/2306.02437), [Sakr et al.](https://arxiv.org/abs/2412.14309), [RINSE](https://arxiv.org/abs/2604.23000), [PSD](https://arxiv.org/abs/2605.01544) | `jerk_rms`, `sparc`, `path_length` | 🟡 detects random action noise (AUROC 0.95–1.00); ❌ inconsistency between demonstrators (different strategies for the same state) not measured | −1.4 (random noise only, action-noise-s30) |
+| **Frozen frames** | A frozen frame paired with moving actions is a stale observation baked into training (same mechanism as test-time staleness above) | [LIBERO-VPro](https://arxiv.org/abs/2609.24350) (test-time staleness) | `freeze_frames`, `longest_freeze_s` (pixel-exact repeats ≥ 1 s) | ✅ recall 1.00, precision 1.00 | +0.6 (freeze-r50: 2 s in half the episodes) |
+| **Blur / darkness / lighting** | Mostly a robustness issue: lighting changes cost ~15 points at test time, while varied images in training act as augmentation and help generalization | [LIBERO-Plus](https://arxiv.org/abs/2510.13626), [SEVO](https://arxiv.org/abs/2605.11114) | `blur_frames`, `dark_frames` per camera (v1 rules) | ✅ blur (AUROC 1.00); 🟡 darkness not yet validated | +4.0 (blur-r50) |
 | **Failed or bad-strategy demonstrations** | Removing failed or brittle-strategy demos gives the largest curation gains (+15 to +35 points; CUPID: 44% → 84% on one task) | [Demo-SCORE](https://arxiv.org/abs/2503.03707), [CUPID](https://arxiv.org/abs/2506.19121), [OpenVLA](https://arxiv.org/abs/2406.09246) | — (`length_ratio` is only a weak proxy) | ❌ needs success labels or policy rollouts | not tested |
 | **Low diversity / small size** | Generalization follows the number of environments and objects (power law); varied lighting and backgrounds was "the single most important factor"; too much diversity can hurt pretrained VLAs | [Lin et al.](https://arxiv.org/abs/2410.18647), [SEVO](https://arxiv.org/abs/2605.11114), [Geometric Entropy](https://arxiv.org/abs/2606.20871) | `episodes`, `hours` only | ❌ scene / object / task diversity not measured | not tested |
 | **Task text and camera-role metadata** | SmolVLA rewrote vague task strings and standardised camera names before pretraining on community data (51.7% → 78.3% with that data) | [SmolVLA](https://arxiv.org/abs/2506.01844) | — | ❌ not measured | not tested |