Add defect reference table (literature, LeBoard v2 metric, status)

README.md

@@ -73,3 +73,32 @@ sampling error of 500 episodes (±2.2).
   one folder per run with `run.json` (recipe, pinned data revision, success per checkpoint),
   training and evaluation logs, `eval_info.json` per task, evaluation videos and checkpoints.
 - Episode tables: [`rogation/leboard-episode-tables`](https://huggingface.co/datasets/rogation/leboard-episode-tables).
+
+## Defect reference: what hurts, how LeBoard measures it, and the evidence
+
+The weight a LeBoard flag should carry comes from the **literature first**; our own calibration
+runs (SmolVLA, LIBERO-10, one seed, defect in 50% of episodes) are one extra data point, not a
+replacement. Status: ✅ implemented and validated on injected defects · 🟡 partial · ❌ not measured.
+
+| Defect | What the literature says hurts | References | How LeBoard v2 measures it | v2 status | Our calibration (Δ vs clean) |
+|---|---|---|---|---|---|
+| **Static / no-op actions** | Wrong action labels are the most damaging training defect: an all-zero first action in Bridge made OpenVLA predict zeros and freeze; filtering no-ops was "crucial" | [OpenVLA](https://arxiv.org/abs/2406.09246) | `static_actions` (every action dimension constant for the episode), `static_action_dims` (single frozen dimensions), `zero_first_action` | ✅ recall 1.00, precision 1.00 | **−37.2** (stuck-actions-r50) |
+| **Idle starts and pauses** | Idle actions make policies get stuck, worst for single-step policies; teams filter idle frames before training | [Diffusion Policy](https://arxiv.org/abs/2303.04137), [openpi DROID recipe](https://github.com/Physical-Intelligence/openpi/blob/main/examples/droid/README_train.md), [DemInf](https://arxiv.org/abs/2502.08623), [FrameSkip](https://arxiv.org/abs/2605.13757) | `idle_start_s`, `idle_mid_s`, `idle_end_s`, `longest_idle_s` (robot state moves < 1% of its typical range over a 1 s window); v1 `idle_episodes` | ✅ idle start (recall 1.00); 🟡 mid-episode pauses measured but not yet validated | −3.2 (idle-start-r50, within noise; re-testing) |
+| **Video / action desync** | Timing errors cut success sharply at deployment: −30 points without latency matching; stale camera streams −65 to −77 points | [UMI](https://arxiv.org/abs/2402.10329), [LIBERO-VPro](https://arxiv.org/abs/2609.24350), [Diffusion Policy](https://arxiv.org/abs/2303.04137) | `per_camera.sync.lag_frames`: correlation of video motion with the best-tracking state dimension, pooled over episodes | ✅ dataset lag exact (1 and 3 frames); 🟡 per-episode lag noisy | −4.4 (desync-k3, within noise; re-testing) |
+| **Missing camera / dropouts** | Losing a view, the wrist view above all, is among the largest effects: −30 points without the wrist camera in training; −64 to −75 at test time | [robomimic study](https://robomimic.github.io/study/), [MINT](https://arxiv.org/abs/2609.35003), [LIBERO-VPro](https://arxiv.org/abs/2609.24350) | camera `absent` (every frame near-blank), `near_blank_frames` (dropouts) | ✅ absent camera (recall 1.00); 🟡 mid-episode dropouts not yet validated | pending (wrist-absent-r50) |
+| **Operator quality / inconsistent actions** | Demonstrations from less skilled or inconsistent operators lower success (robomimic Square BC 78.7% → 52.7%); action noise costs most on small datasets; smoothness and consistency predict success | [robomimic](https://arxiv.org/abs/2108.03298), [Belkhale et al.](https://arxiv.org/abs/2306.02437), [Sakr et al.](https://arxiv.org/abs/2412.14309), [RINSE](https://arxiv.org/abs/2604.23000), [PSD](https://arxiv.org/abs/2605.01544) | `jerk_rms`, `sparc`, `path_length` | 🟡 detects random action noise (AUROC 0.95–1.00); ❌ inconsistency between demonstrators (different strategies for the same state) not measured | −1.6 (random noise only, action-noise-s30) |
+| **Frozen frames** | A frozen frame paired with moving actions is a stale observation baked into training (same mechanism as test-time staleness above) | [LIBERO-VPro](https://arxiv.org/abs/2609.24350) (test-time staleness) | `freeze_frames`, `longest_freeze_s` (pixel-exact repeats ≥ 1 s) | ✅ recall 1.00, precision 1.00 | +0.4 (freeze-r50: 2 s in half the episodes) |
+| **Blur / darkness / lighting** | Mostly a robustness issue: lighting changes cost ~15 points at test time, while varied images in training act as augmentation and help generalization | [LIBERO-Plus](https://arxiv.org/abs/2510.13626), [SEVO](https://arxiv.org/abs/2605.11114) | `blur_frames`, `dark_frames` per camera (v1 rules) | ✅ blur (AUROC 1.00); 🟡 darkness not yet validated | +3.8 (blur-r50) |
+| **Failed or bad-strategy demonstrations** | Removing failed or brittle-strategy demos gives the largest curation gains (+15 to +35 points; CUPID: 44% → 84% on one task) | [Demo-SCORE](https://arxiv.org/abs/2503.03707), [CUPID](https://arxiv.org/abs/2506.19121), [OpenVLA](https://arxiv.org/abs/2406.09246) | — (`length_ratio` is only a weak proxy) | ❌ needs success labels or policy rollouts | not tested |
+| **Low diversity / small size** | Generalization follows the number of environments and objects (power law); varied lighting and backgrounds was "the single most important factor"; too much diversity can hurt pretrained VLAs | [Lin et al.](https://arxiv.org/abs/2410.18647), [SEVO](https://arxiv.org/abs/2605.11114), [Geometric Entropy](https://arxiv.org/abs/2606.20871) | `episodes`, `hours` only | ❌ scene / object / task diversity not measured | not tested |
+| **Task text and camera-role metadata** | SmolVLA rewrote vague task strings and standardised camera names before pretraining on community data (51.7% → 78.3% with that data) | [SmolVLA](https://arxiv.org/abs/2506.01844) | — | ❌ not measured | not tested |
+| **Constant state dimensions** | A dimension that never moves gets ~0 std, so tiny deviations at deployment become huge normalised inputs | [TheWisp/lerobot PR #233](https://github.com/TheWisp/lerobot/pull/233) | `static_state_dims` in the summary | ✅ reported | not tested |
+| **Metadata vs video duration** | No published training-time measurement | — | v1 metadata/video duration check | ✅ (not injected) | not tested |
+
+**Reading the last column.** Our calibration has one seed per defect: a gap within about ±5
+points is noise. Where it differs from the literature (random action noise, frozen frames,
+desync, idle start), the setups differ too: SmolVLA predicts 50-action chunks (the literature's
+large idle effects are for single-step policies), the timing results in the literature are
+measured at test time rather than in training data, our action noise is random rather than an
+operator's structured mistakes, and 2 s of freeze in half the episodes is only ~7% of frames.
+Until more policies, datasets and doses are tested, the literature's ranking is the default.