segment_eval/README.md

# Segment-level physics violations — human-rating examples

Nine annotated clips (3 per robot/view group) showing **which time segment of a
generated video violates physics**, so a human rater can agree or disagree with
the judge segment by segment instead of on a whole-clip percentage.

Each mp4 is: **PRED** (Ctrl-World / DreamDojo generated) on top, **GT** (the real
video it was conditioned on) below, and a bar at the bottom with one cell per
segment. The playing segment carries a white border; `Y` = violating, `n` = not.

## The rule

| | statistic per segment | why |
|---|---|---|
| **rigidity** | **MEDIAN** of the per-frame score | a warped limb is a *persistent state* — the median says "the arm was wrong for most of this segment", and one bad frame cannot fake it |
| **jerk** | **MAX** of the per-frame score | a jolt is an *impulsive event* — one frame is the whole thing, so averaging it away would be wrong |

**Segmentation.** `K = clamp(round(duration / 2s), 6, 20)`, then capped so every
segment holds ≥ 5 frames. ~2 s is about as long as a person can hold "was the arm
warped just now?" in their head; the 6–20 clamp keeps the rating task bounded and
the bar readable. Clips here land at 6–20 segments. The cost of the cap is
honest: a 165 s singleview clip gets 20 × 8 s segments, coarser than the 2 s
target.

**Thresholds are calibrated at the segment level**, as the 95th percentile of the
*same* segment statistic measured on real (gt) video of the same robot+view. So
both detectors have a **5% false-positive rate on real footage by construction**,
and "YES" means *outside the top 5% of what real video of this robot ever does*.

> This is a deliberate change from the per-frame thresholds in `thresholds.json`.
> Those are 95th percentiles of per-**frame** values, and a MAX over ~10 frames
> clears a per-frame p95 about 40% of the time by chance. Measured: that rule fired
> on **20–43% of segments of real video** — unusable as a yes/no. Everything here
> uses the recalibrated segment thresholds below.

## Groups, thresholds, and what the rule measures

| group | robot | view | fps | thr rigidity (median) | thr jerk (max) | clips | segments |
|---|---|---|---|---|---|---|---|
| `single_arm_multiview` | franka_panda | 3 cams, ctrlworld | 5 | 16.70 mm | 195.92 mm/f³ | 257 | 2430 |
| `bimanual_multiview` | aloha_bimanual | 3 cams, ctrlworld | 5 | 31.07 mm | 173.07 mm/f³ | 300 | 4174 |
| `bimanual_singleview` | aloha_bimanual | cam_high, dreamdojo | 10 | 69.27 mm | 141.50 mm/f³ | 188 | 3252 |

Segment violation rate over the whole group (GT is 5.0% by construction):

| group | PRED rigidity | GT rigidity | sep | PRED jerk | GT jerk | sep |
|---|---|---|---|---|---|---|
| `single_arm_multiview` | **27.7%** | 4.9% | **5.6x** | **25.1%** | 5.0% | **5.0x** |
| `bimanual_multiview` | 7.0% | 5.0% | 1.4x | 7.2% | 5.0% | 1.4x |
| `bimanual_singleview` | 10.1% | 5.0% | 2.0x | 12.4% | 5.0% | 2.5x |

**Read this before rating.** Only `single_arm_multiview` separates cleanly — 5–6x
over real video. `bimanual_multiview` at 1.4x is barely above the judge's own
error floor; a rater disagreeing with those labels is probably right, and that
group is included so the disagreement can be measured, not because the labels are
trusted. `bimanual_singleview` sits in between (its reader has a 73.69 mm
validation error, one camera on a 14-DoF arm).

`single_arm` has **no singleview group**: those cells ship no joint ground truth,
so no reader could be trained or calibrated for them.

## Files

| group | example | segments violating |
|---|---|---|
| `single_arm_multiview` | `worst_rigidity__…episode_002168` | rigidity 13/13, jerk 8/13 |
| | `worst_jerk__…episode_006884` | rigidity 9/11, jerk 11/11 |
| | `clean__…episode_003714` | 0/8 and 0/8 |
| `bimanual_multiview` | `worst_rigidity__…spoons_on_the_cloth__000023` | rigidity 8/8, jerk 3/8 |
| | `worst_jerk__…spoons_on_the_cloth__000018` | rigidity 6/8, jerk 6/8 |
| | `clean__…close_toolbox__000002` | 0/6 and 0/6 |
| `bimanual_singleview` | `worst_rigidity__…episode_0001` | rigidity 10/12, jerk 0/12 |
| | `worst_jerk__…episode_0041` | rigidity 5/20, jerk 18/20 |
| | `clean__…episode_0016` | 0/20 and 0/20 |

`worst_rigidity__…episode_0001` is the useful diagnostic case: 10/12 segments
warped with **zero** jerk segments — the arm is drawn wrong but drawn *smoothly*
wrong, which is exactly the failure a joint→FK reader is structurally blind to
(FK pins limb lengths, so rigidity ≡ 0 there by construction).

Every mp4 has a sidecar `.json` with the per-segment `rigidity_median`,
`jerk_max`, the frame range, and the boolean verdict — the labels are auditable
without watching anything.

`manifest.json` carries the rule, the thresholds, and the group rates in machine
-readable form.

## Provenance

Scores come from the direct-keypoint runs already in this bucket
(`direct_keypoint/frame_scores.csv` for multiview, the singleview run for
`bimanual_singleview`); nothing was re-scored. Only the *aggregation* — segment
cutting, median/max, and the segment-level recalibration — is new here.