twanghcmut/result-video-bench/segment_eval - Storage Bucket

Segment-level physics violations — human-rating examples

Nine annotated clips (3 per robot/view group) showing which time segment of a generated video violates physics, so a human rater can agree or disagree with the judge segment by segment instead of on a whole-clip percentage.

Each mp4 is: PRED (Ctrl-World / DreamDojo generated) on top, GT (the real video it was conditioned on) below, and a bar at the bottom with one cell per segment. The playing segment carries a white border; Y = violating, n = not.

The rule

	statistic per segment	why
rigidity	MEDIAN of the per-frame score	a warped limb is a persistent state — the median says "the arm was wrong for most of this segment", and one bad frame cannot fake it
jerk	MAX of the per-frame score	a jolt is an impulsive event — one frame is the whole thing, so averaging it away would be wrong

Segmentation. K = clamp(round(duration / 2s), 6, 20), then capped so every segment holds ≥ 5 frames. ~2 s is about as long as a person can hold "was the arm warped just now?" in their head; the 6–20 clamp keeps the rating task bounded and the bar readable. Clips here land at 6–20 segments. The cost of the cap is honest: a 165 s singleview clip gets 20 × 8 s segments, coarser than the 2 s target.

Thresholds are calibrated at the segment level, as the 95th percentile of the same segment statistic measured on real (gt) video of the same robot+view. So both detectors have a 5% false-positive rate on real footage by construction, and "YES" means outside the top 5% of what real video of this robot ever does.

This is a deliberate change from the per-frame thresholds in thresholds.json. Those are 95th percentiles of per-frame values, and a MAX over ~10 frames clears a per-frame p95 about 40% of the time by chance. Measured: that rule fired on 20–43% of segments of real video — unusable as a yes/no. Everything here uses the recalibrated segment thresholds below.

Groups, thresholds, and what the rule measures

group	robot	view	fps	thr rigidity (median)	thr jerk (max)	clips	segments
single_arm_multiview	franka_panda	3 cams, ctrlworld	5	16.70 mm	195.92 mm/f³	257	2430
bimanual_multiview	aloha_bimanual	3 cams, ctrlworld	5	31.07 mm	173.07 mm/f³	300	4174
bimanual_singleview	aloha_bimanual	cam_high, dreamdojo	10	69.27 mm	141.50 mm/f³	188	3252

Segment violation rate over the whole group (GT is 5.0% by construction):

group	PRED rigidity	GT rigidity	sep	PRED jerk	GT jerk	sep
single_arm_multiview	27.7%	4.9%	5.6x	25.1%	5.0%	5.0x
bimanual_multiview	7.0%	5.0%	1.4x	7.2%	5.0%	1.4x
bimanual_singleview	10.1%	5.0%	2.0x	12.4%	5.0%	2.5x

Read this before rating. Only single_arm_multiview separates cleanly — 5–6x over real video. bimanual_multiview at 1.4x is barely above the judge's own error floor; a rater disagreeing with those labels is probably right, and that group is included so the disagreement can be measured, not because the labels are trusted. bimanual_singleview sits in between (its reader has a 73.69 mm validation error, one camera on a 14-DoF arm).

single_arm has no singleview group: those cells ship no joint ground truth, so no reader could be trained or calibrated for them.

Files

group	example	segments violating
single_arm_multiview	worst_rigidity__…episode_002168	rigidity 13/13, jerk 8/13
	worst_jerk__…episode_006884	rigidity 9/11, jerk 11/11
	clean__…episode_003714	0/8 and 0/8
bimanual_multiview	worst_rigidity__…spoons_on_the_cloth__000023	rigidity 8/8, jerk 3/8
	worst_jerk__…spoons_on_the_cloth__000018	rigidity 6/8, jerk 6/8
	clean__…close_toolbox__000002	0/6 and 0/6
bimanual_singleview	worst_rigidity__…episode_0001	rigidity 10/12, jerk 0/12
	worst_jerk__…episode_0041	rigidity 5/20, jerk 18/20
	clean__…episode_0016	0/20 and 0/20

worst_rigidity__…episode_0001 is the useful diagnostic case: 10/12 segments warped with zero jerk segments — the arm is drawn wrong but drawn smoothly wrong, which is exactly the failure a joint→FK reader is structurally blind to (FK pins limb lengths, so rigidity ≡ 0 there by construction).

Every mp4 has a sidecar .json with the per-segment rigidity_median, jerk_max, the frame range, and the boolean verdict — the labels are auditable without watching anything.

manifest.json carries the rule, the thresholds, and the group rates in machine -readable form.

Provenance

Scores come from the direct-keypoint runs already in this bucket (direct_keypoint/frame_scores.csv for multiview, the singleview run for bimanual_singleview); nothing was re-scored. Only the aggregation — segment cutting, median/max, and the segment-level recalibration — is new here.