eval/robopulse.md

robopulse-v1: 1800 items

model	answered	acc (95% CI)	balanced acc	macro-F1	F1 pos / neg	AUROC	NLL	ECE	pred pos rate
rj4b-v02-final	1800/1800	71.1 [68.8, 73.2]	71.1	71.0	70.6 / 71.5	0.806	0.518	0.040	48.4
JevAny-4B	1800/1800	52.7 [50.5, 55.0]	52.7	44.2	66.0 / 22.3	0.588	0.746	0.151	89.2

Timing

model	GPUs	load s	wall s (loop)	mean / median / p95 ms per item	items/s	median input tokens	GPU-hours
rj4b-v02-final	1x NVIDIA A100-SXM4-40GB	11	586	325 / 294 / 540	3.07	1269	0.17
JevAny-4B	1x NVIDIA A100-SXM4-40GB	13	573	318 / 287 / 541	3.14	1269	0.16

Slices (accuracy / macro-F1, %)

slice	n	rj4b-v02-final	JevAny-4B
hop_bin=large	552	81.9 / 81.8	57.1 / 52.5
hop_bin=medium	500	76.0 / 75.9	52.4 / 39.9
hop_bin=small	748	59.8 / 59.7	49.7 / 39.6
image_dataset=agibotworld	200	62.5 / 62.2	51.0 / 36.3
image_dataset=agilex_newdragon	200	69.5 / 69.4	50.5 / 39.6
image_dataset=droid_oxe	200	73.0 / 72.9	52.0 / 39.1
image_dataset=galaxea_r1lite	200	64.5 / 64.2	51.5 / 47.2
image_dataset=human_egodex	200	68.0 / 68.0	55.5 / 52.7
image_dataset=human_pika	200	67.5 / 67.4	50.0 / 41.6
image_dataset=libero_data	200	84.0 / 84.0	59.0 / 51.2
image_dataset=robocasa_data	200	72.0 / 71.8	50.5 / 34.4
image_dataset=robotwin2_agilex_part1	200	78.5 / 78.5	54.5 / 47.9

Paired: rj4b-v02-final − JevAny-4B

• accuracy: 18.3 pts, 95% CI [15.3, 21.5]
• macro-F1: 26.9 pts, 95% CI [24.0, 29.8]
• McNemar: rj4b-v02-final only correct 584, JevAny-4B only correct 254, exact p = 1.062e-30