# robopulse-v1: 1800 items

| model | answered | acc (95% CI) | balanced acc | macro-F1 | F1 pos / neg | AUROC | NLL | ECE | pred pos rate |
|---|---|---|---|---|---|---|---|---|---|
| rj4b-v02-final | 1800/1800 | 71.1 [68.8, 73.2] | 71.1 | 71.0 | 70.6 / 71.5 | 0.806 | 0.518 | 0.040 | 48.4 |
| JevAny-4B | 1800/1800 | 52.7 [50.5, 55.0] | 52.7 | 44.2 | 66.0 / 22.3 | 0.588 | 0.746 | 0.151 | 89.2 |

## Timing

| model | GPUs | load s | wall s (loop) | mean / median / p95 ms per item | items/s | median input tokens | GPU-hours |
|---|---|---|---|---|---|---|---|
| rj4b-v02-final | 1x NVIDIA A100-SXM4-40GB | 11 | 586 | 325 / 294 / 540 | 3.07 | 1269 | 0.17 |
| JevAny-4B | 1x NVIDIA A100-SXM4-40GB | 13 | 573 | 318 / 287 / 541 | 3.14 | 1269 | 0.16 |

## Slices (accuracy / macro-F1, %)

| slice | n | rj4b-v02-final | JevAny-4B |
|---|---|---|---|
| hop_bin=large | 552 | 81.9 / 81.8 | 57.1 / 52.5 |
| hop_bin=medium | 500 | 76.0 / 75.9 | 52.4 / 39.9 |
| hop_bin=small | 748 | 59.8 / 59.7 | 49.7 / 39.6 |
| image_dataset=agibotworld | 200 | 62.5 / 62.2 | 51.0 / 36.3 |
| image_dataset=agilex_newdragon | 200 | 69.5 / 69.4 | 50.5 / 39.6 |
| image_dataset=droid_oxe | 200 | 73.0 / 72.9 | 52.0 / 39.1 |
| image_dataset=galaxea_r1lite | 200 | 64.5 / 64.2 | 51.5 / 47.2 |
| image_dataset=human_egodex | 200 | 68.0 / 68.0 | 55.5 / 52.7 |
| image_dataset=human_pika | 200 | 67.5 / 67.4 | 50.0 / 41.6 |
| image_dataset=libero_data | 200 | 84.0 / 84.0 | 59.0 / 51.2 |
| image_dataset=robocasa_data | 200 | 72.0 / 71.8 | 50.5 / 34.4 |
| image_dataset=robotwin2_agilex_part1 | 200 | 78.5 / 78.5 | 54.5 / 47.9 |

## Paired: rj4b-v02-final − JevAny-4B

- accuracy: 18.3 pts, 95% CI [15.3, 21.5]
- macro-F1: 26.9 pts, 95% CI [24.0, 29.8]
- McNemar: rj4b-v02-final only correct 584, JevAny-4B only correct 254, exact p = 1.062e-30