# RoboRewardBench (roboreward-v1): 2831/2831 items scored by every run

| model | Overall (group-wise MAE) | RoboArena | OXE subsets | micro MAE | median-label MAE | expected-score MAE | exact acc | within-1 | Spearman | QWK | mean pred |
|---|---|---|---|---|---|---|---|---|---|---|---|
| rj4b-v02-final | **0.443** | 0.687 | 0.432 | 0.523 | 0.419 | 0.458 | 62.1% | 89.0% | 0.827 | 0.788 | 2.82 |
| JevAny-4B | **1.057** | 1.110 | 1.055 | 1.090 | 0.769 | 0.835 | 40.8% | 69.0% | 0.684 | 0.528 | 2.24 |

Leaderboard (paper Table 1/7; Overall / RoboArena / OXE): RoboReward-8B 0.665 / 0.768 / 0.660; GPT-5 mini 0.691 / 0.862 / 0.683; GPT-5 0.811 / 1.028 / 0.801; RoboReward-4B 0.845 / 0.806 / 0.847; Gemini 3 Pro (preview) 0.851 / 1.234 / 0.833; Qwen3-VL-8B Instruct 0.892 / 0.847 / 0.894; Gemini 2.5 Pro 0.902 / 0.936 / 0.900; Qwen3-VL-30B Instruct 0.903; Gemini Robotics-ER 1.5 0.906 / 1.002 / 0.902; Qwen3-VL-4B Instruct 1.032 / 0.929 / 1.036; Qwen2.5-VL-72B Instruct 0.991

## rj4b-v02-final: confusion (rows = true 1-5, cols = predicted 1-5)

| true | pred 1 | pred 2 | pred 3 | pred 4 | pred 5 | n | mean pred |
|---|---|---|---|---|---|---|---|
| 1 | 573 | 84 | 23 | 5 | 4 | 689 | 1.23 |
| 2 | 94 | 322 | 153 | 42 | 30 | 641 | 2.36 |
| 3 | 40 | 76 | 304 | 89 | 47 | 556 | 3.05 |
| 4 | 29 | 27 | 110 | 162 | 90 | 418 | 3.61 |
| 5 | 7 | 12 | 44 | 67 | 397 | 527 | 4.58 |

## JevAny-4B: confusion (rows = true 1-5, cols = predicted 1-5)

| true | pred 1 | pred 2 | pred 3 | pred 4 | pred 5 | n | mean pred |
|---|---|---|---|---|---|---|---|
| 1 | 672 | 0 | 3 | 0 | 14 | 689 | 1.09 |
| 2 | 518 | 6 | 38 | 0 | 79 | 641 | 1.62 |
| 3 | 321 | 7 | 117 | 5 | 106 | 556 | 2.22 |
| 4 | 181 | 2 | 49 | 3 | 183 | 418 | 3.01 |
| 5 | 120 | 3 | 48 | 0 | 356 | 527 | 3.89 |

## Micro MAE / exact acc by item type

| model | robo_arena (n=1000) | oxe_success (n=371) | oxe_relabelled (n=669) | oxe_clipped (n=791) |
|---|---|---|---|---|
| rj4b-v02-final | 0.687 / 51.7% | 0.358 / 76.5% | 0.297 / 77.3% | 0.584 / 55.6% |
| JevAny-4B | 1.110 / 38.1% | 1.124 / 67.1% | 1.021 / 42.9% | 1.106 / 30.0% |

## Per-subset MAE (argmax)

| subset | n | rj4b-v02-final | JevAny-4B |
|---|---|---|---|
| robo_arena | 1000 | 0.687 | 1.110 |
| austin_sirius_dataset_converted_externally_to_rlds | 87 | 0.414 | 0.448 |
| berkeley_autolab_ur5 | 100 | 0.360 | 1.390 |
| berkeley_fanuc_manipulation | 91 | 0.220 | 1.352 |
| berkeley_mvp_converted_externally_to_rlds | 71 | 0.099 | 0.577 |
| berkeley_rpt_converted_externally_to_rlds | 86 | 0.826 | 0.605 |
| bridge | 100 | 0.240 | 1.010 |
| cmu_play_fusion | 92 | 0.413 | 1.500 |
| dlr_edan_shared_control_converted_externally_to_rlds | 19 | 0.000 | 0.579 |
| droid | 100 | 0.710 | 1.280 |
| fractal20220817_data | 100 | 0.240 | 1.100 |
| iamlab_cmu_pickup_insert_converted_externally_to_rlds | 83 | 0.133 | 1.024 |
| jaco_play | 100 | 0.270 | 1.280 |
| kaist_nonprehensile_converted_externally_to_rlds | 53 | 0.415 | 1.377 |
| roboturk | 97 | 0.505 | 0.701 |
| stanford_hydra_dataset_converted_externally_to_rlds | 91 | 0.923 | 1.297 |
| taco_play | 91 | 0.198 | 1.626 |
| tokyo_u_lsmo_converted_externally_to_rlds | 71 | 0.549 | 1.761 |
| ucsd_kitchen_dataset_converted_externally_to_rlds | 95 | 0.337 | 1.042 |
| ucsd_pick_and_place_dataset_converted_externally_to_rlds | 100 | 0.150 | 0.740 |
| utokyo_pr2_tabletop_manipulation_converted_externally_to_rlds | 73 | 0.288 | 0.671 |
| utokyo_xarm_bimanual_converted_externally_to_rlds | 71 | 1.451 | 1.408 |
| viola | 60 | 0.767 | 0.433 |

Paired: Overall MAE rj4b-v02-final - JevAny-4B = -0.614, 95% cluster-bootstrap CI [-0.725, -0.554]; per-item |err| smaller for rj4b-v02-final on 1135, larger on 253, tie 1443