robojudge-4b-v02 SFT final (step 5780)
fac5b68 verified 1 day ago
RoboRewardBench (roboreward-v1): 2831/2831 items scored by every run
model
Overall (group-wise MAE)
RoboArena
OXE subsets
micro MAE
median-label MAE
expected-score MAE
exact acc
within-1
Spearman
QWK
mean pred
rj4b-v02-final
0.443 0.687
0.432
0.523
0.419
0.458
62.1%
89.0%
0.827
0.788
2.82
JevAny-4B
1.057 1.110
1.055
1.090
0.769
0.835
40.8%
69.0%
0.684
0.528
2.24
Leaderboard (paper Table 1/7; Overall / RoboArena / OXE): RoboReward-8B 0.665 / 0.768 / 0.660; GPT-5 mini 0.691 / 0.862 / 0.683; GPT-5 0.811 / 1.028 / 0.801; RoboReward-4B 0.845 / 0.806 / 0.847; Gemini 3 Pro (preview) 0.851 / 1.234 / 0.833; Qwen3-VL-8B Instruct 0.892 / 0.847 / 0.894; Gemini 2.5 Pro 0.902 / 0.936 / 0.900; Qwen3-VL-30B Instruct 0.903; Gemini Robotics-ER 1.5 0.906 / 1.002 / 0.902; Qwen3-VL-4B Instruct 1.032 / 0.929 / 1.036; Qwen2.5-VL-72B Instruct 0.991
rj4b-v02-final: confusion (rows = true 1-5, cols = predicted 1-5)
true
pred 1
pred 2
pred 3
pred 4
pred 5
n
mean pred
1
573
84
23
5
4
689
1.23
2
94
322
153
42
30
641
2.36
3
40
76
304
89
47
556
3.05
4
29
27
110
162
90
418
3.61
5
7
12
44
67
397
527
4.58
JevAny-4B: confusion (rows = true 1-5, cols = predicted 1-5)
true
pred 1
pred 2
pred 3
pred 4
pred 5
n
mean pred
1
672
0
3
0
14
689
1.09
2
518
6
38
0
79
641
1.62
3
321
7
117
5
106
556
2.22
4
181
2
49
3
183
418
3.01
5
120
3
48
0
356
527
3.89
Micro MAE / exact acc by item type
model
robo_arena (n=1000)
oxe_success (n=371)
oxe_relabelled (n=669)
oxe_clipped (n=791)
rj4b-v02-final
0.687 / 51.7%
0.358 / 76.5%
0.297 / 77.3%
0.584 / 55.6%
JevAny-4B
1.110 / 38.1%
1.124 / 67.1%
1.021 / 42.9%
1.106 / 30.0%
Per-subset MAE (argmax)
subset
n
rj4b-v02-final
JevAny-4B
robo_arena
1000
0.687
1.110
austin_sirius_dataset_converted_externally_to_rlds
87
0.414
0.448
berkeley_autolab_ur5
100
0.360
1.390
berkeley_fanuc_manipulation
91
0.220
1.352
berkeley_mvp_converted_externally_to_rlds
71
0.099
0.577
berkeley_rpt_converted_externally_to_rlds
86
0.826
0.605
bridge
100
0.240
1.010
cmu_play_fusion
92
0.413
1.500
dlr_edan_shared_control_converted_externally_to_rlds
19
0.000
0.579
droid
100
0.710
1.280
fractal20220817_data
100
0.240
1.100
iamlab_cmu_pickup_insert_converted_externally_to_rlds
83
0.133
1.024
jaco_play
100
0.270
1.280
kaist_nonprehensile_converted_externally_to_rlds
53
0.415
1.377
roboturk
97
0.505
0.701
stanford_hydra_dataset_converted_externally_to_rlds
91
0.923
1.297
taco_play
91
0.198
1.626
tokyo_u_lsmo_converted_externally_to_rlds
71
0.549
1.761
ucsd_kitchen_dataset_converted_externally_to_rlds
95
0.337
1.042
ucsd_pick_and_place_dataset_converted_externally_to_rlds
100
0.150
0.740
utokyo_pr2_tabletop_manipulation_converted_externally_to_rlds
73
0.288
0.671
utokyo_xarm_bimanual_converted_externally_to_rlds
71
1.451
1.408
viola
60
0.767
0.433
Paired: Overall MAE rj4b-v02-final - JevAny-4B = -0.614, 95% cluster-bootstrap CI [-0.725, -0.554]; per-item |err| smaller for rj4b-v02-final on 1135, larger on 253, tie 1443
