How often does RoboJev say success when the robot failed?

#1
by Sattyam - opened

Hi Josef, RoboJev landed right where I'm stuck. I red-team robot policies in simulation (Provael), and every rate I publish depends on a success or unsafe check, so the judge's false successes matter more to me than balanced accuracy.

On the SO-101 runs, how many of the failed attempts did it score as success? And how did you decide the 27B checkpoint was ready to release?

If you have 20 minutes some evening, I'd love to hear the second one properly. I'll be candid about how I do it on my side: a benign control next to every rate.

Sign up or log in to comment