SimpleJev/JevAny-Robotics-4B-LoRA

JevAny Qwen3.5 4B RoboJudge LoRA (v0.2, SFT)

A 4B robot judge / reward model: a robotics continuation of SimpleJev/JevAny-Robotics-4B-LoRA (pointer readout on Qwen/Qwen3.5-4B), fine-tuned on SimpleJev/JevAny-Robotics-Train-v0.2 — 184,942 judgement questions over real and simulated robot images and videos (task progress, 1–5 rollout scores, execution success, failure mode, plan checking, failure localisation, trajectory preference), built only from train-side splits and decontaminated against every test set below.

Highlights (all models run by us in the same harness, on the same items with the same media protocol):

• RoboPulse: 71.1% accuracy / 0.806 AUROC — best of every judge we ran, including 27B models (OneJev-27B 60.4% / 0.651, JevAny-27B 56.6% / 0.627); +18.3 points over its parent.
• RoboRewardBench: MAE 0.443 — lower than every model we ran (OneJev-27B 0.805, JevAny-27B 0.692, Intern-Decision-4B 0.820) and than the best entry of the paper leaderboard (RoboReward-8B 0.665, their protocol); paired MAE difference vs the best 27B model −0.249 [−0.374, −0.220].
• RoboPulse++: best AUROC (0.728) and balanced accuracy among 4B models (65.9 vs Intern-Decision-4B 60.5); at the level of the best 27B model (OneJev-27B 0.700 AUROC, paired +0.028 [−0.004, +0.063]).
• JevJudge-Robotics-Public: skill_role 0.567, micro accuracy 80.4% (highest of all models) — 4× its parent, +0.14 over OneJev-4B, statistically tied with the strongest 27B judge (OneJev-27B 0.605, paired −0.038 [−0.081, +0.006]).
• Same 4B backbone and speed as the parent (median 161 ms per JevJudge-Robotics item on an A100).

Robot benchmarks

Model	Size	RoboPulse acc ↑	RoboPulse AUROC ↑	RoboPulse++ AUROC ↑	RoboPulse++ bal. acc ↑	RoboRewardBench MAE ↓	JevJudge-Robotics skill_role ↑
JevAny-Robotics-4B-LoRA (this model)	4B	71.1	0.806	0.728	65.9	0.443	0.567
JevAny-Qwen3.5-4B-LoRA (parent)	4B	52.7	0.588	0.641	59.1	1.057	0.149
Qwen3.5-4B zero-shot (constrained choice)	4B	49.9	0.519	0.592	50.0	1.261	0.203
OneJev-4B	4B	53.9	0.586	0.617	58.3	1.005	0.425
Intern-Decision-4B	4B	51.6	0.547	0.653	60.5	0.820	0.303
OmniJev-4B	4B	49.7	0.536	0.556	54.6	0.811*	0.264
OneJev-9B	9B	57.6	0.595	0.564	54.0	0.927	0.442
Jev-Omni-12B	12B	49.4	0.492	0.537	52.3	1.085	0.194
Qwen3.8-27B zero-shot (constrained choice)	27B	56.7	0.604	0.569	55.2	0.992	0.472
JevAny-Qwen3.8-27B-LoRA	27B	56.6	0.627	0.692	55.7	0.692	0.549
OneJev-27B	27B	60.4	0.651	0.700	62.7	0.805	0.605

RoboPulse: 1,800 pairs, native image resolution. RoboPulse++: 7,720 consecutive-step pairs, native resolution. RoboRewardBench: 2,831 videos, full media, official group-wise MAE (RoboArena 0.687 / OXE subsets 0.432 for this model). JevJudge-Robotics-Public v0.1: 3,990 items, skill_role is the role-balanced chance-corrected skill. *OmniJev-4B's training data overlaps RoboRewardBench. Every model answered every item.

Paired comparisons (this model minus baseline, 95% cluster-bootstrap CI over episodes / task groups):

Baseline	RoboPulse acc	RoboPulse++ AUROC	RoboRewardBench MAE	JevJudge-Robotics skill_role
Parent JevAny-4B	+18.3 [+15.1, +21.4]	+0.087 [+0.049, +0.131]	−0.614 [−0.725, −0.554]	+0.419 [+0.365, +0.463]
Qwen3.5-4B zero-shot	+21.2 [+18.1, +24.4]	+0.136 [+0.091, +0.187]	−0.817 [−0.875, −0.732]	+0.364 [+0.318, +0.411]
OneJev-4B	+17.2 [+14.1, +20.1]	+0.111 [+0.072, +0.151]	−0.562 [−0.811, −0.541]	+0.142 [+0.100, +0.189]
Intern-Decision-4B	+19.4 [+16.5, +22.5]	+0.075 [+0.047, +0.104]	−0.376 [−0.518, −0.341]	+0.264 [+0.218, +0.312]
JevAny-27B	+14.4 [+11.4, +17.0]	+0.036 [−0.003, +0.076]	−0.249 [−0.374, −0.220]	+0.019 [−0.027, +0.052]
OneJev-27B	+10.7 [+8.0, +13.3]	+0.028 [−0.004, +0.063]	−0.362 [−0.488, −0.336]	−0.038 [−0.081, +0.006]

Every difference in favour of this model is significant except RoboPulse++ AUROC and JevJudge-Robotics against the two 27B judges (ties). Per-benchmark reports with all slices are in eval/comparison/.

For reference, published numbers under the benchmarks' own protocols: RoboRewardBench leaderboard — RoboReward-8B 0.665, GPT-5 mini 0.691, GPT-5 0.811, RoboReward-4B 0.845, Gemini 3 Pro (preview) 0.851, Qwen3-VL-8B 0.892; RoboPulse paper — Gemini 3 Pro Preview 0.66, Qwen3-VL-8B 0.59, Qwen3-VL-4B 0.56.

Baselines

• Zero-shot Qwen3.5-4B / Qwen3.8-27B (constrained choice): the untrained base VLM with the same judge prompt; options are lettered and the answer is restricted to the option letters (one forward pass, probabilities = softmax over the letter logits). This is the same scoring as vLLM structured-output choice decoding (structured_outputs.choice + letter logprobs); the robot suites were run with the transformers implementation, JevJudge-Public with vLLM.
• OneJev / Intern-Decision / OmniJev / Jev-Omni: released checkpoints run with their own inference code inside the same harness and items.
• JevAny-Qwen3.8-27B-LoRA and the parent JevAny-Qwen3.5-4B-LoRA: released JevAny checkpoints, same media settings as this model.

General judgement (retention)

Benchmark	Qwen3.5-4B zero-shot (vLLM choice)	Parent JevAny-4B	This model	This − parent
Transfer v9 (1,046) acc / NLL	—	78.68% / 0.587	78.39% / 0.664	acc −0.29 [−1.75, +1.05]
JevBench public (local rebuild, 231) acc	—	77.92%	77.92%	0.00 [−3.10, +2.64]
JevJudge-Public v0.4 (3,208) skill_role / acc	0.223 / 53.6%	0.201 / 52.6%	0.176 / 51.4%	acc −1.2 (n.s., McNemar p = 0.12)

Accuracy on general (non-robot) judgement is retained within noise; probabilities become over-confident outside robotics (NLL +0.077 on Transfer, +0.163 on JevJudge-Public), so recalibrate before using them there. For general-purpose judging use the parent checkpoint; this one is specialised for robotics. The JevBench number is on a local rebuild of the public suite (same 231 items, different manifest hash) and is not directly comparable to the official 80.09%.

Training

• Init: LoRA adapter + pointer head of SimpleJev/JevAny-Qwen3.5-4B-LoRA (delta mode); base revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a.
• 1 epoch, 5,780 optimizer steps, global batch 32, lr 1e-5 (OneCycle), LoRA r=8 on all projection modules, head dim 256 + residual 256, bf16.
• Full media: images at native resolution, every frame of the derived video clips (≈1–2 fps, 8–32 frames), up to 16,384 packed tokens.
• training_config.json / training_metrics.json hold the exact arguments; eval/training_eval_history.jsonl the in-training evaluations.

step	development acc / NLL (3,026)	calibration acc / NLL (2,228)
0 (parent)	0.463 / 1.069	0.496 / 0.969
1,000	0.640 / 0.757	0.688 / 0.633
3,000	0.723 / 0.650	0.778 / 0.499
5,780 (this checkpoint)	0.733 / 0.638	0.786 / 0.475

Limits

• Source overlap. RoboPulse and RoboRewardBench draw on the same public robot datasets as parts of the training data (GRM, RoboReward train). Training data were decontaminated against all test sets (episode blocklists + perceptual-hash frame dedup), but scenes and robots overlap.
• RoboPulse++ threshold. Training balanced rising/falling 1:1, while the RoboPulse++ test is 81% rising; at the default 0.5 threshold the model under-predicts "rising" (accuracy 60.4% vs 69.9% for the parent; a constant "rising" answer scores 81.1%). Ranking quality (AUROC) and balanced accuracy are the meaningful metrics here and both improve; set the threshold on your own data. Multi-view (2–3 camera) items and long, slow-progress intervals remain the weakest cases.
• Calibration outside robotics (see above).

Usage

pip install -e '.[serve,multimodal]'
jevany serve --checkpoint SimpleJev/JevAny-Qwen3.5-4B-RoboJudge-LoRA --device cuda --dtype bf16

For video inputs use the same media handling as training/evaluation (native resolution, all frames of short derived clips). The repository contains LoRA adapter weights and JevAny readout metadata, not the base weights; the base model's license and access terms also apply, as do the licenses of the training data sources listed in the dataset card.