---
base_model: Qwen/Qwen3.5-4B
library_name: peft
license: apache-2.0
tags:
- jevany
- lora
- decision-model
- pointer
- robotics
- reward-model
- judge
datasets:
- SimpleJev/JevAny-Robotics-Train-v0.2
---

# JevAny-Robotics-4B-LoRA (v0.2, SFT)

A 4B robot judge / reward model: a robotics continuation of [`SimpleJev/JevAny-Qwen3.5-4B-LoRA`](https://huggingface.co/SimpleJev/JevAny-Qwen3.5-4B-LoRA)
(pointer readout on [`Qwen/Qwen3.5-4B`](https://huggingface.co/Qwen/Qwen3.5-4B)), fine-tuned on [`SimpleJev/JevAny-Robotics-Train-v0.2`](https://huggingface.co/datasets/SimpleJev/JevAny-Robotics-Train-v0.2) — 184,942 judgement questions over
real and simulated robot images and videos (task progress, 1–5 rollout scores, execution success, failure mode, plan checking,
failure localisation, trajectory preference), built only from train-side splits and decontaminated against every test set below.

**Highlights** (all models run by us in the same harness, on the same items with the same media protocol):

- **RoboPulse: 71.1% accuracy / 0.806 AUROC** — best of every judge we ran, including 27B models (OneJev-27B 60.4% / 0.651, JevAny-27B 56.6% / 0.627); +18.3 points over its parent.
- **RoboRewardBench: MAE 0.443** — lower than every model we ran (OneJev-27B 0.805, JevAny-27B 0.692, Intern-Decision-4B 0.820) and than the best entry of the paper leaderboard (RoboReward-8B 0.665, their protocol); paired MAE difference vs the best 27B model −0.249 [−0.374, −0.220].
- **RoboPulse++: best AUROC (0.728)** and balanced accuracy among 4B models (65.9 vs Intern-Decision-4B 60.5); at the level of the best 27B model (OneJev-27B 0.700 AUROC, paired +0.028 [−0.004, +0.063]).
- **JevAny-Robotics-Public: skill_role 0.567, micro accuracy 80.4% (highest of all models)** — 4× its parent, +0.14 over OneJev-4B, statistically tied with the strongest 27B judge (OneJev-27B 0.605, paired −0.038 [−0.081, +0.006]).
- Same 4B backbone and speed as the parent (median 161 ms per JevAny-Robotics-Public item on an A100).

## Robot benchmarks

| Model | Size | RoboPulse acc ↑ | RoboPulse AUROC ↑ | RoboPulse++ AUROC ↑ | RoboPulse++ bal. acc ↑ | RoboRewardBench MAE ↓ | JevAny-Robotics-Public skill_role ↑ |
|---|---:|---:|---:|---:|---:|---:|---:|
| **JevAny-Robotics-4B-LoRA (this model)** | 4B | **71.1** | **0.806** | **0.728** | **65.9** | **0.443** | 0.567 |
| JevAny-Qwen3.5-4B-LoRA (parent) | 4B | 52.7 | 0.588 | 0.641 | 59.1 | 1.057 | 0.149 |
| Qwen3.5-4B zero-shot (constrained choice) | 4B | 49.9 | 0.519 | 0.592 | 50.0 | 1.261 | 0.203 |
| OneJev-4B | 4B | 53.9 | 0.586 | 0.617 | 58.3 | 1.005 | 0.425 |
| Intern-Decision-4B | 4B | 51.6 | 0.547 | 0.653 | 60.5 | 0.820 | 0.303 |
| OmniJev-4B | 4B | 49.7 | 0.536 | 0.556 | 54.6 | 0.811* | 0.264 |
| OneJev-9B | 9B | 57.6 | 0.595 | 0.564 | 54.0 | 0.927 | 0.442 |
| Jev-Omni-12B | 12B | 49.4 | 0.492 | 0.537 | 52.3 | 1.085 | 0.194 |
| Qwen3.8-27B zero-shot (constrained choice) | 27B | 56.7 | 0.604 | 0.569 | 55.2 | 0.992 | 0.472 |
| JevAny-Qwen3.8-27B-LoRA | 27B | 56.6 | 0.627 | 0.692 | 55.7 | 0.692 | 0.549 |
| OneJev-27B | 27B | 60.4 | 0.651 | 0.700 | 62.7 | 0.805 | **0.605** |

RoboPulse: 1,800 pairs, native image resolution. RoboPulse++: 7,720 consecutive-step pairs, native resolution. RoboRewardBench: 2,831 videos, full media,
official group-wise MAE (RoboArena 0.687 / OXE subsets 0.432 for this model). JevAny-Robotics-Public v0.1: 3,990 items, skill_role is the role-balanced
chance-corrected skill. *OmniJev-4B's training data overlaps RoboRewardBench. Every model answered every item.

**Paired comparisons** (this model minus baseline, 95% cluster-bootstrap CI over episodes / task groups):

| Baseline | RoboPulse acc | RoboPulse++ AUROC | RoboRewardBench MAE | JevAny-Robotics-Public skill_role |
|---|---:|---:|---:|---:|
| Parent JevAny-4B | +18.3 [+15.1, +21.4] | +0.087 [+0.049, +0.131] | −0.614 [−0.725, −0.554] | +0.419 [+0.365, +0.463] |
| Qwen3.5-4B zero-shot | +21.2 [+18.1, +24.4] | +0.136 [+0.091, +0.187] | −0.817 [−0.875, −0.732] | +0.364 [+0.318, +0.411] |
| OneJev-4B | +17.2 [+14.1, +20.1] | +0.111 [+0.072, +0.151] | −0.562 [−0.811, −0.541] | +0.142 [+0.100, +0.189] |
| Intern-Decision-4B | +19.4 [+16.5, +22.5] | +0.075 [+0.047, +0.104] | −0.376 [−0.518, −0.341] | +0.264 [+0.218, +0.312] |
| JevAny-27B | +14.4 [+11.4, +17.0] | +0.036 [−0.003, +0.076] | −0.249 [−0.374, −0.220] | +0.019 [−0.027, +0.052] |
| OneJev-27B | +10.7 [+8.0, +13.3] | +0.028 [−0.004, +0.063] | −0.362 [−0.488, −0.336] | −0.038 [−0.081, +0.006] |

Every difference in favour of this model is significant except RoboPulse++ AUROC and JevAny-Robotics-Public against the two 27B judges (ties).
Per-benchmark reports with all slices are in `eval/comparison/`.

For reference, published numbers under the benchmarks' own protocols: RoboRewardBench leaderboard — RoboReward-8B 0.665, GPT-5 mini 0.691,
GPT-5 0.811, RoboReward-4B 0.845, Gemini 3 Pro (preview) 0.851, Qwen3-VL-8B 0.892; RoboPulse paper — Gemini 3 Pro Preview 0.66, Qwen3-VL-8B 0.59,
Qwen3-VL-4B 0.56.

### Baselines

- **Zero-shot Qwen3.5-4B / Qwen3.8-27B (constrained choice)**: the untrained base VLM with the same judge prompt; options are lettered and the answer is
  restricted to the option letters (one forward pass, probabilities = softmax over the letter logits). This is the same scoring as vLLM
  structured-output choice decoding (`structured_outputs.choice` + letter logprobs); the robot suites were run with the transformers implementation,
  JevAny-Multimodal-agentic-Public with vLLM.
- **OneJev / Intern-Decision / OmniJev / Jev-Omni**: released checkpoints run with their own inference code inside the same harness and items.
- **JevAny-Qwen3.8-27B-LoRA** and the parent **JevAny-Qwen3.5-4B-LoRA**: released JevAny checkpoints, same media settings as this model.

## General judgement (retention)

| Benchmark | Qwen3.5-4B zero-shot (vLLM choice) | Parent JevAny-4B | This model | This − parent |
|---|---:|---:|---:|---|
| Transfer v9 (1,046) acc / NLL | — | 78.68% / 0.587 | 78.39% / 0.664 | acc −0.29 [−1.75, +1.05] |
| JevBench public (local rebuild, 231) acc | — | 77.92% | 77.92% | 0.00 [−3.10, +2.64] |
| JevAny-Multimodal-agentic-Public v0.4 (3,208) skill_role / acc | 0.223 / 53.6% | 0.201 / 52.6% | 0.176 / 51.4% | acc −1.2 (n.s., McNemar p = 0.12) |

Accuracy on general (non-robot) judgement is retained within noise; probabilities become over-confident outside robotics
(NLL +0.077 on Transfer, +0.163 on JevAny-Multimodal-agentic-Public), so recalibrate before using them there. For general-purpose judging use the parent
checkpoint; this one is specialised for robotics. The JevBench number is on a local rebuild of the public suite (same 231 items, different
manifest hash) and is not directly comparable to the official 80.09%.

## Training

- Init: LoRA adapter + pointer head of `SimpleJev/JevAny-Qwen3.5-4B-LoRA` (delta mode); base revision `851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a`.
- 1 epoch, 5,780 optimizer steps, global batch 32, lr 1e-5 (OneCycle), LoRA r=8 on all projection modules, head dim 256 + residual 256, bf16.
- Full media: images at native resolution, every frame of the derived video clips (≈1–2 fps, 8–32 frames), up to 16,384 packed tokens.
- `training_config.json` / `training_metrics.json` hold the exact arguments; `eval/training_eval_history.jsonl` the in-training evaluations.

| step | development acc / NLL (3,026) | calibration acc / NLL (2,228) |
|---:|---:|---:|
| 0 (parent) | 0.463 / 1.069 | 0.496 / 0.969 |
| 1,000 | 0.640 / 0.757 | 0.688 / 0.633 |
| 3,000 | 0.723 / 0.650 | 0.778 / 0.499 |
| 5,780 (this checkpoint) | 0.733 / 0.638 | 0.786 / 0.475 |

## Limits

- **Source overlap.** RoboPulse and RoboRewardBench draw on the same public robot datasets as parts of the training data (GRM, RoboReward train).
  Training data were decontaminated against all test sets (episode blocklists + perceptual-hash frame dedup), but scenes and robots overlap.
- **RoboPulse++ threshold.** Training balanced rising/falling 1:1, while the RoboPulse++ test is 81% rising; at the default 0.5 threshold the model
  under-predicts "rising" (accuracy 60.4% vs 69.9% for the parent; a constant "rising" answer scores 81.1%). Ranking quality (AUROC) and balanced
  accuracy are the meaningful metrics here and both improve; set the threshold on your own data. Multi-view (2–3 camera) items and long, slow-progress
  intervals remain the weakest cases.
- **Calibration outside robotics** (see above).

## Usage

```bash
pip install -e '.[serve,multimodal]'
jevany serve --checkpoint SimpleJev/JevAny-Robotics-4B-LoRA --device cuda --dtype bf16
```

For video inputs use the same media handling as training/evaluation (native resolution, all frames of short derived clips).
The repository contains LoRA adapter weights and JevAny readout metadata, not the base weights; the base model's license and
access terms also apply, as do the licenses of the training data sources listed in the dataset card.