QEV-2B

A 2B open multimodal decision model: text and image in, calibrated option probabilities out, in one forward pass with no generated tokens.

QEV-2B is the full-size member of the QEV family. It is built on Qwen3.5-2B. You supply the evidence (text, one or more images) and the candidate options with their meanings. The model returns a probability for each option, a typed answer, and an abstain signal.

Architecture, training, evaluation protocol, calibration, speed, ablations and limitations for the earlier release are described in the technical report.

What changed

This version retrains QEV-2B to reduce format overfitting. Public scores move little; the gain is on DEV items whose options are built differently from the training format.

• Diversified distractors. The earlier training data built each wrong option from the answer to another question about the same image, always with 4 options. The public InfographicVQA rows use the same construction. The v3 mix keeps that scheme for at most 35% of items. The rest use answers from other images, perturbed values and same-image annotations, with 3 to 6 options. 60% of Mind2Web items are re-rendered.
• Targeted v3 fixes. 707 CORD field-presence items now include receipt-text decoys. 157 new upstream train rows (127 Hateful Memes, 30 A-OKVQA) replace rows removed by the near-duplicate filter.
• DEV-only selection. The checkpoint score is the mean of the original-format selection set and a shifted-distractor DEV set. No public evaluation rows are used in training or selection. Training images within dHash distance 4 of any evaluation image are excluded.
• Private scores are unknown until the board re-scores this version.

	Previous release	This version
Shifted-distractor DEV (9 directories, mean accuracy)	84.90	88.79
Original-format holdout (18 sources, mean accuracy)	89.15	88.96
Public vision index (self-scored, same engine)	61.99	62.27

Results

Leaderboard comparison

Historical comparison models are copied from the old card's 2026-10-16 public board (edition 0.3.2). The QEV row is this version's self-score (previous release: 61.99 with the same engine). Row order does not assert this version's rank.

Model	Params	Index (public)
d3-mini	4.54B	67.67
RSI-Jev v6.1 VL 4B	4.66B	63.01
RSI-Jev v5.0 VL 3B	4.66B	62.58
QEV-2B (self-scored)	2.21B	62.27
Intern-Decision-4B	4.66B	61.69
Jev-Omni	11.96B	60.10
d3-nano	2.21B	59.15
Intern-Decision-2B	2.27B	52.92
OneJev1.5-0.8B	1.11B	52.32

Per-benchmark scores

Public evaluation only; self-scored, pending the board's re-score. These are post-selection results.

Benchmark	Chance-corrected score (%)	Raw accuracy (%)	Rows	Weight
CV-Bench	72.07	83.07	2038	1
BLINK	48.29	66.60	961	1
RealWorldQA	55.33	72.67	589	1
CharXiv	65.41	74.06	532	1
InfographicVQA	80.87	85.66	244	1
Mind2Web	82.73	87.05	973	1
Winoground	66.00	83.00	800	1
KIE (CORD+FUNSD)	92.52	95.46	2465	1
Moderation (Hateful Memes)	32.07	66.03	1213	1
R-Bench-M	15.36	29.47	665	0.5
MMMU-Pro vision	16.66	27.34	1730	0.25
Index	62.27	—	—	—

Index is the weighted mean of chance-corrected scores, not the mean raw accuracy.

Shift DEV and original-format holdout

Shift DEV

Directory	Accuracy (%)	Rows
A/chartqa	92.50	400
B/chartqa	93.50	400
Bn/chartqa	93.00	400
A/cord_train	95.12	246
B/cord_train	95.53	246
B/infographic_vqa	89.75	400
B/docvqa	99.75	400
A/mind2web_all	83.33	60
B/mind2web_all	56.67	60
Unweighted mean across the 9 directories above	88.79	—

The mean treats each listed directory equally; paired variants are separate directories.

Original-format holdout

Directory	Accuracy (%)	Rows
mix-holdout/ai2d	90.62	96
mix-holdout/aokvqa	85.00	120
mix-holdout/chartqa	90.00	120
mix-holdout/cord_train	98.33	120
mix-holdout/docvqa	99.17	120
mix-holdout/funsd_train	95.95	74
mix-holdout/hateful_memes	83.54	79
mix-holdout/iconqa	93.62	94
mix-holdout/infographic_vqa	84.95	93
mix-holdout/mind2web	87.50	120
mix-holdout/mind2web_k	82.30	113
mix-holdout/options_in_image	79.00	100
mix-holdout/scienceqa	95.00	120
mix-holdout/screen2words	93.33	90
mix-holdout/synthetic	91.67	120
mix-holdout/tallyqa	80.22	91
mix-holdout/visual7w	88.00	100
mix-holdout/vsr	83.15	89
Unweighted mean across the 18 directories above	88.96	—

The mean treats each listed directory equally; paired variants are separate directories.

Speed and fast mode

Mode	Image cap	Depth	Accuracy	Worst-case p95	Median p50	Peak GPU memory
Accurate (default)	1.6 MP	24 layers	89.5% DEV	738.6 ms	135.3 ms	4.44 GiB
Fast	0.26 MP	16 layers	84.7% DEV	225.0 ms	91.6 ms	4.13 GiB

_{Release loader only (qev.QEV.load + predict, no CUDA graphs or custom kernels), RTX 4060 Ti 8 GB, batch 1, warm. 47 request profiles (text and image requests with up to 16 options), 3 warm-up and 30 timed runs each. Worst-case p95 is the highest per-profile p95; median p50 is the median over profiles of each profile's p50. Peak GPU memory is PyTorch's peak allocation. Latency and memory were measured on the previous release, which has the same architecture, configuration and runtime path as this version; a re-measurement for this version was discarded because the control run did not reproduce under desktop load. DEV is the internal selection set (3,000 rows).}

Fast mode exits at layer 16 of 24 with a distilled head (exit16.safetensors) and caps images at 0.26 MP. At the same 0.26 MP input, the layer-16 head matches full depth on DEV (exit vs full: fam-text 91.5% vs 90.9% (n=485); nlvr2 88.3% vs 87.5% (n=128); fam-image 90.0% vs 90.5% (n=400); mix 81.1% vs 81.3% (n=1859)). The drop against accurate mode comes from the lower resolution and is largest on document, chart and screen sources. No index is reported for fast mode.

Calibration and abstention

Per-type temperatures and abstain thresholds use the DEV selection rows, with image/group-disjoint halves: temperature on one half, policy on the other. The policy targets at most 15% error with at least 20 accepted questions. These are internal DEV estimates. Serving calibration is fitted in accurate mode and shared with fast mode, as in the published releases. Fast mode is evaluated with those settings; a separate fast-mode fit is retained for inspection and is not applied to serving. A response still carries its answer when the separate abstain flag is set.

Type	Temperature rows	Policy rows	Temperature	Threshold	Status
choice	1392	1296	0.841395	0.296769	fitted
score	87	79	0.398107	0.419354	fitted
noul	66	80	0.446684	0.513213	fitted

On the held-out policy half, choice questions (1,296) have ECE 0.036 before and 0.018 after temperature scaling, with 100% coverage at 10.0% accepted error.

Architecture

• Backbone: Qwen3.5-2B language model (hidden size 2048, 24 layers: 18 GatedDeltaNet + 6 full attention), 2.21B parameters.
• Vision: Qwen3.5 ViT encoder (frozen).
• Readout: option probabilities read from the hidden states.
• Condition-modulated binding head, per-type temperature calibration and an abstain decision (fitted on DEV; see Calibration and abstention).
• Single forward pass, no generated tokens.
• Fast mode: a distilled exit head reads the options at layer 16 of 24 (exit16.safetensors); images are capped at 0.26 MP.
• GPU inference uses a BF16 backbone and FP32 readouts; the CPU path uses FP32.
• Fine-tuning: LoRA (rank 8, alpha 16) on the language model plus trained readout and binding heads; the vision encoder and the other base weights are frozen (board category: LoRA + head).
• Weights are shipped merged: model.safetensors (4.43 GB, BF16, both LoRA stages folded in) plus head.safetensors (0.7 MB, readout and binding head).

Training data

The following inherited source families are documented by the old card. The v3 distractor generators and exclusions are described under What changed.

Coverage mix (train split, 18 sources). Each source contributes items from its train split. 5% of each source (up to 120 items) is held out as a selection set and is never trained on.

Source	Split
AI2D	train
A-OKVQA	train
ChartQA	train
CORD	train
DocVQA	train
FUNSD	train
Hateful Memes	train
IconQA	train
InfographicVQA	train
Mind2Web	train
Mind2Web (keyed variant, mind2web_k)	train
Options-in-image	train
ScienceQA	train
Screen2Words	train
Synthetic (internal)	train
TallyQA	train
Visual7W	train
VSR	train

Additional sources

• NLVR2 train split (teacher-filtered targets). The NLVR2 dev split is used only for checkpoint selection.
• Replay of internal decision units (image and text families, train split), with teacher outputs as targets.
• Teacher outputs: option-probability distillation from QEV 0.2.1, the earlier QEV-2B release (ken-jo/qev at revision 0d2d13f): a LoRA (rank 8, alpha 16) with a condition-bound binding head on Qwen3.5-2B. The same teacher was used for QEV-2B and QEV-0.8B; it is not the released QEV-2B checkpoint.

Selection: DEV split only. Public rows are not used in training or selection.

v3 training data. 51,091 coverage items (49,275 trained, 1,816 held out for selection) from the 18 sources above, plus NLVR2 (2,048 rows) and decision replay (4,658 rows) in each pass. A fresh audit of all 46,971 training images found no image within dHash distance 4 of the 11,542 evaluation images or the 1,636 DEV images, and no gold answer was changed by the rebuild.

Training. Start: QEV 0.2.1 (Qwen3.5-2B + LoRA, merged), as for the previous release. One pass over the v3 mix (3,652 optimizer steps). The selected checkpoint is the end of the pass, the highest DEV selection score of the saved checkpoints. LoRA rank 8, alpha 16, merged into model.safetensors with FP32 accumulation.

Fast-mode head. The layer-16 head is distilled from this checkpoint's full-depth outputs, with the original probe recipe and data: a seeded sample of the earlier coverage mix and decision replay (24,658 rows), KL to the teacher plus 0.3 gold cross-entropy, 6 epochs. It does not change the backbone.

Quickstart

This checkpoint needs the qev source (the published qev==0.2.1 package loads QEV-2B 0.2.1 only).

python -m pip install "qev @ git+https://github.com/ken-jo/qev" huggingface_hub

from huggingface_hub import snapshot_download
import qev
from qev import DecisionRequest

path = snapshot_download("ken-jo/QEV-2B")
model = qev.QEV.load(path, device="cuda")               # accurate: full depth, up to 1.6 MP per image
# model = qev.QEV.load(path, device="cuda", mode="fast")  # fast: exit at layer 16, 0.26 MP
request = DecisionRequest.model_validate({
    "state": {
        "text": "I was charged twice. Please refund the duplicate payment.",
    },
    "questions": {
        "department": {
            "type": "choice",
            "instructions": "Which team should handle this request?",
            "criteria": {"billing": "Payments and refunds", "technical": "Software faults"},
        }
    },
})
print(model.predict(request)["answers"])

Note on the name: the PyPI package qev is also the import name used by OneJev (from qev import Client). Install one of them per environment.

Limitations

• Public numbers are self-scored after DEV-only checkpoint selection. Private scores are unknown until the board re-scores this version.
• Contamination: a post-hoc audit compared every v3 coverage-mix training image with every evaluation image. Candidates were dHash distance <= 10, confirmed by 64x64 grayscale mean absolute difference <= 6/255. It flagged 52 evaluation images (27 mind2web, 17 mmmu_pro_vision, 2 charxiv, 2 hateful_memes, 2 blink, 1 funsd, 1 winoground). On inspection, most flags are low-texture images such as near-blank web pages and plain backgrounds. The genuine matches are Hateful Memes base photos reused with different captions (one also appears in Winoground) and a FUNSD blank form template. Rescoring without all 65 affected rows changes the index from 62.27 to 62.31. NLVR2, the replay units and the inherited base training were not re-audited. If you find an overlapping item, please report it.
• Lower than the previous release on some public benchmarks: InfographicVQA 83.1 -> 80.9; R-Bench-M 16.8 -> 15.4; KIE (CORD+FUNSD) 93.9 -> 92.5. Part of the InfographicVQA drop is expected, since the earlier training format matched that benchmark's construction.
• The model returns probabilities, not guarantees. Calibration can fail on other domains; refit it on your own DEV data where abstention matters.
• The model scores only caller-supplied options and evidence. Evaluation focuses on English.
• This release does not include a 4B model; the original project used one RTX 4060 Ti 8 GB.

License

Apache-2.0, the same license as the base model. The base model Qwen3.5-2B is subject to its upstream license.

The training data is not redistributed with the model. For the third-party sources, ken-jo/qev-data (vision-mix-2026-10/) publishes item ids, gold labels and hashes, plus the build scripts, for the earlier mix; the v3 build is not published there yet. It also publishes the synthetic set in full and the NLVR2 teacher targets. Each training source keeps its own terms; some, such as FUNSD and Hateful Memes, restrict how their data may be used. Check those terms before reusing the data or retraining on it.

Citation

@misc{jo2026qev2b,
  title        = {QEV-2B: A Calibrated Multimodal Decision Model},
  author       = {Jo, Ken},
  year         = {2026},
  howpublished = {Hugging Face model card},
  url          = {https://huggingface.co/ken-jo/QEV-2B}
}