QEV-0.8B

A 0.8B open multimodal decision model: text and image in, calibrated option probabilities out, in one forward pass with no generated tokens.

QEV-0.8B is the small member of the QEV family. It is built on Qwen3.5-0.8B and distilled, like QEV-2B, from the QEV 0.2.1 teacher (see Training data). You supply the evidence (text, one or more images) and the candidate options with their meanings. The model returns a probability for each option, a typed answer, and an abstain signal.

Architecture, training, evaluation protocol, calibration, speed, ablations and limitations for the earlier release are described in the technical report.

What changed

This version retrains QEV-0.8B to reduce format overfitting. Public scores move little; the gain is on DEV items whose options are built differently from the training format.

• Diversified distractors. The earlier training data built each wrong option from the answer to another question about the same image, always with 4 options. The public InfographicVQA rows use the same construction. The v3 mix keeps that scheme for at most 35% of items. The rest use answers from other images, perturbed values and same-image annotations, with 3 to 6 options. 60% of Mind2Web items are re-rendered.
• Targeted v3 fixes. 707 CORD field-presence items now include receipt-text decoys. 157 new upstream train rows (127 Hateful Memes, 30 A-OKVQA) replace rows removed by the near-duplicate filter.
• DEV-only selection. The checkpoint score is the mean of the original-format selection set and a shifted-distractor DEV set. No public evaluation rows are used in training or selection. Training images within dHash distance 4 of any evaluation image are excluded.
• Private scores are unknown until the board re-scores this version.

	Previous release	This version
Shifted-distractor DEV (9 directories, mean accuracy)	81.06	88.64
Original-format holdout (18 sources, mean accuracy)	86.06	85.15
Public vision index (self-scored, same engine)	53.02	52.70

Results

Leaderboard comparison

Historical comparison models are copied from the old card's 2026-10-16 public board (edition 0.3.2). The QEV row is this version's self-score (previous release: 53.02 with the same engine). Row order does not assert this version's rank.

Model	Params	Index (public)
d3-nano	2.21B	59.15
Intern-Decision-2B	2.27B	52.92
QEV-0.8B (self-scored)	0.8B	52.70
OneJev1.5-0.8B	1.11B	52.32
d3-lite	0.85B	47.88
JPT-0.8B	0.87B	44.55
Intern-Decision-0.8B	0.87B	38.96
d3-edge	0.59B	36.61

Per-benchmark scores

Public evaluation only; self-scored, pending the board's re-score. These are post-selection results.

Benchmark	Chance-corrected score (%)	Raw accuracy (%)	Rows	Weight
CV-Bench	58.64	74.93	2038	1
BLINK	30.89	55.36	961	1
RealWorldQA	38.96	62.65	589	1
CharXiv	55.89	66.92	532	1
InfographicVQA	70.49	77.87	244	1
Mind2Web	77.53	83.14	973	1
Winoground	51.75	75.88	800	1
KIE (CORD+FUNSD)	91.78	95.01	2465	1
Moderation (Hateful Memes)	28.11	64.06	1213	1
R-Bench-M	15.72	29.77	665	0.5
MMMU-Pro vision	7.85	19.65	1730	0.25
Index	52.70	—	—	—

Index is the weighted mean of chance-corrected scores, not the mean raw accuracy.

Shift DEV and original-format holdout

Shift DEV

Directory	Accuracy (%)	Rows
A/chartqa	90.50	400
B/chartqa	92.25	400
Bn/chartqa	91.75	400
A/cord_train	96.34	246
B/cord_train	94.72	246
B/infographic_vqa	91.00	400
B/docvqa	99.50	400
A/mind2web_all	83.33	60
B/mind2web_all	58.33	60
Unweighted mean across the 9 directories above	88.64	—

The mean treats each listed directory equally; paired variants are separate directories.

Original-format holdout

Directory	Accuracy (%)	Rows
mix-holdout/ai2d	85.42	96
mix-holdout/aokvqa	73.33	120
mix-holdout/chartqa	90.83	120
mix-holdout/cord_train	98.33	120
mix-holdout/docvqa	99.17	120
mix-holdout/funsd_train	90.54	74
mix-holdout/hateful_memes	82.28	79
mix-holdout/iconqa	87.23	94
mix-holdout/infographic_vqa	80.65	93
mix-holdout/mind2web	79.17	120
mix-holdout/mind2web_k	76.11	113
mix-holdout/options_in_image	68.00	100
mix-holdout/scienceqa	90.00	120
mix-holdout/screen2words	91.11	90
mix-holdout/synthetic	85.83	120
mix-holdout/tallyqa	78.02	91
mix-holdout/visual7w	89.00	100
mix-holdout/vsr	87.64	89
Unweighted mean across the 18 directories above	85.15	—

The mean treats each listed directory equally; paired variants are separate directories.

Speed and fast mode

Mode	Image cap	Depth	Accuracy	Worst-case p95	Median p50	Peak GPU memory
Accurate (default)	1.6 MP	24 layers	85.6% DEV	377.3 ms	66.0 ms	1.86 GiB
Fast	0.26 MP	16 layers	81.9% DEV	123.2 ms	44.6 ms	1.6 GiB

_{Release loader only (qev.QEV.load + predict, no CUDA graphs or custom kernels), RTX 4060 Ti 8 GB, batch 1, warm. 47 request profiles (text and image requests with up to 16 options), 3 warm-up and 30 timed runs each. Worst-case p95 is the highest per-profile p95; median p50 is the median over profiles of each profile's p50. Peak GPU memory is PyTorch's peak allocation. Latency and memory were measured on the previous release, which has the same architecture, configuration and runtime path as this version; a re-measurement for this version was discarded because the control run did not reproduce under desktop load. DEV is the internal selection set (3,000 rows).}

Fast mode exits at layer 16 of 24 with a distilled head (exit16.safetensors) and caps images at 0.26 MP. At the same 0.26 MP input, the layer-16 head matches full depth on DEV (exit vs full: fam-text 81.0% vs 81.0% (n=485); nlvr2 99.2% vs 100.0% (n=128); fam-image 84.2% vs 85.0% (n=400); mix 79.2% vs 79.2% (n=1859)). The drop against accurate mode comes from the lower resolution and is largest on document, chart and screen sources. No index is reported for fast mode.

Calibration and abstention

Per-type temperatures and abstain thresholds use the DEV selection rows, with image/group-disjoint halves: temperature on one half, policy on the other. The policy targets at most 15% error with at least 20 accepted questions. These are internal DEV estimates. Serving calibration is fitted in accurate mode and shared with fast mode, as in the published releases. Fast mode is evaluated with those settings; a separate fast-mode fit is retained for inspection and is not applied to serving. A response still carries its answer when the separate abstain flag is set.

Type	Temperature rows	Policy rows	Temperature	Threshold	Status
choice	1392	1296	1.0292	0.212789	fitted
score	87	79	0.707946	0.395743	fitted
noul	66	80	0.749894	0.690195	fitted

On the held-out policy half, choice questions (1,296) have ECE 0.038 before and 0.039 after temperature scaling, with 100% coverage at 14.0% accepted error.

Architecture

• Backbone: Qwen3.5-0.8B language model (hidden size 1024, 24 layers: 18 GatedDeltaNet + 6 full attention).
• Vision: Qwen3.5 ViT encoder (frozen).
• Readout: option probabilities read from the hidden states of the option tokens.
• Reasoning slots: 4 slots plus a result marker.
• Binding head, per-type temperature calibration and an abstain decision (fitted on DEV; see Calibration and abstention).
• Single forward pass, no generated tokens.
• Fast mode: a distilled exit head reads the options at layer 16 of 24 (exit16.safetensors); images are capped at 0.26 MP.
• Fine-tuning: LoRA (rank 8, alpha 16) on the language model plus trained readout and binding heads; the vision encoder and the other base weights are frozen (board category: LoRA + head).
• Weights are shipped merged (LoRA folded into the base weights).

Training data

The following inherited source families are documented by the old card. The v3 distractor generators and exclusions are described under What changed.

Coverage mix (train split, 18 sources). Each source contributes items from its train split. 5% of each source (up to 120 items) is held out as a selection set and is never trained on.

Source	Split
AI2D	train
A-OKVQA	train
ChartQA	train
CORD	train
DocVQA	train
FUNSD	train
Hateful Memes	train
IconQA	train
InfographicVQA	train
Mind2Web	train
Mind2Web (keyed variant, mind2web_k)	train
Options-in-image	train
ScienceQA	train
Screen2Words	train
Synthetic (internal)	train
TallyQA	train
Visual7W	train
VSR	train

Additional sources

• NLVR2 train split (teacher-filtered targets). The NLVR2 dev split is used only for checkpoint selection.
• Replay of internal decision units (image and text families, train split), with teacher outputs as targets.
• Teacher outputs: option-probability distillation from QEV 0.2.1, the earlier QEV-2B release (ken-jo/qev at revision 0d2d13f): a LoRA (rank 8, alpha 16) with a condition-bound binding head on Qwen3.5-2B. The same teacher was used for QEV-2B and QEV-0.8B; it is not the released QEV-2B checkpoint.

Selection: DEV split only. Public rows are not used in training or selection.

v3 training data. 51,091 coverage items (49,275 trained, 1,816 held out for selection) from the 18 sources above, plus NLVR2 (2,048 rows) and decision replay (4,658 rows) in each pass. A fresh audit of all 46,971 training images found no image within dHash distance 4 of the 11,542 evaluation images or the 1,636 DEV images, and no gold answer was changed by the rebuild.

Training. Start: the QEV-0.8B base checkpoint used by the previous release (Qwen3.5-0.8B distilled from the QEV 0.2.1 teacher). Two passes over the v3 mix (5,419 optimizer steps each; the second pass restarts the optimizer from the end of the first). The selected checkpoint is the end of pass 2, the highest DEV selection score of the saved checkpoints. LoRA rank 8, alpha 16, merged into model.safetensors with FP32 accumulation.

Fast-mode head. The layer-16 head is distilled from this checkpoint's full-depth outputs, with the original probe recipe and data: a seeded sample of the earlier coverage mix and decision replay (20,000 rows), KL to the teacher plus 0.3 gold cross-entropy, 6 epochs. It does not change the backbone.

Quickstart

The 0.8B checkpoint needs the qev source (the published qev==0.2.1 package loads QEV-2B 0.2.1 only).

python -m pip install "qev @ git+https://github.com/ken-jo/qev" huggingface_hub

from huggingface_hub import snapshot_download
import qev
from qev import DecisionRequest

path = snapshot_download("ken-jo/QEV-0.8B")
model = qev.QEV.load(path, device="cuda")               # accurate: full depth, up to 1.6 MP per image
# model = qev.QEV.load(path, device="cuda", mode="fast")  # fast: exit at layer 16, 0.26 MP
request = DecisionRequest.model_validate({
    "state": {
        "text": "Is this photo of a parked car blocking a driveway?",
        "images": [{"path": "photo.jpg"}],
    },
    "questions": {
        "blocking": {
            "type": "choice",
            "instructions": "Does the car block the driveway?",
            "criteria": {"yes": "The car blocks the driveway.", "no": "The driveway is clear."},
        }
    },
})
print(model.predict(request)["answers"])

Note on the name: the PyPI package qev is also the import name used by OneJev (from qev import Client). Install one of them per environment.

Limitations

• Public numbers are self-scored after DEV-only checkpoint selection. Private scores are unknown until the board re-scores this version.
• Contamination: a post-hoc audit compared every v3 coverage-mix training image with every evaluation image. Candidates were dHash distance <= 10, confirmed by 64x64 grayscale mean absolute difference <= 6/255. It flagged 52 evaluation images (27 mind2web, 17 mmmu_pro_vision, 2 charxiv, 2 hateful_memes, 2 blink, 1 funsd, 1 winoground). On inspection, most flags are low-texture images such as near-blank web pages and plain backgrounds. The genuine matches are Hateful Memes base photos reused with different captions (one also appears in Winoground) and a FUNSD blank form template. Rescoring without all 65 affected rows changes the index from 52.70 to 52.76. NLVR2, the replay units and the inherited base training were not re-audited. If you find an overlapping item, please report it.
• Lower than the previous release on some public benchmarks: BLINK 39.1 -> 30.9; InfographicVQA 75.4 -> 70.5; Moderation (Hateful Memes) 31.9 -> 28.1; MMMU-Pro vision 9.6 -> 7.8; CV-Bench 60.0 -> 58.6. Part of the InfographicVQA drop is expected, since the earlier training format matched that benchmark's construction.
• The model returns probabilities, not guarantees. Calibration can fail on other domains; refit it on your own DEV data where abstention matters.
• The model scores only caller-supplied options and evidence. Evaluation focuses on English.
• This release does not include a 4B model; the original project used one RTX 4060 Ti 8 GB.

License

Apache-2.0, the same license as the base model. The base model Qwen3.5-0.8B is subject to its upstream license.

The training data is not redistributed with the model. For the third-party sources, ken-jo/qev-data (vision-mix-2026-10/) publishes item ids, gold labels and hashes, plus the build scripts, for the earlier mix; the v3 build is not published there yet. It also publishes the synthetic set in full and the NLVR2 teacher targets. Each training source keeps its own terms; some, such as FUNSD and Hateful Memes, restrict how their data may be used. Check those terms before reusing the data or retraining on it.

Citation

@misc{jo2026qev08b,
  title        = {QEV-0.8B: A Calibrated Multimodal Decision Model},
  author       = {Jo, Ken},
  year         = {2026},
  howpublished = {Hugging Face model card},
  url          = {https://huggingface.co/ken-jo/QEV-0.8B}
}