QEV-2B v3: diversified-distractor retrain (weights, heads, card)

Browse files

Files changed (4) hide show

README.md +105 -60
exit16.safetensors +1 -1
head.safetensors +1 -1
model.safetensors +1 -1

README.md CHANGED Viewed

@@ -17,95 +17,140 @@ tags:
 - abstention
 - early-exit
 ---
 # QEV-2B
 **A 2B open multimodal decision model: text and image in, calibrated option probabilities out, in one forward pass with no generated tokens.**
-[![JEV Decision Index](https://img.shields.io/badge/JEV%20Decision%20Index%20vision-61.99%20(self--scored)-2563eb)](#results)
 [![License](https://img.shields.io/badge/license-Apache--2.0-blue)](#license)
 [![Base model](https://img.shields.io/badge/base-Qwen3.5--2B-6b7280)](https://huggingface.co/Qwen/Qwen3.5-2B)
 [![Technical report](https://img.shields.io/badge/technical%20report-PDF-b91c1c)](https://huggingface.co/datasets/ken-jo/decision-index-results/blob/main/reports/QEV-technical-report.pdf)
 QEV-2B is the full-size member of the QEV family. It is built on Qwen3.5-2B. You supply the evidence (text, one or more images) and the candidate options with their meanings. The model returns a probability for each option, a typed answer, and an abstain signal.
-Architecture, training, evaluation protocol, calibration, speed, ablations and limitations are described in the [technical report](https://huggingface.co/datasets/ken-jo/decision-index-results/blob/main/reports/QEV-technical-report.pdf).
-**Headline (self-scored, pending official run):** On the JEV Decision Index vision board (edition 0.3.2, public part), QEV-2B scores **61.99**, the highest score on the board among models up to 2.3B parameters: 2.84 points above vLLM-SR d3-nano (2.21B, 59.15), and above Intern-Decision-4B (4.66B, 61.69) and Jev-Omni (11.96B, 60.10). It ranks first among models up to 2.3B on five of the eleven benchmarks: RealWorldQA, Mind2Web, KIE, Moderation and R-Bench-M.
 ## Results
 ### Leaderboard comparison
-JEV Decision Index, vision, edition 0.3.2, public part. Board generated 2026-10-16.
 | Model | Params | Index (public) |
 | --- | ---: | ---: |
 | d3-mini | 4.54B | 67.67 |
 | RSI-Jev v6.1 VL 4B | 4.66B | 63.01 |
 | RSI-Jev v5.0 VL 3B | 4.66B | 62.58 |
-| **QEV-2B (self-scored)** | **2.21B** | **61.99** |
 | Intern-Decision-4B | 4.66B | 61.69 |
 | Jev-Omni | 11.96B | 60.10 |
 | d3-nano | 2.21B | 59.15 |
 | Intern-Decision-2B | 2.27B | 52.92 |
 | OneJev1.5-0.8B | 1.11B | 52.32 |
-<sub>Board values are copied as published (JEV Decision Index vision, edition 0.3.2, public part, generated 2026-10-16). The QEV-2B value is **self-scored, pending official run**: public evaluation kit, native resolution up to 1.6 MP per image (no shared multi-image budget), single run, full depth, scored on the exact released files. It is not an official leaderboard entry and will be replaced when the official run is available.</sub>
-### Where QEV-2B leads (self-scored)
-| Benchmark | QEV-2B | Best other model up to 2.3B (board, public) |
-| --- | ---: | --- |
-| Mind2Web | **82.3** | d3-nano (2.21B) 63.7 |
-| KIE | **93.9** | Intern-Decision-2B (2.27B) 84.4 |
-| RealWorldQA | **56.2** | d3-nano (2.21B) 53.1 |
-| Moderation | **29.9** | OneJev1.5-0.8B (1.11B) 24.7 |
-| R-Bench-M | **16.8** | d3-nano (2.21B) 15.5 |
-### Per-benchmark scores
-Index = weighted mean of the benchmarks below. Weights: 1.0 unless noted. R-Bench-M 0.5, MMMU-Pro 0.25.
-| Benchmark | QEV-2B (self-scored) | Best other model up to 2.3B (board) | Weight |
-| --- | ---: | ---: | ---: |
-| CV-Bench | 70.6 | 71.9 (d3-nano) | 1.0 |
-| BLINK | 47.2 | 53.8 (OneJev1.5-0.8B) | 1.0 |
-| RealWorldQA | **56.2** | 53.1 (d3-nano) | 1.0 |
-| CharXiv | 63.4 | 65.2 (d3-nano) | 1.0 |
-| InfographicVQA | 83.1 | 83.6 (d3-nano) | 1.0 |
-| Mind2Web | **82.3** | 63.7 (d3-nano) | 1.0 |
-| Winoground | 65.5 | 69.0 (d3-nano) | 1.0 |
-| KIE | **93.9** | 84.4 (Intern-Decision-2B) | 1.0 |
-| Moderation | **29.9** | 24.7 (OneJev1.5-0.8B) | 1.0 |
-| R-Bench-M | **16.8** | 15.5 (d3-nano) | 0.5 |
-| MMMU-Pro | 15.6 | 17.8 (d3-nano) | 0.25 |
-| **Index** | **61.99** | 59.15 (d3-nano) | |
-Evaluation: public evaluation kit, native resolution up to 1.6 MP per image (no shared multi-image budget), single run, full depth, on the released files (merged weights). 95% bootstrap interval of the index (items resampled within each benchmark, 2,000 replicates): 60.63 to 63.34. With rows that share an image resampled together: 60.59 to 63.43. Benchmarks were not used for checkpoint selection. The previous QEV-2B release (0.2.1) has no score on this board.
 ### Speed and fast mode
-| Mode | Image cap | Depth | Worst-case p95 | Median p50 | Peak GPU memory |
-| --- | ---: | ---: | ---: | ---: | ---: |
-| Accurate (default) | 1.6 MP | 24 layers | 738.6 ms | 135.3 ms | 4.44 GiB |
-| Fast | 0.26 MP | 16 layers | 225.0 ms | 91.6 ms | 4.13 GiB |
-<sub>Release loader only (`qev.QEV.load` + `predict`, no CUDA graphs or custom kernels), RTX 4060 Ti 8 GB, batch 1, warm. 47 request profiles (text and image requests with up to 16 options; the largest is 3,229 input tokens in accurate mode), 3 warm-up and 30 timed runs each. Worst-case p95 is the highest per-profile p95; median p50 is the median over profiles of each profile's p50. Peak GPU memory is PyTorch's peak allocation. In both modes the slowest profile is a 16-option image request near the token limit.</sub>
-Fast mode exits at layer 16 of 24 with a distilled exit head (`exit16.safetensors`) and caps images at 0.26 MP. At the same 0.26 MP input, exiting at layer 16 matched full depth on our internal DEV set: 85.0% versus 84.9% on the coverage-mix selection split (1,859 items), 89.8% versus 90.6% on NLVR2 dev (128), 88.8% versus 89.2% on image diagnostics (400) and 93.0% versus 92.8% on text diagnostics (485). The lower resolution costs accuracy against accurate mode, most on document, chart and screen sources. No index is reported for fast mode, so the numbers above apply to the full-depth mode only. Calibration is fitted for accurate mode and has not been refitted for fast mode.
 ### Calibration and abstention
-Per-type temperatures and abstain thresholds are fitted on the DEV selection split (the split also used for checkpoint selection; benchmarks are never used), on group-disjoint halves: temperature on one half, threshold on the other. Policy: the lowest confidence threshold at which answered questions have at most 15% error, with at least 20 answered. Numbers are on the held-out policy half, accurate mode. A response always carries the answer; `abstained` is a separate flag, so benchmark scores are unaffected.
-| Question type | DEV questions | Temperature | Threshold | Answered | Error on answered | ECE (raw -> calibrated) |
-| --- | ---: | ---: | ---: | ---: | ---: | --- |
-| Choice | 1,296 | 0.84 | 0.31 | 100% | 10.5% | 0.025 -> 0.013 |
-| Score | 79 | 0.43 | 0.52 | 100% | 7.6% | 0.140 -> 0.036 |
-| Yes/no | 80 | 0.49 | 0.58 | 100% | 6.3% | 0.196 -> 0.088 |
-For all three question types the 15% target is met on all DEV questions, so abstention only triggers below the lowest confidence seen on DEV and is rare. The score and yes/no thresholds rest on about 80 DEV questions each (internal text diagnostics) and are less certain than the choice threshold.
 ## Architecture
@@ -123,7 +168,7 @@ For all three question types the 15% target is met on all DEV questions, so abst
 ## Training data
-Every source used in training, read from the training script (`qev_mix.py`) and its training log.
 **Coverage mix (train split, 18 sources).** Each source contributes items from its train split. 5% of each source (up to 120 items) is held out as a selection set and is never trained on.
@@ -154,13 +199,14 @@ Every source used in training, read from the training script (`qev_mix.py`) and
 - Replay of internal decision units (image and text families, train split), with teacher outputs as targets.
 - Teacher outputs: option-probability distillation from QEV 0.2.1, the earlier QEV-2B release ([ken-jo/qev](https://huggingface.co/ken-jo/qev/tree/0d2d13ffb3c392071ea00b6bdb903c4b84dff48d) at revision 0d2d13f): a LoRA (rank 8, alpha 16) with a condition-bound binding head on Qwen3.5-2B. The same teacher was used for QEV-2B and QEV-0.8B; it is not the released QEV-2B checkpoint.
-**Deduplication and contamination audit:** all sources use train splits only (evaluation uses test/validation splits). Ten document/KIE/perception sources were filtered at build time: any item whose image is within Hamming distance 4 (64-bit dHash) of an evaluation image was dropped. A post-hoc audit of every training image against every evaluation image (dHash candidate, confirmed by 64x64 grayscale mean absolute difference <= 6/255) found **37 evaluation items** with a near-identical training image: 30 Hateful Memes (shared base images across the dataset's splits), 6 BLINK, 1 Mind2Web. Mind2Web has no action_uid or annotation_id overlap with the evaluation rows. Rescoring without those 37 items changes the index from 61.99 to 62.09 (Moderation 29.93 -> 30.85, BLINK 47.16 -> 47.07, Mind2Web 82.32 -> 82.44).
-**Teacher cleanup:** a teacher target is kept only where the teacher's top option matches the gold answer. Otherwise only the gold label is trained on.
-**Selection:** checkpoint chosen on held-out DEV data only (coverage-mix holdout, NLVR2 DEV, internal diagnostics). Public benchmarks were not used for selection.
-**Training recipe:** continued from QEV-2B 0.2.1 (its LoRA merged into the backbone) with a fresh LoRA (rank 8, alpha 16) on the coverage mix plus replay, 56k items; the released checkpoint is optimizer step 3,616 (end of the cosine schedule). Resolution-elastic batches (0.26 / 1.05 / 1.64 MP). Answer options are permuted per visit, except for sources whose options refer to letters or boxes in the image (Mind2Web, Mind2Web-keyed, options-in-image).
 ## Quickstart
@@ -197,19 +243,18 @@ Note on the name: the PyPI package `qev` is also the import name used by OneJev
 ## Limitations
-- Weak areas: Moderation (29.9), BLINK (47.2), MMMU-Pro (15.6). QEV-2B trails d3-nano on CV-Bench, CharXiv, InfographicVQA, Winoground and MMMU-Pro, and OneJev1.5-0.8B on BLINK.
-- Scores are self-scored on the public part, single run, up to 1.6 MP per image. The private-part scores are unknown and may differ.
-- Contamination: the training mix includes train splits of several benchmark families (Mind2Web, InfographicVQA, CORD, FUNSD, Hateful Memes). The audit above found 37 near-duplicate evaluation images (0.3% of 12,210 rows); removing them does not lower the score. If you find an overlapping item, please report it.
-- The model returns probabilities, not guarantees. Calibration and abstention are fitted on internal DEV data and can be wrong on other domains; refit on your own data where the abstain decision matters.
-- Visual reasoning is limited; see the evaluation notes in the QEV repository.
-- Evaluation focuses on English.
-- No larger model: training and evaluating a 4B QEV did not fit our hardware (one RTX 4060 Ti 8 GB), so we could not build or measure one. We would welcome compute support to do so.
 ## License
 Apache-2.0, the same license as the base model. The base model Qwen3.5-2B is subject to its upstream license.
-The training data is not redistributed with the model. For the third-party sources, [ken-jo/qev-data](https://huggingface.co/datasets/ken-jo/qev-data/tree/main/vision-mix-2026-10) (`vision-mix-2026-10/`) publishes item ids, gold labels and hashes, plus the build scripts. It also publishes the synthetic set in full and the NLVR2 teacher targets. Each training source keeps its own terms; some, such as FUNSD and Hateful Memes, restrict how their data may be used. Check those terms before reusing the data or retraining on it.
 ## Citation

 - abstention
 - early-exit
 ---
 # QEV-2B
 **A 2B open multimodal decision model: text and image in, calibrated option probabilities out, in one forward pass with no generated tokens.**
+[![JEV Decision Index](https://img.shields.io/badge/JEV%20Decision%20Index%20vision-62.09%20(self--scored)-2563eb)](#results)
 [![License](https://img.shields.io/badge/license-Apache--2.0-blue)](#license)
 [![Base model](https://img.shields.io/badge/base-Qwen3.5--2B-6b7280)](https://huggingface.co/Qwen/Qwen3.5-2B)
 [![Technical report](https://img.shields.io/badge/technical%20report-PDF-b91c1c)](https://huggingface.co/datasets/ken-jo/decision-index-results/blob/main/reports/QEV-technical-report.pdf)
 QEV-2B is the full-size member of the QEV family. It is built on Qwen3.5-2B. You supply the evidence (text, one or more images) and the candidate options with their meanings. The model returns a probability for each option, a typed answer, and an abstain signal.
+Architecture, training, evaluation protocol, calibration, speed, ablations and limitations for the earlier release are described in the [technical report](https://huggingface.co/datasets/ken-jo/decision-index-results/blob/main/reports/QEV-technical-report.pdf).
+## What changed
+This version retrains QEV-2B to reduce format overfitting. Public scores move little; the gain is on DEV items whose options are built differently from the training format.
+- **Diversified distractors.** The earlier training data built each wrong option from the answer to another question about the same image, always with 4 options. The public InfographicVQA rows use the same construction. The v3 mix keeps that scheme for at most 35% of items. The rest use answers from other images, perturbed values and same-image annotations, with 3 to 6 options. 60% of Mind2Web items are re-rendered.
+- **Targeted v3 fixes.** 707 CORD field-presence items now include receipt-text decoys. 157 new upstream train rows (127 Hateful Memes, 30 A-OKVQA) replace rows removed by the near-duplicate filter.
+- **DEV-only selection.** The checkpoint score is the mean of the original-format selection set and a shifted-distractor DEV set. No public evaluation rows are used in training or selection. Training images within dHash distance 4 of any evaluation image are excluded.
+- **Private scores are unknown** until the board re-scores this version.
+| | Previous release | This version |
+| --- | ---: | ---: |
+| Shifted-distractor DEV (9 directories, mean accuracy) | 84.90 | 88.79 |
+| Original-format holdout (18 sources, mean accuracy) | 89.15 | 88.96 |
+| Public vision index (self-scored, same harness) | 61.95 | 62.09 |
 ## Results
 ### Leaderboard comparison
+Historical comparison models are copied from the old card's 2026-10-16 public board (edition 0.3.2). The QEV row is this version's self-score (previous release: 61.95 on the same harness). Row order does not assert this version's rank.
 | Model | Params | Index (public) |
 | --- | ---: | ---: |
 | d3-mini | 4.54B | 67.67 |
 | RSI-Jev v6.1 VL 4B | 4.66B | 63.01 |
 | RSI-Jev v5.0 VL 3B | 4.66B | 62.58 |
+| **QEV-2B (self-scored)** | **2.21B** | **62.09** |
 | Intern-Decision-4B | 4.66B | 61.69 |
 | Jev-Omni | 11.96B | 60.10 |
 | d3-nano | 2.21B | 59.15 |
 | Intern-Decision-2B | 2.27B | 52.92 |
 | OneJev1.5-0.8B | 1.11B | 52.32 |
+### Per-benchmark scores
+Public evaluation only; self-scored, pending the board's re-score. These are post-selection results.
+| Benchmark | Chance-corrected score (%) | Raw accuracy (%) | Rows | Weight |
+| --- | ---: | ---: | ---: | ---: |
+| CV-Bench | 72.07 | 83.07 | 2038 | 1 |
+| BLINK | 46.52 | 65.45 | 961 | 1 |
+| RealWorldQA | 55.33 | 72.67 | 589 | 1 |
+| CharXiv | 65.41 | 74.06 | 532 | 1 |
+| InfographicVQA | 80.87 | 85.66 | 244 | 1 |
+| Mind2Web | 82.73 | 87.05 | 973 | 1 |
+| Winoground | 66.00 | 83.00 | 800 | 1 |
+| KIE (CORD+FUNSD) | 92.52 | 95.46 | 2465 | 1 |
+| Moderation (Hateful Memes) | 32.07 | 66.03 | 1213 | 1 |
+| R-Bench-M | 15.36 | 29.47 | 665 | 0.5 |
+| MMMU-Pro vision | 16.66 | 27.34 | 1730 | 0.25 |
+| **Index** | **62.09** | — | — | — |
+Index is the weighted mean of chance-corrected scores, not the mean raw accuracy.
+### Shift DEV and original-format holdout
+**Shift DEV**
+| Directory | Accuracy (%) | Rows |
+| --- | ---: | ---: |
+| A/chartqa | 92.50 | 400 |
+| B/chartqa | 93.50 | 400 |
+| Bn/chartqa | 93.00 | 400 |
+| A/cord_train | 95.12 | 246 |
+| B/cord_train | 95.53 | 246 |
+| B/infographic_vqa | 89.75 | 400 |
+| B/docvqa | 99.75 | 400 |
+| A/mind2web_all | 83.33 | 60 |
+| B/mind2web_all | 56.67 | 60 |
+| Unweighted mean across the 9 directories above | 88.79 | — |
+The mean treats each listed directory equally; paired variants are separate directories.
+**Original-format holdout**
+| Directory | Accuracy (%) | Rows |
+| --- | ---: | ---: |
+| mix-holdout/ai2d | 90.62 | 96 |
+| mix-holdout/aokvqa | 85.00 | 120 |
+| mix-holdout/chartqa | 90.00 | 120 |
+| mix-holdout/cord_train | 98.33 | 120 |
+| mix-holdout/docvqa | 99.17 | 120 |
+| mix-holdout/funsd_train | 95.95 | 74 |
+| mix-holdout/hateful_memes | 83.54 | 79 |
+| mix-holdout/iconqa | 93.62 | 94 |
+| mix-holdout/infographic_vqa | 84.95 | 93 |
+| mix-holdout/mind2web | 87.50 | 120 |
+| mix-holdout/mind2web_k | 82.30 | 113 |
+| mix-holdout/options_in_image | 79.00 | 100 |
+| mix-holdout/scienceqa | 95.00 | 120 |
+| mix-holdout/screen2words | 93.33 | 90 |
+| mix-holdout/synthetic | 91.67 | 120 |
+| mix-holdout/tallyqa | 80.22 | 91 |
+| mix-holdout/visual7w | 88.00 | 100 |
+| mix-holdout/vsr | 83.15 | 89 |
+| Unweighted mean across the 18 directories above | 88.96 | — |
+The mean treats each listed directory equally; paired variants are separate directories.
 ### Speed and fast mode
+| Mode | Image cap | Depth | Accuracy | Worst-case p95 | Median p50 | Peak GPU memory |
+| --- | ---: | ---: | --- | --- | --- | --- |
+| Accurate (default) | 1.6 MP | 24 layers | 89.5% DEV | 738.6 ms | 135.3 ms | 4.44 GiB |
+| Fast | 0.26 MP | 16 layers | 84.7% DEV | 225.0 ms | 91.6 ms | 4.13 GiB |
+<sub>Release loader only (`qev.QEV.load` + `predict`, no CUDA graphs or custom kernels), RTX 4060 Ti 8 GB, batch 1, warm. 47 request profiles (text and image requests with up to 16 options), 3 warm-up and 30 timed runs each. Worst-case p95 is the highest per-profile p95; median p50 is the median over profiles of each profile's p50. Peak GPU memory is PyTorch's peak allocation. Latency and memory were measured on the previous release, which has the same architecture, configuration and runtime path as this version; a re-measurement for this version was discarded because the control run did not reproduce under desktop load. DEV is the internal selection set (3,000 rows).</sub>
+Fast mode exits at layer 16 of 24 with a distilled head (`exit16.safetensors`) and caps images at 0.26 MP. At the same 0.26 MP input, the layer-16 head matches full depth on DEV (exit vs full: fam-text 91.5% vs 90.9% (n=485); nlvr2 88.3% vs 87.5% (n=128); fam-image 90.0% vs 90.5% (n=400); mix 81.1% vs 81.3% (n=1859)). The drop against accurate mode comes from the lower resolution and is largest on document, chart and screen sources. No index is reported for fast mode.
 ### Calibration and abstention
+Per-type temperatures and abstain thresholds use the DEV selection rows, with image/group-disjoint halves: temperature on one half, policy on the other. The policy targets at most 15% error with at least 20 accepted questions. These are internal DEV estimates. Serving calibration is fitted in accurate mode and shared with fast mode, as in the published releases. Fast mode is evaluated with those settings; a separate fast-mode fit is retained for inspection and is not applied to serving. A response still carries its answer when the separate abstain flag is set.
+| Type | Temperature rows | Policy rows | Temperature | Threshold | Status |
+| --- | ---: | ---: | ---: | ---: | --- |
+| choice | 1392 | 1296 | 0.841395 | 0.296769 | fitted |
+| score | 87 | 79 | 0.398107 | 0.419354 | fitted |
+| noul | 66 | 80 | 0.446684 | 0.513213 | fitted |
+On the held-out policy half, choice questions (1,296) have ECE 0.036 before and 0.018 after temperature scaling, with 100% coverage at 10.0% accepted error.
 ## Architecture
 ## Training data
+The following inherited source families are documented by the old card. The v3 distractor generators and exclusions are described under What changed.
 **Coverage mix (train split, 18 sources).** Each source contributes items from its train split. 5% of each source (up to 120 items) is held out as a selection set and is never trained on.
 - Replay of internal decision units (image and text families, train split), with teacher outputs as targets.
 - Teacher outputs: option-probability distillation from QEV 0.2.1, the earlier QEV-2B release ([ken-jo/qev](https://huggingface.co/ken-jo/qev/tree/0d2d13ffb3c392071ea00b6bdb903c4b84dff48d) at revision 0d2d13f): a LoRA (rank 8, alpha 16) with a condition-bound binding head on Qwen3.5-2B. The same teacher was used for QEV-2B and QEV-0.8B; it is not the released QEV-2B checkpoint.
+**Selection:** DEV split only. Public rows are not used in training or selection.
+**v3 training data.** 51,091 coverage items (49,275 trained, 1,816 held out for selection) from the 18 sources above, plus NLVR2 (2,048 rows) and decision replay (4,658 rows) in each pass. A fresh audit of all 46,971 training images found no image within dHash distance 4 of the 11,542 evaluation images or the 1,636 DEV images, and no gold answer was changed by the rebuild.
+**Training.** Start: QEV 0.2.1 (Qwen3.5-2B + LoRA, merged), as for the previous release. One pass over the v3 mix (3,652 optimizer steps). The selected checkpoint is the end of the pass, the highest DEV selection score of the saved checkpoints. LoRA rank 8, alpha 16, merged into `model.safetensors` with FP32 accumulation.
+**Fast-mode head.** The layer-16 head is distilled from this checkpoint's full-depth outputs, with the original probe recipe and data: a seeded sample of the earlier coverage mix and decision replay (24,658 rows), KL to the teacher plus 0.3 gold cross-entropy, 6 epochs. It does not change the backbone.
 ## Quickstart
 ## Limitations
+- Public numbers are self-scored after DEV-only checkpoint selection. Private scores are unknown until the board re-scores this version.
+- Contamination: a post-hoc audit compared every v3 coverage-mix training image with every evaluation image. Candidates were dHash distance <= 10, confirmed by 64x64 grayscale mean absolute difference <= 6/255. It flagged **52** evaluation images (27 mind2web, 17 mmmu_pro_vision, 2 charxiv, 2 hateful_memes, 2 blink, 1 funsd, 1 winoground). On inspection, most flags are low-texture images such as near-blank web pages and plain backgrounds. The genuine matches are Hateful Memes base photos reused with different captions (one also appears in Winoground) and a FUNSD blank form template. Rescoring without all 65 affected rows changes the index from 62.09 to 62.12. NLVR2, the replay units and the inherited base training were not re-audited. If you find an overlapping item, please report it.
+- Lower than the previous release on some public benchmarks: InfographicVQA 83.1 -> 80.9; R-Bench-M 16.8 -> 15.4; KIE (CORD+FUNSD) 93.9 -> 92.5. Part of the InfographicVQA drop is expected, since the earlier training format matched that benchmark's construction.
+- The model returns probabilities, not guarantees. Calibration can fail on other domains; refit it on your own DEV data where abstention matters.
+- The model scores only caller-supplied options and evidence. Evaluation focuses on English.
+- This release does not include a 4B model; the original project used one RTX 4060 Ti 8 GB.
 ## License
 Apache-2.0, the same license as the base model. The base model Qwen3.5-2B is subject to its upstream license.
+The training data is not redistributed with the model. For the third-party sources, [ken-jo/qev-data](https://huggingface.co/datasets/ken-jo/qev-data/tree/main/vision-mix-2026-10) (`vision-mix-2026-10/`) publishes item ids, gold labels and hashes, plus the build scripts, for the earlier mix; the v3 build is not published there yet. It also publishes the synthetic set in full and the NLVR2 teacher targets. Each training source keeps its own terms; some, such as FUNSD and Hateful Memes, restrict how their data may be used. Check those terms before reusing the data or retraining on it.
 ## Citation

exit16.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:fc61287b393b8e6d4c88621b7839f21a77f02f3be5f3366ce5ce805f135208cc
 size 16880144

 version https://git-lfs.github.com/spec/v1
+oid sha256:dc7fa5a4f9c935c9e46ad5a011f82b8e06fe4cb2ff2e809ddadfba0534036173
 size 16880144

head.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:a0057040e29d834d7c0658102ba29b0b7c11119beeac318bb83f7a8d264200d3
 size 709496

 version https://git-lfs.github.com/spec/v1
+oid sha256:9afbeb27c6864dbfae28026f0cf477ca5ab9a2403cf414cd9f6edcdeb018c82c
 size 709496

model.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:7f1b8dd81ab0ebd219159a5979bf0e9fb9d7d6c3352aa599d31da5559be42df5
 size 4426555160

 version https://git-lfs.github.com/spec/v1
+oid sha256:d75ade07f5cbc089d56ebd1e4efe7724b850e104df6d4bce2d49abd868d26875
 size 4426555160