docs/BENCHMARKS.md

Standard One — full benchmark report

Note. The Standard One 3B v2 figures are on its model card. The benchmark figures in this report are still those of v1.1 and will be updated when both sizes are on v2; the training-data provenance and overlap-audit sections already describe the v2 weights.

This report covers both released systems, Standard One 8B (StandardThinking/StandardOne-8B / -LoRA) and Standard One 3B (StandardThinking/StandardOne-3B / -LoRA). It is our own measurement, not an official JevBench leaderboard score. The only comparisons in this report are: the untuned base model (accuracy, see § Decision quality vs. the untuned base), Qwen-based models served on the same H200 stack (speed only, untuned, no accuracy comparison implied), and one authorized peer comparison against TypeSafe Jev 1.13 (see § Measured against Jev 1.13).

Contents

• Suite name key
• Scope
• Measurement paths: offline HF runner vs. served endpoint
• Text suites
• Served parity
• Public classification and decision suites
• Latency and throughput
• Measured against Jev 1.13
• Calibration
• Decision quality vs. the untuned base
• Data provenance and licences
• Training data provenance
• Public-overlap audit and disclosure
• Evidence file hashes

Suite name key

The model cards use readable names for four internal evaluation suites; this report keeps the original identifiers so figures trace back to release-manifest.json and evidence/. Mapping: judge-proxy-v1 = judge proxy (600 questions: routing + answer adequacy); internal-eval-v2-realistic = realistic transfer set (600); distprob-val-v2 = stated-distribution probability (1,036); internal-eval-hardproxy-v1 = hard proxy (600). JevBench's own public-tier names (easy / standard / hard) are used as-is in both places.

Scope

• Both candidates: merged BF16 weights, LoRA r=16/α=32 on q/k/v/o/gate/up/down of the language model only (vision tower + projector excluded), same five-stage training recipe on the identical final training mixture (34 cohorts, 382,576 rows) — see MODEL_CARD_8B.md / MODEL_CARD_3B.md § Training for the recipe and per-cohort breakdown, and release-manifest.json for the exact per-cohort row counts and hashes. The 3B is a smaller base/adapter pair, not a reduced-data variant. Image input is supported (the Pixtral vision tower and projector are retained; images are passed as data URLs), and image-side data was included in the final training stage — this report does not detail image or content-moderation training or results further.
• Release serving configuration for both: native wording, no system prompt (--prompt-wording native --native-system-prompt none), single option order, server-side default temperature T = 1.65 (8B) / T = 1.55 (3B) — both refit directly on served probabilities (see § Calibration). See QUICKSTART.md for the exact commands.
• What is not covered here: any official JevBench sealed-set score (we have none yet), any additional peer-system comparison beyond Jev 1.13, and image/content-moderation training detail beyond the one sentence above.

Measurement paths: offline HF runner vs. served endpoint

Two different measurement paths appear throughout this report and are not interchangeable:

1. Offline HF runner: transformers-based scoring harness, computes label-logit probabilities directly from a loaded checkpoint (adapter or merged), no network serving stack involved.
2. Served endpoint: stock SGLang 0.5.20 + jev-adapter, POST /v1/systemone, the same path an external caller would use — see QUICKSTART.md for how to reproduce this endpoint.

These two paths do not agree exactly. Served probabilities differ measurably from offline-runner probabilities for both candidates: mean total-variation distance ≈0.07–0.09 on the hard tier, with 8–11 of 111 argmax label changes, even though the rendered prompts were verified byte-identical between the two paths. Treat served numbers as authoritative and offline numbers as indicative.

Temperature is final and served-path-fit for both candidates: T = 1.65 (8B) and T = 1.55 (3B) were refit directly on served probabilities from the running endpoint, on internal-eval-hardproxy-calib-v1 + distprob-calib-v2, no system prompt. This supersedes an earlier offline-harness fit (T = 1.55 / T = 1.5).

Most of the offline text-suite table below reflects the with-system-prompt configuration, while the release ships without the system prompt. Where we independently re-measured the no-system-prompt configuration (hard, original, judge-proxy-v1, internal-eval-v2-realistic, distprob-val-v2, easy), both columns are shown.

Text suites

Standard One 8B

Suite	n	With system prompt	No system prompt (shipped)	Notes
JevBench public easy	48	100.00 %	100.00 %
JevBench public original	72	95.83 %	97.22 %
JevBench public hard	111	51.35 %	52.25 %	never used for training or T-fitting; informed the prompt-wording choice
judge-proxy-v1 (routing + adequacy)	600	89.00 %	88.83 %	leaderboard judge text not public; proxy only
internal-eval-v2-realistic	600	91.33 %	90.00 %
distprob-val-v2	1,036	80.89 %	81.56 %
internal-eval-v1	600	76.50 %	not separately re-measured
internal-eval-hardproxy-v1	600	53.00 %	not separately re-measured	hard-tier proxy
games-harness-eval-v1	360	96.94 %	not separately re-measured	othello/chess/tic-tac-toe/snake/graph, text
tetris-drought-eval-v1 (text)	120	67.50 %	not separately re-measured

Standard One 3B

Suite	n	With system prompt	No system prompt (shipped)	Notes
JevBench public easy	48	100.00 %	100.00 %
JevBench public original	72	93.06 %	93.06 %	unchanged with/without system prompt
JevBench public hard	111	48.65 %	46.85 %	never used for training or T-fitting; informed the prompt-wording choice
judge-proxy-v1 (routing + adequacy)	600	89.00 %	88.83 %
internal-eval-v2-realistic	600	90.00 %	90.67 %
distprob-val-v2	1,036	81.18 %	79.83 %
internal-eval-v1	600	71.67 %	not separately re-measured
internal-eval-hardproxy-v1	600	43.50 %	not separately re-measured	hard-tier proxy
games-harness-eval-v1	360	96.39 %	not separately re-measured
tetris-drought-eval-v1 (text)	120	68.33 %	not separately re-measured

Served parity

Both offline and served measurement paths were run; served numbers below are authoritative, on the final served-path temperature fit:

	Standard One 8B (no system prompt, T=1.65)	Standard One 3B (no system prompt, T=1.55)
easy (48)	100.00 %	100.00 %
original (72)	94.44 %	88.89 %
hard (111)	54.95 %	45.95 %
judge-proxy-v1	90.50 %	89.00 %
internal-eval-hardproxy-v1	52.50 %	42.17 %
internal-eval-v2-realistic	90.33 %	90.83 %
distprob-val-v2	81.37 %	80.50 %
Calibration axis (leaderboard formula)	78.1	73.3

As noted above, served numbers differ somewhat from the corresponding offline numbers for both candidates (mean TV ≈0.07–0.09 hard tier, 8–11/111 argmax changes) — these are the figures used for the Accuracy does not depend on temperature, so the earlier T=1.0-forced served run (before the temperature refit) reported the same easy/original/hard accuracy: 8B 100.00/94.44/54.95, 3B 100.00/88.89/45.95 no-system-prompt.

Public classification and decision suites

400 cases per suite, seed 13, sampled from public test splits, one request per case, served endpoints (merged BF16, SGLang 0.5.20, jev-adapter, native wording, no system prompt, single option order), measured 25 September 2026. Standard One only; no comparison column.

Suite	Standard One 8B	Standard One 3B
AG News (4 topics)	84.2 %	83.0 %
DAIR Emotion (6 labels)	56.5 %	58.8 %
Typed decisions (400 cases, 2,000 decisions)	69.3 %	69.0 %
MASSIVE intent, 20 options, 9 languages, mean (en 88.0 / 85.5, ko 88.2 / 83.8)	85.6 %	83.3 %
XNLI, 3 labels, 6 languages, mean	60.1 %	62.0 %
Email spam	92.2 %	93.0 %
Phishing	86.0 %	89.8 %
Moderation (toxic vs benign)	85.0 %	81.5 %
Model routing (domain)	93.0 %	83.7 %

Option-order flip rate (share of cases whose answer changes under a permuted option order) ranges from 3.3 % (AG News) to 13 % (MASSIVE en) for the 8B. Full per-language and per-workflow numbers, macro-F1 and ECE: public-classification-suites.md.

Latency and throughput

Same H200, SGLang 0.5.20, jev-adapter, 242-decision profile, serial, ≈280 input tokens/decision. Qwen checkpoints are served untuned, as a speed-only reference — no accuracy comparison is implied.

Model	p50	p95	Input tokens
Standard One 3B	22.6 ms	33.2 ms	≈280
Standard One 8B	25.8 ms	41.9 ms	≈280
Qwen3-8B (untuned)	28.5 ms	57.2 ms	278
Qwen3.5-4B (untuned)	48.8 ms	72.7 ms	283

242-decision profile, one H200, no system prompt:

	Standard One 8B	Standard One 3B
Raw p50 / p95	25.8 ms / 41.9 ms	22.6 ms / 33.2 ms
Leaderboard-adjusted (×2 + 0.15 s) p50 / p95	0.202 s / 0.234 s	0.195 s / 0.216 s
Speed axis	93.3	93.7
Mean input tokens/decision	280	280 (hard tier alone: 1,242)
With system prompt: mean tokens / Speed axis	803 / —	803 / 93.6

Throughput at concurrency 8, same 242-decision/280-token profile: ≈40 decisions/s (32.3k tokens/s) on one H200, measured for the 8B only. On the heavier hard+standard, 1,322-token/request mix (below): 29.3k tok/s at concurrency 1 → 39.5k tok/s at concurrency 64.

*Standard One 8B merged BF16, SGLang 0.5.20, one H200, hard+standard mix at 1,322 tokens/request — a heavier, different profile than the 280-token/decision figure above, so the two throughput numbers (37.0k tok/s here at concurrency 8 vs. 32.3k tok/s above) are not directly comparable; both are genuine measurements of the same hardware under different request mixes. 8B only.

Concurrency	Throughput (k tok/s)	p50 (ms)
1	29.3	31
8	37.0	282
32	39.3	1,033
64	39.5	1,998

Measured against Jev 1.13

The one peer comparison in this report: we measured TypeSafe Jev 1.13 (typesafe/jev-1.13-20260917) through OpenRouter's /v1/systemone, on the same items, same option order, raw probabilities as returned (no temperature applied), 3,067 questions, 0 failures, measured 24 September 2026 from the same host used for our own runs.

Suite (n)	Jev 1.13	Standard One 8B	Standard One 3B
distprob-val-v2 (1,036)	72.97 %	81.37 %	80.50 %
internal-eval-v2-realistic (600)	86.67 %	90.33 %	90.83 %
judge-proxy-v1 (600)	90.50 %	90.50 %	89.00 %
JevBench easy (48)	100.00 %	100.00 %	100.00 %
JevBench standard (72)	98.61 %	94.44 %	88.89 %
internal-eval-hardproxy-v1 (600)	54.83 %	52.50 %	42.17 %
JevBench hard (111)	72.07 %	54.95 %	45.95 %

Jev leads on the public hard tier and the hard proxy; Standard One leads on stated-distribution probability and the realistic transfer set; judge and easy tie.

Calibration on the same items, Jev raw (no temperature applied):

	Jev 1.13	Standard One 8B	Standard One 3B
Hard-tier ECE (10-bin, top label)	0.099	0.231 raw (0.147 at T=1.65)	0.286 raw (0.198 at T=1.55)
Mean TV to stated distributions	0.192	0.109 raw (0.145 at T)	0.112 raw (0.139 at T)

Latency, same 242-decision profile, same host:

	p50	p95	Input tokens/decision
Jev 1.13 (hosted API via OpenRouter)	227 ms	321 ms	≈538 (OpenRouter-reported)
Standard One 8B	25.8 ms	41.9 ms	≈280
Standard One 3B	22.6 ms	33.2 ms	≈280

Jev's latency includes network round-trip to a hosted API; Standard One's is local-GPU serial on the same host.

Calibration

• T fit (release, no system prompt, served path, final): 8B T = 1.65, 3B T = 1.55, both refit directly on served probabilities from the running endpoint, on internal-eval-hardproxy-calib-v1 + distprob-calib-v2. This is the final, shipped configuration.
• Historical reference (offline-harness fit, superseded): 8B T = 1.55 (no system prompt) / T = 1.45 (with system prompt, merged-checkpoint offline check, hard-tier ECE 0.269 at T=1, 0.206 at T=1.45); 3B T = 1.5 (no system prompt) / T = 1.35 (with system prompt, merged-checkpoint offline check, hard-tier ECE 0.269 at T=1, 0.212 at T=1.35). These predate the served-path refit and are kept for record only.
• Calibration axis (leaderboard formula): 8B 78.1, 3B 73.3 (see § Served parity).
• Distribution (TV) fidelity, distprob-val-v2, offline, with system prompt, at T=1: 8B mean TV ≈0.103; 3B mean TV ≈0.110. See § Measured against Jev 1.13 for the served-path TV comparison against Jev.

Decision quality vs. the untuned base

Accuracy change versus the untuned Ministral 3 base, same prompt, same offline runner, no system prompt, T=1. Public JevBench tiers are the 231 public items; the other suites are held-out proxies built from public data. The hard tier is placed last on purpose (see below).

The only base-model comparison in this report. On the public hard tier, the untuned Ministral 3 8B base is measured ahead of Standard One 8B (60.4 % vs. 52.3 %, a 9-of-111-item gap) — hard is 30 % of the v1.3.0 Intelligence axis, so this is material, not a rounding artifact. All rows below come from one run of the same offline runner (native wording, no system prompt, T=1, single option order); the untuned 3B base is behind on every suite except the public hard tier (by 1 of 111 items), the untuned 8B base is ahead only on the public hard tier.

Suite	Base 8B	Standard One 8B	Base 3B	Standard One 3B
distprob-val-v2 (stated-distribution probability, 1,036)	34.85 %	81.56 % (+46.7)	31.56 %	79.83 % (+48.3)
internal-eval-v2-realistic (realistic transfer set, 600)	72.83 %	90.00 % (+17.2)	62.50 %	90.67 % (+28.2)
JevBench public standard tier (72)	79.17 %	97.22 % (+18.0)	70.83 %	93.06 % (+22.2)
judge-proxy-v1 (routing + adequacy, 600)	79.33 %	88.83 % (+9.5)	63.17 %	88.83 % (+25.7)
JevBench public easy tier (48)	100.00 %	100.00 % (+0.0)	97.92 %	100.00 % (+2.1)
JevBench public hard tier (111)	60.36 %	52.25 % (-8.1)	47.75 %	46.85 % (-0.9)

Data provenance and licences

Source	Licence
Every synthetic cohort (deterministic generators + visible checkers; subset teacher-reviewed)	Apache-2.0 (ours)
Base models mistralai/Ministral-3-8B-Instruct-2512(-BF16) / -3B-	Apache-2.0
jev-adapter server code	Apache-2.0
JevBench harness + public tiers (evaluation only, commit fd51755e…)	MIT

All training-cohort details (per-cohort row counts, sha256) are in § Training data provenance below and in release-manifest.json, identical for both candidates.

Training data provenance

The following cohort families make up the 46-cohort, 520,754-row training mixture used by both Standard One 8B and Standard One 3B. Apart from the public-dataset items listed below, every row is synthetic (deterministic generators with visible checkers; a subset was teacher-reviewed) or format-augmented. No JevBench evaluation scenario appears in any training cohort.

Cohort family	What it covers	Licence
Document & field normalisation (document-v3, exact, field-roles, synthetic-en)	Extracting and normalising fields from documents and records	Apache-2.0 (ours)
Judge / routing & adequacy (judge-proxy-v1, judge-realistic-train, judge-multilingual-train)	Routing decisions and answer-adequacy checks	Apache-2.0 (ours)
Stated-distribution probability (distprob-scale30k, distprob-multilingual, distprob-pilot2k, narrative-prob-train, probability)	Estimating a probability distribution over stated outcomes	Apache-2.0 (ours)
Verification & scoring (noul-score-10k)	Yes/no verification and ordinal scoring	Apache-2.0 (ours)
Adequacy-rubric style (hardstyle-train)	Hard-style adequacy questions (see § Public-overlap audit and disclosure)	Apache-2.0 (ours)
Abstention & robustness (abstain, paraphrase, trap, hard-natural)	Abstaining when the state does not settle the question, paraphrased and adversarial wording, harder natural-language decisions	Apache-2.0 (ours)
Game & board-state (games-text-train, games-harness-train, games-harness-multilingual)	Othello, chess, tic-tac-toe, snake and graph-problem decisions	Apache-2.0 (ours)
Tetris board-state (games-text-train-tetris, games-harness-tetris, tetris-drought-train)	Tetris-specific board-state decisions	Apache-2.0 (ours)
Synthetic and format-augmented decision data (v2)	Decision items in varied layouts	Apache-2.0 (ours)
Public-dataset decision items	Decision items converted from public datasets (table below)	Each dataset's own licence (table below)
JevBench public tiers (evaluation only, not training)	Easy / standard / hard held-out accuracy measurement	MIT

Public datasets used (train splits where a dataset has one; CUAD: its single release file). Labels come from the datasets; distractor options and derived labels are generated by code. Licences as stated by each dataset; attribution to the dataset authors.

Dataset / upstream id	Licence	Cohort
SQuAD 2.0 rajpurkar/squad_v2	CC BY-SA 4.0	pub-abstain-v1
ARC allenai/ai2_arc	CC BY-SA 4.0	pub-breadth-v1
BoolQ google/boolq	CC BY-SA 3.0	pub-breadth-v1
CommonsenseQA tau/commonsense_qa	MIT	pub-breadth-v1
HellaSwag Rowan/hellaswag	MIT	pub-breadth-v1
Banking77 PolyAI/banking77	CC BY 4.0	pub-classify-v1
Bias in Bios LabHC/bias_in_bios	MIT	pub-classify-v1
Bitext customer support bitext/Bitext-customer-support-llm-chatbot-training-dataset	CDLA-Sharing-1.0	pub-classify-v1
CLINC150 clinc/clinc_oos	CC BY 3.0	pub-classify-v1
Amazon Counterfactual mteb/amazon_counterfactual	CC BY 4.0	pub-classify-v1
DBpedia-14 fancyzhx/dbpedia_14	CC BY-SA 3.0	pub-classify-v1
Dolly 15k databricks/databricks-dolly-15k	CC BY-SA 3.0	pub-classify-v1, safety-judge-v1
GoEmotions google-research-datasets/go_emotions	Apache-2.0	pub-classify-v1
MASSIVE AmazonScience/massive	CC BY 4.0	pub-classify-v1
Twitter Financial News Sentiment zeroshot/twitter-financial-news-sentiment	MIT	pub-classify-v1
HelpSteer3 nvidia/HelpSteer3	CC BY 4.0	pub-judge-v1
HelpSteer2 nvidia/HelpSteer2	CC BY 4.0	pub-judge-v1
2WikiMultihopQA framolfese/2WikiMultihopQA	Apache-2.0	pub-multihop-v1
HotpotQA hotpotqa/hotpot_qa	CC BY-SA 4.0	pub-multihop-v1
MuSiQue dgslibisey/MuSiQue	CC BY 4.0	pub-multihop-v1
QASC allenai/qasc	CC BY 4.0	pub-multihop-v1
DROP ucinlp/drop	CC BY-SA 4.0	pub-numeric-v1
GSM8K openai/gsm8k	MIT	pub-numeric-v1
TempReason tonytan48/TempReason	CC BY-SA 3.0	pub-numeric-v1
MultiNLI nyu-mll/multi_nli	OANC / CC BY-SA 3.0 / CC BY 3.0	pub-paraphrase-v1
PAWS google-research-datasets/paws	Google terms, free for any purpose	pub-paraphrase-v1
PAWS-X google-research-datasets/paws-x	Google terms, free for any purpose	pub-paraphrase-v1
SNLI stanfordnlp/snli	CC BY-SA 4.0	pub-paraphrase-v1
WANLI alisawuffles/WANLI	CC BY 4.0	pub-paraphrase-v1
ContractNLI stanfordnlp.github.io/contract-nli	CC BY 4.0	pub-rules-v1
CUAD theatticusproject/cuad	CC BY 4.0	pub-rules-v1
ShARC sharc-data.github.io	CC BY-SA 3.0	pub-rules-v1
Jailbreak classification jackhhao/jailbreak-classification	Apache-2.0	safety-judge-v1
Prompt injections deepset/prompt-injections	Apache-2.0	safety-judge-v1
Aegis AI Content Safety 2.0 nvidia/Aegis-AI-Content-Safety-Dataset-2.0	CC BY 4.0	content-safety-text-v1
Jigsaw Toxic Comment Classification (mirror of the Kaggle data) tasksource/jigsaw_toxicity	CC0 (data); comment text CC BY-SA 3.0 (Wikipedia)	content-safety-text-v1
Measuring Hate Speech ucberkeley-dlab/measuring-hate-speech	CC BY 4.0	content-safety-text-v1
Image safety classes deepghs/nsfw_detect	MIT	content-safety-image-v1, content-safety-image-2d-v1

Public-overlap audit and disclosure

Run on the 46-cohort, 520,754-row training data that both candidates train on:

• 0 exact state matches — no training row's state/scenario duplicates any public JevBench item.
• 181 exact instruction matches: hardstyle-train-v1 (87 of 3,900 rows) and hardstyle-train-v2 (94 of 3,879 rows) contain rows whose entire instruction is byte-identical, after normalization, to one public jevbench-hard instruction — a 58-character generic adequacy-rubric question ("does the response fully and correctly satisfy the request?"), reused as the complete instruction rather than as a substring.
• 24 distinct shared word 8-grams (20,400 rows), all in known families: the rubric sentence above (also in judge-realistic-train-v3, judge-multilingual-train-v1, judge-proxy-v1 and the format-augmented cohort), an insurance "cancellation takes effect" boilerplate clause, a "coverage reviewer" claims-adjudication phrase, and generic contract boilerplate (for example "controlled by or under common control with") in one public-dataset cohort.
• All other 44 cohorts: 0 exact state and 0 exact instruction matches.

These 181 rows are kept and disclosed here, not regenerated. They are 0.03 % of the mixture, and the exposure is limited to one hard-tier question's generic wording, never a state or an answer.

Evidence file hashes

evidence/ (8B):

File	sha256
adapter_model.safetensors	9551923c162970ea8bc2c3a5faf3ea3762b48591c9f460064e531d09427fa41a
adapter_config.json	545101ebcafd9fb90723284518d744ca01de46534e04a5dfd4e8f86bc4de0ef2
manifest.json	62bd99b6d4a49926840c258874a3b804a868626761fa57a39e8bfe10e3192904
MERGE_REPORT.json	1f39839e969d874d397d3842eb702f39516ec784f546ce18220e9836bffb2793
calibration-leaderboard.json	a28ada7681a3ae8edcce3f7885dec63ecfb9bc6a9c9794204fa040ae1dd1065f
latency-report-shared-gpu.json	f6734e5cb4a210e9284632ef97c922876f4aaced3106461ec9cede475a640fa9
served-easy-report.json	f68aa8f1d1bbc60b045592713188563643e07d0e8d8337df4df9387bf87254f3
served-original-report.json	b87e0a9a48ebb9fa933fe0db470b506cbe1850824abf65dfebf6ad6c5f58cdda
served-hard-report.json	c0eb0fde38c44b1f3f62f6a79cc45c3e89cad1f782d1c6cc40526f69e8595ff6

evidence/ (3B):

File	sha256
adapter_model.safetensors	7bc8bfff4be52e0c9c6f605faf1eebc5e8a966ce103793c7f6970e29056ef5d3
adapter_config.json	43b99a646a535b15e0f8bceebe2cfff03dd8cc519f5e21f68856b885657d7179
manifest.json	ecfd448e212e09df55c86c93233abe857282b8d9d6154dceddb681663b6ebea2
calibration-leaderboard.json	de13ebd925123ade9f065d7ae3a3955e7f55ddc29ce1d598624039fcdc75567e
offline-native-result.json	2a53c2dc7c82dcf971110e2f788c750c63a09c1c0ea844a371e6abdd087adf24

docs/assets/ (figures referenced by this report):

File	sha256
02-latency-vs-qwen.png	d383eb64f146c6be069455d1631e10ef120ffb534f0000f7470e715e9fe99872
04-throughput.png	bdb8cb392f6f288fe7ce7fde78e39759cd688340be4fe605890dc79c58e83f73

The 3B evidence set does not yet include a separate MERGE_REPORT.json or served-endpoint report files; the merge-tensor numbers (182 tensors changed, max |Δ| 0.0023) and served numbers (§ Served parity) are recorded narratively in MODEL_CARD_3B.md.