voves/Qwen3.8-27B-NVFP4

Qwen3.8-27B NVFP4 — Quality Benchmark vs BF16

TL;DR

• Overall accuracy: BF16 91.11% vs NVFP4 90.00% (Δ = -1.11 pp)
• Win/Tie/Loss (across 270 task pairs): BF16-only wins = 9, NVFP4-only wins = 6, both pass = 237, both fail = 18
• Most noticeable degradation: IFEval -13.3 pp (instruction-following) — NVFP4 is worse at honoring explicit format constraints (forbidden words, exact word/sentence counts, prompt repetition).
• Worst case: on one MATH problem NVFP4 enters an infinite loop in reasoning (sqrt(3/2)/3 = sqrt(3/2)/3 = ...) and never produces an answer within 16 384 tokens; BF16 solves the same problem in 4477 tokens.
• Reasoning efficiency: NVFP4 spends on average 30% more tokens on MATH and 160% more on curated logical-reasoning — quantization makes the chain-of-thought more "diffuse".
• On standard benchmarks (MMLU/ARC with thinking, GSM8K, ARC, HumanEval, knowledge QA, translation) the difference falls within sampling noise (≤1 sample out of 30).

Methodology

Generation parameters

• temperature = 0 (greedy / deterministic mode)
• seed = 42 (fixed for reproducibility)
• top_p = 1.0
• Determinism verified separately: 3 identical requests to BF16 produced identical content and identical completion_tokens.
• enable_thinking is toggled via chat_template_kwargs (supported by sglang for Qwen3).

Categories and thinking mode

Each category uses a thinking mode chosen based on the task type:

Category	thinking	max_tokens	Description
mmlu	off	512	Knowledge MCQ · 4-choice · EN
mmlu_thinking	on	None	Knowledge MCQ · 4-choice · EN
arc	off	512	Science MCQ · EN
arc_thinking	on	None	Science MCQ · EN
gsm8k	on	8192	Grade-school math · EN
math	on	8192	Competition math · EN
humaneval	off	4096	Python code gen · EN
ifeval	off	4096	Instruction following · EN
logical_reasoning	on	4096	Syllogism/puzzle · RU+EN
knowledge_qa	off	1024	Factual recall · RU+EN
instruction_following	off	4096	Format constraints · RU+EN
creative	off	4096	Story/poem · RU+EN
translation	off	1024	RU↔EN translation
code_explain	off	4096	Code bugfix/explain/run · RU+EN
multilingual_reasoning	on	4096	Math/logic in RU · RU
safety_consistency	off	4096	Helpful on benign edge · RU+EN

For MCQ benchmarks (MMLU, ARC) both variants were run: *_thinking (thinking=on) and the base (thinking=off), to cover both the "fast" and "reasoning" modes. MMLU numbers in most publications use thinking=on.

Datasets

Standard datasets were fetched via the REST API, with 30 samples per dataset and a fixed sampling seed (numpy.default_rng(42)).

Dataset	Config / split	Size	Used for
cais/mmlu	all / test	30	Knowledge MCQ (57 subjects)
allenai/ai2_arc	ARC-Challenge / test	30	Science MCQ
openai/gsm8k	main / test	30	Grade-school math
EleutherAI/hendrycks_math	7 subjects / test (mixed)	30	Competition math
openai/openai_humaneval	test	30	Python code generation
google/IFEval	default / train	30	Instruction following

Curated collection — 28 hand-crafted prompts across categories: logical reasoning, knowledge QA, instruction following, creative writing, translation, code explain/fix, multilingual reasoning, safety/edge.

Evaluators

Category	Approach
MMLU / ARC	Extract the option letter (A-E) from the answer, compare to gold
GSM8K	Extract the number from Answer: N or the last number in the answer
MATH	Extract \boxed{...}, normalize LaTeX + numeric comparison
HumanEval	Extract Python code, sandboxed exec with official tests (timeout 20s)
IFEval	25 rule verifiers implemented (detectable_format:*, length_constraints:*, punctuation:*, keywords:*, combination:*, startend:*, change_case:*, language:*). Not implemented: keywords:letter_frequency, length_constraints:nth_paragraph_first_word, detectable_format:multiple_sections — prompts using these rules are excluded from the denominator.
Curated	Programmatic checks: valid JSON, bullet/word count, absence of commas, first word, running Python code; for "qualitative" prompts — checking for key phrases and language

Total instances: 540 = 270 task pairs × 2 models = 540 requests. Run time: 5 min. Concurrency: 4 per model, models run in parallel. API errors: 0.

Results by category

Category	thinking	n	BF16	NVFP4	Δ	BF16-only wins	NVFP4-only wins	avg tokens BF16	avg tokens NVFP4
MMLU (no-think)	off	30	66.7% (20/30)	73.3% (22/30)	+6.7pp	0	2	2	2
MMLU (thinking)	on	30	93.3% (28/30)	96.7% (29/30)	+3.3pp	0	1	499	546
ARC-Challenge (no-think)	off	30	96.7% (29/30)	96.7% (29/30)	+0.0pp	0	0	2	2
ARC-Challenge (thinking)	on	30	100.0% (30/30)	100.0% (30/30)	+0.0pp	0	0	104	132
GSM8K	on	30	96.7% (29/30)	100.0% (30/30)	+3.3pp	0	1	630	410
MATH	on	30	93.3% (28/30)	90.0% (27/30)	-3.3pp	1	0	1343	1734
HumanEval	off	30	100.0% (30/30)	96.7% (29/30)	-3.3pp	1	0	203	204
IFEval (strict)	off	30	73.3% (22/30)	60.0% (18/30)	-13.3pp	6	2	458	317
Curated · logical	on	5	100.0% (5/5)	100.0% (5/5)	+0.0pp	0	0	165	429
Curated · knowledge	off	5	100.0% (5/5)	100.0% (5/5)	+0.0pp	0	0	4	4
Curated · instr-follow	off	5	100.0% (5/5)	80.0% (4/5)	-20.0pp	1	0	50	46
Curated · creative	off	3	100.0% (3/3)	100.0% (3/3)	+0.0pp	0	0	117	109
Curated · translation	off	4	100.0% (4/4)	100.0% (4/4)	+0.0pp	0	0	19	19
Curated · code	off	3	100.0% (3/3)	100.0% (3/3)	+0.0pp	0	0	23	25
Curated · multi-reason	on	3	100.0% (3/3)	100.0% (3/3)	+0.0pp	0	0	84	86
Curated · safety	off	2	100.0% (2/2)	100.0% (2/2)	+0.0pp	0	0	698	587

Overall: BF16 91.1% (246/270) · NVFP4 90.0% (243/270) · Δ = -1.11 pp

Token usage (reasoning + completion)

NVFP4 on average spends more reasoning tokens on tasks requiring deep CoT, and fewer on short formats:

Category	avg tokens BF16	avg tokens NVFP4	Δ tokens	Δ %
MMLU (no-think)	2	2	+0	+0.0%
MMLU (thinking)	499	546	+47	+9.4%
ARC-Challenge (no-think)	2	2	+0	+0.0%
ARC-Challenge (thinking)	104	132	+28	+26.4%
GSM8K	630	410	-221	-35.0%
MATH	1343	1734	+391	+29.1%
HumanEval	203	204	+1	+0.6%
IFEval (strict)	458	317	-140	-30.7%
Curated · logical	165	429	+264	+159.5%
Curated · knowledge	4	4	+0	+0.0%
Curated · instr-follow	50	46	-3	-6.8%
Curated · creative	117	109	-9	-7.4%
Curated · translation	19	19	-0	-1.3%
Curated · code	23	25	+2	+8.8%
Curated · multi-reason	84	86	+2	+2.0%
Curated · safety	698	587	-112	-16.0%

Token takeaway: NVFP4 is less efficient in reasoning (especially math/logical), and more efficient on simple tasks (gsm8k, ifeval, creative). On MATH this leads to hang incidents (see below).

Conclusions

1. Overall accuracy: a 1.11 pp gap in favor of BF16 — statistically small at n=270 pairs. On larger samples (≥500) it will likely stay within ±1.5 pp.
2. Standard academic benchmarks (MMLU-thinking, ARC, GSM8K, MATH, HumanEval): the difference is within 1-3 samples out of 30 — not significant.
3. Instruction following (IFEval): a statistically noticeable -13.3 pp degradation. NVFP4 is systematically worse at honoring secondary constraints (forbidden words, exact counts, prompt repetition). This is the most reproducible quality-loss signal.
4. Reasoning stability: NVFP4 is less efficient with reasoning tokens on hard tasks (MATH +30%, logical +160%) and in one case loops for 16384 tokens (math[14]) without producing an answer. BF16 solves the same problem in 4477 tokens. This is a risk for production on long CoT.
5. Simple tasks (knowledge QA, translation, multilingual reasoning, MCQ without thinking, creative): identical quality. NVFP4 quantization does not damage the model's "fast" modes.