nvidia

/

Qwen3.6-27B-NVFP4

like 428

Follow

NVIDIA 66k

TL;DR

Same model, two quantizations — nvidia/Qwen3.6-27B-NVFP4 (4‑bit) vs Qwen/Qwen3.6-27B-FP8 (8‑bit) — head‑to‑head, each on its own RTX PRO 6000 Blackwell (96 GB), with vLLM and MTP speculative decoding (3 tokens) on both.

Even today, on the latest vLLM (0.24.0) and Blackwell, NVFP4 is not faster than FP8. FP8 is ~8–12 % faster on decode in every single‑stream and high‑concurrency scenario; the two only draw level on prefill‑bound big‑context concurrent load. Warm TTFT is identical.

Quality is effectively tied on everything with an objective answer (logic puzzle, edge‑case Python, strict‑schema long‑context JSON). Czech is native and artifact‑free on both, with FP8 a hair cleaner and better‑behaved in "thinking" mode. No catastrophic degradation on either.

Caveat: the endpoints ran different vLLM builds (NVFP4 0.24.0, FP8 0.23.1rc1‑dev). That plausibly favors NVFP4 — and it still doesn't win.

Setup

Verified via nvidia-smi: each quant on its own GPU (NVFP4→GPU0, FP8→GPU1), so both were benchmarked in parallel without competing for compute. OpenAI‑compatible streaming endpoints. Non‑thinking mode unless noted.

Methodology

Client‑side streaming, timed send→[DONE], tokens from the server usage block. Four throughput scenarios + a 7‑prompt quality suite (Czech poem, reklamace e‑mail, HTML to‑do, HTML Snake, logic→JSON, merge_intervals, 59 KB→JSON analysis) in both modes.

Total wall‑clock: ~20 min end‑to‑end (both GPUs in parallel).

Throughput (non‑thinking, MTP=3) — decode tok/s, higher is better

No failures or truncation at concurrency 16. FP8 wins the decode‑bound scenarios by a consistent ~8–12 %; the only place NVFP4 draws level (and edges ahead) is heavy prefill‑bound concurrent load (par_big), where both are limited by chewing through 22k‑token prompts rather than by weight bandwidth. Warm TTFT is identical.

The first NVFP4 request showed ~3.7 s TTFT — one‑time vLLM warm‑up (CUDA‑graph capture), not the quant; warm value is 0.09 s.

Per‑prompt answer latency (non‑thinking)

* short outputs → tok/s is noisy on those two rows.

Quality (code & JSON actually executed/parsed)

Thinking mode: both over‑reason on the trivial Python task, but NVFP4 is less disciplined — it never finished the function even at a 16k‑token budget (12 139 reasoning tokens), while FP8 finished correctly. For deterministic work, non‑thinking is the sane choice on both.

Bottom line: no measurable gap in reasoning correctness or instruction‑following; no gibberish, broken diacritics, or language switching on either. FP8 keeps a small edge in Czech polish and thinking‑mode restraint. 4‑bit NVFP4 does not noticeably "get dumber."

Caveats

1. vLLM builds differ (0.24.0 vs 0.23.1rc1‑dev) + different attention/MoE backends — a confound that plausibly favors NVFP4, making "FP8 is faster" the conservative read.
2. Single run per scenario, not averaged; short‑output tok/s is noisy.
3. One 27B MoE model, one Blackwell card, MTP=3 — don't extrapolate to dense models / other GPUs / non‑speculative serving.
4. HTML judged structurally (valid single file, inline JS, required APIs present), not by driving it in a browser.
5. MTP=3 is on for both, so decode tok/s already bakes in speculative acceptance.

Legend

• TTFT — time to first token (prefill + queue proxy).
• decode tok/s — per‑request output tokens/sec after the first token (excludes prefill).
• aggregate tok/s — total output tokens/sec across all concurrent requests (total_completion_tokens / wall).
• prefill tok/s — input tokens/sec before first output (prompt_tokens / TTFT); meaningful only on the ~22.3k‑token big‑prompt rows (≈ a 59 KB Czech document).
• e2e p50/p95 — end‑to‑end request‑latency percentiles across the concurrent batch.