Qwen3.8-27B-DFlash2-NVFP4

Native NVFP4 (W4A16, group-16) requantization of the Qwen3.8-27B DFlash2 block drafter (5 Qwen3-style layers, 1.92B params). Built for the all-NVFP4 vLLM stack: NVFP4 target weights + NVFP4 draft weights + NVFP4 KV cache on SM120 (RTX 5090), verified at 262,144-token context with DFlash2 K=3.

Format

compressed-tensors nvfp4-pack-quantized:

• weights: float4, strategy: tensor_group, group_size: 16, symmetric, dynamic: false, observer: memoryless_minmax, scale_dtype: torch.float8_e4m3fn
• packed tensors: weight_packed (uint8, [out, in/2]), weight_scale (fp8_e4m3fn, [out, in/16]), weight_global_scale (float32, divisor convention)
• quantized: self_attn.{q,k,v,o}_proj, mlp.{gate,up,down}_proj, fc (36 matrices total)
• kept BF16: kernel_projection (grouped convs), candidate_selector, hidden_projection, norms

Quantization

Round-to-nearest weight-only quantization (no calibration) from the BF16 checkpoint, using the release image's compressed_tensors 0.17.0 (compressors/nvfp4/helpers.pack_fp4_to_uint8). Reproducible with scripts/build_nvfp4_draft.py (source: vLLM-sm12x DFlash2 release repo).

Checksums

• model.safetensors (1361734592 bytes): db19f8496af66a5b8cc26c682a7c80a754af0fa62929e6b2f14aa14fe97d7126
• config.json: 106a0e81a35acfbdfcd8d40786216f260a51e12e690ea820e9c05bc5d7ab7045

Usage (vLLM, SM120, all-NVFP4)

vllm serve <Qwen3.8-27B NVFP4 target>   --speculative-config '{"method":"dflash","model":"<this repo>",
  '  "num_speculative_tokens":3,"kv_cache_dtype":"nvfp4"}'   --kv-cache-dtype nvfp4   --max-model-len 262144 --max-num-seqs 8   --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE",
  '  "cudagraph_capture_sizes":[4,8,12,16,20,24,28,32]}'

Measured on the all-NVFP4 262k profile: 341,765-token NVFP4 KV pool (1.30x of 262k), greedy determinism byte-identical, acceptance mean 2.35 / 44.9% avg, and a 254,658-token request completing end-to-end.

Built for the all-NVFP4 vLLM build — try it

This drafter is the draft half of the all-NVFP4 vLLM stack in seanyourhighness/vllm-sm12x-nvfp4-dflash2: NVFP4 target weights + NVFP4 draft weights + NVFP4 KV cache on a single RTX 5090 (SM120, 32 GB). Most public 5090 recipes pair NVFP4 weights with an FP8 KV cache; keeping the KV cache in NVFP4 as well is what lets an 8 GiB explicit pin hold a 325,139-token pool at 262K max context.

Why quantize the drafter at all? The DFlash2 block drafter is small (1.92B), but at NVFP4 it drops from ~3.6 GiB (BF16) to ~1.3 GiB, and — because it uses the target's own KV cache dtype — every byte saved on the draft model and its state becomes target KV capacity. The quantization is round-to-nearest weight-only (no calibration), and the numerically sensitive parts — the candidate selector, grouped convolutions, and norms — are deliberately kept in BF16 so acceptance length doesn't take a hit.

Measured on the production profile (RTX 5090, K=7)

• 616 tok/s aggregate at 4-way concurrency on 1,536-token code outputs, thinking off (per-request latency 8–10 s at c4)
• ~61% draft acceptance overall — 55–64% on code, 42–44% when reasoning tokens are mixed in
• 325,139-token KV pool from an 8 GiB pin; needle recovered exactly at 184,024 tokens
• Tools 10/10, greedy output byte-identical to autoregressive, zero restarts, zero OOM

The full build log — every bug, what it cost, and the measurements behind each fix — is in the article (tweet). Highlights:

• The XQA cliff. The attention kernel ran 5.2x slower in integrated execution than a standalone replay of the exact same server tensors. A dedicated CUDA stream brought 1.437 ms/call down to 0.278 ms/call — now upstream as vLLM PR #53543.
• GDN runtime-K width fix: +40.9% at c8 (212.17 → 298.91 tok/s, vLLM PR #53542).
• FlashInfer PR #4346 backports SM120 NVFP4 paged prefill (57/57 tests, 10–12% prefill win).

Run it

Two commands on a 5090 (DGX Spark / GB10 via SPARK=1 ./build.sh, not yet validated on GB10 hardware):

git clone https://github.com/seanyourhighness/vllm-sm12x-nvfp4-dflash2.git
cd vllm-sm12x-nvfp4-dflash2 && ./start.sh   # add --vision for the CPU vision sidecar

start.sh pulls the pinned runtime, downloads the target and this draft model, boots vLLM, and runs a deterministic canary. If you'd rather wire this drafter into your own stack, the K=7 production profile is:

vllm serve <Qwen3.8-27B NVFP4 target> \
  --speculative-config '{"method":"dflash","model":"YourHighnessLA/Qwen3.8-27B-DFlash2-NVFP4",
  "num_speculative_tokens":7,"kv_cache_dtype":"nvfp4"}' \
  --kv-cache-dtype nvfp4 --max-model-len 262144 --max-num-seqs 4 \
  --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE","cudagraph_capture_sizes":[8,16,24,32]}'

The runtime image is bit-identical to the one running in production, and the 51-file Python-only overlay applies cleanly to pristine vLLM v0.27.1 — so every number above traces to a pinned commit, digest, and model revision. Run it, break it, and tell us what you find.

The drafter shares the target's embeddings and lm_head (it ships neither), matching the upstream DFlash2 design. Apache-2.0, like the original drafter by Inco.