philbert440/Qwen3.8-27B-NVFP4

Qwen3.8-27B — NVFP4

NVFP4 (4-bit floating point, weight-only) quantization of Qwen/Qwen3.8-27B — the whole model, nothing stripped: vision tower intact, MTP speculative-decoding head intact in BF16, thinking mode calibrated with real reasoning traces. 55 GB BF16 → 19.8 GB.

Built as compressed-tensors nvfp4-pack-quantized deliberately: 1Cat-vLLM's SM70 NVFP4 path accepts this format at compute capability 7.0, so it serves on 2×V100-32GB under 1Cat-vLLM 1.2.2 — whereas ModelOpt-exported NVFP4 checkpoints require capability 7.5+ and reject Volta. On SM100+ (Blackwell) the FP4 tensor cores run this format natively.

At a glance

Base model	Qwen/Qwen3.8-27B — dense 27B vision-language model, 64 layers (16 full-attention + 48 GatedDeltaNet linear-attention), thinking on by default
Format	compressed-tensors nvfp4-pack-quantized — E2M1 4-bit weights, FP8-E4M3 per-group scales (group 16), FP32 per-tensor global scale, weight-only
Method	GPTQ (Hessian-based, sequential over Qwen3_5DecoderLayer)
Files	model.safetensors 19.0 GB (packed body) + model-mtp.safetensors 0.83 GB (BF16 MTP head)
Quantized	400 linears — all attention + MLP projections, incl. 144 GatedDeltaNet projections
Kept in BF16	vision tower (333 tensors) · linear_attn.in_proj_a/b · embed_tokens · lm_head · MTP head
Context length	262,144 native · extensible to 1M with YaRN (see below)
Reference hardware	2×V100-32GB (TP2) via 1Cat-vLLM 1.2.2 · native FP4 on SM100+

Measured performance

Methodology: warm serve (2 discarded warmup generations), fixed-length generations via ignore_eos so every run produces exactly the stated token count, official Qwen3.8 sampling per mode (thinking 1.0/0.95/20, instruct 0.7/0.80/20 + presence 1.5), varied prompts. Reported as mean ± sd tokens/s. Rig: 2×V100-32GB, 1Cat-vLLM 1.2.2, TP2, fp8_e5m2 KV, max_num_seqs 4, MTP K=2.

Regime	greedy draft	probabilistic draft
512-tok, thinking (n=10)	53.0 ± 2.5	54.4 ± 1.3
2048-tok, thinking (n=3)	51.1 ± 2.5	54.1 ± 1.1
512-tok, instruct (n=6)	50.5 ± 1.8	50.0 ± 1.6
Mean acceptance length, whole workload	2.25	2.46

Concurrency (4-way, 512-tok, aggregate): ~170 tok/s with {"cudagraph_mode":"piecewise"} (auto capture sizes), ~165 with full_and_piecewise — on par with the W4A16 sibling. One sizing rule matters: with MTP, each sequence schedules K+1 tokens per step, so never set explicit cudagraph_capture_sizes below max_num_seqs × (K+1) — a cap of [1,2,4,8] at batch 4 pushes concurrent decode off CUDA graphs and collapses aggregate throughput ~3× (measured 55–72 tok/s; reproduces identically on the W4A16 sibling, so it's a config trap, not a format property). Engine-default auto sizing is correct.

Pick the draft mode by workload: verification rejection-samples against the target model, so output quality is identical either way. At the official temp-1.0 thinking sampling, probabilistic matches the verified distribution and wins (+2–6%); on low-temperature workloads the two converge (see instruct row); at temperature 0 greedy is the natural choice.

Quality validation (passed on this rig): factual coherence, think-tag discipline (zero <think> leakage with thinking disabled), vision (image understanding through the VLM path), GSM8K sample 3/3, and long-form generation with no repetition/degeneration.

The base model

Qwen3.8-27B is the compact dense model of the Qwen3.8 generation: a native vision-language model with flexible thinking control (reasoning_effort, preserve_thinking), built for long-horizon agentic work. The hidden layout is 16 blocks of 3×(GatedDeltaNet→FFN) → 1×(GatedAttention→FFN).

Official base-model (BF16) benchmarks, from the Qwen3.8-27B model card — quantization applies on top of these; they are not re-measured on this checkpoint:

Benchmark	Qwen3.8-27B	Qwen3.6-27B
Terminal Bench 2.1 (Terminus)	73.0	63.4
SWE-bench Pro	53.5	57.6
NL2Repo-Bench	42.3	36.2
GPQA Diamond	89.2	87.8
HLE	30.8	24.0
LiveCodeBench v6	83.9	89.6
IFBench	69.1	79.1
OmniDocBench 1.5 (VL)	91.1	89.4
RealWorldQA (VL)	85.9	84.1
ERQA (VL)	65.5	62.5
OSWorld-Verified (VL)	63.9	73.3

Why this build is careful

Thinking-mode calibration

Qwen3.8 deploys with thinking on by default — so this quant was calibrated in thinking mode. 256 samples @ 2048 tokens of real <think>…</think> reasoning traces from Magpie-Reasoning-V2-250K-CoT (filtered to complete traces), applied with enable_thinking=True. Calibrating a thinking model with enable_thinking=False shows the quantizer an empty think block on every sample — a calibration/deployment mismatch documented to corrupt <think>-tag behavior after quantization (llm-compressor #2680 / #2681).

Scale packaging, verified

NVFP4 dequant is w = e2m1 × weight_scale / weight_global_scale. This build stores the fp8 block scales with the global factor folded in (block-scale max saturates fp8 at 448.0, FP32 global per tensor) — the packaging convention verified against known-good checkpoints. GPTQ Hessian calibration ran on the same thinking-mode dataset as the weight quantization.

MTP head — BF16 by design

The base model's 15-tensor MTP head is grafted back verbatim in BF16 as model-mtp.safetensors and listed in quantization_config.ignore. High-precision MTP heads are standard NVFP4 practice (NVIDIA's ModelOpt recipes do the same): the head drives speculative decoding, where draft quality directly sets acceptance rate. One MTP layer; K=2 is the sweet spot on V100.

Serving

2×V100 / SM70 — 1Cat-vLLM 1.2.2

VLLM_SM70_QUANT_BACKEND=turbomind VLLM_SM70_FLASH_ATTN_V100=1 VLLM_SM70_NVFP4_TURBOMIND=1 \
VLLM_SM70_MTP_DYNAMIC_DRAFT_VOCAB_DEFAULT=0 NCCL_P2P_DISABLE=1 \
python -m vllm.entrypoints.openai.api_server \
  --model philbert440/Qwen3.8-27B-NVFP4 \
  --dtype half --attention-backend FLASH_ATTN_V100 \
  --tensor-parallel-size 2 --gpu-memory-utilization 0.78 \
  --max-model-len 32768 --kv-cache-dtype fp8_e5m2 \
  --enable-prefix-caching --reasoning-parser qwen3 \
  --compilation-config '{"cudagraph_mode":"piecewise"}' \
  --speculative-config '{"method":"mtp","num_speculative_tokens":2,"attention_backend":"FLASH_ATTN_V100","draft_sample_method":"probabilistic"}'

SM70 notes, learned the hard way:

• ninja must be on PATH — the dequant kernels JIT-compile at startup.
• VLLM_SM70_MTP_DYNAMIC_DRAFT_VOCAB_DEFAULT=0 is required for max_num_seqs > 1 with MTP.
• NCCL_P2P_DISABLE=1 on V100 pairs without NVLink — leaving P2P on costs ~4× throughput.
• Boot takes ~3–5 min (weight load + torch.compile + CUDA graph capture); poll /v1/models.

Modern GPUs

On SM100+ the compressed-tensors NVFP4 format runs on native FP4 tensor cores in recent vLLM; on Ampere/Hopper it runs via dequant paths (Marlin). If you only need Ampere+, consider the W4A16-AWQ sibling as well.

Recommended sampling (official)

Mode	temperature	top_p	top_k	min_p	presence_penalty
Thinking (default)	1.0	0.95	20	0.0	0.0
Instruct (non-thinking)	0.7	0.80	20	0.0	1.5

Thinking is on by default; disable per request with "chat_template_kwargs": {"enable_thinking": false}. For agentic work, give it generous output budgets rather than tight max_tokens limits.

Long context — 262K native, 1M with YaRN

Native 262,144-token context. For longer totals, Qwen's official YaRN override applies:

VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 vllm serve philbert440/Qwen3.8-27B-NVFP4 \
  --max-model-len 1000000 \
  --hf-overrides '{"text_config": {"rope_parameters": {"mrope_interleaved": true, "mrope_section": [11, 11, 10], "rope_type": "yarn", "rope_theta": 10000000, "partial_rotary_factor": 0.25, "factor": 4.0, "original_max_position_embeddings": 262144}}}'

Static YaRN can slightly hurt short-text quality — enable only when needed, and size factor to your real context.

Provenance

Quantized from the official BF16 release with llm-compressor (GPTQModifier, recipe in recipe.yaml). No fine-tuning, no ablation, no chat-template edits — tokenizer, chat template, and preprocessor configs are verbatim from the base model.

Citation

Per the Qwen team, for the base model:

@misc{qwen38,
    title = {{Qwen3.8-Max}: A New Bar for Coding and Cowork},
    url = {https://qwen.ai/blog?id=qwen3.8},
    author = {{Qwen Team}},
    month = {August},
    year = {2026}
}

Changelog

• 2026-08-15 — tokenizer fix. tokenizer.json / tokenizer_config.json were re-serialized by the llm-compressor calibration run and shipped with an active truncation block (max_length 1024/2048) plus a drifted pre-tokenizer regex (and, on the Qwen3.6-based repos, 7 phantom audio/TTS special tokens the base model does not define). That broke image inputs larger than the limit under transformers 5 / vLLM (Mismatch in image token count, surfacing as an HTTP 400 Failed to apply Qwen3VLProcessor). Both files are now byte-identical to the upstream base model's (vocab/merges/added tokens were always identical — this is a metadata-only restore). If you downloaded before this date, re-fetch those two files. Thanks to @elBuffo for the report.