Qwen3.8-27B-NVFP4

Mixed-precision NVIDIA ModelOpt quantization of Qwen/Qwen3.8-27B, pinned to upstream revision 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0.

• MLP layers 0-55: W4A16 NVFP4, group size 16
• MLP layers 56-63: FP8 W8A8
• lm_head: FP8 W8A8
• Attention and large linear-attention projections: FP8 W8A8
• KV cache: FP8
• Vision tower: preserved in BF16
• MTP head: preserved in BF16

The checkpoint was quantized with NVIDIA ModelOpt 0.45.0. Calibration used 1,024 fully populated 512-token rows from the deterministic version-5 corpus, covering instruction following, science, code, function calling, web search, math, software engineering, multilingual data, and natural text.

On tool-eval-bench v2.2.0 with vLLM 0.27.1, temperature 0, seed 42, and all 69 scenarios, this checkpoint scored 125/138 (91/100). It failed the benchmark's TC-34 and TC-58 prompt-injection checks; deployment should apply independent tool authorization and untrusted-content defenses.

Use a runtime that supports mixed ModelOpt W4A16 NVFP4 + FP8 checkpoints. See the upstream model card for capabilities and limitations.

vLLM

Text only

vllm serve acyildirimer/Qwen3.8-27B-NVFP4 \
  --quantization modelopt \
  --max-model-len auto \
  --kv-cache-dtype fp8 \
  --gpu-memory-utilization 0.95 \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --performance-mode interactivity \
  --language-model-only \
  --attention-backend auto \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --generation-config auto \
  --override-generation-config \
    '{"temperature":1.0,"top_p":0.95,"top_k":20,"min_p":0.0,"presence_penalty":0.0,"repetition_penalty":1.0}' \
  --tool-call-parser qwen3_coder

Text only with MTP

vllm serve acyildirimer/Qwen3.8-27B-NVFP4 \
  --quantization modelopt \
  --max-model-len auto \
  --kv-cache-dtype fp8 \
  --gpu-memory-utilization 0.92 \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --performance-mode interactivity \
  --language-model-only \
  --speculative-config '{"method": "mtp", "num_speculative_tokens": 2}' \
  --attention-backend auto \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --generation-config auto \
  --override-generation-config \
    '{"temperature":1.0,"top_p":0.95,"top_k":20,"min_p":0.0,"presence_penalty":0.0,"repetition_penalty":1.0}' \
  --tool-call-parser qwen3_coder

Text, MTP, and vision

vllm serve acyildirimer/Qwen3.8-27B-NVFP4 \
  --quantization modelopt \
  --max-model-len auto \
  --kv-cache-dtype fp8 \
  --gpu-memory-utilization 0.91 \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --performance-mode interactivity \
  --speculative-config '{"method": "mtp", "num_speculative_tokens": 2}' \
  --attention-backend auto \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --generation-config auto \
  --override-generation-config \
    '{"temperature":1.0,"top_p":0.95,"top_k":20,"min_p":0.0,"presence_penalty":0.0,"repetition_penalty":1.0}' \
  --tool-call-parser qwen3_coder

Check Avg Draft acceptance rate in the vLLM logs to confirm MTP speculative decoding is effective for the deployed workload.