gh0stx-nvfp4

Expert-pruned and NVFP4-quantized derivative of Qwen3.5-MoE, with a grafted MTP head for speculative decoding. Fits on a single NVIDIA Blackwell card.

Specs

• Base: Qwen3.5-MoE (A17B active), expert-pruned to ~141B total params
• Quantization: NVFP4 (compressed-tensors) — ~84 GB weights
• Context: 128K (native up to 256K)
• Acceleration: built-in MTP head (qwen3_5_mtp) → ~1.7× tokens/s via speculative decoding

Requirements

• NVIDIA Blackwell GPU (sm_120+) — GB10 / B200 / RTX 50xx. NVFP4 is Blackwell-only; it will not run on H100/Ampere. For those, use the bf16 sibling and quantize yourself.
• vLLM with Qwen3.5-MoE + qwen3_5_mtp support (aeon-vllm-ultimate or a compatible build).

Serve (vLLM)

vllm serve promzeus/gh0stx-nvfp4 \
  --served-model-name gh0stx --quantization compressed-tensors \
  --trust-remote-code --max-model-len 131072 \
  --gpu-memory-utilization 0.87 --max-num-seqs 2 \
  --enable-chunked-prefill --enable-prefix-caching \
  --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":3}'

Notes

• The MTP head is bf16 (excluded from NVFP4 quant — see ignore in config). num_speculative_tokens=3 is the sweet spot (~11 tok/s on a single GB10, ×1.7 vs no MTP).
• KV cache: keep bf16 (fp8 KV is unstable with the MTP + linear-attention path).
• Thinking mode is ON by default (Qwen3.5). To disable per request, pass chat_template_kwargs: {"enable_thinking": false}.
• At 128K context use --max-num-seqs 2 to keep KV within a single card; raise --gpu-memory-utilization cautiously (0.90 was tight → cuBLAS workspace errors on peaks).

Sibling

• promzeus/gh0stx-bf16 — full bf16 weights (~141B). Runs on any stack; quantize to INT4/AWQ for H100/Intel. Add model-mtp-grafted.safetensors + mtp_num_hidden_layers: 1 to enable MTP there too.