ThinkingCap-Qwen3.6-27B-FP8

FP8 (compressed-tensors, FP8_DYNAMIC W8A8) quantisation of bottlecapai/ThinkingCap-Qwen3.6-27B.

This is our default FP8 build. It keeps the entire self-attention path in bf16 (the attnbf16 recipe) and ships a working MTP / self-speculative-decoding head.

What's kept in bf16

lm_head, the MTP head, the vision tower, the whole linear_attn (Gated-DeltaNet / SSM) block, and — the distinctive choice here — the entire self_attn path, including the attention output gate that Qwen3.5/3.6 fuses into q_proj. Only the MLPs (17 B params) are FP8. Quantisation error on the fused multiplicative gate is qualitatively worse than on an additive projection, and only ~1/4 of the layers are full-attention, so keeping attention bf16 costs ~1.5 GiB (4.6 %) and measurably lowers KLD. See recipe.yaml.

Accuracy vs the bf16 parent: KLD ≈ 0.0132 nats, ΔPPL ≈ +0.96 % (kld_thinkingcap-fp8-attnbf16.json, ppl_thinkingcap-fp8-attnbf16.json).

Speculative decoding (MTP)

The bf16 MTP head is declared in quantization_config.ignore so vLLM loads it correctly (a bf16 MTP head regrafted into a compressed-tensors quant is otherwise mis-loaded and yields 0 % draft acceptance).

Measured MTP acceptance (vLLM 0.26.0, Blackwell SM120, greedy, {"method":"mtp","num_speculative_tokens":3})

Build	MTP draft acceptance
this repo	75.0 %
bottlecapai/…-FP8 (upstream, reference)	74.2 %

vllm serve huginnfork/ThinkingCap-Qwen3.6-27B-FP8 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
  --max-num-seqs 32

--max-num-seqs 32 (or lower) is required — Qwen3.6 is a hybrid linear-attention model and the default max_num_seqs exceeds the available Mamba cache blocks.

Serving without speculation

vllm serve huginnfork/ThinkingCap-Qwen3.6-27B-FP8 --max-num-seqs 32

Loads natively in vLLM / SGLang. Text + vision + MTP-presence smoke tests pass; the vision tower correctly describes images.