Qwen3.6-27B-heretic-MTP-FP8

FP8 quantization of Youssofal/Qwen3.6-27B-Abliterated-Heretic-Uncensored-BF16, with MTP speculative decoding working and the vision tower kept in BF16.

• Base: Qwen3.6-27B (27.8B dense, 64 layers, hybrid linear/full attention), abliterated with a HERETIC-style two-stage MPOA pipeline. KL 0.0282 against base on harmless prompts.
• Quantization: FP8 E4M3. 128x128 block-scaled weights, dynamic per-group activations. Same scheme Qwen uses for its own FP8 releases.
• Size: 30GB. 1599 tensors across 7 shards plus separate MTP and visual shards.
• Quantized: 400 linear layers in the language backbone (attention + MLP).
• Left in BF16: vision tower (333 tensors), MTP heads (15), lm_head, embed_tokens, linear-attention input projections.
• MTP: works. The mtp.* tensors are BF16, spliced from official Qwen3.6-27B.
• Tokenizer: official Qwen3.6-27B's, not the source's.

Fits on one 96GB card with room for a large KV cache. Image and video input both work.

Usage with vLLM

Tested on vLLM 0.24 through 0.26.

vllm serve OptimizeLLM/Qwen3.6-27B-heretic-MTP-FP8 \
  --max-num-seqs 32 \
  --reasoning-parser qwen3 \
  --reasoning-config '{}' \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --enable-prefix-caching \
  --speculative-config '{"method":"mtp","num_speculative_tokens":4}'

--max-num-seqs 32 is required. vLLM's default of 1024 exceeds the available Mamba cache blocks (933) on this hybrid architecture, and the engine won't start.

Measured

Single RTX PRO 6000 Blackwell, vLLM 0.26.0, MTP=4, thinking off:

Metric	Value
Single-stream generation	~86 tok/s
MTP acceptance length	2.5-3.4 tokens/step
Per-position acceptance	0.81 / 0.65 / 0.51 / 0.42
Weights on GPU	~30GB

Acceptance varies with workload. That range covers conversational and tool-calling traffic.

Notes on the flags

This is what we run, not a suggested starting point.

• num_speculative_tokens: 4. Positions 3 and 4 accept least, so 2 or 3 looks better on paper. 4 has been faster for us on conversational and tool-calling traffic. It's one flag, so test it against your own.
• --enable-prefix-caching. How much this buys you depends on your traffic rather than the model. A long stable system prompt with short turns on top caches well; one-shot requests sharing no prefix gain nothing. Watch anything that varies near the front of the prompt, like a timestamp or session id, since it invalidates everything after it.
• --reasoning-parser qwen3 with --reasoning-config '{}' for thinking control. thinking_token_budget works. Cap it. Uncapped, the model will sometimes think at length before an easy answer, and that shows up in tail latency.
• --tool-call-parser qwen3_coder. Clean tool-call JSON, including against large MCP schemas.
• --gpu-memory-utilization at its 0.9 default gives a large KV cache on a 96GB card. Lower it first if you're sharing the card with something else.

One template constraint: system messages are only accepted at position 0. A mid-conversation system turn fails with a template error. Merge runtime context into a user message instead.

Building this yourself

Quantized with llm-compressor (compressed-tensors 0.14.1.dev28), FP8_BLOCK preset. Needs transformers >= 5.0 for the Qwen3.6 classes.

Scheme:

• Format float-quantized
• Weights FP8 E4M3, static, 128x128 blocks
• Activations FP8 E4M3, dynamic, group size 128
• No calibration. FP8_BLOCK is data-free RTN, about 30 minutes on CPU.
• Ignore list follows Qwen's official FP8 release: lm_head, embed_tokens, visual.*, the linear_attn in_proj_a/in_proj_b/in_proj_ba gates, and mtp.*. The delta-rule gates are precision-sensitive. in_proj_qkv, in_proj_z and out_proj are quantized.

Five things this checkpoint does that a plain oneshot() run won't:

1. Run with CUDA_VISIBLE_DEVICES="". The data-free pipeline dispatches to visible GPUs and will OOM against a card that's already busy.
2. Restore the vision tower. llm-compressor loads this architecture text-only, which drops the 333 visual.* tensors and flattens config.json to a text-only causal LM. Splice the tensors back and restore the multimodal config with quantization_config grafted in.
3. Splice in the MTP heads. Abliterated re-uploads don't carry mtp.*, so take all 15 in BF16 from official Qwen3.6-27B. Splicing only needs the source shards holding those keys, not the whole checkpoint.
4. Take the tokenizer from official Qwen3.6-27B, not from the abliterated source. tokenizer.json, tokenizer_config.json and chat_template.jinja. The source's tokenizer was damaged by a round-trip through an older tokenizers version: a 768-token truncation cap, a pre-tokenizer regex missing the Unicode Mark class, and seven audio tokens mapped to untrained embedding rows. The chat template is byte-identical either way, so prompting is unchanged.
5. Write the ignore list as prefix-agnostic re: patterns. vLLM fuses in_proj_a and in_proj_b into in_proj_ba, and llm-compressor writes resolved concrete names, so a literal ignore list misses the fused module and vLLM refuses to load the checkpoint.

Then check the index against the shards before serving: 1599 tensors, no orphans in either direction.

Thanks

• Qwen Team, for Qwen3.6-27B itself: the architecture, the vision tower, the MTP heads, and the open release.
• p-e-w, for HERETIC.
• Youssofal, for the two-stage MPOA abliteration this is built on, and for documenting the method and its KL divergence.
• The vLLM and llm-compressor teams.