Model Card for Qwen3.6-27B-MTP-8bit

8-bit quantized version of Qwen3.6-27B with multi-token prediction (MTP) capabilities preserved. Quantization performed using mlx-lm pull request https://github.com/ml-explore/mlx-lm/pull/990 to preserve MTP weights. Original model: https://huggingface.co/qwen/qwen3.6-27b

Key Features

• Multi-Token Prediction (MTP): This quantized model preserves the MTP capabilities from the original Qwen3.6-27B, enabling faster inference through speculative decoding.
• YaRN Support: The repository includes config_1M.json for hot-swapping to enable YaRN (Yet another RoPE extensioN) for ultra-long context processing up to 1M tokens. For details on using YaRN with this model, see https://huggingface.co/Qwen/Qwen3.6-27B#processing-ultra-long-texts

Performance on Apple M4 Max

This model provides the best quality MTP quantization for Apple M4 Max hardware. Due to memory bandwidth saturation of the full bf16 weights, enabling MTP there actually decreases TPS.

Throughput Comparison:

• Full bf16: ~9 TPS
• 8-bit quantized: ~18 TPS
• 8-bit with MTP: ~23-24 TPS

Prefill Speed (Full bf16 vs 8-bit):

• Short contexts (~100k tokens): ~14% faster with bf16
• Medium contexts (~200k tokens): ~13% faster with bf16
• Long contexts (~750k tokens): ~12% faster with bf16

Optimal Usage Strategy: Running the full weights wherever possible is strongly recommend for maximum quality, but for large output tasks, 8-bit with MTP is the perfect balance for this model and hardware. For the optimal balance of speed and quality, you can re-use your bf16 KV cache and hot toggle between bf16 and MTP-8bit within the same conversation:

• Use bf16 for: Prefill, thinking, planning, and design tasks
• Use MTP-8bit for: Large output tasks such as code editing or generation

You can give your agent a toggle_mtp tool with these instructions so it can dynamically switch between the appropriate weights for maximum performance for both prefill and decode.

Thinking Preservation: Take this one step further with Thinking Preservation enabled (https://huggingface.co/Qwen/Qwen3.6-27B#qwen36-highlights) so that the MTP-8bit has visibility of the planning/reasoning thinking traces from the full weights, providing the ultimate dynamic hybrid of maximum performance and quality.

Model Details

• Base Model: Qwen/Qwen3.6-27B
• Library: mlx-lm
• Quantization: 8-bit
• Special Configurations: config_1M.json available for YaRN-enabled 1M context length
• License: apache-2.0
• Pipeline Tag: text-generation

Installation

MTP support for this model currently requires installing mlx-lm from PR #990 (not yet in main/release):

# Option 1: pip install
pip install https://github.com/ml-explore/mlx-lm/archive/refs/pull/990/head.zip

# Option 2: uv run (no virtual env needed)
uv run --with https://github.com/ml-explore/mlx-lm/archive/refs/pull/990/head.zip python example.py

Usage

import mlx_lm
from mlx_lm.sample_utils import make_sampler

model_path = "petergilani/Qwen3.6-27B-MTP-8bit"
model, tokenizer = mlx_lm.load(model_path)

sampler = make_sampler(temp=1.0, top_p=0.95, top_k=20)

prompt = "Explain multi-token prediction in language models."
response = mlx_lm.generate(
    model, 
    tokenizer, 
    prompt=prompt,
    sampler=sampler,
    max_tokens=512,
    mtp=True  # Enable Multi-Token Prediction for faster generation
)
print(response)