ToPo-ToPo/Qwen3.8-27B-MTP-4bit

MTP (multi-token prediction) drafter for speculative decoding with Qwen/Qwen3.8-27B on Apple Silicon (mlx-vlm).

This is not a standalone chat model — it only works bound to a Qwen3.8-27B target model.

Provenance

• Split from the official checkpoint's built-in mtp.* weights (15 tensors, isolated in model-00018-of-00018.safetensors) with mlx-vlm 0.6.9: python -m mlx_vlm.speculative.drafters.qwen3_5_mtp.split --model Qwen/Qwen3.8-27B --output .
• Precision: affine 4bit / group 64 — 4.501 bits/weight
• Size: 247 MB, block_size: 3, model_type: qwen3_5_mtp

Speculative decoding notes

Gains are largest on structured output (code, JSON tool calls) and smallest on free-form prose. Stable over 589 rounds with rejections — no rollback crash.

The bf16 and 4bit drafters accept identically under greedy decoding (quantization rarely flips the argmax); bf16 was marginally better on code, 4bit uses less memory.

Measure it yourself. Speedup depends on hardware, target quantization, workload and the mlx-vlm version — on the machine used here the reference pair Qwen3.6-27B-4bit + mlx-community/Qwen3.6-27B-MTP-4bit showed no speedup at all, so any number quoted elsewhere may not transfer to your setup.

Usage

mlx_vlm.generate --model ToPo-ToPo/Qwen3.8-27B-mlx-4bit \
  --draft-model ToPo-ToPo/Qwen3.8-27B-MTP-4bit --draft-kind mtp \
  --prompt "..." --max-tokens 400