---
license: apache-2.0
base_model: Qwen/Qwen3.8-27B
library_name: mlx
tags:
- mlx
- mlx-vlm
- speculative-decoding
- mtp
---

# ToPo-ToPo/Qwen3.8-27B-MTP-4bit

MTP (multi-token prediction) **drafter** for speculative decoding with
[`Qwen/Qwen3.8-27B`](https://huggingface.co/Qwen/Qwen3.8-27B) on Apple Silicon (mlx-vlm).

This is **not** a standalone chat model — it only works bound to a Qwen3.8-27B target model.

## Provenance
- Split from the official checkpoint's built-in `mtp.*` weights (15 tensors, isolated in
  `model-00018-of-00018.safetensors`) with mlx-vlm 0.6.9:
  `python -m mlx_vlm.speculative.drafters.qwen3_5_mtp.split --model Qwen/Qwen3.8-27B --output .`
- Precision: affine 4bit / group 64 — 4.501 bits/weight
- Size: 247 MB, `block_size: 3`, `model_type: qwen3_5_mtp`

## Speculative decoding notes

Gains are largest on structured output (code, JSON tool calls) and smallest on free-form prose.
Stable over 589 rounds with rejections — no rollback crash.

The bf16 and 4bit drafters accept identically under greedy decoding (quantization rarely flips
the argmax); bf16 was marginally better on code, 4bit uses less memory.

**Measure it yourself.** Speedup depends on hardware, target quantization, workload and the
mlx-vlm version — on the machine used here the reference pair
`Qwen3.6-27B-4bit` + `mlx-community/Qwen3.6-27B-MTP-4bit` showed no speedup at all, so any
number quoted elsewhere may not transfer to your setup.

## Usage
```bash
mlx_vlm.generate --model ToPo-ToPo/Qwen3.8-27B-mlx-4bit \
  --draft-model ToPo-ToPo/Qwen3.8-27B-MTP-4bit --draft-kind mtp \
  --prompt "..." --max-tokens 400
```