Quatfit/Quatfit-Mini-MTP

Quatfit Mini — MTP

Standalone Multi-Token-Prediction Drafter for Speculative Decoding

---

Overview

Quatfit Mini MTP is the standalone multi-token-prediction (MTP) drafter used for speculative decoding with Quatfit Mini. Rather than a separate small language model, it is a lightweight autoregressive head that cross-attends directly to the target model's own KV cache — it has its own token embedder and a compact 4-layer transformer block (3 local sliding-window layers + 1 global layer), but no separate prefill pass and no independent context of its own to maintain.

This repository packages that drafter as a standalone download so it can be attached to any Quatfit Mini serving deployment (FP32, BF16, FP8, or GGUF) to accelerate decoding.

This is not a general-purpose chat model — it drafts candidate continuations that the target model verifies. Use it together with Quatfit/Quatfit-Mini, Quatfit/Quatfit-Mini-FP8, or Quatfit/Quatfit-Mini-GGUF.

---

Headline Results

Metric	Result
Decode throughput vs. standard autoregressive decoding	2.4×
Draft token acceptance rate vs. a same-size standalone draft-model baseline	+97% relative (≈91% vs. ≈46%)

See Methodology below for exactly what these numbers measure and the conditions they were measured under — read that section before quoting these figures in your own comparisons.

---

Why It's Faster and More Accurate Than a Standalone Draft Model

Most speculative-decoding setups pair a large target model with a small, independently-trained draft model that has to run its own forward pass and maintain its own context. Quatfit Mini MTP takes a different approach:

1. Shared KV cache, no separate prefill. The drafter cross-attends to the target model's existing KV cache instead of recomputing its own — it never needs to catch up on context the target model has already processed. This removes the drafter-prefill overhead that standalone draft models pay on every request.
2. Trained jointly with the target model's representations. Because the drafter consumes the target model's own last-layer activations and embeddings rather than approximating them from scratch, its predictions track the target model's actual output distribution more closely than an independently trained model of similar size — this is the main driver of the higher acceptance rate.
3. Efficient decoding via top-k clustering. Instead of projecting over the full 262,144-token vocabulary at every drafted step, the drafter's output head uses a top-k clustering operation, shrinking the final matrix multiplication from $d \times 262{,}144$ to $d \times 4{,}096$ — this keeps the drafter's own per-token cost low enough that its overhead doesn't eat into the speedup from higher acceptance.
4. Arbitrary draft length. Because there's no separate prefill to amortize, the drafter can propose a variable number of tokens per step rather than being locked to a fixed draft length, letting the serving stack tune the depth/acceptance trade-off per workload.

---

Architecture

Component	Value
Type	Autoregressive MTP drafter, cross-attends to target model KV cache
Target model	Quatfit Mini (8B, Gemma 4 architecture)
Drafter layers	4 (3 local sliding-window + 1 global)
Hidden dimension	512
Attention heads	4
Own embedder	Yes — separate from target model's input embeddings
Own KV cache	No — cross-attends to target model's cache
Output head	Top-k clustered projection ($d \times 4{,}096$, not full vocab)
Parameters	~180M
Precision	BF16 (FP8 variant available for FP8-served targets)

---

Usage

vLLM (speculative decoding)

vllm serve Quatfit/Quatfit-Mini \
  --speculative-model Quatfit/Quatfit-Mini-MTP \
  --num-speculative-tokens 5 \
  --max-model-len 131072

from vllm import LLM, SamplingParams

llm = LLM(
    model="Quatfit/Quatfit-Mini",
    speculative_model="Quatfit/Quatfit-Mini-MTP",
    num_speculative_tokens=5,
    max_model_len=131072,
)
sampling_params = SamplingParams(temperature=0.7, max_tokens=512)

outputs = llm.generate(["Explain the difference between GQA and MHA."], sampling_params)
print(outputs[0].outputs[0].text)

With the FP8-served target model

vllm serve Quatfit/Quatfit-Mini-FP8 \
  --quantization fp8 \
  --speculative-model Quatfit/Quatfit-Mini-MTP \
  --num-speculative-tokens 5

llama.cpp

./llama-server \
  -hf Quatfit/Quatfit-Mini-GGUF:Q4_K_M \
  --model-draft Quatfit-Mini-MTP-Q4_K_M.gguf \
  --draft-max 5 \
  -c 131072

🤗 Transformers (assisted generation)

import torch
from transformers import AutoModelForImageTextToText, AutoProcessor

target = AutoModelForImageTextToText.from_pretrained(
    "Quatfit/Quatfit-Mini", torch_dtype="auto", device_map="auto"
)
drafter = AutoModelForImageTextToText.from_pretrained(
    "Quatfit/Quatfit-Mini-MTP", torch_dtype="auto", device_map="auto"
)
processor = AutoProcessor.from_pretrained("Quatfit/Quatfit-Mini")

messages = [{"role": "user", "content": "Write a Python implementation of binary search."}]
inputs = processor.apply_chat_template(messages, tokenize=True, return_tensors="pt").to(target.device)

outputs = target.generate(**inputs, assistant_model=drafter, max_new_tokens=512)
print(processor.decode(outputs[0]))

---

Methodology

The headline numbers above were measured as follows — please validate against your own workload before relying on them for capacity planning.

Decode throughput (2.4×): measured as end-to-end tokens/second during decode (post-prefill), comparing standard single-token autoregressive decoding against speculative decoding with this drafter, both targeting Quatfit Mini at the same precision and batch size, on a single H100. Throughput gain varies with batch size, sequence length, and task mix — larger batches and less-predictable generation (e.g. open-ended creative writing vs. structured code) typically see a smaller multiple than the figure quoted.

Draft acceptance rate (+97% relative): measured as the fraction of drafted tokens accepted by the target model's verification step, averaged across a mixed evaluation set of code, chat, reasoning, and tool-use prompts, at draft length 5. The "baseline" is an independently trained standalone draft model of comparable parameter count (~180M) with its own embedder, own KV cache, and no cross-attention into the target model — i.e., the conventional speculative-decoding setup this drafter is designed to improve on, not a specific named third-party product.

---

Relationship to Other Quatfit Mini Repositories

	Quatfit Mini (base)	Quatfit Mini FP8	Quatfit Mini GGUF	Quatfit Mini MTP (this repo)
Role	Target model, full fidelity	Target model, GPU serving	Target model, local/CPU	Drafter, attaches to any of the above
Format	safetensors, FP32	safetensors, FP8 + BF16	.gguf	safetensors
Runtime	🤗 Transformers	vLLM, TensorRT-LLM, SGLang	llama.cpp	vLLM, TensorRT-LLM, llama.cpp, Transformers
Size	~32 GB	~9.5 GB	~3–16.8 GB per quant	180M params (360 MB BF16)

Full architecture details for both the target model and this drafter are documented in the Quatfit Mini Technical Report.

---

Responsible AI

This drafter does not generate final output on its own — all drafted tokens are verified by the target model before being returned, so speculative decoding does not change what the target model would otherwise produce; it only changes how fast those tokens are produced. Standard responsible-use guidance for Quatfit Mini itself still applies:

• Verify critical information
• Apply RAG for factual grounding
• Use application-level safety filters
• Keep human oversight for high-risk domains

---

Citation

@article{quatfitminimtp2026,
    title={Quatfit Mini MTP: A Cross-Attending Multi-Token-Prediction Drafter for Speculative Decoding},
    author={Quatfit AI Research},
    year={2026}
}

---

License

Apache License 2.0. See LICENSE for details.

---

🤗 MTP Repository • 🧠 Base Model (FP32) • ⚡ FP8 Build • ⚙️ GGUF Builds • 📄 Technical Report