---
license: apache-2.0
license_name: apache-2.0
license_link: https://www.apache.org/licenses/LICENSE-2.0
base_model: Quatfit/Quatfit-Mini
language:
- en
- hi
- multilingual
- mr
- fr
- zh
- gu
- ja
- te
pipeline_tag: text-generation
tags:
- mtp
- multi token prediction
- speculative-decoding
- draft-model
- gemma4
- quatfit
- text-generation
---
Quatfit Mini — MTP
Standalone Multi-Token-Prediction Drafter for Speculative Decoding
---
## Overview
**Quatfit Mini MTP** is the standalone **multi-token-prediction (MTP) drafter** used for speculative decoding with [Quatfit Mini](https://huggingface.co/Quatfit/Quatfit-Mini). Rather than a separate small language model, it is a lightweight autoregressive head that **cross-attends directly to the target model's own KV cache** — it has its own token embedder and a compact 4-layer transformer block (3 local sliding-window layers + 1 global layer), but no separate prefill pass and no independent context of its own to maintain.
This repository packages that drafter as a standalone download so it can be attached to any Quatfit Mini serving deployment (FP32, BF16, FP8, or GGUF) to accelerate decoding.
> This is not a general-purpose chat model — it drafts candidate continuations that the target model verifies. Use it together with **[Quatfit/Quatfit-Mini](https://huggingface.co/Quatfit/Quatfit-Mini)**, **[Quatfit/Quatfit-Mini-FP8](https://huggingface.co/Quatfit/Quatfit-Mini-FP8)**, or **[Quatfit/Quatfit-Mini-GGUF](https://huggingface.co/Quatfit/Quatfit-Mini-GGUF)**.
---
## Headline Results
| Metric | Result |
|---|---|
| Decode throughput vs. standard autoregressive decoding | **2.4×** |
| Draft token acceptance rate vs. a same-size standalone draft-model baseline | **+97% relative** (≈91% vs. ≈46%) |
See [Methodology](#methodology) below for exactly what these numbers measure and the conditions they were measured under — read that section before quoting these figures in your own comparisons.
---
## Why It's Faster and More Accurate Than a Standalone Draft Model
Most speculative-decoding setups pair a large target model with a small, independently-trained draft model that has to run its own forward pass and maintain its own context. Quatfit Mini MTP takes a different approach:
1. **Shared KV cache, no separate prefill.** The drafter cross-attends to the target model's existing KV cache instead of recomputing its own — it never needs to catch up on context the target model has already processed. This removes the drafter-prefill overhead that standalone draft models pay on every request.
2. **Trained jointly with the target model's representations.** Because the drafter consumes the target model's own last-layer activations and embeddings rather than approximating them from scratch, its predictions track the target model's actual output distribution more closely than an independently trained model of similar size — this is the main driver of the higher acceptance rate.
3. **Efficient decoding via top-k clustering.** Instead of projecting over the full 262,144-token vocabulary at every drafted step, the drafter's output head uses a top-k clustering operation, shrinking the final matrix multiplication from $d \times 262{,}144$ to $d \times 4{,}096$ — this keeps the drafter's own per-token cost low enough that its overhead doesn't eat into the speedup from higher acceptance.
4. **Arbitrary draft length.** Because there's no separate prefill to amortize, the drafter can propose a variable number of tokens per step rather than being locked to a fixed draft length, letting the serving stack tune the depth/acceptance trade-off per workload.
---
## Architecture
| Component | Value |
|---|---|
| Type | Autoregressive MTP drafter, cross-attends to target model KV cache |
| Target model | Quatfit Mini (8B, Gemma 4 architecture) |
| Drafter layers | 4 (3 local sliding-window + 1 global) |
| Hidden dimension | 512 |
| Attention heads | 4 |
| Own embedder | Yes — separate from target model's input embeddings |
| Own KV cache | No — cross-attends to target model's cache |
| Output head | Top-k clustered projection ($d \times 4{,}096$, not full vocab) |
| Parameters | ~180M |
| Precision | BF16 (FP8 variant available for FP8-served targets) |
---
## Usage
### vLLM (speculative decoding)
```bash
vllm serve Quatfit/Quatfit-Mini \
--speculative-model Quatfit/Quatfit-Mini-MTP \
--num-speculative-tokens 5 \
--max-model-len 131072
```
```python
from vllm import LLM, SamplingParams
llm = LLM(
model="Quatfit/Quatfit-Mini",
speculative_model="Quatfit/Quatfit-Mini-MTP",
num_speculative_tokens=5,
max_model_len=131072,
)
sampling_params = SamplingParams(temperature=0.7, max_tokens=512)
outputs = llm.generate(["Explain the difference between GQA and MHA."], sampling_params)
print(outputs[0].outputs[0].text)
```
### With the FP8-served target model
```bash
vllm serve Quatfit/Quatfit-Mini-FP8 \
--quantization fp8 \
--speculative-model Quatfit/Quatfit-Mini-MTP \
--num-speculative-tokens 5
```
### llama.cpp
```bash
./llama-server \
-hf Quatfit/Quatfit-Mini-GGUF:Q4_K_M \
--model-draft Quatfit-Mini-MTP-Q4_K_M.gguf \
--draft-max 5 \
-c 131072
```
### 🤗 Transformers (assisted generation)
```python
import torch
from transformers import AutoModelForImageTextToText, AutoProcessor
target = AutoModelForImageTextToText.from_pretrained(
"Quatfit/Quatfit-Mini", torch_dtype="auto", device_map="auto"
)
drafter = AutoModelForImageTextToText.from_pretrained(
"Quatfit/Quatfit-Mini-MTP", torch_dtype="auto", device_map="auto"
)
processor = AutoProcessor.from_pretrained("Quatfit/Quatfit-Mini")
messages = [{"role": "user", "content": "Write a Python implementation of binary search."}]
inputs = processor.apply_chat_template(messages, tokenize=True, return_tensors="pt").to(target.device)
outputs = target.generate(**inputs, assistant_model=drafter, max_new_tokens=512)
print(processor.decode(outputs[0]))
```
---
## Methodology
The headline numbers above were measured as follows — please validate against your own workload before relying on them for capacity planning.
**Decode throughput (2.4×):** measured as end-to-end tokens/second during decode (post-prefill), comparing standard single-token autoregressive decoding against speculative decoding with this drafter, both targeting Quatfit Mini at the same precision and batch size, on a single H100. Throughput gain varies with batch size, sequence length, and task mix — larger batches and less-predictable generation (e.g. open-ended creative writing vs. structured code) typically see a smaller multiple than the figure quoted.
**Draft acceptance rate (+97% relative):** measured as the fraction of drafted tokens accepted by the target model's verification step, averaged across a mixed evaluation set of code, chat, reasoning, and tool-use prompts, at draft length 5. The "baseline" is an independently trained standalone draft model of comparable parameter count (~180M) with its own embedder, own KV cache, and no cross-attention into the target model — i.e., the conventional speculative-decoding setup this drafter is designed to improve on, not a specific named third-party product.
---
## Relationship to Other Quatfit Mini Repositories
| | Quatfit Mini (base) | Quatfit Mini FP8 | Quatfit Mini GGUF | **Quatfit Mini MTP (this repo)** |
|---|---|---|---|---|
| Role | Target model, full fidelity | Target model, GPU serving | Target model, local/CPU | Drafter, attaches to any of the above |
| Format | `safetensors`, FP32 | `safetensors`, FP8 + BF16 | `.gguf` | `safetensors` |
| Runtime | 🤗 Transformers | vLLM, TensorRT-LLM, SGLang | llama.cpp | vLLM, TensorRT-LLM, llama.cpp, Transformers |
| Size | ~32 GB | ~9.5 GB | ~3–16.8 GB per quant | ~180M params (~360 MB BF16) |
Full architecture details for both the target model and this drafter are documented in the [Quatfit Mini Technical Report](https://huggingface.co/Quatfit/Quatfit-Mini/resolve/main/Quatfit-Mini_Technical_Report.pdf).
---
## Responsible AI
This drafter does not generate final output on its own — all drafted tokens are verified by the target model before being returned, so speculative decoding does not change what the target model would otherwise produce; it only changes how fast those tokens are produced. Standard responsible-use guidance for Quatfit Mini itself still applies:
- Verify critical information
- Apply RAG for factual grounding
- Use application-level safety filters
- Keep human oversight for high-risk domains
---
## Citation
```bibtex
@article{quatfitminimtp2026,
title={Quatfit Mini MTP: A Cross-Attending Multi-Token-Prediction Drafter for Speculative Decoding},
author={Quatfit AI Research},
year={2026}
}
```
---
## License
Apache License 2.0. See [LICENSE](https://www.apache.org/licenses/LICENSE-2.0) for details.
---
🤗 MTP Repository •
🧠 Base Model (FP32) •
⚡ FP8 Build •
⚙️ GGUF Builds •
📄 Technical Report