README.md

---
license: apache-2.0
license_name: apache-2.0
license_link: https://www.apache.org/licenses/LICENSE-2.0
base_model: Quatfit/Quatfit-Mini
language:
- en
- hi
- multilingual
- mr
- fr
- zh
- gu
- ja
- te
pipeline_tag: text-generation
tags:
- mtp
- multi token prediction
- speculative-decoding
- draft-model
- gemma4
- quatfit
- text-generation
---

<p align="center">
  <img src="https://huggingface.co/Quatfit/Quatfit-Mini/resolve/main/banner.png" alt="Quatfit Mini MTP Banner" width="100%">
</p>

<h1 align="center">Quatfit Mini — MTP</h1>
<h3 align="center">Standalone Multi-Token-Prediction Drafter for Speculative Decoding</h3>

<p align="center">
  <a href="https://huggingface.co/Quatfit/Quatfit-Mini-MTP"><img src="https://img.shields.io/badge/🤗%20Hugging%20Face-Model%20Hub-FFD21E?style=flat-square" alt="Hugging Face"></a>
  <a href="https://huggingface.co/Quatfit/Quatfit-Mini"><img src="https://img.shields.io/badge/🧠%20Target%20Model-Quatfit%20Mini-4052F5?style=flat-square" alt="Target Model"></a>
  <a href="https://huggingface.co/Quatfit/Quatfit-Mini-FP8"><img src="https://img.shields.io/badge/⚡%20Serving-FP8%20%7C%20GGUF-00C853?style=flat-square" alt="Serving"></a>
  <a href="https://www.apache.org/licenses/LICENSE-2.0"><img src="https://img.shields.io/badge/📜%20License-Apache%202.0-E53935?style=flat-square" alt="License"></a>
</p>

---

## Overview

**Quatfit Mini MTP** is the standalone **multi-token-prediction (MTP) drafter** used for speculative decoding with [Quatfit Mini](https://huggingface.co/Quatfit/Quatfit-Mini). Rather than a separate small language model, it is a lightweight autoregressive head that **cross-attends directly to the target model's own KV cache** — it has its own token embedder and a compact 4-layer transformer block (3 local sliding-window layers + 1 global layer), but no separate prefill pass and no independent context of its own to maintain.

This repository packages that drafter as a standalone download so it can be attached to any Quatfit Mini serving deployment (FP32, BF16, FP8, or GGUF) to accelerate decoding.

> This is not a general-purpose chat model — it drafts candidate continuations that the target model verifies. Use it together with **[Quatfit/Quatfit-Mini](https://huggingface.co/Quatfit/Quatfit-Mini)**, **[Quatfit/Quatfit-Mini-FP8](https://huggingface.co/Quatfit/Quatfit-Mini-FP8)**, or **[Quatfit/Quatfit-Mini-GGUF](https://huggingface.co/Quatfit/Quatfit-Mini-GGUF)**.

---

## Headline Results

| Metric | Result |
|---|---|
| Decode throughput vs. standard autoregressive decoding | **2.4×** |
| Draft token acceptance rate vs. a same-size standalone draft-model baseline | **+97% relative** (≈91% vs. ≈46%) |

See [Methodology](#methodology) below for exactly what these numbers measure and the conditions they were measured under — read that section before quoting these figures in your own comparisons.

---

## Why It's Faster and More Accurate Than a Standalone Draft Model

Most speculative-decoding setups pair a large target model with a small, independently-trained draft model that has to run its own forward pass and maintain its own context. Quatfit Mini MTP takes a different approach:

1. **Shared KV cache, no separate prefill.** The drafter cross-attends to the target model's existing KV cache instead of recomputing its own — it never needs to catch up on context the target model has already processed. This removes the drafter-prefill overhead that standalone draft models pay on every request.
2. **Trained jointly with the target model's representations.** Because the drafter consumes the target model's own last-layer activations and embeddings rather than approximating them from scratch, its predictions track the target model's actual output distribution more closely than an independently trained model of similar size — this is the main driver of the higher acceptance rate.
3. **Efficient decoding via top-k clustering.** Instead of projecting over the full 262,144-token vocabulary at every drafted step, the drafter's output head uses a top-k clustering operation, shrinking the final matrix multiplication from $d \times 262{,}144$ to $d \times 4{,}096$ — this keeps the drafter's own per-token cost low enough that its overhead doesn't eat into the speedup from higher acceptance.
4. **Arbitrary draft length.** Because there's no separate prefill to amortize, the drafter can propose a variable number of tokens per step rather than being locked to a fixed draft length, letting the serving stack tune the depth/acceptance trade-off per workload.

---

## Architecture

| Component | Value |
|---|---|
| Type | Autoregressive MTP drafter, cross-attends to target model KV cache |
| Target model | Quatfit Mini (8B, Gemma 4 architecture) |
| Drafter layers | 4 (3 local sliding-window + 1 global) |
| Hidden dimension | 512 |
| Attention heads | 4 |
| Own embedder | Yes — separate from target model's input embeddings |
| Own KV cache | No — cross-attends to target model's cache |
| Output head | Top-k clustered projection ($d \times 4{,}096$, not full vocab) |
| Parameters | ~180M |
| Precision | BF16 (FP8 variant available for FP8-served targets) |

---

## Usage

### vLLM (speculative decoding)

```bash
vllm serve Quatfit/Quatfit-Mini \
  --speculative-model Quatfit/Quatfit-Mini-MTP \
  --num-speculative-tokens 5 \
  --max-model-len 131072
```

```python
from vllm import LLM, SamplingParams

llm = LLM(
    model="Quatfit/Quatfit-Mini",
    speculative_model="Quatfit/Quatfit-Mini-MTP",
    num_speculative_tokens=5,
    max_model_len=131072,
)
sampling_params = SamplingParams(temperature=0.7, max_tokens=512)

outputs = llm.generate(["Explain the difference between GQA and MHA."], sampling_params)
print(outputs[0].outputs[0].text)
```

### With the FP8-served target model

```bash
vllm serve Quatfit/Quatfit-Mini-FP8 \
  --quantization fp8 \
  --speculative-model Quatfit/Quatfit-Mini-MTP \
  --num-speculative-tokens 5
```

### llama.cpp

```bash
./llama-server \
  -hf Quatfit/Quatfit-Mini-GGUF:Q4_K_M \
  --model-draft Quatfit-Mini-MTP-Q4_K_M.gguf \
  --draft-max 5 \
  -c 131072
```

### 🤗 Transformers (assisted generation)

```python
import torch
from transformers import AutoModelForImageTextToText, AutoProcessor

target = AutoModelForImageTextToText.from_pretrained(
    "Quatfit/Quatfit-Mini", torch_dtype="auto", device_map="auto"
)
drafter = AutoModelForImageTextToText.from_pretrained(
    "Quatfit/Quatfit-Mini-MTP", torch_dtype="auto", device_map="auto"
)
processor = AutoProcessor.from_pretrained("Quatfit/Quatfit-Mini")

messages = [{"role": "user", "content": "Write a Python implementation of binary search."}]
inputs = processor.apply_chat_template(messages, tokenize=True, return_tensors="pt").to(target.device)

outputs = target.generate(**inputs, assistant_model=drafter, max_new_tokens=512)
print(processor.decode(outputs[0]))
```

---

## Methodology

The headline numbers above were measured as follows — please validate against your own workload before relying on them for capacity planning.

**Decode throughput (2.4×):** measured as end-to-end tokens/second during decode (post-prefill), comparing standard single-token autoregressive decoding against speculative decoding with this drafter, both targeting Quatfit Mini at the same precision and batch size, on a single H100. Throughput gain varies with batch size, sequence length, and task mix — larger batches and less-predictable generation (e.g. open-ended creative writing vs. structured code) typically see a smaller multiple than the figure quoted.

**Draft acceptance rate (+97% relative):** measured as the fraction of drafted tokens accepted by the target model's verification step, averaged across a mixed evaluation set of code, chat, reasoning, and tool-use prompts, at draft length 5. The "baseline" is an independently trained standalone draft model of comparable parameter count (~180M) with its own embedder, own KV cache, and no cross-attention into the target model — i.e., the conventional speculative-decoding setup this drafter is designed to improve on, not a specific named third-party product.

---

## Relationship to Other Quatfit Mini Repositories

| | Quatfit Mini (base) | Quatfit Mini FP8 | Quatfit Mini GGUF | **Quatfit Mini MTP (this repo)** |
|---|---|---|---|---|
| Role | Target model, full fidelity | Target model, GPU serving | Target model, local/CPU | Drafter, attaches to any of the above |
| Format | `safetensors`, FP32 | `safetensors`, FP8 + BF16 | `.gguf` | `safetensors` |
| Runtime | 🤗 Transformers | vLLM, TensorRT-LLM, SGLang | llama.cpp | vLLM, TensorRT-LLM, llama.cpp, Transformers |
| Size | ~32 GB | ~9.5 GB | ~3–16.8 GB per quant | ~180M params (~360 MB BF16) |

Full architecture details for both the target model and this drafter are documented in the [Quatfit Mini Technical Report](https://huggingface.co/Quatfit/Quatfit-Mini/resolve/main/Quatfit-Mini_Technical_Report.pdf).

---

## Responsible AI

This drafter does not generate final output on its own — all drafted tokens are verified by the target model before being returned, so speculative decoding does not change what the target model would otherwise produce; it only changes how fast those tokens are produced. Standard responsible-use guidance for Quatfit Mini itself still applies:

- Verify critical information
- Apply RAG for factual grounding
- Use application-level safety filters
- Keep human oversight for high-risk domains

---

## Citation

```bibtex
@article{quatfitminimtp2026,
    title={Quatfit Mini MTP: A Cross-Attending Multi-Token-Prediction Drafter for Speculative Decoding},
    author={Quatfit AI Research},
    year={2026}
}
```

---

## License

Apache License 2.0. See [LICENSE](https://www.apache.org/licenses/LICENSE-2.0) for details.

---

<p align="center">
<a href="https://huggingface.co/Quatfit/Quatfit-Mini-MTP">🤗 MTP Repository</a> •
<a href="https://huggingface.co/Quatfit/Quatfit-Mini">🧠 Base Model (FP32)</a> •
<a href="https://huggingface.co/Quatfit/Quatfit-Mini-FP8">⚡ FP8 Build</a> •
<a href="https://huggingface.co/Quatfit/Quatfit-Mini-GGUF">⚙️ GGUF Builds</a> •
<a href="https://huggingface.co/Quatfit/Quatfit-Mini/resolve/main/Quatfit-Mini_Technical_Report.pdf">📄 Technical Report</a>
</p>