| --- |
| title: Speculative Decoding on a 16 GB Consumer GPU |
| emoji: ⚡ |
| colorFrom: blue |
| colorTo: green |
| sdk: static |
| pinned: false |
| license: mit |
| --- |
| |
| # Speculative Decoding on a 16 GB Consumer GPU |
|
|
| **Acceptance, modern drafters, and quantization in Gemma 4 12B and Qwen3-8B** |
|
|
| *Martin Ulises Millan Guerrero · 2026-08-08* |
|
|
| --- |
|
|
| ## Abstract |
|
|
| Speculative decoding (SD) is a mature family of lossless inference acceleration |
| techniques, yet the evidence for its modern drafters — EAGLE-3, DFlash, and |
| DSpark — comes almost entirely from datacenter GPUs serving batched workloads. |
| The community that runs quantized GGUF models on a single 16 GB consumer GPU has |
| no empirical guidance on which drafter to use, at which draft length, or with |
| which quantization. |
|
|
| This study presents the first systematic evaluation of modern drafters on a |
| 16 GB consumer NVIDIA GPU (RTX 4060 Ti) with official checkpoints in llama.cpp: |
| **26 target × drafter configurations** over **1,474 greedy-decoded prompts** |
| across nine benchmarks in three cognitive domains, per-position acceptance |
| curves for 20 configurations, and a draft-length sweep (k ∈ {5, 7, 10}, 21 |
| runs). |
|
|
| ## Key results |
|
|
| | Finding | Result | |
| |---|---| |
| | **DSpark** fastest Qwen3-8B drafter | 1.64× at Q4 · 1.93× at Q8_0 | |
| | **EAGLE-3** robust, quantization-invariant | 1.39×–1.63× (acceptance nearly constant) | |
| | **Vanilla 1.7B** surprisingly competitive | 1.41× | |
| | **DFlash** fails on Qwen3-8B | 0.63× (α ≈ 0.009) | |
| | **DFlash / MTP on Gemma 4 12B** | 2.09×–2.30× / 2.39×–2.71× | |
| | **Confidence gating (p-min 0.6)** | acceptance 0.616 → 0.714, but throughput 87.6 → 80.7 tok/s (single-user) | |
| | **Quantization** | barely affects acceptance; Q4 draft saves ≈ 1 GiB VRAM | |
| |
| ## Resources |
| |
| - **Companion repository (code, data, analysis):** [reyden009/speculative-decoding-lab](https://huggingface.co/reyden009/speculative-decoding-lab) |
| - **Paper (PDF):** [paper.pdf](https://huggingface.co/reyden009/speculative-decoding-lab/resolve/main/paper/paper.pdf) |
| - **arXiv-ready source bundle:** [arxiv-bundle.tar.gz](https://huggingface.co/reyden009/speculative-decoding-lab/resolve/main/paper/arxiv-bundle.tar.gz) |
| - **Markdown source:** [paper.md](https://huggingface.co/reyden009/speculative-decoding-lab/resolve/main/paper/paper.md) |
| |
| ## Full paper |
| |
| <iframe src="https://huggingface.co/reyden009/speculative-decoding-lab/resolve/main/paper/paper.pdf" width="100%" height="1100px" style="border: 1px solid #ddd; border-radius: 4px;"></iframe> |
| |
| --- |
| |
| *Setup: RTX 4060 Ti 16 GB · Intel i5-12400 · llama.cpp (CUDA) · Gemma 4 12B and Qwen3-8B GGUF · everything user-space, no sudo.* |
| |
