File size: 2,644 Bytes
397c7e9 dee07e0 397c7e9 dee07e0 397c7e9 dee07e0 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 | ---
title: Speculative Decoding on a 16 GB Consumer GPU
emoji: ⚡
colorFrom: blue
colorTo: green
sdk: static
pinned: false
license: mit
---
# Speculative Decoding on a 16 GB Consumer GPU
**Acceptance, modern drafters, and quantization in Gemma 4 12B and Qwen3-8B**
*Martin Ulises Millan Guerrero · 2026-08-08*
---
## Abstract
Speculative decoding (SD) is a mature family of lossless inference acceleration
techniques, yet the evidence for its modern drafters — EAGLE-3, DFlash, and
DSpark — comes almost entirely from datacenter GPUs serving batched workloads.
The community that runs quantized GGUF models on a single 16 GB consumer GPU has
no empirical guidance on which drafter to use, at which draft length, or with
which quantization.
This study presents the first systematic evaluation of modern drafters on a
16 GB consumer NVIDIA GPU (RTX 4060 Ti) with official checkpoints in llama.cpp:
**26 target × drafter configurations** over **1,474 greedy-decoded prompts**
across nine benchmarks in three cognitive domains, per-position acceptance
curves for 20 configurations, and a draft-length sweep (k ∈ {5, 7, 10}, 21
runs).
## Key results
| Finding | Result |
|---|---|
| **DSpark** fastest Qwen3-8B drafter | 1.64× at Q4 · 1.93× at Q8_0 |
| **EAGLE-3** robust, quantization-invariant | 1.39×–1.63× (acceptance nearly constant) |
| **Vanilla 1.7B** surprisingly competitive | 1.41× |
| **DFlash** fails on Qwen3-8B | 0.63× (α ≈ 0.009) |
| **DFlash / MTP on Gemma 4 12B** | 2.09×–2.30× / 2.39×–2.71× |
| **Confidence gating (p-min 0.6)** | acceptance 0.616 → 0.714, but throughput 87.6 → 80.7 tok/s (single-user) |
| **Quantization** | barely affects acceptance; Q4 draft saves ≈ 1 GiB VRAM |
## Resources
- **Companion repository (code, data, analysis):** [reyden009/speculative-decoding-lab](https://huggingface.co/reyden009/speculative-decoding-lab)
- **Paper (PDF):** [paper.pdf](https://huggingface.co/reyden009/speculative-decoding-lab/resolve/main/paper/paper.pdf)
- **arXiv-ready source bundle:** [arxiv-bundle.tar.gz](https://huggingface.co/reyden009/speculative-decoding-lab/resolve/main/paper/arxiv-bundle.tar.gz)
- **Markdown source:** [paper.md](https://huggingface.co/reyden009/speculative-decoding-lab/resolve/main/paper/paper.md)
## Full paper
<iframe src="https://huggingface.co/reyden009/speculative-decoding-lab/resolve/main/paper/paper.pdf" width="100%" height="1100px" style="border: 1px solid #ddd; border-radius: 4px;"></iframe>
---
*Setup: RTX 4060 Ti 16 GB · Intel i5-12400 · llama.cpp (CUDA) · Gemma 4 12B and Qwen3-8B GGUF · everything user-space, no sudo.*
|