title: Speculative Decoding on a 16 GB Consumer GPU
emoji: ⚡
colorFrom: blue
colorTo: green
sdk: static
pinned: false
license: mit
Speculative Decoding on a 16 GB Consumer GPU
Acceptance, modern drafters, and quantization in Gemma 4 12B and Qwen3-8B
Martin Ulises Millan Guerrero · 2026-08-08
Abstract
Speculative decoding (SD) is a mature family of lossless inference acceleration techniques, yet the evidence for its modern drafters — EAGLE-3, DFlash, and DSpark — comes almost entirely from datacenter GPUs serving batched workloads. The community that runs quantized GGUF models on a single 16 GB consumer GPU has no empirical guidance on which drafter to use, at which draft length, or with which quantization.
This study presents the first systematic evaluation of modern drafters on a 16 GB consumer NVIDIA GPU (RTX 4060 Ti) with official checkpoints in llama.cpp: 26 target × drafter configurations over 1,474 greedy-decoded prompts across nine benchmarks in three cognitive domains, per-position acceptance curves for 20 configurations, and a draft-length sweep (k ∈ {5, 7, 10}, 21 runs).
Key results
| Finding | Result |
|---|---|
| DSpark fastest Qwen3-8B drafter | 1.64× at Q4 · 1.93× at Q8_0 |
| EAGLE-3 robust, quantization-invariant | 1.39×–1.63× (acceptance nearly constant) |
| Vanilla 1.7B surprisingly competitive | 1.41× |
| DFlash fails on Qwen3-8B | 0.63× (α ≈ 0.009) |
| DFlash / MTP on Gemma 4 12B | 2.09×–2.30× / 2.39×–2.71× |
| Confidence gating (p-min 0.6) | acceptance 0.616 → 0.714, but throughput 87.6 → 80.7 tok/s (single-user) |
| Quantization | barely affects acceptance; Q4 draft saves ≈ 1 GiB VRAM |
Resources
- Companion repository (code, data, analysis): reyden009/speculative-decoding-lab
- Paper (PDF): paper.pdf
- arXiv-ready source bundle: arxiv-bundle.tar.gz
- Markdown source: paper.md
Full paper
Setup: RTX 4060 Ti 16 GB · Intel i5-12400 · llama.cpp (CUDA) · Gemma 4 12B and Qwen3-8B GGUF · everything user-space, no sudo.