File size: 2,896 Bytes
c941abf d4c2896 9421f4f d4c2896 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 | ---
license: mit
tags:
- speculative-decoding
- llm-inference
- benchmark
- quantization
- gguf
- llama.cpp
---
# Speculative Decoding Lab
Systematic, reproducible empirical evaluation of **modern speculative decoding**
(EAGLE-3, DFlash, DSpark vs. vanilla and model-free) on an **RTX 4060 Ti 16 GB**
with **Gemma 4 12B** and **Qwen3-8B** GGUF-quantized models: acceptance by
cognitive domain, quantization effects, and hardware break-even maps.
> Paper: [PDF](paper/paper.pdf) · [arXiv-ready source bundle](paper/arxiv-bundle.tar.gz) · [source (markdown)](paper/paper.md) · [paper page](https://huggingface.co/spaces/reyden009/speculative-decoding-paper)
## Hardware used
The study was developed and benchmarked on this machine, so all reported
numbers (speedups, acceptance, break-even) are tied to these specs. The code
itself is portable: it only needs llama.cpp (CUDA build) and enough VRAM for
the target model, and can be re-run on any other hardware.
| Component | Detail |
| --- | --- |
| GPU | NVIDIA RTX 4060 Ti **16 GB** (Ada Lovelace, compute 8.9) |
| CPU | Intel i5-12400 (6C/12T, 4.4 GHz) |
| RAM / Disk | 31 GB / ~185 GB free |
| OS | Ubuntu 24.04 · driver 595.84 |
Environment principle: **everything user-space, no sudo** (uv + CUDA runfile in `$HOME`).
## Stack
- **Python 3.12 + uv** — `pyproject.toml` + `uv.lock` (numpy, ruff, pyright, cmake, ninja).
- **CUDA toolkit 12.8.1** in `~/cuda` (runfile `--toolkitpath`). *Note: the `nvcc` binary is not on PyPI (the wheels only ship ptxas).*
- **llama.cpp master 0713275 (v22)** (CUDA build) in `~/llama.cpp` (built from source; no CUDA binaries are published for Linux).
- **GGUF models** in `models/` (not versioned; downloaded with `curl -C -`).
## Setup (once)
```bash
uv sync # create .venv with all dependencies
source scripts/spec-env.sh # PATH + CUDA_HOME + env vars (idempotent)
```
## Quality (required before committing)
```bash
uv run ruff check . # lint
uv run ruff format --check . # formatting
uv run pyright # types
```
## Execution regime
Overnight sessions with **pause/resume**: `scripts/bench_accept.py --resume`
(incremental JSONL per prompt; server-based runner with α/τ), downloads with
`scripts/dl.py`, runs in tmux/nohup. Measurement campaigns are launched with
`scripts/run_final_chain.sh`, `scripts/run_curves_chain.sh` and
`scripts/run_ksweep_chain.sh`; final analysis with `scripts/analyze_final.py`.
## Structure
```
scripts/ environment, downloads, benchmark runner and analysis (bench_accept.py, analyze_final.py)
experiments/ runs/ (raw, not versioned) and prompts/ (JSONL subsamples)
models/ GGUFs (not versioned)
```
## For agents
Read [`AGENTS.md`](AGENTS.md) before working in this repo (conventions, git
rules, sessions and handoff).
## License
MIT — see [`LICENSE`](LICENSE).
|