| --- |
| license: mit |
| tags: |
| - speculative-decoding |
| - llm-inference |
| - benchmark |
| - quantization |
| - gguf |
| - llama.cpp |
| --- |
| |
| # Speculative Decoding Lab |
|
|
| Systematic, reproducible empirical evaluation of **modern speculative decoding** |
| (EAGLE-3, DFlash, DSpark vs. vanilla and model-free) on an **RTX 4060 Ti 16 GB** |
| with **Gemma 4 12B** and **Qwen3-8B** GGUF-quantized models: acceptance by |
| cognitive domain, quantization effects, and hardware break-even maps. |
|
|
| > Paper: [PDF](paper/paper.pdf) · [arXiv-ready source bundle](paper/arxiv-bundle.tar.gz) · [source (markdown)](paper/paper.md) · [paper page](https://huggingface.co/spaces/reyden009/speculative-decoding-paper) |
|
|
| ## Hardware used |
|
|
| The study was developed and benchmarked on this machine, so all reported |
| numbers (speedups, acceptance, break-even) are tied to these specs. The code |
| itself is portable: it only needs llama.cpp (CUDA build) and enough VRAM for |
| the target model, and can be re-run on any other hardware. |
|
|
| | Component | Detail | |
| | --- | --- | |
| | GPU | NVIDIA RTX 4060 Ti **16 GB** (Ada Lovelace, compute 8.9) | |
| | CPU | Intel i5-12400 (6C/12T, 4.4 GHz) | |
| | RAM / Disk | 31 GB / ~185 GB free | |
| | OS | Ubuntu 24.04 · driver 595.84 | |
|
|
| Environment principle: **everything user-space, no sudo** (uv + CUDA runfile in `$HOME`). |
|
|
| ## Stack |
|
|
| - **Python 3.12 + uv** — `pyproject.toml` + `uv.lock` (numpy, ruff, pyright, cmake, ninja). |
| - **CUDA toolkit 12.8.1** in `~/cuda` (runfile `--toolkitpath`). *Note: the `nvcc` binary is not on PyPI (the wheels only ship ptxas).* |
| - **llama.cpp master 0713275 (v22)** (CUDA build) in `~/llama.cpp` (built from source; no CUDA binaries are published for Linux). |
| - **GGUF models** in `models/` (not versioned; downloaded with `curl -C -`). |
|
|
| ## Setup (once) |
|
|
| ```bash |
| uv sync # create .venv with all dependencies |
| source scripts/spec-env.sh # PATH + CUDA_HOME + env vars (idempotent) |
| ``` |
|
|
| ## Quality (required before committing) |
|
|
| ```bash |
| uv run ruff check . # lint |
| uv run ruff format --check . # formatting |
| uv run pyright # types |
| ``` |
|
|
| ## Execution regime |
|
|
| Overnight sessions with **pause/resume**: `scripts/bench_accept.py --resume` |
| (incremental JSONL per prompt; server-based runner with α/τ), downloads with |
| `scripts/dl.py`, runs in tmux/nohup. Measurement campaigns are launched with |
| `scripts/run_final_chain.sh`, `scripts/run_curves_chain.sh` and |
| `scripts/run_ksweep_chain.sh`; final analysis with `scripts/analyze_final.py`. |
|
|
| ## Structure |
|
|
| ``` |
| scripts/ environment, downloads, benchmark runner and analysis (bench_accept.py, analyze_final.py) |
| experiments/ runs/ (raw, not versioned) and prompts/ (JSONL subsamples) |
| models/ GGUFs (not versioned) |
| ``` |
|
|
| ## For agents |
|
|
| Read [`AGENTS.md`](AGENTS.md) before working in this repo (conventions, git |
| rules, sessions and handoff). |
|
|
| ## License |
|
|
| MIT — see [`LICENSE`](LICENSE). |
|
|
