Hugging Face's logo

reyden009
/
speculative-decoding-lab

GGUF
speculative-decoding
llm-inference
benchmark
quantization
llama.cpp
Model card Files
xet
 Community
speculative-decoding-lab / README.md
reyden009's picture
reyden009
Upload README.md with huggingface_hub
9421f4f verified
preview code
|
Raw
History Blame Contribute Delete
2.9 kB
metadata 
license: mit
tags:
  - speculative-decoding
  - llm-inference
  - benchmark
  - quantization
  - gguf
  - llama.cpp

Speculative Decoding Lab

Systematic, reproducible empirical evaluation of modern speculative decoding (EAGLE-3, DFlash, DSpark vs. vanilla and model-free) on an RTX 4060 Ti 16 GB with Gemma 4 12B and Qwen3-8B GGUF-quantized models: acceptance by cognitive domain, quantization effects, and hardware break-even maps.

Paper: PDF · arXiv-ready source bundle · source (markdown) · paper page

Hardware used

The study was developed and benchmarked on this machine, so all reported numbers (speedups, acceptance, break-even) are tied to these specs. The code itself is portable: it only needs llama.cpp (CUDA build) and enough VRAM for the target model, and can be re-run on any other hardware.

Component Detail
GPU NVIDIA RTX 4060 Ti 16 GB (Ada Lovelace, compute 8.9)
CPU Intel i5-12400 (6C/12T, 4.4 GHz)
RAM / Disk 31 GB / ~185 GB free
OS Ubuntu 24.04 · driver 595.84

Environment principle: everything user-space, no sudo (uv + CUDA runfile in $HOME).

Stack

  • Python 3.12 + uv — pyproject.toml + uv.lock (numpy, ruff, pyright, cmake, ninja).
  • CUDA toolkit 12.8.1 in ~/cuda (runfile --toolkitpath). Note: the nvcc binary is not on PyPI (the wheels only ship ptxas).
  • llama.cpp master 0713275 (v22) (CUDA build) in ~/llama.cpp (built from source; no CUDA binaries are published for Linux).
  • GGUF models in models/ (not versioned; downloaded with curl -C -).

Setup (once) 

uv sync                        # create .venv with all dependencies
source scripts/spec-env.sh     # PATH + CUDA_HOME + env vars (idempotent)

Quality (required before committing) 

uv run ruff check .            # lint
uv run ruff format --check .   # formatting
uv run pyright                 # types

Execution regime

Overnight sessions with pause/resume: scripts/bench_accept.py --resume (incremental JSONL per prompt; server-based runner with α/τ), downloads with scripts/dl.py, runs in tmux/nohup. Measurement campaigns are launched with scripts/run_final_chain.sh, scripts/run_curves_chain.sh and scripts/run_ksweep_chain.sh; final analysis with scripts/analyze_final.py.

Structure 

scripts/       environment, downloads, benchmark runner and analysis (bench_accept.py, analyze_final.py)
experiments/   runs/ (raw, not versioned) and prompts/ (JSONL subsamples)
models/        GGUFs (not versioned)

For agents

Read AGENTS.md before working in this repo (conventions, git rules, sessions and handoff).

License

MIT — see LICENSE.