README.md

metadata

title: DSSD exact self-speculative decoding
emoji: ⚡
colorFrom: indigo
colorTo: pink
sdk: gradio
sdk_version: 6.29.1
python_version: '3.12'
app_file: app.py
license: apache-2.0
short_description: Exact self-speculative decoding with exit heads + copying
models:
  - FlorianVal/DSSD-Qwen3-8B-Base
  - FlorianVal/Llama-3.1-8B-DSSD
  - Qwen/Qwen3-8B-Base
  - meta-llama/Llama-3.1-8B

DSSD demo

Exact self-speculative decoding: a small exit head on layer 4 of the frozen model, plus n-gram copies from the context, propose a tree of draft tokens. The full model verifies it in one forward pass. Outputs are identical to greedy decoding.

The demo shows:

• each token coloured by its source: copy, exit head, head top-4 candidate, or full model;
• the same prompt decoded autoregressively for comparison;
• the measured speedup, tokens per verification, and a check that both outputs are identical;
• a side-by-side replay of both generations at the measured speed.

Models: Qwen3-8B-Base and Llama-3.1-8B (base models, plain-text completion). The heads are in FlorianVal/DSSD-Qwen3-8B-Base and FlorianVal/Llama-3.1-8B-DSSD. The Llama heads are built with Llama.

Configuration

Variable	Default	Effect
HF_TOKEN (secret)	—	needed for the gated Llama base model and for private head repositories
DSSD_DEMO_MODELS	both	comma-separated subset of Qwen3-8B-Base,Llama-3.1-8B
DSSD_GRAPHS	off on ZeroGPU, on otherwise	1 / 0 forces CUDA graphs
DSSD_DEMO_BACKEND	hub	tiny: random tiny model on CPU, for local UI work

Local development

uv venv .venv && uv pip install --python .venv/bin/python -r requirements.txt pytest
DSSD_DEMO_BACKEND=tiny .venv/bin/python app.py      # http://localhost:7860, no GPU needed
.venv/bin/python -m pytest -q tests
scripts/vendor_dssd.sh ../BranchyLLM-pub           # refresh the vendored dssd package

The dssd/ directory is a copy of the inference package from BranchyLLM (see dssd/VENDORED.txt); do not edit it here.