Spaces:
Sleeping
Sleeping
File size: 2,152 Bytes
c0c2411 74679d4 c0c2411 74679d4 c0c2411 74679d4 c0c2411 74679d4 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 | ---
title: DSSD exact self-speculative decoding
emoji: ⚡
colorFrom: indigo
colorTo: pink
sdk: gradio
sdk_version: 6.29.1
python_version: "3.12"
app_file: app.py
license: apache-2.0
short_description: Exact self-speculative decoding with exit heads + copying
models:
- FlorianVal/DSSD-Qwen3-8B-Base
- FlorianVal/Llama-3.1-8B-DSSD
- Qwen/Qwen3-8B-Base
- meta-llama/Llama-3.1-8B
---
# DSSD demo
Exact self-speculative decoding: a small exit head on layer 4 of the frozen model, plus n-gram copies from the
context, propose a tree of draft tokens. The full model verifies it in one forward pass. Outputs are identical to
greedy decoding.
The demo shows:
- each token coloured by its source: copy, exit head, head top-4 candidate, or full model;
- the same prompt decoded autoregressively for comparison;
- the measured speedup, tokens per verification, and a check that both outputs are identical;
- a side-by-side replay of both generations at the measured speed.
Models: Qwen3-8B-Base and Llama-3.1-8B (base models, plain-text completion). The heads are in
[FlorianVal/DSSD-Qwen3-8B-Base](https://huggingface.co/FlorianVal/DSSD-Qwen3-8B-Base) and
[FlorianVal/Llama-3.1-8B-DSSD](https://huggingface.co/FlorianVal/Llama-3.1-8B-DSSD). The Llama heads are built with Llama.
## Configuration
| Variable | Default | Effect |
|---|---|---|
| `HF_TOKEN` (secret) | — | needed for the gated Llama base model and for private head repositories |
| `DSSD_DEMO_MODELS` | both | comma-separated subset of `Qwen3-8B-Base,Llama-3.1-8B` |
| `DSSD_GRAPHS` | off on ZeroGPU, on otherwise | `1` / `0` forces CUDA graphs |
| `DSSD_DEMO_BACKEND` | `hub` | `tiny`: random tiny model on CPU, for local UI work |
## Local development
```bash
uv venv .venv && uv pip install --python .venv/bin/python -r requirements.txt pytest
DSSD_DEMO_BACKEND=tiny .venv/bin/python app.py # http://localhost:7860, no GPU needed
.venv/bin/python -m pytest -q tests
scripts/vendor_dssd.sh ../BranchyLLM-pub # refresh the vendored dssd package
```
The `dssd/` directory is a copy of the inference package from BranchyLLM (see `dssd/VENDORED.txt`); do not edit it here.
|