Spaces:
Sleeping
Sleeping
| title: DSSD exact self-speculative decoding | |
| emoji: ⚡ | |
| colorFrom: indigo | |
| colorTo: pink | |
| sdk: gradio | |
| sdk_version: 6.29.1 | |
| python_version: "3.12" | |
| app_file: app.py | |
| license: apache-2.0 | |
| short_description: Exact self-speculative decoding with exit heads + copying | |
| models: | |
| - FlorianVal/DSSD-Qwen3-8B-Base | |
| - FlorianVal/Llama-3.1-8B-DSSD | |
| - Qwen/Qwen3-8B-Base | |
| - meta-llama/Llama-3.1-8B | |
| # DSSD demo | |
| Exact self-speculative decoding: a small exit head on layer 4 of the frozen model, plus n-gram copies from the | |
| context, propose a tree of draft tokens. The full model verifies it in one forward pass. Outputs are identical to | |
| greedy decoding. | |
| The demo shows: | |
| - each token coloured by its source: copy, exit head, head top-4 candidate, or full model; | |
| - the same prompt decoded autoregressively for comparison; | |
| - the measured speedup, tokens per verification, and a check that both outputs are identical; | |
| - a side-by-side replay of both generations at the measured speed. | |
| Models: Qwen3-8B-Base and Llama-3.1-8B (base models, plain-text completion). The heads are in | |
| [FlorianVal/DSSD-Qwen3-8B-Base](https://huggingface.co/FlorianVal/DSSD-Qwen3-8B-Base) and | |
| [FlorianVal/Llama-3.1-8B-DSSD](https://huggingface.co/FlorianVal/Llama-3.1-8B-DSSD). The Llama heads are built with Llama. | |
| ## Configuration | |
| | Variable | Default | Effect | | |
| |---|---|---| | |
| | `HF_TOKEN` (secret) | — | needed for the gated Llama base model and for private head repositories | | |
| | `DSSD_DEMO_MODELS` | both | comma-separated subset of `Qwen3-8B-Base,Llama-3.1-8B` | | |
| | `DSSD_GRAPHS` | off on ZeroGPU, on otherwise | `1` / `0` forces CUDA graphs | | |
| | `DSSD_DEMO_BACKEND` | `hub` | `tiny`: random tiny model on CPU, for local UI work | | |
| ## Local development | |
| ```bash | |
| uv venv .venv && uv pip install --python .venv/bin/python -r requirements.txt pytest | |
| DSSD_DEMO_BACKEND=tiny .venv/bin/python app.py # http://localhost:7860, no GPU needed | |
| .venv/bin/python -m pytest -q tests | |
| scripts/vendor_dssd.sh ../BranchyLLM-pub # refresh the vendored dssd package | |
| ``` | |
| The `dssd/` directory is a copy of the inference package from BranchyLLM (see `dssd/VENDORED.txt`); do not edit it here. | |