Florian valade commited on
Commit Β·
f835254
1
Parent(s): 43e76c1
Add HuggingFace Spaces metadata to README
Browse files
README.md
CHANGED
|
@@ -1,34 +1,38 @@
|
|
| 1 |
-
|
| 2 |
-
|
| 3 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 4 |
|
| 5 |
## Features
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 6 |
|
| 7 |
-
|
| 8 |
-
- **
|
| 9 |
-
- **
|
| 10 |
-
- **Model selection**: Switch between different DSSD models
|
| 11 |
|
| 12 |
-
## Quick Start
|
| 13 |
|
| 14 |
```bash
|
| 15 |
-
# Install dependencies
|
| 16 |
pip install -r requirements.txt
|
| 17 |
-
|
| 18 |
-
# Run the demo
|
| 19 |
python app.py
|
| 20 |
```
|
| 21 |
-
|
| 22 |
-
Then open http://localhost:7860 in your browser.
|
| 23 |
-
|
| 24 |
-
## Models
|
| 25 |
-
|
| 26 |
-
- **DSSD-Llama3-8B**: Llama 3 8B with 3 early exit heads at layers 8, 16, 24
|
| 27 |
-
- **DSSD-Qwen3-0.6B**: Qwen3 0.6B with 4 early exit heads at layers 5, 11, 16, 22
|
| 28 |
-
|
| 29 |
-
## Color Legend
|
| 30 |
-
|
| 31 |
-
- π΄ **Red**: Head 0 (earliest layer)
|
| 32 |
-
- π **Orange**: Head 1
|
| 33 |
-
- π΅ **Teal/Blue**: Head 2-3
|
| 34 |
-
- π’ **Light Green**: Full model (all layers)
|
|
|
|
| 1 |
+
---
|
| 2 |
+
title: DSSD Demo
|
| 3 |
+
emoji: π
|
| 4 |
+
colorFrom: blue
|
| 5 |
+
colorTo: green
|
| 6 |
+
sdk: gradio
|
| 7 |
+
sdk_version: 4.44.0
|
| 8 |
+
app_file: app.py
|
| 9 |
+
pinned: false
|
| 10 |
+
license: apache-2.0
|
| 11 |
+
---
|
| 12 |
+
|
| 13 |
+
# π Dynamic Self-Speculative Decoding (DSSD) Demo
|
| 14 |
+
|
| 15 |
+
This demo showcases **early exit inference** with true speculative decoding.
|
| 16 |
+
Tokens are generated from intermediate layers when the model is confident, resulting in faster generation while **guaranteeing output identical to the full model**.
|
| 17 |
|
| 18 |
## Features
|
| 19 |
+
- **Speculative Decoding**: Uses early exit heads to draft tokens, then verifies them with the full model.
|
| 20 |
+
- **Streaming Output**: Watch the generation process live, including drafting and verification statuses.
|
| 21 |
+
- **Model Comparison**: Compare performance and output between DSSD and the full model side-by-side.
|
| 22 |
+
- **Color-coded Visualization**: Each token is colored based on which head/layer generated it.
|
| 23 |
+
|
| 24 |
+
## How it works
|
| 25 |
+
1. **Draft Phase**: The model tries to predict the next token(s) using early exit heads placed at intermediate layers.
|
| 26 |
+
2. **Verification Phase**: The full model checks the drafted tokens in a single forward pass.
|
| 27 |
+
3. **Acceptance**: Matching tokens are kept. The first mismatch is corrected, and the process restarts.
|
| 28 |
|
| 29 |
+
## Models
|
| 30 |
+
- **Llama 3 8B**: Using 3 auxiliary heads at layers 8, 16, and 24.
|
| 31 |
+
- **Qwen 3 0.6B**: Using 4 auxiliary heads at layers 5, 11, 16, and 22.
|
|
|
|
| 32 |
|
| 33 |
+
## Quick Start (Local)
|
| 34 |
|
| 35 |
```bash
|
|
|
|
| 36 |
pip install -r requirements.txt
|
|
|
|
|
|
|
| 37 |
python app.py
|
| 38 |
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|