Florian valade commited on
Commit
f835254
Β·
1 Parent(s): 43e76c1

Add HuggingFace Spaces metadata to README

Browse files
Files changed (1) hide show
  1. README.md +29 -25
README.md CHANGED
@@ -1,34 +1,38 @@
1
- # DSSD Demo - Dynamic Self-Speculative Decoding
2
-
3
- A Gradio demo showcasing early exit inference with color-coded token visualization.
 
 
 
 
 
 
 
 
 
 
 
 
 
4
 
5
  ## Features
 
 
 
 
 
 
 
 
 
6
 
7
- - **Color-coded tokens**: Each token shows which head/layer generated it
8
- - **True early exit**: Actual speedup by stopping layer computation early
9
- - **Compare mode**: Side-by-side comparison with full model
10
- - **Model selection**: Switch between different DSSD models
11
 
12
- ## Quick Start
13
 
14
  ```bash
15
- # Install dependencies
16
  pip install -r requirements.txt
17
-
18
- # Run the demo
19
  python app.py
20
  ```
21
-
22
- Then open http://localhost:7860 in your browser.
23
-
24
- ## Models
25
-
26
- - **DSSD-Llama3-8B**: Llama 3 8B with 3 early exit heads at layers 8, 16, 24
27
- - **DSSD-Qwen3-0.6B**: Qwen3 0.6B with 4 early exit heads at layers 5, 11, 16, 22
28
-
29
- ## Color Legend
30
-
31
- - πŸ”΄ **Red**: Head 0 (earliest layer)
32
- - 🟠 **Orange**: Head 1
33
- - πŸ”΅ **Teal/Blue**: Head 2-3
34
- - 🟒 **Light Green**: Full model (all layers)
 
1
+ ---
2
+ title: DSSD Demo
3
+ emoji: πŸš€
4
+ colorFrom: blue
5
+ colorTo: green
6
+ sdk: gradio
7
+ sdk_version: 4.44.0
8
+ app_file: app.py
9
+ pinned: false
10
+ license: apache-2.0
11
+ ---
12
+
13
+ # πŸš€ Dynamic Self-Speculative Decoding (DSSD) Demo
14
+
15
+ This demo showcases **early exit inference** with true speculative decoding.
16
+ Tokens are generated from intermediate layers when the model is confident, resulting in faster generation while **guaranteeing output identical to the full model**.
17
 
18
  ## Features
19
+ - **Speculative Decoding**: Uses early exit heads to draft tokens, then verifies them with the full model.
20
+ - **Streaming Output**: Watch the generation process live, including drafting and verification statuses.
21
+ - **Model Comparison**: Compare performance and output between DSSD and the full model side-by-side.
22
+ - **Color-coded Visualization**: Each token is colored based on which head/layer generated it.
23
+
24
+ ## How it works
25
+ 1. **Draft Phase**: The model tries to predict the next token(s) using early exit heads placed at intermediate layers.
26
+ 2. **Verification Phase**: The full model checks the drafted tokens in a single forward pass.
27
+ 3. **Acceptance**: Matching tokens are kept. The first mismatch is corrected, and the process restarts.
28
 
29
+ ## Models
30
+ - **Llama 3 8B**: Using 3 auxiliary heads at layers 8, 16, and 24.
31
+ - **Qwen 3 0.6B**: Using 4 auxiliary heads at layers 5, 11, 16, and 22.
 
32
 
33
+ ## Quick Start (Local)
34
 
35
  ```bash
 
36
  pip install -r requirements.txt
 
 
37
  python app.py
38
  ```