Add epoch-2 vLLM acceptance-length results (64-prompt bench); restore deploy command
Browse files
README.md
CHANGED
|
@@ -137,8 +137,8 @@ Notes:
|
|
| 137 |
|
| 138 |
## Deployment
|
| 139 |
|
| 140 |
-
DSpark inference support
|
| 141 |
-
|
| 142 |
|
| 143 |
```bash
|
| 144 |
vllm serve zai-org/GLM-5.2-FP8 \
|
|
@@ -146,9 +146,10 @@ vllm serve zai-org/GLM-5.2-FP8 \
|
|
| 146 |
--max-model-len 16384 \
|
| 147 |
--trust-remote-code \
|
| 148 |
--speculative-config '{
|
| 149 |
-
"model": "
|
| 150 |
"num_speculative_tokens": 7,
|
| 151 |
-
"method": "dspark"
|
|
|
|
| 152 |
}'
|
| 153 |
```
|
| 154 |
|
|
@@ -169,6 +170,34 @@ Per-position accuracy (positions 1-7), epoch 2:
|
|
| 169 |
|
| 170 |
Still improving epoch-over-epoch (epoch 3 in training).
|
| 171 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 172 |
## References
|
| 173 |
|
| 174 |
- **DFlash**: Block Diffusion for Flash Speculative Decoding (arXiv:2602.06036) — the
|
|
|
|
| 137 |
|
| 138 |
## Deployment
|
| 139 |
|
| 140 |
+
DSpark inference support is available on the vLLM nightly
|
| 141 |
+
(`uv pip install vllm --extra-index-url https://wheels.vllm.ai/nightly`):
|
| 142 |
|
| 143 |
```bash
|
| 144 |
vllm serve zai-org/GLM-5.2-FP8 \
|
|
|
|
| 146 |
--max-model-len 16384 \
|
| 147 |
--trust-remote-code \
|
| 148 |
--speculative-config '{
|
| 149 |
+
"model": "RedHatAI/GLM-5.2-speculator.dspark",
|
| 150 |
"num_speculative_tokens": 7,
|
| 151 |
+
"method": "dspark",
|
| 152 |
+
"draft_sample_method": "probabilistic"
|
| 153 |
}'
|
| 154 |
```
|
| 155 |
|
|
|
|
| 170 |
|
| 171 |
Still improving epoch-over-epoch (epoch 3 in training).
|
| 172 |
|
| 173 |
+
### Acceptance length in vLLM (revision `epoch-2`)
|
| 174 |
+
|
| 175 |
+
Measured end-to-end in vLLM speculative decoding (nightly
|
| 176 |
+
`0.23.1rc1.dev709+g2b753ad20`), serving `zai-org/GLM-5.2-FP8` on 4xB300 with
|
| 177 |
+
`num_speculative_tokens=7`, `draft_sample_method="probabilistic"`, greedy
|
| 178 |
+
sampling, batch size 1, 64 single-turn chat prompts (32 `HumanEval` + 32
|
| 179 |
+
`math_reasoning` from
|
| 180 |
+
[`RedHatAI/speculator_benchmarks`](https://huggingface.co/datasets/RedHatAI/speculator_benchmarks)),
|
| 181 |
+
1024 output tokens each. Acceptance is computed from vLLM's
|
| 182 |
+
`spec_decode_num_accepted_tokens_per_pos` / `num_drafts` counter deltas.
|
| 183 |
+
The earlier
|
| 184 |
+
[`GLM-5.2-speculator.dspark-preview`](https://huggingface.co/RedHatAI/GLM-5.2-speculator.dspark-preview)
|
| 185 |
+
is included for reference (measured on a 16-prompt subset of the same set).
|
| 186 |
+
|
| 187 |
+
> This table is updated as later epochs are published; `main` currently
|
| 188 |
+
> points to **`epoch-2`**.
|
| 189 |
+
|
| 190 |
+
| Checkpoint | Pos 1 | Pos 2 | Pos 3 | Pos 4 | Pos 5 | Pos 6 | Pos 7 | Accept Len | Decode tok/s |
|
| 191 |
+
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|
| 192 |
+
| `epoch-2` (= `main`) | 74.7% | 56.0% | 41.5% | 31.1% | 23.7% | 18.0% | 13.2% | **3.58** | **225** |
|
| 193 |
+
| `epoch-1` | 74.7% | 55.4% | 40.2% | 29.5% | 21.5% | 15.4% | 10.9% | 3.48 | 219 |
|
| 194 |
+
| `dspark-preview` | 57.5% | 31.7% | 15.4% | 7.4% | 3.3% | 1.5% | 0.7% | 2.18 | 139 |
|
| 195 |
+
|
| 196 |
+
The epoch-2 gain comes from deeper draft positions (position-1 acceptance is
|
| 197 |
+
unchanged), consistent with continued training. For reference, the same
|
| 198 |
+
server without speculative decoding decodes at 102 tok/s (2.2x speedup for
|
| 199 |
+
`epoch-2`).
|
| 200 |
+
|
| 201 |
## References
|
| 202 |
|
| 203 |
- **DFlash**: Block Diffusion for Flash Speculative Decoding (arXiv:2602.06036) — the
|