mgoin commited on
Commit
7598747
·
verified ·
1 Parent(s): 2bf6542

Add epoch-2 vLLM acceptance-length results (64-prompt bench); restore deploy command

Browse files
Files changed (1) hide show
  1. README.md +33 -4
README.md CHANGED
@@ -137,8 +137,8 @@ Notes:
137
 
138
  ## Deployment
139
 
140
- DSpark inference support in vLLM is landing; once available, deploy with speculative
141
- decoding:
142
 
143
  ```bash
144
  vllm serve zai-org/GLM-5.2-FP8 \
@@ -146,9 +146,10 @@ vllm serve zai-org/GLM-5.2-FP8 \
146
  --max-model-len 16384 \
147
  --trust-remote-code \
148
  --speculative-config '{
149
- "model": "mgoin/GLM-5.2-speculator.dspark",
150
  "num_speculative_tokens": 7,
151
- "method": "dspark"
 
152
  }'
153
  ```
154
 
@@ -169,6 +170,34 @@ Per-position accuracy (positions 1-7), epoch 2:
169
 
170
  Still improving epoch-over-epoch (epoch 3 in training).
171
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
172
  ## References
173
 
174
  - **DFlash**: Block Diffusion for Flash Speculative Decoding (arXiv:2602.06036) — the
 
137
 
138
  ## Deployment
139
 
140
+ DSpark inference support is available on the vLLM nightly
141
+ (`uv pip install vllm --extra-index-url https://wheels.vllm.ai/nightly`):
142
 
143
  ```bash
144
  vllm serve zai-org/GLM-5.2-FP8 \
 
146
  --max-model-len 16384 \
147
  --trust-remote-code \
148
  --speculative-config '{
149
+ "model": "RedHatAI/GLM-5.2-speculator.dspark",
150
  "num_speculative_tokens": 7,
151
+ "method": "dspark",
152
+ "draft_sample_method": "probabilistic"
153
  }'
154
  ```
155
 
 
170
 
171
  Still improving epoch-over-epoch (epoch 3 in training).
172
 
173
+ ### Acceptance length in vLLM (revision `epoch-2`)
174
+
175
+ Measured end-to-end in vLLM speculative decoding (nightly
176
+ `0.23.1rc1.dev709+g2b753ad20`), serving `zai-org/GLM-5.2-FP8` on 4xB300 with
177
+ `num_speculative_tokens=7`, `draft_sample_method="probabilistic"`, greedy
178
+ sampling, batch size 1, 64 single-turn chat prompts (32 `HumanEval` + 32
179
+ `math_reasoning` from
180
+ [`RedHatAI/speculator_benchmarks`](https://huggingface.co/datasets/RedHatAI/speculator_benchmarks)),
181
+ 1024 output tokens each. Acceptance is computed from vLLM's
182
+ `spec_decode_num_accepted_tokens_per_pos` / `num_drafts` counter deltas.
183
+ The earlier
184
+ [`GLM-5.2-speculator.dspark-preview`](https://huggingface.co/RedHatAI/GLM-5.2-speculator.dspark-preview)
185
+ is included for reference (measured on a 16-prompt subset of the same set).
186
+
187
+ > This table is updated as later epochs are published; `main` currently
188
+ > points to **`epoch-2`**.
189
+
190
+ | Checkpoint | Pos 1 | Pos 2 | Pos 3 | Pos 4 | Pos 5 | Pos 6 | Pos 7 | Accept Len | Decode tok/s |
191
+ | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
192
+ | `epoch-2` (= `main`) | 74.7% | 56.0% | 41.5% | 31.1% | 23.7% | 18.0% | 13.2% | **3.58** | **225** |
193
+ | `epoch-1` | 74.7% | 55.4% | 40.2% | 29.5% | 21.5% | 15.4% | 10.9% | 3.48 | 219 |
194
+ | `dspark-preview` | 57.5% | 31.7% | 15.4% | 7.4% | 3.3% | 1.5% | 0.7% | 2.18 | 139 |
195
+
196
+ The epoch-2 gain comes from deeper draft positions (position-1 acceptance is
197
+ unchanged), consistent with continued training. For reference, the same
198
+ server without speculative decoding decodes at 102 tok/s (2.2x speedup for
199
+ `epoch-2`).
200
+
201
  ## References
202
 
203
  - **DFlash**: Block Diffusion for Flash Speculative Decoding (arXiv:2602.06036) — the