Upload README.md with huggingface_hub
Browse files
README.md
CHANGED
|
@@ -21,6 +21,8 @@ intra-block token dependency) and a **per-position confidence head** (accept-rat
|
|
| 21 |
prediction). Trained with the [speculators](https://github.com/vllm-project/speculators)
|
| 22 |
library.
|
| 23 |
|
|
|
|
|
|
|
| 24 |
## Model Specifications
|
| 25 |
|
| 26 |
- **Base Model**: zai-org/GLM-5.2-FP8
|
|
@@ -32,22 +34,39 @@ library.
|
|
| 32 |
|
| 33 |
## Checkpoint series
|
| 34 |
|
| 35 |
-
|
| 36 |
-
|
| 37 |
|
| 38 |
| revision | epoch | status |
|
| 39 |
| --- | --- | --- |
|
| 40 |
| `epoch-1` | 1 / 3 | ✅ available |
|
| 41 |
-
| `epoch-2` | 2 / 3 | ✅
|
| 42 |
-
| `epoch-3` | 3 / 3 |
|
| 43 |
|
| 44 |
```python
|
| 45 |
from transformers import AutoModel
|
| 46 |
model = AutoModel.from_pretrained(
|
| 47 |
-
"mgoin/GLM-5.2-speculator.dspark", revision="epoch-
|
| 48 |
)
|
| 49 |
```
|
| 50 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 51 |
## Training Details
|
| 52 |
|
| 53 |
The model was trained using the Speculators library on prompts from
|
|
@@ -133,12 +152,11 @@ Notes:
|
|
| 133 |
- DSpark-specific flags: `--markov-rank`, `--enable-confidence-head`,
|
| 134 |
`--confidence-head-with-markov`, `--confidence-head-alpha`. Dropping them (and
|
| 135 |
using `--speculator-type dflash`) recovers a plain DFlash draft.
|
| 136 |
-
- `--checkpoint-freq 0.2` writes sub-epoch checkpoints (~5 per epoch) for resumability.
|
| 137 |
|
| 138 |
## Deployment
|
| 139 |
|
| 140 |
-
DSpark inference support is
|
| 141 |
-
|
| 142 |
|
| 143 |
```bash
|
| 144 |
vllm serve zai-org/GLM-5.2-FP8 \
|
|
@@ -146,58 +164,12 @@ vllm serve zai-org/GLM-5.2-FP8 \
|
|
| 146 |
--max-model-len 16384 \
|
| 147 |
--trust-remote-code \
|
| 148 |
--speculative-config '{
|
| 149 |
-
"model": "
|
| 150 |
"num_speculative_tokens": 7,
|
| 151 |
-
"method": "dspark"
|
| 152 |
-
"draft_sample_method": "probabilistic"
|
| 153 |
}'
|
| 154 |
```
|
| 155 |
|
| 156 |
-
## Evaluation Results
|
| 157 |
-
|
| 158 |
-
Train-set metrics at the end of epoch 2 (per-epoch validation passes did not
|
| 159 |
-
complete due to server restarts; per-dataset acceptance evaluation will accompany
|
| 160 |
-
the final checkpoint):
|
| 161 |
-
|
| 162 |
-
| metric | epoch 1 | epoch 2 |
|
| 163 |
-
| --- | --- | --- |
|
| 164 |
-
| mean accepted length | 3.376 | **3.819** |
|
| 165 |
-
| full accuracy | 0.532 | 0.587 |
|
| 166 |
-
| mean acceptance rate | 0.497 | 0.557 |
|
| 167 |
-
|
| 168 |
-
Per-position accuracy (positions 1-7), epoch 2:
|
| 169 |
-
`0.809 / 0.693 / 0.619 / 0.549 / 0.515 / 0.478 / 0.445`
|
| 170 |
-
|
| 171 |
-
Still improving epoch-over-epoch (epoch 3 in training).
|
| 172 |
-
|
| 173 |
-
### Acceptance length in vLLM (revision `epoch-2`)
|
| 174 |
-
|
| 175 |
-
Measured end-to-end in vLLM speculative decoding (nightly
|
| 176 |
-
`0.23.1rc1.dev709+g2b753ad20`), serving `zai-org/GLM-5.2-FP8` on 4xB300 with
|
| 177 |
-
`num_speculative_tokens=7`, `draft_sample_method="probabilistic"`, greedy
|
| 178 |
-
sampling, batch size 1, 64 single-turn chat prompts (32 `HumanEval` + 32
|
| 179 |
-
`math_reasoning` from
|
| 180 |
-
[`RedHatAI/speculator_benchmarks`](https://huggingface.co/datasets/RedHatAI/speculator_benchmarks)),
|
| 181 |
-
1024 output tokens each. Acceptance is computed from vLLM's
|
| 182 |
-
`spec_decode_num_accepted_tokens_per_pos` / `num_drafts` counter deltas.
|
| 183 |
-
The earlier
|
| 184 |
-
[`GLM-5.2-speculator.dspark-preview`](https://huggingface.co/RedHatAI/GLM-5.2-speculator.dspark-preview)
|
| 185 |
-
is included for reference (measured on a 16-prompt subset of the same set).
|
| 186 |
-
|
| 187 |
-
> This table is updated as later epochs are published; `main` currently
|
| 188 |
-
> points to **`epoch-2`**.
|
| 189 |
-
|
| 190 |
-
| Checkpoint | Pos 1 | Pos 2 | Pos 3 | Pos 4 | Pos 5 | Pos 6 | Pos 7 | Accept Len | Decode tok/s |
|
| 191 |
-
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|
| 192 |
-
| `epoch-2` (= `main`) | 74.7% | 56.0% | 41.5% | 31.1% | 23.7% | 18.0% | 13.2% | **3.58** | **225** |
|
| 193 |
-
| `epoch-1` | 74.7% | 55.4% | 40.2% | 29.5% | 21.5% | 15.4% | 10.9% | 3.48 | 219 |
|
| 194 |
-
| `dspark-preview` | 57.5% | 31.7% | 15.4% | 7.4% | 3.3% | 1.5% | 0.7% | 2.18 | 139 |
|
| 195 |
-
|
| 196 |
-
The epoch-2 gain comes from deeper draft positions (position-1 acceptance is
|
| 197 |
-
unchanged), consistent with continued training. For reference, the same
|
| 198 |
-
server without speculative decoding decodes at 102 tok/s (2.2x speedup for
|
| 199 |
-
`epoch-2`).
|
| 200 |
-
|
| 201 |
## References
|
| 202 |
|
| 203 |
- **DFlash**: Block Diffusion for Flash Speculative Decoding (arXiv:2602.06036) — the
|
|
|
|
| 21 |
prediction). Trained with the [speculators](https://github.com/vllm-project/speculators)
|
| 22 |
library.
|
| 23 |
|
| 24 |
+
`main` is the final epoch-3 checkpoint (best validation).
|
| 25 |
+
|
| 26 |
## Model Specifications
|
| 27 |
|
| 28 |
- **Base Model**: zai-org/GLM-5.2-FP8
|
|
|
|
| 34 |
|
| 35 |
## Checkpoint series
|
| 36 |
|
| 37 |
+
Per-epoch checkpoints of a single 3-epoch run. `main` = the final (epoch-3) checkpoint;
|
| 38 |
+
each epoch is also a permanent revision.
|
| 39 |
|
| 40 |
| revision | epoch | status |
|
| 41 |
| --- | --- | --- |
|
| 42 |
| `epoch-1` | 1 / 3 | ✅ available |
|
| 43 |
+
| `epoch-2` | 2 / 3 | ✅ available |
|
| 44 |
+
| `epoch-3` | 3 / 3 | ✅ final (= `main`) |
|
| 45 |
|
| 46 |
```python
|
| 47 |
from transformers import AutoModel
|
| 48 |
model = AutoModel.from_pretrained(
|
| 49 |
+
"mgoin/GLM-5.2-speculator.dspark", trust_remote_code=True # or revision="epoch-3"
|
| 50 |
)
|
| 51 |
```
|
| 52 |
|
| 53 |
+
## Evaluation Results
|
| 54 |
+
|
| 55 |
+
Validation metrics after epoch 3 (held-out split):
|
| 56 |
+
|
| 57 |
+
| metric | value |
|
| 58 |
+
| --- | --- |
|
| 59 |
+
| **mean accepted length** | **3.967** |
|
| 60 |
+
| full accuracy | 0.613 |
|
| 61 |
+
| mean acceptance rate | 0.584 |
|
| 62 |
+
| confidence abs error | 0.044 |
|
| 63 |
+
|
| 64 |
+
Per-position acceptance (positions 1-7):
|
| 65 |
+
`0.829 / 0.723 / 0.646 / 0.587 / 0.539 / 0.500 / 0.464`
|
| 66 |
+
|
| 67 |
+
Epoch-over-epoch mean accepted length (train-set proxy for epochs 1-2, val for epoch 3):
|
| 68 |
+
3.376 → 3.819 → **3.967 (val)**.
|
| 69 |
+
|
| 70 |
## Training Details
|
| 71 |
|
| 72 |
The model was trained using the Speculators library on prompts from
|
|
|
|
| 152 |
- DSpark-specific flags: `--markov-rank`, `--enable-confidence-head`,
|
| 153 |
`--confidence-head-with-markov`, `--confidence-head-alpha`. Dropping them (and
|
| 154 |
using `--speculator-type dflash`) recovers a plain DFlash draft.
|
|
|
|
| 155 |
|
| 156 |
## Deployment
|
| 157 |
|
| 158 |
+
DSpark inference support in vLLM is landing; once available, deploy with speculative
|
| 159 |
+
decoding:
|
| 160 |
|
| 161 |
```bash
|
| 162 |
vllm serve zai-org/GLM-5.2-FP8 \
|
|
|
|
| 164 |
--max-model-len 16384 \
|
| 165 |
--trust-remote-code \
|
| 166 |
--speculative-config '{
|
| 167 |
+
"model": "mgoin/GLM-5.2-speculator.dspark",
|
| 168 |
"num_speculative_tokens": 7,
|
| 169 |
+
"method": "dspark"
|
|
|
|
| 170 |
}'
|
| 171 |
```
|
| 172 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 173 |
## References
|
| 174 |
|
| 175 |
- **DFlash**: Block Diffusion for Flash Speculative Decoding (arXiv:2602.06036) — the
|