mgoin commited on
Commit
8ff0f28
·
verified ·
1 Parent(s): 7598747

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +28 -56
README.md CHANGED
@@ -21,6 +21,8 @@ intra-block token dependency) and a **per-position confidence head** (accept-rat
21
  prediction). Trained with the [speculators](https://github.com/vllm-project/speculators)
22
  library.
23
 
 
 
24
  ## Model Specifications
25
 
26
  - **Base Model**: zai-org/GLM-5.2-FP8
@@ -32,22 +34,39 @@ library.
32
 
33
  ## Checkpoint series
34
 
35
- This repo publishes **per-epoch checkpoints** of a single 3-epoch run. `main` tracks
36
- the latest available epoch; each epoch is also a permanent revision.
37
 
38
  | revision | epoch | status |
39
  | --- | --- | --- |
40
  | `epoch-1` | 1 / 3 | ✅ available |
41
- | `epoch-2` | 2 / 3 | ✅ this checkpoint (= `main`) |
42
- | `epoch-3` | 3 / 3 | training |
43
 
44
  ```python
45
  from transformers import AutoModel
46
  model = AutoModel.from_pretrained(
47
- "mgoin/GLM-5.2-speculator.dspark", revision="epoch-2", trust_remote_code=True
48
  )
49
  ```
50
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
51
  ## Training Details
52
 
53
  The model was trained using the Speculators library on prompts from
@@ -133,12 +152,11 @@ Notes:
133
  - DSpark-specific flags: `--markov-rank`, `--enable-confidence-head`,
134
  `--confidence-head-with-markov`, `--confidence-head-alpha`. Dropping them (and
135
  using `--speculator-type dflash`) recovers a plain DFlash draft.
136
- - `--checkpoint-freq 0.2` writes sub-epoch checkpoints (~5 per epoch) for resumability.
137
 
138
  ## Deployment
139
 
140
- DSpark inference support is available on the vLLM nightly
141
- (`uv pip install vllm --extra-index-url https://wheels.vllm.ai/nightly`):
142
 
143
  ```bash
144
  vllm serve zai-org/GLM-5.2-FP8 \
@@ -146,58 +164,12 @@ vllm serve zai-org/GLM-5.2-FP8 \
146
  --max-model-len 16384 \
147
  --trust-remote-code \
148
  --speculative-config '{
149
- "model": "RedHatAI/GLM-5.2-speculator.dspark",
150
  "num_speculative_tokens": 7,
151
- "method": "dspark",
152
- "draft_sample_method": "probabilistic"
153
  }'
154
  ```
155
 
156
- ## Evaluation Results
157
-
158
- Train-set metrics at the end of epoch 2 (per-epoch validation passes did not
159
- complete due to server restarts; per-dataset acceptance evaluation will accompany
160
- the final checkpoint):
161
-
162
- | metric | epoch 1 | epoch 2 |
163
- | --- | --- | --- |
164
- | mean accepted length | 3.376 | **3.819** |
165
- | full accuracy | 0.532 | 0.587 |
166
- | mean acceptance rate | 0.497 | 0.557 |
167
-
168
- Per-position accuracy (positions 1-7), epoch 2:
169
- `0.809 / 0.693 / 0.619 / 0.549 / 0.515 / 0.478 / 0.445`
170
-
171
- Still improving epoch-over-epoch (epoch 3 in training).
172
-
173
- ### Acceptance length in vLLM (revision `epoch-2`)
174
-
175
- Measured end-to-end in vLLM speculative decoding (nightly
176
- `0.23.1rc1.dev709+g2b753ad20`), serving `zai-org/GLM-5.2-FP8` on 4xB300 with
177
- `num_speculative_tokens=7`, `draft_sample_method="probabilistic"`, greedy
178
- sampling, batch size 1, 64 single-turn chat prompts (32 `HumanEval` + 32
179
- `math_reasoning` from
180
- [`RedHatAI/speculator_benchmarks`](https://huggingface.co/datasets/RedHatAI/speculator_benchmarks)),
181
- 1024 output tokens each. Acceptance is computed from vLLM's
182
- `spec_decode_num_accepted_tokens_per_pos` / `num_drafts` counter deltas.
183
- The earlier
184
- [`GLM-5.2-speculator.dspark-preview`](https://huggingface.co/RedHatAI/GLM-5.2-speculator.dspark-preview)
185
- is included for reference (measured on a 16-prompt subset of the same set).
186
-
187
- > This table is updated as later epochs are published; `main` currently
188
- > points to **`epoch-2`**.
189
-
190
- | Checkpoint | Pos 1 | Pos 2 | Pos 3 | Pos 4 | Pos 5 | Pos 6 | Pos 7 | Accept Len | Decode tok/s |
191
- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
192
- | `epoch-2` (= `main`) | 74.7% | 56.0% | 41.5% | 31.1% | 23.7% | 18.0% | 13.2% | **3.58** | **225** |
193
- | `epoch-1` | 74.7% | 55.4% | 40.2% | 29.5% | 21.5% | 15.4% | 10.9% | 3.48 | 219 |
194
- | `dspark-preview` | 57.5% | 31.7% | 15.4% | 7.4% | 3.3% | 1.5% | 0.7% | 2.18 | 139 |
195
-
196
- The epoch-2 gain comes from deeper draft positions (position-1 acceptance is
197
- unchanged), consistent with continued training. For reference, the same
198
- server without speculative decoding decodes at 102 tok/s (2.2x speedup for
199
- `epoch-2`).
200
-
201
  ## References
202
 
203
  - **DFlash**: Block Diffusion for Flash Speculative Decoding (arXiv:2602.06036) — the
 
21
  prediction). Trained with the [speculators](https://github.com/vllm-project/speculators)
22
  library.
23
 
24
+ `main` is the final epoch-3 checkpoint (best validation).
25
+
26
  ## Model Specifications
27
 
28
  - **Base Model**: zai-org/GLM-5.2-FP8
 
34
 
35
  ## Checkpoint series
36
 
37
+ Per-epoch checkpoints of a single 3-epoch run. `main` = the final (epoch-3) checkpoint;
38
+ each epoch is also a permanent revision.
39
 
40
  | revision | epoch | status |
41
  | --- | --- | --- |
42
  | `epoch-1` | 1 / 3 | ✅ available |
43
+ | `epoch-2` | 2 / 3 | ✅ available |
44
+ | `epoch-3` | 3 / 3 | ✅ final (= `main`) |
45
 
46
  ```python
47
  from transformers import AutoModel
48
  model = AutoModel.from_pretrained(
49
+ "mgoin/GLM-5.2-speculator.dspark", trust_remote_code=True # or revision="epoch-3"
50
  )
51
  ```
52
 
53
+ ## Evaluation Results
54
+
55
+ Validation metrics after epoch 3 (held-out split):
56
+
57
+ | metric | value |
58
+ | --- | --- |
59
+ | **mean accepted length** | **3.967** |
60
+ | full accuracy | 0.613 |
61
+ | mean acceptance rate | 0.584 |
62
+ | confidence abs error | 0.044 |
63
+
64
+ Per-position acceptance (positions 1-7):
65
+ `0.829 / 0.723 / 0.646 / 0.587 / 0.539 / 0.500 / 0.464`
66
+
67
+ Epoch-over-epoch mean accepted length (train-set proxy for epochs 1-2, val for epoch 3):
68
+ 3.376 → 3.819 → **3.967 (val)**.
69
+
70
  ## Training Details
71
 
72
  The model was trained using the Speculators library on prompts from
 
152
  - DSpark-specific flags: `--markov-rank`, `--enable-confidence-head`,
153
  `--confidence-head-with-markov`, `--confidence-head-alpha`. Dropping them (and
154
  using `--speculator-type dflash`) recovers a plain DFlash draft.
 
155
 
156
  ## Deployment
157
 
158
+ DSpark inference support in vLLM is landing; once available, deploy with speculative
159
+ decoding:
160
 
161
  ```bash
162
  vllm serve zai-org/GLM-5.2-FP8 \
 
164
  --max-model-len 16384 \
165
  --trust-remote-code \
166
  --speculative-config '{
167
+ "model": "mgoin/GLM-5.2-speculator.dspark",
168
  "num_speculative_tokens": 7,
169
+ "method": "dspark"
 
170
  }'
171
  ```
172
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
173
  ## References
174
 
175
  - **DFlash**: Block Diffusion for Flash Speculative Decoding (arXiv:2602.06036) — the