README.md

---
base_model: zai-org/GLM-5.2-FP8
library_name: speculators
pipeline_tag: text-generation
license: mit
datasets:
- mgoin/GLM-5.2-FP8-magpie-ultrachat
tags:
- speculative-decoding
- dspark
- speculators
---
# GLM-5.2 DSpark speculator

## Overview

A DSpark speculator model for the `zai-org/GLM-5.2-FP8` base model, enabling faster
inference through speculative decoding. DSpark extends the DFlash parallel draft
backbone with two lightweight heads: a **Markov logit-bias head** (low-rank
intra-block token dependency) and a **per-position confidence head** (accept-rate
prediction). Trained with the [speculators](https://github.com/vllm-project/speculators)
library.

`main` is the final epoch-3 checkpoint (best validation).

## Model Specifications

- **Base Model**: zai-org/GLM-5.2-FP8
- **Chat Template**: GLM-5.2 (compatible with `/chat/completions`)
- **Format**: Safetensors
- **License**: MIT
- **Draft**: 5 layers, `block_size=8`, full vocabulary (154,880), aux layers `[8, 23, 39, 55, 70]`
- **Validation Hardware**: NVIDIA B300

## Checkpoint series

Per-epoch checkpoints of a single 3-epoch run. `main` = the final (epoch-3) checkpoint;
each epoch is also a permanent revision.

| revision | epoch | status |
| --- | --- | --- |
| `epoch-1` | 1 / 3 | ✅ available |
| `epoch-2` | 2 / 3 | ✅ available |
| `epoch-3` | 3 / 3 | ✅ final (= `main`) |

```python
from transformers import AutoModel
model = AutoModel.from_pretrained(
    "RedHatAI/GLM-5.2-speculator.dspark", trust_remote_code=True  # or revision="epoch-3"
)
```

## Evaluation Results

Validation metrics after epoch 3 (held-out split):

| metric | value |
| --- | --- |
| **mean accepted length** | **3.967** |
| full accuracy | 0.613 |
| mean acceptance rate | 0.584 |
| confidence abs error | 0.044 |

Per-position acceptance (positions 1-7):
`0.829 / 0.723 / 0.646 / 0.587 / 0.539 / 0.500 / 0.464`

Epoch-over-epoch mean accepted length (train-set proxy for epochs 1-2, val for epoch 3):
3.376 → 3.819 → **3.967 (val)**.

## Training Details

The model was trained using the Speculators library on prompts from
`Magpie-Align/Magpie-Llama-3.1-Pro-300K-Filtered` and `HuggingFaceH4/ultrachat_200k`,
with responses regenerated by GLM-5.2-FP8 itself (published as
[`mgoin/GLM-5.2-FP8-magpie-ultrachat`](https://huggingface.co/datasets/mgoin/GLM-5.2-FP8-magpie-ultrachat)).

Training is **online**: the draft consumes hidden states streamed on-the-fly from a
live GLM-5.2-FP8 vLLM server, with the trainer running FSDP data-parallel on separate
GPUs. The three commands below (data prep → server → trainer) reproduce the run.
Install [speculators](https://github.com/vllm-project/speculators) and vLLM from main.
GPU indices/parallelism are examples — adjust to your hardware.

### Data Preparation

```bash
python scripts/prepare_data.py \
  --model zai-org/GLM-5.2-FP8 \
  --trust-remote-code \
  --data ./regenerated_data.jsonl \
  --output ./output \
  --seq-length 8192 \
  --assistant-pattern '<\|assistant\|>((?:(?!<\|user\|>|<\|assistant\|>).)*)'
```

> `--assistant-pattern` is currently needed for GLM-5.2's inline-reasoning chat
> format (the `<think>...</think>` trace is kept inside the assistant turn); it may be
> auto-detected by future speculators versions.

### vLLM Server Launch (hidden-states server)

```bash
CUDA_VISIBLE_DEVICES=0,1,2,3 python scripts/launch_vllm.py \
  zai-org/GLM-5.2-FP8 \
  --target-layer-ids 8 23 39 55 70 \
  -- --port 8000 \
  --tensor-parallel-size 4 \
  --gpu-memory-utilization 0.9 \
  --max-model-len 8192 \
  --trust-remote-code
```

### Training Command

```bash
CUDA_VISIBLE_DEVICES=4,5,6,7 torchrun \
  --standalone \
  --nproc_per_node 4 \
  scripts/train.py \
  --verifier-name-or-path zai-org/GLM-5.2-FP8 \
  --speculator-type dspark \
  --num-layers 5 \
  --block-size 8 \
  --data-path ./output \
  --vllm-endpoint http://localhost:8000/v1 \
  --save-path ./output/checkpoints \
  --epochs 3 \
  --lr 0.0006 \
  --scheduler-type cosine \
  --total-seq-len 4096 \
  --draft-arch qwen3 \
  --draft-hidden-act silu \
  --target-layer-ids 8 23 39 55 70 \
  --max-anchors 1024 \
  --markov-rank 256 \
  --enable-confidence-head \
  --confidence-head-with-markov \
  --loss-fn '{"ce": 0.1, "tv": 0.9}' \
  --confidence-head-alpha 1.0 \
  --checkpoint-freq 0.2 \
  --on-missing generate \
  --on-generate delete \
  --seed 42 \
  --log-freq 100 \
  --prefetch-factor 2 \
  --num-workers 8 \
  --trust-remote-code
```

Notes:
- Omitting `--draft-vocab-size` trains on the **full vocabulary**; pass
  `--draft-vocab-size 32000` for a reduced draft vocab.
- DSpark-specific flags: `--markov-rank`, `--enable-confidence-head`,
  `--confidence-head-with-markov`, `--confidence-head-alpha`. Dropping them (and
  using `--speculator-type dflash`) recovers a plain DFlash draft.

## Deployment

DSpark inference support in vLLM is landing; once available, deploy with speculative
decoding:

```bash
vllm serve zai-org/GLM-5.2-FP8 \
    --tensor-parallel-size 4 \
    --max-model-len 16384 \
    --trust-remote-code \
    --speculative-config '{
        "model": "RedHatAI/GLM-5.2-speculator.dspark",
        "num_speculative_tokens": 7,
        "method": "dspark"
    }'
```

## References

- **DFlash**: Block Diffusion for Flash Speculative Decoding (arXiv:2602.06036) — the
  parallel draft backbone DSpark builds on.
- **DSpark** (DeepSeek) — the Markov + confidence-head additions replicated here.
- [speculators](https://github.com/vllm-project/speculators) — training library.

AI assistance was used to build the training pipeline and run these experiments.