RedHatAI/GLM-5.2-speculator.dspark

GLM-5.2 DSpark speculator

Overview

A DSpark speculator model for the zai-org/GLM-5.2-FP8 base model, enabling faster inference through speculative decoding. DSpark extends the DFlash parallel draft backbone with two lightweight heads: a Markov logit-bias head (low-rank intra-block token dependency) and a per-position confidence head (accept-rate prediction). Trained with the speculators library.

main is the final epoch-3 checkpoint (best validation).

Model Specifications

• Base Model: zai-org/GLM-5.2-FP8
• Chat Template: GLM-5.2 (compatible with /chat/completions)
• Format: Safetensors
• License: MIT
• Draft: 5 layers, block_size=8, full vocabulary (154,880), aux layers [8, 23, 39, 55, 70]
• Validation Hardware: NVIDIA B300

Checkpoint series

Per-epoch checkpoints of a single 3-epoch run. main = the final (epoch-3) checkpoint; each epoch is also a permanent revision.

revision	epoch	status
epoch-1	1 / 3	✅ available
epoch-2	2 / 3	✅ available
epoch-3	3 / 3	✅ final (= main)

from transformers import AutoModel
model = AutoModel.from_pretrained(
    "mgoin/GLM-5.2-speculator.dspark", trust_remote_code=True  # or revision="epoch-3"
)

Evaluation Results

Validation metrics after epoch 3 (held-out split):

metric	value
mean accepted length	3.967
full accuracy	0.613
mean acceptance rate	0.584
confidence abs error	0.044

Per-position acceptance (positions 1-7): 0.829 / 0.723 / 0.646 / 0.587 / 0.539 / 0.500 / 0.464

Epoch-over-epoch mean accepted length (train-set proxy for epochs 1-2, val for epoch 3): 3.376 → 3.819 → 3.967 (val).

Training Details

The model was trained using the Speculators library on prompts from Magpie-Align/Magpie-Llama-3.1-Pro-300K-Filtered and HuggingFaceH4/ultrachat_200k, with responses regenerated by GLM-5.2-FP8 itself (published as mgoin/GLM-5.2-FP8-magpie-ultrachat).

Training is online: the draft consumes hidden states streamed on-the-fly from a live GLM-5.2-FP8 vLLM server, with the trainer running FSDP data-parallel on separate GPUs. The three commands below (data prep → server → trainer) reproduce the run. Install speculators and vLLM from main. GPU indices/parallelism are examples — adjust to your hardware.

Data Preparation

python scripts/prepare_data.py \
  --model zai-org/GLM-5.2-FP8 \
  --trust-remote-code \
  --data ./regenerated_data.jsonl \
  --output ./output \
  --seq-length 8192 \
  --assistant-pattern '<\|assistant\|>((?:(?!<\|user\|>|<\|assistant\|>).)*)'

--assistant-pattern is currently needed for GLM-5.2's inline-reasoning chat format (the <think>...</think> trace is kept inside the assistant turn); it may be auto-detected by future speculators versions.

vLLM Server Launch (hidden-states server)

CUDA_VISIBLE_DEVICES=0,1,2,3 python scripts/launch_vllm.py \
  zai-org/GLM-5.2-FP8 \
  --target-layer-ids 8 23 39 55 70 \
  -- --port 8000 \
  --tensor-parallel-size 4 \
  --gpu-memory-utilization 0.9 \
  --max-model-len 8192 \
  --trust-remote-code

Training Command

CUDA_VISIBLE_DEVICES=4,5,6,7 torchrun \
  --standalone \
  --nproc_per_node 4 \
  scripts/train.py \
  --verifier-name-or-path zai-org/GLM-5.2-FP8 \
  --speculator-type dspark \
  --num-layers 5 \
  --block-size 8 \
  --data-path ./output \
  --vllm-endpoint http://localhost:8000/v1 \
  --save-path ./output/checkpoints \
  --epochs 3 \
  --lr 0.0006 \
  --scheduler-type cosine \
  --total-seq-len 4096 \
  --draft-arch qwen3 \
  --draft-hidden-act silu \
  --target-layer-ids 8 23 39 55 70 \
  --max-anchors 1024 \
  --markov-rank 256 \
  --enable-confidence-head \
  --confidence-head-with-markov \
  --loss-fn '{"ce": 0.1, "tv": 0.9}' \
  --confidence-head-alpha 1.0 \
  --checkpoint-freq 0.2 \
  --on-missing generate \
  --on-generate delete \
  --seed 42 \
  --log-freq 100 \
  --prefetch-factor 2 \
  --num-workers 8 \
  --trust-remote-code

Notes:

• Omitting --draft-vocab-size trains on the full vocabulary; pass --draft-vocab-size 32000 for a reduced draft vocab.
• DSpark-specific flags: --markov-rank, --enable-confidence-head, --confidence-head-with-markov, --confidence-head-alpha. Dropping them (and using --speculator-type dflash) recovers a plain DFlash draft.

Deployment

DSpark inference support in vLLM is landing; once available, deploy with speculative decoding:

vllm serve zai-org/GLM-5.2-FP8 \
    --tensor-parallel-size 4 \
    --max-model-len 16384 \
    --trust-remote-code \
    --speculative-config '{
        "model": "mgoin/GLM-5.2-speculator.dspark",
        "num_speculative_tokens": 7,
        "method": "dspark"
    }'

References

• DFlash: Block Diffusion for Flash Speculative Decoding (arXiv:2602.06036) — the parallel draft backbone DSpark builds on.
• DSpark (DeepSeek) — the Markov + confidence-head additions replicated here.
• speculators — training library.

AI assistance was used to build the training pipeline and run these experiments.