mgoin/GLM-5.2-speculator.dspark-block16

GLM-5.2 DSpark (block16) Speculator Model Card

Overview

This is a DSpark speculator model for the zai-org/GLM-5.2-FP8 base model, enabling faster inference through speculative decoding. The architecture extends DFlash with dual lightweight components: a Markov logit-bias head addressing token dependencies and a per-position confidence head for acceptance-rate forecasting. Development utilized the speculators library.

This is the block_size=16 variant — identical to RedHatAI/GLM-5.2-speculator.dspark except for the draft block size (16 instead of 8). See Difference from the reference.

Model Specifications

• Base Model: zai-org/GLM-5.2-FP8
• Chat Template: GLM-5.2 (compatible with /chat/completions)
• Format: Safetensors
• License: MIT
• Draft: 5 layers, block_size=16, full attention, full vocabulary (154,880), aux layers [8, 23, 39, 55, 70]
• Training/Validation Hardware: 60× NVIDIA GB300 on a GB300 NVL72 rack (9 vLLM producer nodes + 6 FSDP trainer nodes, DP=24)

Difference from the dspark reference

This variant changes one knob versus RedHatAI/GLM-5.2-speculator.dspark:

• block_size = 16 (versus 8).

Everything else matches: full attention, full vocabulary, aux layers [8, 23, 39, 55, 70], lr 6e-4 cosine, 3 epochs, --max-anchors 1024, loss {"ce": 0.1, "tv": 0.9}, Markov + confidence heads.

Checkpoint series

This repository distributes per-epoch checkpoints from a single 3-epoch training run. The main branch tracks the most recent epoch; each epoch remains permanently accessible as a separate revision.

revision	epoch	status
epoch-1	1 / 3	✅ this checkpoint
epoch-2	2 / 3	training
epoch-3	3 / 3	training

Training Details

Training employed the Speculators library on prompts from Magpie-Align and HuggingFaceH4 collections, with GLM-5.2-FP8 generating responses (available as mgoin/GLM-5.2-FP8-magpie-ultrachat).

The approach is "online": the draft receives hidden states from a live GLM-5.2-FP8 vLLM server, while the trainer runs FSDP data-parallel on separate hardware. Three commands reproduce this process — install speculators and vLLM from main branches first.

Data Preparation

python scripts/prepare_data.py \
  --model zai-org/GLM-5.2-FP8 \
  --trust-remote-code \
  --data ./regenerated_data.jsonl \
  --output ./output \
  --seq-length 8192 \
  --assistant-pattern '<\|assistant\|>((?:(?!<\|user\|>|<\|assistant\|>).)*)'

The --assistant-pattern flag addresses GLM-5.2's inline-reasoning format where reasoning traces remain within assistant turns; future versions may auto-detect this.

vLLM Server Launch (hidden-states server)

CUDA_VISIBLE_DEVICES=0,1,2,3 python scripts/launch_vllm.py \
  zai-org/GLM-5.2-FP8 \
  --target-layer-ids 8 23 39 55 70 \
  -- --port 8000 \
  --tensor-parallel-size 4 \
  --gpu-memory-utilization 0.9 \
  --max-model-len 8192 \
  --trust-remote-code

Training Command

CUDA_VISIBLE_DEVICES=4,5,6,7 torchrun \
  --standalone \
  --nproc_per_node 4 \
  scripts/train.py \
  --verifier-name-or-path zai-org/GLM-5.2-FP8 \
  --speculator-type dspark \
  --num-layers 5 \
  --block-size 16 \
  --data-path ./output \
  --vllm-endpoint http://localhost:8000/v1 \
  --save-path ./output/checkpoints \
  --epochs 3 \
  --lr 0.0006 \
  --scheduler-type cosine \
  --total-seq-len 4096 \
  --draft-arch qwen3 \
  --draft-hidden-act silu \
  --target-layer-ids 8 23 39 55 70 \
  --max-anchors 1024 \
  --markov-rank 256 \
  --enable-confidence-head \
  --confidence-head-with-markov \
  --loss-fn '{"ce": 0.1, "tv": 0.9}' \
  --confidence-head-alpha 1.0 \
  --checkpoint-freq 0.2 \
  --on-missing generate \
  --on-generate delete \
  --seed 42 \
  --log-freq 100 \
  --prefetch-factor 2 \
  --num-workers 8 \
  --trust-remote-code

Notes:

• The published checkpoint was trained at scale on 60× GB300 (9 vLLM producer nodes serving the FP8 verifier + a 6-node, DP=24 FSDP trainer) with hidden states streamed over a Mooncake RDMA store; the command above is the equivalent single-node recipe.
• The only difference from the reference dspark is --block-size 16 (versus 8).
• Excluding --draft-vocab-size trains on full vocabulary; pass --draft-vocab-size 32000 to reduce it.
• DSpark-specific parameters: --markov-rank, --enable-confidence-head, --confidence-head-with-markov, --confidence-head-alpha. Removing these (with --speculator-type dflash) produces standard DFlash.
• Sub-epoch checkpoints enable resumability with --checkpoint-freq 0.2.

Deployment

vLLM nightly provides DSpark inference support:

vllm serve zai-org/GLM-5.2-FP8 \
    --tensor-parallel-size 4 \
    --max-model-len 16384 \
    --trust-remote-code \
    --speculative-config '{
        "model": "mgoin/GLM-5.2-speculator.dspark-block16",
        "num_speculative_tokens": 7,
        "method": "dspark",
        "draft_sample_method": "probabilistic"
    }'

Evaluation Results

End-of-epoch-1 metrics (teacher-forced validation over the held-out split):

metric	value
mean accepted length	3.741
full accuracy	0.444
mean acceptance rate	0.399
confidence abs error	0.008

Per-position accuracy across positions 1-7: 0.796 / 0.675 / 0.589 / 0.528 / 0.482 / 0.446 / 0.417

Acceptance length in vLLM (revision epoch-1)

[ Eval pending — to be added. ]

End-to-end measurement in vLLM speculative decoding serving GLM-5.2-FP8 with this speculator, versus the reference dspark under identical conditions: num_speculative_tokens=7, greedy sampling, batch size 1, 16 single-turn prompts (8 HumanEval + 8 math reasoning), 1024 output tokens each.

Checkpoint	Pos 1	Pos 2	Pos 3	Pos 4	Pos 5	Pos 6	Pos 7	Accept Len	Decode tok/s
epoch-1 (this repo, block16)	—	—	—	—	—	—	—	—	—
dspark (reference, block8)	75.8%	55.2%	40.3%	29.3%	21.5%	15.5%	11.1%	3.49	219

References

• DFlash: Block Diffusion for Flash Speculative Decoding (arXiv:2602.06036) — the parallel backbone foundation.
• DSpark (DeepSeek) — the Markov and confidence-head extensions replicated here.
• speculators — the training framework.
• RedHatAI/GLM-5.2-speculator.dspark — the block_size=8 reference.

AI support contributed to pipeline development and experimental execution.