b1rd/GLM-5.2-speculator.dspark-quanttrio-int4-ft

GLM-5.2 DSpark speculator — finetuned for QuantTrio Int4-Int8Mix

A finetune of RedHatAI/GLM-5.2-speculator.dspark re-aligned to the hidden states of the int4/int8 target QuantTrio/GLM-5.2-Int4-Int8Mix.

The public speculator was trained against FP8-GLM hidden states; served against the QuantTrio quant it accepts 1.45 extra tokens/step. This finetune reaches **2.1**, which on a 4× DGX Spark (GB10) cluster is the difference between 19.6 and **24.7 tok/s** decode at up to 1,048,576 context — lossless (exact rejection sampling), acceptance flat in depth.

Full serving recipe: bird/GLM-spark → recipes/4x-dspark-1m (vLLM branch bird/vllm-lil@dspark-ring-1m-20260711 with ring-buffer windowed drafting — required; stock vLLM's full-context DSpark port suffers acceptance depth-collapse).

Files

path	what
model.safetensors + config.json	full-vocab finetune — exact drop-in for the RedHatAI original (same speculators layout)
ft32k/	reduced draft vocab (top-32,768 ids, 99.94% coverage, d2t mapping) — what our production runs; same speed, ~400 MB/rank less memory

Usage (vLLM, with the recipe branch)

--speculative-config '{"model":"/path/to/this-repo","num_speculative_tokens":3,
                       "attention_backend":"TRITON_ATTN","quantization":"fp8"}'
# plus: VLLM_DSPARK_DRAFT_RING=1  VLLM_DSPARK_DRAFT_WINDOW=1024

num_speculative_tokens must keep 1+k a power of two on the b12x path (k∈{3,7}); k=3 is optimal.

Training

Teacher-forced on ~3M tokens of self-generated capture from the target quant itself (raw per-step target aux hidden states + token ids recorded while serving natural technical text at depths up to ~880k tokens), using the pipeline published in the recipe (speculator-training/): stitched full-window streams, bonus-anchor fill-in readout, window-excludes-anchor, length-weighted depth sampling, best-rolling-checkpoint selection. 2,000 steps, 3× GB10 DDP, ~75 minutes. Numerically verified against the inference kernels (bit-exact hidden-norm, bf16-noise rope at 1M positions).

Measured (2026-07-11, 4× GB10, TP4+DCP4, 1M pool)

• Decode: 24.4/24.4/24.6/24.3/25.2/24.8 and 25.6/24.1/24.5 tok/s across two boots (greedy 2k-token completions) — mean 24.66
• Acceptance: 2.03–2.23 extra tokens/step, flat 16k→486k at fixed content
• vs base speculator on the same stack: accept ~1.45, ~19.6 tok/s

Quality is unaffected by construction: DSpark verification is exact rejection sampling against the target.