---
license: other
library_name: torchspec
tags:
  - speculative-decoding
  - draft-model
  - dspark
  - glm-5.2
  - code
---

# dspark-glm52-draft

DSpark (DFlash-family single-shot parallel drafting) **draft models for GLM-5.2
speculative decoding**, optimized for code / coding-chat traffic. This repo consolidates
the lineage; each historical checkpoint is an immutable **git tag** (pick it from the
revision dropdown or pass `revision=` to `from_pretrained`/`hf_hub_download`).

`main` tracks the current best checkpoint.

## Revisions

| tag | arch | checkpoint | trend `official_accept_len` | note |
|---|---|---|---|---|
| **`bs8-moe8-lr7xct-20260722-iter7651`** (= `main`) | block-8 MoE-8 + DFlare | lr7x-ct iter_7651 | **4.704 / 8** | current best; high-LR warm-restart continuation |
| `bs8-moe8-dflare-20260715-iter10001` | block-8 MoE-8 + DFlare | dflare iter_10001 | 4.59 / 8 | predecessor run's pre-bump champion |
| `bs8-moe8-dflare-20260715-iter5251` | block-8 MoE-8 + DFlare | dflare iter_5251 | ~4.49 | earlier dflare snapshot |
| `bs8-moe8-dflare-20260714` | block-8 MoE-8 + DFlare | dflare (earlier) | — | earliest block-8 export |
| `bs16-moe-20260705` | **block-16** MoE | 2026-07-05 | — | distinct architecture (γ=16), research arm — NOT a version of the block-8 line |

**All trend numbers are teacher-forced metrics on a fixed 96-row set that overlaps the
training view (no holdout) — NOT serving measurements and NOT SOTA claims.** Keep/kill
belongs to the offline 3-task screen (gsm8k + humaneval + mt-bench) and an sglang serving
bench. Trend-eval noise is ±0.03–0.04.

## Shared architecture (block-8 line)

- DSpark block size 8, 5 draft layers, hidden 6144, 64 heads (head_dim 64).
- Vanilla MoE-8 (8 routed top-2 + 1 shared per layer, moe_intermediate 3072); Jakiro OFF.
- DFlare per-layer context fusion, Markov/GRIFFIN correction head (rank 256) + confidence
  head (`confidence_detach_input`, `griffin_fuse`).
- Reads GLM-5.2 target hidden states from layers [8, 23, 39, 55, 70] of 78.
- 4.44 B total params (~2.7 B active/token), bf16, 8.3 GB.
- `draft_vocab_size` 32000 vs target 154880 → serve via sglang's DSpark loader with the
  **d2t full-vocab scatter** (`d2t` int64 [32000], `t2d` bool [154880] ship in the
  safetensors; do NOT bf16-cast them). Tokenizer: use GLM-5.2's (not bundled).

## Target & data (block-8 line)

- Target (generation AND extraction): **lukealonso/GLM-5.2-NVFP4** (modelopt_fp4, fp8-e4m3
  KV), live target prefill — fully on-policy, no disk target-state cache.
- SFT view `sha256:c17516837c…` (1,379,922 production DB conversations + 522,176 on-policy
  OPB rows), seq 4096, GLM chat template.

See each tag's own README for its exact step, LR schedule, and provenance hashes.