jmeadlock/GLM-5.3-Flash-DFlash2-JM-FAILED

❌ FAILED EXPERIMENT: GLM-5.3-Flash DFlash2 Drafter -JM (Apache-2.0)

This drafter is a failure. Do not use it for serving. It loads and works, but it accepts fewer draft tokens than the existing incoai/GLM-5.3-Flash-DFlash2 drafter on almost every test we ran (the two tie only on agent prompts, where both hit the K=4 ceiling, and on structured output). It is 16–19 % slower on prose, and it missed 3 of its 4 pre-registered pass bars (prose acceptance 1.97 against a bar of 2.6; incoai gets 2.34–2.44). It is published as a documented failure so others can learn from it.

A self-trained DFlash2 block-diffusion speculative-decoding drafter for GLM-5.3-Flash. It was trained against the NVFP4 quant LibertAIDAI/GLM-5.3-Flash-NVFP4@aa28e1f, the exact bytes we serve. Training started from random init (initial_draft_model_path: null). No third-party drafter weights or third-party model outputs were used anywhere in training: every completion was regenerated by the target.

Read this first: this experiment failed. The drafter is a working drop-in on the incoai/GLM-5.3-Flash-DFlash2 geometry. But it accepts fewer tokens per verify step than incoai on every fixture we measured, including our own in-distribution holdout. Prose at K=4 is 1.97 vs 2.34–2.44, and the canada-quant thinking-ON holdout at K=4 is 2.712 vs 3.134. We are publishing it for two reasons. It is the only Apache-2.0 drop-in for this geometry that we know of (incoai is CC-BY-NC-ND-4.0). And the measurement is useful to anyone who expects a drafter trained on their own serving distribution to beat a general one. At 150K samples and one pass, it did not.

If you can use the incoai drafter under its licence, use incoai. It is faster on everything we measured.

Architecture: the incoai drop-in geometry, unchanged

This is not a new architecture. config.json is byte-identical to incoai's (sha256 c4aeac01…) and the tensor set matches incoai's exactly: 81 tensors with the same names, shapes and dtypes, and 1,171,080,448 parameters. We copied the geometry on purpose so the drafter drops into any launcher already serving incoai 7d74cdd.

Field	Value
Type	DFlash2 block-diffusion drafter (DFlash2DraftModel, Qwen3-style backbone, model_type: qwen3)
Layers	5 decoder layers, sliding-window attention (window 2048)
Hidden / heads	4096 (intermediate 12288) · 32 attention heads / 8 KV heads · head_dim 128
Target taps	5 hidden-state taps, target_layer_ids [5, 14, 24, 33, 42] (vLLM resolves these to layers 6, 15, 25, 34, 43; verified at boot)
Block size	8 (trained). Served at K = 4; K = 7 also measured
Selector	rank 256, top_k 16, grouped dynamic conv (kernel 2, group 16)
Mask token	mask_token_id 154856. No mask_embedding.pt: incoai ships none, and the trainer emits none
Embeddings / LM head	Not shipped. They come from the target at serve time (tie_word_embeddings: false; no embed_tokens / lm_head tensors in the file)
Size	1.171 B parameters · 2.34 GB bf16 model.safetensors
Vocab / max positions	154,880 / 1,048,576

Training

Item	Value
Samples	150,000 in one pass (no epochs, no soups), mixed thinking ON/OFF
Tokens	147,169,407 total, of which 92,053,163 supervised. Pretokenized from the target's own emitted token ids, not re-tokenized text
Mix (kept)	40 % chat (60,000) · 30 % agent / tool-calling (45,000) · 20 % code / STEM (30,000) · 10 % structured output (15,000)
Regimes	thinking OFF, greedy T=0 (the regime we serve): 114,976 (76.65 %) · thinking ON, T=1.0 / top_p 0.95: 35,024 (23.35 %). The plan was 80/20; 24,992 thinking-off rows were dropped for hitting the 2,048-token cap, which pushed the kept share of thinking-ON up
Length cap	≤ 2,048 tokens prompt + completion. 16,768 thinking-ON rows hit the cap inside the think span and were kept as-is, with no answer
Steps	18,750 optimizer steps · global batch 8 sequences (~7.85K tokens/step)
LR	1e-4 → 1e-5 cosine over all 18,750 steps, warmup 2 %, grad-clip 1.0, bf16
Loss	per token, (1 − w)·CE + w·top-20 KL, with w = 0.5. Gamma-6.5 exponential tail weighting. KL teacher = target lm_head(norm(last hidden)). The canada-quant recipe; w = 0.5 is our choice, because they don't publish theirs
Final loss	mean of the last 200 steps: CE 2.791, KL 2.468, train acc 0.430 (per position 0.767 · 0.588 · 0.466 · 0.378 · 0.315 · 0.263 · 0.222). The curve was still falling at the end
Hardware	1× NVIDIA GB300 (one GPU), the same GPU used for extraction and training
Wall time	12 h 25 m 21 s end to end (2026-10-07 01:46:52 → 14:12:13 CDT). That is 7.26 h of hidden-state extraction (offline, 15 chunks × 10K rows, 7.235 TB materialized and deleted chunk by chunk) plus 5.13 h of training. Peak training HBM was 54,235 MiB
Trainer	lightseekorg/TorchSpec @076a738 (MIT) plus four local patches: the GLM-5-next aux taps, the KV-spec fix, the offline-saver fix, and a port of Tencent/AngelSpec @d3412be's top-K KL (_kl_variant_b). Target activations come from vLLM extract_hidden_states (vLLM 0.30.1rc1.dev493+gac9126e58)

PROVENANCE.txt in this repo is the verbatim training record: image digests, commits, commands, config and data manifest sha.

Data and licences

Prompts only. Every assistant turn was regenerated by LibertAIDAI/GLM-5.3-Flash-NVFP4@aa28e1f (SGLang, plain autoregressive, no speculation). No dataset-provided answers, reasoning traces or third-party model outputs were kept as targets.

Bucket	Source	Licence	Kept
agent	Team-ACE/ToolACE	Apache-2.0	10,940
agent	glaiveai/glaive-function-calling-v2	Apache-2.0	34,060
chat	allenai/WildChat-1M (English, moderation-filtered, first user turn)	ODC-BY-1.0	60,000
code_stem	nvidia/OpenCodeInstruct	CC-BY-4.0	17,413
code_stem	nvidia/OpenCodeReasoning (rows with NC or missing per-row licences dropped)	CC-BY-4.0	6,264
code_stem	open-thoughts/OpenThoughts3-1.2M (prompts only; QwQ traces dropped)	Apache-2.0	6,323
structured	synthetic templates (count, csv, enum, JSON list/object, markdown table, ranked, yes/no)	campaign-original, released Apache-2.0	15,000
	Total		150,000

• Agent rows use the stock Hermes Agent default system prompt and its 50 public tool schemas (MIT, v0.21.5). They contain no private session content. Multi-turn agent rows had a synthetic scaffold turn, which was stripped and replaced with the target's own tool call before the target continued.
• Planned sources that were not used: lmsys/lmsys-chat-1m and Salesforce/xlam-function-calling-60k returned HTTP 403 at download time. Chat is WildChat-only and agent is ToolACE + glaive.
• The prompt shards are not redistributed.

Evaluation: holdout, same lane, same morning

All rows except the weekend/AR columns were measured on one GB300 vLLM lane (TP1, vllm 0.30.1rc1.dev493+gac9126e58 + aux/KV patches, CUDA graphs on, NVFP4 target aa28e1f). They ran on 2026-10-15 between 03:03 and 07:55 CDT, with one boot per arm and drafter sha checked before every boot. 0 HTTP errors in any cell; 500/500 or 100/100 ok on every cq run.

Columns: JM = this drafter. INCOAI = incoai 7d74cdd (safetensors 8931dc52…). TONY_K7 (wknd) = incoai 7d74cdd on the 2× DGX Spark TP2 eager lane (Tony's image funland/glm53-dflash2:ccc5a78e), measured 2026-10-04/05. It is cross-day and cross-lane, so compare acceptance only and never tok/s. AR = plain autoregressive on that same weekend Spark lane. No AR baseline was booted on the GB300, so its only number is Spark tok/s.

Fixture (acceptance length = tokens per verify step)	JM K=4	JM K=7	INCOAI K=4	INCOAI K=7	TONY_K7 (wknd, Spark)	AR (wknd, Spark)
E1 prose C1, greedy, thinking OFF (median of 3 reps, pass 1 / all 9 reps)	1.97 / 1.97	1.99 / 2.00	2.44 / 2.34	2.42 / 2.40	2.31	1.00 (n/a)
E1 prose C1 output tok/s (GB300 only; pass 1 / all 9)	150.2 / 150.2	130.1 / 131.1	185.5 / 178.1	157.4 / 156.9	—	15.2 (Spark)
probe_code0 / probe_code1 (median of 3 passes)	2.72 / 2.46	2.67 / 2.52	3.28 / 3.03	3.88 / 3.22	3.63 / 3.35	—
structured C1 (count 1..300; ceiling = K+1)	4.55	5.45	5.00	7.89	—	—
agent C1 (13-token tool call; ceiling, not a discriminator)	4.33	6.50	4.33	6.50	—	—
canada-quant holdout, thinking ON T=1.0, c6 × 500	2.712	2.919	3.134	3.649	—	—
canada-quant holdout, thinking ON, c16 × 500 / c1 × 100	2.731 / 2.720	2.919 / 2.895	3.146 / 3.131	3.655 / 3.623	—	—
our P1 holdout, thinking OFF greedy, 300 prompts, overall	2.65	2.82	3.03	3.43	—	—
· chat (120) / agent (90) / code_stem (60) / structured (30)	2.39 / 2.60 / 3.30 / 4.30	2.44 / 2.72 / 3.85 / 5.43	2.77 / 2.88 / 3.75 / 4.17	3.02 / 3.20 / 4.71 / 5.48	—	—

Lane cross-check: on this GB300 lane, incoai at K=7 on the canada-quant kit reads 3.65, inside canada-quant's published 3.58–3.68 for incoai on B300. Its per-position prose curve tracks the weekend Spark rows within 0.03, so acceptance here is comparable to the Spark lane even though tok/s is not. A newer incoai revision (bf582e4e) is within noise of 7d74cdd (prose 2.34 vs 2.34, cq 3.133 vs 3.134).

Scheduler note: on this lane the sliding-window drafter KV group capped the engine at 6 running requests (K=4) or 3 (K=7), so "c16" ran as c6/c3. Acceptance does not depend on this: c1, c6 and c16 agree within 0.03 on every arm.

Per-position acceptance (probability that a verify step keeps draft position k)

Drafter, K	Fixture	pos0	pos1	pos2	pos3	pos4	pos5	pos6
JM, 4	E1 prose	0.61	0.25	0.07	0.03
INCOAI, 4	E1 prose	0.65	0.34	0.20	0.09
JM, 7	E1 prose	0.61	0.26	0.09	0.03	0.01	0.00	0.00
INCOAI, 7	E1 prose	0.65	0.37	0.21	0.08	0.05	0.02	0.02
TONY_K7 (wknd, Spark)	E1 prose	0.62	0.36	0.19	0.07	0.05	0.01	0.00
JM, 4	probe_code0	0.72	0.49	0.30	0.18
INCOAI, 4	probe_code0	0.82	0.63	0.46	0.37
JM, 7	probe_code0	0.68	0.45	0.23	0.14	0.07	0.03	0.01
INCOAI, 7	probe_code0	0.81	0.62	0.47	0.39	0.29	0.21	0.16
TONY_K7 (wknd, Spark)	probe_code0	0.78	0.57	0.40	0.31	0.24	0.19	0.13
JM, 4	cq thinking ON, c6×500	0.71	0.47	0.32	0.21
INCOAI, 4	cq thinking ON, c6×500	0.77	0.58	0.44	0.34
JM, 7	cq thinking ON, c6×500	0.70	0.46	0.30	0.19	0.13	0.09	0.06
INCOAI, 7	cq thinking ON, c6×500	0.75	0.56	0.42	0.32	0.25	0.20	0.16

Honest read

• It works, and it is weaker. The drafter loads as a drop-in, serves without errors, and its per-position counters decay sensibly. It never pins at 1.00. On every distribution we have it accepts less than incoai. Position 0 is close (0.61 vs 0.65 on prose). The gap opens at positions 1–3: prose pos2 is 0.07 vs 0.20. At K=7, positions 4–6 contribute almost nothing.
• Training on the serving regime bought nothing on prose. The premise was that a drafter trained on our thinking-off prose/agent mix could lift prose acceptance from ~2.3 to 2.6 or more. It went to 1.97. Even on our own in-distribution holdout, drawn from the same sources and regime as training, incoai wins chat (−0.38), agent (−0.28) and code (−0.45) at K=4. Structured output is the only place we are level (+0.13 at K=4, n=30, within noise).
• Thinking-ON delta vs incoai: −0.42 at K=4 (2.712 vs 3.134, −13.5 %) and −0.73 at K=7. The thinking-OFF gap on our own holdout is about the same size (−0.38 at K=4, −0.61 at K=7). So the extra thinking-ON penalty is only about 0.04 at K=4 and 0.12 at K=7. The drafter is uniformly weaker; this is not a regime-specific regression. Within our own drafter, thinking-ON T=1.0 prose (2.22) is not below thinking-OFF T=0 (2.00).
• Pass bars from the plan: C1 prose ≥ 2.6 at K=4 failed (1.97, below the 2.4 negative-result line). Agent ≥ 4.33 passed (4.33, a ceiling). probe_code within −0.2 of incoai at K=7 failed (−1.21 / −0.70). Thinking-ON holdout within −0.1 failed (−0.42).
• Likely causes, untested: (1) data volume: one pass over 150K samples, against canada-quant's 736K for their -G, and our loss curve was still falling at the end; (2) the KL weight of 0.5 and the loss port are our choices, not a reproduced recipe; (3) Inco's data and step count are unpublished and are probably much larger. This release measures the result; it does not diagnose it.
• Cost of the licence: on this lane, choosing this drafter over incoai costs −16 to −19 % C1 prose tok/s (150.2 vs 178.1–185.5) and about −0.4 acceptance on the cq kit.

Serving

Pairs with the NVFP4 target it was trained against, LibertAIDAI/GLM-5.3-Flash-NVFP4@aa28e1f. It should also work with other GLM-5.3-Flash checkpoints that keep the same hidden states, but we have not measured that. vLLM speculative config, at K=4 (the measured serving default):

{"method": "dflash", "model": "/models/GLM-5.3-Flash-DFlash2-JM", "num_speculative_tokens": 4}

Generic vLLM (x86 / Grace-Blackwell). This is what we measured: GB300 TP1, vLLM 0.30.1rc1.dev493+gac9126e58. That build needs two patches: the GLM-5-next aux-tap patch, and the kv_cache_utils patch that lets the GLM-5 KV fast path accept the drafter's sliding-window layers. Without the second, boot fails with page size is not divisible by the maximum page size.

vllm serve /models/GLM-5.3-Flash-NVFP4 --served-model-name glm-5.3-flash --trust-remote-code \
  --max-model-len 16384 --max-num-seqs 16 --gpu-memory-utilization 0.85 \
  --speculative-config '{"method":"dflash","model":"/models/GLM-5.3-Flash-DFlash2-JM","num_speculative_tokens":4}' \
  --tool-call-parser glm47 --enable-auto-tool-choice --reasoning-parser glm45 \
  --default-chat-template-kwargs '{"enable_thinking":false}'

2× DGX Spark (SM121), Tony's lane. Image funland/glm53-dflash2:ccc5a78e (digest sha256:5be2a57f…), TP2 eager, K=4. The drafter is bind-mounted at runtime. Only the drafter mount changes from the incoai setup:

-v /home/milo/GLM-5.3-Flash-DFlash2-JM:/models/dflash2-draft:ro \
...
--speculative-config '{"method":"dflash","model":"/models/dflash2-draft","num_speculative_tokens":4}' \
--enforce-eager --tensor-parallel-size 2 --max-num-seqs 6 --block-size 2304 --kv-cache-dtype fp8_e4m3

Unmeasured on Spark. The weights are sha-verified on both Spark nodes, but this drafter has not been booted on the Spark image: the TP2 interconnect was down during evaluation. The geometry is identical to incoai, which that image serves, so it is expected to load. Boot it and re-measure before quoting any Spark number.

Constraints:

• The serving stack must honor dflash_config.target_layer_ids of length 5 and the drafter's sliding-window KV spec.
• Verify in the boot log that the DFlash2 head loads with aux layers (6, 15, 25, 34, 43).
• There is no mask embedding to check. Its absence is correct for this geometry.
• K=4 is the measured serving value. K=7 boots and is measured above, but it is slower on prose for this drafter.

Provenance

• model.safetensors: sha256 bfdf92d2d92527ca5e6a8202457d0ae6c1b7948e4102e201fb45f6b52b3adf31, exported from TorchSpec checkpoint iter_0018751
• config.json: sha256 c4aeac0101196a6e26705b34c45230bcd0c7c68ee2d2d1efdb242087f3712573 (byte-identical to incoai 7d74cdd's config.json; geometry asserted at export)
• config.torchspec.json: sha256 87b9b0fada5abc74da18e6a561fc84ea67410085ecc19edd9b3778d1d65b38f4 (the exporter's own config; differs only in metadata keys)
• Target: LibertAIDAI/GLM-5.3-Flash-NVFP4@aa28e1f54130286c95fee10d0705c74ce8743734
• Training data manifest (manifest.json, 30 shards × 5,000): sha256 0510302d25ef1f125feeee853edab5f44e2afcff06fe20ea744752ab0b568e4c; pretokenized corpus train.parquet: b614b478…10ef
• Holdout prompt file (2,000 prompts, disjoint from training ids): sha256 f1f796aa33abc4648c65c037cce500ca3508d451ba939073619c32f87c7bd634
• PROVENANCE.txt: full record, including images, commits, commands and config

Credits

• canada-quant: the training recipe (CE + top-20 KL, gamma-6.5 tail weighting, lr schedule), the 500-prompt holdout eval kit and holdout_client.py (used unmodified, canada-quant/vllm-glm53-flash-sm121 @1c142c95), the GLM-5-next aux-tap overlay we ported, and the card format.
• Inco (incoai/GLM-5.3-Flash-DFlash2, CC-BY-NC-ND-4.0): the drafter geometry we matched for drop-in compatibility, and the reference drafter we measured against. Inco's weights were never a training input or an initialization.
• Tony: the DGX Spark NVFP4 DFlash2 lane (funland/glm53-dflash2:ccc5a78e) and the chat template (byte-identical template used for generation and eval).
• lightseekorg/TorchSpec (MIT): the trainer. Tencent/AngelSpec (Apache-2.0): the top-K KL loss we ported.
• DFlash2 method: z-lab/dflash · arXiv 2602.06036 · inco.ai blog
• Upstream vLLM DFlash2 integration: vllm-project/vllm#52816
• Dataset authors listed in the table above. WildChat-1M is used under ODC-BY-1.0 with attribution.