k3f_exp_ckpts — DSpark draft checkpoints from the GB300 (g3*) experiments

Final-iteration draft weights for six single-variable arms trained against moonshotai/Kimi-K3-Flash on the GB300 cluster. Staged for batch export + evaluation on another cluster: each folder holds the model/ shards, the run's own config, and the draft architecture JSON.

optimizer/ state is deliberately excluded — it is ~80% of the bytes and only needed to resume training.

Shard halves do not live on a fixed node. g3g/g3j had ranks 0-3 on .101 and 4-7 on .31; g3l is the reverse, with .metadata on .31. Read the layout per arm.

Every arm's model/ needs all 8 shards (__0_0.distcp … __7_0.distcp plus .metadata, which names all eight). The 2×4 trainer wrote ranks 0-3 and ranks 4-7 to different nodes, so an assembly that reads only one node yields a 4-of-8 set that cannot be loaded at all. Verify the count, not just that model/ exists — an earlier revision of this repo shipped four arms with half their shards and passed an existence check.

The arms, and what each one changes

The chain is single-variable: g3d → g3e → g3f → {g3g, g3h, g3j}. Each arm differs from its parent by one thing, so adjacent arms are differenceable.

g3j is the one arm that varies initialisation rather than architecture: it has g3f's draft JSON byte-for-byte but starts from g3e's converged weights instead of scratch, so g3f vs g3j isolates warm-start. Note it loads block-5 weights and then trains at block 3 — the load was clean (no shape warnings in the trainer log), but serve it with num_speculative_tokens 3, not 5.

arm	final iter	fc_norm	block	taps	engines	changes vs parent
g3c	6413	false	3	[1,29,57]	16	all-turn corpus (not single-variable vs g3d)
g3d	651	true	5	[1,29,57]	16	baseline — v3d recipe on GB300
g3e	4581	false	5	[1,29,57]	16	fc_norm true→false
g3f	4581	false	3	[1,29,57]	16	block_size 5→3
g3g	4581	false	3	[46,57,59]	16	tap placement
g3h	2751	false	3	[1,29,57]	12	4→3 producer engines
g3j	4581	false	3	[1,29,57]	16	warm-start from g3e iter 4581 (continual_training)
g3l	7527	false	3	[1,29,57]	16	v3h6 draft + v34 corpus, 2 epochs — not single-variable
g3m	4455	false	3	[1,29,57]	16	max_seq_length 65536→40000 (also drops 8000 long rows)
g3n	4455	false	3	[1,29,57]	16	rope yarn → default (theta 50000→1e6), vs g3m
g3o	4455	false	3	[1,29,57]	16	rope_theta 1e6→5e4 vs g3n — measured inert
g3p	4581	false	3	[1,29,57]	16	v5 corpus @ 65536 — two variables vs g3n
g3q	9161	false	3	[1,29,57]	16	num_epochs 2→4 on v5, vs g3p
g3r	14325	false	3	[1,29,57]	16	v6 text-only corpus (zero tool calls), vs g3p
g3s	18019	false	3	[1,29,57]	16	v7 text-only corpus (1.15M rows, 66.5% reasoning), vs g3r

Shared by every arm: 3-layer MLA draft, enable_confidence_head: true, markov_head_type: vanilla, target depth 61, max_seq_length: 65536, last_turn_loss_only: true, v3 blended corpus (except g3c).

Read these before comparing numbers

• g3e, g3f, g3g and g3j all completed the same 4580-step schedule and are step-aligned with each other.
• g3d stopped at iteration 651. It is an early checkpoint, not a converged model — comparing its acceptance against the 4581-step arms at face value is wrong.
• g3h stopped at 2751, also short of the full schedule.
• g3h is not a modelling change. Same draft JSON path as g3f; only producer capacity differs (4 engines → 3). It answers "was g3f producer-bound", nothing about architecture.
• g3c changes the corpus, so it is not on the single-variable chain.
• g3c was OOM-killed and resumed, and its resume job (g3cr) wrote into the same directory name on different nodes (.61/.62 instead of .101/.31). An earlier revision of this repo therefore shipped the pre-OOM iteration 6001; the real endpoint is 6413 (global_step 6412, the full planned schedule). When an arm has a resume job, read that job's pod spec for the output hostPath — the original arm's nodes keep a stale copy.
• g3p → g3q isolates epoch count (2→4, one config line; 4580→9160 derived steps on the same v5 data) and g3p → g3r isolates the corpus (one line). g3q vs g3r differ in two keys, so difference each against g3p, not each other.
• g3s is g3r with v7 instead of v6 (one config line), but v7 differs from v6 in three ways at once: 1.15M rows vs 917K, median row ~1.8x longer, and only 66.5% of rows carry reasoning against v6's ~100%. Since supervision is last_turn_loss_only, a third of v7's supervised spans are bare answers with no think channel. v7 also looks single-source (every sampled id is lc_-prefixed) where v6 was mixed. Both are still zero tool calls, so g3r vs g3s compares within the prose-only regime.
• g3r's v6 corpus contains NO tool calls — 0 of 3000 sampled rows, against v5's 148 (4.9%); the _text_ in v6_text_train is literal. Acceptance is governed by whether the supervised turn emits a tool call (tool-call turns draft at tau ~4.26, prose at 2.05-2.96), so g3r is trained entirely on the harder class. Expect it to read worse on mixed traffic; that is a supervision-mix difference, not a corpus-quality one. It is also 3.1x larger (916820 rows), which is why 2 epochs yields 14324 steps.
• g3p moves two things at once (corpus v3→v5 and max_seq_length 40000→65536 vs g3n), so it is not differenceable against any arm here.
• dataset_size cannot identify the corpus. v5 has exactly v3's row count (293182 both) while being a different, much smaller file (3.36 GB vs 8.33 GB, different content). g3p and g3f therefore both log dataset_size=293182 on different corpora — read train_data_path, not the row count.
• The rope trio g3m/g3n/g3o decomposes one effect. Non-rope keys are byte-identical across all three and all three stop at the same derived iteration, so: default/1e6 costs −5.45% acc against g3m's yarn/5e4, but dropping theta 1e6→5e4 at fixed rope_type (g3o vs g3n) costs only −0.15%. Base frequency is not what matters — 97.3% of the gap is yarn's interpolation plus the attention temperature that rides with rope_type (softmax_scale is derived in the rope block and the yarn branch only fires under rope_type: yarn). Details in arthur/docs/runs/53-g3o-rope-theta-isolated.md.
• g3m vs g3n isolates rope and is the cleanest pair here: identical in every other key, same derived schedule (lr_total_steps=4454, dataset_size=285182), both stopped at 4455. g3m is yarn (factor 256, theta 50000); g3n is default rope (theta 1e6). Serve each under the rope it trained with.
• g3m lowers max_seq_length to 40000, which does not merely truncate: the same corpus file yields 285182 rows against g3f's 293182, so 8000 long rows (2.7%) are excluded entirely. A g3f-vs-g3m difference therefore mixes "shorter context" with "fewer long examples"; it does not isolate either.
• g3l changes four things at once (draft architecture, corpus, min_loss_tokens, and a derived rather than pinned LR schedule), so it is not differenceable against g3f either. Its 7527 iterations come from num_epochs: 2 over a 481781-row corpus (lr_total_steps=7526, derived) — not from training the same thing for longer.
• Step noise on these runs is 5-9%; single-step reads are not resolvable. Use windowed medians.

Using a checkpoint

These are TorchSpec distcp shards, not an HF model folder. Convert first (arthur/scripts/export_k3flash_dspark_vllm.sh, wrapping TorchSpec tools/convert_to_hf.py --vllm), then pass the result as the draft in vLLM's --speculative-config.

Two gates that fail silently if skipped:

1. num_speculative_tokens must equal the arm's block_size (5 for g3d/g3e, 3 for the rest). A mismatch drafts a different width and reads as low acceptance rather than as an error.
2. The target must be 61 layers deep. Against a different depth, vLLM rescales the aux capture ids and logs only a WARNING, after which the draft reads layers it never trained on — again indistinguishable from a weak draft.

Serving K3-Flash on agent traffic additionally needs --enable-auto-tool-choice --tool-call-parser kimi_k3; without them every tools-carrying request 400s before inference and the run looks like spec decode is off.