vanilla-small-20B-valmatched — dense baselines for the small 20B cell

12L/d=1024 GPT, sparsity_mode=none, trained on 20B tokens of FineWeb-Edu. Produces both arms of the small-cell comparison in one run: the val-matched dense baselines (early stops) and the token-matched dense baseline (full budget).

Best val_loss: 2.8586 (iter 12200); final at 20B 2.8664
12,716 iters, 1,572,864 tok/iter → 20,000,538,624 tokens
2× NVIDIA H200 (143 GB), wall clock 8h32m (12:29 → 21:01 PDT, 2026-07-24)
Wandb: 9iezcto0
HF: markhenry/vanilla-small-20B-valmatched

Headline numbers

Run	Val loss	Δ vs dense@20B
`vanilla-small-20B` (this, iter 12200)	2.8586	—
`vanilla-small-20B` (this, final 20B)	2.8664	+0.008 (noise)
`cayley-small-3L-mlp_in-20B`	3.1330	+0.274 (worse)
`cayley-small-2L-mlp_in-20B`	3.1584	+0.300 (worse)

At matched 20B tokens the dense model is 0.274–0.300 nats better. That is the alignment tax in loss terms at this scale, and it is the expected direction: the CayleySAE bottleneck buys disentanglement by spending capability.

Alignment tax (token efficiency)

Dense reaches each sibling's final val far inside the budget:

Sibling	its val	dense first eval ≤	interpolated crossing	tax
2L	3.1584	iter 1900 (3.1536)	~iter 1840, 2.89B	~6.9×
3L	3.1330	iter 2300 (3.1162)	~iter 2215, 3.48B	~5.7×

Both well inside the <15× target. Crossings are linearly interpolated between adjacent evals; eval spacing (100 iters = 157M tokens) caps resolution at ~±5%, so treat these as one significant figure. Quoting the saved-checkpoint iter instead would understate the tax (6.7× / 5.5×).

The save-at-val ladder, and what it taught us

Four rungs were requested: each sibling's final val plus a companion 0.01 nats below (3.1584 / 3.1484 / 3.1330 / 3.1230). All four fired inside 400 iters.

rung	iter	tokens	val at trigger	overshoot
3.1584	1900	2.99B	3.1536	0.005
3.1484	2100	3.30B	3.1354	0.013
3.1330	2300	3.62B	3.1162	0.017
3.1230	2300	3.62B	3.1162	—

The 3L pair collapsed into one model. 3.1330 and 3.1230 fired on the same eval; weights are bit-identical (md5 4d7c75acd798 over the sorted state dict). Only 3 distinct checkpoints exist, and only 3 were uploaded.

Root cause is eval spacing, not noise: dense descends ~0.017–0.020 nats per 100-iter window in this region, wider than the 0.01-nat companion gap. A companion can only be distinct if the per-window descent is smaller than the gap. For the large sibling, either drop companions or tighten --eval-interval to ~25–50 in the crossing region.

The 2L pair did stay distinct, and earns its keep for a different reason than designed. ckpt_val_3.1584.pt beats 2L by only 0.0048 nats — a statistical tie at 4.9M eval tokens, so it does not support "dense was at least as good." ckpt_val_3.1484.pt beats it by 0.0230 (~5× the noise floor) and does. The tight rung is for the tax crossing; the companion is for capability parity.

Eval noise floor

The run-end sequence: 2.8603, 2.8676, 2.8665, 2.8652, 2.8641, 2.8664 — spread 0.0073 with no trend, so ~0.005–0.008 nats at 150 eval iters × bs 32 × 1024. In-crossing region the same wobble shows as 3.1354 → 3.1359 → 3.1162.

Note in-run eval is master-rank only (train.py:899), so eval tokens do not scale with world size: 150 × 32 × 1024 = 4.9M, not the 47M the parent script's header claimed (it multiplied by 8 ranks). Any threshold decision from in-run evals inherits the 4.9M noise floor.

Training health

Throughput 688–697k tok/s from iter 10 to the end; no drift, no warmup pathology (compile warmup cost only iter 0 at 77k). Exactly half the 4×H200 1,389k of report 37, i.e. clean linear scaling in world size.
grad_norm fell to ~0.20 by iter 3400 and ~0.09 at the end; monotone, no spikes. Peak VRAM 27.6 GB / 143 GB per GPU — bs could go far higher, but was held at 32 to keep tok/iter identical to the recorded recipe.
Loss monotone modulo the noise floor throughout.

Node-shape delta (2× H200, not 4×)

Recipe unchanged from train_vanilla_small_20B_valmatched_4xh200.sh. Only --nproc_per_node 4→2. Because ga is the global micro-step count, tok/iter stays 1,572,864 and the LR schedule occupies the same 12,716-iter space; each rank runs 24 micro-steps instead of 12. Wall clock doubled, nothing else changed. Script: mh/scripts/train_vanilla_small_20B_valmatched_2xh200.sh.

Two defects found in the existing scripts

1. The vanilla-small script family cannot launch. train_vanilla_small_20B.sh, ..._valmatched.sh, and ..._valmatched_4xh200.sh all pass --n-embd 1024 without --n-head. train.py applies mode-dependent arch defaults (train.py:1131): cayley → n_head=8, else n_head=12. With sparsity_mode=none that resolves to 12, and assert n_embd % n_head == 0 fails instantly since 1024 % 12 ≠ 0. The cayley-small siblings silently took the cayley branch and got n_head=8, which is why only the vanilla side is broken. Fixed in the 2xh200 script with an explicit --n-head 8 (verified 202.53M params, matching the siblings exactly). The three parents remain un-runnable.

2. quick_eval silently skipped for the entire run — 6 attempts, all import failed (No module named 'evals'); skipping. The script does cd deeptopk then uv run, which resolves to deeptopk/.venv; the evals package is declared by the root pyproject.toml and is only present in the root .venv. So no pile/hellaswag/lambada during training. Harmless here (val_loss unaffected, report 42 does these post-hoc) but it means the quick-eval columns are absent from this wandb run.

Artifacts

Five checkpoints in one HF repo, all also at sparse-nanogpt-private/out/vanilla-small-20B/ (md5-verified against /dev/shm):

file	iter	tokens	val
`ckpt_val_3.1584.pt`	1900	2.99B	3.1536
`ckpt_val_3.1484.pt`	2100	3.30B	3.1354
`ckpt_val_3.1330.pt`	2300	3.62B	3.1162
`ckpt_final_20B.pt`	12716	20.001B	2.8664
`ckpt_best.pt`	12200	19.19B	2.8586

Recommended use

Interpretability comparison vs 2L → ckpt_val_3.1484.pt (unambiguously better than the sibling, so no handicapped-baseline objection).
Interpretability comparison vs 3L → ckpt_val_3.1330.pt.
Single comparator for both → ckpt_val_3.1330.pt (val 3.1162 beats 2L by 0.042 and 3L by 0.017).
Equal-budget tax in nats → ckpt_final_20B.pt.
Token-efficiency tax → interpolated crossings above, not the saved iters.

All val numbers above are in-run trigger values at 4.9M eval tokens. Re-evaluate with a large eval before quoting any as a val-matched number; the purpose of that pass is to confirm the dense ≥ cayley inequality still holds, not to find the closest match.

Data note

fineweb-edu-25B was re-tokenized on this node from sample-100BT (24,975,629,888 train / 24,433,651 val tokens, 40 streams, ~6 min). Same subset and distribution as the siblings, but a different stream count means a different shard→token layout, so this is not a byte-identical data replay. Val comparisons hold; exact-trajectory comparisons do not.