README.md

---
license: mit
datasets:
  - HuggingFaceFW/fineweb-edu
language:
  - en
tags:
  - gpt
  - baseline
  - interpretability
  - paritytransformer
---

# vanilla-large-20B — dense baselines (24L / d=2048, 1.31B)

Dense (`sparsity_mode=none`) counterparts to the ParityTransformer large-chassis
flagship [`markhenry/cayley-large-2L-mlp_in-20B`](https://huggingface.co/markhenry/cayley-large-2L-mlp_in-20B).
All checkpoints come from **one 20B-token run**, so both baseline definitions
share a single trajectory.

## Checkpoints

Files are named by **iter**. (They were previously named by the `--save-at-val`
threshold that fired; those names misstated the actual loss — `ckpt_val_2.8000.pt`
held a model at 2.7950 — so they were renamed. Two thresholds fired on the same
eval and produced byte-identical files; only one copy is kept.)

| file | iter | tokens | **measured val_loss** | vs flagship |
|---|---|---|---|---|
| `ckpt_val_3400.pt` | 3400 | 5.35B | **2.7981** | +0.0023 worse (7σ) |
| `ckpt_val_3800.pt` | 3800 | 5.98B | **2.7764** | −0.0194 better (60σ) |
| `ckpt_final_20B.pt` | 12716 | 20.001B | **2.5419** | −0.2538 better (338σ) |

Reference, measured in the same pass on the same batches:

| reference | measured val_loss | published (in-run) |
|---|---|---|
| `cayley-large-2L-mlp_in-20B` (flagship) | **2.7957** | 2.8081 |
| `cayley-flagship-1.3b-20b` (2L + aux losses) | **2.8309** | 2.842 |

`ckpt_val_3400.pt` fired a `--save-at-val` trigger but is **0.0023 nats worse**
than the flagship (7σ) — it does *not* clear it. It does clear the aux-loss
variant by 0.0328 (86σ). `ckpt_val_3800.pt` clears both.

`ckpt_val_3800.pt` being *better* than the flagship is the useful direction: a
dense model that beats the ParityTransformer on val and is still less
interpretable cannot be dismissed as a handicapped baseline.

`ckpt_best.pt` (iter 12400) is **not** uploaded — `ckpt_final_20B.pt` measures better, so the "best" label was a noise artifact of the 3.28M-token
in-run eval and the file carries no information the final one lacks.

## How these numbers were measured

`evals/language_modeling/eval_val_loss.py` over the **entire** FineWeb-Edu val
split — contiguous non-overlapping windows, bf16 autocast, batch 8 × 1024,
`--data fineweb-edu-25B`, seed 0.

| quantity | value |
|---|---|
| tokens per model | **25,161,728** (whole val split) |
| batches per model | 3,072, **byte-identical across all models** |
| absolute SEM (single model) | ±0.0047 |
| paired SEM (difference of two models) | **±0.0003** |

The paired SEM is ~17× tighter because shared passage difficulty cancels. Quote
the paired differences, not error bars added in quadrature.

Two reasons the run log disagrees, both of which bite if you mix sources:

| source of disagreement | effect |
|---|---|
| In-run evals are master-rank only, `eval_iters × bs × seq_len` = **3.28M tokens** regardless of world size; adjacent evals wobble 0.005–0.017 nats. | `--save-at-val` triggers and "best" labels inherit that noise. |
| `best_val_loss` inside a checkpoint is the **run's** best, not that checkpoint's val. | Differencing it against a reference `ckpt.pt` saved at another iter is meaningless. |

The flagship's published **2.8081** is an in-run number; on this protocol it is
**2.7957**. Use 2.7957 against these checkpoints and do not mix the two.

## Alignment tax

**Equal budget (20B tokens both sides):**

| pair | dense | cayley | gap |
|---|---|---|---|
| vs `cayley-large-2L` | 2.5419 | 2.7957 | **0.254 nats** |
| vs `cayley-large-2L`+aux | 2.5419 | 2.8309 | **0.289 nats** |

**Token efficiency** — where dense first reaches the flagship's measured val,
interpolated between the two bracketing measured points:

| sibling | its measured val | dense crossing | tokens | tax |
|---|---|---|---|---|
| `cayley-large-2L` | 2.7957 | iter ~3444 | 5.42B | **~3.7×** |

Caveats before this goes in a paper table:

| caveat | detail |
|---|---|
| not scale-invariant | the small chassis gives **6.3–7.5×** on the same construction ([`vanilla-small-20B`](https://huggingface.co/markhenry/vanilla-small-20B)); the tax shrinks as the backbone grows |
| not comparable to `vanilla-large-parity-3B`'s 6.7× | that run had a 3B budget and was *warming down into* the target, which buys val per token; this one passes the crossing at flat peak LR (warmdown starts iter 6358) |
| resolution | eval spacing (100 iters = 157M tokens) caps it; two significant figures |

## Recipe: identical to the flagship except `sparsity_mode`

| setting | value |
|---|---|
| arch | 24L / 16H / d=2048, learned abs. pos enc, bf16 |
| muon lr | 8e-3 → 1e-4 |
| adamw lr | 3e-4 → 1e-5 |
| schedule | `linear_warmdown`, wf=0.5, warmup 200 |
| tokens/iter | 32 × 1024 × 48 = **1,572,864** (12,716 iters) |
| data | FineWeb-Edu `sample-100BT`, GPT-2 tok, 24.97B train tokens |
| run | 4× H200, 23h57m, ~252k tok/s, peak 67.8 GB/GPU |
| wandb | [`700d4i1i`](https://wandb.ai/markhenrysoftware/sparse-nanogpt/runs/700d4i1i) |

Deltas vs the flagship:

| delta | note |
|---|---|
| `--sparsity-mode none` | drops the five cayley flags — the only intended difference |
| `--eval-interval 100` (vs 500) | pins the val-matched crossing |
| `--save-at-val` | writes the first-crossing checkpoints |
| 4× H200 instead of 8 | **no effect on optimization** — `--gradient-accumulation-steps` is the *global* micro-step count, so ga stays 48 and tok/iter is identical |

## Loading

```python
import torch
ck = torch.load("ckpt_val_3800.pt", map_location="cpu", weights_only=False)
ck["iter_num"]                              # 3800
cfg, sd = ck["model_config"], ck["model"]   # GPTConfig fields, 148 tensors
```

Checkpoints carry `optimizer_states` (Muon momentum + AdamW), so they are
resumable, not inference-only.

## Siblings

- [`markhenry/cayley-large-2L-mlp_in-20B`](https://huggingface.co/markhenry/cayley-large-2L-mlp_in-20B) — the flagship this is matched to
- [`markhenry/vanilla-small-20B`](https://huggingface.co/markhenry/vanilla-small-20B) — same construction, small chassis

---

*Measurements and card by Claude Code, from the run log and a paired
re-evaluation of every checkpoint over the whole val split.*