| --- |
| license: mit |
| tags: |
| - pythia |
| - gpt-neox |
| - pretraining |
| - mech-interp |
| --- |
| |
| # Readout recipe-control models (31M, 4 matched arms) |
|
|
| Four matched 31M Pythia-style (GPTNeoX architecture) pretraining runs used as |
| the optimizer-recipe sensitivity control in **Learning to Read Out: |
| Unembedding Dynamics in Language Model Pretraining** (appendix, |
| `fig:app-recipe-control-geometry`). All arms share one tokenized Pile slice |
| (`pile_10B_seed1234.bin`, GPT-NeoX-20B tokenizer, data_seed=1234), the same |
| parameter seed (0), data order, fp16 precision, global batch 1024 |
| (2,097,152 tokens/step), weight decay 0.1, and a 10B-token budget; each arm |
| perturbs exactly one recipe axis: |
| |
| | Arm | Perturbation | |
| |---|---| |
| | `baseline/` | none (peak_lr 1e-3, warmup 1430 steps, W_U lr multiplier 1.0) | |
| | `long_warmup/` | extended LR warmup | |
| | `wu_lr_0p25/` | output-readout (W_U) learning-rate multiplier 0.25× | |
| | `wu_lr_4x/` | output-readout (W_U) learning-rate multiplier 4× | |
|
|
| Each arm ships `config.json` (full training config), `metrics.csv` |
| (train/val curves), and `ckpts/step<N>/model_fp16.pt` checkpoint trajectories |
| at token-milestone steps. |
|
|
| The trainer, tokenizer pipeline, and slice-building scripts are in the code |
| release under `experiments/ablations/pretraining_recipe_control/` |
| (https://github.com/hematteo/learning-to-read-out): rebuild the exact slice |
| with `trainer/tokenize_slice.py` or `scripts/fetch_pythia_preshuffled.py` |
| (sources and licences in `docs/DATA.md`). These are research artifacts for |
| analyzing W_U readout geometry across training, not general-purpose language |
| models. |
| |
| ## Citation |
| |
| ```bibtex |
| @misc{he2026learningtoreadout, |
| title = {Learning to Read Out: Unembedding Dynamics in Language Model Pretraining}, |
| author = {He, Matteo and Shen, William F. and Iacob, Alex and Jovanovic, Andrej |
| and Qiu, Xinchi and Lane, Nicholas D.}, |
| year = {2026}, |
| note = {Under review. Code: https://github.com/hematteo/learning-to-read-out}, |
| } |
| ``` |
| |
| MIT. Trained on a slice of the Pile (`monology/pile-uncopyrighted` / |
| `EleutherAI/pile-standard-pythia-preshuffled`); see the Pile's data statement |
| for upstream text provenance. |
| |
