# Training a small LLM from scratch — an operating guide

For the next agent session that has to build, train, publish and evaluate a ~100M-parameter language model
on rented, capped, per-week GPU. It is the distilled version of one real project: 106,194,240 params,
999,817,216 tokens, 2×Tesla T4 on Kaggle, storage on the Hugging Face Hub, eight academic benchmarks, and a
public report — plus 54 written-up failure records spanning E-001…E-056 that are the actual source of these rules.

Read in order once; then use `06-checklists.md` as the working page.

| File | What it covers |
|---|---|
| `01-before-you-start.md` | The constitution: what to fix in writing, and why memory files are the real state |
| `02-data.md` | Mix design, tokenisation, sharding, the contamination audit, publishing reproducibly |
| `03-preflight.md` | The evidence ladder: free rehearsal → cheap smoke → measurement probe → run |
| `04-the-run.md` | Checkpoints, resume, session sizing, quota arithmetic, what to do when it breaks |
| `05-publish-and-evaluate.md` | Model card, clean-room load, benchmark protocol, honest reporting |
| `06-checklists.md` | Copy-paste gates for each phase, and the recurring failure patterns |
| `07-platform-notes.md` | Kaggle and HF mechanics measured rather than believed, with the numbers |

## The five sentences that matter most

1. **Every claim about a tool is a hypothesis until you have executed it** — the expensive mistakes in this
   project were all "documented behaviour" that the installed version did not have.
2. **Verify bytes, never listings.** A commit that returned, a file that is listed, and bytes that
   download are three different facts, and only the third authorises deleting your copy.
3. **Design for interruption; it is not an exception.** Every interval must end on a verified, pointed-at
   checkpoint on shared storage, and the only resume test that counts deletes the local disk first.
4. **Free compute is a strategy, not a fallback.** A CPU rehearsal that costs nine minutes buys back a
   five-hour session; run everything that does not need the accelerator there, forever.
5. **Write the report before the results exist.** Frozen choices, pre-registered targets and a contamination
   statement composed after you can see the scores are a different kind of document.

## What "done" means

A public model repo that a stranger loads with `from_pretrained` from an empty cache; a dataset repo whose
manifest, hashes and exclusion masks let someone rebuild that exact mix; a benchmark table where every row
names its metric, its shot count and its split, and can be re-run from published config; and a report that
says what failed. Model quality is not the deliverable — a trustworthy number is.