cookbook/quickstart.md

Quickstart — from zero to a small trained model

1. The loop that makes it possible

For every phase (pretrain, growth, SFT, DPO, eval) do exactly this:

1. Research the phase (seriously; 20-30 min of reading/searching, not vibes).
2. Write a short "skill" (a markdown repo + decision record) capturing:
   • what big-tech actually says,
   • measured numbers from YOUR device,
   • what failed last time (so you never re-run a dead end).
3. Apply the skill (one narrow change).
4. Gate it with a small eval probe battery before moving on.

That's it. The "skill" is the artifact that separates "a build I guessed at" from "a build I own and can hand to a team."

2. Suggested order

1. Token + corpus: build/acquire tokenizer, then a balanced training corpus (see corpus.md). Don't concatenate big blocks; window-shuffle.
2. Pretrain a base (~7M) on a constrained domain until it produces coherent simple text (that's your floor). Checkpoint every ~500 steps, resume-aware.
3. Grow it (tower / identity-blocks) — keep the trained trunk, add identity-initialized layers so the loss is unchanged, then continue-pretrain. Never start from scratch again.
4. Curriculum SFT on hand-written gold (a few hundred consistent examples) where labels are checkable from the prompt (verdict classes).
5. Eval the probe battery; iterate on real mistakes.
6. Export GGUF/Q8 for on-device use.

3. Copy config minimum (on an 8 GB tablet)

• ≈ 7-25M params, fp/bf16, batch 16, seq 256.
• 1 epoch ≈ 11-20 h depending on size. Always resumable.
• One heavy job at a time — two torch processes starve each other.

4. Honest expectations

• Small + niche + honest beats big + generic for decision-support work.
• Read device.md for real across what a tablet can train in a day.