docs/BIGTECH_RECIPE.md

# TinyLiquid — Big-Tech Recipe and Current-Situation Audit

Date: 2026-08-01. Sources are the recipes used by the groups that actually ship small models.

## What the sources say

- **SmolLM (Hugging Face, 2024)** — `https://huggingface.co/blog/smollm`
  - 135M/360M trained on 600B tokens; 1.7B on 1T tokens. They explicitly trained small
    models "even beyond the Chinchilla optimal point" because gains kept coming.
  - Corpus = synthetic "textbook" data generated by a strong teacher (Cosmopedia v2,
    28B tokens, generated by Mixtral) + curated web (FineWeb-Edu 220B) + code (4B).
  - Instruct = SFT then DPO, following the Zephyr/Gemma alignment-handbook recipe.
- **SmolLM2 (Hugging Face, 2025)** — `https://huggingface.co/HuggingFaceTB/SmolLM2-135M`
  - 135M trained on 2T tokens (FineWeb-Edu, DCLM, The Stack + curated).
  - Instruct = SFT (smoltalk) then DPO on UltraFeedback.
- **Phi-3 (Microsoft, 2024)** — arXiv:2404.14219
  - 3.8B trained on 3.3T tokens of "heavily filtered web data and synthetic data";
    data quality is the lever that makes small models competitive.
- **TinyStories (OpenAI, 2023)** — arXiv:2305.07759
  - Direct evidence that models BELOW 10M parameters (our scale) become fluent and
    consistent when trained on large, high-quality synthetic corpora.
- **LoRA (Hu et al., 2021)** — arXiv:2106.09685
  - Freeze pretrained weights, train low-rank adapters; the standard defense against
    catastrophic forgetting during adaptation.
- **DPO (Rafailov et al., 2023)** — arXiv:2305.18290
  - Preference optimization without a separate reward model; used after SFT by both
    SmolLM generations.

## Audit of this project

- Architecture: 7.8M params, liquid blocks, no attention. Fine for this exercise.
- Pretraining used only ~47M tokens (TinyStories slice + code): about 6 tokens/param.
- Reference ratios: SmolLM1 135M = 600B tokens = ~4,400 tok/param; SmolLM2 135M =
  2T tokens = ~15,000 tok/param; TinyStories sub-10M models were trained on roughly
  a billion+ tokens of synthetic stories.
- Implication: the base is under-trained by 2-3 orders of magnitude. Every SFT/DPO on
  top of it either collapses (overfit to a few hundred examples) or stays story-like
  (base never learned the domain). This is why iterative fine-tuning "goes in circles".
- Full corpus: `data/TinyStoriesV2-GPT4-train.txt` (2.2 GB, ~550M tokens) was never
  tokenized; it is now being encoded to `data/train_full.bin` by
  `data/encode_full.py`.

## What we are doing about it (in order)

1. **Continue pretraining** on the full ~550M-token corpus with
   `train/train_lm.py --resume ckpt/nlp --data data/train_full.bin ...`
   (memmap loader added; saves every N steps; resumable). Realistic device rate is
   ~800 tok/s, so this is a multi-day job: 100M tokens ~= 35h. Run in chunks.
2. **Teacher-generated synthetic data** (Phi/Cosmopedia lever): expand the analyst
   voice dataset to thousands of high-quality examples across general chat, truth
   Q&A, SOP, forensic, tool use, and skeptic attacks.
3. **LoRA SFT** (LoRA paper): freeze the base, train low-rank adapters + persona,
   KL-anchor to the base, eval-driven checkpoint selection with a TinyStories PPL
   guard.
4. **DPO** on a larger generated preference set (SmolLM instruct recipe), only after
   SFT quality is verified.
5. Ship: HF export, Q8 GGUF, benchmark, model card, TUI.

## Commands

```bash
# 1. encode full corpus (done once, ~25 min)
export PYTHONPATH=$PWD
.venv/bin/python data/encode_full.py

# 2. continue pretraining (resumable; each invocation adds --steps)
.venv/bin/python -u train/train_lm.py --resume ckpt/nlp \
  --data data/train_full.bin --val data/valid.bin --config tiny10m \
  --ckpt ckpt/nlp_full --batch 16 --seq 256 --lr 1.5e-4 --warmup 200 \
  --steps 5000 --save-every 500 --eval-every 500 --threads 4

# 3. LoRA SFT (see train/train_lora.py when ready)
```

## Honest expectations

- A 7.8M general chatbot is below every published instruct-model floor (135M+).
- With the TinyStories result in mind, the achievable target is a coherent,
  narrow analyst assistant for claim verification / OSINT research via large
  synthetic data + LoRA SFT + DPO, plus honest limits on the model card.
- A competitive general chatbot at this size is not realistic on this device;
  the pretraining gap alone is ~35-100+ hours at ~800 tok/s.