File size: 4,370 Bytes
d83b47a | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 | # TinyLiquid — Big-Tech Recipe and Current-Situation Audit
Date: 2026-08-01. Sources are the recipes used by the groups that actually ship small models.
## What the sources say
- **SmolLM (Hugging Face, 2024)** — `https://huggingface.co/blog/smollm`
- 135M/360M trained on 600B tokens; 1.7B on 1T tokens. They explicitly trained small
models "even beyond the Chinchilla optimal point" because gains kept coming.
- Corpus = synthetic "textbook" data generated by a strong teacher (Cosmopedia v2,
28B tokens, generated by Mixtral) + curated web (FineWeb-Edu 220B) + code (4B).
- Instruct = SFT then DPO, following the Zephyr/Gemma alignment-handbook recipe.
- **SmolLM2 (Hugging Face, 2025)** — `https://huggingface.co/HuggingFaceTB/SmolLM2-135M`
- 135M trained on 2T tokens (FineWeb-Edu, DCLM, The Stack + curated).
- Instruct = SFT (smoltalk) then DPO on UltraFeedback.
- **Phi-3 (Microsoft, 2024)** — arXiv:2404.14219
- 3.8B trained on 3.3T tokens of "heavily filtered web data and synthetic data";
data quality is the lever that makes small models competitive.
- **TinyStories (OpenAI, 2023)** — arXiv:2305.07759
- Direct evidence that models BELOW 10M parameters (our scale) become fluent and
consistent when trained on large, high-quality synthetic corpora.
- **LoRA (Hu et al., 2021)** — arXiv:2106.09685
- Freeze pretrained weights, train low-rank adapters; the standard defense against
catastrophic forgetting during adaptation.
- **DPO (Rafailov et al., 2023)** — arXiv:2305.18290
- Preference optimization without a separate reward model; used after SFT by both
SmolLM generations.
## Audit of this project
- Architecture: 7.8M params, liquid blocks, no attention. Fine for this exercise.
- Pretraining used only ~47M tokens (TinyStories slice + code): about 6 tokens/param.
- Reference ratios: SmolLM1 135M = 600B tokens = ~4,400 tok/param; SmolLM2 135M =
2T tokens = ~15,000 tok/param; TinyStories sub-10M models were trained on roughly
a billion+ tokens of synthetic stories.
- Implication: the base is under-trained by 2-3 orders of magnitude. Every SFT/DPO on
top of it either collapses (overfit to a few hundred examples) or stays story-like
(base never learned the domain). This is why iterative fine-tuning "goes in circles".
- Full corpus: `data/TinyStoriesV2-GPT4-train.txt` (2.2 GB, ~550M tokens) was never
tokenized; it is now being encoded to `data/train_full.bin` by
`data/encode_full.py`.
## What we are doing about it (in order)
1. **Continue pretraining** on the full ~550M-token corpus with
`train/train_lm.py --resume ckpt/nlp --data data/train_full.bin ...`
(memmap loader added; saves every N steps; resumable). Realistic device rate is
~800 tok/s, so this is a multi-day job: 100M tokens ~= 35h. Run in chunks.
2. **Teacher-generated synthetic data** (Phi/Cosmopedia lever): expand the analyst
voice dataset to thousands of high-quality examples across general chat, truth
Q&A, SOP, forensic, tool use, and skeptic attacks.
3. **LoRA SFT** (LoRA paper): freeze the base, train low-rank adapters + persona,
KL-anchor to the base, eval-driven checkpoint selection with a TinyStories PPL
guard.
4. **DPO** on a larger generated preference set (SmolLM instruct recipe), only after
SFT quality is verified.
5. Ship: HF export, Q8 GGUF, benchmark, model card, TUI.
## Commands
```bash
# 1. encode full corpus (done once, ~25 min)
export PYTHONPATH=$PWD
.venv/bin/python data/encode_full.py
# 2. continue pretraining (resumable; each invocation adds --steps)
.venv/bin/python -u train/train_lm.py --resume ckpt/nlp \
--data data/train_full.bin --val data/valid.bin --config tiny10m \
--ckpt ckpt/nlp_full --batch 16 --seq 256 --lr 1.5e-4 --warmup 200 \
--steps 5000 --save-every 500 --eval-every 500 --threads 4
# 3. LoRA SFT (see train/train_lora.py when ready)
```
## Honest expectations
- A 7.8M general chatbot is below every published instruct-model floor (135M+).
- With the TinyStories result in mind, the achievable target is a coherent,
narrow analyst assistant for claim verification / OSINT research via large
synthetic data + LoRA SFT + DPO, plus honest limits on the model card.
- A competitive general chatbot at this size is not realistic on this device;
the pretraining gap alone is ~35-100+ hours at ~800 tok/s.
|