TinyLiquid — Big-Tech Recipe and Current-Situation Audit

Date: 2026-08-01. Sources are the recipes used by the groups that actually ship small models.

What the sources say

SmolLM (Hugging Face, 2024) — https://huggingface.co/blog/smollm
- 135M/360M trained on 600B tokens; 1.7B on 1T tokens. They explicitly trained small models "even beyond the Chinchilla optimal point" because gains kept coming.
- Corpus = synthetic "textbook" data generated by a strong teacher (Cosmopedia v2, 28B tokens, generated by Mixtral) + curated web (FineWeb-Edu 220B) + code (4B).
- Instruct = SFT then DPO, following the Zephyr/Gemma alignment-handbook recipe.
SmolLM2 (Hugging Face, 2025) — https://huggingface.co/HuggingFaceTB/SmolLM2-135M
- 135M trained on 2T tokens (FineWeb-Edu, DCLM, The Stack + curated).
- Instruct = SFT (smoltalk) then DPO on UltraFeedback.
Phi-3 (Microsoft, 2024) — arXiv:2404.14219
- 3.8B trained on 3.3T tokens of "heavily filtered web data and synthetic data"; data quality is the lever that makes small models competitive.
TinyStories (OpenAI, 2023) — arXiv:2305.07759
- Direct evidence that models BELOW 10M parameters (our scale) become fluent and consistent when trained on large, high-quality synthetic corpora.
LoRA (Hu et al., 2021) — arXiv:2106.09685
- Freeze pretrained weights, train low-rank adapters; the standard defense against catastrophic forgetting during adaptation.
DPO (Rafailov et al., 2023) — arXiv:2305.18290
- Preference optimization without a separate reward model; used after SFT by both SmolLM generations.

Audit of this project

Architecture: 7.8M params, liquid blocks, no attention. Fine for this exercise.
Pretraining used only ~47M tokens (TinyStories slice + code): about 6 tokens/param.
Reference ratios: SmolLM1 135M = 600B tokens = ~4,400 tok/param; SmolLM2 135M = 2T tokens = ~15,000 tok/param; TinyStories sub-10M models were trained on roughly a billion+ tokens of synthetic stories.
Implication: the base is under-trained by 2-3 orders of magnitude. Every SFT/DPO on top of it either collapses (overfit to a few hundred examples) or stays story-like (base never learned the domain). This is why iterative fine-tuning "goes in circles".
Full corpus: data/TinyStoriesV2-GPT4-train.txt (2.2 GB, ~550M tokens) was never tokenized; it is now being encoded to data/train_full.bin by data/encode_full.py.

What we are doing about it (in order)

Continue pretraining on the full ~550M-token corpus with train/train_lm.py --resume ckpt/nlp --data data/train_full.bin ... (memmap loader added; saves every N steps; resumable). Realistic device rate is ~800 tok/s, so this is a multi-day job: 100M tokens ~= 35h. Run in chunks.
Teacher-generated synthetic data (Phi/Cosmopedia lever): expand the analyst voice dataset to thousands of high-quality examples across general chat, truth Q&A, SOP, forensic, tool use, and skeptic attacks.
LoRA SFT (LoRA paper): freeze the base, train low-rank adapters + persona, KL-anchor to the base, eval-driven checkpoint selection with a TinyStories PPL guard.
DPO on a larger generated preference set (SmolLM instruct recipe), only after SFT quality is verified.
Ship: HF export, Q8 GGUF, benchmark, model card, TUI.

Commands

# 1. encode full corpus (done once, ~25 min)
export PYTHONPATH=$PWD
.venv/bin/python data/encode_full.py

# 2. continue pretraining (resumable; each invocation adds --steps)
.venv/bin/python -u train/train_lm.py --resume ckpt/nlp \
  --data data/train_full.bin --val data/valid.bin --config tiny10m \
  --ckpt ckpt/nlp_full --batch 16 --seq 256 --lr 1.5e-4 --warmup 200 \
  --steps 5000 --save-every 500 --eval-every 500 --threads 4

# 3. LoRA SFT (see train/train_lora.py when ready)

Honest expectations

A 7.8M general chatbot is below every published instruct-model floor (135M+).
With the TinyStories result in mind, the achievable target is a coherent, narrow analyst assistant for claim verification / OSINT research via large synthetic data + LoRA SFT + DPO, plus honest limits on the model card.
A competitive general chatbot at this size is not realistic on this device; the pretraining gap alone is ~35-100+ hours at ~800 tok/s.