language: en
license: mit
tags:
- tiny-llm
- tinystories
- from-scratch
- educational
- speculative-decoding
- dpo
datasets:
- roneneldan/TinyStories
pipeline_tag: text-generation
tiny-llm — a 27M-parameter LLM built from scratch, end to end
A complete modern LLM pipeline built by hand as a learning mission — every stage written from scratch in readable PyTorch (no HF Transformers, no external tokenizer), trained on free Colab T4 and a Mac CPU.
Pipeline: own byte-level BPE tokenizer → pretraining → SFT (instruction following) → DPO (preference alignment) → distilled draft model + speculative decoding → RAG with copy-tuning → this release.
Sizes
| File | Model | Params | What it is |
|---|---|---|---|
dpo-m4c.pt |
main | 27.0M (dim 512, 8 layers, 8 heads, vocab 4096, ctx 512) | pretrained → SFT → DPO story-writer |
rag-m6.pt |
RAG variant | 27.0M | same model copy-tuned to quote retrieved context faithfully |
draft-m5.pt |
draft | 1.28M (dim 128, 4 layers) | distilled from the main model for speculative decoding (1.4× on CPU) |
tokenizer.json |
BPE | 4096 merges | byte-level BPE trained from scratch on TinyStories |
Training data
- Only dataset: TinyStories (synthetic children's stories, English). The validation file was deduplicated against train (33% of it duplicated train stories) before any evaluation.
- SFT pairs:
"Write a story about X.\n\n" + story, where X is strictly the story's first-sentence subject appearing ≥2 times (data-quality lesson: a noisier heuristic taught the model to ignore instructions). - DPO pairs: chosen = human story, rejected = the SFT model's own sample.
Honest limitations
- Writes only simple children's stories in English. Nothing else.
- It does not know facts about the world, cannot do math, cannot chat in Russian or any other language.
- Instruction following ("about X") works on ~7/10 held-out topics; very rare words (e.g. "unicorn") may be ignored.
- The RAG variant only quotes from a supplied context line; without context it invents.
- Trained on ~500M tokens of synthetic text at 27M params — a toy, roughly 25,000× smaller than frontier models. Built for learning, not for use.
Usage
import torch
from model import TinyLLM, ModelConfig # model.py from this repo
from bpe import BPETokenizer # bpe.py from this repo
tok = BPETokenizer.load("tokenizer.json")
ck = torch.load("dpo-m4c.pt", map_location="cpu")
m = TinyLLM(ModelConfig(**ck["cfg"])); m.load_state_dict(ck["model"]); m.eval()
# IMPORTANT: encode word-by-word (the same word-cache encoding used in training)
import re
ids = []
for w in re.findall(r"\S+\s*", "Write a story about dragon.\n\n"):
ids += tok.encode(w)
out = m.generate(torch.tensor([ids]), 120, temperature=0.7)
print(tok.decode(out[0].tolist()))
Speculative decoding (speculative.py) makes greedy generation ~1.4× faster
on CPU with byte-identical output; rag.py + rag-corpus.txt show the RAG
setup.
Validation
Every stage was gated by a frozen validation contract (18 assertions): pretrain val loss 1.656 (≤2.3), instruction following judged blind 7/10, DPO preferred blind in 90% of 20 pairs, speculative decoding exact on 20/20 prompts at 1.40×, RAG answers contain the fact on 9/10 questions (0/10 without retrieval). Fresh-context judges never saw the training code.