---
language: en
license: mit
tags:
  - tiny-llm
  - tinystories
  - from-scratch
  - educational
  - speculative-decoding
  - dpo
datasets:
  - roneneldan/TinyStories
pipeline_tag: text-generation
---

# tiny-llm — a 27M-parameter LLM built from scratch, end to end

A complete modern LLM pipeline built by hand as a learning mission — every
stage written from scratch in readable PyTorch (no HF Transformers, no
external tokenizer), trained on free Colab T4 and a Mac CPU.

**Pipeline:** own byte-level BPE tokenizer → pretraining → SFT (instruction
following) → DPO (preference alignment) → distilled draft model + speculative
decoding → RAG with copy-tuning → this release.

## Sizes

| File | Model | Params | What it is |
|------|-------|-------:|------------|
| `dpo-m4c.pt` | main | **27.0M** (dim 512, 8 layers, 8 heads, vocab 4096, ctx 512) | pretrained → SFT → DPO story-writer |
| `rag-m6.pt` | RAG variant | 27.0M | same model copy-tuned to quote retrieved context faithfully |
| `draft-m5.pt` | draft | **1.28M** (dim 128, 4 layers) | distilled from the main model for speculative decoding (1.4× on CPU) |
| `tokenizer.json` | BPE | 4096 merges | byte-level BPE trained from scratch on TinyStories |

## Training data

- **Only dataset:** [TinyStories](https://huggingface.co/datasets/roneneldan/TinyStories)
  (synthetic children's stories, English). The validation file was deduplicated
  against train (33% of it duplicated train stories) before any evaluation.
- SFT pairs: `"Write a story about X.\n\n" + story`, where X is strictly the
  story's first-sentence subject appearing ≥2 times (data-quality lesson:
  a noisier heuristic taught the model to ignore instructions).
- DPO pairs: chosen = human story, rejected = the SFT model's own sample.

## Honest limitations

- Writes **only simple children's stories in English**. Nothing else.
- It does not know facts about the world, cannot do math, cannot chat in
  Russian or any other language.
- Instruction following ("about X") works on ~7/10 held-out topics; very rare
  words (e.g. "unicorn") may be ignored.
- The RAG variant only quotes from a supplied context line; without context it
  invents.
- Trained on ~500M tokens of synthetic text at 27M params — a *toy*, roughly
  25,000× smaller than frontier models. Built for learning, not for use.

## Usage

```python
import torch
from model import TinyLLM, ModelConfig      # model.py from this repo
from bpe import BPETokenizer                # bpe.py from this repo

tok = BPETokenizer.load("tokenizer.json")
ck = torch.load("dpo-m4c.pt", map_location="cpu")
m = TinyLLM(ModelConfig(**ck["cfg"])); m.load_state_dict(ck["model"]); m.eval()

# IMPORTANT: encode word-by-word (the same word-cache encoding used in training)
import re
ids = []
for w in re.findall(r"\S+\s*", "Write a story about dragon.\n\n"):
    ids += tok.encode(w)
out = m.generate(torch.tensor([ids]), 120, temperature=0.7)
print(tok.decode(out[0].tolist()))
```

Speculative decoding (`speculative.py`) makes greedy generation ~1.4× faster
on CPU with byte-identical output; `rag.py` + `rag-corpus.txt` show the RAG
setup.

## Validation

Every stage was gated by a frozen validation contract (18 assertions):
pretrain val loss 1.656 (≤2.3), instruction following judged blind 7/10,
DPO preferred blind in 90% of 20 pairs, speculative decoding exact on 20/20
prompts at 1.40×, RAG answers contain the fact on 9/10 questions (0/10
without retrieval). Fresh-context judges never saw the training code.