---
license: mit
language: en
datasets: [roneneldan/TinyStories]
tags: [tiny, from-scratch, pytorch, educational, text-generation]
---
# Tiny Storyteller

A 5.9M-parameter GPT-style language model trained **from scratch** (random init, no pretrained weights)
on 19,800 TinyStories plus 3,000 short synthetic stories about two characters, Momo the monkey and
Ella the elephant (1,000 unique stories, each repeated 3 times, generated by `momo_stories.py`), as a learning project.
Use case: a bedtime story starter for young kids.

- Architecture: 6 transformer blocks, 4 heads, 256-dim embeddings, 256-token context
- Tokenizer: byte-level BPE, 4,096 tokens, trained on the same data
- Training: 1500 steps, batch 32, AdamW, cosine schedule
- Final validation loss 2.202 (perplexity 9.0) on 200 held-out TinyStories (no Momo stories in the validation set)

The full build, explained step by step, is in `tiny_llm_from_scratch.ipynb` in this repo.
Load with plain PyTorch: `GPT(cfg).load_state_dict(torch.load("model.pt"))`.