File size: 1,022 Bytes
00fbfe5 c9b67bf 00fbfe5 c9b67bf 00fbfe5 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 | ---
license: mit
language: en
datasets: [roneneldan/TinyStories]
tags: [tiny, from-scratch, pytorch, educational, text-generation]
---
# Tiny Storyteller
A 5.9M-parameter GPT-style language model trained **from scratch** (random init, no pretrained weights)
on 19,800 TinyStories plus 3,000 short synthetic stories about two characters, Momo the monkey and
Ella the elephant (1,000 unique stories, each repeated 3 times, generated by `momo_stories.py`), as a learning project.
Use case: a bedtime story starter for young kids.
- Architecture: 6 transformer blocks, 4 heads, 256-dim embeddings, 256-token context
- Tokenizer: byte-level BPE, 4,096 tokens, trained on the same data
- Training: 1500 steps, batch 32, AdamW, cosine schedule
- Final validation loss 2.202 (perplexity 9.0) on 200 held-out TinyStories (no Momo stories in the validation set)
The full build, explained step by step, is in `tiny_llm_from_scratch.ipynb` in this repo.
Load with plain PyTorch: `GPT(cfg).load_state_dict(torch.load("model.pt"))`.
|