Upload README.md with huggingface_hub

README.md

@@ -1,51 +1,53 @@
-# tinyllm
+---
+license: mit
+language:
+- en
+tags:
+- pytorch
+- transformer
+- language-model
+- from-scratch
+- educational
+- shakespeare
+- rope
+- swiglu
+- rmsnorm
+- kv-cache
+datasets:
+- tiny-shakespeare
+pipeline_tag: text-generation
+---
+
+# tiny-gpt-shakespeare
 
 <p align="center">
-  <img src="images/robot_shakespeare.png" alt="A tiny robot reading Shakespeare by candlelight, with transformer layers glowing inside its transparent head" width="700">
+  <img src="images/brain_book.png" alt="A glowing neural network brain floating above an open Shakespeare book" width="600">
 </p>
 
 <p align="center">
-  <strong>I built a tiny LLM from scratch to understand how GPT-4, Claude, and LLaMA actually work.</strong>
+  <strong>I built a tiny LLM from scratch to understand how GPT-4 and LLaMA actually work.</strong>
 </p>
 
 <p align="center">
-  <em>10M parameters. Trained on Shakespeare. Modernized with the same architecture as LLaMA and Qwen. Every line of code written from scratch.</em>
+  <em>10M parameters. Trained on Shakespeare. Every line of code written from scratch. Every mistake documented.</em>
 </p>
 
 <p align="center">
-  <a href="DEVLOG.md">Learning Journal</a> |
-  <a href="https://huggingface.co/bmeyer2025/tiny-gpt-shakespeare">HuggingFace</a> |
-  <a href="MODEL_CARD.md">Model Card</a>
+  <a href="https://github.com/brianmeyer/tinyllm">GitHub</a> |
+  <a href="https://github.com/brianmeyer/tinyllm/blob/main/DEVLOG.md">Learning Journal</a>
 </p>
 
 ---
 
-## The idea
-
-GPT-4, Claude, and LLaMA are all scaled-up versions of the same architecture. I wanted to understand it from the ground up — not by reading papers, but by building it myself.
-
-So I built a 10M parameter transformer, trained it on Shakespeare, then upgraded it piece by piece with the same components used in production LLMs. Every mistake, crash, and debugging session is documented in the [DEVLOG](DEVLOG.md).
+## What is this?
 
-## What makes it "modern"?
+A ~10M parameter decoder-only transformer — no HuggingFace Transformers library, no pretrained weights, no shortcuts. Built from an empty file to a working Shakespeare generator, then modernized with the same architecture used in LLaMA, Qwen, and Mistral.
 
-I started with a vanilla GPT-2-style transformer, then swapped in four upgrades — one at a time, measuring each:
+This is a learning project. The model itself is tiny and toy-scale. The value is in the code, the [DEVLOG](https://github.com/brianmeyer/tinyllm/blob/main/DEVLOG.md), and the 9 things that went wrong along the way.
 
-| Component | GPT-2 era | Modern (LLaMA/Qwen) | Impact |
-|-----------|-----------|---------------------|--------|
-| Normalization | LayerNorm | **RMSNorm** | Free efficiency win |
-| FFN | ReLU | **SwiGLU** | **-0.11** val loss |
-| Position | Learned embeddings | **RoPE** | **-0.31** val loss |
-| Inference | Recompute all | **KV Cache** | Faster generation |
-
-**RoPE was the star** — biggest improvement, fewer parameters, and the position encoding math is genuinely beautiful.
-
-## Results
-
-| Model | Best Val Loss | Training Time |
-|-------|-------------|--------------|
-| Vanilla (10.8M params) | 1.4804 | 57 min |
-| **Modern (10.6M params)** | **1.4754** | **64 min** |
+## It generates Shakespeare
 
+**Modern model, temp=0.8 (RMSNorm + SwiGLU + RoPE + KV cache):**
 ```
 ROMEO:
 A gallant-house! what says the woe?
@@ -59,116 +61,105 @@ Which hath a sin by him come to the crown,
 That he is reports for me; for ever is he.
 ```
 
-*A 10M parameter model generating Shakespeare dialogue after 67 minutes of training.*
+**Vanilla model, temp=0.5:**
+```
+KING HENRY:
+The father of the marriage of my son,
+And then we will be no longer to be then,
+And but the Lord Hastings of Semiram Stanley.
+```
+
+Not perfect. But recognizable Shakespeare — proper character names, dialogue formatting, verse rhythm — from a 10M param model trained for ~60 minutes on 1MB of text.
 
-## Project structure
+## Architecture
 
 ```
-tinyllm/
-├── src/                        # Core model code (built from scratch)
-│   ├── tokenizer.py            #   Character-level tokenizer + data loading
-│   ├── attention.py            #   Single-head causal self-attention
-│   ├── transformer.py          #   Multi-head attention, FFN, transformer Block
-│   ├── model.py                #   Full vanilla GPT (10.8M params)
-│   ├── modernize.py            #   Modern components: RMSNorm, SwiGLU, RoPE, KV cache
-│   ├── model_modern.py         #   Modernized GPT (10.6M params)
-│   └── generate.py             #   Text generation with sampling
-│
-├── experiments/                # Per-swap A/B comparisons
-│   ├── swap1_rmsnorm.py        #   LayerNorm → RMSNorm (2000 steps)
-│   ├── swap2_swiglu.py         #   ReLU → SwiGLU (2000 steps)
-│   ├── swap3_rope.py           #   Learned pos → RoPE (2000 steps)
-│   └── swap4_kvcache.py        #   KV cache speed benchmark
-│
-├── training/                   # Training scripts
-│   ├── train.py                #   Vanilla GPT (5000 steps)
-│   ├── train_modern.py         #   Modern GPT with early stopping
-│   ├── train_bpe.py            #   BPE + gradient accumulation
-│   └── benchmark.py            #   Samples, latency, throughput comparison
-│
-├── colab/                      # Google Colab
-│   └── train_colab.py          #   All-in-one: vanilla + modern + BPE + benchmarks
-│
-├── data/input.txt              # Tiny Shakespeare (~1.1MB)
-├── images/                     # Generated graphics
-├── DEVLOG.md                   # Full learning journal (the real value)
-├── MODEL_CARD.md               # HuggingFace model card
-└── publish.py                  # Upload to HuggingFace
+ModernGPT (10.6M params)
+  token_emb:   Embedding(65, 384)
+  blocks × 6:
+    RMSNorm → MultiHeadAttention(6 heads, RoPE, KV cache) → residual
+    RMSNorm → SwiGLU(384 → 1024 → 384) → residual
+  RMSNorm → lm_head (tied with token_emb)
 ```
 
-## Quick start
+Four upgrades over vanilla GPT-2, each tested in isolation:
 
-**On Google Colab (recommended):**
-```python
-!git clone https://github.com/brianmeyer/tinyllm.git
-%cd tinyllm
-!pip install tiktoken
-!python -u colab/train_colab.py
-```
+| Upgrade | What changed | Impact |
+|---------|-------------|--------|
+| **RMSNorm** | Drop mean subtraction from LayerNorm | Free efficiency win |
+| **SwiGLU** | Smooth gating replaces hard ReLU cutoff | **-0.11** val loss at step 500 |
+| **RoPE** | Rotate Q/K vectors instead of adding position embeddings | **-0.31** val loss at step 500 |
+| **KV Cache** | Cache keys/values during generation | Faster inference |
 
-**Locally (M4 Mac / any GPU):**
-```bash
-git clone https://github.com/brianmeyer/tinyllm.git
-cd tinyllm
-python3 -m venv .venv && source .venv/bin/activate
-pip install -r requirements.txt
-python -u training/train.py              # vanilla, ~60 min
-python -u training/train_modern.py       # modern, ~67 min
-python src/generate.py --demo            # see the output
-```
+## Results
 
-## The learning journey
+| Model | Params | Best Val Loss | Time |
+|-------|--------|-------------|------|
+| Vanilla | 10.8M | 1.4804 | 57 min |
+| **Modern** | **10.6M** | **1.4754** | **64 min** |
 
-**Phase 1 — Build from scratch:** Tokenizer, attention mechanism, multi-head attention, feed-forward network, transformer block, full GPT model. Every component explained in the [DEVLOG](DEVLOG.md).
+Modern beats vanilla with fewer params. RoPE was the star — biggest single improvement.
 
-**Phase 2 — Modernize one swap at a time:** Replace LayerNorm, ReLU, learned positions, and naive inference with RMSNorm, SwiGLU, RoPE, and KV cache. Each swap tested in isolation so you can see exactly what it does.
+## 9 things that went wrong
 
-**Phase 3 — Scale up:** BPE tokenization (50K vocab), mixed precision, gradient accumulation. Learned why BPE needs way more data than 1MB of Shakespeare.
+Building this was not smooth. Every failure is documented in the [DEVLOG](https://github.com/brianmeyer/tinyllm/blob/main/DEVLOG.md):
 
-**Phase 4 — Break everything:** MPS memory leaks, silent process kills, float16 divergence, RoPE position bugs, Colab runtime evictions, lost checkpoints. Each failure documented with root cause and fix.
+1. MPS training died silently (memory leak)
+2. Bundled all 4 architecture swaps together instead of testing one at a time
+3. Python stdout buffering hid training progress
+4. RoPE position bug in KV cache made the model generate garbage
+5. Modern model memorized Shakespeare (overfitting on 1MB)
+6. Float16 diverged on MPS with 50K BPE vocab
+7. MPS kept killing every retrain attempt
+8. Lost all Colab checkpoints when runtime disconnected
+9. Ran out of free Colab GPU quota
 
-## What I learned
+## Training details
 
-1. **RoPE is the most impactful modern change** — 0.31 better loss, fewer params, beautiful math
-2. **More powerful models overfit faster on small data** — early stopping is essential
-3. **MPS (Apple Silicon) silently kills training** after 60-80 min due to memory leaks
-4. **When loss is good but output is garbage, the bug is in inference** — our RoPE position bug only appeared during KV cache generation
-5. **Change one thing at a time** — the per-swap comparison approach is how real ML research works
-6. **Always save checkpoints to persistent storage** — we lost 3 hours of Colab training to a runtime disconnect
+| | |
+|---|---|
+| Dataset | Tiny Shakespeare (~1.1MB, 65 unique characters) |
+| Optimizer | AdamW, lr=3e-4 |
+| Batch size | 64, block size 256 |
+| Steps | 5,000 (best checkpoint via early stopping) |
+| Hardware | Google Colab T4 (and an M4 Mac that kept crashing) |
+| Dropout | 0.3 (increased from 0.2 to fight overfitting) |
 
-## Architecture
+## How to use
 
-```
-ModernGPT (10.6M params)
-  token_emb:   Embedding(65, 384)
-  blocks × 6:
-    RMSNorm → MultiHeadAttention(6 heads, RoPE, KV cache) → residual
-    RMSNorm → SwiGLU(384 → 1024 → 384) → residual
-  RMSNorm → lm_head (tied with token_emb)
+```python
+import torch
+import sys
+sys.path.append('src')
+from model_modern import ModernGPT
+from tokenizer import encode, decode
+
+device = "cuda" if torch.cuda.is_available() else "cpu"
+ckpt = torch.load("model.pt", map_location=device, weights_only=False)
+model = ModernGPT(**ckpt["config"]).to(device)
+model.load_state_dict(ckpt["model_state"])
+model.eval()
+
+idx = torch.tensor([encode("ROMEO:")], dtype=torch.long, device=device)
+out = model.generate(idx, max_new_tokens=200, temperature=0.8)
+print(decode(out[0].tolist()))
 ```
 
-## 9 things that went wrong
-
-| # | What happened | Root cause |
-|---|--------------|-----------|
-| 1 | MPS training died silently | Memory leak in PyTorch MPS backend |
-| 2 | Bundled all 4 swaps together | Rushing — should test one at a time |
-| 3 | Python output hidden during training | stdout buffering — use `python -u` |
-| 4 | Modern model generated garbage | RoPE position bug in KV cache inference |
-| 5 | Modern model memorized Shakespeare | 10M params too powerful for 1MB data |
-| 6 | BPE training diverged | float16 on MPS overflows with 50K vocab |
-| 7 | MPS kept killing all retrains | Memory leak unfixable on 16GB |
-| 8 | Lost all Colab checkpoints | Runtime disconnected — ephemeral storage |
-| 9 | Colab GPU quota exhausted | Used all free T4 hours in one session |
+## What I learned
 
-Full analysis of each: [DEVLOG.md](DEVLOG.md)
+1. **RoPE is the most impactful modern architecture change** — beautiful math, fewer params, better results
+2. **More powerful models overfit faster on small data** — early stopping is essential
+3. **When loss is good but output is garbage, the bug is in inference code** — not the model
+4. **MPS is not ready for serious training** — use CUDA
+5. **Always save checkpoints to persistent storage** — Colab runtimes are ephemeral
+6. **Change one thing at a time and measure** — this is how real ML research works
 
 ## References
 
 - [build-nanogpt](https://github.com/karpathy/build-nanogpt) — Karpathy's step-by-step GPT build
-- [nanochat](https://github.com/karpathy/nanochat) — nanoGPT successor
 - [RoPE paper](https://arxiv.org/abs/2104.09864) — Su et al.
 - [SwiGLU paper](https://arxiv.org/abs/2002.05202) — Shazeer
+- [RMSNorm paper](https://arxiv.org/abs/1910.07467) — Zhang & Sennrich
 
 ## License