| # nanoGPT Tutorial |
|
|
| A step-by-step implementation of a tiny GPT model from scratch in pure PyTorch. |
|
|
| ## What is this? |
|
|
| This repository contains a complete, tutorial-style implementation of a small GPT (Generative Pre-trained Transformer) trained on tiny Shakespeare. Every line of code is commented to explain **what** it does and **why**. |
|
|
| ## Files |
|
|
| | File | Purpose | |
| |------|---------| |
| | `model.py` | The full GPT architecture: CausalSelfAttention, MLP, Block, GPT | |
| | `prepare.py` | Data preparation: character-level tokenization, train/val split | |
| | `train.py` | Training loop with AdamW, cosine LR schedule, and generation | |
| | `input.txt` | The tiny Shakespeare dataset (~1.1M characters, 65 unique chars) | |
| | `data.pt` | Preprocessed tensors (generated by `prepare.py`) | |
| | `best.pt` | Best model checkpoint (generated by `train.py`) | |
|
|
| ## Model Architecture |
|
|
| ``` |
| GPT( |
| wte (Embedding): vocab_size -> n_embd (token embeddings) |
| wpe (Embedding): block_size -> n_embd (position embeddings) |
| h (6x Block): |
| ln_1 (LayerNorm) |
| attn (CausalSelfAttention: multi-head self-attention with causal mask) |
| ln_2 (LayerNorm) |
| mlp (MLP: expand 4x -> GELU -> project back) |
| ln_f (LayerNorm) |
| lm_head (Linear): n_embd -> vocab_size (next-token prediction) |
| ) |
| ``` |
|
|
| **Key design choices:** |
| - **Character-level vocabulary** — no tokenizer library needed |
| - **Pre-LayerNorm** residuals — standard in modern transformers |
| - **Weight tying** — shared weights between input embedding and output projection |
| - **Causal (autoregressive) attention** — can only attend to past tokens |
|
|
| ## How to Run |
|
|
| ```bash |
| # 1. Prepare data |
| python prepare.py |
| |
| # 2. Train (requires GPU for speed, CPU works too) |
| python train.py |
| |
| # 3. The model will print generated Shakespeare-style text at the end! |
| ``` |
|
|
| ## Training Details |
|
|
| | Hyperparameter | Value | |
| |---------------|-------| |
| | Layers | 6 | |
| | Heads | 6 | |
| | Embedding dim | 384 | |
| | Context length | 256 | |
| | Batch size | 64 | |
| | Training steps | 5,000 | |
| | Optimizer | AdamW (β₁=0.9, β₂=0.95) | |
| | Learning rate | 1e-3 (cosine decay to 1e-4) | |
| | Warmup | 200 steps | |
| | Gradient clip | 1.0 | |
|
|
| ## Acknowledgments |
|
|
| Based on Andrej Karpathy's legendary [build-nanogpt](https://github.com/karpathy/build-nanogpt) and [nanoGPT](https://github.com/karpathy/nanoGPT) repositories. |
|
|
