# nanoGPT: Step-by-Step Tutorial

This tutorial walks through building and training a **tiny GPT from scratch** in pure PyTorch. No `transformers` library, no pre-trained weights — just ~200 lines of clean code.

---

## Table of Contents

1. [Overview](#1-overview)
2. [Dataset Preparation](#2-dataset-preparation)
3. [Model Architecture](#3-model-architecture)
4. [Training Loop](#4-training-loop)
5. [Generation / Inference](#5-generation--inference)
6. [Results](#6-results)
7. [Files in this Repo](#7-files-in-this-repo)

---

## 1. Overview

We train a **character-level** language model on [tiny Shakespeare](https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt) (~1.1M characters, 65 unique characters).

The model learns to predict the next character given all previous characters, autoregressively. This is exactly how GPT-2, GPT-3, and ChatGPT work — just at character scale instead of word/BPE token scale.

**Model size**: ~10.8M parameters  
**Architecture**: 6 layers, 6 heads, 384 embedding dim, 256 context length

---

## 2. Dataset Preparation (`prepare.py`)

### What happens:
1. **Download** tiny Shakespeare text
2. **Discover vocabulary**: find all unique characters → 65 chars
3. **Build mappings**:
   - `stoi` (string-to-int): `'a' → 0`, `'b' → 1`, ...
   - `itos` (int-to-string): reverse lookup
4. **Encode** the entire text as integers
5. **Split** 90% train / 10% validation
6. **Save** as `data.pt` (PyTorch tensors for fast loading)

### Key concept: Character-level tokenization
```python
chars = sorted(list(set(text)))          # vocabulary
vocab_size = len(chars)                  # 65
encode = lambda s: [stoi[c] for c in s]  # "hello" -> [46, 43, 50, 50, 53]
decode = lambda l: "".join([itos[i] for i in l])
```

No tokenizer library needed! For English text, ~65 chars is enough.

---

## 3. Model Architecture (`model.py`)

### 3.1 Configuration (`GPTConfig`)
```python
@dataclass
class GPTConfig:
    block_size: int = 256    # max sequence length
    vocab_size: int = 65     # number of unique characters
    n_layer: int = 6         # transformer blocks
    n_head: int = 6          # attention heads per block
    n_embd: int = 384        # embedding dimension
```

### 3.2 Causal Self-Attention
The core idea: every token can "look at" all previous tokens to decide what comes next.

```
For each token:
  Query = "What am I looking for?"
  Key   = "What do I contain?"
  Value = "What information do I have?"

Attention score = Query · Key  (scaled)
Causal mask     = prevent looking at future tokens
Output          = weighted sum of Values
```

We use **multi-head attention**: split embeddings into 6 parallel attention operations (heads), run them simultaneously, then concatenate.

**Code flow:**
```
Input (B, T, C)
  → c_attn → (Q, K, V) each (B, T, C)
  → reshape to (B, n_head, T, head_size)
  → Q @ K.T → attention scores (B, n_head, T, T)
  → causal mask → softmax → weighted sum of V
  → reshape back → c_proj → Output (B, T, C)
```

### 3.3 MLP (Feed-Forward)
After attention, each token gets a private "thinking step":
```
(B, T, C) → Linear(4*C) → GELU → Linear(C) → (B, T, C)
```
The 4× expansion is standard in transformers.

### 3.4 Transformer Block
```
x = x + Attention(LayerNorm(x))   # pre-norm residual
x = x + MLP(LayerNorm(x))         # pre-norm residual
```
**Pre-LayerNorm** (normalize before sublayer) is used by GPT-2/3/Llama.

### 3.5 Full GPT Model
```
1. Token Embedding  (wte): char index → vector
2. Position Embedding (wpe): position index → vector
3. Sum them: x = wte + wpe
4. Pass through N transformer blocks
5. Final LayerNorm
6. Language Model Head: project to vocab_size logits
7. Cross-entropy loss against next-character targets
```

**Weight tying**: `wte` (input embedding) shares weights with `lm_head` (output projection). Saves parameters, improves training.

---

## 4. Training Loop (`train.py` / `train_standalone.py`)

### 4.1 Batch sampling
For each training step, grab random contiguous chunks:
```python
def get_batch(split):
    ix = torch.randint(len(data) - BLOCK_SIZE, (BATCH_SIZE,))
    x = data[ix : ix+BLOCK_SIZE]      # input
    y = data[ix+1 : ix+BLOCK_SIZE+1]  # target (shifted by 1)
```

### 4.2 Learning rate schedule
**Cosine with linear warmup**:
```
Step 0-200:   LR ramps up from 0 → 1e-3   (warmup)
Step 200-5000: LR decays cosine to 1e-4   (cosine annealing)
```
Warmup prevents early loss spikes when gradients are large.

### 4.3 Optimizer
**AdamW** with separated weight decay:
- 2D parameters (weights) → weight_decay = 0.1
- 1D parameters (biases, LayerNorm) → weight_decay = 0.0

### 4.4 Gradient clipping
`torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)` prevents exploding gradients.

### 4.5 Evaluation
Every 500 steps, we evaluate on 200 random validation batches and report:
```
step  500 | train loss 1.8234 | val loss 1.9012 | lr 9.12e-04 | time 45.2s
```
The best validation checkpoint is saved as `best.pt`.

---

## 5. Generation / Inference (`generate.py`)

Autoregressive generation:
```
1. Encode a prompt (e.g., "\nROMEO:")
2. Run forward pass → get logits for last token
3. Apply temperature + top-k sampling → probability distribution
4. Sample next token from distribution
5. Append token to sequence
6. Repeat from step 2
```

**Temperature**: lower = more conservative/deterministic, higher = more random/creative  
**Top-k**: only sample from the k most likely tokens (prevents gibberish)

---

## 6. Results

Expected after 5000 steps on T4 GPU (~30-60 minutes):

| Metric | Value |
|--------|-------|
| Initial loss | ~4.3 (random guessing among 65 chars) |
| Final train loss | ~1.2–1.5 |
| Final val loss | ~1.3–1.6 |
| Parameters | 10.77 M |

**Generated sample** (should look vaguely Shakespeare-like):
```
ROMEO:
What say you, then? I have heard you say
The hour is come, and I must hence depart.
```

---

## 7. Files in this Repo

| File | Purpose |
|------|---------|
| `model.py` | Pure PyTorch GPT architecture (standalone) |
| `prepare.py` | Downloads data, builds char-level vocab, saves `data.pt` |
| `train.py` | Training script (imports from `model.py`) |
| `train_standalone.py` | Self-contained training script (model + train in one file) |
| `generate.py` | Inference script — load checkpoint and generate text |
| `input.txt` | Raw tiny Shakespeare text |
| `data.pt` | Preprocessed train/val tensors + vocab mappings |
| `best.pt` | Best model checkpoint (saved during training) |

---

## How to Run

```bash
# 1. Prepare data
python prepare.py

# 2. Train (GPU recommended)
python train_standalone.py

# 3. Generate
python generate.py --prompt "ROMEO:" --length 500 --temperature 0.8
```

---

## Learning Checklist

- [ ] Read `model.py` — understand attention masking, pre-norm, weight tying
- [ ] Read `prepare.py` — understand character-level tokenization
- [ ] Read `train.py` — understand batching, LR schedule, gradient clipping
- [ ] Run training and watch loss go down
- [ ] Tweak hyperparameters (n_layer, n_embd, learning rate) and observe changes
- [ ] Generate with different temperatures and top-k values

---

Based on Andrej Karpathy's [build-nanogpt](https://github.com/karpathy/build-nanogpt) and [nanoGPT](https://github.com/karpathy/nanoGPT).