README.md

---
license: mit
language:
  - en
library_name: pytorch
tags:
  - gpt
  - from-scratch
  - nanogpt
  - language-model
  - scaling-laws
  - educational
datasets:
  - Skylion007/openwebtext
pipeline_tag: text-generation
---

# slm-from-scratch-phase3-scaling

Three GPT-style language models, hand-written and trained from scratch in PyTorch on a single RTX 5070 Ti, produced by a controlled scaling-law experiment: same architecture family, varying only model size and tokens seen. No HuggingFace `transformers` model classes were used — every component (attention, MLP, blocks, embeddings, training loop) was hand-coded.

This is Phase 3 of a larger build documented at [github.com/Ishaanred/slm-from-scratch](https://github.com/Ishaanred/slm-from-scratch). See [`docs/phase3/results.md`](https://github.com/Ishaanred/slm-from-scratch/blob/main/docs/phase3/results.md) for the full scaling-law writeup and plots. These are portfolio and learning artifacts, not production models.

## The three checkpoints

| Checkpoint | Params | Tokens | Tokens/param | Best val loss |
|---|---|---|---|---|
| `75m-500m-tokens/` | 69.4M | 500M | ~7.2x | **4.0773** |
| `75m-2b-tokens/` | 69.4M | 2B | ~28.8x | **3.8496** |
| `150m-500m-tokens/` | 135.0M | 500M | ~3.7x | **3.7378** |

Each subfolder is independently loadable: `model.safetensors` + `config.json`. All three share the same `model.py` and tokenizer (this repo's root).

Note: these are the *best* validation-loss checkpoints saved during each run (the lowest point reached), not necessarily the literal last training step — the scaling-law docs report final-iteration loss instead, for a compute-matched comparison across runs. The two numbers differ slightly because validation loss is noisy near the end of a cosine-decay schedule.

## What they're for

- **75m-500m-tokens vs 75m-2b-tokens** — same model size, 4x the tokens. Tests whether more data helps at fixed size.
- **75m-500m-tokens vs 150m-500m-tokens** — same tokens, ~1.9x the parameters. Tests whether more parameters help at fixed, scarce data.

## Architecture

| | |
|---|---|
| Type | Decoder-only GPT (pre-LayerNorm) |
| Attention | Flash Attention via PyTorch SDPA |
| Context length | 1024 |
| Vocabulary | Custom 32K BPE tokenizer (trained on this project's own filtered OpenWebText corpus, not GPT-2's) |
| Precision | weights stored fp32, trained in bf16 |

## Training

| | |
|---|---|
| Data | ~9.14B-token OpenWebText corpus, filtered (language-ID, quality heuristics, exact dedup) and re-tokenized with the project's own 32K BPE tokenizer |
| Optimizer | AdamW, cosine LR with warmup, weight decay on 2D params |
| Hardware | 1x RTX 5070 Ti (16GB) |

## Honest assessment

These are small models trained on a token budget far below Chinchilla-optimal for their size (compute-optimal would be roughly 20x tokens/param; the largest run here reaches ~29x on the small model, but only ~3.7x on the 150M model). They generate grammatical English with local coherence but no long-range consistency, and score near the random-chance floor on standard benchmarks (see [`docs/phase5/evaluation.html`](https://github.com/Ishaanred/slm-from-scratch/blob/main/docs/phase5/evaluation.html)) — expected at this scale, not a bug.

## Files

- `<checkpoint-name>/model.safetensors` — model weights, fp32, optimizer state stripped
- `<checkpoint-name>/config.json` — architecture and training metadata for that checkpoint
- `model.py` — the from-scratch model definition needed to load any of these weights
- `tokenizer/` — the project's own 32K BPE tokenizer (`tokenizer.json`, `vocab.json`, `merges.txt`)

## Usage

This is not a `transformers` architecture, so load it with the included `model.py`.

```python
import json, torch
from safetensors.torch import load_file
from model import GPT, GPTConfig
from tokenizers import Tokenizer

CHECKPOINT = "75m-2b-tokens"  # or "75m-500m-tokens" / "150m-500m-tokens"

cfg = json.load(open(f"{CHECKPOINT}/config.json"))
model = GPT(GPTConfig(
    n_layer=cfg["n_layer"], n_head=cfg["n_head"], n_embd=cfg["n_embd"],
    block_size=cfg["block_size"], vocab_size=cfg["vocab_size"], dropout=0.0,
))
model.load_state_dict(load_file(f"{CHECKPOINT}/model.safetensors"))
model.eval()

tok = Tokenizer.from_file("tokenizer/tokenizer.json")
ids = torch.tensor([tok.encode("The meaning of life is").ids])

with torch.no_grad():
    for _ in range(50):
        logits, _ = model(ids[:, -cfg["block_size"]:])
        nxt = torch.softmax(logits[:, -1, :] / 0.8, dim=-1).multinomial(1)
        ids = torch.cat([ids, nxt], dim=1)
print(tok.decode(ids[0].tolist()))
```

## Limitations

- Small and undertrained relative to Chinchilla-optimal, especially the 150M checkpoint.
- Trained on OpenWebText, so it inherits the biases and noise of unfiltered web text scraped from Reddit-linked URLs.
- No instruction tuning, no safety alignment. Raw next-token predictors.

## License

MIT. Built as an educational project.