metadata
license: mit
tags:
- transformer
- interpretability
- from-scratch
library_name: pytorch
tiny-transformer
Trained runs for tiny-transformer: a readable, hookable GPT-style decoder-only transformer built from scratch in PyTorch, trained here on its demo task, sorting lists of 16 values. The code, the full write-up and the figures are on GitHub.
Each folder is one run: evaluation weights (model.safetensors), the architecture and vocabulary (model_config.json), the full training record (metrics.json: config, seeds, data definition, code state, environment, history), a diagnostic plot, and the final analysis under analysis/.
| Run | What it is | Lists sorted exactly (test, 2,048 lists) |
|---|---|---|
default-seed0 |
4 layers, 3.2M parameters | 100% |
default-seed1 |
same, seed 1 | 100% |
default-seed2 |
same, seed 2 | 99.8% |
bigram-seed0 |
baseline: no layers (--model.n-layers 0), sees only the previous token |
0% |
To load one, with the tiny-transformer package installed:
from pathlib import Path
from huggingface_hub import snapshot_download
from tiny_transformer.hub import load_model
folder = Path(snapshot_download('coeiico/tiny-transformer', allow_patterns='default-seed0/*')) / 'default-seed0'
model, vocab = load_model(folder)
These are evaluation weights, not resumable checkpoints. The runs were trained on CUDA (an RTX 3060); a seed fixes a run on one kind of device.