---
license: mit
tags:
  - transformer
  - interpretability
  - from-scratch
library_name: pytorch
---

# tiny-transformer

Trained runs for [tiny-transformer](https://github.com/coeiico/tiny-transformer): a readable, hookable GPT-style decoder-only transformer built from scratch in PyTorch, trained here on its demo task, sorting lists of 16 values. The code, the full write-up and the figures are on GitHub.

Each folder is one run: evaluation weights (`model.safetensors`), the architecture and vocabulary (`model_config.json`), the full training record (`metrics.json`: config, seeds, data definition, code state, environment, history), a diagnostic plot, and the final analysis under `analysis/`.

| Run | What it is | Lists sorted exactly (test, 2,048 lists) |
| --- | --- | --- |
| `default-seed0` | 4 layers, 3.2M parameters | 100% |
| `default-seed1` | same, seed 1 | 100% |
| `default-seed2` | same, seed 2 | 99.8% |
| `bigram-seed0` | baseline: no layers (`--model.n-layers 0`), sees only the previous token | 0% |

To load one, with the [tiny-transformer](https://github.com/coeiico/tiny-transformer) package installed:

```python
from pathlib import Path
from huggingface_hub import snapshot_download
from tiny_transformer.hub import load_model

folder = Path(snapshot_download('coeiico/tiny-transformer', allow_patterns='default-seed0/*')) / 'default-seed0'
model, vocab = load_model(folder)
```

These are evaluation weights, not resumable checkpoints. The runs were trained on CUDA (an RTX 3060); a seed fixes a run on one kind of device.