File size: 1,556 Bytes
4a570eb | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 | ---
license: mit
tags:
- transformer
- interpretability
- from-scratch
library_name: pytorch
---
# tiny-transformer
Trained runs for [tiny-transformer](https://github.com/coeiico/tiny-transformer): a readable, hookable GPT-style decoder-only transformer built from scratch in PyTorch, trained here on its demo task, sorting lists of 16 values. The code, the full write-up and the figures are on GitHub.
Each folder is one run: evaluation weights (`model.safetensors`), the architecture and vocabulary (`model_config.json`), the full training record (`metrics.json`: config, seeds, data definition, code state, environment, history), a diagnostic plot, and the final analysis under `analysis/`.
| Run | What it is | Lists sorted exactly (test, 2,048 lists) |
| --- | --- | --- |
| `default-seed0` | 4 layers, 3.2M parameters | 100% |
| `default-seed1` | same, seed 1 | 100% |
| `default-seed2` | same, seed 2 | 99.8% |
| `bigram-seed0` | baseline: no layers (`--model.n-layers 0`), sees only the previous token | 0% |
To load one, with the [tiny-transformer](https://github.com/coeiico/tiny-transformer) package installed:
```python
from pathlib import Path
from huggingface_hub import snapshot_download
from tiny_transformer.hub import load_model
folder = Path(snapshot_download('coeiico/tiny-transformer', allow_patterns='default-seed0/*')) / 'default-seed0'
model, vocab = load_model(folder)
```
These are evaluation weights, not resumable checkpoints. The runs were trained on CUDA (an RTX 3060); a seed fixes a run on one kind of device.
|