| --- |
| license: mit |
| tags: |
| - transformer |
| - interpretability |
| - from-scratch |
| library_name: pytorch |
| --- |
| |
| # tiny-transformer |
|
|
| Trained runs for [tiny-transformer](https://github.com/coeiico/tiny-transformer): a readable, hookable GPT-style decoder-only transformer built from scratch in PyTorch, trained here on its demo task, sorting lists of 16 values. The code, the full write-up and the figures are on GitHub. |
|
|
| Each folder is one run: evaluation weights (`model.safetensors`), the architecture and vocabulary (`model_config.json`), the full training record (`metrics.json`: config, seeds, data definition, code state, environment, history), a diagnostic plot, and the final analysis under `analysis/`. |
|
|
| | Run | What it is | Lists sorted exactly (test, 2,048 lists) | |
| | --- | --- | --- | |
| | `default-seed0` | 4 layers, 3.2M parameters | 100% | |
| | `default-seed1` | same, seed 1 | 100% | |
| | `default-seed2` | same, seed 2 | 99.8% | |
| | `bigram-seed0` | baseline: no layers (`--model.n-layers 0`), sees only the previous token | 0% | |
|
|
| To load one, with the [tiny-transformer](https://github.com/coeiico/tiny-transformer) package installed: |
|
|
| ```python |
| from pathlib import Path |
| from huggingface_hub import snapshot_download |
| from tiny_transformer.hub import load_model |
| |
| folder = Path(snapshot_download('coeiico/tiny-transformer', allow_patterns='default-seed0/*')) / 'default-seed0' |
| model, vocab = load_model(folder) |
| ``` |
|
|
| These are evaluation weights, not resumable checkpoints. The runs were trained on CUDA (an RTX 3060); a seed fixes a run on one kind of device. |
|
|
