tiny-transformer

Trained runs for tiny-transformer: a readable, hookable GPT-style decoder-only transformer built from scratch in PyTorch, trained here on its demo task, sorting lists of 16 values. The code, the full write-up and the figures are on GitHub.

Each folder is one run: evaluation weights (model.safetensors), the architecture and vocabulary (model_config.json), the full training record (metrics.json: config, seeds, data definition, code state, environment, history), a diagnostic plot, and the final analysis under analysis/.

Run	What it is	Lists sorted exactly (test, 2,048 lists)
default-seed0	4 layers, 3.2M parameters	100%
default-seed1	same, seed 1	100%
default-seed2	same, seed 2	99.8%
bigram-seed0	baseline: no layers (--model.n-layers 0), sees only the previous token	0%

To load one, with the tiny-transformer package installed:

from pathlib import Path
from huggingface_hub import snapshot_download
from tiny_transformer.hub import load_model

folder = Path(snapshot_download('coeiico/tiny-transformer', allow_patterns='default-seed0/*')) / 'default-seed0'
model, vocab = load_model(folder)

These are evaluation weights, not resumable checkpoints. The runs were trained on CUDA (an RTX 3060); a seed fixes a run on one kind of device.