deja-vu-tfm

Trained runs for deja-vu-tfm: induction heads forming as a phase change in a 2-layer attention-only transformer, and the heads that do the copying. The code, the full write-up and the figures are on GitHub.

Each folder is one run: evaluation weights (model.safetensors), the architecture and vocabulary (model_config.json), the full training record (metrics.json: config, seeds, data definition, code state, environment, history), a diagnostic plot, and the final analysis under analysis/final/.

Run	What it is	Copy accuracy (test)
default-seed0	2 layers, synthetic repeats; induction heads fed by a previous-token head (K-composition)	92.2%
default-seed1	same, seed 1; fed by duplicate-token heads (Q-composition)	91.8%
default-seed2	same, seed 2; fed by a duplicate-token head (Q-composition)	90.5%
one-layer-seed0	baseline: one layer, cannot form an induction head	11.8%
gpt2-init-seed0	baseline: two layers with GPT-2's init (std 0.02); no induction heads	16.6%
stories-seed0	2 layers on TinyStories, 330M tokens; a previous-token head, no induction head yet	1.7% (probe)

The models are tiny-transformer instances. To load one:

from pathlib import Path
from huggingface_hub import snapshot_download
from tiny_transformer.hub import load_model

folder = Path(snapshot_download('coeiico/deja-vu-tfm', allow_patterns='default-seed0/*')) / 'default-seed0'
model, vocab = load_model(folder)

These are evaluation weights, not resumable checkpoints. Seeds fix a run on one kind of device; the runs here were trained on CUDA (an RTX 3060).