--- license: apache-2.0 library_name: pytorch tags: - looped-transformer - latent-recurrence - mixture-of-experts - ngpt --- # Spidory: small looped nGPT models with experts Weights of the 30 models in Table 2 of the paper draft *Choose Again Each Loop: Expert Re-selection and the One-Loop Tax in a Small Looped Transformer* (David Ternoposkiy, 2026). Code, paper and training scripts: https://github.com/Davidooff/Spidory The models follow NVIDIA's Dory recipe at 26M parameters: a 4-layer input block, a 10-layer shared centre block run `k` times (`s0 = x`, `u = N(s + x)`, `s = N(F(u))`) and a 4-layer output block, with nGPT layers, SwiGLU and RoPE. Some variants add four small experts to the centre block, a per-loop run index, or a block between loops. Each was trained on 1.2B tokens, on seeds 0, 1 and 2. These are research checkpoints for reproducing the paper, not general-purpose language models. They were trained on TinyStories, generated arithmetic, short QA and a few small maths sets, with a 4,096-token vocabulary and a 256-token context. ## Models Test loss is the mean over 7 test sets. Arithmetic is exact-match accuracy on 500 held-out problems. Looped models are scored at `k = 3`, no-loop models at `k = 1`. Mean ± standard deviation over the 3 seeds; per-seed numbers are in each folder's `results.json`. | Model | Folders | Params | Test loss | Arithmetic (%) | |---|---|---|---|---| | No-loop, 10 layers (k=1) | `noloop_s0` `noloop_s1` `noloop_s2` | 26.1M | 1.710 ± 0.002 | 65.5 ± 7.3 | | No-loop, 30 layers (k=1) | `noloop_deep_s0` `noloop_deep_s1` `noloop_deep_s2` | 53.6M | 1.694 ± 0.010 | 70.6 ± 4.8 | | Plain | `plain_s0` `plain_s1` `plain_s2` | 26.1M | 1.708 ± 0.006 | 65.6 ± 4.6 | | Plain + run index | `plain_signal_s0` `plain_signal_s1` `plain_signal_s2` | 26.1M | 1.708 ± 0.009 | 66.7 ± 5.6 | | Plain + run index, no norm | `plain_signal_nonorm_s0` `plain_signal_nonorm_s1` `plain_signal_nonorm_s2` | 26.1M | 1.711 ± 0.004 | 69.3 ± 4.1 | | Per-sequence experts, re-selected each loop | `perloop_s0` `perloop_s1` `perloop_s2` | 26.6M | 1.768 ± 0.035 | 66.6 ± 10.6 | | Per-token experts, re-selected each loop | `token_s0` `token_s1` `token_s2` | 26.6M | 1.716 ± 0.005 | 69.9 ± 4.0 | | Per-token experts + run index, re-selected each loop | `token_signal_s0` `token_signal_s1` `token_signal_s2` | 26.6M | 1.711 ± 0.013 | 71.3 ± 10.0 | | Per-token experts + run index, chosen once | `token_signal_once_s0` `token_signal_once_s1` `token_signal_once_s2` | 26.6M | 1.711 ± 0.012 | 59.7 ± 8.4 | | Between-loop block + run index | `between_signal_s0` `between_signal_s1` `between_signal_s2` | 34.9M | 1.752 ± 0.012 | 64.1 ± 9.5 | ## Files - `_s/model.safetensors`: fp32 weights, no optimizer state. - `_s/config.json`: model and training settings, and the final step. - `_s/results.json`: every test score at `k = 1..4`, as reported in the paper. - `tokenizer.json`: the BPE tokenizer (4,096 tokens) shared by all models. - `manifest.json`: parameters, test loss and the SHA-256 of the training checkpoint each export came from. Each export was checked against its training checkpoint: every tensor is equal, and logits at `k = 1` and `k = 3` match exactly. ## Use ```bash pip install git+https://github.com/Davidooff/Spidory ``` ```python from spidory.evaluate import generate_text from spidory.hub import load model, tok = load("token_signal_s0") print(generate_text(model, tok, "Q: Calculate 12 + 7 * 3.\nA:", 8, k=3)) ``` `k` is the number of passes through the centre block (1 to 4). `model.infer(ids, k=3)` returns the logits, and for models with experts also the expert and run count each token or sequence used. ## License Apache-2.0, for both the weights and the code.