spidory / README.md
Davidow's picture
Paper models: 10 variants x 3 seeds, fp32 safetensors
038a279 verified
|
Raw History Blame Contribute Delete
3.79 kB
---
license: apache-2.0
library_name: pytorch
tags:
- looped-transformer
- latent-recurrence
- mixture-of-experts
- ngpt
---
# Spidory: small looped nGPT models with experts
Weights of the 30 models in Table 2 of the paper draft
*Choose Again Each Loop: Expert Re-selection and the One-Loop Tax in a Small Looped Transformer*
(David Ternoposkiy, 2026). Code, paper and training scripts: https://github.com/Davidooff/Spidory
The models follow NVIDIA's Dory recipe at 26M parameters: a 4-layer input block, a 10-layer shared
centre block run `k` times (`s0 = x`, `u = N(s + x)`, `s = N(F(u))`) and a 4-layer output block, with
nGPT layers, SwiGLU and RoPE. Some variants add four small experts to the centre block, a per-loop
run index, or a block between loops. Each was trained on 1.2B tokens, on seeds 0, 1 and 2.
These are research checkpoints for reproducing the paper, not general-purpose language models.
They were trained on TinyStories, generated arithmetic, short QA and a few small maths sets, with a
4,096-token vocabulary and a 256-token context.
## Models
Test loss is the mean over 7 test sets. Arithmetic is exact-match accuracy on 500 held-out
problems. Looped models are scored at `k = 3`, no-loop models at `k = 1`. Mean ± standard
deviation over the 3 seeds; per-seed numbers are in each folder's `results.json`.
| Model | Folders | Params | Test loss | Arithmetic (%) |
|---|---|---|---|---|
| No-loop, 10 layers (k=1) | `noloop_s0` `noloop_s1` `noloop_s2` | 26.1M | 1.710 ± 0.002 | 65.5 ± 7.3 |
| No-loop, 30 layers (k=1) | `noloop_deep_s0` `noloop_deep_s1` `noloop_deep_s2` | 53.6M | 1.694 ± 0.010 | 70.6 ± 4.8 |
| Plain | `plain_s0` `plain_s1` `plain_s2` | 26.1M | 1.708 ± 0.006 | 65.6 ± 4.6 |
| Plain + run index | `plain_signal_s0` `plain_signal_s1` `plain_signal_s2` | 26.1M | 1.708 ± 0.009 | 66.7 ± 5.6 |
| Plain + run index, no norm | `plain_signal_nonorm_s0` `plain_signal_nonorm_s1` `plain_signal_nonorm_s2` | 26.1M | 1.711 ± 0.004 | 69.3 ± 4.1 |
| Per-sequence experts, re-selected each loop | `perloop_s0` `perloop_s1` `perloop_s2` | 26.6M | 1.768 ± 0.035 | 66.6 ± 10.6 |
| Per-token experts, re-selected each loop | `token_s0` `token_s1` `token_s2` | 26.6M | 1.716 ± 0.005 | 69.9 ± 4.0 |
| Per-token experts + run index, re-selected each loop | `token_signal_s0` `token_signal_s1` `token_signal_s2` | 26.6M | 1.711 ± 0.013 | 71.3 ± 10.0 |
| Per-token experts + run index, chosen once | `token_signal_once_s0` `token_signal_once_s1` `token_signal_once_s2` | 26.6M | 1.711 ± 0.012 | 59.7 ± 8.4 |
| Between-loop block + run index | `between_signal_s0` `between_signal_s1` `between_signal_s2` | 34.9M | 1.752 ± 0.012 | 64.1 ± 9.5 |
## Files
- `<model>_s<seed>/model.safetensors`: fp32 weights, no optimizer state.
- `<model>_s<seed>/config.json`: model and training settings, and the final step.
- `<model>_s<seed>/results.json`: every test score at `k = 1..4`, as reported in the paper.
- `tokenizer.json`: the BPE tokenizer (4,096 tokens) shared by all models.
- `manifest.json`: parameters, test loss and the SHA-256 of the training checkpoint each export
came from.
Each export was checked against its training checkpoint: every tensor is equal, and logits at
`k = 1` and `k = 3` match exactly.
## Use
```bash
pip install git+https://github.com/Davidooff/Spidory
```
```python
from spidory.evaluate import generate_text
from spidory.hub import load
model, tok = load("token_signal_s0")
print(generate_text(model, tok, "Q: Calculate 12 + 7 * 3.\nA:", 8, k=3))
```
`k` is the number of passes through the centre block (1 to 4). `model.infer(ids, k=3)` returns the
logits, and for models with experts also the expert and run count each token or sequence used.
## License
Apache-2.0, for both the weights and the code.