|
Download README.md from Davidow/spidory: direct link, hf CLI and curl.
- Browser
- Download file 3.79 kB
-
https://huggingface.co/Davidow/spidory/resolve/main/README.md
- Command line
-
hf download hf://Davidow/spidory/README.md
-
curl -L -o README.md https://huggingface.co/Davidow/spidory/resolve/main/README.md
3.79 kB
| license: apache-2.0 | |
| library_name: pytorch | |
| tags: | |
| - looped-transformer | |
| - latent-recurrence | |
| - mixture-of-experts | |
| - ngpt | |
| # Spidory: small looped nGPT models with experts | |
| Weights of the 30 models in Table 2 of the paper draft | |
| *Choose Again Each Loop: Expert Re-selection and the One-Loop Tax in a Small Looped Transformer* | |
| (David Ternoposkiy, 2026). Code, paper and training scripts: https://github.com/Davidooff/Spidory | |
| The models follow NVIDIA's Dory recipe at 26M parameters: a 4-layer input block, a 10-layer shared | |
| centre block run `k` times (`s0 = x`, `u = N(s + x)`, `s = N(F(u))`) and a 4-layer output block, with | |
| nGPT layers, SwiGLU and RoPE. Some variants add four small experts to the centre block, a per-loop | |
| run index, or a block between loops. Each was trained on 1.2B tokens, on seeds 0, 1 and 2. | |
| These are research checkpoints for reproducing the paper, not general-purpose language models. | |
| They were trained on TinyStories, generated arithmetic, short QA and a few small maths sets, with a | |
| 4,096-token vocabulary and a 256-token context. | |
| ## Models | |
| Test loss is the mean over 7 test sets. Arithmetic is exact-match accuracy on 500 held-out | |
| problems. Looped models are scored at `k = 3`, no-loop models at `k = 1`. Mean ± standard | |
| deviation over the 3 seeds; per-seed numbers are in each folder's `results.json`. | |
| | Model | Folders | Params | Test loss | Arithmetic (%) | | |
| |---|---|---|---|---| | |
| | No-loop, 10 layers (k=1) | `noloop_s0` `noloop_s1` `noloop_s2` | 26.1M | 1.710 ± 0.002 | 65.5 ± 7.3 | | |
| | No-loop, 30 layers (k=1) | `noloop_deep_s0` `noloop_deep_s1` `noloop_deep_s2` | 53.6M | 1.694 ± 0.010 | 70.6 ± 4.8 | | |
| | Plain | `plain_s0` `plain_s1` `plain_s2` | 26.1M | 1.708 ± 0.006 | 65.6 ± 4.6 | | |
| | Plain + run index | `plain_signal_s0` `plain_signal_s1` `plain_signal_s2` | 26.1M | 1.708 ± 0.009 | 66.7 ± 5.6 | | |
| | Plain + run index, no norm | `plain_signal_nonorm_s0` `plain_signal_nonorm_s1` `plain_signal_nonorm_s2` | 26.1M | 1.711 ± 0.004 | 69.3 ± 4.1 | | |
| | Per-sequence experts, re-selected each loop | `perloop_s0` `perloop_s1` `perloop_s2` | 26.6M | 1.768 ± 0.035 | 66.6 ± 10.6 | | |
| | Per-token experts, re-selected each loop | `token_s0` `token_s1` `token_s2` | 26.6M | 1.716 ± 0.005 | 69.9 ± 4.0 | | |
| | Per-token experts + run index, re-selected each loop | `token_signal_s0` `token_signal_s1` `token_signal_s2` | 26.6M | 1.711 ± 0.013 | 71.3 ± 10.0 | | |
| | Per-token experts + run index, chosen once | `token_signal_once_s0` `token_signal_once_s1` `token_signal_once_s2` | 26.6M | 1.711 ± 0.012 | 59.7 ± 8.4 | | |
| | Between-loop block + run index | `between_signal_s0` `between_signal_s1` `between_signal_s2` | 34.9M | 1.752 ± 0.012 | 64.1 ± 9.5 | | |
| ## Files | |
| - `<model>_s<seed>/model.safetensors`: fp32 weights, no optimizer state. | |
| - `<model>_s<seed>/config.json`: model and training settings, and the final step. | |
| - `<model>_s<seed>/results.json`: every test score at `k = 1..4`, as reported in the paper. | |
| - `tokenizer.json`: the BPE tokenizer (4,096 tokens) shared by all models. | |
| - `manifest.json`: parameters, test loss and the SHA-256 of the training checkpoint each export | |
| came from. | |
| Each export was checked against its training checkpoint: every tensor is equal, and logits at | |
| `k = 1` and `k = 3` match exactly. | |
| ## Use | |
| ```bash | |
| pip install git+https://github.com/Davidooff/Spidory | |
| ``` | |
| ```python | |
| from spidory.evaluate import generate_text | |
| from spidory.hub import load | |
| model, tok = load("token_signal_s0") | |
| print(generate_text(model, tok, "Q: Calculate 12 + 7 * 3.\nA:", 8, k=3)) | |
| ``` | |
| `k` is the number of passes through the centre block (1 to 4). `model.infer(ids, k=3)` returns the | |
| logits, and for models with experts also the expert and run count each token or sequence used. | |
| ## License | |
| Apache-2.0, for both the weights and the code. | |