SPIRA v3: cortex representation repair β direct input pathway, multi-timescale trace banks, structured trigram inputs; controlled A/B (SNN alone 0.40% -> 2.07% in-vocab held-out vs bigram 4.94% at matched vocab 4096); paper, README and docs updated to third iteration
6923c51 verified | # SPIRA β Spiking Plastic Recurrent Architecture | |
| A CPU-only sequence-learning system **outside the transformer/BPE/backpropagation | |
| paradigm**, built from three components: | |
| | Component | What it replaces | | |
| |---|---| | |
| | **EPT** β Entropy-Patch Tokenization | BPE / WordPiece / SentencePiece | | |
| | **Hierarchical spiking cortex** β LIF reservoir + slow area, reward-modulated plasticity | Transformer + backprop | | |
| | **Hippocampus** β one-shot associative memory + replay consolidation | Fine-tuning / gradient learning | | |
| This is **v2** of the prototype: a 16,384-column readout (chosen from a measured | |
| coverage curve), a two-timescale hierarchical cortex (fast 512-neuron reservoir + | |
| slow 256-neuron area reading the pooled spike volley every 8 tokens), and | |
| hippocampal replay that consolidates one-shot memories into cortical weights. | |
| ## Headline results (2 vCPU, 4 GB RAM, 13.2 MB mixed text+code corpus) | |
| | Proof | Result | | |
| |---|---| | |
| | One-shot learning | **99.3%** exact recall after a single 0.4 s pass over an unseen book | | |
| | **Systems consolidation (v2)** | cortex-only recall of that book **0.3% β 46.9%** (64.2% in-vocab, 138Γ) after replaying 100k hippocampal tokens β the AM itself stays at 99.3% | | |
| | Pattern completion | **99.0%** recall with 1 of 4 context patches corrupted (sparse AM); 52.3% in the deployed full AM (46% load) | | |
| | Replay interference | none β held-out test accuracy *improves* slightly after replay (0.34% β 0.75% cortex-only) | | |
| | Zero-shot coverage | 6.3% patch novelty on held-out text+code; the 16k readout covers 64% of held-out targets (v1: 38% at 4k) | | |
| | Efficiency | β9.2 M synaptic MACs/token β β**81Γ fewer ops/token** than GPT-2-small training (v1 at 4k vocab: 2.4 M, 314Γ) | | |
| | Generation | β2,000 chars/s closed-loop on 2 vCPU, in-browser at ~450-700 chars/s | | |
| | One-shot write rate | 1,199,996 associations in 18 s, with an exact checkpoint round-trip | | |
| Honest negative results (reported in the paper): the cortex does not amortize | |
| generalization to unseen documents (test in-vocab 0.49% vs a bigram's 7.7%), and | |
| the slow area's token-level fusion self-discounts to Ξ²β0.06 at this scale β its | |
| value is as a second consolidation substrate during replay. See | |
| `../paper/spira_paper.pdf`, Sections 7β8. | |
| ## What's new in v3 (cortex representation repair) | |
| Diagnosis probes on the shipped v2 checkpoint showed the held-out weakness was | |
| representational: the readout state carried β1 token of memory (current patch | |
| decodable at 99.94%, previous patch at 39.7% vs 33.0% majority chance), and an | |
| offline ridge readout on the same states scored 0.36% β worse than the online | |
| delta readout (0.51%). v3 changes the readout's *input*, not its learning rule: | |
| - **Direct input pathway** (`direct_input=True`, `direct_tau=0.5`): a decaying | |
| trace of the exact input-neuron pattern is concatenated to the readout state, | |
| so the delta rule sees the recent input identities directly. | |
| - **Multi-timescale trace banks** (`trace_taus=(0.4, 0.8, 0.95)`): the reservoir | |
| keeps one spike-trace bank per Ο; slow banks are normalized by (1 β Ο) so the | |
| Ο=0.95 bank does not swamp the fast one. With the direct pathway, the readout | |
| state is `n_state = 512 Γ 3 + 1024 = 2560` (v2: 512), and `W_out` is | |
| `(2560, V+1)`. | |
| - **Structured inputs** (`structured_inputs=True`, `k_tri=6`): each patch | |
| activates 6 full-hash input neurons **plus** 6 byte-trigram hash neurons, so | |
| patches sharing byte trigrams share input neurons. | |
| - **Compatibility mode:** `direct_input=False, trace_taus=(0.4,), | |
| structured_inputs=False` reproduces the v2 architecture exactly | |
| (`n_state = 512`); v3 checkpoints record the new fields and restore them on | |
| `SPIRA.load()`. | |
| - **Bug fix:** v2's `model.train()` referenced an undefined `ok_mask` | |
| (NameError) β training only ever ran through `scripts/train_chunk.py`. Fixed | |
| in v3; `SPIRA.train()` now runs standalone. | |
| Controlled A/B (same protocol both arms β rebuilt corpus, vocab 4096, 200k | |
| tokens Γ 2 epochs, AM 50k contexts, held-out test 20k positions, coverage | |
| 45.24%): SNN alone, in-vocab β **0.40% (v2) β 2.07% (v3)**, patch bigram @4096 | |
| on the same positions **4.94%**. The beat-the-bigram criterion is **not met** | |
| (gap Γ12.4 β Γ2.4); trained region 3.76% β 13.29%. Training costs 189 t/s vs | |
| ~530 t/s and the checkpoint grows 14.6 MB β 45.9 MB. See `../V3_REPORT.md` and | |
| `artifacts/metrics_v3_ab.json`. | |
| ## Repository layout | |
| ``` | |
| spira/ Python package (NumPy only, CPU-only) | |
| hashing.py FNV-1a-64, splitmix64, mulberry32 β bit-compatible with the TS port | |
| ept.py entropy-patch tokenizer (order-1 Markov surprisal, content addressing) | |
| spiking.py LIF reservoir + Dale's law + homeostatic thresholds, | |
| Rescorla-Wagner readout gated by dopaminergic RPE, | |
| reward-modulated recurrent STDP, and the hierarchical | |
| slow area G (windowed spike volleys, top-down prior, | |
| counterfactual trust gain beta); v3: multi-timescale | |
| trace banks + direct input pathway to the readout | |
| hippocampus.py one-shot memory: primary full-key hash (pattern separation) | |
| + leave-one-out lanes (CA3 pattern completion), exact | |
| derivable-slot checkpoint round-trip | |
| model.py SPIRA wrapper: build / train / generate / teach / replay / | |
| calibrate_fusion / eval / save-load; v3 config: | |
| direct_input, trace_taus, direct_tau, structured_inputs, | |
| k_tri (defaults reproduce v3; flags off = v2 architecture) | |
| baselines.py gzip, byte-level BPE (HuggingFace tokenizers), sparse | |
| n-gram predictors | |
| train.py training CLI | |
| evaluate.py full evaluation harness (coverage curve, compression, | |
| prediction + hierarchy ablation, one-shot, REPLAY | |
| consolidation experiment, correction, generation) β | |
| resumable by section | |
| export_deploy.py JSON artifact export for the browser demo (format v2) | |
| artifacts/ trained checkpoint, metrics.json, deploy artifact, | |
| figures, browser-demo screenshots | |
| paper/ LaTeX source + compiled research paper (spira_paper.pdf) | |
| ``` | |
| ## Quick start | |
| ```bash | |
| # build the corpus (Gutenberg books + Python stdlib code) | |
| python3 scripts/build_corpus.py | |
| # chunked training (background processes are killed between calls in sandboxes; | |
| # each chunk fits a single tool/shell call). v2 defaults: vocab 16384, hierarchy on. | |
| python3 scripts/train_chunk.py --start 0 --count 150000 # chunk 1 (fresh build) | |
| python3 scripts/train_chunk.py --start 150000 --count 150000 # ... continue to 600k | |
| python3 scripts/train_chunk.py --start 0 --count 150000 # epoch 2 | |
| python3 scripts/train_chunk.py --start 450000 --count 150000 --raster # epoch 2 end + raster | |
| python3 scripts/train_chunk.py --am-only # hippocampal write pass (1.2M) | |
| python3 scripts/train_chunk.py --start 450000 --warmup 150000 # re-adapt operating point | |
| # full evaluation (resumable: completed sections are skipped on re-run) | |
| python3 -m spira.evaluate --model artifacts/spira_model.npz \ | |
| --train data/train.bin --test data/test.bin \ | |
| --oneshot data/oneshot.bin --out artifacts/metrics.json | |
| # hippocampal replay -> cortical consolidation (in a few lines) | |
| python3 - << 'EOF' | |
| import sys; sys.path.insert(0, ".") | |
| from spira import SPIRA | |
| m = SPIRA.load("artifacts/spira_model.npz") | |
| print(m.teach_bytes(open("data/oneshot.bin", "rb").read()[:60000])) | |
| print("before replay:", m.eval_stream(open("data/oneshot.bin","rb").read()[:60000], use_am=False, max_positions=4000)["hier_acc"]) | |
| print(m.replay(100000)) # the "sleep" pass | |
| print("after replay:", m.eval_stream(open("data/oneshot.bin","rb").read()[:60000], use_am=False, max_positions=4000)["hier_acc"]) | |
| EOF | |
| # export the browser-demo artifact (format v2, includes the slow area) | |
| python3 -m spira.export_deploy --model artifacts/spira_model.npz \ | |
| --out artifacts/spira_deploy.json --am-cap 100000 --am-min-count 1 | |
| ``` | |
| ## Design notes (the interesting parts) | |
| - **Constant-information tokens.** Each EPT patch carries β15.9 bits of accumulated | |
| surprisal; low-entropy text yields long patches, high-entropy text short ones. | |
| Patch identity is the FNV-1a-64 hash of the bytes β the same substring is always | |
| the same token, anywhere, with zero vocabulary learning. | |
| - **The readout rule is local.** Readout units apply a WidrowβHoff / RescorlaβWagner | |
| delta rule on their own activity, gated by dopamine `d = r β baseline(r)`. No | |
| gradients, no chain rule, no autodiff. | |
| - **Mean-centering the spike trace matters.** The raw trace carries a dense shared | |
| background that otherwise makes every column's score correlate with frequency; | |
| subtracting a slow EMA of the trace (homeostatic scaling) is what makes per-context | |
| learning converge. | |
| - **Learned state includes the operating point.** Homeostatic thresholds, the trace | |
| mean, the slow area's homeostatic scalars, the top-down gain Ξ², and STDP-shaped | |
| recurrent weights must be checkpointed together β omitting any of them (or | |
| resetting Ξ² to its init) silently destroys the readout's regime. | |
| - **The hierarchy must be variance-matched and counterfactually credited.** The slow | |
| readout's raw scores live at ~20Γ the fast one's scale; fusing raw scores lets a | |
| tiny gain swamp the fast argmax, so the prior is rescaled by Ο_fast/Ο_slow | |
| (measured once, like the EPT budget). And a gain rule that rewards the slow area | |
| for ranking frequent truths high degenerates into a frequency prior β only | |
| saved-vs-corrupted counterfactual attribution calibrates Ξ² honestly (it | |
| self-discounts to β0.06 at this scale). | |
| - **Hippocampal replay closes the complementary-systems loop.** Teaching retains the | |
| episodic token trace; replay re-runs random 512-token slices through the cortex in | |
| full training mode, so one-shot material becomes slow cortical weight while the | |
| hippocampus stays intact (0.3% β 46.9% cortex-only recall, zero interference β | |
| held-out accuracy actually improves slightly). | |
| - **Associative-memory checkpoints are easy to get subtly wrong.** Entries must be | |
| packed and restored at their derivable slots (primary + leave-one-out hashes); an | |
| early format that rebuilt entries at lane-value addresses made the reloaded corpus | |
| memory invisible to queries β and inflated v1's corruption-robustness number, | |
| which had been measured against an accidentally-empty hippocampus. | |
| - **Hybrid hippocampal addressing.** A primary full-key hash gives every distinct | |
| context its own slot (capacity); leave-one-out key hashes let a query with one | |
| corrupted lane still reach the entry (completion). Completion strength trades | |
| against table load (99% at 0.8% load vs 52% at 46% load); two corrupted lanes is | |
| an explicit architectural cliff. | |
| - **The TS port is compatible.** `web/src/lib/spira/` regenerates the reservoir from | |
| the same seeded PRNG (mulberry32), shares the exact hashing spec, ports the slow | |
| area (stepG + variance-matched fusion), and loads the same deploy artifact, so | |
| browser inference matches the Python prototype. (Replay is Python-only.) | |
| ## Citation | |
| See `../paper/spira_paper.pdf` β *SPIRA: Sequence Learning Without Backpropagation or | |
| GPU Clusters*, preprint, October 2026 (third iteration). | |