--- license: mit library_name: numpy tags: - neural-cellular-automata - mixture-of-experts - arithmetic - tiny-models - numpy - from-scratch datasets: [] pipeline_tag: other --- # gary-neuron 🧠➕ **A mesh of ~100 neurons that fire asynchronously and, between them, do arithmetic.** gary-neuron is an **asynchronous Neural Cellular Automaton** whose per-cell update rule is a **top-2 Mixture-of-Experts**. It is not a transformer. It adds integers the way silicon actually does — by letting a **carry ripple across a strip of cells** — and it does so in **26,448 parameters of pure numpy** (34 KB int8), with a hand-written autograd engine and **zero ML frameworks**. Same numpy-only soul as [gary-4-petite](https://huggingface.co/gary23w/gary-4-petite). Different question. petite asked *"can a tiny model speak?"* gary-neuron asks *"what if the model isn't one network, but a mesh of tiny neurons firing out of sync — can that compute?"* It can. **99.97% exact-match on held-out 7-digit addition; 100% with a 9-vote ensemble.** --- ## The three ideas gary-neuron is the intersection of three research lines, each contributing one piece: | Idea | What it gives | Source | |---|---|---| | **Neural Cellular Automaton** | A strip of identical cells with one **shared** local update rule. Cell *i* perceives only `[left, self, right]`. | Mordvintsev et al., *Growing NCA* (Distill 2020); *Self-Organising Textures* (Distill 2021) | | **Asynchrony** | Each step, only a **random subset of cells fire**. Breaks grid symmetry, buys robustness, and — crucially — lets carries settle in any order. | *Mesh Neural Cellular Automata* (arXiv:2311.02820, ACM TOG 2024) | | **Mixture-of-Experts rule** | The shared rule is a router + **K=6 experts, top-2 gating** — so each firing cell uses only *some* of its neurons. A load-balancing loss makes them specialize. | Shazeer et al., *Sparsely-Gated MoE* (2017); Fedus et al., *Switch Transformer* (2021) | And the task itself rides on a fourth: > **Reversed-digit format.** The answer is emitted **least-significant digit first** — `12+34 → 64`, not `46`. This is the single change that flips tiny-model addition from "never quite right" to a sharp phase transition to ~100%, because the model predicts the LSB first, the same direction carries flow. *(Lee et al., "Teaching Arithmetic to Small Transformers", arXiv:2307.03381.)* The beautiful part: **addition-with-carry *is* a cellular automaton.** Cell *i* holds digit *i* of each operand; it needs its own two digits and the carry from cell *i−1*. Carry propagation is local message-passing. So the NCA substrate isn't a gimmick bolted onto arithmetic — it's the natural shape of the problem. --- ## Stats | | | |---|---| | Parameters | **26,448** | | Weights (int8) | **34 KB** | | Full release (model + engine + trainer) | ~40 KB | | Architecture | async 1-D NCA, 8 cells · state dim 32 · 6 experts (top-2) · 3d→32→d expert MLPs | | Substrate | reversed-digit strip, carry ripples low→high | | Training | pure-numpy, CPU only, ~9k steps in 35-s bursts, from-scratch autograd | | Inference | numpy. that's it. no tokenizer, no torch. | | Hardware | anything that runs python | The 6 experts end up **evenly used** (utilization 0.16–0.18 each) — the mesh genuinely distributes work across specialists rather than collapsing to one. --- ## How well it adds (measured, held-out, never-trained pairs) The test space is ~10¹⁴ operand pairs; random train/test overlap is negligible. | Benchmark | Result | |---|---| | **Held-out 10k, ≤7-digit, single async order** | **99.97%** exact-match (mean over 8 random orders, **std 0.02%**) | | **Held-out 10k, 9-vote async ensemble** | **100.000%** exact-match | | Exact-match by operand length (1→7 digits) | 99.9% – 100% across the board | | Adversarial maximal-carry ripples (22 hand-picked) | 21/22 (the one miss is an 8-digit input — out of range for an 8-cell strip) | | Random spot-check, 300 sums, vote(9) | 300/300 | **Robustness to update order** is the headline an async CA should own: across 8 totally different random firing orders, exact-match moves by only **±0.02%**. The computation does not depend on *when* each neuron fires. ### Train short, run a little longer The mesh is trained at 20 async steps but you can run it longer at inference — classic NCA "iterate toward a fixed point": ``` steps : 8 12 16 20 24 28 exact%: 84.7 98.7 99.9 99.95 99.97 99.94 ``` 24 steps is the sweet spot; past ~28 it drifts slightly (it's a learned attractor, not a perfect fixed point). The released engine defaults to 24. > **Fully-synchronous (every cell fires every step) is *worse*, not better** — the model learned to rely on asynchrony, exactly the symmetry-breaking the ANCA literature predicts. --- ## Watch the mesh think `python solve.py 9999999 1 --show` runs the hardest case — a single `+1` that must ripple a carry through all 8 cells — and prints every step. `·` = a cell that didn't fire that step; the number on the right is the live readout. ``` 9999999 + 1 (mesh = 8 cells, 6 experts, top-2, async p=0.5, 24 steps) digit place (10^): 7 6 5 4 3 2 1 0 ---------------------------------------------------- step 0 digits: 1 1 1 1 9 9 9 1 | fired(expert#): · · · · 4 4 4 · = 11119991 step 4 digits: 0 1 9 9 9 9 9 0 | fired(expert#): 4 · · 5 5 · · 2 = 1999990 step 8 digits: 1 9 9 9 9 0 0 0 | fired(expert#): · · 5 5 · 4 · 4 = 19999000 step 12 digits: 0 9 0 0 0 0 0 0 | fired(expert#): · · 3 4 · · · 4 = 9000000 step 16 digits: 1 0 0 0 0 0 0 0 | fired(expert#): 4 2 · · · · 0 0 = 10000000 step 23 digits: 1 0 0 0 0 0 0 0 | fired(expert#): · 2 · · · 1 1 · = 10000000 ---------------------------------------------------- => 9999999 + 1 = 10000000 OK ``` You can see the carry climb from cell 0 to cell 7 and the readout lock onto `10000000` by step ~16, then hold steady — a stable attractor. Different experts (4, 5, 2, 3, 0, 1) fire at different cells: *some neurons fire*, which ones depends on the local situation. --- ## Run it ```bash pip install numpy python solve.py 1234567 + 7654321 # -> 8888888 python solve.py 9999999 1 --show # watch the carry ripple, step by step python solve.py --vote 9 48591 + 9732 # robust ensemble over 9 async orders python solve.py # interactive ``` No tokenizer, no weights download step beyond this repo, no GPU. ## Reproduce / keep training it The full pure-numpy pipeline is in `training/` — including the from-scratch reverse-mode autograd (`garyneuron.py`), the finite-difference gradient check (`test_grad.py`), the carry-heavy hard-case miner (`data.py`), and the benchmark harness. ```bash cd training python test_grad.py # verify the autograd (analytic vs numeric) SEC=40 MAXDIG=7 HARD=0.35 python train.py # one 40-s training burst (resumes from ckpt) python benchmark.py main # held-out + adversarial + by-length python export_int8.py # re-quantize -> release ``` Trained and served entirely in numpy. The autograd, the MoE, the async CA, the int8 packing — all of it, ~700 lines, no frameworks. --- ## What it can't do (yet) - **8 cells = 8 output digits.** Sums ≥ 10⁸ don't fit; widen `S` and retrain. - **The single hardest full-length ripple** is right at the edge of the 24-step dynamics; the 9-vote ensemble cleans it up, but a maximally adversarial carry chain longer than the strip will defeat a fixed step budget. (This is the known hard case for *any* fixed-iteration local model.) - It adds. That's the whole job. Subtraction/multiplication are future meshes. ## Why this exists To show that "intelligence" at tiny scale doesn't have to be one monolithic network. gary-neuron is **a hundred-odd neurons, firing out of sync, passing notes to their neighbors**, and the *collective* computes something exact. It's a toy — but it's a toy that makes the mesh-of-specialists idea concrete, measurable, and 34 KB. ## Citations - N. Lee, K. Sreenivasan, J. D. Lee, K. Lee, D. Papailiopoulos. *Teaching Arithmetic to Small Transformers.* arXiv:2307.03381 (2023). - A. Mordvintsev, E. Niklasson, et al. *Growing Neural Cellular Automata.* Distill (2020). · E. Niklasson et al. *Self-Organising Textures.* Distill (2021). - *Mesh Neural Cellular Automata.* arXiv:2311.02820, ACM TOG (2024). - N. Shazeer et al. *Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer.* (2017). · W. Fedus, B. Zoph, N. Shazeer. *Switch Transformers.* (2021). *Built with numpy. That's it.*