| --- |
| license: mit |
| library_name: numpy |
| tags: |
| - neural-cellular-automata |
| - mixture-of-experts |
| - arithmetic |
| - tiny-models |
| - numpy |
| - from-scratch |
| datasets: [] |
| pipeline_tag: other |
| --- |
| |
| # gary-neuron 🧠➕ |
|
|
| **A mesh of ~100 neurons that fire asynchronously and, between them, do arithmetic.** |
|
|
| gary-neuron is an **asynchronous Neural Cellular Automaton** whose per-cell update rule is a **top-2 Mixture-of-Experts**. It is not a transformer. It adds integers the way silicon actually does — by letting a **carry ripple across a strip of cells** — and it does so in **26,448 parameters of pure numpy** (34 KB int8), with a hand-written autograd engine and **zero ML frameworks**. |
|
|
| Same numpy-only soul as [gary-4-petite](https://huggingface.co/gary23w/gary-4-petite). Different question. petite asked *"can a tiny model speak?"* gary-neuron asks *"what if the model isn't one network, but a mesh of tiny neurons firing out of sync — can that compute?"* |
|
|
| It can. **99.97% exact-match on held-out 7-digit addition; 100% with a 9-vote ensemble.** |
|
|
| --- |
|
|
| ## The three ideas |
|
|
| gary-neuron is the intersection of three research lines, each contributing one piece: |
|
|
| | Idea | What it gives | Source | |
| |---|---|---| |
| | **Neural Cellular Automaton** | A strip of identical cells with one **shared** local update rule. Cell *i* perceives only `[left, self, right]`. | Mordvintsev et al., *Growing NCA* (Distill 2020); *Self-Organising Textures* (Distill 2021) | |
| | **Asynchrony** | Each step, only a **random subset of cells fire**. Breaks grid symmetry, buys robustness, and — crucially — lets carries settle in any order. | *Mesh Neural Cellular Automata* (arXiv:2311.02820, ACM TOG 2024) | |
| | **Mixture-of-Experts rule** | The shared rule is a router + **K=6 experts, top-2 gating** — so each firing cell uses only *some* of its neurons. A load-balancing loss makes them specialize. | Shazeer et al., *Sparsely-Gated MoE* (2017); Fedus et al., *Switch Transformer* (2021) | |
|
|
| And the task itself rides on a fourth: |
|
|
| > **Reversed-digit format.** The answer is emitted **least-significant digit first** — `12+34 → 64`, not `46`. This is the single change that flips tiny-model addition from "never quite right" to a sharp phase transition to ~100%, because the model predicts the LSB first, the same direction carries flow. *(Lee et al., "Teaching Arithmetic to Small Transformers", arXiv:2307.03381.)* |
|
|
| The beautiful part: **addition-with-carry *is* a cellular automaton.** Cell *i* holds digit *i* of each operand; it needs its own two digits and the carry from cell *i−1*. Carry propagation is local message-passing. So the NCA substrate isn't a gimmick bolted onto arithmetic — it's the natural shape of the problem. |
|
|
| --- |
|
|
| ## Stats |
|
|
| | | | |
| |---|---| |
| | Parameters | **26,448** | |
| | Weights (int8) | **34 KB** | |
| | Full release (model + engine + trainer) | ~40 KB | |
| | Architecture | async 1-D NCA, 8 cells · state dim 32 · 6 experts (top-2) · 3d→32→d expert MLPs | |
| | Substrate | reversed-digit strip, carry ripples low→high | |
| | Training | pure-numpy, CPU only, ~9k steps in 35-s bursts, from-scratch autograd | |
| | Inference | numpy. that's it. no tokenizer, no torch. | |
| | Hardware | anything that runs python | |
|
|
| The 6 experts end up **evenly used** (utilization 0.16–0.18 each) — the mesh genuinely distributes work across specialists rather than collapsing to one. |
|
|
| --- |
|
|
| ## How well it adds (measured, held-out, never-trained pairs) |
|
|
| The test space is ~10¹⁴ operand pairs; random train/test overlap is negligible. |
|
|
| | Benchmark | Result | |
| |---|---| |
| | **Held-out 10k, ≤7-digit, single async order** | **99.97%** exact-match (mean over 8 random orders, **std 0.02%**) | |
| | **Held-out 10k, 9-vote async ensemble** | **100.000%** exact-match | |
| | Exact-match by operand length (1→7 digits) | 99.9% – 100% across the board | |
| | Adversarial maximal-carry ripples (22 hand-picked) | 21/22 (the one miss is an 8-digit input — out of range for an 8-cell strip) | |
| | Random spot-check, 300 sums, vote(9) | 300/300 | |
|
|
| **Robustness to update order** is the headline an async CA should own: across 8 totally different random firing orders, exact-match moves by only **±0.02%**. The computation does not depend on *when* each neuron fires. |
|
|
| ### Train short, run a little longer |
|
|
| The mesh is trained at 20 async steps but you can run it longer at inference — classic NCA "iterate toward a fixed point": |
|
|
| ``` |
| steps : 8 12 16 20 24 28 |
| exact%: 84.7 98.7 99.9 99.95 99.97 99.94 |
| ``` |
|
|
| 24 steps is the sweet spot; past ~28 it drifts slightly (it's a learned attractor, not a perfect fixed point). The released engine defaults to 24. |
|
|
| > **Fully-synchronous (every cell fires every step) is *worse*, not better** — the model learned to rely on asynchrony, exactly the symmetry-breaking the ANCA literature predicts. |
|
|
| --- |
|
|
| ## Watch the mesh think |
|
|
| `python solve.py 9999999 1 --show` runs the hardest case — a single `+1` that must ripple a carry through all 8 cells — and prints every step. `·` = a cell that didn't fire that step; the number on the right is the live readout. |
|
|
| ``` |
| 9999999 + 1 (mesh = 8 cells, 6 experts, top-2, async p=0.5, 24 steps) |
| digit place (10^): 7 6 5 4 3 2 1 0 |
| ---------------------------------------------------- |
| step 0 digits: 1 1 1 1 9 9 9 1 | fired(expert#): · · · · 4 4 4 · = 11119991 |
| step 4 digits: 0 1 9 9 9 9 9 0 | fired(expert#): 4 · · 5 5 · · 2 = 1999990 |
| step 8 digits: 1 9 9 9 9 0 0 0 | fired(expert#): · · 5 5 · 4 · 4 = 19999000 |
| step 12 digits: 0 9 0 0 0 0 0 0 | fired(expert#): · · 3 4 · · · 4 = 9000000 |
| step 16 digits: 1 0 0 0 0 0 0 0 | fired(expert#): 4 2 · · · · 0 0 = 10000000 |
| step 23 digits: 1 0 0 0 0 0 0 0 | fired(expert#): · 2 · · · 1 1 · = 10000000 |
| ---------------------------------------------------- |
| => 9999999 + 1 = 10000000 OK |
| ``` |
|
|
| You can see the carry climb from cell 0 to cell 7 and the readout lock onto `10000000` by step ~16, then hold steady — a stable attractor. Different experts (4, 5, 2, 3, 0, 1) fire at different cells: *some neurons fire*, which ones depends on the local situation. |
|
|
| --- |
|
|
| ## Run it |
|
|
| ```bash |
| pip install numpy |
| python solve.py 1234567 + 7654321 # -> 8888888 |
| python solve.py 9999999 1 --show # watch the carry ripple, step by step |
| python solve.py --vote 9 48591 + 9732 # robust ensemble over 9 async orders |
| python solve.py # interactive |
| ``` |
|
|
| No tokenizer, no weights download step beyond this repo, no GPU. |
|
|
| ## Reproduce / keep training it |
|
|
| The full pure-numpy pipeline is in `training/` — including the from-scratch reverse-mode autograd (`garyneuron.py`), the finite-difference gradient check (`test_grad.py`), the carry-heavy hard-case miner (`data.py`), and the benchmark harness. |
|
|
| ```bash |
| cd training |
| python test_grad.py # verify the autograd (analytic vs numeric) |
| SEC=40 MAXDIG=7 HARD=0.35 python train.py # one 40-s training burst (resumes from ckpt) |
| python benchmark.py main # held-out + adversarial + by-length |
| python export_int8.py # re-quantize -> release |
| ``` |
|
|
| Trained and served entirely in numpy. The autograd, the MoE, the async CA, the int8 packing — all of it, ~700 lines, no frameworks. |
|
|
| --- |
|
|
| ## What it can't do (yet) |
|
|
| - **8 cells = 8 output digits.** Sums ≥ 10⁸ don't fit; widen `S` and retrain. |
| - **The single hardest full-length ripple** is right at the edge of the 24-step dynamics; the 9-vote ensemble cleans it up, but a maximally adversarial carry chain longer than the strip will defeat a fixed step budget. (This is the known hard case for *any* fixed-iteration local model.) |
| - It adds. That's the whole job. Subtraction/multiplication are future meshes. |
|
|
| ## Why this exists |
|
|
| To show that "intelligence" at tiny scale doesn't have to be one monolithic network. gary-neuron is **a hundred-odd neurons, firing out of sync, passing notes to their neighbors**, and the *collective* computes something exact. It's a toy — but it's a toy that makes the mesh-of-specialists idea concrete, measurable, and 34 KB. |
|
|
| ## Citations |
|
|
| - N. Lee, K. Sreenivasan, J. D. Lee, K. Lee, D. Papailiopoulos. *Teaching Arithmetic to Small Transformers.* arXiv:2307.03381 (2023). |
| - A. Mordvintsev, E. Niklasson, et al. *Growing Neural Cellular Automata.* Distill (2020). · E. Niklasson et al. *Self-Organising Textures.* Distill (2021). |
| - *Mesh Neural Cellular Automata.* arXiv:2311.02820, ACM TOG (2024). |
| - N. Shazeer et al. *Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer.* (2017). · W. Fedus, B. Zoph, N. Shazeer. *Switch Transformers.* (2021). |
|
|
| *Built with numpy. That's it.* |
|
|
