Some notes on swapping out the "important" parts of neural nets
A couple of years ago I got diagnosed with a neurological issue, except for most of that time nobody could tell me what it was. They treated me for dystonia. I had never even heard of it, and there wasn't much sign I'd ever get a real answer. Just last year it finally got a name: Functional Neurological Disorder. Also something I'd never heard of.
I'm putting this up here so it doesn't leak in sideways later. I had a lot of empty hours. I used to joke I could've done one of those NASA isolation simulations three times over. I decided to spend them learning things I figured I'd need when I could work again.
So this is a build log. A few experiments I could actually check. I am mostly self-taught, and I wanted to feel these ideas by making them. Nothing below is new. MLP-Mixer, gMLP, state-space models, AKOrN, distillation, mixture of experts, Sutton's Bitter Lesson. George Box already said the only line that matters here: all models are wrong, but some are useful.
SpikeWhale
The first thing I built from scratch was SpikeWhale-SNN-216M. 216M, spiking, leaky-integrate-and-fire, surrogate gradients, all of it written in plain PyTorch. No snntorch, no norse. Too much for a first project. I failed at parts of it and kept the parts that didn't fail.
Three of those stuck.
The tokenizer is byte-level, 16,512 vocab. It sits under every LM I have made since.
The "traits" started there too. An n-gram memory, HRM refinement, multi-token prediction, Kuramoto coupling, a JEPA head. Each one is wired so it does almost nothing at init and only wakes up if it helps.
And there was this odd biological bridge where each neuron's excitability came from a frozen virtual-cell gene model. That one left a habit. Steal a structure from some other field, bolt it on, and see if the net learns around it.
The Kuramoto mixer and the HRM-vs-JEPA question were already in that first net. I just didn't know that's what I had.
Byrne vs Escarda
I wanted to know what a trait actually does. So I ran the cleanest test I could set up.
Two ~86M models. Same decoder, same tokenizer, same data. One difference.
- Byrne-86M: HRM refinement only
- Escarda-86M: HRM plus a JEPA objective
Escarda came out of a 28-checkpoint bake-off.
The trait didn't add a capability I could name. It changed how the model packed what it learned. Same knowledge, different compression.
That is the question that ate the next year. If a trait is just a compression change, how much does the rest of the architecture even matter? Could I pull out the block people treat as the thinking and still get a generalist?
The mixer
People treat the per-layer MLP as the place the thinking happens. I left the tokenizer, the traits, and the data alone, and I swapped only that block.
Positronic-144M puts a bank of 508 coupled phase oscillators where the MLP was. Kuramoto dynamics on seven concentric rings.
Mycel-LM-79M grows a fungal colony instead. Tips in a bounded space, sensing their own density, steering away from it.
Then I stopped being reasonable. Per-tip oscillator rings, and then a mixer built out of the Wheeler–DeWitt equation. That's the "wavefunction of the universe." A quantum-gravity constraint with no time in it. The actual equation looks like this, and you don't need to parse it:
No $\partial/\partial t$. Nothing evolves. The state has to hold. I made a small differentiable version and used it as the channel mixer. Wheeler-63M, 63M params. 64 modes under a learnable Lorentzian metric. The trained checkpoint keeps that indefinite signature. I checked. There's a chat.
Then I stopped picking. Chimera-64M runs the oscillators, the growth, and the Wheeler–DeWitt wave in parallel, and a small GRPO rule sits on top as a council. It learns which law of physics to apply to each token. About 40k pretraining steps in. Chat is here.
They all learned language.
Worse than a plain MLP at the same compute. I don't have a matched baseline, so I can't quote the gap, only a curve. Chimera went from about 300 perplexity to about 42.
As far as I know nobody has made a language layer out of Wheeler–DeWitt. That mostly means it was a strange thing to try. It still trained. So the mixer is more interchangeable than its reputation.
Neural CPUs
The result I would actually defend is the other direction.
I took deterministic CPU logic and built processors out of small networks.
Break a CPU into datapath units. Train a net to reproduce each unit over its complete, enumerable input domain. Verify it bit-exact. N of N. Never 99%. Stitch those units into a golden core, then prove, per instruction, that a fast reference matches it exactly.
Five instruction sets:
- Neural GB — Game Boy. ALU, DAA, tile decode, sprite priority. All neural, all exhaustively verified.
- Neural i386 — DOOM. One full frame, 5,952,699 instructions, replayed fully neurally. Bit-identical to the real run. Framebuffer and all 128 MB of state.
- Neural N64 — CPU, FPU, RSP, RDP, checked against hardware test ROMs. Runs Portal 64.
- Neural RISC-V — boots Linux 6.8, runs karpathy's llama2.c inside the emulated OS. There's an int8 build, 3.6× smaller, that re-proves bit-exact. Accuracy is binary, so the smaller machine is the same machine.
- Neural AArch64 — boots Linux, hits the real internet through an in-emulator NAT, runs llama.cpp on a graphical framebuffer. Coherent text, drawn pixel by pixel by the neural CPU.
If you generalize a function over its whole finite domain, the approximation stops being an approximation. The net didn't mostly learn the ALU. It learned the ALU. I checked that on five ISAs. Bit-identical either holds or it doesn't. I used to treat approximation as the ceiling. It isn't. It's a stage.
Physics
In between those two poles is a Neural Physics Engine I had fun with.
Projective dynamics. Cloth, soft bodies, fluid. The local per-element constraint projection is a tiny net, about 10k parameters, shared across five solid materials and water via material tokens. Rotations stay analytic. The global solve stays analytic. I only handed the network the part that's usually derived by hand per material.
A second net, about 1.5k, predicts the solver's converged correction so the loop needs fewer iterations.
WebGPU demo. Unfinished, and the card says so. Learned pieces inside an exact skeleton. That's the picture I trust.
Other modalities
Same primitives on other data. RMSNorm, RoPE / QK-Norm, SwiGLU, HRM, the usual family stuff.
Byrne-Speech: ~12M TTS. 8.4M acoustic model, 3.6M HiFi-GAN, one 12 GB GPU, distilled from a VITS teacher. Clear speech at a size people assume can't do speech.
Byrne-ASR-English: ~12M CTC. CER ≈ 0.085 on LibriTTS-R dev-clean. CPU.
Byrne-VE: 39M vision encoder, distilled from DINOv2, then improved with the teacher gone. About 88% of DINOv2's k-NN accuracy at about 45% of the params.
Byrne-VLM-131M: that encoder wired to a 90M LM, LLaVA-style.
Documents next. Byrne-Docling-131M reads a page image and emits DocTags (tables, formulas, code, bounding boxes), the same markup Granite-Docling and SmolDocling use. First version tiled a 224px encoder, learned the format, and looped. I went to 448px / 784 patches and it got worse. Center-crop was throwing away about 90% of tall pages. Letterbox fixed it. I also put the 140 DocTags markers into the tokenizer as atomic tokens, so <fcel> is one token instead of a spelling problem. Not a production parser. Space is here.
None of this is SOTA. The VLM's COCO CIDEr is about 0.06. Scene gist, not much else. That's on the card. I wasn't chasing a board. I wanted to see if a small net, given something to attend to and time to compress, would generalize the domain. Phonemes, pixels, tokens. It did, within the limits of the size.
DaisyChain
This is the half with numbers I trust.
DaisyChain takes Carbon-500M and distills it into four ~74M DNA/RNA specialists: eukaryote, prokaryote, mRNA, splice. About 295M total, under Carbon's budget. A small router looks at how surprised each specialist is by a sequence (bits/base) plus a bit of hidden state, and sends the sequence home.
Held-out routing is 100.0%. A plain argmin-perplexity rule gets 87.5%. Only one 74M specialist runs per query, so it's about 7× cheaper per token than the monolith. On Carbon's own base-pair metric we sit at 1.862 vs Carbon's 1.787, a +0.0752 gap, verified to 6e-08. That gap started at +0.164 and has been closing every round.
Carbon still wins. No domain is ahead. That's on the card. We did not beat the teacher. We recovered most of a 500M / 1T-token model as four small routed pieces, at a fraction of the compute, and the gap is still moving.
You don't have to keep the whole generalist hot to keep what it knows. Cut it up. Send each input to the expert that is least surprised to see it.
So what do I think
A trained net is a frozen generalist. Experience goes in, it gets weighted by whatever got attended to, you do that until it settles, you compress. The block in the middle is more replaceable than people act like it is. I watched a colony, an oscillator bank, and a timeless constraint all become next-token predictors under the same pipeline.
Push that far enough on a finite domain and it goes exact. That's the CPUs.
The cheap form is small specialists routed by their own surprise. That's DaisyChain.
I didn't invent that shape. I just kept building versions of it until I believed it.
The part I don't have nailed is attention. If a lot of intelligence is experience compressed against whatever you managed to look at, then a lot of the difference between seeing far and not is just what got attended to. I get distracted constantly. I have wondered how much that alone decides what I can see. I don't think these systems are alive. I also don't have a clean reason they aren't.
What's wrong with this
The CPUs are the rigorous work. Bit-exact, exhaustive, five ISAs.
The exotic LMs do not have a matched MLP at equal compute. Chimera's curve (~300 → ~42) is a curve, not a comparison. "It generalized" there means it learned coherent-ish language. It does not mean it won a benchmark.
The small stack is a demo.
DaisyChain still trails Carbon. Carbon wins about 36 of 100, and Carbon is a draft model, not the 3B.
I found this useful because I built it. Reading it would not have done the same thing. Once I stopped treating the papers as the work, the work was more approachable than it looked.
Links
- Origin: SpikeWhale-SNN-216M
- Mixers: Positronic-144M · Mycel-LM-79M · Wheeler-63M (chat) · Chimera-64M (chat) · Byrne-86M vs Escarda-86M
- Neural CPUs: GB · i386 / DOOM · N64 · RISC-V · AArch64
- Physics: engine · demo
- Small stack: Speech · ASR · Vision · VLM · Docling · Embed · Anon
- DaisyChain: genomics · demo