Some notes on swapping out the "important" parts of neural nets
Quazim0t0
a bit of setting, first (Happy 4th of July!)
A couple of years ago I got diagnosed with a neurological issue — except for most of that time, nobody could actually tell me what it was. The doctors treated me for dystonia, which I'd never even heard of, and there wasn't much sign I'd ever get a real answer. Just last year it finally got a name: Functional Neurological Disorder. Another thing I'd never heard of, and another kind of unknown.
I'm not writing this so you'll read the rest a certain way — honestly the opposite. I just want to set the scene, because the scene is most of the reason any of this exists. I suddenly had a lot of time and not much to do with it. (I used to joke I could've done one of those NASA "stay inside for months" isolation simulations three times over.) So I decided to spend the waiting absorbing as much as I could — the plan being that whenever I got better, I'd come out the other side with a wider range of skills to bring to the job market.
So what follows isn't a "look what I did." It's a slight story mixed with some experiments I could actually verify — mostly me learning in the open, and answering a few questions for myself.
Now, about the work itself — I'll say the honest thing up front: none of this is new, and I'd rather be the one to point that out than have someone do it for me. Everything below rhymes with stuff the field already knows — MLP-Mixer, gMLP, state-space models, the Kuramoto-oscillator-neuron work (AKOrN), plain old distillation and mixture-of-experts, and Sutton's Bitter Lesson. I didn't set out to discover anything. I'm mostly self-taught, and I wanted to feel an idea by building it instead of reading about it. So this is a build log with a thesis leaning against it, and the whole thing sits under George Box's line: all models are wrong, but some are useful. I'm after useful.
The idea, if I had to compress it:
A trained network is a frozen generalist. You pour experience in, weight it by whatever the thing paid attention to, repeat it enough times that it settles, and compress it. What comes out the far side is generalization — and the machinery in the middle matters way less than you'd think. Attend, compress, repeat. That's the loop.
And then a second half I only really believed after building it: once you have a generalist, you usually don't need all of it at once. You can chop it into small specialists and hand each input to whichever one is least surprised to see it. Surprise is a nice routing signal because nobody has to label it — it just falls out.
Two things surprised me along the way, and they're the reason I bothered to write any of this down. Pushed far enough, generalization stops being fuzzy and becomes exact. And the efficient shape of a generalist isn't one big model, it's a handful of small ones that route by their own uncertainty. I'll get to both.
it started with a spiking network I probably shouldn't have attempted
The first thing I built from scratch was SpikeWhale-SNN-216M — a 216M spiking language model, with the leaky-integrate-and-fire neurons and the surrogate-gradient training loop all hand-written in plain PyTorch. No snntorch, no norse. It was too ambitious for a first project and I learned a ton failing at parts of it, but it quietly turned into the seed of everything since. Without really meaning to, I put three things in it that I've reused ever since: a byte-level tokenizer (16,512 vocab) that now runs under every LM I make; a little toolbox of "traits" — an n-gram memory, HRM refinement, multi-token prediction, Kuramoto oscillator coupling, a JEPA head — each wired so it does basically nothing at init and only wakes up if it helps; and a genuinely weird biological bridge where each neuron's excitability is set by a frozen virtual-cell gene model. That last one taught me a habit I never dropped: borrow structure from some other field, bolt it in, and see if the network learns around it anyway.
The Kuramoto mixer and the HRM-vs-JEPA question that became the core of this whole thing? They were already sitting in that first spiking net. I just didn't understand what I'd built yet.
the experiment that turned a curiosity into a thesis
I got stuck on a simple question: what does a "trait" actually do? So I ran the cleanest test I could think of. Two ~86M models, identical in every single way except one objective:
- Byrne-86M — HRM refinement only.
- Escarda-86M — HRM plus a JEPA objective.
Same decoder, same tokenizer, same data, same everything. (Escarda got picked from a 28-checkpoint bake-off, for what it's worth.) And here's the thing that reframed everything for me: the trait didn't add a capability I could point at. It changed how the model compressed what it learned. Not what it knew — the shape of how it packed it. Which made me ask the obvious follow-up: if a trait just changes the compression, how much does the rest of the architecture even matter? Could I rip out the block everyone treats as the important one and still get a generalist?
so I started ripping out the important block
In a transformer, the per-layer MLP is where people assume the thinking happens. So I kept the tokenizer, traits, and data fixed and swapped only that block for increasingly unreasonable things.
Positronic-144M replaces the MLP with a bank of 508 coupled phase oscillators (Kuramoto dynamics on seven concentric rings). Mycel-LM-79M replaces it with a fungal-colony growth model — little tips growing in a bounded space, sensing their own density, steering away from it. And I kept going down that road, into per-tip oscillator rings, and then one built entirely from the Wheeler–DeWitt equation — the "wavefunction of the universe," a timeless quantum-gravity constraint. Here's the actual thing (you really don't need to parse it to follow along):
The odd part is there's no in it — nothing evolves; the state just has to hold. I made a small differentiable version of that and used it as a channel mixer. That's Wheeler-63M (63M params; the 64 modes evolve under a learnable Lorentzian metric, and the trained checkpoint keeps that indefinite signature — I checked). You can actually chat with it.
Then I stopped picking one. Chimera-64M runs all three physics cores — oscillators, growth, and the Wheeler–DeWitt wave — in parallel, and a small reinforcement-learning rule (GRPO) sits on top as a "council," learning which law of physics to apply to each token. It's the one that ties all the others together, ~40k pretraining steps in so far, and you can talk to it here.
They all learned language. Rougher than a plain MLP at the same compute, and I want to be honest about that. But unmistakably generalizing. A colony, a bank of oscillators, and an equation with no built-in idea of time all became next-token predictors under the exact same pipeline. As far as I know nobody has made a language layer out of Wheeler–DeWitt, which mostly tells you it's a strange thing to do, not a smart one. It still worked. That was half my answer: the mixer is way more interchangeable than its reputation.
the other extreme: generalizing all the way to exact
The part I'd actually stake something on is the opposite direction. I took the least neural-friendly thing there is — deterministic CPU logic — and built processors out of neural networks.
The trick is one idea reused: break a CPU into its datapath units, train a small net to reproduce each one over its complete, enumerable input domain, and verify it bit-exact. N of N. Never 99%. Then stitch the units into a golden core and prove, per instruction, that a fast reference core matches it exactly.
I've done this across five instruction sets now — the demos were the fun part:
- Neural GB — a Game Boy: ALU, DAA, tile decode, sprite priority, all neural, all exhaustively verified.
- Neural i386 — DOOM. A full frame (5,952,699 instructions) replayed fully neurally, bit-identical to the real run, framebuffer and all 128 MB of state.
- Neural N64 — CPU/FPU/RSP/RDP units checked against hardware test ROMs, running real homebrew (Portal 64).
- Neural RISC-V — boots Linux 6.8 and runs karpathy's llama2.c inside the emulated OS; an int8 build, 3.6× smaller, re-proves bit-exact (accuracy is binary, so the smaller machine is literally indistinguishable).
- Neural AArch64 — boots Linux, reaches the real internet through an in-emulator NAT, runs llama.cpp on a graphical framebuffer — coherent text drawn pixel-by-pixel by the neural CPU.
This is where that first surprise lives. When you generalize a function completely over a finite domain, the approximation stops being fuzzy and just becomes the thing. The net didn't "mostly" learn the ALU — it learned it exactly, and that's checkable, so I checked it, across five different instruction sets. And unlike everything else here, "bit-identical" isn't a vibe. It matches or it doesn't. Approximation turned out to be a stage, not a ceiling.
a physics engine that's half-learned, half-exact
Somewhere in the middle of exact and fuzzy is a Neural Physics Engine I had a good time with. It's a projective-dynamics simulator — cloth, soft bodies, fluid — where the local per-element constraint projection is a tiny learned network (about 10k parameters, shared across five solid materials and water via material tokens), but the rotations and the global solve stay exactly analytic. I don't ask the network to learn what math already does perfectly; I only hand it the part that's usually hand-derived per material, and let one small net cover all of them. There's a second ~1.5k net that just predicts the solver's converged correction so it needs fewer iterations. You can poke the WebGPU demo in a browser. It's unfinished and I say so on the card — but the shape of it is the point: learned pieces slotted inside an exact skeleton, which is maybe the most honest picture of where these things actually belong.
the same parts, every modality
If the loop is what matters, the same toolbox should work no matter the data. So I built a whole little stack out of the same primitives (RMSNorm, RoPE/QK-Norm, SwiGLU, HRM, and friends):
Byrne-Speech is a ~12M text-to-speech model (an 8.4M acoustic model + a 3.6M HiFi-GAN vocoder), trained on a single 12 GB GPU, distilled from a VITS teacher — clear speech at a size people assume is too small. Byrne-ASR-English is a ~12M CTC recognizer, CER ≈ 0.085 on LibriTTS-R dev-clean, runs on CPU. Byrne-VE is a 39M vision encoder distilled from DINOv2 and then improved teacher-free, landing around 88% of DINOv2's k-NN accuracy at ~45% of the params. And Byrne-VLM-131M wires the vision encoder to a 90M language model, LLaVA-style. I also tried it on documents: Byrne-Docling-131M reads a page image and emits DocTags — the structured markup (tables, formulas, code, bounding boxes) that Granite-Docling and SmolDocling use. Getting there was a very honest slog: the first version tiled a 224px encoder and learned the format but looped; a 448px encoder (784 patches) made it worse until I realized center-crop was throwing away ~90% of tall pages, and switching to letterbox fixed it. I also added the 140 DocTags markers to the tokenizer as atomic tokens so it emits <fcel> as one token instead of spelling it out. It's a research artifact, not a production parser — but you can try it.
None of these are state of the art and I don't pretend they are — the VLM scores a rough CIDEr ≈ 0.06 on COCO and gets scene gist but not much more; I say so on its card. The point was never the leaderboard. It's that a small net, given something to attend to and time to compress, generalized the domain whether it was phonemes, pixels, or tokens.
and then: compress it, route by surprise
The last piece is the second half of the thesis made real, and it's the one with the cleanest numbers. DaisyChain takes a 500M genomics model (Carbon-500M) and distills it into four ~74M DNA/RNA specialists — eukaryote, prokaryote, mRNA, splice — about 295M total, under the original budget. A little router reads how surprised each specialist is by a sequence (its bits/base) plus a bit of its hidden state, and sends the sequence to its home domain.
The measured results are what I actually care about here:
- routing accuracy 100.0% on held-out data (a plain argmin-perplexity rule only gets 87.5%),
- ~7× cheaper per token than the monolith, since only one 74M specialist runs per query,
- and it lands within +0.0752 bits/base of Carbon-500M on Carbon's own base-pair metric (1.862 vs 1.787, verified to 6e-08), a gap that's shrunk steadily round over round from +0.164.
Carbon still wins overall — no single domain is ahead yet, and I keep that on the card too. The claim isn't "we beat the teacher." It's "we recovered most of a 500M/1T-token model in four small routed pieces at a fraction of the compute, and the gap keeps closing." That's the whole endpoint: you don't have to keep the entire generalist hot to keep its knowledge. Chop it up, and let each input go to the expert least surprised to see it.
what I actually think, and what I don't
Across a CPU, a genome, a voice, images, and language, the same shape kept showing up: give a network something to attend to, a way to compress from it, and enough time, and it becomes a frozen generalist — mostly regardless of what's sitting in the middle. Push that far enough and it goes exact (the CPUs). And the efficient form of it is small specialists routed by their own surprise (DaisyChain). The idea underneath isn't mine and isn't new. All I can honestly say is I kept testing it from every angle I could build, and it kept holding.
The part I'm least sure about is the part I think about most, so I'll just leave it as wondering. If intelligence is mostly experience compressed uniquely against whatever attention could reach, then maybe the difference between seeing far and not is mostly a difference in what you were able to attend to. I get distracted constantly and I've genuinely wondered how much that alone shapes what I can see. I don't think these systems are alive. I'm also not sure I could cleanly tell you why they aren't. Those are questions I'm holding, not answers I'm handing you.
where it falls short (because it does)
I'd rather undersell this than get caught overselling it. The neural CPUs are the rigorous part — bit-exactness is exhaustively checkable and I checked it, on all five. The exotic language models don't have a matched plain-MLP baseline at equal compute, so I can quote trajectories (Chimera's perplexity fell ~300 → ~42) but not the honest gap to a normal block; "generalize" there means "learns coherent-ish language," not a benchmark. The small stack is a demonstration, not a leaderboard entry. And DaisyChain, for all its nice routing numbers, still sits behind its teacher — Carbon wins about 36 out of 100, and Carbon itself is only a draft model, not a benchmark-grade 3B.
Some wrong models are still useful to think with. Building these is the only reason I ever found this one useful — and if there's one thing I'd want you to take away, it's just how approachable the whole thing turned out to be once I stopped reading about it and started making it.
everything here, if you want to poke at it
Quazim0t0 and DaisyChainAI on Hugging Face.
- Origin Story: SpikeWhale-SNN-216M
- The mixer experiments: Positronic-144M · Mycel-LM-79M · Wheeler-63M (chat) · Chimera-64M (chat) — and the trait A/B Byrne-86M vs Escarda-86M
- Neural CPUs: GB · i386 / DOOM · N64 · RISC-V · AArch64
- Neural physics: Neural Physics Engine (demo)
- The small-model stack: Speech · ASR · Vision · VLM · Docling · Embed · Anon
- Compression: DaisyChain genomics · demo