ESP32 Markov Mixer, 973k parameters
We wanted to see whether a language model made mostly from low-rank Markov lookups, two tiny recurrent heads, and a little nonlinear context mixing could fit on an ESP32 and still say recognizably different things. It can.
This repository is the exact model artifact from that experiment. It is a demonstration, not a useful general-purpose assistant or a supported model release.
The bare-metal Rust inference engine and serial TUI are published at arm64be/esp32-serial-tui.
What is here
| File | Description |
|---|---|
release-binding-allparam-r8-30ep.pt |
PyTorch checkpoint with 973,471 total and trainable parameters |
release-binding-allparam-r8-30ep.q8mm |
Deployed fixed-point model, tokenizer included, 1,075,652 bytes |
release-binding-allparam-r8-30ep.json |
Integer format, tensor layout, quantization, and architecture manifest |
tokenizer.json |
Lowercasing 4,096-token byte-level BPE tokenizer |
release-binding-run-notes.md |
Training, validation, export, and release-gate notes |
The architecture uses a 192-token context, eight direct Markov lags at rank 21, four hashed context features, separate 32-hidden-unit intent and grammar GRUs, a 64-unit context MLP, and a 48-unit gate. It has no positional embeddings, prompt-copy path, shared recurrent trunk, feature mixer, or rank normalization.
The integer export uses square-companded signed int8 lookup tables, per-row signed int8 matrices, Q10 int16 activations, and integer accumulators. In a 1,024-context oracle comparison against the float checkpoint, it reached 98.24% top-1 agreement; fixed and float greedy decoding matched on all release prompts.
What it says
Greedy decoding on the ESP32 produced:
tell me a story about a fish named bob
-> Bob was a little fish who was very happy. He was so happy to see the little bird.
what is the capital of france?
-> Hello, the little boy's mom and dad are glad.
hi
-> Hello. I am a little boy.
The capital answer is plainly wrong. The release gate was narrower: three different prompts had to yield distinct English instead of collapsing into the same tree-and-ball story. The final checkpoint was corrected for those three prompt families using 120 hand-authored examples and strong replay against the pre-correction checkpoint.
Training snapshot
The correction ran for 30 full epochs over 114 training records, with AdamW,
peak learning rates of 2e-4 for context computation and 1e-4 for lookup and
output parameters, 1e-4 weight decay, a 5% warmup, and replay KL weight 8.0.
Its six-record correction validation result was CE 2.7134 / PPL 15.08; a broader
60-batch mixed validation was CE 2.7845 / PPL 16.19.
Prompt format
Text is normalized to lowercase and wrapped as:
<|user|>
{prompt}
<|assistant|>
Greedy decoding is intentional. Sampling experiments reduced some repetition but damaged prompt binding on this model.
Checksums
491047766597fdb8375fa7bde4fe3d069858a6305c9fcac9464d022bcc20e518 release-binding-allparam-r8-30ep.pt
c9b2d7102c72b3ba61d14d47fff85906d28e69446114f71ff1eba6c922df0b30 release-binding-allparam-r8-30ep.q8mm
ea17b665872be0d59d485b54d056f39b84b2cd7bfc18a86c408a8c219ed9bad1 release-binding-allparam-r8-30ep.json
4d06b372a8e7ba5f5b015c3901cbc8a2cdac8d248e3e7275e08b27899e76946b tokenizer.json
Limitations
This model is tiny, narrow, repetitive, factually unreliable, and strongly shaped by TinyStories-like data. It should not be used for decisions, factual questions, or safety-critical tasks. The point is the constrained architecture and embedded execution, not model quality.