license: apache-2.0
tags:
- kuramoto
- oscillator
- language-model
- tinystories
library_name: pytorch
KOLM-Alpha — Kuramoto Oscillator Language Model
Delon Swartz — AI Researcher | Engineer
KOLM-Alpha is, to our knowledge, the first language model whose per-token processing layers are networks of coupled Kuramoto oscillators, trained natively from scratch. Each layer uses standard causal attention for routing between tokens and a Kuramoto block for processing: H oscillators, each a unit vector on a sphere, relax over K settling steps under trained pairwise couplings, conditioned on the attention output. The oscillator block replaces the Transformer's feed-forward network; attention is retained.
The result
Under a strictly controlled twin protocol — identical tokenizer, data, data order, context length, optimizer, schedule and seed, with only the feed-forward slot differing — KOLM-Alpha is compared against TMT (Transformer Model Twin), a standard attention + MLP model. Both were trained from scratch on 11.3M tokens of TinyStories on a single consumer laptop.
TMT leads early; KOLM-Alpha closes the gap steadily, crosses near 7M tokens, and finishes ahead — with 3.5% fewer parameters.
| Model | Parameters | Val loss | Perplexity |
|---|---|---|---|
| KOLM-Alpha | 16,916,224 | 2.6946 | 14.80 |
| TMT (Transformer twin) | 17,538,816 | 2.7188 | 15.16 |
Both models generate coherent TinyStories-grade prose. Prompt "Once upon a time there was a little red shoe":
KOLM-Alpha: …The little boy would come home and play with his friends. One day, the little girl was playing with the big castle in the sky…
Why it matters
A Transformer computes with a fixed number of operations per token. A Kuramoto block computes by iterative settling — the oscillator field relaxes toward a configuration over K steps — which opens a second axis of scale that costs no additional memory, only settling time. KOLM-Alpha establishes the first point: at matched parameters, oscillatory processing is not merely competitive with a Transformer feed-forward network — it edges it out.
Files
| File | Purpose |
|---|---|
native_kolm.py |
model + twin trainer (--arch kolm / --arch transformer) |
kuramoto_torch.py |
the Kuramoto block (parity-tested to 2.2e-16 vs a NumPy reference) |
kolm.py, olm.py |
pure-NumPy reference core with hand-derived, gradient-checked backprop |
sample_native.py |
generation from the trained weights |
native_kolm.pt |
KOLM-Alpha weights (16.9M) |
native_transformer.pt |
TMT weights (17.5M) |
tiny8k.json |
tokenizer (required to run the model) |
curve_kolm.csv, curve_transformer.csv |
validation-loss curves |
Run it
python3 -m venv --system-site-packages .venv
.venv/bin/pip install torch tokenizers
# generate from the released weights
.venv/bin/python sample_native.py --prompt "Once upon a time"
# verify the hand-derived gradients of the NumPy reference (no PyTorch needed)
python3 kolm.py --gradcheck
Reproduce the comparison
.venv/bin/pip install transformers accelerate
.venv/bin/python native_kolm.py --prep # build tokenizer + data
.venv/bin/python native_kolm.py --arch kolm --steps 5500
.venv/bin/python native_kolm.py --arch transformer --steps 5500
Roadmap
KOLM-Alpha is the first release. In progress:
- KOLM-Beta — a larger model with frustrated coupling and a broader training corpus, grown from Alpha via function-preserving depth growth.
- KOLM-Chat — an instruction-tuned conversational KOLM, exposing the settling depth as a user-facing "think-harder" control.
Details
- Architecture: 8 layers, d=384, 6 heads, context 256, 8k byte-level BPE. Feed-forward slot = Kuramoto block (H=320 oscillators on S³, K=4 settling steps, 32 order-parameter readouts).
- Training: AdamW, lr 1e-4 cosine, gradient clip 1.0, single seed.
- Hardware: one Apple-silicon laptop (MPS), ~6 hours.
- Status: research preview. The headline comparison is single-seed; a second seed is queued.
License
Apache-2.0.
Acknowledgements
Builds on the Kuramoto-oscillator formulation of AKOrN (Miyato et al., ICLR 2025) and the TinyStories corpus (Eldan & Li, 2023).