| --- |
| license: apache-2.0 |
| tags: |
| - kuramoto |
| - oscillator |
| - language-model |
| - tinystories |
| library_name: pytorch |
| --- |
| |
| # KOLM-Alpha — Kuramoto Oscillator Language Model |
|
|
| *Delon Swartz — AI Researcher | Engineer* |
|
|
| **KOLM-Alpha is, to our knowledge, the first language model whose per-token |
| processing layers are networks of coupled Kuramoto oscillators, trained |
| natively from scratch.** Each layer uses standard causal attention for |
| routing between tokens and a **Kuramoto block** for processing: *H* |
| oscillators, each a unit vector on a sphere, relax over *K* settling steps |
| under trained pairwise couplings, conditioned on the attention output. The |
| oscillator block replaces the Transformer's feed-forward network; attention |
| is retained. |
|
|
| ## The result |
|
|
| Under a strictly controlled **twin protocol** — identical tokenizer, data, |
| data order, context length, optimizer, schedule and seed, with *only* the |
| feed-forward slot differing — KOLM-Alpha is compared against **TMT** |
| (Transformer Model Twin), a standard attention + MLP model. Both were |
| trained from scratch on 11.3M tokens of TinyStories on a single consumer |
| laptop. |
|
|
|  |
|
|
| TMT leads early; KOLM-Alpha closes the gap steadily, crosses near 7M tokens, |
| and finishes ahead — **with 3.5% fewer parameters.** |
|
|
|  |
|
|
| | Model | Parameters | Val loss | Perplexity | |
| |---|---|---|---| |
| | **KOLM-Alpha** | **16,916,224** | **2.6946** | **14.80** | |
| | TMT (Transformer twin) | 17,538,816 | 2.7188 | 15.16 | |
|
|
| Both models generate coherent TinyStories-grade prose. Prompt *"Once upon a |
| time there was a little red shoe"*: |
|
|
| > **KOLM-Alpha:** …The little boy would come home and play with his friends. |
| > One day, the little girl was playing with the big castle in the sky… |
|
|
| ## Why it matters |
|
|
| A Transformer computes with a fixed number of operations per token. A |
| Kuramoto block computes by **iterative settling** — the oscillator field |
| relaxes toward a configuration over *K* steps — which opens a second axis of |
| scale that costs no additional memory, only settling time. KOLM-Alpha |
| establishes the first point: at matched parameters, oscillatory processing |
| is not merely competitive with a Transformer feed-forward network — it edges |
| it out. |
|
|
| ## Files |
|
|
| | File | Purpose | |
| |---|---| |
| | `native_kolm.py` | model + twin trainer (`--arch kolm` / `--arch transformer`) | |
| | `kuramoto_torch.py` | the Kuramoto block (parity-tested to 2.2e-16 vs a NumPy reference) | |
| | `kolm.py`, `olm.py` | pure-NumPy reference core with hand-derived, gradient-checked backprop | |
| | `sample_native.py` | generation from the trained weights | |
| | `native_kolm.pt` | **KOLM-Alpha weights** (16.9M) | |
| | `native_transformer.pt` | TMT weights (17.5M) | |
| | `tiny8k.json` | tokenizer (required to run the model) | |
| | `curve_kolm.csv`, `curve_transformer.csv` | validation-loss curves | |
|
|
| ## Run it |
|
|
| ```bash |
| python3 -m venv --system-site-packages .venv |
| .venv/bin/pip install torch tokenizers |
| |
| # generate from the released weights |
| .venv/bin/python sample_native.py --prompt "Once upon a time" |
| |
| # verify the hand-derived gradients of the NumPy reference (no PyTorch needed) |
| python3 kolm.py --gradcheck |
| ``` |
|
|
| ## Reproduce the comparison |
|
|
| ```bash |
| .venv/bin/pip install transformers accelerate |
| .venv/bin/python native_kolm.py --prep # build tokenizer + data |
| .venv/bin/python native_kolm.py --arch kolm --steps 5500 |
| .venv/bin/python native_kolm.py --arch transformer --steps 5500 |
| ``` |
|
|
| ## Roadmap |
|
|
| KOLM-Alpha is the first release. In progress: |
|
|
| - **KOLM-Beta** — a larger model with frustrated coupling and a broader |
| training corpus, grown from Alpha via function-preserving depth growth. |
| - **KOLM-Chat** — an instruction-tuned conversational KOLM, exposing the |
| settling depth as a user-facing "think-harder" control. |
|
|
| ## Details |
|
|
| - **Architecture:** 8 layers, d=384, 6 heads, context 256, 8k byte-level BPE. |
| Feed-forward slot = Kuramoto block (H=320 oscillators on S³, K=4 settling |
| steps, 32 order-parameter readouts). |
| - **Training:** AdamW, lr 1e-4 cosine, gradient clip 1.0, single seed. |
| - **Hardware:** one Apple-silicon laptop (MPS), ~6 hours. |
| - **Status:** research preview. The headline comparison is single-seed; a |
| second seed is queued. |
|
|
| ## License |
|
|
| Apache-2.0. |
|
|
| ## Acknowledgements |
|
|
| Builds on the Kuramoto-oscillator formulation of **AKOrN** (Miyato et al., |
| ICLR 2025) and the **TinyStories** corpus (Eldan & Li, 2023). |
|
|
