---
license: apache-2.0
tags:
  - kuramoto
  - oscillator
  - language-model
  - tinystories
library_name: pytorch
---

# KOLM-Alpha — Kuramoto Oscillator Language Model

*Delon Swartz — AI Researcher | Engineer*

**KOLM-Alpha is, to our knowledge, the first language model whose per-token
processing layers are networks of coupled Kuramoto oscillators, trained
natively from scratch.** Each layer uses standard causal attention for
routing between tokens and a **Kuramoto block** for processing: *H*
oscillators, each a unit vector on a sphere, relax over *K* settling steps
under trained pairwise couplings, conditioned on the attention output. The
oscillator block replaces the Transformer's feed-forward network; attention
is retained.

## The result

Under a strictly controlled **twin protocol** — identical tokenizer, data,
data order, context length, optimizer, schedule and seed, with *only* the
feed-forward slot differing — KOLM-Alpha is compared against **TMT**
(Transformer Model Twin), a standard attention + MLP model. Both were
trained from scratch on 11.3M tokens of TinyStories on a single consumer
laptop.

![Validation loss](assets/val_loss_curve.png)

TMT leads early; KOLM-Alpha closes the gap steadily, crosses near 7M tokens,
and finishes ahead — **with 3.5% fewer parameters.**

![Final scores](assets/final_scores.png)

| Model | Parameters | Val loss | Perplexity |
|---|---|---|---|
| **KOLM-Alpha** | **16,916,224** | **2.6946** | **14.80** |
| TMT (Transformer twin) | 17,538,816 | 2.7188 | 15.16 |

Both models generate coherent TinyStories-grade prose. Prompt *"Once upon a
time there was a little red shoe"*:

> **KOLM-Alpha:** …The little boy would come home and play with his friends.
> One day, the little girl was playing with the big castle in the sky…

## Why it matters

A Transformer computes with a fixed number of operations per token. A
Kuramoto block computes by **iterative settling** — the oscillator field
relaxes toward a configuration over *K* steps — which opens a second axis of
scale that costs no additional memory, only settling time. KOLM-Alpha
establishes the first point: at matched parameters, oscillatory processing
is not merely competitive with a Transformer feed-forward network — it edges
it out.

## Files

| File | Purpose |
|---|---|
| `native_kolm.py` | model + twin trainer (`--arch kolm` / `--arch transformer`) |
| `kuramoto_torch.py` | the Kuramoto block (parity-tested to 2.2e-16 vs a NumPy reference) |
| `kolm.py`, `olm.py` | pure-NumPy reference core with hand-derived, gradient-checked backprop |
| `sample_native.py` | generation from the trained weights |
| `native_kolm.pt` | **KOLM-Alpha weights** (16.9M) |
| `native_transformer.pt` | TMT weights (17.5M) |
| `tiny8k.json` | tokenizer (required to run the model) |
| `curve_kolm.csv`, `curve_transformer.csv` | validation-loss curves |

## Run it

```bash
python3 -m venv --system-site-packages .venv
.venv/bin/pip install torch tokenizers

# generate from the released weights
.venv/bin/python sample_native.py --prompt "Once upon a time"

# verify the hand-derived gradients of the NumPy reference (no PyTorch needed)
python3 kolm.py --gradcheck
```

## Reproduce the comparison

```bash
.venv/bin/pip install transformers accelerate
.venv/bin/python native_kolm.py --prep                 # build tokenizer + data
.venv/bin/python native_kolm.py --arch kolm        --steps 5500
.venv/bin/python native_kolm.py --arch transformer --steps 5500
```

## Roadmap

KOLM-Alpha is the first release. In progress:

- **KOLM-Beta** — a larger model with frustrated coupling and a broader
  training corpus, grown from Alpha via function-preserving depth growth.
- **KOLM-Chat** — an instruction-tuned conversational KOLM, exposing the
  settling depth as a user-facing "think-harder" control.

## Details

- **Architecture:** 8 layers, d=384, 6 heads, context 256, 8k byte-level BPE.
  Feed-forward slot = Kuramoto block (H=320 oscillators on S³, K=4 settling
  steps, 32 order-parameter readouts).
- **Training:** AdamW, lr 1e-4 cosine, gradient clip 1.0, single seed.
- **Hardware:** one Apple-silicon laptop (MPS), ~6 hours.
- **Status:** research preview. The headline comparison is single-seed; a
  second seed is queued.

## License

Apache-2.0.

## Acknowledgements

Builds on the Kuramoto-oscillator formulation of **AKOrN** (Miyato et al.,
ICLR 2025) and the **TinyStories** corpus (Eldan & Li, 2023).