README.md

---
license: apache-2.0
tags:
  - kuramoto
  - oscillator
  - language-model
  - synchronization-routing
  - tinystories
library_name: pytorch
---

# KOLM-Alpha — Fully-Oscillatory Kuramoto Language Model

*Delon Swartz — AI Researcher | Engineer*

**KOLM-Alpha is, to our knowledge, the first language model with *no
transformer computation anywhere* — both routing between tokens and
processing within them are performed by networks of coupled Kuramoto
oscillators.** It is the successor to [KOLM-Hybrid-1](https://huggingface.co/Dbrent/KOLM-Hybrid-1),
which kept standard attention and made only the feed-forward slot
oscillatory. KOLM-Alpha replaces attention itself with **synchronization
routing**.

> **Naming note:** the name *KOLM-Alpha* previously labelled the hybrid twin
> study, now published as **KOLM-Hybrid-1**. Under the project naming
> registry (2026-07), *KOLM-Alpha* denotes this fully-oscillatory line
> (working name "KOLM-True").

## The mechanism: KuramotoRouter A

Each layer's routing is a competitive synchronization process, not static
attention:

- **Softmax competition** over the causal past — sharp selection, not
  blurry averaging.
- **RoPE on the routing scores** — position is *in* the routing, not just
  the moved content.
- **Dynamic query modulation** — routing weights are *recomputed at every
  settling step*, modulated by the evolving oscillator state. This is
  genuinely dynamic routing that static attention cannot express.

The routed value drives each token's oscillators, which settle via Kuramoto
dynamics under trained couplings, natural rotations, and (optionally)
frustrated (phase-offset) coupling. Normalization is **SphereNorm**. Every
block is an exact identity at initialization.

## The result

Under the same controlled protocol as KOLM-Hybrid-1 (identical tokenizer,
data, order, budget, seed; 11.26M TinyStories tokens; single laptop):

| Model | Params | Val loss | Perplexity |
|---|---|---|---|
| **KOLM-Alpha (Router A)** | 17.06M | **2.3170** | **10.14** |
| KOLM-Hybrid-1 (attention + osc-FFN) | 16.92M | 2.6946 | 14.80 |
| Transformer twin (TMT) | 17.54M | 2.7188 | 15.16 |

**KOLM-Alpha wins decisively — 0.38 nats below the hybrid, 0.40 below the
transformer twin** — while doing *zero* transformer computation. It crossed
the hybrid's final score at 5M tokens (41% through training).

An honest negative precedes this win: **Router v1** (sigmoid-gated,
position-blind, routing frozen across settle steps) *lost* at 3.0273. The
three diagnosed failures motivated Router A's softmax + RoPE + dynamic
modulation.

## Scaling to 42M from scratch (the flagship)

**KOLM-Alpha-42M** ("W42") is the same architecture at 16 layers, d=384,
H=320, frustrated coupling, randomized settling depth *K ~ U[1,4]*, trained
from scratch on 188M tokens (32k vocabulary, TinyStories + Simple English
Wikipedia):

- **Val loss 1.7141 / perplexity 5.55** — from scratch, matching
  function-preserving-transfer-warmed comparables (K-sweep K1 1.7162 /
  K2 1.6848 / K4 1.7044).
- **Chat**, via staged SFT: broad chat mixture → single-turn discipline
  (Alpaca). The staged order beats Alpaca-from-base by 0.35 nats
  (alpaca-val 3.19 vs 3.55). Turn discipline is clean (answers once, stops
  at `<|end|>`); simple facts land ("The capital of France is Paris").
  Entity binding and counting remain beyond 42M — capacity limits, not
  recipe bugs.

## Why it matters

The hybrid showed oscillators can *process* as well as an MLP. KOLM-Alpha
shows they can *route* better than attention — the whole layer is now
synchronization dynamics, with a settling-depth **compute dial** that adds a
memory-free axis of scale. Randomized-depth training makes that dial
extrapolate past its trained depth.

## Files (weights to follow)

| File | Purpose |
|---|---|
| `kolm_true.py` | full-oscillator model + trainer (Router A) |
| `kuramoto_torch.py` | Kuramoto block (parity-tested to 2.2e-16 vs NumPy) |
| `chat_true.py` | REPL / sampler with the `/think K` settling dial |
| `native_true_A.pt` | **KOLM-Alpha 17M weights** (Router A) |
| `native_w42.pt` | **KOLM-Alpha-42M** pretrain weights |
| `native_w42chat_alp.pt` | 42M chat model (staged SFT) |
| `tiny32k.json` | 32k tokenizer |

## Roadmap

- **KOLM-Beta-T** ([repo](https://huggingface.co/Dbrent/KOLM-Beta-T)) —
  transplanting pretrained transformer weights into this architecture to
  reach billion-parameter capability without pretraining.
- Larger from-scratch KOLM-Alpha rungs exposing the settling dial as a
  user-facing "think-harder" control.

## Status & limitations

Research preview. Headline comparisons are single-seed; the 17M Router-A run
used a settling/lr protocol that differs slightly from the twin runs (a
matched rerun is queued before external citation). Corpora are easy and all
budgets are far below Chinchilla-optimal; factual recall at 42M is limited.

## License

Apache-2.0.

## Acknowledgements

Builds on the Kuramoto-oscillator formulation of **AKOrN** (Miyato et al.,
ICLR 2025), the frustrated-synchronization principle of **FSN**
(arXiv:2606.18694), and the **TinyStories** corpus (Eldan & Li, 2023).