---
license: other
license_name: lolm-community-license-1.0
license_link: LICENSE
library_name: lolm
tags:
  - language-model
  - hybrid-transformer-ssm
  - state-space-model
  - mamba
  - latent-order
  - research
pipeline_tag: text-generation
---

# LOLM-304M — Latent Order Language Model

**LOLM** is a hybrid Transformer–SSM architecture that separates surface token
prediction from latent state tracking. This is the **304M** research checkpoint
behind the published WikiText-103 results.

> Authors: Bryan Leonard & Brandyn Leonard · Qira LLC · Provisional patent 64002166
> Code: https://github.com/TheArtOfSound/LOLM

## TL;DR

At 304M parameters, LOLM reaches **68.37 PPL on WikiText-103**, beating
**Pythia-410M (142.93 PPL)** by **52%** at matched compute with 26% fewer
parameters. It is a **base language model** — evaluate it on perplexity and
representation quality, not on instruction-following (it is not instruction-tuned).

## Architecture

Each token flows through five parallel streams that converge via learned fusion:

```
o_t = g · LN(W_h·h_t) + (1−g) · LN(W_z·z_t) + W_m·m_t + W_r·r_t
```

| Stream | Role | Implementation |
|--------|------|----------------|
| Surface decoder | local token relationships | pre-norm Transformer + RoPE |
| Latent SSM core | slow latent dynamics | selective SSM (Mamba-style) |
| Regime layer | discrete phase detection | Gumbel-Softmax + causal conv1d |
| Persistent memory | cross-sequence state | 3-bank gated read/write |
| Manifestation gate | surface↔latent arbitration | per-dimension sigmoid MLP |

## Results

| Metric | LOLM-304M | Pythia-410M | Δ |
|--------|-----------|-------------|---|
| Parameters | 304M | 410M | −26% |
| WikiText-103 eval PPL | **68.37** | 142.93 | **−52%** |
| Late-position BPC | 1.02 | 1.23 | −17% |
| Distinct-2 (generation) | 0.687 | 0.607 | +13% |

Inference-time ablations confirm every component contributes (regime, SSM, gate,
memory). See the paper for full tables and the 1.57B-scale comparison.

## Usage

LOLM is a custom architecture — load it with the LOLM code, not `transformers`:

```bash
git clone https://github.com/TheArtOfSound/LOLM.git && cd LOLM
pip install -r requirements.txt
```
```python
import torch, tiktoken
from lolm.config import load_config
from lolm.model import LOLM

cfg = load_config("configs/scale/300m_lolm_full_tpu.yaml")
model = LOLM(cfg.model)
ckpt = torch.load("ckpt_26000.pt", map_location="cpu")
model.load_state_dict(ckpt["model"], strict=False)
model.eval()
```

It exposes its own latent dynamics every step (`out.gate_values`,
`out.regime_probs`) — used by the NFET runtime governor in the
[lolm-bridge](https://github.com/TheArtOfSound/LOLM) workspace.

## License

Released under the **LOLM Community License Agreement v1.0** (see `LICENSE`):
free for academic research, education, and personal/non-commercial use;
commercial license required for qualifying entities.

## Citation

```bibtex
@article{leonard2026lolm,
  title  = {LOLM: Language Modeling Beyond the Surface with Hybrid
            Transformer-SSM Latent Order Fields},
  author = {Leonard, Bryan and Leonard, Brandyn},
  year   = {2026},
  note   = {Qira LLC. Provisional patent application No. 64002166.}
}
```