| --- |
| license: other |
| license_name: lolm-community-license-1.0 |
| license_link: LICENSE |
| library_name: lolm |
| tags: |
| - language-model |
| - hybrid-transformer-ssm |
| - state-space-model |
| - mamba |
| - latent-order |
| - research |
| pipeline_tag: text-generation |
| --- |
| |
| # LOLM-304M — Latent Order Language Model |
|
|
| **LOLM** is a hybrid Transformer–SSM architecture that separates surface token |
| prediction from latent state tracking. This is the **304M** research checkpoint |
| behind the published WikiText-103 results. |
|
|
| > Authors: Bryan Leonard & Brandyn Leonard · Qira LLC · Provisional patent 64002166 |
| > Code: https://github.com/TheArtOfSound/LOLM |
|
|
| ## TL;DR |
|
|
| At 304M parameters, LOLM reaches **68.37 PPL on WikiText-103**, beating |
| **Pythia-410M (142.93 PPL)** by **52%** at matched compute with 26% fewer |
| parameters. It is a **base language model** — evaluate it on perplexity and |
| representation quality, not on instruction-following (it is not instruction-tuned). |
|
|
| ## Architecture |
|
|
| Each token flows through five parallel streams that converge via learned fusion: |
|
|
| ``` |
| o_t = g · LN(W_h·h_t) + (1−g) · LN(W_z·z_t) + W_m·m_t + W_r·r_t |
| ``` |
|
|
| | Stream | Role | Implementation | |
| |--------|------|----------------| |
| | Surface decoder | local token relationships | pre-norm Transformer + RoPE | |
| | Latent SSM core | slow latent dynamics | selective SSM (Mamba-style) | |
| | Regime layer | discrete phase detection | Gumbel-Softmax + causal conv1d | |
| | Persistent memory | cross-sequence state | 3-bank gated read/write | |
| | Manifestation gate | surface↔latent arbitration | per-dimension sigmoid MLP | |
|
|
| ## Results |
|
|
| | Metric | LOLM-304M | Pythia-410M | Δ | |
| |--------|-----------|-------------|---| |
| | Parameters | 304M | 410M | −26% | |
| | WikiText-103 eval PPL | **68.37** | 142.93 | **−52%** | |
| | Late-position BPC | 1.02 | 1.23 | −17% | |
| | Distinct-2 (generation) | 0.687 | 0.607 | +13% | |
|
|
| Inference-time ablations confirm every component contributes (regime, SSM, gate, |
| memory). See the paper for full tables and the 1.57B-scale comparison. |
|
|
| ## Usage |
|
|
| LOLM is a custom architecture — load it with the LOLM code, not `transformers`: |
|
|
| ```bash |
| git clone https://github.com/TheArtOfSound/LOLM.git && cd LOLM |
| pip install -r requirements.txt |
| ``` |
| ```python |
| import torch, tiktoken |
| from lolm.config import load_config |
| from lolm.model import LOLM |
| |
| cfg = load_config("configs/scale/300m_lolm_full_tpu.yaml") |
| model = LOLM(cfg.model) |
| ckpt = torch.load("ckpt_26000.pt", map_location="cpu") |
| model.load_state_dict(ckpt["model"], strict=False) |
| model.eval() |
| ``` |
|
|
| It exposes its own latent dynamics every step (`out.gate_values`, |
| `out.regime_probs`) — used by the NFET runtime governor in the |
| [lolm-bridge](https://github.com/TheArtOfSound/LOLM) workspace. |
|
|
| ## License |
|
|
| Released under the **LOLM Community License Agreement v1.0** (see `LICENSE`): |
| free for academic research, education, and personal/non-commercial use; |
| commercial license required for qualifying entities. |
|
|
| ## Citation |
|
|
| ```bibtex |
| @article{leonard2026lolm, |
| title = {LOLM: Language Modeling Beyond the Surface with Hybrid |
| Transformer-SSM Latent Order Fields}, |
| author = {Leonard, Bryan and Leonard, Brandyn}, |
| year = {2026}, |
| note = {Qira LLC. Provisional patent application No. 64002166.} |
| } |
| ``` |
|
|
