metadata
license: mit
language:
- en
tags:
- liquid-neural-networks
- hybrid
- attention
- long-context
- text-generation
- edge
pipeline_tag: text-generation
M1-128M — native hybrid (attention + liquid core), trained from scratch
Honest status first: research backbone, not a chat model. M1-128M is the M-Series hybrid of the AwareLiquid research project — window attention + a liquid core (selective decay, exp parameterization) in every layer, 12 layers, d_model 832, trained from scratch on WikiText-103. English text continuation. Language-modeling quality is not the selling point (see below); the case is the liquid memory mechanisms.
What it is
- 128.6M params — 12 layers × 832d, GQA (13 heads / 1 KV head)
- Hybrid: MicrotubuleAttention + MultiScaleResonance liquid core per layer
- Trained from scratch on WikiText-103 (50K steps), fp32, stable — no NaN
- Served live at awareliquid.ai/demo (CPU)
Measured results (multi-seed, reproducible — RESULTS.md is canonical)
| Result | Number |
|---|---|
| LM quality at convergence (20K steps, 3 seeds) | modern Transformer 78.86 ± 0.25 < MT-LNN 88.93 ± 0.33 < simple Transformer 94.14 ± 0.78 |
| Cross-window associative recall (fast-weight) | 0.56 vs 0.000 (attention/LoRA) |
| Cross-session snapshot/restore | bit-exact round-trip |
| O(1) inference state (attention-free O-series variant only) | 0.381 MB flat → 8,063× smaller than a KV-cache at 1M tokens |
The hybrid M-series is not O(1) (it keeps attention); the O(1) claim belongs to the attention-free O-series (O1-48M).
Serve it
CKPT_PATH=hybrid_125m_serve.pt TOKENIZER=gpt2 python -m uvicorn serve.server:app
Paper
Related
- Code: AwareLiquid/M1 (MIT)
- Live demo: awareliquid.ai/demo
- Model family: awareliquid.ai/#models