File size: 2,264 Bytes
5fb154d | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 | ---
license: mit
language:
- en
tags:
- liquid-neural-networks
- hybrid
- attention
- long-context
- text-generation
- edge
pipeline_tag: text-generation
---
# M1-128M — native hybrid (attention + liquid core), trained from scratch
**Honest status first:** research backbone, not a chat model. M1-128M is the
M-Series hybrid of the [AwareLiquid](https://github.com/AwareLiquid/M1) research
project — window attention + a liquid core (selective decay, exp parameterization)
in every layer, 12 layers, d_model 832, trained from scratch on WikiText-103.
English text continuation. Language-modeling quality is **not** the selling
point (see below); the case is the liquid memory mechanisms.
## What it is
- **128.6M params** — 12 layers × 832d, GQA (13 heads / 1 KV head)
- **Hybrid**: MicrotubuleAttention + MultiScaleResonance liquid core per layer
- Trained from scratch on WikiText-103 (50K steps), fp32, stable — no NaN
- Served live at [awareliquid.ai/demo](https://awareliquid.ai/demo) (CPU)
## Measured results (multi-seed, reproducible — RESULTS.md is canonical)
| Result | Number |
|---|---|
| LM quality at convergence (20K steps, 3 seeds) | modern Transformer **78.86 ± 0.25** < MT-LNN **88.93 ± 0.33** < simple Transformer **94.14 ± 0.78** |
| Cross-window associative recall (fast-weight) | **0.56 vs 0.000** (attention/LoRA) |
| Cross-session snapshot/restore | **bit-exact** round-trip |
| O(1) inference state (attention-free O-series variant only) | 0.381 MB flat → 8,063× smaller than a KV-cache at 1M tokens |
The hybrid M-series is **not** O(1) (it keeps attention); the O(1) claim belongs
to the attention-free O-series ([O1-48M](https://huggingface.co/AwareLiquid/O1-48M)).
## Serve it
```bash
CKPT_PATH=hybrid_125m_serve.pt TOKENIZER=gpt2 python -m uvicorn serve.server:app
```
## Paper
- [English PDF](https://github.com/AwareLiquid/M1/blob/main/mt_lnn_v2_reliable_long_pretraining_arxiv.pdf)
- [中文 PDF](https://huggingface.co/AwareLiquid/O1-48M/resolve/main/mt_lnn_arxiv_zh.pdf)
## Related
- Code: [AwareLiquid/M1](https://github.com/AwareLiquid/M1) (MIT)
- Live demo: [awareliquid.ai/demo](https://awareliquid.ai/demo)
- Model family: [awareliquid.ai/#models](https://awareliquid.ai/#models)
|