| --- |
| license: mit |
| language: |
| - en |
| tags: |
| - liquid-neural-networks |
| - hybrid |
| - attention |
| - long-context |
| - text-generation |
| - edge |
| pipeline_tag: text-generation |
| --- |
| |
| # M1-128M — native hybrid (attention + liquid core), trained from scratch |
|
|
| **Honest status first:** research backbone, not a chat model. M1-128M is the |
| M-Series hybrid of the [AwareLiquid](https://github.com/AwareLiquid/M1) research |
| project — window attention + a liquid core (selective decay, exp parameterization) |
| in every layer, 12 layers, d_model 832, trained from scratch on WikiText-103. |
| English text continuation. Language-modeling quality is **not** the selling |
| point (see below); the case is the liquid memory mechanisms. |
| |
| ## What it is |
| |
| - **128.6M params** — 12 layers × 832d, GQA (13 heads / 1 KV head) |
| - **Hybrid**: MicrotubuleAttention + MultiScaleResonance liquid core per layer |
| - Trained from scratch on WikiText-103 (50K steps), fp32, stable — no NaN |
| - Served live at [awareliquid.ai/demo](https://awareliquid.ai/demo) (CPU) |
| |
| ## Measured results (multi-seed, reproducible — RESULTS.md is canonical) |
| |
| | Result | Number | |
| |---|---| |
| | LM quality at convergence (20K steps, 3 seeds) | modern Transformer **78.86 ± 0.25** < MT-LNN **88.93 ± 0.33** < simple Transformer **94.14 ± 0.78** | |
| | Cross-window associative recall (fast-weight) | **0.56 vs 0.000** (attention/LoRA) | |
| | Cross-session snapshot/restore | **bit-exact** round-trip | |
| | O(1) inference state (attention-free O-series variant only) | 0.381 MB flat → 8,063× smaller than a KV-cache at 1M tokens | |
| |
| The hybrid M-series is **not** O(1) (it keeps attention); the O(1) claim belongs |
| to the attention-free O-series ([O1-48M](https://huggingface.co/AwareLiquid/O1-48M)). |
| |
| ## Serve it |
| |
| ```bash |
| CKPT_PATH=hybrid_125m_serve.pt TOKENIZER=gpt2 python -m uvicorn serve.server:app |
| ``` |
| |
| ## Paper |
| |
| - [English PDF](https://github.com/AwareLiquid/M1/blob/main/mt_lnn_v2_reliable_long_pretraining_arxiv.pdf) |
| - [中文 PDF](https://huggingface.co/AwareLiquid/O1-48M/resolve/main/mt_lnn_arxiv_zh.pdf) |
| |
| ## Related |
| |
| - Code: [AwareLiquid/M1](https://github.com/AwareLiquid/M1) (MIT) |
| - Live demo: [awareliquid.ai/demo](https://awareliquid.ai/demo) |
| - Model family: [awareliquid.ai/#models](https://awareliquid.ai/#models) |
| |
