README.md

---
license: mit
tags:
- cpu
- gguf
- llama.cpp
- ternary
- mixture-of-experts
- edge
---

# CPU-Only Inference Models

Small, efficient LLMs that run well on **ordinary CPUs — no GPU needed**. All models
below were measured on a modest 2019-era laptop CPU:

> **thinkpad2**: Intel Core i5-8350U (4C/8T, AVX2), 64 GB DDR4-2400, AC power, 4 threads.

If your machine has no discrete GPU (or you want to keep the GPU free), these are the
models and setups that actually work — with real token/s numbers, not promises.

---

## ⭐ The star: Maple Preview 20B-A1B (ternary 2-bit)

**The fastest useful model on a CPU we have found — over 28 tokens/s on a 4-core laptop.**

| Model | Size | Quant | CPU decode (thinkpad2, 4 threads) |
|---|---|---|---|
| **Maple Preview** (20B-A1B, 256-expert MoE, 8 active) | 5.5 GiB | TQ2_0 ternary, **2.06 bpw** | **33.9 t/s** (tg128) · **28.2 t/s** (benchy tg64) |

Maple Preview is DeepGrove's open-source reasoning model, designed from the start for
efficient on-device inference (24 layers, 3:1 SWA-512:GA attention, 131k context, MIT
license). It is the **real star of this collection**: on our CPU it runs at
**28–34 tokens/s** — comfortably interactive — while its 20B total / 1B-active ternary
weights keep it to a 5.5 GB file that fits any machine with 16 GB of RAM.

Reference points (from the DeepGrove team and our own measurements):
- **this CPU (i5-8350U, 4 threads)**: prompt 512 tok → 100.8 t/s · decode tg128 → 33.9 t/s
- **Apple M2 Max CPU**: ~360 t/s prompt · ~77 t/s decode
- **Apple M4 mini (DeepGrove's measurement)**: 200+ t/s

The engine is the **DeepGrove llama.cpp fork** (the `maple` architecture + TQ2_0
support); the GGUF we use is their `maple-preview-TQ2_0-head-Q4_K.gguf`.

- Model: [deepgrove/maple-preview](https://huggingface.co/deepgrove/maple-preview) (MIT)
- GGUF: [deepgrove/maple-preview-GGUF](https://huggingface.co/deepgrove/maple-preview-GGUF)
- Engine: [deepgrove-ai/llama.cpp](https://github.com/deepgrove-ai/llama.cpp)
- Announcement: [deepgrove on X](https://x.com/deepgrove_ai/status/2085190212427411715)

```bash
llama-server -m maple-preview-TQ2_0-head-Q4_K.gguf \
  --ctx-size 131072 --cache-type-k q8_0 --cache-type-v q8_0 \
  --threads 4   # 4 beats 8 on this CPU (33.9 vs 21.8 t/s)
```

---

## LiquidAI LFM2.5 family — mixed 4-bit GGUFs by ljupco

The LFM2.5 models (2.6B dense, 1.2B-Thinking, 8B-A1B MoE) with mixed quantizations:
the bulk of the weights at 4-bit, the most sensitive tensors kept at higher precision.

| Model | GGUF repo | Quant | CPU decode (thinkpad2, 4 threads) |
|---|---|---|---|
| LFM2.5-2.6B | [ljupco/LFM2.5-2.6B-GGUF](https://huggingface.co/ljupco/LFM2.5-2.6B-GGUF) | Q4_K_M | **13.1 t/s** (benchy tg64) |
| LFM2.5-1.2B-Thinking | [ljupco/LFM2.5-1.2B-Thinking-GGUF](https://huggingface.co/ljupco/LFM2.5-1.2B-Thinking-GGUF) | Q4_0h | **25.3 t/s** (benchy tg64) |
| LFM2.5-8B-A1B | [ljupco/LFM2.5-8B-A1B-GGUF](https://huggingface.co/ljupco/LFM2.5-8B-A1B-GGUF) | Q4_0h | **13.8 t/s** (benchy tg64) |

Original models by **Liquid AI** — [LFM2.5-2.6B](https://huggingface.co/LiquidAI/LFM2.5-2.6B),
[LFM2.5-1.2B-Thinking](https://huggingface.co/LiquidAI/LFM2.5-1.2B-Thinking),
[LFM2.5-8B-A1B](https://huggingface.co/LiquidAI/LFM2.5-8B-A1B).

```bash
llama-server -m LFM2.5-2.6B-Q4_K_M.gguf --threads 4
```

---

## The full three-engine report

These numbers come from a systematic porting and benchmarking exploration across three
engines — stock llama.cpp (with the DeepGrove fork for Maple), `ik-llama.cpp`, and
`vllm.cpp` — including kernel-level work (fused ops, integer-dot kernels, a ternary
gemv) and a detailed analysis of why the DeepGrove 8x8 gemv cannot run on
standard-quantized weights:

**[The LFM2.5 / Maple Preview three-engine report](docs/lfm2-maple-4bit-report.md)**

---

## Credits and Acknowledgements

This collection is entirely built on the work of others, and we are deeply grateful:

- **DeepGrove AI** — for the Maple Preview model, the TQ2_0 ternary quantization, and
  their llama.cpp fork with the Maple architecture support. The ternary design and the
  on-device focus are what make 28+ t/s on a CPU possible. Thank you!
- **Liquid AI** — for the LFM2.5 family and its gated-delta / shortconv architecture,
  and for publishing the weights openly. Thank you!
- **llama.cpp / ggml** — the core inference engine and its maintainers and contributors.
- The **HuggingFace / GGUF** ecosystem for the format, the tooling, and the platform.

Any remaining errors are ours. Benchmark numbers are single-machine measurements;
expect ±10–20% day-to-day noise.