LFM2.5-2.6B — LiteRT-LM

LiquidAI/LFM2.5-2.6B converted to the LiteRT-LM (.litertlm) format for on-device inference with Google's LiteRT-LM runtime (requires litert-lm ≥ 0.14; the bundled executor-metadata section makes the same file run on 0.15's new state binding as well).

LFM2.5-2.6B is the flagship of Liquid AI's LFM2.5 on-device family: a hybrid architecture (22 gated short-convolution blocks + 8 grouped-query attention layers) and a thinking model — it reasons inside a <think>…</think> block before answering. The bundle declares the thought channel, so LiteRT-LM apps receive the reasoning and the answer as separate streams.

File	Recipe	Size	GSM8K (n=100)
LFM2.5-2.6B_int8.litertlm	int8 dynamic (linears + convs + embedding)	2.87 GB	88% (bf16 reference: 92%)
LFM2.5-2.6B_int4.litertlm	int4 blockwise-32 + OCTAV linears, int8 embedding, convs float	1.67 GB	83%

Context (KV cache)	4096 max (see --max-num-tokens note)
Backend	CPU (the hybrid conv graph is not supported by current mobile GPU delegates)
Template	bundled — ChatML-style with tool-list support; generation prompt pre-fills <think> (the model's trained convention); thought channel declared
Base model	LiquidAI/LFM2.5-2.6B (LFM Open License v1.0)

Quality — GSM8K

GSM8K, greedy, 0-shot chain-of-thought, max-tokens 2048 (thinking model), n=100, same harness for all rows:

Configuration	GSM8K
PyTorch bf16 (reference, MPS)	92%
LiteRT int8 (this repo)	88%
LiteRT int4-b32 OCTAV (this repo)	83%

Both files pass an 8-question sanity gate 8/8 with zero degenerate outputs (CPU), a 42-length prefill sweep with zero corrupt first tokens (fresh engine per length), and a 3-turn conversation gate (fact recall + arithmetic).

Usage

pip install litert-lm
litert-lm run ./LFM2.5-2.6B_int4.litertlm --prompt "What is 17 + 25?"

The model thinks before answering — through the Python/C++ API the reasoning arrives on the thought channel and the reply stays clean; the CLI prints both.

--max-num-tokens tip: reasoning turns are long — give the model at least 2048 tokens of budget for math/complex questions, or the think block may not close. Decode speed drops as the token budget grows, so don't set it higher than you need.

Multi-turn note: on the current runtime the conversation context retains previous turns' reasoning, so long chats fill the context faster than the visible text suggests. For unrelated questions, prefer fresh conversations.

Speed

litert-lm benchmark, CPU backend, Mac M4 Max, max-num-tokens 1024 (prefill-1024 rows measured at max-num-tokens 2048):

Variant	Prefill (256)	Prefill (1024)	Decode	TTFT
int8	169 tok/s	434 tok/s	38.0 tok/s	1.54 s
int4	156 tok/s	193 tok/s	43.7 tok/s	1.66 s

iPhone 17 Pro (CPU backend, cold first runs): the int4 file answers the 8-question quality gate 8/8 and decodes at ~20 tok/s warm (the first, cold run is slower while caches build). The int8 file also loads and runs on iPhone 17 Pro when the host app carries the extended-virtual-addressing / increased-memory entitlements (its 2.87 GB single weight section exceeds what a default-entitlement app will memory-map) — int4 is the recommended phone variant (smaller, faster on phone-class memory bandwidth).

Run on Android

Install a recent Google AI Edge Gallery (1.0.16+ imports .litertlm directly from Hugging Face), import this repo (or adb push a file and use local import), select the CPU backend, and chat.

Conversion notes

Converted with released litert-torch 0.9.2 (upstream lfm2 hybrid support incl. the ShortConv prefill-pad fix) and packaged for litert-lm ≥ 0.14 (the executor-metadata section binds the 22 conv states + 16 KV caches; files exported without it do not run on 0.15).

• The generation prompt pre-fills <think> exactly as the vendor chat template does. This matters: with a bare assistant prompt, think-block emission becomes the model's choice, and the int4 quantization degrades that discipline first (unscaffolded rambling in place of answers on multi-turn). With the pre-fill, all variants answer tersely and the thought channel routes cleanly.
• The checkpoint is notably sparse: OCTAV int4-b32 produced 746k all-zero weight blocks whose zero scales the XNNPACK delegate rejects at load. The zero scales are patched to the tensor's smallest nonzero scale (dequantization unchanged — the blocks are all zeros).
• int8 quantizes convs at export time (safe); post-hoc conv int8 breaks generation. int4 keeps convs float.
• Multi-length prefill signatures (1–1024) are exported so the runtime picks tight chunks; a 42-length first-token sweep verifies no prefill-padding state corruption.