README.md

---
license: other
license_name: lfm-open-license-v1.0
license_link: LICENSE
base_model: LiquidAI/LFM2.5-1.2B-Instruct
tags:
  - litert
  - litert-lm
  - litertlm
  - on-device
  - edge
  - hybrid
  - liquid
pipeline_tag: text-generation
library_name: litert-lm
---

# LFM2.5-1.2B-Instruct — LiteRT-LM

[LiquidAI/LFM2.5-1.2B-Instruct](https://huggingface.co/LiquidAI/LFM2.5-1.2B-Instruct) converted to the **LiteRT-LM** (`.litertlm`) format for on-device inference with Google's [LiteRT-LM](https://github.com/google-ai-edge/litert-lm) runtime (requires **litert-lm ≥ 0.14** / a recent AI Edge Gallery).

> **Update (2026-08-04):** the `.litertlm` files were updated in place to add the `ExecutorMetadata` section that **litert-lm ≥ 0.15** requires to bind the hybrid conv/attention state buffers (without it, 0.15 fails at inference with `missing some output TensorBuffers`). Weights and graph are byte-identical to the original release, and the files continue to run on litert-lm 0.14.

LFM2.5-1.2B-Instruct is Liquid AI's flagship on-device model: a **hybrid architecture** mixing gated short-convolution blocks with grouped-query attention. Per the base model card it outperforms Qwen3-1.7B and Gemma 3 1B on knowledge and instruction-following benchmarks at a smaller footprint.

| File | Recipe | Size | GSM8K (n=100) |
|---|---|---|---|
| `LFM2.5-1.2B-Instruct_int8.litertlm` | int8 dynamic (linears + convs + embedding) | 1.25 GB | **81%** (bf16 reference: 79%) |
| `LFM2.5-1.2B-Instruct_int4.litertlm` | int4 blockwise-32 + OCTAV linears, int8 embedding, convs float | 736 MB | 72% |

| | |
|---|---|
| **Context (KV cache)** | 4096 max (see `--max-num-tokens` note) |
| **Backend** | **CPU** (the hybrid conv graph is not supported by current GPU delegates) |
| **Template** | bundled — full chat template incl. tool-list support + `<think>` channel declaration |
| **Base model** | LiquidAI/LFM2.5-1.2B-Instruct (LFM Open License v1.0) |

## Quality — GSM8K

GSM8K, greedy, 0-shot chain-of-thought, max-tokens 1024, n=100, same harness for all rows:

| Configuration | GSM8K |
|---|---|
| PyTorch bf16 (reference) | 79% |
| **LiteRT int8 (this repo)** | **81%** |
| LiteRT int4-b32 OCTAV (this repo) | 72% |

The int8 file is at full parity with the PyTorch reference (+2pt, within noise). Both files pass an 8-question sanity gate (7/8, zero degenerate outputs). The int4 file trades −7pt for a 41% smaller download.

## Usage

```bash
litert-lm run ./LFM2.5-1.2B-Instruct_int8.litertlm --prompt "Explain what a hybrid conv-attention LLM is in two sentences."
```

The bundle carries the tokenizer and the full chat template (ChatML-style `<|im_start|>…<|im_end|>` with tool-calling support). Generation stops cleanly at `<|im_end|>`.

**`--max-num-tokens` tip:** decode speed drops as the token budget grows (int8 decode: 101 tok/s at 1024 → 77 tok/s at 4096 on an M4 Max) — set it to the smallest value your use case needs (1024 is a good chat default; the file allows up to 4096).

## Speed

`litert-lm benchmark`, CPU backend, Mac M4 Max, max-num-tokens 1024:

| Variant | Prefill (256) | Prefill (1024) | Decode | TTFT |
|---|---|---|---|---|
| int8 | 402 tok/s | 1603 tok/s | 101 tok/s | 0.65 s |
| int4 | 304 tok/s | — | 91 tok/s | 0.85 s |

On a Pixel 8a (Tensor G3, CPU backend, measured in AI Edge Gallery): int8 decodes at ~19 tok/s and int4 at ~31 tok/s — on phone-class memory bandwidth the int4 file is ~1.7× faster as well as 41% smaller, so prefer int4 on mid-range devices unless you need the last few GSM8K points.

The hybrid short-conv blocks are what make a 1.2B decode at ~100 tok/s on CPU — faster than same-size pure-attention models. Long prompts prefill through the large (up to 1024-token) signatures at >1500 tok/s.

## Run on Android

Install a recent [Google AI Edge Gallery](https://github.com/google-ai-edge/gallery) (1.0.16+ imports `.litertlm` directly from Hugging Face), import this repo (or `adb push` a file and use local import), select the **CPU** backend, and chat.

## Conversion notes

Converted with released [`litert-torch`](https://github.com/google-ai-edge/litert) 0.9.1 (which ships the `lfm2` hybrid model support) and packaged for litert-lm ≥ 0.14, with one exporter patch: the stock LFM2 short-conv block saves its conv state from the **last columns of the prefill chunk**, which are padding whenever the prompt does not exactly fill a prefill signature — corrupting the first generated token of nearly every reply. The patch derives the valid length from the attention mask in-graph and gathers the conv state from the last *valid* columns instead; with it, engine output is token-identical to an exact per-token reference loop at every prompt length. Multi-length prefill signatures (1–1024) are exported so the runtime can pick tight chunks.

Quantization targets fully-connected + embedding ops (plus convs in the int8 file, quantized at export time); post-hoc int8 of the convs with ai-edge-quantizer breaks generation — quantize convs only at export time.

## License and changes

Distributed under the **LFM Open License v1.0** (see `LICENSE`, inherited from the base model). Note the license's commercial-use limitation for organizations above US$10M annual revenue. **Changes from the original work:** weights converted from safetensors bf16 to LiteRT flatbuffers and quantized as described above; tokenizer and chat template repackaged unmodified into the `.litertlm` bundle; exporter conv-state fix as described in Conversion notes. This repository is a community conversion and is not affiliated with Liquid AI.