README.md

---
license: other
license_name: lfm-open-license-v1.0
license_link: https://huggingface.co/LiquidAI/LFM2.5-2.6B/blob/dca1825886789bd40b94368f53b1d9ada4c94598/LICENSE
base_model: LiquidAI/LFM2.5-2.6B
base_model_relation: quantized
pipeline_tag: text-generation
library_name: coreai
language:
- ar
- zh
- en
- fr
- de
- hi
- id
- it
- ja
- ko
- pl
- pt
- ru
- es
- th
- vi
tags:
- coreai
- aimodel
- aimodelc
- apple-silicon
- lfm2
- lfm2.5
- liquid
- edge
- int8
- community-model-zoo
---

# LFM2.5-2.6B — Core AI (int8)

An int8 Core AI conversion of **[LiquidAI/LFM2.5-2.6B](https://huggingface.co/LiquidAI/LFM2.5-2.6B)**
for Apple silicon. This repository contains no trained weights of its own: it is a quantized
format conversion of Liquid AI's model, and all model credit belongs to **Liquid AI**.

> **Runtime status:** runnable with public tooling, but not with an unmodified
> `apple/coreai-models` checkout. LFM's fixed-shape convolution state requires two patches from
> the public, community-maintained
> [`john-rocky/coreai-model-zoo`](https://github.com/john-rocky/coreai-model-zoo). Exact public
> commits and commands are below.

Converted from source revision `dca1825886789bd40b94368f53b1d9ada4c94598`. Both upstream
safetensors shards were SHA-256 verified against that revision before conversion.

## Runtime requirements

This graph has the normal growing key/value cache plus one fixed-shape `conv_state`. The stock
pipelined engine handles only the KV pair. Apply these **two** public community patches—no
per-token-input, static-input, or prefix-cache patch is required for this model:

1. `apps/coreai-shared-product.patch`
2. `apps/coreai-pipelined-extra-states.patch`

Both come from
[`john-rocky/coreai-model-zoo@95a29d41`](https://github.com/john-rocky/coreai-model-zoo/tree/95a29d41affed3bdf3ea5992ece094a908e21a04),
a public community repository maintained by Daisuke Majima (`john-rocky`). It is not an Apple
repository. The patches target Apple's public
[`coreai-models`](https://github.com/apple/coreai-models) runtime at
`5ed9981303b38d5a44aa6b45509bc4f6945029f5`.

```bash
git clone https://github.com/apple/coreai-models.git
git -C coreai-models checkout 5ed9981303b38d5a44aa6b45509bc4f6945029f5

git clone https://github.com/john-rocky/coreai-model-zoo.git
git -C coreai-model-zoo checkout 95a29d41affed3bdf3ea5992ece094a908e21a04

git -C coreai-models apply ../coreai-model-zoo/apps/coreai-shared-product.patch
git -C coreai-models apply ../coreai-model-zoo/apps/coreai-pipelined-extra-states.patch
swift build --package-path coreai-models -c release --product llm-runner
```

The public community zoo also supplies the LFM2 exporter and its `coreai-models` overlay. The
small toolkit published here contains the 2.6B-specific wrapper, tokenizer/config corrections,
quality gates, and locked Python environment used for this artifact. Full reproduction details
are in [`RECIPE.md`](RECIPE.md).

## Which file do I want?

**The repository root *is* the portable bundle.** The runtime wants a directory containing
`metadata.json`, the `*.aimodel/` graph and `tokenizer/`, and those sit at the top level here.
So the recommended download is the root minus the compiled variant — the exact command is under
[Download](#download).

`aimodelc-h16c/` is that same model pre-compiled for one GPU architecture: identical output and
speed, about half the cold-load time, and the runtime rejects it on a different architecture.
Take it only if you are on `h16c` and care about load time.

## What this is and is not

- It **is** a decode-optimized single-token-step graph, the shape a chat/completion loop uses.
- It **is not** a chunked-prefill or batch-serving asset.
- Quality here means **teacher-forced top-1 agreement and cosine similarity against an
  independent fp32 Hugging Face reference**, over 5 sequences / 125 positions. That is a
  regression probe, **not** a benchmark suite. No MMLU/GSM8K-style numbers are claimed.

## Artifacts

Full SHA-256 of each file as published.

| File | Bytes | SHA-256 | Recipe | Hardware scope | Use |
| --- | ---: | --- | --- | --- | --- |
| `lfm2_5_2_6b_decode_int8hu_attnfp16_block32_sym.aimodel/main.mlirb` | 3469367807 | `80540b2ee9183b756adb1ce51f334a94f13c093a554a2538aec66f9d9f3a07b3` | int8 blockwise-32, fp16 attention + embedding | any Apple silicon | **recommended** |
| `aimodelc-h16c/…h16c.aimodelc/…/resources.bin` | 3468949676 | `793907c101a01331ffa72aff7c9db5049c89a02ea03acee3174513a93c394e22` | same weights, AOT compiled | **`h16c` only** | faster cold load |
| `aimodelc-h16c/…h16c.aimodelc/…/original_model_0.mpsgraph` | 363905 | `74a24870a8853bf797a6f12f40e7fcb988de014d98166da772b24cab643d5d4f` | compiled graph | `h16c` only | part of the above |
| `tokenizer/tokenizer.json` | 17905598 | `695be7802a0e4b8a81048f0ff5ebb7fc811a0ba5a6be63dbb24deb5a81096f41` | upstream, unmodified | — | required |
| `lfm2.5-2.6b-coreai-conversion-e0e375b.tar.gz` | 57118 | `665285044be7c0171e1b737167b9892f2799bc2e01e3c5441212887903456f2d` | exact wrapper + gates at local conversion commit `e0e375b` | — | reproduction |

- **The two `tokenizer/tokenizer.json` copies are byte-identical** (same hash above). The one
  under `aimodelc-h16c/tokenizer/` is a **convenience copy**; you do not need both.
- Portable and compiled are **the same model**, published together because the compiled one
  halves cold load but only runs on one architecture, so neither dominates.
- Both were produced from the same pinned commit and have identical compiled storage budgets.

## Recipe, and what was rejected

Oracle for every quality number: **an independent fp32 reference, Hugging Face's own
`Lfm2ForCausalLM` at transformers ≥ 5.2**, teacher-forced over 5 sequences / 125 positions.
Cosine is the minimum per-position cosine.

| Variant | Bundle | Top-1 vs fp32 oracle | Min cosine | Conversion | Shipped |
| --- | ---: | ---: | ---: | ---: | --- |
| **int8 blockwise-32, fp16 attention + embedding** | 3.25 GB | **122/125** | **0.997050** | 5/5 | **yes** |
| + attention q/k/v/out to int8 | 3.19 GB | 121/125 | 0.996949 | 5/5 | no |
| + embedding to int8 | 3.03 GB | 123/125 | 0.996848 | 5/5 | no |
| attention at fp32 (converter default) | 3.42 GB | — | 0.997210 | 5/5 | no |
| int4 blockwise-32 family | 2.07–2.34 GB | — | 0.51–0.80 | 4–5/5 | no |

The shipped arm has the highest minimum cosine of the int8 arms. The ±1 position differences
between the three int8 rows are near-tie argmax flips at 125 positions, not a systematic
ordering — do not read the embedding row's 123 as "better". int4 is excluded on **quality**, not
size: a minimum cosine of 0.51–0.80 is a different model. **Rejected variants are documented and
deliberately not uploaded.** Full detail in [`RECIPE.md`](RECIPE.md).

## Performance

Decode here is **memory-bandwidth bound** — throughput tracks bytes read per token.

| Measurement | Protocol | Result |
| --- | --- | --- |
| This published artifact | 128 prompt tokens, 256 generated, greedy, 5 trials, M4 Max | **107.98 decode tok/s**, sd 0.500; **111.86 prompt tok/s** |
| Earlier development build of the same recipe | separate runs on the same M4 Max | 138.15 decode tok/s |

Only **107.98 tok/s** is attached to the published artifact in `evidence/benchmark.json`. The
earlier 138.15 result was not reproduced after publication, so it is retained as historical
context rather than advertised as this download's performance. The workload is
memory-bandwidth-bound and was measured while the machine was busy; measure on your own hardware.

For reference, on the same Mac, MLX reported ~100 tok/s at 8-bit and ~60 tok/s at BF16. Those
were throughput-only observations; MLX quality was not measured, so no quality comparison against
MLX is claimed.

AOT compilation is throughput-neutral within noise and halves cold load, 9.8 s → 5.1 s.

## Reproduce

Every pin, the full quantization spec, the four gates, the measurement protocol and the rejected
variants are in **[`RECIPE.md`](RECIPE.md)**. Summary:

```
Source model : LiquidAI/LFM2.5-2.6B @ dca1825886789bd40b94368f53b1d9ada4c94598          (public)
Converter    : github.com/apple/coreai-models @ b1cb71b8522d99408059fa0b98b8742171bcb0b8  (public)
Runtime      : github.com/apple/coreai-models @ 5ed9981303b38d5a44aa6b45509bc4f6945029f5  (public)
Community LFM2 exporter, overlay and two runtime patches:
               github.com/john-rocky/coreai-model-zoo
               @ 95a29d41affed3bdf3ea5992ece094a908e21a04                     (public)
2.6B wrapper : lfm2.5-2.6b-coreai-conversion-e0e375b.tar.gz
               SHA-256 665285044be7c0171e1b737167b9892f2799bc2e01e3c5441212887903456f2d
Toolchain    : macOS 27.0 (26A5388g), Xcode 27.0 (27A5228h), Python 3.11.15,
               torch 2.9.0, coreai-core 1.0.0b2, coreai-torch 0.4.1,
               coreai-opt 0.2.1, coremltools 9.0
```

**Verify a rebuild by the gates and the storage budget, not by hashing.** The exporter names each
externalized call site with a generated UUID (391 in this graph), so two exports of identical
weights differ in a few bytes and therefore in SHA-256. The budget to match is
`Int8 2,621,243,392` + `Float16 428,342,276` + `Float32 34` + small index types.

## Download

```bash
# The portable bundle: the .aimodel graph, the tokenizer and the bundle metadata.
# This is the whole repository except the compiled h16c variant (~3.5 GB rather than ~7 GB).
hf download harshav/LFM2.5-2.6B-CoreAI \
  --exclude 'aimodelc-h16c/*' \
  --local-dir ./LFM2.5-2.6B-CoreAI

# The directory you then point the runtime at is ./LFM2.5-2.6B-CoreAI itself.
```

With a runtime built as described above, the asset is driven as a pipelined Core AI language
model with `COREAI_CHUNK_THRESHOLD=1`, greedy decoding, using the model's own chat template at
`tokenizer/chat_template.jinja`. Operational notes:

```bash
COREAI_CHUNK_THRESHOLD=1 coreai-models/.build/release/llm-runner \
  --model ./LFM2.5-2.6B-CoreAI \
  --prompt "The capital of France is" \
  --max-tokens 64 --temperature 0 \
  --inference-engine-variant coreai-pipelined \
  --warmup exact --warmup-length 1
```

- **macOS 27.0+** on **Apple silicon**. Validated on M4 Max.
- Runs on the **GPU** via an `MPSGraph` delegate. This is **not** an ANE asset: the KV dimension
  is dynamic, and `--preferred-compute` does not change the emitted delegate.
- Budget roughly bundle size plus KV cache, about **5 GB** at 4096 context.
- `aimodelc-h16c/` is **architecture-locked to `h16c`**; the runtime names the architecture it
  wanted when it refuses.

## Validation evidence

Machine-readable under `evidence/`: `authored_parity_vs_huggingface.json`, `fp32_reference.json`,
`recipe_quality.json`, `conversion_gate.json`, `benchmark.json`, `compiled_storage_stats.json`.

Four **separate** questions, not interchangeable:

- **Authoring fidelity** — re-authored module vs Hugging Face, fp32: 21/21 top-1, cosine
  1.000000. This caught a real bug: the checkpoint sets `rope_parameters.rope_theta = 1e7`, and
  code reading only the legacy top-level key silently defaults to `1e6` — a 10× wrong RoPE that
  still produces fluent short text. Both the converter overlay and transformers 4.x hit it.
- **Quantization damage** — vs the fp32 oracle: 122/125, min cosine 0.997050.
- **Conversion fidelity** — bundle vs **its own** quantized weights run eagerly: 5/5 exact.
  Comparing to fp32 here would conflate quantization damage with conversion bugs.
- **Throughput** — see the caveat above.

## Limitations and negative results

- Requires the two public community runtime patches listed above; it does not run on an
  unmodified `apple/coreai-models` runtime.
- No task-benchmark evaluation; quality is a 125-position regression probe.
- Multilingual support is inherited from upstream and was **not** re-verified per language; the
  probe is English.
- int4 rejected at block 32 (cosine 0.51–0.66) and at block 16 (quality recovers, 42 tok/s,
  ~3× slower than int8).
- `--expect-frequent-reshapes` measured 84 tok/s and 8.3 GB; not used.
- Speculative decoding not shipped: a static-S verify graph exports and its contract gates, but
  per-position logits do not match stepped decode.
- 300 tok/s was a target and was not reached by any tested configuration.

## License and attribution

The model is **Liquid AI's**. This repository redistributes a converted, quantized copy under the
upstream license, and claims **no authorship of the model**.

- Upstream: [LiquidAI/LFM2.5-2.6B](https://huggingface.co/LiquidAI/LFM2.5-2.6B) by **Liquid AI**.
- Upstream license: **LFM Open License v1.0** — pinned copy
  [here](https://huggingface.co/LiquidAI/LFM2.5-2.6B/blob/dca1825886789bd40b94368f53b1d9ada4c94598/LICENSE),
  included verbatim as [`LICENSE.upstream`](LICENSE.upstream). Your use of these weights is
  governed by it.
- Official Apple tooling: [`apple/coreai-models`](https://github.com/apple/coreai-models),
  `coreai-core`, `coreai-torch`, and `coreai-opt`.
- Community conversion/runtime work: public
  [`john-rocky/coreai-model-zoo`](https://github.com/john-rocky/coreai-model-zoo), maintained
  by **Daisuke Majima**, under its BSD-3-Clause license. Its repository includes third-party
  notices for portions derived from Apple's BSD-licensed `coreai-models` project.
- This repository contributes the conversion recipe, the gates, and the measurements.