README.md

metadata

license: other
license_name: lfm-open-license-v1.0
license_link: LICENSE
base_model: LiquidAI/LFM2.5-8B-A1B
pipeline_tag: text-generation
library_name: core-ai
tags:
  - core-ai
  - coreml
  - apple
  - moe
  - on-device
  - metal

LFM2.5-8B-A1B — Core AI (the zoo's first MoE on iPhone)

Apple Core AI (.aimodel) conversion of LiquidAI/LFM2.5-8B-A1B: a conv + full-attention MoE hybrid decoder (24 layers = 18 short-conv mixers + 6 GQA attention; hidden 2048, vocab 128k; first 2 layers dense, the rest 32-expert top-4 sparse MoE). 8.3B total / ~1.5B active per token.

Part of the community Core AI model zoo: https://github.com/john-rocky/coreai-model-zoo (full card: zoo/lfm2.5-8b-a1b-moe.md).

Use it

▶️ Run it (source) — the ChatDemo runner (GUI + CLI, one app for every chat model in the catalog):

git clone https://github.com/john-rocky/coreai-kit
open coreai-kit/Examples/ChatDemo/ChatDemo.xcodeproj
# → Run, then pick "LFM2.5-8B-A1B (MoE)" in the model picker

# agents / headless (macOS):
cd coreai-kit/Examples/ChatDemo
swift run chat-cli --model lfm2.5-8b-a1b --prompt "What can you do, offline?"

💻 Build with it — complete; the glue is kit API, copy-paste runs:

import CoreAIKit

let chat = try await ChatSession(catalog: "lfm2.5-8b-a1b")
let reply = try await chat.respond(to: prompt)
// reply: the answer, generated fully on-device

The take-home is Examples/ChatDemo/Sources/QuickStart.swift — this exact code as one typed function, no UI; the CLI is an argument shell over it, and the GUI drives the same ChatSession across turns for its transcript. Multi-turn? Hold the ChatSession and call respond(to:) per turn — it keeps the conversation history; streamResponse(to:) yields tokens as they decode.

Integration checklist

• SPM: https://github.com/john-rocky/coreai-kit → product CoreAIKit
• Info.plist: none needed
• Entitlements: none needed (macOS)
• First run downloads the model — 9.0 GB (Mac) — then it loads from the local cache (Application Support; progress via the downloadProgress callback)
• Measure in Release — Debug is ~3× slower on per-token host work

The gather_qmm kernel

MoE decode normally reads all 32 experts' weights every token via the GatherMM composite even though only the top-4 are routed — bandwidth-bound at 39 tok/s. This bundle uses a custom coreai_torch.TorchMetalKernel that takes the routed indices as a kernel input and reads only the 4 routed experts' weight slabs → 3.6× faster (141 tok/s) at the same active-param bandwidth.

Bundles & honest quality

Shipped here (Mac-only):

dir	size	platform	decode tok/s	quality (fp32-oracle margin gate)
gpu-pipelined/lfm2_5_8b_a1b_decode_sym8_gather/	8.8 GB	Mac	140	CLEAN — +1 flip/41 (= fp16 ceiling) ✅

Honest bottom line. The sym8 (symmetric-linear int8) Mac bundle is both 3.6× faster AND clean — at the fp16 ceiling, matching the shipped int8-linear quality. The kernel itself is bit-exact; quality is purely the expert quantization scheme. An int4 bundle (4.7 GB) was validated to run on the iPhone 17 Pro (~32 tok/s, the first MoE on the phone) — but the iPhone needs int4 for size and non-QAT int4 is a hard quality wall (two independent 4-bit schemes both land at ~12 introduced flips/41 with large margins; clean int4 would need QAT weights LiquidAI doesn't ship). So only the clean Mac bundle is shipped; rebuild the int4 variant locally if you want the on-device version. On a bare prompt the base model itself greedy-degenerates into repetition (present in fp16 too) — use the chat template + sampling.

Run

COREAI_CHUNK_THRESHOLD=1 llm-benchmark --model gpu-pipelined/lfm2_5_8b_a1b_decode_sym8_gather -p 128 -g 256 -n 3

The decode graph's input_ids is static [1,1]; prefill runs as S=1 pipelined steps. Convert your own with conversion/export_lfm2_moe_metal_decode_pipelined.py (sym8 = clean Mac; int4km = iPhone-compact, not shipped).

License

LFM Open License v1.0 (upstream LiquidAI license, shipped as LICENSE). Conversion/kernel: community.