LFM2.5-Embedding-350M — LiteRT

LiquidAI/LFM2.5-Embedding-350M converted to LiteRT (.tflite) for on-device inference. A multilingual (11 languages) dense bi-encoder on the LFM2 hybrid backbone (gated short-convolutions + grouped-query attention) — one 1024-d vector per text, for retrieval, semantic search and RAG, fully offline on CPU.

File	Recipe	Size
LFM2.5-Embedding-350M_wi8fc.tflite	int8 dynamic-range (linears + embedding, convs float)	371 MB	mobile + desktop (iPhone-verified bit-exact, 1.2 GiB peak)
LFM2.5-Embedding-350M_fp16.tflite	fp16 weights, float compute	712 MB	desktop; loads on a 12 GB iPhone but needs 5.8 GiB peak — see Speed

Signatures

All signatures take batch-1, right-padded static shapes: input_ids int32 [1, S], attention_mask int32 [1, S] (1 = real token, 0 = pad).

Signature	Output
embed_64 / embed_128 / embed_256 / embed_512	output_0 float32 [1, 1024] — CLS token, L2-normalized

Pick the smallest signature that fits your token count. The vector does not depend on which one you pick: padding is zeroed inside the graph before the short-convolution and masked in attention, so the same text through embed_64 and through embed_512 returns a bitwise identical vector — verified on Mac and again on-device. Model max length is 512 tokens.

Contract

Two things must match the upstream model or results silently degrade:

• Asymmetric prompts, with the trailing space. Prefix queries with query: and passages with document: , exactly as stored in the base repo's config_sentence_transformers.json.
• CLS pooling and normalization are already in the graph. The output is the position-0 token vector, L2-normalized, so a plain dot product is cosine similarity. Do not pool or normalize again.

Quality

Measured against the PyTorch fp32 reference (Lfm2BidirectionalModel + CLS + normalize) on the same inputs. fp16 is bitwise identical to the reference on every task metric below.

Variant	STS17 Spearman (11 pairs)	NanoSciFact nDCG@10	recall@5	hit@1
PyTorch fp32 (reference)	0.6720	0.8540	0.920	0.780
fp16	0.6720	0.8540	0.920	0.780
int8 (wi8fc)	0.6721	0.8494	0.900	0.780

int8 costs 0.005 nDCG@10 and is a rounding difference on STS17. Per-vector agreement with the reference is cosine 0.9945–0.9955 for int8 and ≥0.9999999 for fp16.

Graph mechanics on the shipped artifacts, all three variants: cross-signature max|diff| 0.0 at 64/128/256/512, pad-content invariance 0.0 (garbage in the padded region cannot move the vector), and a live attention_mask (shortening it moves the output by 4.8e-02).

Speed

Apple-silicon Mac, XNNPACK, 16 threads, 20 iterations after warmup:

Variant	embed_64	embed_128	embed_256	embed_512
int8 (wi8fc)	31.7 ms	45.7 ms	73.6 ms	123.0 ms (4164 tok/s)
fp16	55.5 ms	74.1 ms	108.2 ms	200.8 ms
fp32	37.9 ms	53.1 ms	88.2 ms	168.8 ms

fp16 is slower than fp32 on CPU: XNNPACK unpacks fp16 weights to fp32 at run time, so the smaller file buys disk, not latency.

iPhone 17 Pro, XNNPACK, 6 threads (warm, after the first call on each signature):

Variant	load+delegate	peak footprint	embed_128	embed_512	vs Mac
int8 (wi8fc)	1.02 s	1245 MiB	40–44 ms	137–142 ms	cosine 1.000000, max diff 0.0
fp16	1.50 s	5799 MiB	73–98 ms	256–409 ms	cosine 1.000000, max diff 2e-07

The int8 model reproduces the Mac outputs bit-exactly on device across all tested cases (en/ja/de/ar/hi, short and long, both prompt forms). The fp16 model does run on an iPhone 17 Pro, but its 5.8 GiB peak is a function of that device's memory — treat fp16 as the desktop artifact. Its first invocation on each signature costs 355–538 ms of one-time XNNPACK fp16→fp32 unpacking, excluded from the warm figures above.

GPU (2026-08-13 re-export)

The re-export respells the one idiom mobile GPU delegates refuse — transformers' rank-5 repeat_kv expand — into an equivalent rank-4 matmul (outputs bitwise-identical on CPU), and the int8 file now fully delegates on mobile GPUs. Measured with the LiteRT CompiledModel API (fp32 GPU precision, real inputs, best of 3 warm runs): Pixel 8a OpenCL embed_512 20.1 ms, cosine 0.9992 vs the fp32 desktop reference; iPhone 17 Pro Metal embed_512 171–194 ms (eight fixture cases, cosine ≥ 0.9995). Set the GPU precision to fp32 — at fp16 GPU precision this family's norm reductions overflow and every output is NaN; CompiledModel timings are not comparable to classic-delegate benchmark_model timings.

Usage (Python)

import numpy as np
from ai_edge_litert.interpreter import Interpreter
from tokenizers import Tokenizer

tok = Tokenizer.from_file("tokenizer.json")
it = Interpreter(model_path="LFM2.5-Embedding-350M_wi8fc.tflite")
embed = it.get_signature_runner("embed_128")

def encode(text, prompt):           # prompt = "query: " or "document: "
    ids = tok.encode(prompt + text).ids[:128]
    x = np.zeros((1, 128), np.int32)
    m = np.zeros((1, 128), np.int32)
    x[0, :len(ids)] = ids
    m[0, :len(ids)] = 1
    return list(embed(input_ids=x, attention_mask=m).values())[0][0]  # [1024]

q = encode("What is the capital of France?", "query: ")
d = encode("Paris is the capital and largest city of France.", "document: ")
print(float(q @ d))                 # already normalized -> dot product is cosine

On Android/iOS use the LiteRT runtime's SignatureRunner APIs with the same signature names; the tokenizer is the standard Hugging Face tokenizer.json (works with the tokenizers libraries for Rust/Swift/Kotlin).

License

LFM Open License v1.0 (see LICENSE, unchanged from the base model). Note the license's commercial-use threshold (Section 5). This repository redistributes converted Derivative Works of LiquidAI/LFM2.5-Embedding-350M with modification notices per Section 4; all credit for the model to Liquid AI.