granite-embedding-311m-multilingual-r2 — LiteRT

ibm-granite/granite-embedding-311m-multilingual-r2 converted to LiteRT (.tflite) for on-device inference. A multilingual ModernBERT bi-encoder for retrieval, search and RAG, producing 768-dimensional L2-normalized vectors — fully offline, on CPU.

CLS pooling and L2 normalization are inside the graph: one call in, one finished embedding out.

File	Recipe	Signatures	Size
granite-embedding-311m-r2_wi8fc.tflite	int8 dynamic-range (linears + embedding table)	64, 128, 256, 512	336 MB	mobile + desktop
granite-embedding-311m-r2_fp16.tflite	fp16 weights, float compute	64, 128, 256, 512	629 MB	desktop (runs on phone, but see below)

Both files are verified bit-exact against desktop on an iPhone 17 Pro (cosine 1.000000, max diff 0.0 across five scripts and both signature lengths). Use int8 on device: it peaks at 518 MiB for all four signatures — no special memory entitlement needed — where fp16 peaks at 3700 MiB and is up to 6× slower, because XNNPACK expands fp16 weights to fp32 while packing each signature subgraph.

Signatures

Batch-1, right-padded static shapes: input_ids int32 [1, S], attention_mask int32 [1, S] (1 = real token, 0 = pad). Output output_0 float32 [1, 768] — the CLS token, L2-normalized.

Pad the token ids into the smallest signature that fits and set the mask accordingly. The result is independent of which signature you route through: the same text through embed_64 / 128 / 256 / 512 returns bitwise identical vectors, and pad-region token ids cannot influence the output at all.

Embeddings are L2-normalized, so cosine similarity is a dot product. IBM documents Matryoshka truncation on the base model — slice the first 256 dimensions and re-normalize (not independently verified here).

Prompts

This model takes plain text — no prefix. Its config_sentence_transformers.json ships empty query/document prompts, and that is the contract used for every number below. (If you are coming from an E5-style model like nvidia/Nemotron-3-Embed, note the difference: there the query: /passage: prefix is mandatory, here it is not part of the model's contract.)

Footnote: a prefix helps symmetric similarity but hurts retrieval

Out of curiosity we measured an unofficial query: prefix on both sides: en-en STS17 rises from 0.783 to 0.826. But on retrieval it reverses — nDCG@10 0.836 → 0.832 and recall@5 0.900 → 0.860. If your workload is purely symmetric (clustering, dedup, similarity scoring) a constant prefix may be worth testing on your own data; for retrieval, use bare text.

Usage (Python)

import numpy as np
from ai_edge_litert.interpreter import Interpreter
from transformers import AutoTokenizer

PAD_ID = 0
tok = AutoTokenizer.from_pretrained("ibm-granite/granite-embedding-311m-multilingual-r2")
it = Interpreter(model_path="granite-embedding-311m-r2_wi8fc.tflite", num_threads=8)

LENS = sorted(int(n.split("_")[1]) for n in it.get_signature_list())
runners = {s: it.get_signature_runner(f"embed_{s}") for s in LENS}

def embed(text):
    ids = tok(text)["input_ids"][:LENS[-1]]
    S = next(s for s in LENS if len(ids) <= s)
    x = np.full((1, S), PAD_ID, np.int32)
    m = np.zeros((1, S), np.int32)
    x[0, :len(ids)] = ids
    m[0, :len(ids)] = 1
    return list(runners[S](input_ids=x, attention_mask=m).values())[0][0]

q = embed("What is the tallest mountain in Japan?")
d = embed("富士山は、静岡県と山梨県にまたがる活火山で、標高3776.12 mで日本最高峰の独立峰である。")
print("cosine:", float(q @ d))   # cross-lingual match

Texts longer than 512 tokens must be chunked (the upstream model accepts 32768, but a static on-device graph at that length is not practical).

Quality

Three independent checks, each on every variant.

1. The base card's own cross-lingual matrix. IBM publishes an exact 3×3 cosine matrix (EN/DE/JA queries × JA/EN/DE passages). fp32 and fp16 reproduce it to every published digit (max abs 0.0000 vs both the card and the PyTorch reference); int8 is 0.0039 away. All variants rank the correct cross-lingual passage first, 3/3.

2. STS17 semantic similarity, 11 language pairs × 100 pairs, Spearman:

Variant	mean	en-en	ar-ar	es-es	ko-ko	en-de	en-ar	en-tr	es-en	fr-en	it-en	nl-en
fp32	0.7363	0.783	0.748	0.792	0.836	0.666	0.762	0.601	0.726	0.726	0.752	0.709
int8	0.7327	0.779	0.746	0.791	0.837	0.664	0.753	0.596	0.727	0.720	0.744	0.702
fp16	0.7364	0.783	0.748	0.792	0.836	0.667	0.762	0.601	0.726	0.726	0.752	0.709

int8 costs 0.0036 mean, per-language ≤ 0.01. fp16 is indistinguishable from fp32.

3. Retrieval (SciFact-derived, 50 queries over a 600-document corpus): fp32 nDCG@10 0.8400, int8 0.8360, fp16 0.8400, with recall@5 0.9000 and hit@1 0.7600 identical across all three. The corpus is subsampled, so the absolute number is not comparable to published BEIR scores — read the variant deltas.

Speed

CPU/XNNPACK, median of 12 runs, Apple M4 Max at 12 threads:

Variant	embed_64	embed_128	embed_256	embed_512
int8	28.6 ms	35.2 ms	45.5 ms	71.3 ms (5383 tok/s)
fp16	29.8 ms	37.5 ms	53.5 ms	84.5 ms

A static signature computes all S positions regardless of how many are real, so route to the smallest signature that fits. Latency scales gently with length (2.5× from 64→512) because 14 of the 22 layers attend over a 64-wide window rather than the full sequence.

Conversion

Encoder lane — a direct multi-signature litert_torch trace of the HF model, not an LLM export. Two things worth knowing if you reproduce it:

• ModernBERT alternates local and global attention (22 layers, every 3rd global, 64-wide half-window) with a separate rope frequency set per layer type. ModernBertModel.forward accepts attention_mask as a dict of pre-built masks, so both are built explicitly rather than through transformers' mask machinery.
• A sliding window plus right padding creates fully-masked query rows — once a pad position is further than the window from every real token, softmax runs over all -inf. Eager PyTorch absorbs it; the exported graph emits NaN (and int8 hides it). The masks here always allow self-attention, which removes the NaN and is provably output-neutral.

Script and full notes: hf-to-litertlm.

License

Apache-2.0, inherited from the base model; LICENSE is included.

Modification notice: these files are converted, not original. The weights were exported to LiteRT and quantized (int8 dynamic-range / fp16); CLS pooling and L2 normalization were folded into the graph. No fine-tuning or weight modification beyond quantization was performed.