Nemotron-3-Embed-1B — LiteRT

nvidia/Nemotron-3-Embed-1B-BF16 converted to LiteRT (.tflite) for on-device inference. A multilingual text-embedding model for retrieval and RAG, evaluated by NVIDIA across 34 languages, producing 2048-dimensional L2-normalized vectors — fully offline, on CPU.

Pooling and normalization are inside the graph: one call in, one finished embedding out.

File	Recipe	Signatures	Size	Peak RAM at load
Nemotron-3-Embed-1B_wi8fc_128_512.tflite	int8 dynamic-range	128, 512	1155 MB	1.9 GiB	recommended for phones
Nemotron-3-Embed-1B_wi8fc.tflite	int8 dynamic-range	64, 128, 256, 512	1167 MB	3.6 GiB	desktop, or an app with extra memory headroom
Nemotron-3-Embed-1B_fp16.tflite	fp16 weights, float compute	64, 128, 256, 512	2286 MB	—	desktop

All three return bitwise identical embeddings — the choice is purely about memory and which input lengths you need (see Memory). Both int8 files are verified bit-exact against desktop on an iPhone 17 Pro.

⚠️ The prefix is mandatory — including for symmetric similarity

This is an E5-style model: it only ever saw a prefix during training. Add query: before queries and passage: before documents. For symmetric tasks (semantic similarity, clustering, dedup) use query: on both sides.

Feeding raw text is out of distribution and quietly degrades — badly, and worst exactly where the model is supposed to shine. Measured on STS17, 100 pairs, int8 artifact:

	no prefix	query: both sides
en-en	0.636	0.865
es-en (cross-lingual)	0.064	0.803
en-ar (cross-lingual)	−0.029	0.724

Note that the upstream config_sentence_transformers.json sets default_prompt_name: null, so a stock SentenceTransformer.encode() call sends raw text and lands in the left-hand column. Use encode_query / encode_document, or pass the prefix yourself.

Signatures

Batch-1, right-padded static shapes: input_ids int32 [1, S], attention_mask int32 [1, S] (1 = real token, 0 = pad).

Signature	Output
embed_<S> for each length S in the file	output_0 float32 [1, 2048] — mean-pooled over valid positions, L2-normalized

The 4-signature file has embed_64/128/256/512; the phone file has embed_128/512. Pad the token ids into the smallest signature that fits and set the mask accordingly. Padding is fully masked inside the graph, so the result is independent of which signature you route through: the same text through embed_64 / 128 / 256 / 512 returns bitwise identical vectors, and pad-region token ids cannot influence the output at all.

Embeddings are L2-normalized, so cosine similarity is just a dot product. NVIDIA additionally documents Matryoshka-style truncation on the base model — slice the first 1024 or 512 dimensions and re-normalize (not independently verified here).

Usage (Python)

import numpy as np
from ai_edge_litert.interpreter import Interpreter
from transformers import AutoTokenizer

PAD_ID = 11
tok = AutoTokenizer.from_pretrained("nvidia/Nemotron-3-Embed-1B-BF16")
it = Interpreter(model_path="Nemotron-3-Embed-1B_wi8fc_128_512.tflite", num_threads=8)

# whatever lengths this particular file was built with
LENS = sorted(int(n.split("_")[1]) for n in it.get_signature_list())
runners = {s: it.get_signature_runner(f"embed_{s}") for s in LENS}

def embed(text, prefix="query: "):
    ids = tok(prefix + text)["input_ids"][:LENS[-1]]
    S = next(s for s in LENS if len(ids) <= s)
    x = np.full((1, S), PAD_ID, np.int32)
    m = np.zeros((1, S), np.int32)
    x[0, :len(ids)] = ids
    m[0, :len(ids)] = 1
    return list(runners[S](input_ids=x, attention_mask=m).values())[0][0]

q = embed("How can someone reduce exposure to pollen during allergy season?")
d = embed("Staying indoors on dry, windy days lowers pollen exposure.", "passage: ")
print("cosine:", float(q @ d))

Texts longer than 512 tokens must be chunked (the upstream model accepts 32768, but a static on-device graph at that length is not practical; chunk-and-average or chunk-and-max is the usual approach).

Quality

Three independent checks, each run on every variant.

1. The base card's own similarity matrix. NVIDIA publishes an exact 4×4 query/document matrix. Every variant reproduces it and ranks the right document first (4/4). fp32 and fp16 match the PyTorch reference to 0.0000; int8 to 0.0071. The residual against the published numbers (0.0041) is the card's bf16-on-CUDA vs our fp32-on-CPU setup, not the conversion — the PyTorch reference shows the same gap.

2. STS17 semantic similarity, 11 language pairs × 100 pairs, Spearman, query: on both sides:

Variant	mean	en-en	ar-ar	es-es	ko-ko	en-de	en-ar	es-en	fr-en	it-en	nl-en
fp32	0.7292	0.868	0.837	0.830	0.869	0.770	0.724	0.806	0.768	0.826	0.686
int8	0.7296	0.865	0.837	0.830	0.870	0.772	0.724	0.803	0.769	0.825	0.681
fp16	0.7292	0.868	0.837	0.830	0.869	0.770	0.724	0.806	0.768	0.826	0.686

int8 is task-lossless — every per-language delta is ≤ 0.005 and they fall in both directions.

3. Retrieval (SciFact-derived, 50 queries over a 600-document corpus, query: /passage: ): fp32 nDCG@10 0.8683, int8 0.8628. Note this corpus is subsampled, so the absolute number is not comparable to published BEIR scores, and at 50 queries the gate cannot resolve differences below a couple of percent.

One language boundary worth knowing: Turkish is not among the 34 languages NVIDIA evaluated, and it shows — STS17 en-tr scores ~0.04 for every variant, including the original PyTorch model.

Speed

CPU/XNNPACK, median of ≥10 runs, measured on the 2-signature build:

Variant	Machine	embed_128	embed_512
int8	iPhone 17 Pro, 6 threads	103 ms	~400 ms
int8	M4 Max Mac, 12 threads	150 ms	561 ms
fp16	M4 Max Mac, 12 threads	197 ms	764 ms

A static signature computes all S positions regardless of how many are real, so a 20-token query in embed_128 costs the same as a 120-token one — route to the smallest signature that fits. On the Mac, latency did not improve going from 6 to 12 to 16 threads (574 / 595 / 561 ms), so this workload is not thread-bound there.

Memory

The interpreter allocates and XNNPACK-delegates every signature subgraph at creation time, whether or not you call it — so peak RAM scales with the signatures present in the file, not the ones you use. Measured on an iPhone 17 Pro (6 threads), roughly 846 MiB per signature:

Build	Signatures	Load time	Peak footprint
wi8fc_128_512	2	1.10 s	1.9 GiB
wi8fc	4	2.34 s	3.6 GiB
(embed_512 alone, if you build it)	1	0.57 s	1.0 GiB

Outputs are bitwise identical across all of them, so trimming signatures costs nothing but flexibility. If 1.9 GiB is still too much for your app, build a single-signature file with --seqs 512 (see the conversion script) for a ~1.0 GiB peak at a flat ~400 ms per text.

The 4-signature file was gated in an app holding com.apple.developer.kernel.increased-memory-limit; on the default iOS budget it will likely be jetsammed at load.

The fp16 file does not run on iPhone — it is killed by the OS during interpreter creation, before the first inference (XNNPACK expands fp16 weights while packing, per subgraph). Use int8 on device; it is bit-exact with desktop anyway.

Conversion

Encoder lane — a direct multi-signature litert_torch trace of the HF model, not an LLM export. Two things worth knowing if you reproduce it:

• architectures: ["Ministral3Model"] is the stock causal-LM class; config.is_causal: false is what makes attention bidirectional in transformers. (Ministral3Attention.is_causal is hardcoded True and is not the switch — don't "fix" it.)
• The bidirectional mask is skipped entirely when nothing is padded, which under torch.export bakes a graph that ignores attention_mask. Trace with padded samples and gate on pad-content invariance.

Script and full notes: hf-to-litertlm.

License

The base model is licensed under the OpenMDW License Agreement v1.1 (LICENSE), which permits redistribution provided the agreement and origin notices travel with it — both are included here, along with the upstream NOTICE covering the Apache-2.0 attribution for the Ministral-3-3B parent model.

Modification notice: these files are converted, not original. The weights were exported to LiteRT and quantized (int8 dynamic-range / fp16); mean pooling and L2 normalization were folded into the graph. No fine-tuning or weight modification beyond quantization was performed.