README.md

---
license: apache-2.0
tags:
- executorch
- xnnpack
- pte
- on-device
- feature-extraction
- sentence-similarity
base_model:
- nomic-ai/nomic-embed-text-v1.5
---
# nomic-embed-text-v1.5 — ExecuTorch

A BERT with rotary embeddings and SwiGLU, trained so that a prefix of the vector is still a usable vector. Text in, one
768-dimensional vector out, for search and retrieval that never leaves the
device.

- **Source**: nomic-ai/nomic-embed-text-v1.5 — 12 layers, 768 dimensions, 30,528 vocabulary
- **License**: apache-2.0
- **Input**: `input_ids` and `attention_mask`, both `[1, 256]` int64
- **Output**: `[1, 768]`, mean-pooled and **not** normalised inside the graph

## The recipe is in the graph, and it was read off this repo

sentence-transformers stores it per model, and the shelf's seven embedding models do
not agree. This one pools **mean** and
**does not normalise**, read from
`1_Pooling/config.json` and `modules.json` rather than inferred from the family name.
Getting it wrong does not throw; it returns vectors that look fine and rank wrong.

## The prefix is not in the graph

This model is trained with `search_query: ` in front of the text and expects it at
inference. That happens before tokenisation, so the `.pte` never sees it as anything
but tokens — and leaving it out does not throw. It returns a plausible vector that
retrieves worse.

## Verification

| build | file | size (MB) | Mac ms* | backend takes | worst cosine vs eager | retrieval budget |
|---|---|---|---|---|---|---|
| fp32 | `embed_nomic_embed_text_v15_xnnpack_fp32.pte` | 547.2 | 45.1 | 64.5% | 1.000000 | 0% |
| fp16 | `embed_nomic_embed_text_v15_xnnpack_fp16.pte` | 273.8 | 97.0 | 58.6% | 0.999999 | 7% |
| Core ML (fp16, iOS) | `embed_nomic_embed_text_v15_coreml_all.pte` | 274.8 | 8.1 | 100.0% | 0.999793 | 46% |

\*Mac arm64, one 256-token sequence, **fastest of five medians of ten** — a reference
point for relative cost, not a device number. The host shares its cores with other work,
and a single median does not survive that: the same eager model here measured 19.6 ms and
182.8 ms twenty minutes apart. Contention only ever adds time, so the fastest repetition is
the one that means something. Torch eager fp32, measured the same way, is
44.9 ms.

Cosine is measured against the model run in eager through its own pooling, over eight
sentences. The last column is the one that decides: rank those eight against each
other, and ask whether this build's score error is smaller than the gap between the
document a query retrieves and the runner-up. Every shipped build keeps all eight
top-1 results.

## The attention is eager, and that is the faster export

`F.scaled_dot_product_attention` does not survive export as one operation. The edge
dialect lowers it through `_safe_softmax`, whose guard against a row with no unmasked key
at all leaves **8 operations XNNPACK cannot take, in every attention
block** — `where`, `mul.Scalar`, `logical_not`, `eq`, `full_like`, `any.dim`. Each one cuts the subgraph in two.

**This model does not answer to `attn_implementation` at all.** Its attention lives
in `nomic-ai/nomic-bert-2048` and calls `F.sdpa` itself, so the setting goes nowhere —
measured, 58.8% delegated either way. That code already carries an explicit-softmax
arm, behind a module-level `scaled_dot_product_attention` it binds at import because
the symbol existed; clearing that symbol is the switch. It already passes an additive
`finfo(dtype).min` mask rather than a boolean one, which is why its guard costs eight
operations a block here rather than eleven — there is no boolean mask to convert.

The guard is emitted whether or not it can ever fire, and here it cannot: it triggers only
on `-inf`, and this arm never produces one. So the two differ only about rows that have no
unmasked key at all — sdpa zeroes them, this one gives them a uniform row — and those are
padding rows, which the pooling discards and which every real query row masks out anyway.
Measured with all but eight positions masked, as adversarial as this shape gets, the two
graphs agree to 1.4e-07.

XNNPACK fp32 goes from **58.8% to 64.5%** delegated.

## Matryoshka: the vector truncates

This model is trained so that a **prefix** of the vector is still a usable vector —
768 down to 512, 256, 128 or 64 dimensions, trading accuracy for index size. The
graph returns the full 768, because the dimension is the caller's choice, and the
truncation recipe is three lines:

```python
import torch.nn.functional as F
v = F.layer_norm(v, (v.shape[1],))     # before truncating, not after
v = v[:, :dim]                         # 512 / 256 / 128 / 64
v = F.normalize(v, p=2, dim=1)
```

**The `layer_norm` is what makes the prefix usable, and it is easy to skip.** At the
full 768 it barely matters — measured on this shelf, going through the layer_norm
changes the direction of the vector by a cosine of 0.999944 and leaves the test pair's
score at 0.8233 either way. It earns its place only once you truncate.

**The four prefixes are a real part of the model.** `search_document: ` for what goes
in the index, `search_query: ` for what is asked of it, plus `classification: ` and
`clustering: `. Unlike E5's symmetric mode, the two retrieval prefixes are not
interchangeable.

**The architecture is not stock BERT.** Rotary embeddings and a SwiGLU MLP, with the
modelling code in `nomic-ai/nomic-bert-2048` rather than in transformers — loading it
needs `trust_remote_code=True` and `einops` installed. None of that reaches the `.pte`,
which is a plain graph once exported.

## Not shipped: int8

`embed_nomic_embed_text_v15_xnnpack_int8.pte` is **208.0 MB** — smaller than fp16's 273.8 MB, because the token embedding
table is only 94 MB of the 547.2 MB model (17%), leaving most of the
weight in linears for int8 to shrink.

It is withheld on the number that decides. Ranking the eight test sentences against
each other, this build moves a pair score by at most **0.0221** while the
closest fp32 decision — the gap between the document a query retrieves and the
runner-up — is **0.0077**. That is **288%** of
the room available, against a bar of 50%.

Correlation reads 0.992203 for this build, which no correlation gate
would stop.

torch.export -> to_edge_transform_and_lower(partitioner) -> .pte
(conversion scripts: [executorch-models](https://github.com/john-rocky/executorch-models))