| --- |
| license: apache-2.0 |
| tags: |
| - executorch |
| - xnnpack |
| - pte |
| - on-device |
| - feature-extraction |
| - sentence-similarity |
| base_model: |
| - nomic-ai/nomic-embed-text-v1.5 |
| --- |
| # nomic-embed-text-v1.5 — ExecuTorch |
|
|
| A BERT with rotary embeddings and SwiGLU, trained so that a prefix of the vector is still a usable vector. Text in, one |
| 768-dimensional vector out, for search and retrieval that never leaves the |
| device. |
|
|
| - **Source**: nomic-ai/nomic-embed-text-v1.5 — 12 layers, 768 dimensions, 30,528 vocabulary |
| - **License**: apache-2.0 |
| - **Input**: `input_ids` and `attention_mask`, both `[1, 256]` int64 |
| - **Output**: `[1, 768]`, mean-pooled and **not** normalised inside the graph |
|
|
| ## The recipe is in the graph, and it was read off this repo |
|
|
| sentence-transformers stores it per model, and the shelf's seven embedding models do |
| not agree. This one pools **mean** and |
| **does not normalise**, read from |
| `1_Pooling/config.json` and `modules.json` rather than inferred from the family name. |
| Getting it wrong does not throw; it returns vectors that look fine and rank wrong. |
|
|
| ## The prefix is not in the graph |
|
|
| This model is trained with `search_query: ` in front of the text and expects it at |
| inference. That happens before tokenisation, so the `.pte` never sees it as anything |
| but tokens — and leaving it out does not throw. It returns a plausible vector that |
| retrieves worse. |
|
|
| ## Verification |
|
|
| | build | file | size (MB) | Mac ms* | backend takes | worst cosine vs eager | retrieval budget | |
| |---|---|---|---|---|---|---| |
| | fp32 | `embed_nomic_embed_text_v15_xnnpack_fp32.pte` | 547.2 | 45.1 | 64.5% | 1.000000 | 0% | |
| | fp16 | `embed_nomic_embed_text_v15_xnnpack_fp16.pte` | 273.8 | 97.0 | 58.6% | 0.999999 | 7% | |
| | Core ML (fp16, iOS) | `embed_nomic_embed_text_v15_coreml_all.pte` | 274.8 | 8.1 | 100.0% | 0.999793 | 46% | |
|
|
| \*Mac arm64, one 256-token sequence, **fastest of five medians of ten** — a reference |
| point for relative cost, not a device number. The host shares its cores with other work, |
| and a single median does not survive that: the same eager model here measured 19.6 ms and |
| 182.8 ms twenty minutes apart. Contention only ever adds time, so the fastest repetition is |
| the one that means something. Torch eager fp32, measured the same way, is |
| 44.9 ms. |
| |
| Cosine is measured against the model run in eager through its own pooling, over eight |
| sentences. The last column is the one that decides: rank those eight against each |
| other, and ask whether this build's score error is smaller than the gap between the |
| document a query retrieves and the runner-up. Every shipped build keeps all eight |
| top-1 results. |
| |
| ## The attention is eager, and that is the faster export |
| |
| `F.scaled_dot_product_attention` does not survive export as one operation. The edge |
| dialect lowers it through `_safe_softmax`, whose guard against a row with no unmasked key |
| at all leaves **8 operations XNNPACK cannot take, in every attention |
| block** — `where`, `mul.Scalar`, `logical_not`, `eq`, `full_like`, `any.dim`. Each one cuts the subgraph in two. |
| |
| **This model does not answer to `attn_implementation` at all.** Its attention lives |
| in `nomic-ai/nomic-bert-2048` and calls `F.sdpa` itself, so the setting goes nowhere — |
| measured, 58.8% delegated either way. That code already carries an explicit-softmax |
| arm, behind a module-level `scaled_dot_product_attention` it binds at import because |
| the symbol existed; clearing that symbol is the switch. It already passes an additive |
| `finfo(dtype).min` mask rather than a boolean one, which is why its guard costs eight |
| operations a block here rather than eleven — there is no boolean mask to convert. |
| |
| The guard is emitted whether or not it can ever fire, and here it cannot: it triggers only |
| on `-inf`, and this arm never produces one. So the two differ only about rows that have no |
| unmasked key at all — sdpa zeroes them, this one gives them a uniform row — and those are |
| padding rows, which the pooling discards and which every real query row masks out anyway. |
| Measured with all but eight positions masked, as adversarial as this shape gets, the two |
| graphs agree to 1.4e-07. |
| |
| XNNPACK fp32 goes from **58.8% to 64.5%** delegated. |
| |
| ## Matryoshka: the vector truncates |
| |
| This model is trained so that a **prefix** of the vector is still a usable vector — |
| 768 down to 512, 256, 128 or 64 dimensions, trading accuracy for index size. The |
| graph returns the full 768, because the dimension is the caller's choice, and the |
| truncation recipe is three lines: |
| |
| ```python |
| import torch.nn.functional as F |
| v = F.layer_norm(v, (v.shape[1],)) # before truncating, not after |
| v = v[:, :dim] # 512 / 256 / 128 / 64 |
| v = F.normalize(v, p=2, dim=1) |
| ``` |
| |
| **The `layer_norm` is what makes the prefix usable, and it is easy to skip.** At the |
| full 768 it barely matters — measured on this shelf, going through the layer_norm |
| changes the direction of the vector by a cosine of 0.999944 and leaves the test pair's |
| score at 0.8233 either way. It earns its place only once you truncate. |
| |
| **The four prefixes are a real part of the model.** `search_document: ` for what goes |
| in the index, `search_query: ` for what is asked of it, plus `classification: ` and |
| `clustering: `. Unlike E5's symmetric mode, the two retrieval prefixes are not |
| interchangeable. |
| |
| **The architecture is not stock BERT.** Rotary embeddings and a SwiGLU MLP, with the |
| modelling code in `nomic-ai/nomic-bert-2048` rather than in transformers — loading it |
| needs `trust_remote_code=True` and `einops` installed. None of that reaches the `.pte`, |
| which is a plain graph once exported. |
| |
| ## Not shipped: int8 |
| |
| `embed_nomic_embed_text_v15_xnnpack_int8.pte` is **208.0 MB** — smaller than fp16's 273.8 MB, because the token embedding |
| table is only 94 MB of the 547.2 MB model (17%), leaving most of the |
| weight in linears for int8 to shrink. |
| |
| It is withheld on the number that decides. Ranking the eight test sentences against |
| each other, this build moves a pair score by at most **0.0221** while the |
| closest fp32 decision — the gap between the document a query retrieves and the |
| runner-up — is **0.0077**. That is **288%** of |
| the room available, against a bar of 50%. |
| |
| Correlation reads 0.992203 for this build, which no correlation gate |
| would stop. |
| |
| torch.export -> to_edge_transform_and_lower(partitioner) -> .pte |
| (conversion scripts: [executorch-models](https://github.com/john-rocky/executorch-models)) |
| |
