| --- |
| license: apache-2.0 |
| tags: |
| - executorch |
| - xnnpack |
| - pte |
| - on-device |
| - feature-extraction |
| - sentence-similarity |
| base_model: |
| - sentence-transformers/all-mpnet-base-v2 |
| --- |
| # all-mpnet-base-v2 — ExecuTorch |
|
|
| The most downloaded sentence-transformer there is. Text in, one |
| 768-dimensional vector out, for search and retrieval that never leaves the |
| device. |
|
|
| - **Source**: sentence-transformers/all-mpnet-base-v2 — 12 layers, 768 dimensions, 30,527 vocabulary |
| - **License**: apache-2.0 |
| - **Input**: `input_ids` and `attention_mask`, both `[1, 256]` int64 |
| - **Output**: `[1, 768]`, mean-pooled and L2-normalised inside the graph |
|
|
| ## The recipe is in the graph, and it was read off this repo |
|
|
| sentence-transformers stores it per model, and the shelf's seven embedding models do |
| not agree. This one pools **mean** and |
| **normalises**, read from |
| `1_Pooling/config.json` and `modules.json` rather than inferred from the family name. |
| Getting it wrong does not throw; it returns vectors that look fine and rank wrong. |
|
|
| ## Verification |
|
|
| | build | file | size (MB) | Mac ms* | backend takes | worst cosine vs eager | retrieval budget | |
| |---|---|---|---|---|---|---| |
| | fp32 | `embed_all_mpnet_xnnpack_fp32.pte` | 435.8 | 35.2 | 75.5% | 1.000000 | 0% | |
| | fp16 | `embed_all_mpnet_xnnpack_fp16.pte` | 218.1 | 56.2 | 65.7% | 1.000000 | 11% | |
| | Core ML (fp16, iOS) | `embed_all_mpnet_coreml_all.pte` | 220.2 | 6.2 | 100.0% | 0.999993 | 32% | |
|
|
| \*Mac arm64, one 256-token sequence, **fastest of five medians of ten** — a reference |
| point for relative cost, not a device number. The host shares its cores with other work, |
| and a single median does not survive that: the same eager model here measured 19.6 ms and |
| 182.8 ms twenty minutes apart. Contention only ever adds time, so the fastest repetition is |
| the one that means something. Torch eager fp32, measured the same way, is |
| 37.0 ms. |
| |
| Cosine is measured against the model run in eager through its own pooling, over eight |
| sentences. The last column is the one that decides: rank those eight against each |
| other, and ask whether this build's score error is smaller than the gap between the |
| document a query retrieves and the runner-up. Every shipped build keeps all eight |
| top-1 results. |
| |
| ## Not shipped: int8 |
| |
| `embed_all_mpnet_xnnpack_int8.pte` is **181.3 MB** — smaller than fp16's 218.1 MB, because the token embedding |
| table is only 94 MB of the 435.8 MB model (22%), leaving most of the |
| weight in linears for int8 to shrink. |
| |
| It is withheld on the number that decides. Ranking the eight test sentences against |
| each other, this build moves a pair score by at most **0.0081** while the |
| closest fp32 decision — the gap between the document a query retrieves and the |
| runner-up — is **0.0026**. That is **316%** of |
| the room available, against a bar of 50%. |
| |
| Correlation reads 0.998871 for this build, which no correlation gate |
| would stop. |
| |
| torch.export -> to_edge_transform_and_lower(partitioner) -> .pte |
| (conversion scripts: [executorch-models](https://github.com/john-rocky/executorch-models)) |
| |
