Upload README.md with huggingface_hub

README.md

@@ -12,95 +12,85 @@ base_model:
 ---
 # multilingual-e5-base — ExecuTorch
 
-Multilingual sentence embeddings: text in any of about a hundred languages in, one
-768-dimensional vector out, comparable across languages. For search and retrieval that
-never leaves the device.
+Multilingual E5, base-sized — one index over 100 languages. Text in, one
+768-dimensional vector out, for search and retrieval that never leaves the
+device.
 
-- **Source**: intfloat/multilingual-e5-base — 278M parameters, 12 XLM-RoBERTa layers,
-  250k vocabulary, 768-dimensional output
-- **License**: MIT
+- **Source**: intfloat/multilingual-e5-base — 12 layers, 768 dimensions, 250,002 vocabulary
+- **License**: mit
 - **Input**: `input_ids` and `attention_mask`, both `[1, 256]` int64
-- **Output**: `[1, 768]`, mean-pooled over the mask and L2-normalised inside the graph
+- **Output**: `[1, 768]`, mean-pooled and L2-normalised inside the graph
 
-## Two parts of the recipe, and only one of them is in the graph
+## The recipe is in the graph, and it was read off this repo
 
-**The pooling is in.** sentence-transformers keeps it per model, and the six embedding
-models on this shelf do not agree:
+sentence-transformers stores it per model, and the shelf's seven embedding models do
+not agree. This one pools **mean** and
+**normalises**, read from
+`1_Pooling/config.json` and `modules.json` rather than inferred from the family name.
+Getting it wrong does not throw; it returns vectors that look fine and rank wrong.
 
-| | pooling | normalised |
-|---|---|---|
-| all-MiniLM-L6-v2 | mean | yes |
-| all-MiniLM-L12-v2 | mean | yes |
-| bge-small-en-v1.5 | **CLS** | yes |
-| **multilingual-e5-base** | **mean** | **yes** |
-| multilingual-e5-large | mean | yes |
-| paraphrase-multilingual-L12 | mean | **no** |
+## The prefix is not in the graph
 
-Read off `modules.json` and `1_Pooling/config.json` rather than assumed from the family
-name — paraphrase-multilingual is the same architecture family with the same pooling, and
-it does *not* normalise.
-
-**The prefix is not.** E5 is trained with `"query: "` in front of a search query and
-`"passage: "` in front of a document, and it expects them at inference:
-
-```
-query:   query: how do I keep data on the phone?
-passage: passage: On-device inference keeps the data on the phone.
-```
-
-That is text, so it happens before tokenisation and the `.pte` never sees it as anything
-but tokens. Leaving it out does not throw and does not look wrong — it returns a
-plausible vector that retrieves worse. The conversion repo's checker applies it, so the
-numbers below are for the recipe as the model intends it.
+This model is trained with `query: ` in front of the text and expects it at
+inference. That happens before tokenisation, so the `.pte` never sees it as anything
+but tokens — and leaving it out does not throw. It returns a plausible vector that
+retrieves worse.
 
 ## Verification
 
-| build | file | size | latency | worst cosine vs eager |
-|---|---|---|---|---|
-| XNNPACK fp32 | `embed_multilingual_e5_base_xnnpack_fp32.pte` | 1110.0 MB | 40.9 ms | 1.000000 |
-| XNNPACK fp16 | `embed_multilingual_e5_base_xnnpack_fp16.pte` | 555.2 MB | 88.6 ms | 0.999999 |
-| Core ML fp32 | `embed_multilingual_e5_base_coreml_all.pte` | 555.4 MB | **6.8 ms** | 0.999988 |
-
-Mac arm64, median of 10, one 256-token sequence — a reference point for relative cost,
-not a device number. Eager fp32 on the same input is 32.9 ms. Cosine is measured against
-the model run in eager through its own pooling, over eight sentences.
-
-**And that the vectors are useful**, which agreement alone cannot show. A paraphrase
-against an unrelated sentence, and then the same test across languages —
-"機械学習のモデルを端末の上で動かす" against "On-device inference keeps the data on the
-phone":
-
-```
-same language:   0.848 same meaning  vs  0.669 unrelated
-across languages: 0.781 same meaning  vs  0.701 unrelated
-```
-
-The cross-lingual row is what this model is for, and it is the row the English-only
-models on this shelf fail. Note the scale: E5 puts everything high, so 0.70 for an
-unrelated pair is normal and the **gap** is what carries the signal, not the absolute
-number.
-
-```bash
-python convert/check_embed.py multilingual_e5_base fp32     # or fp16, int8, coreml
-```
-
-## Not shipped
-
-**int8** converts and holds — worst cosine 0.996673, and it still separates the pairs —
-but it comes out at **855.6 MB against fp16's 555.2 MB**. Dynamic int8 quantises the
-linear weights and leaves the token embedding table alone, and with a 250k vocabulary at
-768 dimensions that table is **768 MB of the 1110 MB model**, 69% of it. Quantising every
-linear saves 254 MB; fp16 halves the table too. This shelf's rule of thumb holds: int8
-beats fp16 only when the embedding table is under a third of the weights, and a
-multilingual vocabulary is never under a third.
-
-## Worth knowing about the speed
-
-XNNPACK fp32 is slower than PyTorch eager here (40.9 ms against 32.9), and fp16 is slower
-again while halving the file — XNNPACK has no fp16 kernels for this graph and inserts
-casts instead. Core ML is the one that pays: **6.8 ms**, five times eager, 100% delegated
-in a single subgraph. On iOS take the Core ML build; on Android the choice is fp32 for
-speed or fp16 for half the disk.
+| build | file | size (MB) | Mac ms* | backend takes | worst cosine vs eager | retrieval budget |
+|---|---|---|---|---|---|---|
+| fp32 | `embed_multilingual_e5_base_xnnpack_fp32.pte` | 1110.0 | 40.0 | 77.3% | 1.000000 | 0% |
+| fp16 | `embed_multilingual_e5_base_xnnpack_fp16.pte` | 555.2 | 65.1 | 66.7% | 0.999999 | 6% |
+| Core ML (fp16, iOS) | `embed_multilingual_e5_base_coreml_all.pte` | 555.7 | 6.9 | 100.0% | 0.999994 | 19% |
+
+\*Mac arm64, median of 10, one 256-token sequence — a reference point for relative
+cost, not a device number. Torch eager fp32 on the same machine is
+47.9 ms.
+
+Cosine is measured against the model run in eager through its own pooling, over eight
+sentences. The last column is the one that decides: rank those eight against each
+other, and ask whether this build's score error is smaller than the gap between the
+document a query retrieves and the runner-up. Every shipped build keeps all eight
+top-1 results.
+
+## The attention is eager, and that is the faster export
+
+`F.scaled_dot_product_attention` does not survive export as one operation. The edge
+dialect lowers it through `_safe_softmax`, whose guard against a row with no unmasked
+key at all leaves **eleven operations XNNPACK cannot take, in every attention block** —
+`scalar_tensor`, `where`, `mul.Scalar`, `logical_not`, `eq`, `full_like`, `any.dim`.
+Each one cuts the subgraph in two. The count is exact and does not vary by family:
+measured across this shelf, from a 4-layer cross-encoder to a 28-layer causal reranker,
+it is 11 per block every time.
+
+The guard is emitted whether or not it can ever fire, and here it cannot. It triggers
+only on `-inf`, which reaches the graph only because the sdpa path hands `F.sdpa` a
+**boolean** mask for PyTorch to fill; `attn_implementation="eager"` masks with
+`torch.finfo(dtype).min`, a large finite number, and never produces one. So the two
+arms differ only on rows that have no unmasked key — sdpa zeroes them, eager gives them
+a uniform row — and those are padding rows, which the pooling discards and every real
+query row masks out. Measured with all but eight positions masked, as adversarial as
+this shape gets, the two graphs agree to 1.4e-07.
+
+XNNPACK fp32 goes from **61.9% to 77.3%** delegated.
+
+## Not shipped: int8
+
+`embed_multilingual_e5_base_xnnpack_int8.pte` is **855.5 MB** against fp16's 555.2 MB. Dynamic int8 quantises the
+linear weights and leaves the token embedding table in fp32, and here that table is
+768 MB of the 1110.0 MB model — **69%**. The size a build comes out at is
+`0.5 + 1.5 x (table share)` times the fp16 build; at 69% that is
+1.54, so there was never a smaller file to be had.
+
+It is withheld on the number that decides. Ranking the eight test sentences against
+each other, this build moves a pair score by at most **0.0057** while the
+closest fp32 decision — the gap between the document a query retrieves and the
+runner-up — is **0.0022**. That is **255%** of
+the room available, against a bar of 50%.
+
+Correlation reads 0.997142 for this build, which no correlation gate
+would stop.
 
 torch.export -> to_edge_transform_and_lower(partitioner) -> .pte
 (conversion scripts: [executorch-models](https://github.com/john-rocky/executorch-models))