Upload README.md with huggingface_hub

README.md

@@ -12,98 +12,85 @@ base_model:
 ---
 # multilingual-e5-large — ExecuTorch
 
-Multilingual sentence embeddings: text in any of about a hundred languages in, one
-1024-dimensional vector out, comparable across languages. For search and retrieval that
-never leaves the device.
+Multilingual E5 at 24 layers. Text in, one
+1024-dimensional vector out, for search and retrieval that never leaves the
+device.
 
-- **Source**: intfloat/multilingual-e5-large — 560M parameters, 24 XLM-RoBERTa layers,
-  250k vocabulary, 1024-dimensional output
-- **License**: MIT
+- **Source**: intfloat/multilingual-e5-large — 24 layers, 1024 dimensions, 250,002 vocabulary
+- **License**: mit
 - **Input**: `input_ids` and `attention_mask`, both `[1, 256]` int64
-- **Output**: `[1, 1024]`, mean-pooled over the mask and L2-normalised inside the graph
+- **Output**: `[1, 1024]`, mean-pooled and L2-normalised inside the graph
 
-## Two parts of the recipe, and only one of them is in the graph
+## The recipe is in the graph, and it was read off this repo
 
-**The pooling is in.** sentence-transformers keeps it per model, and the six embedding
-models on this shelf do not agree:
+sentence-transformers stores it per model, and the shelf's seven embedding models do
+not agree. This one pools **mean** and
+**normalises**, read from
+`1_Pooling/config.json` and `modules.json` rather than inferred from the family name.
+Getting it wrong does not throw; it returns vectors that look fine and rank wrong.
 
-| | pooling | normalised |
-|---|---|---|
-| all-MiniLM-L6-v2 | mean | yes |
-| all-MiniLM-L12-v2 | mean | yes |
-| bge-small-en-v1.5 | **CLS** | yes |
-| multilingual-e5-base | mean | yes |
-| **multilingual-e5-large** | **mean** | **yes** |
-| paraphrase-multilingual-L12 | mean | **no** |
+## The prefix is not in the graph
 
-Read off `modules.json` and `1_Pooling/config.json` rather than assumed from the family
-name — paraphrase-multilingual is the same architecture family with the same pooling, and
-it does *not* normalise.
-
-**The prefix is not.** E5 is trained with `"query: "` in front of a search query and
-`"passage: "` in front of a document, and it expects them at inference:
-
-```
-query:   query: how do I keep data on the phone?
-passage: passage: On-device inference keeps the data on the phone.
-```
-
-That is text, so it happens before tokenisation and the `.pte` never sees it as anything
-but tokens. Leaving it out does not throw and does not look wrong — it returns a
-plausible vector that retrieves worse. The conversion repo's checker applies it, so the
-numbers below are for the recipe as the model intends it.
+This model is trained with `query: ` in front of the text and expects it at
+inference. That happens before tokenisation, so the `.pte` never sees it as anything
+but tokens — and leaving it out does not throw. It returns a plausible vector that
+retrieves worse.
 
 ## Verification
 
-| build | file | size | latency | worst cosine vs eager |
-|---|---|---|---|---|
-| XNNPACK fp32 | `embed_multilingual_e5_large_xnnpack_fp32.pte` | 2235.7 MB | 106.5 ms | 1.000000 |
-| XNNPACK fp16 | `embed_multilingual_e5_large_xnnpack_fp16.pte` | 1118.3 MB | 221.2 ms | 0.999999 |
-| Core ML fp32 | `embed_multilingual_e5_large_coreml_all.pte` | 1119.3 MB | **27.2 ms** | 0.999993 |
-
-Mac arm64, median of 10, one 256-token sequence — a reference point for relative cost,
-not a device number. Eager fp32 on the same input is 101.1 ms. Cosine is measured against
-the model run in eager through its own pooling, over eight sentences.
-
-**And that the vectors are useful**, which agreement alone cannot show. A paraphrase
-against an unrelated sentence, and then the same test across languages —
-"機械学習のモデルを端末の上で動かす" against "On-device inference keeps the data on the
-phone":
-
-```
-same language:   0.852 same meaning  vs  0.661 unrelated
-across languages: 0.833 same meaning  vs  0.739 unrelated
-```
-
-The cross-lingual row is what this model is for, and it is the row the English-only
-models on this shelf fail. Note the scale: E5 puts everything high, so 0.70 for an
-unrelated pair is normal and the **gap** is what carries the signal, not the absolute
-number.
-
-```bash
-python convert/check_embed.py multilingual_e5_large fp32     # or fp16, int8, coreml
-```
-
-## Not shipped
-
-**int8** converts and holds — worst cosine 0.997435, and it still separates the pairs —
-but it comes out at **1330.7 MB against fp16's 1118.3 MB**. Dynamic int8 quantises the
-linear weights and leaves the token embedding table alone, and with a 250k vocabulary at
-1024 dimensions that table is **1024 MB of the 2235.7 MB model**, 46% of it.
-
-The base model is the same story with a different ratio, and the pair is worth reading
-together: its table is 69% of the file and int8 lands 54% above fp16; here the table is
-46% and int8 lands 19% above. The depth doubled and the vocabulary did not. This shelf's
-rule of thumb — int8 beats fp16 only when the embedding table is under about a third of
-the weights — holds at both points, and large is closer to the crossing.
-
-## Worth knowing about the speed
-
-XNNPACK fp32 is level with PyTorch eager here (106.5 ms against 101.1), and fp16 is
-twice as slow while halving the file — XNNPACK has no fp16 kernels for this graph and
-inserts casts instead. Core ML is the one that pays: **27.2 ms**, roughly four times
-eager, 100% delegated in a single subgraph. On iOS take the Core ML build; on Android the choice is
-fp32 for speed or fp16 for half the disk.
+| build | file | size (MB) | Mac ms* | backend takes | worst cosine vs eager | retrieval budget |
+|---|---|---|---|---|---|---|
+| fp32 | `embed_multilingual_e5_large_xnnpack_fp32.pte` | 2235.7 | 103.6 | 80.7% | 1.000000 | 0% |
+| fp16 | `embed_multilingual_e5_large_xnnpack_fp16.pte` | 1118.2 | 145.5 | 69.6% | 0.999999 | 6% |
+| Core ML (fp16, iOS) | `embed_multilingual_e5_large_coreml_all.pte` | 1119.5 | 28.1 | 100.0% | 0.999992 | 40% |
+
+\*Mac arm64, median of 10, one 256-token sequence — a reference point for relative
+cost, not a device number. Torch eager fp32 on the same machine is
+135.6 ms.
+
+Cosine is measured against the model run in eager through its own pooling, over eight
+sentences. The last column is the one that decides: rank those eight against each
+other, and ask whether this build's score error is smaller than the gap between the
+document a query retrieves and the runner-up. Every shipped build keeps all eight
+top-1 results.
+
+## The attention is eager, and that is the faster export
+
+`F.scaled_dot_product_attention` does not survive export as one operation. The edge
+dialect lowers it through `_safe_softmax`, whose guard against a row with no unmasked
+key at all leaves **eleven operations XNNPACK cannot take, in every attention block** —
+`scalar_tensor`, `where`, `mul.Scalar`, `logical_not`, `eq`, `full_like`, `any.dim`.
+Each one cuts the subgraph in two. The count is exact and does not vary by family:
+measured across this shelf, from a 4-layer cross-encoder to a 28-layer causal reranker,
+it is 11 per block every time.
+
+The guard is emitted whether or not it can ever fire, and here it cannot. It triggers
+only on `-inf`, which reaches the graph only because the sdpa path hands `F.sdpa` a
+**boolean** mask for PyTorch to fill; `attn_implementation="eager"` masks with
+`torch.finfo(dtype).min`, a large finite number, and never produces one. So the two
+arms differ only on rows that have no unmasked key — sdpa zeroes them, eager gives them
+a uniform row — and those are padding rows, which the pooling discards and every real
+query row masks out. Measured with all but eight positions masked, as adversarial as
+this shape gets, the two graphs agree to 1.4e-07.
+
+XNNPACK fp32 goes from **63.8% to 80.7%** delegated.
+
+## Not shipped: int8
+
+`embed_multilingual_e5_large_xnnpack_int8.pte` is **1330.6 MB** against fp16's 1118.2 MB. Dynamic int8 quantises the
+linear weights and leaves the token embedding table in fp32, and here that table is
+1024 MB of the 2235.7 MB model — **46%**. The size a build comes out at is
+`0.5 + 1.5 x (table share)` times the fp16 build; at 46% that is
+1.19, so there was never a smaller file to be had.
+
+It is withheld on the number that decides. Ranking the eight test sentences against
+each other, this build moves a pair score by at most **0.0083** while the
+closest fp32 decision — the gap between the document a query retrieves and the
+runner-up — is **0.0014**. That is **590%** of
+the room available, against a bar of 50%.
+
+Correlation reads 0.997579 for this build, which no correlation gate
+would stop.
 
 torch.export -> to_edge_transform_and_lower(partitioner) -> .pte
 (conversion scripts: [executorch-models](https://github.com/john-rocky/executorch-models))