Upload README.md with huggingface_hub

README.md

@@ -0,0 +1,107 @@
+---
+license: mit
+tags:
+- executorch
+- xnnpack
+- pte
+- on-device
+- sentence-similarity
+- feature-extraction
+base_model:
+- intfloat/multilingual-e5-large
+---
+# multilingual-e5-large — ExecuTorch
+
+Multilingual sentence embeddings: text in any of about a hundred languages in, one
+1024-dimensional vector out, comparable across languages. For search and retrieval that
+never leaves the device.
+
+- **Source**: intfloat/multilingual-e5-large — 560M parameters, 24 XLM-RoBERTa layers,
+  250k vocabulary, 1024-dimensional output
+- **License**: MIT
+- **Input**: `input_ids` and `attention_mask`, both `[1, 256]` int64
+- **Output**: `[1, 1024]`, mean-pooled over the mask and L2-normalised inside the graph
+
+## Two parts of the recipe, and only one of them is in the graph
+
+**The pooling is in.** sentence-transformers keeps it per model, and the four small
+embedding models on this shelf do not agree:
+
+| | pooling | normalised |
+|---|---|---|
+| **multilingual-e5-large** | **mean** | **yes** |
+| all-MiniLM-L6-v2 / L12-v2 | mean | yes |
+| bge-small-en-v1.5 | **CLS** | yes |
+| paraphrase-multilingual-L12 | mean | **no** |
+
+Read off `modules.json` and `1_Pooling/config.json` rather than assumed from the family
+name — the other multilingual model on this shelf uses the same pooling and does *not*
+normalise.
+
+**The prefix is not.** E5 is trained with `"query: "` in front of a search query and
+`"passage: "` in front of a document, and it expects them at inference:
+
+```
+query:   query: how do I keep data on the phone?
+passage: passage: On-device inference keeps the data on the phone.
+```
+
+That is text, so it happens before tokenisation and the `.pte` never sees it as anything
+but tokens. Leaving it out does not throw and does not look wrong — it returns a
+plausible vector that retrieves worse. The conversion repo's checker applies it, so the
+numbers below are for the recipe as the model intends it.
+
+## Verification
+
+| build | file | size | latency | worst cosine vs eager |
+|---|---|---|---|---|
+| XNNPACK fp32 | `embed_multilingual_e5_large_xnnpack_fp32.pte` | 2235.7 MB | 106.5 ms | 1.000000 |
+| XNNPACK fp16 | `embed_multilingual_e5_large_xnnpack_fp16.pte` | 1118.3 MB | 221.2 ms | 1.000000 |
+| Core ML fp32 | `embed_multilingual_e5_large_coreml_all.pte` | 1119.3 MB | **27.2 ms** | 0.999993 |
+
+Mac arm64, median of 10, one 256-token sequence — a reference point for relative cost,
+not a device number. Eager fp32 on the same input is 101.1 ms. Cosine is measured against
+the model run in eager through its own pooling, over eight sentences.
+
+**And that the vectors are useful**, which agreement alone cannot show. A paraphrase
+against an unrelated sentence, and then the same test across languages —
+"機械学習のモデルを端末の上で動かす" against "On-device inference keeps the data on the
+phone":
+
+```
+same language:   0.852 same meaning  vs  0.661 unrelated
+across languages: 0.833 same meaning  vs  0.739 unrelated
+```
+
+The cross-lingual row is what this model is for, and it is the row the English-only
+models on this shelf fail. Note the scale: E5 puts everything high, so 0.70 for an
+unrelated pair is normal and the **gap** is what carries the signal, not the absolute
+number.
+
+```bash
+python convert/check_embed.py multilingual_e5_large fp32     # or fp16, int8, coreml
+```
+
+## Not shipped
+
+**int8** converts and holds — worst cosine 0.997435, and it still separates the pairs —
+but it comes out at **1330.7 MB against fp16's 1118.3 MB**. Dynamic int8 quantises the
+linear weights and leaves the token embedding table alone, and with a 250k vocabulary at
+1024 dimensions that table is **1024 MB of the 2235.7 MB model**, 46% of it.
+
+The base model is the same story with a different ratio, and the pair is worth reading
+together: its table is 69% of the file and int8 lands 54% above fp16; here the table is
+46% and int8 lands 19% above. The depth doubled and the vocabulary did not. This shelf's
+rule of thumb — int8 beats fp16 only when the embedding table is under about a third of
+the weights — holds at both points, and large is closer to the crossing.
+
+## Worth knowing about the speed
+
+XNNPACK fp32 is level with PyTorch eager here (106.5 ms against 101.1), and fp16 is
+twice as slow while halving the file — XNNPACK has no fp16 kernels for this graph and
+inserts casts instead. Core ML is the one that pays: **27.2 ms**, roughly four times
+eager, 100% delegated in a single subgraph. On iOS take the Core ML build; on Android the choice is fp32 for
+speed or fp16 for half the disk.
+
+torch.export -> to_edge_transform_and_lower(partitioner) -> .pte
+(conversion scripts: [executorch-models](https://github.com/john-rocky/executorch-models))