Upload README.md with huggingface_hub

README.md

@@ -0,0 +1,104 @@
+---
+license: mit
+tags:
+- executorch
+- xnnpack
+- pte
+- on-device
+- sentence-similarity
+- feature-extraction
+base_model:
+- intfloat/multilingual-e5-base
+---
+# multilingual-e5-base — ExecuTorch
+
+Multilingual sentence embeddings: text in any of about a hundred languages in, one
+768-dimensional vector out, comparable across languages. For search and retrieval that
+never leaves the device.
+
+- **Source**: intfloat/multilingual-e5-base — 278M parameters, 12 XLM-RoBERTa layers,
+  250k vocabulary, 768-dimensional output
+- **License**: MIT
+- **Input**: `input_ids` and `attention_mask`, both `[1, 256]` int64
+- **Output**: `[1, 768]`, mean-pooled over the mask and L2-normalised inside the graph
+
+## Two parts of the recipe, and only one of them is in the graph
+
+**The pooling is in.** sentence-transformers keeps it per model, and the four small
+embedding models on this shelf do not agree:
+
+| | pooling | normalised |
+|---|---|---|
+| **multilingual-e5-base** | **mean** | **yes** |
+| all-MiniLM-L6-v2 / L12-v2 | mean | yes |
+| bge-small-en-v1.5 | **CLS** | yes |
+| paraphrase-multilingual-L12 | mean | **no** |
+
+Read off `modules.json` and `1_Pooling/config.json` rather than assumed from the family
+name — the other multilingual model on this shelf uses the same pooling and does *not*
+normalise.
+
+**The prefix is not.** E5 is trained with `"query: "` in front of a search query and
+`"passage: "` in front of a document, and it expects them at inference:
+
+```
+query:   query: how do I keep data on the phone?
+passage: passage: On-device inference keeps the data on the phone.
+```
+
+That is text, so it happens before tokenisation and the `.pte` never sees it as anything
+but tokens. Leaving it out does not throw and does not look wrong — it returns a
+plausible vector that retrieves worse. The conversion repo's checker applies it, so the
+numbers below are for the recipe as the model intends it.
+
+## Verification
+
+| build | file | size | latency | worst cosine vs eager |
+|---|---|---|---|---|
+| XNNPACK fp32 | `embed_multilingual_e5_base_xnnpack_fp32.pte` | 1110.0 MB | 40.9 ms | 1.000000 |
+| XNNPACK fp16 | `embed_multilingual_e5_base_xnnpack_fp16.pte` | 555.2 MB | 88.6 ms | 0.999999 |
+| Core ML fp32 | `embed_multilingual_e5_base_coreml_all.pte` | 555.4 MB | **6.8 ms** | 0.999988 |
+
+Mac arm64, median of 10, one 256-token sequence — a reference point for relative cost,
+not a device number. Eager fp32 on the same input is 32.9 ms. Cosine is measured against
+the model run in eager through its own pooling, over eight sentences.
+
+**And that the vectors are useful**, which agreement alone cannot show. A paraphrase
+against an unrelated sentence, and then the same test across languages —
+"機械学習のモデルを端末の上で動かす" against "On-device inference keeps the data on the
+phone":
+
+```
+same language:   0.848 same meaning  vs  0.669 unrelated
+across languages: 0.781 same meaning  vs  0.701 unrelated
+```
+
+The cross-lingual row is what this model is for, and it is the row the English-only
+models on this shelf fail. Note the scale: E5 puts everything high, so 0.70 for an
+unrelated pair is normal and the **gap** is what carries the signal, not the absolute
+number.
+
+```bash
+python convert/check_embed.py multilingual_e5_base fp32     # or fp16, int8, coreml
+```
+
+## Not shipped
+
+**int8** converts and holds — worst cosine 0.996673, and it still separates the pairs —
+but it comes out at **855.6 MB against fp16's 555.2 MB**. Dynamic int8 quantises the
+linear weights and leaves the token embedding table alone, and with a 250k vocabulary at
+768 dimensions that table is **768 MB of the 1110 MB model**, 69% of it. Quantising every
+linear saves 254 MB; fp16 halves the table too. This shelf's rule of thumb holds: int8
+beats fp16 only when the embedding table is under a third of the weights, and a
+multilingual vocabulary is never under a third.
+
+## Worth knowing about the speed
+
+XNNPACK fp32 is slower than PyTorch eager here (40.9 ms against 32.9), and fp16 is slower
+again while halving the file — XNNPACK has no fp16 kernels for this graph and inserts
+casts instead. Core ML is the one that pays: **6.8 ms**, five times eager, 100% delegated
+in a single subgraph. On iOS take the Core ML build; on Android the choice is fp32 for
+speed or fp16 for half the disk.
+
+torch.export -> to_edge_transform_and_lower(partitioner) -> .pte
+(conversion scripts: [executorch-models](https://github.com/john-rocky/executorch-models))