Upload README.md with huggingface_hub
Browse files
README.md
CHANGED
|
@@ -24,19 +24,21 @@ never leaves the device.
|
|
| 24 |
|
| 25 |
## Two parts of the recipe, and only one of them is in the graph
|
| 26 |
|
| 27 |
-
**The pooling is in.** sentence-transformers keeps it per model, and the
|
| 28 |
-
|
| 29 |
|
| 30 |
| | pooling | normalised |
|
| 31 |
|---|---|---|
|
| 32 |
-
|
|
| 33 |
-
| all-MiniLM-
|
| 34 |
| bge-small-en-v1.5 | **CLS** | yes |
|
|
|
|
|
|
|
| 35 |
| paraphrase-multilingual-L12 | mean | **no** |
|
| 36 |
|
| 37 |
Read off `modules.json` and `1_Pooling/config.json` rather than assumed from the family
|
| 38 |
-
name —
|
| 39 |
-
normalise.
|
| 40 |
|
| 41 |
**The prefix is not.** E5 is trained with `"query: "` in front of a search query and
|
| 42 |
`"passage: "` in front of a document, and it expects them at inference:
|
|
@@ -56,7 +58,7 @@ numbers below are for the recipe as the model intends it.
|
|
| 56 |
| build | file | size | latency | worst cosine vs eager |
|
| 57 |
|---|---|---|---|---|
|
| 58 |
| XNNPACK fp32 | `embed_multilingual_e5_large_xnnpack_fp32.pte` | 2235.7 MB | 106.5 ms | 1.000000 |
|
| 59 |
-
| XNNPACK fp16 | `embed_multilingual_e5_large_xnnpack_fp16.pte` | 1118.3 MB | 221.2 ms |
|
| 60 |
| Core ML fp32 | `embed_multilingual_e5_large_coreml_all.pte` | 1119.3 MB | **27.2 ms** | 0.999993 |
|
| 61 |
|
| 62 |
Mac arm64, median of 10, one 256-token sequence — a reference point for relative cost,
|
|
@@ -100,8 +102,8 @@ the weights — holds at both points, and large is closer to the crossing.
|
|
| 100 |
XNNPACK fp32 is level with PyTorch eager here (106.5 ms against 101.1), and fp16 is
|
| 101 |
twice as slow while halving the file — XNNPACK has no fp16 kernels for this graph and
|
| 102 |
inserts casts instead. Core ML is the one that pays: **27.2 ms**, roughly four times
|
| 103 |
-
eager, 100% delegated in a single subgraph. On iOS take the Core ML build; on Android the choice is
|
| 104 |
-
speed or fp16 for half the disk.
|
| 105 |
|
| 106 |
torch.export -> to_edge_transform_and_lower(partitioner) -> .pte
|
| 107 |
(conversion scripts: [executorch-models](https://github.com/john-rocky/executorch-models))
|
|
|
|
| 24 |
|
| 25 |
## Two parts of the recipe, and only one of them is in the graph
|
| 26 |
|
| 27 |
+
**The pooling is in.** sentence-transformers keeps it per model, and the six embedding
|
| 28 |
+
models on this shelf do not agree:
|
| 29 |
|
| 30 |
| | pooling | normalised |
|
| 31 |
|---|---|---|
|
| 32 |
+
| all-MiniLM-L6-v2 | mean | yes |
|
| 33 |
+
| all-MiniLM-L12-v2 | mean | yes |
|
| 34 |
| bge-small-en-v1.5 | **CLS** | yes |
|
| 35 |
+
| multilingual-e5-base | mean | yes |
|
| 36 |
+
| **multilingual-e5-large** | **mean** | **yes** |
|
| 37 |
| paraphrase-multilingual-L12 | mean | **no** |
|
| 38 |
|
| 39 |
Read off `modules.json` and `1_Pooling/config.json` rather than assumed from the family
|
| 40 |
+
name — paraphrase-multilingual is the same architecture family with the same pooling, and
|
| 41 |
+
it does *not* normalise.
|
| 42 |
|
| 43 |
**The prefix is not.** E5 is trained with `"query: "` in front of a search query and
|
| 44 |
`"passage: "` in front of a document, and it expects them at inference:
|
|
|
|
| 58 |
| build | file | size | latency | worst cosine vs eager |
|
| 59 |
|---|---|---|---|---|
|
| 60 |
| XNNPACK fp32 | `embed_multilingual_e5_large_xnnpack_fp32.pte` | 2235.7 MB | 106.5 ms | 1.000000 |
|
| 61 |
+
| XNNPACK fp16 | `embed_multilingual_e5_large_xnnpack_fp16.pte` | 1118.3 MB | 221.2 ms | 0.999999 |
|
| 62 |
| Core ML fp32 | `embed_multilingual_e5_large_coreml_all.pte` | 1119.3 MB | **27.2 ms** | 0.999993 |
|
| 63 |
|
| 64 |
Mac arm64, median of 10, one 256-token sequence — a reference point for relative cost,
|
|
|
|
| 102 |
XNNPACK fp32 is level with PyTorch eager here (106.5 ms against 101.1), and fp16 is
|
| 103 |
twice as slow while halving the file — XNNPACK has no fp16 kernels for this graph and
|
| 104 |
inserts casts instead. Core ML is the one that pays: **27.2 ms**, roughly four times
|
| 105 |
+
eager, 100% delegated in a single subgraph. On iOS take the Core ML build; on Android the choice is
|
| 106 |
+
fp32 for speed or fp16 for half the disk.
|
| 107 |
|
| 108 |
torch.export -> to_edge_transform_and_lower(partitioner) -> .pte
|
| 109 |
(conversion scripts: [executorch-models](https://github.com/john-rocky/executorch-models))
|