| --- |
| license: mit |
| tags: |
| - executorch |
| - xnnpack |
| - pte |
| - on-device |
| - feature-extraction |
| - sentence-similarity |
| base_model: |
| - BAAI/bge-base-en-v1.5 |
| --- |
| # bge-base-en-v1.5 — ExecuTorch |
|
|
| BGE's English workhorse. Text in, one |
| 768-dimensional vector out, for search and retrieval that never leaves the |
| device. |
|
|
| - **Source**: BAAI/bge-base-en-v1.5 — 12 layers, 768 dimensions, 30,522 vocabulary |
| - **License**: mit |
| - **Input**: `input_ids` and `attention_mask`, both `[1, 256]` int64 |
| - **Output**: `[1, 768]`, CLS-pooled and L2-normalised inside the graph |
|
|
| ## The recipe is in the graph, and it was read off this repo |
|
|
| sentence-transformers stores it per model, and the shelf's seven embedding models do |
| not agree. This one pools **CLS** and |
| **normalises**, read from |
| `1_Pooling/config.json` and `modules.json` rather than inferred from the family name. |
| Getting it wrong does not throw; it returns vectors that look fine and rank wrong. |
|
|
| ## Verification |
|
|
| | build | file | size (MB) | Mac ms* | backend takes | worst cosine vs eager | retrieval budget | |
| |---|---|---|---|---|---|---| |
| | fp32 | `embed_bge_base_en_xnnpack_fp32.pte` | 435.7 | 33.3 | 78.8% | 1.000000 | 0% | |
| | fp16 | `embed_bge_base_en_xnnpack_fp16.pte` | 218.0 | 53.7 | 67.8% | 0.999999 | 1% | |
| | Core ML (fp16, iOS) | `embed_bge_base_en_coreml_all.pte` | 218.6 | 6.0 | 100.0% | 0.999997 | 8% | |
|
|
| \*Mac arm64, one 256-token sequence, **fastest of five medians of ten** — a reference |
| point for relative cost, not a device number. The host shares its cores with other work, |
| and a single median does not survive that: the same eager model here measured 19.6 ms and |
| 182.8 ms twenty minutes apart. Contention only ever adds time, so the fastest repetition is |
| the one that means something. Torch eager fp32, measured the same way, is |
| 35.3 ms. |
| |
| Cosine is measured against the model run in eager through its own pooling, over eight |
| sentences. The last column is the one that decides: rank those eight against each |
| other, and ask whether this build's score error is smaller than the gap between the |
| document a query retrieves and the runner-up. Every shipped build keeps all eight |
| top-1 results. |
| |
| ## The attention is eager, and that is the faster export |
| |
| `F.scaled_dot_product_attention` does not survive export as one operation. The edge |
| dialect lowers it through `_safe_softmax`, whose guard against a row with no unmasked key |
| at all leaves **11 operations XNNPACK cannot take, in every attention |
| block** — `scalar_tensor`, `where`, `mul.Scalar`, `logical_not`, `eq`, `full_like`, `any.dim`. Each one cuts the subgraph in two. |
| |
| The switch is `attn_implementation="eager"`: transformers then builds the mask |
| itself, as `torch.finfo(dtype).min`, instead of handing `F.sdpa` a **boolean** mask |
| for PyTorch to fill with `-inf`. |
| |
| The guard is emitted whether or not it can ever fire, and here it cannot: it triggers only |
| on `-inf`, and this arm never produces one. So the two differ only about rows that have no |
| unmasked key at all — sdpa zeroes them, this one gives them a uniform row — and those are |
| padding rows, which the pooling discards and which every real query row masks out anyway. |
| Measured with all but eight positions masked, as adversarial as this shape gets, the two |
| graphs agree to 1.4e-07. |
| |
| XNNPACK fp32 goes from **62.8% to 78.8%** delegated. |
| |
| ## Not shipped: int8 |
| |
| `embed_bge_base_en_xnnpack_int8.pte` is **181.3 MB** — smaller than fp16's 218.0 MB, because the token embedding |
| table is only 94 MB of the 435.7 MB model (22%), leaving most of the |
| weight in linears for int8 to shrink. |
| |
| It is withheld on the number that decides. Ranking the eight test sentences against |
| each other, this build moves a pair score by at most **0.0193** while the |
| closest fp32 decision — the gap between the document a query retrieves and the |
| runner-up — is **0.0125**. That is **154%** of |
| the room available, against a bar of 50%. |
| |
| Correlation reads 0.999366 for this build, which no correlation gate |
| would stop. |
| |
| torch.export -> to_edge_transform_and_lower(partitioner) -> .pte |
| (conversion scripts: [executorch-models](https://github.com/john-rocky/executorch-models)) |
| |
