mlboydaisuke/bge-large-en-v1.5-ExecuTorch

bge-large-en-v1.5 — ExecuTorch

BGE's English model at 24 layers. Text in, one 1024-dimensional vector out, for search and retrieval that never leaves the device.

• Source: BAAI/bge-large-en-v1.5 — 24 layers, 1024 dimensions, 30,522 vocabulary
• License: mit
• Input: input_ids and attention_mask, both [1, 256] int64
• Output: [1, 1024], CLS-pooled and L2-normalised inside the graph

The recipe is in the graph, and it was read off this repo

sentence-transformers stores it per model, and the shelf's seven embedding models do not agree. This one pools CLS and normalises, read from 1_Pooling/config.json and modules.json rather than inferred from the family name. Getting it wrong does not throw; it returns vectors that look fine and rank wrong.

Verification

build	file	size (MB)	Mac ms*	backend takes	worst cosine vs eager	retrieval budget
fp32	embed_bge_large_en_xnnpack_fp32.pte	1336.7	90.4	81.6%	1.000000	0%
fp16	embed_bge_large_en_xnnpack_fp16.pte	668.7	151.7	70.2%	1.000000	2%
Core ML (fp16, iOS)	embed_bge_large_en_coreml_all.pte	669.4	25.2	100.0%	0.999997	10%

*Mac arm64, median of 10, one 256-token sequence — a reference point for relative cost, not a device number. Torch eager fp32 on the same machine is 117.7 ms.

Cosine is measured against the model run in eager through its own pooling, over eight sentences. The last column is the one that decides: rank those eight against each other, and ask whether this build's score error is smaller than the gap between the document a query retrieves and the runner-up. Every shipped build keeps all eight top-1 results.

The attention is eager, and that is the faster export

F.scaled_dot_product_attention does not survive export as one operation. The edge dialect lowers it through _safe_softmax, whose guard against a row with no unmasked key at all leaves eleven operations XNNPACK cannot take, in every attention block — scalar_tensor, where, mul.Scalar, logical_not, eq, full_like, any.dim. Each one cuts the subgraph in two. The count is exact and does not vary by family: measured across this shelf, from a 4-layer cross-encoder to a 28-layer causal reranker, it is 11 per block every time.

The guard is emitted whether or not it can ever fire, and here it cannot. It triggers only on -inf, which reaches the graph only because the sdpa path hands F.sdpa a boolean mask for PyTorch to fill; attn_implementation="eager" masks with torch.finfo(dtype).min, a large finite number, and never produces one. So the two arms differ only on rows that have no unmasked key — sdpa zeroes them, eager gives them a uniform row — and those are padding rows, which the pooling discards and every real query row masks out. Measured with all but eight positions masked, as adversarial as this shape gets, the two graphs agree to 1.4e-07.

XNNPACK fp32 goes from 64.3% to 81.6% delegated.

Not shipped: int8

embed_bge_large_en_xnnpack_int8.pte is 431.7 MB — smaller than fp16's 668.7 MB, because the token embedding table is only 125 MB of the 1336.7 MB model (9%), leaving most of the weight in linears for int8 to shrink.

It is withheld on the number that decides. Ranking the eight test sentences against each other, this build moves a pair score by at most 0.0190 while the closest fp32 decision — the gap between the document a query retrieves and the runner-up — is 0.0090. That is 212% of the room available, against a bar of 50%.

Correlation reads 0.999392 for this build, which no correlation gate would stop.

torch.export -> to_edge_transform_and_lower(partitioner) -> .pte (conversion scripts: executorch-models)