mlboydaisuke
/

Qwen3-Embedding-0.6B-ExecuTorch

@@ -76,15 +76,23 @@ not throw and does not look wrong — it retrieves worse.
 ## Verification
-| build | file | size (MB) | Mac median (ms)* | worst cosine vs eager | retrieval budget spent |
-|---|---|---|---|---|---|
-| XNNPACK fp32 | `embed_qwen3_embedding_0_6b_xnnpack_fp32.pte` | 2383.7 | 203.9 | 1.000000 | 0% |
-| XNNPACK fp16 | `embed_qwen3_embedding_0_6b_xnnpack_fp16.pte` | 1192.5 | 537.9 | 0.999998 | 2% |
-| Core ML fp32 | `embed_qwen3_embedding_0_6b_coreml_all.pte` | 1194.9 | **30.4** | 0.999957 | 17% |
 \*Mac arm64, median of 10, one 256-token sequence — a reference point for relative cost,
-not a device number. Eager fp32 on the same input is 203.9 ms, so the Core ML build is
-**6.7x eager**, 100% delegated in one subgraph.
 Agreement with eager is the first check and it separates nothing here. The one that decides
 is the last column: rank all eight test sentences against each other, then ask whether this
@@ -108,7 +116,7 @@ python convert/check_embed.py qwen3_embedding_0_6b fp32     # or fp16, int8, cor
 ## Not shipped: int8, and this is the first time it was close
-**int8 is 1064.0 MB — smaller than fp16's 1192.5 MB.** No other embedding model on this
 shelf manages that. Dynamic int8 quantises the linear weights and leaves the token
 embedding table in fp32, so it only wins when that table is a small share of the file, and
 here it is: 151669 x 1024 = **621 MB of the 2384 MB model, 26%**. Every other embedding
@@ -118,11 +126,11 @@ It is still not published, because the size question and the quality question ar
 different questions:
 ```
-worst retrieval score shift          0.0233
 closest fp32 top-1 decision          0.0234
 ```
-The error is **100% of the room available**. On these eight sentences it happens not to
 change any result — but eight deliberately dissimilar sentences are the friendliest corpus
 a retrieval model will ever see, and that 0.0234 is an upper bound on what a real corpus
 leaves. For scale, the same eight sentences give multilingual-e5-large a top-1 decision of
@@ -130,8 +138,13 @@ leaves. For scale, the same eight sentences give multilingual-e5-large a top-1 d
 change what one of the eight queries retrieves.
 So the bar here is half the decision distance rather than all of it, and int8 misses it.
-Correlation reads 0.990494, which would also have flagged it; the retrieval measurement is
 what says by how much and why.
 torch.export -> to_edge_transform_and_lower(partitioner) -> .pte
 (conversion scripts: [executorch-models](https://github.com/john-rocky/executorch-models))

 ## Verification
+| build | file | size (MB) | Mac median (ms)* | XNNPACK takes | worst cosine vs eager | retrieval budget spent |
+|---|---|---|---|---|---|---|
+| XNNPACK fp32 | `embed_qwen3_embedding_0_6b_xnnpack_fp32.pte` | 2383.7 | 184.0 | 80.5% | 1.000000 | 0% |
+| XNNPACK fp16 | `embed_qwen3_embedding_0_6b_xnnpack_fp16.pte` | 1192.4 | 521.5 | 70.2% | 0.999992 | 2% |
+| Core ML fp32 | `embed_qwen3_embedding_0_6b_coreml_all.pte` | 1195.1 | **30.0** | 100% | 0.999910 | 17% |
 \*Mac arm64, median of 10, one 256-token sequence — a reference point for relative cost,
+not a device number. Eager fp32 on the same input is 179.7 ms, so the Core ML build is
+**6.0x eager**, 100% delegated in one subgraph.
+**The XNNPACK column moved because the export stopped going through
+`F.scaled_dot_product_attention`.** It decomposes to `_safe_softmax`, whose guard for
+fully-masked rows is six operations XNNPACK cannot take, once per attention block. This
+model is right-padded, so no query row can lose every key and the guard can never fire:
+with 249 of 256 positions masked, `attn_implementation="eager"` agrees with the sdpa graph
+at cosine 1.00000000 while taking delegation from **71.8% to 80.5%** and fp32 from 203.9 ms
+to 184.0 ms.
 Agreement with eager is the first check and it separates nothing here. The one that decides
 is the last column: rank all eight test sentences against each other, then ask whether this
 ## Not shipped: int8, and this is the first time it was close
+**int8 is 1063.9 MB — smaller than fp16's 1192.4 MB.** No other embedding model on this
 shelf manages that. Dynamic int8 quantises the linear weights and leaves the token
 embedding table in fp32, so it only wins when that table is a small share of the file, and
 here it is: 151669 x 1024 = **621 MB of the 2384 MB model, 26%**. Every other embedding
 different questions:
 ```
+worst retrieval score shift          0.0258
 closest fp32 top-1 decision          0.0234
 ```
+The error is **110% of the room available**. On these eight sentences it happens not to
 change any result — but eight deliberately dissimilar sentences are the friendliest corpus
 a retrieval model will ever see, and that 0.0234 is an upper bound on what a real corpus
 leaves. For scale, the same eight sentences give multilingual-e5-large a top-1 decision of
 change what one of the eight queries retrieves.
 So the bar here is half the decision distance rather than all of it, and int8 misses it.
+Correlation reads 0.990459, which would also have flagged it; the retrieval measurement is
 what says by how much and why.
+The attention rewrite above made int8 **13% faster (174.8 ms to 151.7 ms) and 7 points
+more delegated (77.9% to 85.1%)** and did not change this verdict — it moved the score
+error from 100% of the room to 110%. Speed and quality are separate questions here, and
+only one of them was ever the reason this build is unpublished.
 torch.export -> to_edge_transform_and_lower(partitioner) -> .pte
 (conversion scripts: [executorch-models](https://github.com/john-rocky/executorch-models))