mlboydaisuke
/

Qwen3-Reranker-0.6B-ExecuTorch

@@ -29,17 +29,19 @@ attention_mask (1, 512) int64
 ## Variants
-| build | file | size (MB) | worst score error vs eager | Mac median (ms)* |
-|---|---|---|---|---|
-| fp32 | `rerank_qwen3_0_6b_xnnpack_fp32.pte` | 2383.7 | **0.0000 logits** | 381.5 |
-| fp16 | `rerank_qwen3_0_6b_xnnpack_fp16.pte` | 1192.5 | 0.0247 logits | 1058.0 |
-| **Core ML (fp16, iOS)** | `rerank_qwen3_0_6b_coreml_all.pte` | 1195.2 | 0.0573 logits | **85.0** |
-\*Mac arm64, median of 10, one 512-token pair — a reference point for relative cost, not
-a device number. Torch eager fp32 on the same machine is 261 ms. **Core ML is the one to
-use where it exists**: 4.5x faster than XNNPACK fp32 and 100% of the graph delegated,
-where XNNPACK takes 72% across 172 subgraphs. A reranker earns its keep over a list of
-fifty candidates, so that factor is the whole story.
 Correlation cannot judge this model — the output is a single number, and the correlation
 of a one-element vector is undefined. The gate is the score error in the model's own
@@ -113,17 +115,45 @@ The six test pairs include a Japanese passage that says what the query asks. It
 (+6.727). The shelf's `ms-marco-MiniLM` rerankers are English-only and put it far lower —
 which is the right answer for them, not a defect.
 ## Not shipped: int8
-`rerank_qwen3_0_6b_xnnpack_int8.pte` is **1064.0 MB** and runs at 327 ms — smaller and
-faster than the fp32 build. It is withheld on the number that decides.
-Dynamic int8 moves a pair score by **0.6456 logits**, against a narrowest adjacent gap of
 **0.7433** in the ranking it has to preserve. It happens to keep the order on these six
-pairs, but an error that is 87% of the distance between two neighbours is not a build
 that ranks reliably — a slightly different candidate list would reorder.
 The size was predictable before it was built: the token embedding table is
 151,669 x 1024 x 4 = 621 MB of the 2383.7 MB model, a **26.1%** share, and this shelf's
 rule `int8/fp16 = 0.5 + 1.5 x (table share)` puts the file at 0.891 of fp16. It came out
 at **0.892** — the same figure as Qwen3-Embedding-0.6B, which shares this backbone.

 ## Variants
+| build | file | size (MB) | worst score error vs eager | Mac ms* | backend takes |
+|---|---|---|---|---|---|
+| fp32 | `rerank_qwen3_0_6b_xnnpack_fp32.pte` | 2383.7 | **0.0001 logits** | 320.5 | 80.8% |
+| fp16 | `rerank_qwen3_0_6b_xnnpack_fp16.pte` | 1192.4 | 0.0268 logits | 1188.8 | 70.4% |
+| **Core ML (fp16, iOS)** | `rerank_qwen3_0_6b_coreml_all.pte` | 1196.2 | 0.0886 logits | **83.4** | 100% |
+\*Mac arm64, one 512-token pair, **fastest of five medians of ten** — a reference point for
+relative cost, not a device number. The host shares its cores with other work and a single
+median does not survive that; contention only ever adds time, so the fastest repetition is
+the one that means something. Torch eager fp32, measured the same way, is 320.1 ms.
+**Core ML is the one to use where it exists**: 3.8x faster than XNNPACK fp32, the whole
+graph in one delegated subgraph where XNNPACK takes 80.8% across 172. A reranker earns its
+keep over a list of fifty candidates, so that factor is the whole story.
 Correlation cannot judge this model — the output is a single number, and the correlation
 of a one-element vector is undefined. The gate is the score error in the model's own
 (+6.727). The shelf's `ms-marco-MiniLM` rerankers are English-only and put it far lower —
 which is the right answer for them, not a defect.
+## The attention is eager, and the left padding is why that needed checking
+`F.scaled_dot_product_attention` does not survive export as one operation. The edge
+dialect lowers it through `_safe_softmax`, whose guard against a row with no unmasked key
+at all leaves **eleven operations XNNPACK cannot take, in every attention block** —
+`scalar_tensor`, `where`, `mul.Scalar`, `logical_not`, `eq`, `full_like`, `any.dim`. Over
+28 layers that is 308 operations, each one cutting the subgraph in two. Exporting with
+`attn_implementation="eager"` removes them: **72.0% to 80.8% delegated**, fp32 from 381.5
+to 320.5 ms.
+**This is the one model on the shelf where that guard could plausibly have been doing
+work.** Its prompt is left-padded, so the leading rows of a causal mask have no unmasked
+key at all — exactly the case the guard exists for. Every other model here is
+right-padded, where even an all-padding row still sees the real tokens and the question
+never arises.
+So it was measured rather than assumed. With 109 real tokens of 512, sdpa against eager:
+no `NaN` appears on either arm, and the scored last position reads **54.7696 against
+54.7695**. The reason is that the guard fires only on `-inf`, and eager masks with
+`torch.finfo(dtype).min`, a large finite number — a fully-masked row comes back uniform
+instead of zeroed. Both answers are defined; they differ only about padding rows, and
+this graph reads position −1, which is a real token by construction.
 ## Not shipped: int8
+`rerank_qwen3_0_6b_xnnpack_int8.pte` is **1063.9 MB** and runs at 265.4 ms — smaller and
+faster than the fp32 build, and 85.3% of it delegates, the most of any build here. It is
+withheld on the number that decides.
+Dynamic int8 moves a pair score by **0.6336 logits**, against a narrowest adjacent gap of
 **0.7433** in the ranking it has to preserve. It happens to keep the order on these six
+pairs, but an error that is 85% of the distance between two neighbours is not a build
 that ranks reliably — a slightly different candidate list would reorder.
 The size was predictable before it was built: the token embedding table is
 151,669 x 1024 x 4 = 621 MB of the 2383.7 MB model, a **26.1%** share, and this shelf's
 rule `int8/fp16 = 0.5 + 1.5 x (table share)` puts the file at 0.891 of fp16. It came out
 at **0.892** — the same figure as Qwen3-Embedding-0.6B, which shares this backbone.
+Going eager (below) made this build 13% faster and took it from 78.1% to 85.3%
+delegated. It did not move the verdict: speed and quality are separate questions,
+and what withholds this file is the score error, which barely changed.