Upload README.md with huggingface_hub

README.md

@@ -24,15 +24,15 @@ each candidate together with the query and scores it properly.
 
 ## Variants
 
-| build | file | size (MB) | worst score error vs eager | Mac median (ms)* |
-|---|---|---|---|---|
-| fp32 | `rerank_ms_marco_minilm_l12_xnnpack_fp32.pte` | 133.6 | 0.0000 logits | 52.7 |
-| fp16 | `rerank_ms_marco_minilm_l12_xnnpack_fp16.pte` | 67.0 | 0.0070 logits | 102.1 |
-| Core ML (fp16, iOS) | `rerank_ms_marco_minilm_l12_coreml_all.pte` | 67.2 | 0.0428 logits | 13.3 |
+| build | file | size (MB) | worst score error vs eager | Mac median (ms)* | backend takes |
+|---|---|---|---|---|---|
+| fp32 | `rerank_ms_marco_minilm_l12_xnnpack_fp32.pte` | 133.6 | 0.0000 logits | 33.0 | 80.0% |
+| fp16 | `rerank_ms_marco_minilm_l12_xnnpack_fp16.pte` | 66.9 | 0.0070 logits | 52.2 | 69.0% |
+| Core ML (fp16, iOS) | `rerank_ms_marco_minilm_l12_coreml_all.pte` | 68.3 | 0.0369 logits | 13.2 | 100.0% |
 
 \*Mac arm64, single process, median of 10, one query-document pair at 512 tokens — a
 reference point for relative cost, not a device number. PyTorch eager fp32 on the same
-machine: 26.7 ms.
+machine: 34.6 ms.
 
 Correlation is not reported because it cannot be: the output is a single number, and the
 correlation of a one-element vector is undefined. The column above is the error in the
@@ -63,9 +63,30 @@ that answers it, feeding zeros instead of the real segment ids moves the score f
 The graph therefore takes three inputs. XLM-R rerankers (`type_vocab_size: 1`) have no
 second segment and take two; the signature follows the model rather than being made uniform.
 
+## The attention is eager, and that is the faster export
+
+`F.scaled_dot_product_attention` does not survive export as one operation. The edge
+dialect lowers it through `_safe_softmax`, whose guard against a row with no unmasked
+key at all leaves **eleven operations XNNPACK cannot take, in every attention block** —
+`scalar_tensor`, `where`, `mul.Scalar`, `logical_not`, `eq`, `full_like`, `any.dim`.
+Each one cuts the subgraph in two. The count is exact and does not vary by family:
+measured across this shelf, from a 4-layer cross-encoder to a 28-layer causal reranker,
+it is 11 per block every time.
+
+The guard is emitted whether or not it can ever fire, and here it cannot. It triggers
+only on `-inf`, which reaches the graph only because the sdpa path hands `F.sdpa` a
+**boolean** mask for PyTorch to fill; `attn_implementation="eager"` masks with
+`torch.finfo(dtype).min`, a large finite number, and never produces one. So the two
+arms differ only on rows that have no unmasked key — sdpa zeroes them, eager gives them
+a uniform row — and those are padding rows, which the pooling discards and every real
+query row masks out. Measured with all but eight positions masked, as adversarial as
+this shape gets, the two graphs agree to 1.1e-05.
+
+XNNPACK fp32 goes from **63.6% to 80.0%** delegated.
+
 ## Not shipped
 
-- **int8 (dynamic) is not shipped**: at 69.7 MB it is larger than the fp16 build's 67.0 MB, and its score error is 0.1148 logits. Dynamic int8 quantizes the linear weights and leaves the token embedding table in fp32, while fp16 halves that table too. The table here is 47 MB of a 134 MB model, and the arithmetic says int8 only comes out smaller when the table is under a third of the weights (45 MB) — measured on ten models on this shelf, the rule called all ten correctly.
+- **int8 (dynamic) is not shipped**: at 69.7 MB it is larger than the fp16 build's 66.9 MB, and its score error is 0.1073 logits. Dynamic int8 quantizes the linear weights and leaves the token embedding table in fp32, while fp16 halves that table too. The table here is 47 MB of a 134 MB model, and the arithmetic says int8 only comes out smaller when the table is under a third of the weights (45 MB) — measured on ten models on this shelf, the rule called all ten correctly.
 
 ## Verification