Upload README.md with huggingface_hub

README.md

@@ -0,0 +1,129 @@
+---
+license: apache-2.0
+tags:
+- executorch
+- xnnpack
+- pte
+- on-device
+- text-ranking
+base_model:
+- Qwen/Qwen3-Reranker-0.6B
+---
+# Qwen3-Reranker-0.6B — ExecuTorch
+
+A reranker that is not a cross-encoder. The other five on this shelf are BERTs with a
+regression head; this one is a **causal language model asked a yes/no question**, and
+the score is how much more it would answer "yes" than "no". Same job — read a query and
+one document together and score the pair — with a different machine underneath.
+
+```
+input_ids      (1, 512) int64   the prompt, LEFT-padded
+attention_mask (1, 512) int64
+    -> score   (1, 1)   fp32    log-odds: logit("yes") - logit("no")
+```
+
+- **Source**: [Qwen/Qwen3-Reranker-0.6B](https://huggingface.co/Qwen/Qwen3-Reranker-0.6B) — 595.8M parameters, 28 Qwen3 layers, hidden 1024, 151,669 vocabulary
+- **License**: apache-2.0
+- **Output**: one number per pair. Higher is more relevant; `sigmoid(x)` maps it to 0..1
+  and does not change the ordering.
+
+## Variants
+
+| build | file | size (MB) | worst score error vs eager | Mac median (ms)* |
+|---|---|---|---|---|
+| fp32 | `rerank_qwen3_0_6b_xnnpack_fp32.pte` | 2383.7 | **0.0000 logits** | 381.5 |
+| fp16 | `rerank_qwen3_0_6b_xnnpack_fp16.pte` | 1192.5 | 0.0247 logits | 1058.0 |
+| **Core ML (fp16, iOS)** | `rerank_qwen3_0_6b_coreml_all.pte` | 1195.2 | 0.0573 logits | **85.0** |
+
+\*Mac arm64, median of 10, one 512-token pair — a reference point for relative cost, not
+a device number. Torch eager fp32 on the same machine is 261 ms. **Core ML is the one to
+use where it exists**: 4.5x faster than XNNPACK fp32 and 100% of the graph delegated,
+where XNNPACK takes 72% across 172 subgraphs. A reranker earns its keep over a list of
+fifty candidates, so that factor is the whole story.
+
+Correlation cannot judge this model — the output is a single number, and the correlation
+of a one-element vector is undefined. The gate is the score error in the model's own
+units together with whether the ranking survives. Over six real pairs, **every shipped
+build reproduces eager's order**, and the narrowest adjacent gap in that ranking is
+0.7433 logits.
+
+## Running it
+
+**1. Build the prompt.** The model was trained to read one specific frame, and it is not
+a chat model you can prompt freely:
+
+```python
+PREFIX = ('<|im_start|>system\nJudge whether the Document meets the requirements '
+          'based on the Query and the Instruct provided. Note that the answer can '
+          'only be "yes" or "no".<|im_end|>\n<|im_start|>user\n')
+SUFFIX = "<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n"
+INSTRUCT = "Given a web search query, retrieve relevant passages that answer the query"
+
+text = (f"{PREFIX}<Instruct>: {INSTRUCT}\n<Query>: {query}\n"
+        f"<Document>: {document}{SUFFIX}")
+```
+
+`INSTRUCT` is a real input, not decoration: it names the retrieval task, and Qwen
+reports it is worth a few points to write your own instead of the generic default.
+
+**2. Tokenise with LEFT padding to exactly 512.**
+
+```python
+tokenizer = AutoTokenizer.from_pretrained(repo, padding_side="left")
+batch = tokenizer([text], padding="max_length", truncation=True, max_length=512,
+                  return_tensors="pt")
+assert batch["attention_mask"][0, -1] == 1
+```
+
+**This is the one that will bite.** The graph reads position −1 unconditionally, which
+is the last *real* token only when the padding is on the left. Right-padded input scores
+a pad token and returns a confident-looking number that means nothing — nothing raises.
+The assertion above is one line and catches it.
+
+**3. Run it once per candidate** and sort by the score, descending.
+
+## The output projection is folded into one vector
+
+The score needs two of the 151,669 logits, and the difference of two dot products is one
+dot product against the difference of two rows:
+
+```
+logit_yes - logit_no = h . W[yes] - h . W[no] = h . (W[yes] - W[no])
+```
+
+So `lm_head` never runs. A 151,669-wide matmul per pair does not happen, and the delegate
+does not take its own copy of a 621 MB table — which is exactly why this shelf's Whisper
+decoders come out bigger than their weights.
+
+The two vocabulary ids are read from the repo's own `1_LogitScore/config.json`
+(**9693** for "yes", **2152** for "no") rather than looked up through the tokenizer,
+where a leading-space variant is a different token.
+
+**Checked against the reference, at matching precision.** sentence-transformers'
+`CrossEncoder` scores these pairs at +6.6875 / −6.0625 / −7.2812 and the folded head at
++6.7271 / −6.1107 / −7.3159 — a gap of 4.8e-02 that has nothing to do with the fold.
+The reference loads the checkpoint in its own **bfloat16**; run both arms in fp32 and the
+folded head matches it to **4.673e-05**, while the reference's bf16 and fp32 arms differ
+from each other by the full 4.820e-02.
+
+## It ranks across languages
+
+The six test pairs include a Japanese passage that says what the query asks. It comes out
+**first of six** at +7.470, above the English document carrying the actual number
+(+6.727). The shelf's `ms-marco-MiniLM` rerankers are English-only and put it far lower —
+which is the right answer for them, not a defect.
+
+## Not shipped: int8
+
+`rerank_qwen3_0_6b_xnnpack_int8.pte` is **1064.0 MB** and runs at 327 ms — smaller and
+faster than the fp32 build. It is withheld on the number that decides.
+
+Dynamic int8 moves a pair score by **0.6456 logits**, against a narrowest adjacent gap of
+**0.7433** in the ranking it has to preserve. It happens to keep the order on these six
+pairs, but an error that is 87% of the distance between two neighbours is not a build
+that ranks reliably — a slightly different candidate list would reorder.
+
+The size was predictable before it was built: the token embedding table is
+151,669 x 1024 x 4 = 621 MB of the 2383.7 MB model, a **26.1%** share, and this shelf's
+rule `int8/fp16 = 0.5 + 1.5 x (table share)` puts the file at 0.891 of fp16. It came out
+at **0.892** — the same figure as Qwen3-Embedding-0.6B, which shares this backbone.