Publish third-pass feed ranker

README.md

@@ -9,33 +9,32 @@ tags:
 - enterprise-feed
 - learning-to-rank
 pipeline_tag: text-classification
-base_model: cross-encoder/mmarco-mMiniLMv2-L12-H384-v1
+base_model: nreimers/mMiniLMv2-L12-H384-distilled-from-XLMR-Large
 library_name: transformers
 ---
 
 # Third-Pass Feed Ranker (job-title × feed-post relevance)
 
 A lightweight **cross-encoder** that scores how relevant an enterprise social-feed post is to a
-viewer, given only the viewer's **job title** and the **post text**. It is a *third-pass
-reranker*: it re-scores a small candidate slate (~20 items) from earlier passes to surface a
-genuinely job-relevant post that was buried below the top slot.
+viewer, given only the viewer's **job title** and the **post text**. It is a *third-pass reranker*:
+it re-scores a small candidate slate (~20 items) from earlier passes to surface a genuinely
+job-relevant post that was buried below the top slot.
 
 - **Input:** `job_title` (query) + `post_text` (passage) → single relevance score (higher = more relevant)
-- **Base:** `cross-encoder/mmarco-mMiniLMv2-L12-H384-v1` (multilingual MiniLM-L12, ~117M params)
-- **Runtime:** server-class **CPU** at scale (~120 pairs/sec on 8 CPU threads; INT8/ONNX 3–6× more)
-- **Trained with a listwise ranking objective** (optimizes which item wins the slate), not just
-  pointwise regression — so it is markedly better at putting the buried relevant post at #1.
+- **Base:** `nreimers/mMiniLMv2-L12-H384-distilled-from-XLMR-Large` — a **generic** distilled
+  multilingual MiniLM-L12 (~117M params), *not* a search reranker. Fine-tuning defines relevance
+  purely from this task's data, without a general-search "dense-technical-text = relevant" prior.
+- **Runtime:** server-class **CPU** at scale (~120 pairs/sec on 8 CPU threads; INT8/ONNX 3–6× more).
+- **Trained with a listwise ranking objective** (optimizes which item wins the slate), pointwise
+  inference unchanged.
 
-## It ranks by MEANING — and the role keyword is neither a crutch nor a trap
-Earlier iterations keyed on whether a post *named* the viewer's role. This model is trained so the
-role keyword carries **zero** relevance information: during training the role name is injected into
-a random half of *all* posts regardless of their true relevance. As a result:
-
-- Naming the viewer's role changes a relevant post's score by **≈0.00** (it neither helps nor hurts).
-- Masking role words in the text does **not** change the ranking (**0% keyword-dependence**).
-
-Relevance is decided by **substance** (e.g. "patient-monitoring escalation protocol" → nurse),
-which is exactly what a keyword matcher cannot do.
+## What it does — and what it deliberately avoids
+- **Ranks by MEANING, not keywords.** Relevance is decided by substance (e.g. "patient-monitoring
+  escalation protocol" → a clinical role). Masking role words barely changes the ranking, and the
+  role keyword is trained to be uncorrelated with the label (neither a shortcut nor a penalty).
+- **Not fooled by dense technical jargon.** Training injects dense-technical posts as hard negatives
+  for non-technical roles, so a machine-learning or software post does **not** score as broadly
+  relevant to, say, a nurse or a chef — it only wins for roles it actually fits.
 
 ## Usage — scoring + the gate
 ```python
@@ -50,63 +49,57 @@ def scores(title, posts):
     with torch.no_grad():
         return model(**enc).logits.squeeze(-1).tolist()
 
-TAU = 3.0    # promotion margin; calibrate per deployment (see below)
+TAU = 1.0    # promotion margin; calibrate per deployment (see below)
 def third_pass(title, slate):
     s = scores(title, slate)
     challenger = max(range(1, len(slate)), key=lambda i: s[i])
     return ("promote", challenger) if s[challenger] - s[0] > TAU else ("no_change", None)
 ```
 **The model only scores.** The gate (which item to move to slot 1, and the τ threshold) is *your*
-logic — none of it is in the weights. Calibrate τ on your own feeds to trade recovery vs.
-false-promotion. **τ is distribution-sensitive.** This model was trained with a listwise objective
-that widens the score range, so its natural margins are large: **τ≈3.0** holds false-promotion
-around 10% on realistic feeds — much higher than the τ≈0.15 that suited the older pointwise model.
-Re-calibrate on a sample of your own no-relevance feeds.
+logic — none of it is in the weights. **τ is distribution-sensitive**: on realistic feeds a value
+around **0.5–1.5** trades recovery vs. false-promotion; re-calibrate on a sample of your own
+no-relevance feeds. Higher τ is more conservative.
+
+## Evaluation
+Measured on an **external hold-out of 157 job titles that never appear in training** (adjacent
+real-world variants of trained roles + high-volume roles the training taxonomy under-covers), each
+with substance gems buried in 20-item feeds:
 
-## Evaluation (held-out roles AND posts; synthetic)
 | metric | value |
 |---|---|
-| **realistic feed: buried relevant post surfaced to #1** | **0.84** |
-| role-keyword effect on a relevant post's score (target ≈ 0) | **≈ 0.00** |
-| keyword-dependence (masked vs normal ranking) | **0%** |
-| gated recovery on adversarial worst-case slates (τ=3.0) | 0.08–0.26 |
-| protection of high-value org announcements | ~0.81–0.92 |
-
-On a **realistic feed** — the relevant post competing against ordinary filler — the model puts it
-first **84%** of the time. On a deliberately **adversarial worst-case slate** (~9 simultaneous
-strong competitors, including posts genuinely relevant to *other* roles), exact-#1 recovery drops
-to ~0.29 (top-3 ~0.59); that is a stress bound, not a realistic-feed number.
+| relevant post surfaced to #1 in its feed (novel titles) | **~60%** |
+| role's gem beats a dense-technical distractor | **~85%** |
+| realistic-density recovery (buried gem promoted) | **~0.64** @ low τ |
+| false-promotion on no-relevance feeds | **~0.3%** |
 
-**Announcement protection is a deliberate trade.** Because the listwise objective makes a strongly
-relevant post outscore an incumbent announcement more often, must-see-announcement protection is
-~0.81–0.92 (down from ~1.0 in the pointwise model). This model favors *recovering* a buried
-role-relevant post over *protecting* every announcement. If you need stricter announcement
-protection, raise τ or add an explicit announcement guard in your decision logic.
+On a deliberately **adversarial worst-case slate** (~9 simultaneous strong competitors, including
+posts relevant to *other* roles), exact-#1 recovery drops to ~0.3 — a stress bound, not a
+realistic-feed number.
 
 ## Intended use & limitations
 - Re-ranking short enterprise-feed candidate slates by job-title relevance; abstains on role-less
   titles.
-- **Trained entirely on SYNTHETIC data** (LLM-generated posts + synthetic slates) — it has not
-  seen real feed content. **Validate on your own data before production.**
-- Being a small CPU model, it does not perfectly resolve *many* simultaneous strong competitors
-  (e.g. distinguishing this role's post from several other-role-relevant posts at once); it excels
-  when the relevant item competes against ordinary filler.
+- **Trained entirely on SYNTHETIC data** (LLM-generated posts + synthetic slates). **Validate on
+  your own data before production.**
+- **Fine role-discrimination is limited.** It reliably separates a relevant post from ordinary
+  filler, but distinguishing a role's exact post from a *closely adjacent* role's post is near the
+  capacity ceiling of a 117M model.
+- **Out-of-distribution phrasing is the weak axis.** Terse status fragments, log/ticket snippets,
+  and atypical wording score more noisily than well-formed posts — for all roles.
+- **Very high-volume roles absent from the training taxonomy** (a few common titles) generalize at
+  reduced magnitude; adding them to training stabilizes them.
 - Cross-lingual mixing (non-English title vs English-only feed) is weaker than same-language feeds.
-- Relevance is title-driven; recency/importance beyond the relevance signal must be added in your
-  decision logic.
+- Relevance is title-driven; recency/importance beyond relevance must live in your decision logic.
 
 ## Training & method
-O*NET-derived job titles; LLM-generated posts where relevance is by **substance**; adversarial
-20-item slates (gem / abstain / announcement types) with held-out roles+posts. Two things make
-this version different from a plain pointwise regressor:
-1. **Role-keyword rebalance** — the viewer's role name is injected into a random half of *all*
-   posts during training (independent of the label), so the keyword is decorrelated from relevance
-   and the model cannot use it as either a shortcut or a penalty.
-2. **Listwise ranking loss** — training batches whole slates and adds a softmax cross-entropy term
-   that pushes the truly-relevant post to rank #1, on top of a pointwise anchor. Inference stays
-   pointwise (one title×post score at a time), so CPU cost is unchanged.
-Headline metric is **role-masked** recovery so a keyword shortcut cannot inflate it.
+Job titles derived from a public occupation taxonomy; LLM-generated posts where relevance is by
+**substance** and the role is never named; adversarial 20-item slates with held-out roles and posts,
+in both well-formed and terse registers. Objective = pointwise relevance MSE **plus a listwise
+ranking loss** that pushes the relevant post to rank #1 within its slate. Two anti-bias measures:
+the role keyword is **rebalanced** to be label-uncorrelated, and **dense-technical hard negatives**
+are injected into non-technical roles' slates so technical vocabulary is not a global relevance
+signal. Headline metrics are measured on titles **never seen in training**.
 
 ## License & attribution
-Apache-2.0. Inherits from `cross-encoder/mmarco-mMiniLMv2-L12-H384-v1` — verify its license carries
-through. Training posts were generated with a Qwen model; review the applicable terms.
+Apache-2.0. Inherits from `nreimers/mMiniLMv2-L12-H384-distilled-from-XLMR-Large` — verify its
+license carries through. Training posts were generated with a Qwen model; review the applicable terms.

config.json

@@ -26,8 +26,6 @@
   "num_attention_heads": 12,
   "num_hidden_layers": 12,
   "pad_token_id": 1,
-  "position_embedding_type": "absolute",
-  "sbert_ce_default_activation_function": "torch.nn.modules.linear.Identity",
   "tie_word_embeddings": true,
   "transformers_version": "5.16.1",
   "type_vocab_size": 1,

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:16acddaa92b8791ef6d9276250cbd88f7dd6774c95a1b2789489681b3d9c66b4
+oid sha256:6ed0648a19134cc5ce1ca15923bb9bc6d614bd279d93d515b376a7617f9eab32
 size 470588492