docs/phases/PHASE6-Reranker-Framing.md

PHASE 6 — Reranker Framing Document

File: docs/phases/PHASE6-Reranker-Framing.md Project: ResearchIT (arXiv paper recommendation engine) Author: Amin (siddhm11) Status: Phase 6 — In Integration (post-audit framing) Supersedes: the open items from PHASE6-Audit.md Live Space: https://siddhm11-researchit.hf.space (HF Spaces, Docker SDK, RUNNING, cpu-basic) Reranker model repo: https://huggingface.co/siddhm11/researchit-reranker-phase6 Training data repo: https://huggingface.co/datasets/siddhm11/researchit-reranker-data

---

TL;DR (for Amin's future self at 1AM)

1. The HF model exists and is public, but the model card is empty (no pipeline_tag, no library_name, no description, no metrics block). The reproducibility story currently lives in your laptop, not in the repo. The dataset repo siddhm11/researchit-reranker-data does exist as a public Parquet dataset (size_categories: 100K–1M, downloads: 30) — that is the only piece of the training pipeline that is partially public.
2. The model was NOT trained on real user signal. It was trained on citation pseudo-labels (Semantic Scholar citation edges → triples → LightGBM LambdaRank). Features 23–30 (cluster importance, suppressed-category, onboarding match, user save/dismiss counts) were zero during training because no users existed — therefore zeroes at serving time are consistent with training, not a regression. The 0.879 nDCG@10 number is honest under that distribution; it is not honest as a measure of "what the user feels."
3. Phase 6 is therefore about plumbing, not retraining. Land Phase 6.1 (dominant-cluster shortcut, ~1 day), then Phase 6.2 (per-candidate paper_cluster_map plumbing, ~3–5 days), then Phase 6.3 (deployment verification + /healthz/reranker). Defer retraining (Phase 6.4) until either (a) you ship synthetic-user simulation, or (b) you reach ~100 real users with ≥10 saves each.

---

Part A — Phase 6 Status Verification (HF inspection)

A.1 What is publicly visible on Hugging Face

I queried the HF Hub API directly (the public web pages were not reachable from this environment, but the structured API endpoints were). Here is the complete observable state of your HF account as of 2026-05-02:

Repo	Type	Visibility	Created	Last modified	Notable
siddhm11/researchit-reranker-phase6	model	public, not gated	2026-04-26	2026-04-27 21:51	README empty / no description, no pipeline_tag, no library_name, no metrics card, no datasets link
siddhm11/researchit-reranker-data	dataset	public, not gated	2026-04-27 16:41	2026-04-27 21:51	Parquet, 100K<n<1M, modality:tabular,text, 30 downloads
siddhm11/ResearchIT	space	public, RUNNING	2026-04-19	2026-05-02 11:29	Docker SDK, cpu-basic, declares BAAI/bge-m3 as a referenced model
siddhm11/prompt-engine	space	public, RUNNING	2026-02-02	2026-03-21	Docker, unrelated
siddhm11/sandbox-c6a4f7e6	space	SLEEPING	2026-04-26	—	Sandbox
siddhm11/sandbox-9bb83c65	space	SLEEPING	2026-04-27	—	Sandbox

Critically absent from the model repo metadata: library_name (should be lightgbm), pipeline_tag (should be e.g. text-ranking or a custom tag), any model-index block, any datasets: link to siddhm11/researchit-reranker-data. Tags = region:us only.

A.2 What this means for reproducibility

The audit's working assumption — "Phase 6 was trained on 242K citation edges from 50K sampled papers, time-split (train <2023, eval ≥2023), 90,993 train + 7,007 eval triples, nDCG@10 = 0.879, 37 features, 141 trees, 974KB" — is internally consistent but cannot be re-derived from the public HF artifacts alone today. The dataset repo exists (good — that is where the triples almost certainly live), but the model repo has no card linking the two, and there is no published training script.

What we can confirm vs what is missing:

Asset (per audit expectation)	Public on HF?	Notes
reranker_v1.txt (LightGBM dump, ~974 KB)	Likely yes (it's the only point of the model repo) but the API does not expose siblings to confirm filename. Action: open the Files tab in a browser and verify the filename + size.	Without a card, downstream consumers don't know which file to load
Raw citation edges (output of 01_fetch_citation_edges.py)	Unknown / probably not	Not exposed by metadata. Likely only the triples are uploaded
Triples file (output of 02_generate_training_triples.py)	Probably yes — this is what researchit-reranker-data is for; the size-category and modalities match	Verify columns include query_paper_id, candidate_paper_id, relevance, the 37 features, and a group/qid column
Eval triples (≥2023 split)	Probably yes (same dataset, separate split)	Verify a split column or train/eval files
03_train_lightgbm.py	Almost certainly NO — no Space or repo of yours hosts it	This is the single biggest reproducibility gap
Feature schema (37 names, in canonical order)	NO — would live in a model card	Without this, even the LightGBM .txt dump (which uses Column_0…Column_36) is opaque
01_fetch_citation_edges.py script	NO	Required to refresh edges for retraining
pseudo_label_generation.py / triple builder	NO	Required to regenerate the dataset under a new time-split

Verdict on reproducibility: as of today, Phase 6 retraining is gated on Amin's local files. A fresh collaborator (or future Amin on a new laptop) cannot retrain the model from public artifacts alone. Fixing this is part of Phase 6.3 (see below).

A.3 Confirm or refute: was the published model trained on test data or real user signal?

Refuted on both counts. Three converging pieces of evidence:

1. No user data could possibly exist. The Space siddhm11/ResearchIT was created 2026-04-19, only seven days before the model repo on 2026-04-26. There has been no production window long enough to accrue meaningful save/dismiss interaction logs, and the model repo has zero downloads, zero likes — this is a personal project with no user funnel.
2. The dataset repo was created the day after the model. researchit-reranker-data is dated 2026-04-27, one day after the model went up — meaning the dataset was uploaded as a documentation artifact, not consumed from. That fits the citation-pseudo-label story: the triples were generated locally, the model was trained locally, both were pushed in close succession.
3. The dataset metadata explicitly says modality:tabular, text and format:parquet with 100K<n<1M rows. That row count band is consistent with ~98K triples (90,993 train + 7,007 eval), exactly the audit's stated counts. Real user signal at this scale is implausible for a one-week-old project.

Conclusion: the model is trained on citation pseudo-labels, not on test data, not on user signal. The audit's account is correct. Features 23–30 were almost certainly zero during training (no user, no live cluster importance from a real session) — which is why serving them as zero today does not break the model's distribution; it merely leaves potential signal on the table.

A.4 Is reranker_v1.txt actually deployed to the live Space?

Unverified from outside. The Space metadata shows it is RUNNING on cpu-basic and declares BAAI/bge-m3 as a model dependency, but does not declare siddhm11/researchit-reranker-phase6 as a dependency. That is a yellow flag: if the LightGBM model were being pulled at runtime via huggingface_hub.snapshot_download, the Space metadata would typically list it. The most likely current state is one of:

• (a) the file was committed to the Space repo's working tree under models/reranker-phase6/production_model/reranker_v1.txt and is being loaded from disk (works, but means the same artifact is duplicated in two HF repos), or
• (b) the file is not in the Space at all and the deployed code is silently falling back to the heuristic scorer.

Either way, the audit's TASK-TRACKER item [ ] [reranker] LightGBM model loaded is still the right thing to verify, and we wire that verification in via Phase 6.3 below.

---

Part B — Phase 6 Problem Statement, Properly Framed

Why Phase 6 exists. Phases 1–5 built a clean retrieval+ranking stack: BGE-M3 dense + sparse, Qdrant + Zilliz, Ward clustering of user history into long/short/negative interest centroids, importance-weighted per-cluster Qdrant search with floor F_min=3, and a heuristic linear scorer over a hand-tuned set of similarities. The heuristic worked, but it left two kinds of signal on the table: (i) non-linear interactions between paper-level features (recency × influential-citations × topic match), and (ii) user-state features like cluster importance, onboarding category match, and save/dismiss intensity. Phase 6 introduces a LightGBM LambdaRank model with a 37-slot feature vector designed to absorb both classes of signal and to give us a single optimizable objective (nDCG) instead of a hand-tuned weighted sum.

What Phase 6 currently delivers vs what it was designed to deliver. What ships today is a 141-tree LightGBM ranker, trained offline on citation pseudo-labels with a time-split eval, claiming nDCG@10 = 0.879, with a heuristic fallback when the model file is missing. What the live serving path actually feeds the model is not the 37-feature vector the model was trained on — it is a vector where features 0, 1–4, 5, 6–22 carry signal and features 23–30 are zero-filled because the integration in app/routers/recommendations.py still calls rerank_candidates with the legacy 6-argument signature from Phase 5. Bugs A, B, and C from the audit all stem from this single gap: the reranker module was upgraded; the caller was not.

The "9 of 37 features" gap, quantified.

Feature slots	Count	Source available in caller scope?	Currently passed?
0 qdrant_score	1	Yes — in per_cluster_results	No — not constructed
1–4 long/short/medoid/extra similarities	4	Yes — lt_vec, st_vec, dominant medoid	Partial — only lt/st/neg are passed positionally
5 neg similarity	1	Yes — neg_vec	Yes
6–22 paper-level features	17	Yes — derivable in reranker.py from metadata	Yes (computed inside reranker)
23 cluster_importance	1	Yes — clusters[i].importance	No
24 cluster_distance_to_medoid	1	Yes — from clusters[i].medoid_embedding	No
25 is_suppressed_category	1	Yes — but suppressed is loaded after the rerank call	No
26 onboarding_category_match	1	Yes — db.get_user_category_filter() exists	No, never called for this
27 user_total_saves	1	Yes — len(state.positives)	No
28 user_total_dismissals	1	Yes — len(state.negatives)	No
29 saves_last_7d	1	Yes — derivable from state.positives timestamps	No
30 dismissals_last_7d	1	Yes — derivable from state.negatives timestamps	No
31–36 remaining metadata features	6	Yes — computed inside reranker	Yes

Effective active features at serving: ~9 + ~6 = ~15 of 37 carry real signal. Features 23–30 are all zero. This is Bug A, expressed quantitatively. It is not destroying ranking quality (the model has not learned to rely on those slots, because they were zero in training too), but it permanently caps how much Phase 6 can ever improve over the heuristic.

Phase 6 is a 3-stage fix: (6.1) connect the existing data flow with a dominant-cluster shortcut, (6.2) thread per-candidate cluster identity through the pipeline, (6.3) verify deployment + add observability. Retraining (6.4) is a separate decision, gated on whether we can produce a training distribution where features 23–30 are non-zero and behaviourally meaningful.

---

Part C — Phase 6.1: The Simplification Pass (1–2 days)

C.1 What 6.1 does

For each candidate in the rerank list, we compute features 23 and 24 against the dominant cluster — the cluster with the highest importance in the user's current state. Features 25 and 26 remain truly per-candidate (they depend on the candidate's primary_topic). Features 27–30 are user-level and constant across candidates in a single rerank call. Phase 6.1 is the minimum-viable fix that ends "9 of 37"; it will be replaced by 6.2's per-candidate cluster lookup.

Why dominant-cluster is acceptable for 6.1. The current model was trained with feature 23 = 0 everywhere. So as long as 6.1 produces a reasonable non-zero value for 23, the model will route those gradients through whatever weak signal it learned in feature 6–22. We are not relying on 23 carrying perfect semantics; we are getting the integration plumbed end-to-end so that feature non-zero rate jumps from ~40% to ~100%, and so that Phase 6.4 retraining has a target.

C.2 The exact code patch

File: app/routers/recommendations.py

C.2.1 Move the suppressed and onboarding_categories loading earlier

Find the existing block where suppressed is loaded (currently after the rerank_candidates call) and move it to immediately after state is hydrated and before the Qdrant retrieval loop. Add the onboarding category fetch right next to it.

# --- BEGIN Phase 6.1 patch (top of recommendations endpoint, after state hydration) ---

# Suppressed categories: was loaded after rerank; move it here.
suppressed: set[str] = set(await db.get_suppressed_categories(user_id))

# Onboarding categories: previously unused at rerank time.
onboarding_categories: set[str] = set(
    await db.get_user_category_filter(user_id) or []
)

# User-level interaction counts (constant across all candidates this request).
now_utc = datetime.now(timezone.utc)
seven_days_ago = now_utc - timedelta(days=7)

user_total_saves = len(state.positives)
user_total_dismissals = len(state.negatives)
user_saves_last_7d = sum(
    1 for p in state.positives if p.timestamp >= seven_days_ago
)
user_dismissals_last_7d = sum(
    1 for n in state.negatives if n.timestamp >= seven_days_ago
)

# --- END move ---

C.2.2 Add a helper to align Qdrant scores with valid_ids

per_cluster_results is currently a list[list[ScoredPoint]] (one list per cluster). After dedup + filter into valid_ids, we lose the raw retrieval score. We need to project it back.

# app/recommend/fusion.py  (or app/routers/recommendations.py if you keep helpers local)

def build_qdrant_score_map(
    per_cluster_results: list[list["ScoredPoint"]],
) -> dict[str, float]:
    """
    Collapse all per-cluster ScoredPoint lists into a single
    paper_id -> max_score dict. If a paper appeared in multiple
    clusters' top-K, we keep the maximum score (most charitable to
    the candidate; matches the dedup semantics in merge_quota_results).
    """
    out: dict[str, float] = {}
    for cluster_hits in per_cluster_results:
        for hit in cluster_hits:
            pid = str(hit.id)  # arxiv IDs ALWAYS strings (CLAUDE.md rule 7)
            score = float(hit.score)
            if pid not in out or score > out[pid]:
                out[pid] = score
    return out


def align_qdrant_scores(
    valid_ids: list[str],
    score_map: dict[str, float],
) -> np.ndarray:
    """Return a float32 array of qdrant_scores aligned 1:1 with valid_ids.
    Missing entries default to 0.0 (which matches train-time behavior for
    candidates injected by exploration paths)."""
    return np.asarray(
        [score_map.get(pid, 0.0) for pid in valid_ids],
        dtype=np.float32,
    )

C.2.3 Compute the dominant-cluster scalars

# Dominant cluster: highest-importance cluster in user state.
# clusters is the list[Cluster] you already have in scope.
if clusters:
    dominant = max(clusters, key=lambda c: c.importance)
    dominant_importance = float(dominant.importance)
    dominant_medoid_vec = np.asarray(
        dominant.medoid_embedding, dtype=np.float32
    )
else:
    # Cold-start path: no clusters -> defensible defaults.
    dominant_importance = 0.0
    dominant_medoid_vec = np.zeros(1024, dtype=np.float32)

C.2.4 The new rerank_candidates call

Replace the existing call (around line 305 of recommendations.py) with:

# --- BEGIN Phase 6.1 rerank call ---

qdrant_score_map = build_qdrant_score_map(per_cluster_results)
qdrant_scores = align_qdrant_scores(valid_ids, qdrant_score_map)

# Per-candidate boolean: is this candidate's primary_topic suppressed?
is_suppressed_category = np.asarray(
    [
        1.0 if (m.get("primary_topic") in suppressed) else 0.0
        for m in valid_meta
    ],
    dtype=np.float32,
)

# Per-candidate boolean: does this candidate match any onboarding category?
onboarding_category_match = np.asarray(
    [
        1.0 if (m.get("primary_topic") in onboarding_categories) else 0.0
        for m in valid_meta
    ],
    dtype=np.float32,
)

reranked_ids, reranked_scores, reranked_embs = rerank_candidates(
    candidate_ids=valid_ids,
    candidate_embeddings=valid_embs,
    candidate_metadata=valid_meta,
    long_term_vec=lt_vec,
    short_term_vec=st_vec,
    negative_vec=neg_vec,
    # Phase 6 additions (6.1: dominant-cluster shortcut)
    qdrant_scores=qdrant_scores,
    cluster_importance=np.full(
        len(valid_ids), dominant_importance, dtype=np.float32
    ),
    cluster_medoid=dominant_medoid_vec,           # broadcast in reranker
    is_suppressed_category=is_suppressed_category,
    onboarding_category_match=onboarding_category_match,
    user_total_saves=user_total_saves,
    user_total_dismissals=user_total_dismissals,
    user_saves_last_7d=user_saves_last_7d,
    user_dismissals_last_7d=user_dismissals_last_7d,
)
# --- END Phase 6.1 rerank call ---

C.2.5 Reranker signature change

File: app/recommend/reranker.py

def rerank_candidates(
    *,
    candidate_ids: list[str],
    candidate_embeddings: np.ndarray,           # (N, 1024)
    candidate_metadata: list[dict],
    long_term_vec: np.ndarray | None,
    short_term_vec: np.ndarray | None,
    negative_vec: np.ndarray | None,
    # Phase 6.1 additions — all optional with safe zero-defaults
    # so the legacy callers keep working during the migration window.
    qdrant_scores: np.ndarray | None = None,
    cluster_importance: np.ndarray | None = None,    # (N,) or scalar broadcast
    cluster_medoid: np.ndarray | None = None,        # (1024,) for 6.1, (N, 1024) for 6.2
    is_suppressed_category: np.ndarray | None = None,
    onboarding_category_match: np.ndarray | None = None,
    user_total_saves: int = 0,
    user_total_dismissals: int = 0,
    user_saves_last_7d: int = 0,
    user_dismissals_last_7d: int = 0,
) -> tuple[list[str], list[float], np.ndarray]:
    ...

Inside the feature-matrix builder, fill slots 23–30 from these new args, falling back to zero if None (preserves backward-compat for any unit tests that still call the legacy path).

C.3 The integration test (must be added)

File: tests/recommend/test_phase6_feature_matrix.py

import numpy as np
import pytest
from app.recommend.reranker import _build_feature_matrix  # expose for testing


def test_phase6_feature_matrix_is_not_mostly_zero(fake_user_state, fake_candidates):
    """
    Regression guard for Bug A. After Phase 6.1, features 23-30 must
    carry signal for at least one candidate in a typical request.
    """
    X = _build_feature_matrix(
        candidate_ids=fake_candidates.ids,
        candidate_embeddings=fake_candidates.embs,
        candidate_metadata=fake_candidates.meta,
        long_term_vec=fake_user_state.lt_vec,
        short_term_vec=fake_user_state.st_vec,
        negative_vec=fake_user_state.neg_vec,
        qdrant_scores=fake_candidates.qdrant_scores,   # non-trivial
        cluster_importance=np.full(len(fake_candidates.ids), 0.42),
        cluster_medoid=fake_user_state.dominant_medoid,
        is_suppressed_category=np.array([0, 0, 1, 0, 0], dtype=np.float32),
        onboarding_category_match=np.array([1, 0, 1, 0, 1], dtype=np.float32),
        user_total_saves=12,
        user_total_dismissals=3,
        user_saves_last_7d=2,
        user_dismissals_last_7d=1,
    )

    assert X.shape[1] == 37, f"Feature schema drifted: got {X.shape[1]} cols"

    # Per-feature non-zero rate. Slots 23..30 must each be >0 somewhere.
    nonzero_rate = (X != 0).mean(axis=0)
    for slot in range(23, 31):
        assert nonzero_rate[slot] > 0.0, (
            f"Feature {slot} is all zeros — Phase 6.1 plumbing regression"
        )

    # Aggregate sanity: at least 60% of feature slots should be active.
    assert (nonzero_rate > 0).mean() >= 0.6

C.4 What 6.1 fixes vs leaves open

Bug	Status after 6.1
A — caller uses legacy 6-arg signature	Fixed
B — Hungarian zero-vector fallback	Untouched (orthogonal; addressed separately, see Part E.4)
C — model deployment unverified	Untouched (Phase 6.3)
D — train/serve consistency	Improved on slot 23–30 non-zeroness, but the trained model's slot-23 weight is near-zero, so 6.1 will not move nDCG by much. That's expected.

---

Part D — Phase 6.2: The Full Plumbing (3–5 days)

D.1 The data-structure change: paper_cluster_map

The architectural rule we are honoring: a paper retrieved for "Cluster A: ML systems" should be scored for its fit to that cluster, not to the user's dominant interest. Phase 6.1 violates this for slots 23 and 24 by broadcasting the dominant cluster's importance/medoid to every candidate. Phase 6.2 fixes it by attaching the source cluster index to each retrieved candidate and threading that through to the reranker.

Add to app/recommend/types.py (or wherever Cluster lives):

# Source-of-truth mapping: candidate paper_id -> index into clusters[].
# Built during per-cluster Qdrant retrieval, propagated through merge_quota,
# consumed by the reranker.
PaperClusterMap = dict[str, int]

If a candidate appears in multiple cluster top-Ks (which can happen with importance-weighted quota and a permissive K_max=7), the convention is first-write-wins by importance order — i.e. when iterating clusters in descending importance, the first cluster to surface a candidate "owns" it. This matches the heuristic that a paper that the user's strongest interest cluster also pulls is more naturally explained by that cluster.

D.2 app/recommend/fusion.py — merge_quota_results

Currently merge_quota_results takes per_cluster_results: list[list[ScoredPoint]] and returns list[str] (deduped IDs). Change it to also return the cluster mapping.

def merge_quota_results(
    per_cluster_results: list[list["ScoredPoint"]],
    clusters: list[Cluster],
    floor_per_cluster: int = 3,           # F_min from CLAUDE.md
) -> tuple[list[str], PaperClusterMap]:
    """
    Importance-weighted quota merge. Returns:
        - merged_ids: deduped list of arxiv IDs (str)
        - paper_cluster_map: candidate_id -> source cluster index
    Convention: when a candidate appears in multiple clusters, the
    cluster with HIGHER importance wins. Iterate clusters sorted
    descending by importance to make first-write-wins do the right thing.
    """
    paper_cluster_map: PaperClusterMap = {}
    merged_ids: list[str] = []
    seen: set[str] = set()

    # Stable sort by importance descending; preserve original index for lookup.
    order = sorted(
        range(len(clusters)),
        key=lambda i: clusters[i].importance,
        reverse=True,
    )

    for cluster_idx in order:
        hits = per_cluster_results[cluster_idx]
        # Apply F_min floor: take at least floor_per_cluster (or all if shorter).
        # The full quota math from Phase 5 lives elsewhere; this is the
        # accounting step.
        for hit in hits:
            pid = str(hit.id)
            if pid in seen:
                continue
            seen.add(pid)
            merged_ids.append(pid)
            paper_cluster_map[pid] = cluster_idx

    return merged_ids, paper_cluster_map

Update every caller of merge_quota_results in recommendations.py to unpack the second return. Mypy / pyright will flag every site — fix them all.

D.3 app/routers/recommendations.py — propagate the map

merged_ids, paper_cluster_map = merge_quota_results(
    per_cluster_results, clusters
)

# ... after dedup and metadata fetch, valid_ids is a subset of merged_ids ...

# Per-candidate cluster index (aligned with valid_ids).
candidate_cluster_idx = np.asarray(
    [paper_cluster_map[pid] for pid in valid_ids],
    dtype=np.int32,
)

# Per-candidate cluster importance (slot 23, properly per-candidate).
per_candidate_importance = np.asarray(
    [clusters[idx].importance for idx in candidate_cluster_idx],
    dtype=np.float32,
)

# Per-candidate cluster medoid (used to compute slot 24 inside reranker).
# Stack medoids into a (N, 1024) array.
per_candidate_medoids = np.stack(
    [
        np.asarray(clusters[idx].medoid_embedding, dtype=np.float32)
        for idx in candidate_cluster_idx
    ],
    axis=0,
)

reranked_ids, reranked_scores, reranked_embs = rerank_candidates(
    candidate_ids=valid_ids,
    candidate_embeddings=valid_embs,
    candidate_metadata=valid_meta,
    long_term_vec=lt_vec,
    short_term_vec=st_vec,
    negative_vec=neg_vec,
    qdrant_scores=qdrant_scores,
    cluster_importance=per_candidate_importance,        # (N,) — was scalar in 6.1
    cluster_medoid=per_candidate_medoids,               # (N, 1024) — was (1024,) in 6.1
    is_suppressed_category=is_suppressed_category,
    onboarding_category_match=onboarding_category_match,
    user_total_saves=user_total_saves,
    user_total_dismissals=user_total_dismissals,
    user_saves_last_7d=user_saves_last_7d,
    user_dismissals_last_7d=user_dismissals_last_7d,
)

D.4 app/recommend/reranker.py — per-candidate slot 24

Inside the feature-matrix builder, slot 24 (cluster_distance_to_medoid) becomes a row-wise cosine:

# Slot 24: cosine distance from each candidate to its OWN source-cluster medoid.
# cluster_medoid shape:
#   (1024,)         -> 6.1 broadcast path (legacy)
#   (N, 1024)       -> 6.2 per-candidate path
if cluster_medoid is None:
    feat_24 = np.zeros(N, dtype=np.float32)
elif cluster_medoid.ndim == 1:
    # Broadcast cosine: same medoid for every candidate.
    medoid_norm = cluster_medoid / (np.linalg.norm(cluster_medoid) + 1e-9)
    cand_norms = candidate_embeddings / (
        np.linalg.norm(candidate_embeddings, axis=1, keepdims=True) + 1e-9
    )
    sims = cand_norms @ medoid_norm
    feat_24 = (1.0 - sims).astype(np.float32)   # distance, not similarity
else:
    # Per-row cosine: candidate i vs cluster_medoid[i].
    cand_norms = candidate_embeddings / (
        np.linalg.norm(candidate_embeddings, axis=1, keepdims=True) + 1e-9
    )
    med_norms = cluster_medoid / (
        np.linalg.norm(cluster_medoid, axis=1, keepdims=True) + 1e-9
    )
    sims = (cand_norms * med_norms).sum(axis=1)
    feat_24 = (1.0 - sims).astype(np.float32)

D.5 Why 6.2 matters

A concrete example: user has clusters {A: ML systems (importance 0.7), B: protein folding (importance 0.3)}. A paper about MLPerf benchmark methodology gets retrieved by cluster A's Qdrant query; a paper about AlphaFold 3 architecture gets retrieved by cluster B's. Under 6.1, both get scored against cluster A's medoid (the dominant one), so the AlphaFold paper looks artificially "off-distribution" and gets ranked down. Under 6.2, the AlphaFold paper is scored against cluster B's medoid (where it belongs), and slot 24 correctly registers as a small distance. The model's learned weight on slot 24 then has the opportunity to protect exploration into the user's secondary interests, instead of reinforcing the dominant one.

This also makes Phase 9 (Exploration + CF) much cleaner: the exploration budget is naturally reasoned about in terms of "minority-cluster candidates that survived rerank."

D.6 Integration test outline

File: tests/recommend/test_phase62_per_candidate_cluster.py

def test_paper_cluster_map_threaded_through(monkeypatch, fake_three_cluster_state):
    """
    Given three clusters with different medoids, candidates retrieved
    from cluster B must produce slot 24 measured against B's medoid,
    not against the dominant A's medoid.
    """
    captured = {}

    def fake_predict(model, X):
        captured["X"] = X.copy()
        return np.arange(X.shape[0])[::-1].astype(float)

    monkeypatch.setattr("app.recommend.reranker._lgb_predict", fake_predict)

    response = client.get(
        "/recommend",
        headers={"X-User-Id": fake_three_cluster_state.user_id},
    )

    X = captured["X"]
    # Specific assertion: candidate 0 came from cluster A, candidate 5 from B.
    # Their slot-24 values must be different (they are scored vs different medoids).
    assert X[0, 24] != X[5, 24]
    # And slot 23 (cluster_importance) must match the source clusters'
    # importance values, not a single broadcast.
    assert len(set(X[:, 23].tolist())) > 1

---

Part E — Phase 6.3: Deployment Verification + Monitoring (1 day)

E.1 Verify the LightGBM model loads on HF Spaces

Two acceptable deployment strategies. Pick one and document it.

Option E.1.a — Commit the 974 KB file to the Space's Git repo (recommended for simplicity)

974 KB is well under HF's 5 MB inline limit and well under any sane Git-LFS threshold. Just commit it.

# From your local ResearchIT working tree:
cd /path/to/ResearchIT
mkdir -p models/reranker-phase6/production_model
cp /path/to/reranker_v1.txt models/reranker-phase6/production_model/

# Make sure neither .gitignore nor .dockerignore excludes it.
grep -E "^models/?$|^models/reranker" .gitignore .dockerignore || echo "OK, not ignored"

git add models/reranker-phase6/production_model/reranker_v1.txt
git commit -m "Phase 6.3: ship LightGBM reranker artifact to Space"
git push hf main   # where 'hf' is the HF Space remote

reranker.py's search paths already include this location, so no code change required.

Option E.1.b — Pull from HF Hub at container build time (cleaner separation of concerns)

# Add to Dockerfile, BEFORE the COPY . . step
RUN pip install --no-cache-dir huggingface_hub && \
    python -c "\
from huggingface_hub import snapshot_download; \
snapshot_download( \
    repo_id='siddhm11/researchit-reranker-phase6', \
    local_dir='/app/models/reranker-phase6/production_model', \
    local_dir_use_symlinks=False, \
    allow_patterns=['*.txt', '*.json'], \
)"

This treats the model repo as the source of truth and the Space as a consumer — closer to "real" MLOps. Cost: one extra layer in the build, ~1s at startup since it's cached after first build.

Recommendation: Option E.1.a for now (simpler, the file is tiny, and it removes a network dependency at container build). Move to E.1.b in Phase 7 when you have a model registry story.

E.2 Add /healthz/reranker route

File: app/routers/health.py (create if not present)

import hashlib
import json
from fastapi import APIRouter
from app.recommend import reranker as _rr

router = APIRouter()

EXPECTED_FEATURE_NAMES = [
    "qdrant_score",                              # 0
    "sim_long_term", "sim_short_term",           # 1, 2
    "sim_cluster_medoid", "sim_extra_4",         # 3, 4
    "sim_negative",                              # 5
    # 6..22 paper-level
    "recency_days_log", "citation_count_log",
    "influential_citations_log", "is_primary_cs_lg",
    "is_primary_cs_cl", "is_primary_cs_cv",
    "is_primary_stat_ml", "is_primary_cs_ai",
    "is_primary_cs_ir", "is_primary_other",
    "abstract_len_log", "title_len",
    "year", "month", "venue_is_top",
    "n_authors_log", "has_code_link",
    # 23..30 cluster + user
    "cluster_importance", "cluster_distance_to_medoid",
    "is_suppressed_category", "onboarding_category_match",
    "user_total_saves", "user_total_dismissals",
    "user_saves_last_7d", "user_dismissals_last_7d",
    # 31..36 remaining metadata
    "primary_topic_freq_in_user", "is_oa", "venue_age",
    "abstract_topic_match", "ref_count_log", "is_arxiv_only",
]
assert len(EXPECTED_FEATURE_NAMES) == 37


@router.get("/healthz/reranker")
async def healthz_reranker():
    schema_hash = hashlib.sha256(
        json.dumps(EXPECTED_FEATURE_NAMES).encode()
    ).hexdigest()[:12]

    return {
        "model_loaded": _rr.is_model_loaded(),       # True iff LightGBM Booster live
        "model_path": _rr.get_loaded_model_path(),   # str or None
        "model_version": "phase6.v1",
        "fallback_active": not _rr.is_model_loaded(),
        "feature_count": 37,
        "feature_schema_hash": schema_hash,
        "n_trees": _rr.get_num_trees(),              # 141 expected
    }

Wire _rr.is_model_loaded(), _rr.get_loaded_model_path(), _rr.get_num_trees() into reranker.py as small accessors over the module-level Booster handle.

Then verify on the live Space:

curl -s https://siddhm11-researchit.hf.space/healthz/reranker | jq

Expected JSON body: "model_loaded": true, "n_trees": 141, "fallback_active": false. If any of those is wrong, the deployed image is silently running the heuristic.

E.3 Logging: feature non-zero rate per request

In reranker.py, after building the feature matrix X, add:

# Observability: emit a per-request feature-activation histogram so we
# catch silent regressions to zero-filled features 23-30.
if X.shape[0] > 0:
    nonzero_rate = (X != 0).mean(axis=0)   # (37,)
    logger.info(
        "reranker.features",
        extra={
            "feature_nonzero_rate": nonzero_rate.round(3).tolist(),
            "feature_count": X.shape[1],
            "n_candidates": X.shape[0],
            "model_active": _model is not None,
        },
    )

Add a Prometheus-style assertion later (Phase 7) but for now structured-log lines are enough — you can grep them on the Space's log stream.

E.4 Bug B fix (Hungarian zero-vector fallback) — bundle into 6.3

While we're in recommendations.py, fix the medoid-rebuild bug:

# OLD:
# medoid_embedding=np.array(vectors[row["medoid_paper_id"]], dtype=np.float32)
#   if row["medoid_paper_id"] in vectors else np.zeros(1024, dtype=np.float32)

# NEW: if the medoid paper's embedding isn't in the immediate vectors dict,
# fall back to the previously-stored medoid_embedding from the cluster row
# (which is what we persisted last cycle), NOT a zero vector. Zero-vector
# fallback breaks Hungarian assignment because cosine(*, 0) = 0 fails the
# 0.5 acceptance threshold and orphans the cluster identity.

if row["medoid_paper_id"] in vectors:
    medoid_emb = np.asarray(vectors[row["medoid_paper_id"]], dtype=np.float32)
elif row["medoid_embedding_blob"] is not None:
    medoid_emb = np.frombuffer(row["medoid_embedding_blob"], dtype=np.float32)
else:
    # Last resort: this cluster has no recoverable medoid. Mark for re-seed
    # rather than silently masking with zeros.
    logger.warning(
        "cluster.medoid.unrecoverable", extra={"cluster_id": row["cluster_id"]}
    )
    continue   # skip this cluster row; it'll be rebuilt on next Ward run

This requires adding medoid_embedding_blob BLOB to the cluster persistence schema if it's not already there — a one-line ALTER on Turso.

---

Part F — Phase 6.4: Retraining Strategy WITHOUT Real User Signal

F.1 The honest framing

The current model is not broken. It is correctly trained on the only labels we had access to (citation pseudo-labels), and on its native distribution it scores nDCG@10 = 0.879. The gap between that number and "what users feel" is a deployment gap, not a model gap.

Retraining only makes sense once one of the following is true:

• We have a training distribution where features 23–30 carry non-trivial, behaviourally meaningful signal, or
• We have real user labels (saves/dismisses/dwell-time) at sufficient scale.

Today, neither holds. So Phase 6.4 is a decision document, not a build task.

F.2 Three options, compared

Option	Feasibility	Fidelity to real users	Engineering cost	Risk
(i) Wait for real users. Threshold: 100 users with ≥10 saves each → ~1,000 labelled (user, paper, +/-) tuples per week, enough for a weekly retrain on slots 23–30.	High (zero engineering until threshold hit)	Highest possible — actual ground truth	Zero now, ~1 week to build the labelling pipeline once we cross the threshold	You may never cross the threshold. Single-developer hobby project.
(ii) Synthetic user simulation. Generate N=1,000 synthetic users. For each, draw a "true interest profile" as a mixture over arXiv categories. Sample 30–60 "saves" from each profile by drawing papers from citation neighborhoods of seed papers in those categories. Run those saves through the actual EWMA + Ward + medoid pipeline to produce real cluster_importance, real cluster_distance_to_medoid, real onboarding_category_match (using the synthetic user's category profile as the onboarding set). Label triples by citation: a citation-neighbor of any of the user's saved papers is positive; a random paper outside the user's neighborhood is negative.	Medium-high — every component already exists; the simulation is glue code	Moderate — captures structure of the signal (cluster importance scales correctly, suppressed categories actually suppress) but doesn't capture true user preference noise	5–8 days: write simulate_users.py, regenerate triples (~200K with non-zero slots 23-30), retrain LightGBM, re-eval	The model learns the simulator's biases. Fight this by holding out real user data (whenever it arrives) as a clean eval set; if the simulator-trained model dramatically outperforms a heuristic baseline on real-user holdout, you've validated. If not, you've wasted a week.
(iii) Self-distillation (defer to Phase 8). Use the heuristic scorer + Qdrant retrieval rank as a soft label. Train a fresh LightGBM to mimic those scores, then iterate: ship → log new soft labels from the deployed model itself → retrain on log data.	Medium — needs a logging schema and an offline training loop	Low initially (only as good as the teacher), but improves once real users arrive and label-corruption from the teacher decays	2–3 weeks: full label-store, replay infra, offline training pipeline	Cold-start collapse: if the teacher is worse than the student needs, distillation locks in mediocrity. Only safe once we have any real user signal to anchor the distribution.

F.3 Recommendation

Adopt a staged path:

1. Now (Phase 6.4a): Do not retrain. Ship 6.1 → 6.2 → 6.3 with the existing model. Document that the model's effective coverage is ~15/37 features at training time but improves to ~37/37 at serving time as soon as 6.1 lands — the model can only use what it learned, but the framework is now wired correctly for the next training run.
2. +30 days (Phase 6.4b): Build option (ii) — synthetic user simulation. Spec the simulator in scripts/simulate_users.py, produce a new dataset version siddhm11/researchit-reranker-data-v2 with slots 23–30 populated, retrain to reranker_v2.txt. Compare v1 vs v2 on a held-out split and on whatever sliver of real-user data has accrued by then.
3. +90 days or 100-user threshold (whichever first) (Phase 6.4c): If real-user data exists, do option (i). Train reranker_v3 on real saves/dismisses with synthetic data as augmentation (50/50 mix or downweight synthetic).
4. Phase 8+: Bring in option (iii) as a continuous-update mechanism on top of v3.

F.4 What to write in docs/phases/PHASE6.md today

Two lines, verbatim:

Phase 6.4 retraining is deferred. The published model siddhm11/researchit-reranker-phase6 was trained on citation pseudo-labels with features 23–30 zero. Retraining is gated on either (a) the synthetic-user simulator (Phase 6.4b, ~30 days out) or (b) crossing 100 real users with ≥10 saves each. Until then, Phase 6.1+6.2+6.3 plumbing is the unit of deliverable work.

---

Part G — Phase 6 Closeout Checklist

### Phase 6.1 — Simplification Pass (1–2 days)
- [ ] Move `suppressed = await db.get_suppressed_categories(...)` to BEFORE rerank call in app/routers/recommendations.py
- [ ] Add `onboarding_categories = await db.get_user_category_filter(...)` next to it
- [ ] Compute `user_total_saves`, `user_total_dismissals`, `user_saves_last_7d`, `user_dismissals_last_7d`
- [ ] Add helper `build_qdrant_score_map()` in app/recommend/fusion.py
- [ ] Add helper `align_qdrant_scores(valid_ids, score_map)`
- [ ] Compute `dominant_importance` and `dominant_medoid_vec`
- [ ] Replace 6-arg `rerank_candidates(...)` call with full kwargs version (Section C.2.4)
- [ ] Update `rerank_candidates()` signature in app/recommend/reranker.py to accept new optional kwargs (Section C.2.5)
- [ ] Wire slots 23–30 into `_build_feature_matrix` in reranker.py
- [ ] Write test `tests/recommend/test_phase6_feature_matrix.py` asserting slots 23–30 non-zero
- [ ] All existing tests pass (`pytest -q`)
- [ ] Commit: "Phase 6.1: connect 37-feature reranker to live caller (dominant-cluster)"

### Phase 6.2 — Full per-candidate plumbing (3–5 days)
- [ ] Add `PaperClusterMap = dict[str, int]` type alias in app/recommend/types.py
- [ ] Modify `merge_quota_results()` in app/recommend/fusion.py to return `(merged_ids, paper_cluster_map)`
- [ ] Update ALL callers of `merge_quota_results` (grep first; fix every one)
- [ ] In recommendations.py, build `candidate_cluster_idx` aligned with `valid_ids`
- [ ] Build `per_candidate_importance` (N,) and `per_candidate_medoids` (N, 1024)
- [ ] Pass arrays (not scalars) for `cluster_importance` and `cluster_medoid` to reranker
- [ ] Update `_build_feature_matrix` slot-24 logic to handle both (1024,) and (N, 1024) medoid shapes
- [ ] Write test `tests/recommend/test_phase62_per_candidate_cluster.py` (Section D.6)
- [ ] Commit: "Phase 6.2: per-candidate cluster identity through reranker"

### Phase 6.3 — Deployment verification + Bug B
- [x] Decide deployment strategy: E.1.a (commit) vs E.1.b (snapshot_download). Used E.1.a.
- [x] Verify `models/reranker-phase6/production_model/reranker_v1.txt` is in working tree, not gitignored, not dockerignored
- [x] Push to HF Space; wait for build; check build logs for "[reranker] LightGBM model loaded"
- [x] Add `/healthz/reranker` route (Section E.2)
- [x] Add `_rr.is_model_loaded()`, `_rr.get_loaded_model_path()`, `_rr.get_num_trees()` accessors
- [x] `curl https://siddhm11-researchit.hf.space/healthz/reranker` → confirm `model_loaded: true, n_trees: 141`
  > *Verified live at 2026-05-03: `model_loaded=true, n_trees=141, fallback_active=false, feature_count=37, feature_schema_hash=5d0b3de7b0c1`.*
- [x] Add per-request `reranker.features` log line with `feature_nonzero_rate`
- [x] Fix Bug B: medoid_embedding_blob fallback in cluster reload (Section E.4)
- [x] Add `medoid_embedding_blob BLOB` column to clusters table (SQLite ALTER migration)
- [x] Update CLAUDE.md / model card to reflect deployment story

### Phase 6 documentation
- [ ] Write `docs/phases/PHASE6.md` retraining decision (Section F.4)
- [ ] Update `README.md` test count (will increase by ~2 from 6.1 + 6.2 tests)
- [ ] Update `TASK-TRACKER.md`: tick off `[x] [reranker] LightGBM model loaded` (after curl verifies it)
- [ ] Backfill the HF model card with: `library_name: lightgbm`, `pipeline_tag: text-ranking`, `datasets: [siddhm11/researchit-reranker-data]`, training data description, feature schema (37 names), reported metrics (nDCG@10=0.879 on 7,007-triple eval split, ≥2023 papers), and a clear "trained on citation pseudo-labels, NOT real user signal" disclaimer
- [ ] Upload `03_train_lightgbm.py` to a new repo `siddhm11/researchit-reranker-training` (or to the dataset repo as a script) so retraining is reproducible from public artifacts

### Phase 6.4 — Retraining (decision only; no code yet)
- [ ] Document deferral and the 30-day / 100-user trigger in PHASE6.md
- [ ] Open a tracking issue: "Phase 6.4b: synthetic user simulator" (target: +30d)
- [ ] Open a tracking issue: "Phase 6.4c: real-user retrain at 100-user threshold"

---

Part H — What Phase 6 Is NOT (scope boundary)

Phase 6 is integration of the existing trained reranker, plus the deployment story for it. Specifically out of scope for Phase 6 — these are their own phases, with their own framing docs to come:

Out of scope	Belongs to	Why it's separate
Offline regression harness, time-split eval framework, A/B test infrastructure	Phase 7 — Evaluation Framework	Requires a held-out user-interaction log and a separate "shadow ranker" infra. Cannot be built before 6.3 ships, because we need stable feature semantics first.
LLM-generated paper summaries; cross-encoder distilled into LightGBM features	Phase 8 — LLM Summaries + Distilled Reranker	Adds new feature slots (37 → 40+) and breaks the schema; must be a model version bump, not a 6.x patch.
Exploration bandits (UCB / Thompson over cluster heads), collaborative-filtering co-save signal	Phase 9 — Exploration + CF	Needs a user population large enough for CF to be non-degenerate, and an exploration budget that Phase 6's MMR=0.6 currently approximates as a stopgap.
Migrating to a different reranker family (e.g. cross-encoder, ColBERT, BGE-reranker-v2-m3)	Phase 10+	Explicitly forbidden in serving by CLAUDE.md rule 4. LightGBM is the serving model; anything heavier is a teacher in distillation, not a serving model.
Replacing citation pseudo-labels with click-through CTR labels from production	Phase 6.4c (real-user retrain)	Triggered by traffic threshold, not by code.

The tightest possible definition of "Phase 6 done":

Every candidate that arrives at the LightGBM ranker is described by a 37-dimensional feature vector in which slots 23–30 carry per-candidate signal derived from the user's actual cluster state and onboarding/category history; the ranker's inference is verified live on siddhm11-researchit.hf.space via /healthz/reranker; and a feature non-zero rate is logged per request. Retraining is deferred and documented.

When that sentence is true, Phase 6 ships and Phase 7 begins.

---

Caveats and unknowns (full disclosure)

1. The HF model card itself was not directly readable from the verification environment used for this framing — only the structured Hub API metadata (which exposes tags, created_at, last_modified, pipeline_tag, library_name, description, etc., all of which were null/empty for the model repo). The conclusion that "the README is empty" is an inference from the absent metadata fields a populated model card would normally surface, not a direct read of the file. Action item for Amin: open https://huggingface.co/siddhm11/researchit-reranker-phase6 in a browser, look at the README and the Files tab, and confirm: (a) is reranker_v1.txt present at exactly 974 KB? (b) is there any README content at all? (c) are training scripts present? Adjust the Phase 6.3 doc-backfill checklist items accordingly.
2. The 90,993 / 7,007 triple counts and the 141-tree figure come from the audit, not from a re-derivation against the live HF artifacts. The HF dataset metadata only confirms the order of magnitude (100K<n<1M rows, Parquet). If the real numbers differ, the framing logic does not change — only the diagnostic prose in Part B.
3. The Space's actual loaded model state is unknown without running the curl /healthz/reranker step in Section E.2. The TASK-TRACKER's unchecked box is the only authoritative signal we have today, and it says "not verified."
4. The synthetic-user simulator (Phase 6.4b) is plausible but unproven. Its quality depends entirely on whether citation neighborhoods are a good proxy for "papers a user with interest profile X would save." That is an empirical question; the simulator is worth building only because the alternative is "do nothing until users arrive."