docs/phases/PHASE8-Search-And-Recommendation-Design.md

PHASE 8 — Search and Recommendation: the exact design

Status: Partly shipped · Updated: 2026-07-30

The definitive specification of both pipelines: every stage, every constant, where it lives, and whether it is live or pending. Latency figures are measured against the deployed Space, not estimated.

Legend: [LIVE] merged to main · [PENDING] designed, not built

---

1. Search

GET /search?q=… → app/routers/search.py → app/hybrid_search_svc.py:search()

query
  │
  ├─[1]─► Groq rewrite            concurrent, non-blocking        [LIVE]
  ├─[2]─► BGE-M3 encode           in executor, max_length=512     [LIVE]
  │
  ├─[3a]► Qdrant dense    limit=60, rescore=True, oversampling=1  [LIVE]
  └─[3b]► lexical         Zilliz sparse → FTS5                    [PENDING]
            │
          [4] RRF fusion, k=60                                    [LIVE]
            │
          [5] cross-encoder, top 50, full abstracts               [LIVE / PENDING]
            │
          [6] title-match + citation boost, top 50                [LIVE]
            │
          [7] return top 10

Stage detail

#	Stage	Constants	Measured
1	Groq rewrite	groq_svc.rewrite, skipped if ≤2 words or _looks_academic	161–329 ms, overlapped
2	BGE-M3 encode	max_length=512, LRU 128, run_in_executor	285–642 ms
3a	Qdrant dense	limit = 10 × SEARCH_FETCH_K_MULTIPLIER(6) = 60	~190 ms
3b	Zilliz sparse	same limit	~network
4	RRF	SEARCH_RRF_K = 60, 1/(k+rank) summed over all lists	~0 ms
5	Cross-encoder	SEARCH_RERANK_TOP_N = 50, sigmoid → [0,1]	139–182 ms at n=10
6	Boosts	exact 2.0 · substring 1.0 · coverage ≥0.8→1.0 / ≥0.5→0.5 · citation cap 0.2	~1 ms
7	Return	ARXIV_MAX_RESULTS = 10

Rules that must not change

• RRF is correct for search. Many retrievers, one query — rank-based fusion needs no score calibration. Do not replace it with quota (that is the recommendation-side answer to a different problem).
• Rerank window must exceed the result count. At SEARCH_RERANK_TOP_N = 10 the stage re-ordered exactly the set already being returned and could never promote a better paper from the retrieved pool. 50 of 60 is the current setting; below 10 it is worse than useless because it still costs CPU.
• Rescore stays on. rescore=False is 615 ms vs 608 ms — no faster — and drops recall@10 from 100% to 57%. Binary codes alone cannot rank 1024-dim vectors.
• Title boost uses the ORIGINAL query, never the rewrite. The user's literal text is what should match a title.
• Stage timings no longer sum. groq_time_ms overlaps encode_time_ms; search_meta.groq_overlapped flags this.

Pending

1. Lexical from FTS5, not Zilliz — same RRF input shape, no network, one fewer vendor, ~2 GB less storage. A/B against Zilliz before switching: BGE-M3 sparse weights are learned, BM25 is not.
2. Full abstracts to the cross-encoder — 90.0% of stored abstracts are truncated at 500 chars while the retriever saw 1024. The precision stage currently has less information than the stage it refines. Biggest remaining search-quality item.

---

2. Recommendations

GET /api/recommendations → app/routers/recommendations.py

Cascading tiers, first non-empty wins.

Tier 1  ≥5 saves   clustering + quota fusion      ← the actual product
Tier 2  ≥3 saves   EWMA long-term vector
Tier 3  ≥1 save    Qdrant Recommend (BEST_SCORE)
Tier 0  onboarded  trending by category

Tier 1 — the real pipeline

saved papers
  └─[1] fetch vectors            qdrant_svc.get_paper_vectors    1247 ms / 20  ← hot spot
  └─[2] Ward clustering          MIN_PAPERS_FOR_CLUSTERING = 5
  └─[3] Hungarian stabilise      against persisted medoids
  └─[4] quota allocation         total_slots=100, min_slots=3, by importance
  └─[5] per-cluster ANN          limit = quota × _OVERSAMPLE(3), parallel
  └─[6] short-term supplement    _ST_SUPPLEMENT = 20
  └─[7] scoring                  heuristic (default) | LightGBM      [LIVE]
  └─[8] category suppression     ≥3 dismissals / 14 days, arXiv codes [LIVE]
  └─[9] MMR diversity            lambda_param = 0.6, top_k = 10
  └─[10] exploration injection   n_explore = 2  → returns limit + 2

Scoring — why the heuristic is the default

RERANKER_MODE = "heuristic" (app/config.py).

Parsing reranker_v1.txt: features 20–30 have zero splits across all 141 trees — every EWMA similarity, both cluster features, the suppression and onboarding flags, all four interaction counts. A tree only reads features it splits on, so a user's entire profile provably cannot change the output. Not a statistical claim; structural.

candidate_num_cited_by additionally holds 65.2% of importance and is hardcoded to 0 at serving time.

heuristic_score() reads features 20–22:

has_ewma:  relevance = 0.40·lt_sim + 0.25·st_sim
otherwise: relevance = 0.65·qdrant_cosine

Demonstrated: opposite ewma_longterm_similarity produces rankings [0,1,2,3,4,5] vs [5,4,3,2,1,0] — a full reversal from the profile alone.

Set RERANKER_MODE=lightgbm to compare once real engagement data exists. /healthz/reranker reports model_loaded and scoring_with separately.

Rules that must not change

• Quota, not RRF, for recommendations. Many queries (one per interest cluster), one user. RRF would let the dominant cluster win on rank alone and reintroduce interest collapse — the failure the whole product exists to prevent.
• Hungarian matching stays. Without it a user's "NLP cluster" becomes cluster_7 after the next recluster and instrumentation loses continuity.
• Suppression on arXiv codes, not primary_topic. The latter has ~15 coarse buckets; AI/ML alone is 20.2% of the corpus, so three dismissals used to suppress a fifth of everything.
• MMR λ=0.6 — below ~0.5 relevance degrades visibly; above ~0.8 the feed collapses toward one interest.

Pending

1. Candidate vectors local. get_paper_vectors is 1,247 ms for 20; Tier 1 needs 100+. MMR and scoring both need real vectors, not ids. At int8, 1.9M vectors is 1.95 GB — same sidecar pattern as metadata. Biggest remaining recommendation latency item.
2. Persistence to Turso. Profiles, clusters and interactions live in SQLite at /tmp and are destroyed on every rebuild. Gates everything below.
3. Reranker retrain on real interactions, restricted to features that are non-zero at serving time. Requires 2.

---

3. Cold start

Tier	Trigger	Source
0	onboarded, 0 saves	fetch_trending_by_categories
3	1 save	Qdrant Recommend
2	3 saves	EWMA long-term
1	5 saves	full pipeline

Trending ranks by citations within a recency window measured back from the newest paper in the corpus, widening 24 → 48 → 96 months for thin categories, with publication date decoded from the arXiv id (update_date is the revision date, so 2017 classics looked new). TRENDING_RECENCY_MONTHS = 24.

Known gap: the onboarding wizard targets 5 seed saves and Tier 1 needs 5, but nothing tells the user that 5 is a threshold. Save 4 and you silently get Tier 3, the weakest path.

---

4. Build order

#	Task	Depends on	Risk
1	Persistence → Turso	—	none, additive
2	FTS5 in sidecar + A/B vs Zilliz	—	none until switched
3	Full abstracts (ingest + backfill)	Phase 7	needs arXiv job
4	Local candidate vectors	index build	image size
5	Retire Zilliz	2	after A/B
6	Reranker retrain	1 + months of data	—

Shipped already: event-loop fix, Groq overlap, RERANKER_MODE, rerank window, quantization search params, arXiv-code suppression, recency trending, metadata sidecar.