docs/walkthroughs/02-Phase2-MultiInterest-Recommender.md

# Phase 2 — Multi-Interest Recommender Walkthrough

## What Was Built

A PinnerSage-style multi-interest recommendation engine that replaces Phase 1's raw-ID Qdrant queries with computed EWMA user profile embeddings, Ward hierarchical clustering for interest detection, heuristic re-ranking, and MMR diversity enforcement.

**The old pipeline (Phase 1):**
```
User saves papers → raw IDs → Qdrant BEST_SCORE → results
```

**The new pipeline (Phase 2):**
```
User saves papers
    ↓
EWMA profiles update (background, non-blocking)
    ↓
Ward clustering → K distinct interest medoids (auto K per user)
    ↓
Qdrant prefetch + RRF fusion (~15-25ms, single API call)  [⚠️ Replaced by Quota Fusion in Phase 4]
    ↓
Heuristic re-ranking of ~100 candidates (~1-2ms)
    ↓
MMR diversity selection → top 10-12 papers (<1ms)
    ↓
Exploration injection → 1-2 serendipitous papers
    ↓
Render HTML via HTMX
```

**Total pipeline latency: <30ms** (excluding metadata fetch if cold)

---

## Why This Architecture

This architecture was chosen after deep research documented in [03-MultiInterest-Recommender-Architecture.md](../research/03-MultiInterest-Recommender-Architecture.md). The key insights:

### The Interest Collapse Problem
A single average embedding for a user interested in both *NLP* and *computer vision* lands in meaningless embedding space — Pinterest called this the "energy-boosting breakfast" problem. PinnerSage (KDD 2020) solved it with multiple user vectors.

### Why EWMA Over Rolling Windows
Rolling windows (last 30 days) lose valuable historical signal abruptly. EWMA (Exponentially Weighted Moving Average) provides smooth decay:
- **Long-term (α=0.10):** Effective window ~20 interactions. Tracks enduring research interests.
- **Short-term (α=0.40):** Effective window ~3-5 interactions. Captures current session context.
- **Negative (α=0.15):** Tracks papers the user explicitly dislikes.

### Why Ward Over K-Means
K-Means requires pre-specifying K (number of clusters). Ward hierarchical clustering auto-determines K per user via a distance threshold — a user with 2 interests gets 2 clusters, a user with 5 gets 5. No hyperparameter tuning per user.

### Why LightGBM Over BGE-reranker
The older `Research-Recommender_Technical_Roadmap.md` suggested BGE-reranker-v2 at ~800ms for 100 candidates on CPU. LightGBM scores 500 candidates in 2-5ms. On Render Free Tier (CPU-only, 512MB RAM), this is the only viable option. Currently using a heuristic scorer with the same feature interface — drop-in LightGBM upgrade when training data accumulates.

---

## 3-Tier Cascading Fallback

The recommender degrades gracefully based on how much data the user has:

| User State | Tier | Strategy | Latency |
|---|---|---|---|
| ≥5 saves | **Tier 1** | Clustering → RRF → Rerank → MMR → Explore | ~25ms |
| 3-4 saves | **Tier 2** | EWMA long-term vector → ANN search | ~10ms |
| 1-2 saves | **Tier 3** | Qdrant BEST_SCORE (Phase 1 path) | ~15ms |
| 0 saves | Empty | "Save at least 1 paper..." | 0ms |

Each tier falls through to the next if it can't produce results.

---

## New Files Created

### `app/recommend/__init__.py`
Package init for the recommendation engine module.

### `app/recommend/profiles.py`
EWMA temporal embedding profiles:
- `ewma_update(current, new_embedding, alpha)` — core blending function
- `update_on_save(user_id, paper_embedding)` — updates both LT and ST profiles
- `update_on_dismiss(user_id, paper_embedding)` — updates negative profile
- `load_profile()` / `save_profile()` — SQLite persistence as binary numpy blobs (4KB each)

### `app/recommend/clustering.py`
Ward hierarchical clustering:
- `compute_clusters(paper_ids, embeddings)` → list of `InterestCluster`
- Each cluster: medoid paper ID, medoid embedding, member paper IDs, importance score
- Auto K (1-7 clusters), recency-weighted importance
- Falls back to single cluster if <5 saved papers

### `app/recommend/reranker.py`
Heuristic scorer (LightGBM-ready):
- `compute_features()` → 4 features per candidate: cosine_sim_LT, cosine_sim_ST, paper_age, rrf_position
- `heuristic_score()` → weighted sum: 45% relevance, 25% session, 20% recency, 10% rank
- `rerank_candidates()` → end-to-end: features → scores → sorted output

### `app/recommend/diversity.py`
MMR diversity + exploration:
- `mmr_rerank(query, candidates, scores, λ=0.6, top_k=20)` — greedy diverse selection
- `inject_exploration(selected, pool, n_explore=2)` — random serendipity injection

---

## Modified Files

### `app/db.py`
- Added `user_profiles` table — EWMA vectors as BLOBs with interaction counts
- Added `user_clusters` table — Ward clustering results (medoid IDs, importance, paper lists)
- Added 4 helper functions: `get_user_profile`, `upsert_user_profile`, `save_user_clusters`, `get_user_clusters`

### `app/qdrant_svc.py`
- Added `get_paper_vectors()` — fetch actual BGE-M3 embeddings from Qdrant (needed for EWMA)
- Added `search_by_vector()` — raw ANN search by embedding vector
- Added `multi_interest_search()` — prefetch + RRF fusion in a single API call
- Imported new Qdrant models: `Prefetch`, `FusionQuery`, `Fusion`

### `app/routers/events.py`
- Save handler now triggers background EWMA profile update (LT + ST) via `asyncio.create_task`
- Dismiss handler triggers background negative profile update
- Both are non-blocking — user response is sent before the update completes

### `app/routers/recommendations.py`
- Complete rewrite with 3-tier cascading fallback
- Tier 1: full 5-step pipeline (cluster → retrieve → rerank → MMR → explore)
- Tier 2: EWMA long-term single-vector search
- Tier 3: original BEST_SCORE (unchanged from Phase 1)

### `requirements.txt`
- Added `numpy>=1.24` — vector computations
- Added `scipy>=1.11` — Ward hierarchical clustering

---

## What Was NOT Changed

These files are intentionally untouched:
- `app/user_state.py` — still manages ID deques for the hot cache
- `app/routers/search.py` — search is a separate concern (see PHASE2-Hybrid-Search-Plan)
- `app/routers/saved.py` — saved papers page is unaffected
- All templates — no UI changes needed, same HTMX partials

---

## Test Coverage

| Test File | Tests | Description |
|---|---|---|
| `test_profiles.py` | 11 | EWMA math, convergence, normalisation, DB round-trips |
| `test_clustering.py` | 10 | Ward clustering, medoid validity, max clusters, DB persistence |
| `test_reranker_diversity.py` | 13 | Heuristic scoring, MMR diversity, exploration injection |
| Existing tests | 52 | Integration, events, saved page, qdrant_svc |
| **Total** | **86 passed** | 2 pre-existing live Qdrant failures (network-dependent) |

---

## Upgrade Path: Heuristic → LightGBM

The heuristic scorer in `reranker.py` is designed for a zero-data-required drop-in to LightGBM:

1. **When:** Interactions table has ≥500 save/dismiss rows
2. **How:** Train offline with `lgb.train(params={'objective': 'lambdarank'}, ...)`
3. **Where:** Save model to `models/reranker.lgb`, replace `heuristic_score()` with `model.predict(features)`
4. **Impact:** Same features, same interface — zero code changes in the router

---

## Key Design Decisions & Rationale

| Decision | Chosen | Rejected | Why |
|---|---|---|---|
| User profile | EWMA (3 vectors) | Rolling window | Smooth decay, no abrupt signal loss |
| Clustering | Ward hierarchical | Fixed K-Means | Auto-determines K per user |
| Re-ranking | Heuristic → LightGBM | BGE-reranker-v2 | 800ms → 2ms on CPU |
| Diversity | MMR (λ=0.6) | Random sampling | Principled relevance/diversity trade-off |
| Exploration | Random injection (2 papers) | None | Prevents filter bubbles |
| Multi-query | Qdrant prefetch+RRF | Sequential queries | Single network round-trip |