eval/TUNING_LOG.md

# Retrieval Tuning Log — Mean Recall@10

Metric: **Mean Recall@10** = mean over the 10 public traces of
`|expected ∩ top-10| / |expected|`, where the labeled shortlist is mapped to
catalog ids. Measured by `python eval/recall_eval.py [--assemble] [--router]`.

Two query modes are reported:
- **user-text** (deterministic): query = concatenation of the trace's user
  messages. Reproducible; isolates retrieval changes from LLM noise.
- **router** (live): query = the router's synthesized `search_query` (one LLM
  call, temperature 0; ~±0.01 run-to-run).

| Step | Change | Query | Mean Recall@10 | Δ |
|------|--------|-------|---------------:|----:|
| 0 | **Baseline** hybrid dense+BM25+RRF (n=50, rrf_k=60) | user-text | **0.5012** | — |
| a | Sweep RRF constant × candidate N | user-text | 0.48–0.52 | ~0 |
| d | **test_type-aware assembly** (inject OPQ32r `P` + Verify G+ `A` defaults) | user-text | **0.6931** | **+0.1919** |
| b | Enrich embedded text (name emphasis; job_levels already present) | user-text | 0.6931 | 0 |
| c | Router `search_query` synthesis | router | 0.7131 | +0.0200 |
| c′ | Router prompt: "include every distinct skill" | router | ~0.72 (0.713–0.727) | +0.01 |

**Baseline 0.5012 → shipped ~0.72** (router query + assembly). Groundedness is
1.0 by construction (URLs looked up from `catalog.json`; non-catalog ids dropped).

---

## (a) RRF constant + dense/BM25 N — *flat, kept defaults*
Swept `rrf_k ∈ {10,20,40,60,100}` × `n ∈ {20,30,50,80,120,200,370}`. The whole
grid sits in 0.48–0.52; the single best cell (n=30, rrf_k=20 → 0.5155) is not
robust (neighbouring cells drop to 0.4955), i.e. noise. **Kept n=50, rrf_k=60.**
Fusion params are not the bottleneck — the misses are missing *documents*, not
mis-ranked ones.

## (b) Enrich embedded text — *no runway, flat*
CLAUDE.md suggested adding `job_levels` + `competencies`. `job_levels` is already
in `assessment_text`; the catalog has **no** competencies/keywords field beyond
the test-type `keys` (already included). Tested name-emphasis (name embedded
twice) to lift crowded short-named tests (`sql-new`, `ms-excel-new`): **0.6931 →
0.6931, no change.** Reverted.

## (c) Router search_query synthesis — *small, real gain*
Using the router's synthesized query instead of raw user text: **0.6931 →
0.7131**. Adding "include EVERY distinct skill/technology, each as its own term"
to the router prompt (targets the C9 Java-flood that drops `sql-new`/`docker-new`)
nudged to ~0.72. Small but real; kept.

## (d) test_type-aware assembly — *the lever, +0.19*
The labeled shortlists are **batteries**: a role-skills spine + two recurring
defaults the user rarely names — a personality measure (**OPQ32r**, `P`, in 7/10
finals) and a cognitive measure (**SHL Verify Interactive G+**, `A`). After
retrieval we reserve slots and guarantee these defaults for hiring-style requests,
unless the user opts out (`app/assembly.py`). This recovered the single largest
miss class: **0.5012 → 0.6931**. Opt-out uses proximity matching so "keep Verify
G+" alongside "drop the OPQ" only drops personality (see `test_assembly.py`).

---

## What didn't work
- **Per-message / per-skill multi-query fusion** (RRF across sub-queries):
  **0.486**, *worse* than a single query — sub-queries pulled unrelated items and
  diluted the exact-name signal. Abandoned.
- **RRF/N tuning** and **text enrichment**: both flat (above).

## Remaining misses (not recoverable by retrieval alone)
Near-duplicate product families where the short canonical variant loses to
`-365`/`-essentials` siblings (C8 `ms-excel-new`), multi-skill JDs where one
skill family floods the pool (C9 `sql-new`, `docker-new`), and second personality
*reports* (C1 `opq-universal-competency-report`). These are ranking/diversity
problems; the next lever would be per-family de-duplication, not more fusion
tuning. **Returns have flattened — stopped here.**