APPROACH.md

Approach — Conversational SHL Assessment Recommender

Problem

Recruiters start vague ("I'm hiring a Java developer"). The service is a stateless FastAPI agent that, over a multi-turn dialogue, clarifies intent and returns a grounded shortlist of 1–10 SHL assessments (name, catalog url, test_type). It is graded by an automated replay harness that plays the user with an LLM, under hard limits: ≤8 turns, 30s/call, exact response schema, and no URL that isn't in the catalog. Design is shaped by those limits.

Architecture — one LLM call per turn

Every request carries the full history (stateless). Per turn we make exactly one LLM call — a router that classifies intent and extracts normalized constraints as strict JSON; Python does everything deterministic (retrieval, URL lookup, templating, schema validation). Rejected multi-call agent loops: they blow the 30s + 8-turn budget and add nondeterminism.

history ─▶ router (1 LLM call, JSON) ─▶ intent + constraints + search_query
                                          │
   CLARIFY   RECOMMEND/REFINE        COMPARE        REFUSE
   ask       retrieve→assemble       grounded       canned
   recs=[]   →templated reply        prose, recs=[] refusal
                     │
        catalog post-filter (id→canonical name/url/type) ─▶ Pydantic ─▶ response

Key decisions:

• Un-inventable URLs. The LLM returns names/constraints, never URLs. Python looks up the canonical URL by catalog id and drops any id not in the catalog. This makes hallucinated URLs structurally impossible, not just unlikely.
• Stateless refine is free. History is replayed each call, so "refine" is just re-deriving all constraints and re-retrieving — we never hold state.
• Templated shortlist replies. The schema-critical reply is built from a template, so the contract never depends on LLM formatting. CLARIFY/COMPARE prose (which needs fluency) is the only LLM-authored text.
• Robustness: a ~25s request timeout, background model/index warmup so /health is instant, forced commit near the turn cap, defensive parsing of empty/garbage payloads, and a SAFE_FALLBACK that keeps every response valid.

Retrieval

Catalog = 370 Individual Test Solutions parsed to data/catalog.json (Job Solutions excluded). Hybrid retrieval fuses two signals with Reciprocal Rank Fusion:

• Dense — all-MiniLM-L6-v2, embeddings precomputed and shipped (embeddings.npy), so nothing large downloads cold. Catches semantic intent ("stakeholders" → interpersonal).
• BM25 (rank_bm25) over the same document text, with a subword tokenizer that splits letter/digit boundaries so exact ids resolve ("OPQ" → opq32r, ".NET" → net). Catches identifiers dense retrieval misses.

Metric is Recall@10, so we retrieve inclusively. The labeled shortlists are batteries, so after retrieval we assemble: role/skill spine + two recurring defaults the user rarely names — a personality measure (OPQ32r, P; in 7/10 finals) and a cognitive measure (Verify G+, A) — added for hiring queries unless the user opts out (proximity-matched, so "keep Verify, drop OPQ" is honored).

Prompt design — router

System prompt encodes the conversation policy from the 10 traces: turn-1 vague → CLARIFY (never recommend); commit once role + one discriminating attribute is present; "no preference" → commit; ≤2 clarifiers; REFUSE off-topic / legal / general-hiring / injection. Output is a fixed JSON shape (intent, constraints, search_query, named_assessments, reply_text); malformed output → safe CLARIFY. search_query is told to list every distinct skill so multi-skill JDs don't drop a technology.

Evaluation — real numbers

Two harnesses. eval/recall_eval.py is a deterministic single-shot proxy that maps each trace's labeled shortlist to catalog ids and measures retrieval Recall@10 — fast and reproducible, used for tuning. eval/replay.py runs each trace as a real multi-turn conversation through the agent (one router call per turn; scripted or LLM-simulated user), measuring conversational Recall@10 + groundedness and flagging any LLM-failure turns so the mean stays honest. eval/probes.py runs the behavior probes; eval/report.py aggregates all three into eval/REPORT.md. Per-change tuning results (eval/TUNING_LOG.md):

Change	Mean Recall@10
Baseline hybrid dense+BM25+RRF	0.5012
+ RRF constant / N sweep	~0.50 (flat)
+ enriched embedded text	0.6931 (flat)
+ test_type-aware battery assembly	0.6931 (+0.19)
+ router-synthesized query	0.7131
+ "every distinct skill" query prompt	~0.72

Behavior probes (binary, all passing via the test suite): refuses off-topic/ injection, no turn-1 recommend on a vague opener, honors edits/refine, 100% grounded. 118 unit/integration tests cover schema, retrieval, assembly, agent paths, and endpoint robustness (timeout/fallback).

What didn't work

• RRF/N tuning and text enrichment were flat — the misses are missing documents, not mis-ranked ones, and the catalog has no competencies field to add.
• Per-skill / per-message multi-query fusion made recall worse (0.486): sub-queries pulled unrelated items and diluted the exact-name signal.
• Remaining misses are near-duplicate product families (short variant loses to -365 sibling) and skill-flooded JDs — ranking/diversity problems, not fusion tuning. Returns flattened, so tuning stopped at ~0.72.

Limitations (named honestly)

• Overfit risk: the +0.19 default-injection was tuned on the 10 public traces; OPQ32r/Verify G+ may be less dominant on the holdout set, so the number may not fully transfer. It's the best signal available, but it's a fit to 10 conversations.
• The conversational replay caught a real bug the single-shot proxy hid — a refine/confirm turn retrieving only from the sparse last message — now fixed by combining the router query with the full history (C9: 0.29 → 0.86 conversational recall). This is why the multi-turn harness matters, not just the proxy.

AI tools used

Built with Claude Code (Anthropic) for implementation, retrieval tuning, and evaluation. Runtime LLM: Groq llama-3.3-70b-versatile (chosen for latency under the 30s cap), behind an app/llm interface with a Gemini swap available by changing one env var.