eval/runbook.md

Eval Runbook

How to read, run, and triage the eval harness when a metric drops.

---

What runs where

Where	What	When
Local dev	python -m eval.smoke_eval (10 questions)	Iterating on retrieval changes
Local dev	python -m eval.run_full --out eval/baselines/<name>.json	Before opening a PR
Local dev	python -m eval.run_full --with-llm --judge --out ...	Pre-release; uses LLM quota
GitHub Actions	.github/workflows/eval.yml	Every PR touching app/, ingest/, eval/

CI runs retrieval metrics only — no LLM calls, no rate limits, fully deterministic. Generation faithfulness is graded offline before each release.

---

CI gate

The job fails if either:

• recall_at_5 < 0.85 (answerable questions where any top-5 chunk is relevant)
• ooc_refusal_rate < 0.90 (out-of-corpus questions where the gate fired)

Set in eval/run_full.py via the EVAL_CI_GATE=1 env var, so the same script runs locally without gating.

---

Triage: a metric just dropped. What now?

1. Pull the artifact from the failed PR's "eval-results" upload, or rerun locally: python -m eval.run_full --out eval/baselines/diag.json.

2. Diff against the last good baseline (eval/baselines/v0.4.0-rerank-full.json). Look at results[*].verdict_pass flips and results[*].retrieved_top5_urls shifts.

3. Categorise the regression by category:

   Category that dropped	First suspects
   vocab-mismatch	Reranker config, query expansion synonyms
   entity-lookup	Embedder change, index rebuild gap, chunk boundaries
   fact-lookup	Chunking strategy, embed model, top-k
   multi-hop	Top-k too small, chunk granularity, relevance threshold
   ooc-negative	relevance_threshold lowered too far

4. Reproduce in isolation: pick one failing question and run python -m app.ask --no-llm "<question>" to see what dense retrieval gave. If gate=PASS but the right chunk isn't in top-K, the issue is rank order (reranker / query expansion). If gate=FAIL, retrieval missed entirely.

5. Knobs in order of safety to twist (safest first):

   • Add a synonym to _EXPANSIONS in app/providers/reranker.py
   • Bump RERANK_TOP_N (currently 20)
   • Re-tune RELEVANCE_THRESHOLD against eval/questions_v2.jsonl
   • Re-chunk and re-index (last resort; rebuilds 12k vectors)

6. Never silently lower the CI thresholds to make the gate pass. If a threshold is wrong, change it in a dedicated PR with an explanation, not in the same PR that introduced the regression.

---

Updating the eval set

eval/questions_v2.jsonl is the source of truth for retrieval metrics.

When you add a question:

• Pick a category from the five existing ones; don't invent new ones unless you also update this runbook.
• relevant_text_any should be the smallest substrings that uniquely identify the right answer in the corpus (e.g. "Désirée Finnegan", not "is the").
• answer_must_contain_any is checked only in --with-llm mode; it's looser than the retrieval needle (lets the LLM paraphrase).

Re-run the baseline and commit the new eval/baselines/*.json so PR reviewers can see whether your question changed the aggregate metrics.

---

Faithfulness scoring

Faithfulness is judged by an LLM (NVIDIA Llama 70B by default) against the retrieved chunks the answerer was actually shown. It's NOT part of the CI gate because:

1. It's non-deterministic (judge LLM sampling).
2. It costs quota — running it on every PR would exhaust free-tier limits.
3. A faithfulness miss is a generator bug, not a retrieval bug — different triage path.

Run it before tagging a release:

python -m eval.run_full --with-llm --judge \
  --out eval/baselines/v$(date +%Y%m%d)-prerelease.json

If faithfulness_avg < 0.85, investigate the system prompt or grounding discipline before shipping.