eval/runbook.md

# Eval Runbook

How to read, run, and triage the eval harness when a metric drops.

---

## What runs where

| Where | What | When |
| --- | --- | --- |
| Local dev | `python -m eval.smoke_eval` (10 questions) | Iterating on retrieval changes |
| Local dev | `python -m eval.run_full --out eval/baselines/<name>.json` | Before opening a PR |
| Local dev | `python -m eval.run_full --with-llm --judge --out ...` | Pre-release; uses LLM quota |
| GitHub Actions | `.github/workflows/eval.yml` | Every PR touching `app/`, `ingest/`, `eval/` |

CI runs retrieval metrics only — no LLM calls, no rate limits, fully deterministic.
Generation faithfulness is graded offline before each release.

---

## CI gate

The job fails if either:
- `recall_at_5 < 0.85` (answerable questions where any top-5 chunk is relevant)
- `ooc_refusal_rate < 0.90` (out-of-corpus questions where the gate fired)

Set in `eval/run_full.py` via the `EVAL_CI_GATE=1` env var, so the same script
runs locally without gating.

---

## Triage: a metric just dropped. What now?

1. **Pull the artifact** from the failed PR's "eval-results" upload, or rerun
   locally: `python -m eval.run_full --out eval/baselines/diag.json`.

2. **Diff against the last good baseline** (`eval/baselines/v0.4.0-rerank-full.json`).
   Look at `results[*].verdict_pass` flips and `results[*].retrieved_top5_urls`
   shifts.

3. **Categorise the regression** by `category`:

   | Category that dropped | First suspects |
   | --- | --- |
   | `vocab-mismatch` | Reranker config, query expansion synonyms |
   | `entity-lookup` | Embedder change, index rebuild gap, chunk boundaries |
   | `fact-lookup` | Chunking strategy, embed model, top-k |
   | `multi-hop` | Top-k too small, chunk granularity, relevance threshold |
   | `ooc-negative` | `relevance_threshold` lowered too far |

4. **Reproduce in isolation**: pick one failing question and run
   `python -m app.ask --no-llm "<question>"` to see what dense retrieval gave.
   If `gate=PASS` but the right chunk isn't in top-K, the issue is rank order
   (reranker / query expansion). If `gate=FAIL`, retrieval missed entirely.

5. **Knobs in order of safety to twist** (safest first):
   - Add a synonym to `_EXPANSIONS` in `app/providers/reranker.py`
   - Bump `RERANK_TOP_N` (currently 20)
   - Re-tune `RELEVANCE_THRESHOLD` against `eval/questions_v2.jsonl`
   - Re-chunk and re-index (last resort; rebuilds 12k vectors)

6. **Never** silently lower the CI thresholds to make the gate pass. If a
   threshold is wrong, change it in a dedicated PR with an explanation, not
   in the same PR that introduced the regression.

---

## Updating the eval set

`eval/questions_v2.jsonl` is the source of truth for retrieval metrics.

When you add a question:
- Pick a `category` from the five existing ones; don't invent new ones unless
  you also update this runbook.
- `relevant_text_any` should be the smallest substrings that uniquely identify
  the right answer in the corpus (e.g. `"Désirée Finnegan"`, not `"is the"`).
- `answer_must_contain_any` is checked only in `--with-llm` mode; it's looser
  than the retrieval needle (lets the LLM paraphrase).

Re-run the baseline and commit the new `eval/baselines/*.json` so PR reviewers
can see whether your question changed the aggregate metrics.

---

## Faithfulness scoring

Faithfulness is judged by an LLM (NVIDIA Llama 70B by default) against the
retrieved chunks the answerer was actually shown. It's NOT part of the CI gate
because:

1. It's non-deterministic (judge LLM sampling).
2. It costs quota — running it on every PR would exhaust free-tier limits.
3. A faithfulness miss is a generator bug, not a retrieval bug — different
   triage path.

Run it before tagging a release:

```bash
python -m eval.run_full --with-llm --judge \
  --out eval/baselines/v$(date +%Y%m%d)-prerelease.json
```

If `faithfulness_avg < 0.85`, investigate the system prompt or grounding
discipline before shipping.