70-docs/40-evaluation/eval-methodology.md

# 03 — Evaluation Plan

> ⚠️ **Methodology is current; some implementation pointers are historical.**
> The gold-set construction, grader signals, and metrics below still
> describe how we evaluate the bot. But the *system under test* is now the
> single-LLM-with-tools handler (`backend/single_brain.py` — one Gemini
> 2.5-flash call per turn with `save_profile_field` / `retrieve_policies` /
> `mark_recommendation`, structured+vector retrieval, small
> `nim_fallback.py` for transient errors). There is no `orchestrator`, no
> `sales_brain`/`qa_brain` split, no separate faithfulness-judge LLM, and no
> DuckDB hot path. Present-state authority: [`README.md`](../../README.md) §4.

| Field | Value |
| --- | --- |
| Project | Insurance Sales Portfolio Expert |
| Version | 0.1 |
| Date | 2026-05-13 |
| Depends on | `01-requirements.md` §6 (success criteria); README §4 (system under test) |
| Status | Pipeline implemented; full run pending corpus completion |

## 0. Purpose

Evaluation is **how we prove the bot works**. Three artifacts:

1. **Gold Q&A set** — ground-truth question-answer-source triples
2. **Automated grader** — measures factual accuracy + citation accuracy + refusal precision per turn
3. **Results table** — versioned, audit-grade record of every eval run

These directly back the success criteria in Doc 01 §6: C2 (≥95% factual), C3 (≥95% citation), C4 (≥90% refusal precision).

## 1. Gold Q&A construction — three pipelines

### Pipeline A — auto-generated from structured extraction (the bulk)

For every successfully extracted policy, generate templated Q&A where the answer comes directly from the 62-field structured extraction. Per `eval/generate_gold.py`:

- ~15 question templates × ~80 policies = ~1,100 candidate pairs
- Each pair is **fully reproducible** — the answer traces to a specific field which traces to a specific clause
- Auto-tagged: `question_type` ∈ {waiting_period, coverage_scope, exclusions, sub_limit, eligibility, claim, network, bonus}

**Why this scales:** adding the 11th, 12th, 80th policy adds zero new template work. Same generator, more rows.

### Pipeline B — LLM-drafted nuanced questions (curated)

For top-priority policies (top 10–20), run an LLM on the policy text with prompt: *"generate 5 buyer-style questions whose answers are explicitly in this document; include the source clause."* Each generated pair is **human spot-checked** before commit.

Target: 5 × 20 = 100 questions covering multi-clause and edge-case reasoning that Pipeline A can't generate.

### Pipeline C — hand-crafted adversarial set (refusal tests)

~30–40 hand-written questions across these classes:

- **Out-of-corpus** ("Does this cover space tourism?") → expect refusal
- **Out-of-policy-type** ("What is the IRDAI mandate on dental coverage?") → expect refusal (D-017)
- **Multi-policy compare** ("Compare cancer coverage in Star vs HDFC ERGO") → expect cited comparison
- **Hinglish** ("Cataract ke liye waiting period kya hai?") → expect Hindi answer, same factual accuracy
- **Code-switched** ("policy mein maternity cover hai kya?") → expect Hinglish answer

Each pair marked with `expected_refusal: bool`. Refusal cases test that the bot doesn't hallucinate when grounding fails.

## 2. Grader design

**Single grading endpoint:** `eval/run.py` calls the single-LLM-with-tools turn handler (`backend/single_brain.py`) in-process, takes the reply, scores against gold with three signals:

### Signal 1 — Regex hard-checks (deterministic)

Numbers, dates, currency, durations, percentages are extracted via regex from both gold and bot reply. Exact match (after normalization) is the strongest signal. Catches "the premium is ₹15,000" hallucinations without an LLM.

### Signal 2 — LLM-judge faithfulness (offline eval grader only)

The eval grader uses a **different model family** from the runtime Gemini 2.5-flash brain → **non-circular evaluation**. This judge is part of the *offline eval harness only* — it is not a runtime gate in the bot (runtime grounding is structural: the single LLM can only state what `retrieve_policies` returned and must cite it). Judge prompt:

> Given GOLD and BOT answers, output strict JSON: `{factual_match: bool, citation_present: bool, score: 0-1, reason: str}`. Be strict on partial matches.

### Signal 3 — Citation regex check

`[Source: ...]` tag must be present in BOT for non-refusal questions. Caught by `re.search(r'\[Source:'` pattern.

**Final per-question score:** `factual_match AND (citation_present OR expected_refusal)`.

## 3. Metrics computed

| Metric | Doc 01 target | How |
| --- | --- | --- |
| Factual accuracy | C2 ≥ 95% | n_correct / n_total |
| Citation accuracy | C3 ≥ 95% | n_correct_citations / n_non_refusal |
| Refusal precision | C4 ≥ 90% | n_correct_refusals / n_expected_refusals |
| Hindi parity | C8 within 5pp | factual_acc(hi) vs factual_acc(en) |
| Path winners | (config) | factual_acc grouped by which path produced the turn — `gemini` (primary single-LLM call) vs `nim_fallback` (transient-error fallback) |

## 4. Output artifacts per run

- `eval/results.md` — human-readable summary with per-type, per-path accuracy + sample misses
- `eval/results.json` — machine-readable full per-question record
- `logs/hallucinations.jsonl` — every blocked reply with its reason (audit log)

## 5. Run cadence

| Stage | Cadence | Implementation |
| --- | --- | --- |
| Development | Manual, after meaningful changes | `python -m eval.run` |
| Pre-deploy | Every PR | GitHub Actions runs eval, blocks merge if accuracy regresses |
| Production | Nightly synthetic + spot-grading | Scheduled job; live-traffic sampling via Playwright |
| Post-deploy verification | Per deploy | `tests/live_verify.py` runs full eval against the live HF Space URL |

## 6. Bilingual eval

Bilingual sub-set: 20 questions translated to Hindi + Hinglish via Sarvam-M with manual spot-check. Run separately. Hindi factual accuracy compared to English baseline — must be within 5pp per C8.

## 7. Known limitations (transparent)

- **Sample bias:** Pipeline A questions are templated, so accuracy on Pipeline A is upper-bound real-world performance. Pipeline B + C provide the harder signal.
- **Single judge model:** the offline eval grader uses one judge model (a different family from the runtime Gemini brain) per question. Risk of judge-specific bias. v2: 3-judge consensus.
- **No human evaluation:** Cost-prohibitive for 1,100 questions. We rely on the grader; manual spot-check 5%.
- **No latency budget enforcement in eval:** Latency is captured per record but doesn't gate the score. Doc 01 C1 (p50 ≤ 4s) is monitored separately.