eval/intent/README.md

# Intent-Routing Eval (E3)

Scores the live 6-intent router (`OrchestratorAgent.classify`) against a golden
dataset of labelled messages. Run it before any deploy that touches the router
prompt (`src/config/prompts/intent_router.md`) or its few-shots.

## Files

| File | What |
|---|---|
| `intent_dataset.json` | **Golden dataset** — `message` + known-correct `expected_intent` per case. The source of truth scoring compares against. |
| `run_eval.py` | Runner — calls the router per case, scores correctness, records latency + tokens. |
| `results/` | Timestamped run reports, one JSON per run (never overwritten). |

## Run

Run as a module (`-m`), not the file path — module mode puts the repo root on
`sys.path` so `src` imports resolve; `python eval/intent/run_eval.py` fails.

```bash
uv run python -m eval.intent.run_eval            # full dataset
uv run python -m eval.intent.run_eval --limit 6  # quick smoke test
uv run python -m eval.intent.run_eval --langfuse # also stream traces to Langfuse
```

Needs a populated `.env` (Azure OpenAI) — it calls the live model and spends
tokens. Output: a per-case detail table + an aggregate summary in the terminal,
and `results/eval_result_<timestamp>.json`.

**Tracking is the committed result files, not Langfuse** — the JSON reports in
`results/` are the versionable audit trail (see below). `--langfuse` is an
*optional* extra: when set, each case is also sent as a Langfuse trace (grouped
under one `intent_eval_<ts>` session) with a `intent_correct` 1/0 score, so the
same run is browsable in the Langfuse dashboard. It is off by default and the
eval runs fully without Langfuse configured.

## What's measured

- **correctness** — overall + per-intent + per-language accuracy (`got == expected`)
- **runtime** — average ms per case
- **tokens** — input / output / total (read from the model response, no Langfuse)

### Content-filter blocks count as `out_of_scope` passes

Aggressive jailbreak / manipulation inputs are often rejected by Azure's own
content filter (HTTP 400, `code=content_filter`) *before* the router model runs.
The live app treats that as a refusal (`chat_handler._is_content_filter_error`),
so for an `out_of_scope` case the block **is** the correct end-to-end outcome. The
runner mirrors this: such a case is recorded as `got=blocked` and scored **correct**
(not `ERROR:BadRequestError`). This keeps `out_of_scope` accuracy honest — the
router isn't penalised for inputs the platform guardrail caught first. A
content-filter block on any *other* expected intent is still scored wrong (an
unexpected block). Non-filter exceptions remain `ERROR:<type>` and score wrong.

## Commit convention for `results/`

The reports are **versionable**, not a scratch log:

- **Do commit** a result after a meaningful change — e.g. a new
  `intent_router.md` version, or new dataset cases. The new timestamped file
  *adds* to the history; old files are never replaced. This is how we answer
  "did accuracy improve after prompt v2?" — diff two committed result files.
- **Don't commit** throwaway runs while iterating. Just leave them unstaged or
  delete them.

So the audit trail = prompt versions (in `src/config/prompts/`) lined up against
the committed result files here.

## Dataset notes

- 6 intents: `chat`, `help`, `check`, `unstructured_flow`, `structured_flow`,
  `out_of_scope`. Each has 6+ **distinct** scenarios (not EN/ID translation
  pairs), balanced across English + Indonesian. (`problem_statement` was dropped
  from the router on 2026-06-24 — the goal is now user-entered `objective` +
  `business_questions`, no agent validation — so its cases were removed here.)
- `carried_over: true` rows mirror the pre-rework `intent_router.md` examples
  (regression). `lang` enables per-language scoring. `id` is a stable handle for
  diffing the same case across runs.
- Routing labels are decided from the question **phrasing**, not from which file
  holds the answer (the router has no catalog access). See the `_grounding` note
  in `intent_dataset.json`.
- Owner: Rifqi (structured/DB-grounded rows) + Sofhia (unstructured/document +
  tabular-file rows). Merge both into this one file.
```