File size: 3,239 Bytes
5dcf1ee 672926b 5dcf1ee 672926b 5dcf1ee 9f235a2 5dcf1ee | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 | # Intent-Routing Eval (E3)
Scores the live 6-intent router (`OrchestratorAgent.classify`) against a golden
dataset of labelled messages. Run it before any deploy that touches the router
prompt (`src/config/prompts/intent_router.md`) or its few-shots.
## Files
| File | What |
|---|---|
| `intent_dataset.json` | **Golden dataset** — `message` + known-correct `expected_intent` per case. The source of truth scoring compares against. |
| `run_eval.py` | Runner — calls the router per case, scores correctness, records latency + tokens. |
| `results/` | Timestamped run reports, one JSON per run (never overwritten). |
## Run
Run as a module (`-m`), not the file path — module mode puts the repo root on
`sys.path` so `src` imports resolve; `python eval/intent/run_eval.py` fails.
```bash
uv run python -m eval.intent.run_eval # full dataset
uv run python -m eval.intent.run_eval --limit 6 # quick smoke test
uv run python -m eval.intent.run_eval --langfuse # also stream traces to Langfuse
```
Needs a populated `.env` (Azure OpenAI) — it calls the live model and spends
tokens. Output: a per-case detail table + an aggregate summary in the terminal,
and `results/eval_result_<timestamp>.json`.
**Tracking is the committed result files, not Langfuse** — the JSON reports in
`results/` are the versionable audit trail (see below). `--langfuse` is an
*optional* extra: when set, each case is also sent as a Langfuse trace (grouped
under one `intent_eval_<ts>` session) with a `intent_correct` 1/0 score, so the
same run is browsable in the Langfuse dashboard. It is off by default and the
eval runs fully without Langfuse configured.
## What's measured
- **correctness** — overall + per-intent + per-language accuracy (`got == expected`)
- **runtime** — average ms per case
- **tokens** — input / output / total (read from the model response, no Langfuse)
## Commit convention for `results/`
The reports are **versionable**, not a scratch log:
- **Do commit** a result after a meaningful change — e.g. a new
`intent_router.md` version, or new dataset cases. The new timestamped file
*adds* to the history; old files are never replaced. This is how we answer
"did accuracy improve after prompt v2?" — diff two committed result files.
- **Don't commit** throwaway runs while iterating. Just leave them unstaged or
delete them.
So the audit trail = prompt versions (in `src/config/prompts/`) lined up against
the committed result files here.
## Dataset notes
- 6 intents: `chat`, `help`, `problem_statement`, `check`, `unstructured_flow`,
`structured_flow`. Each has 6-7 **distinct** scenarios (not EN/ID translation
pairs), balanced across English + Indonesian.
- `carried_over: true` rows mirror the pre-rework `intent_router.md` examples
(regression). `lang` enables per-language scoring. `id` is a stable handle for
diffing the same case across runs.
- Routing labels are decided from the question **phrasing**, not from which file
holds the answer (the router has no catalog access). See the `_grounding` note
in `intent_dataset.json`.
- Owner: Rifqi (structured/DB-grounded rows) + Sofhia (unstructured/document +
tabular-file rows). Merge both into this one file.
```
|