eval/intent/README.md

Intent-Routing Eval (E3)

Scores the live 6-intent router (OrchestratorAgent.classify) against a golden dataset of labelled messages. Run it before any deploy that touches the router prompt (src/config/prompts/intent_router.md) or its few-shots.

Files

File	What
intent_dataset.json	Golden dataset — message + known-correct expected_intent per case. The source of truth scoring compares against.
run_eval.py	Runner — calls the router per case, scores correctness, records latency + tokens.
results/	Timestamped run reports, one JSON per run (never overwritten).

Run

Run as a module (-m), not the file path — module mode puts the repo root on sys.path so src imports resolve; python eval/intent/run_eval.py fails.

uv run python -m eval.intent.run_eval            # full dataset
uv run python -m eval.intent.run_eval --limit 6  # quick smoke test
uv run python -m eval.intent.run_eval --langfuse # also stream traces to Langfuse

Needs a populated .env (Azure OpenAI) — it calls the live model and spends tokens. Output: a per-case detail table + an aggregate summary in the terminal, and results/eval_result_<timestamp>.json.

Tracking is the committed result files, not Langfuse — the JSON reports in results/ are the versionable audit trail (see below). --langfuse is an optional extra: when set, each case is also sent as a Langfuse trace (grouped under one intent_eval_<ts> session) with a intent_correct 1/0 score, so the same run is browsable in the Langfuse dashboard. It is off by default and the eval runs fully without Langfuse configured.

What's measured

• correctness — overall + per-intent + per-language accuracy (got == expected)
• runtime — average ms per case
• tokens — input / output / total (read from the model response, no Langfuse)

Content-filter blocks count as out_of_scope passes

Aggressive jailbreak / manipulation inputs are often rejected by Azure's own content filter (HTTP 400, code=content_filter) before the router model runs. The live app treats that as a refusal (chat_handler._is_content_filter_error), so for an out_of_scope case the block is the correct end-to-end outcome. The runner mirrors this: such a case is recorded as got=blocked and scored correct (not ERROR:BadRequestError). This keeps out_of_scope accuracy honest — the router isn't penalised for inputs the platform guardrail caught first. A content-filter block on any other expected intent is still scored wrong (an unexpected block). Non-filter exceptions remain ERROR:<type> and score wrong.

Commit convention for results/

The reports are versionable, not a scratch log:

• Do commit a result after a meaningful change — e.g. a new intent_router.md version, or new dataset cases. The new timestamped file adds to the history; old files are never replaced. This is how we answer "did accuracy improve after prompt v2?" — diff two committed result files.
• Don't commit throwaway runs while iterating. Just leave them unstaged or delete them.

So the audit trail = prompt versions (in src/config/prompts/) lined up against the committed result files here.

Dataset notes

• 6 intents: chat, help, check, unstructured_flow, structured_flow, out_of_scope. Each has 6+ distinct scenarios (not EN/ID translation pairs), balanced across English + Indonesian. (problem_statement was dropped from the router on 2026-06-24 — the goal is now user-entered objective + business_questions, no agent validation — so its cases were removed here.)
• carried_over: true rows mirror the pre-rework intent_router.md examples (regression). lang enables per-language scoring. id is a stable handle for diffing the same case across runs.
• Routing labels are decided from the question phrasing, not from which file holds the answer (the router has no catalog access). See the _grounding note in intent_dataset.json.
• Owner: Rifqi (structured/DB-grounded rows) + Sofhia (unstructured/document + tabular-file rows). Merge both into this one file.