eval/help/README.md

Help-skill eval

Scores the live Help skill (src/agents/handlers/help.HelpAgent) — the guide that tells a user where they are and what to do next. Each golden case declares an analysis state + report-readiness + chat history; the runner streams HelpAgent.astream for real and asserts the rules the reply must obey.

Unlike eval/readiness (deterministic, no LLM), this calls the model, so it needs a working .env (Azure OpenAI) and spends tokens. Run it before a deploy that touches config/prompts/help.md — not on every commit. The fast, no-LLM guard is tests/unit/agents/handlers/test_help.py (fake chain); this is the end-to-end "does the model actually obey the prompt" layer on top.

Run

uv run python -m eval.help.run_eval
uv run python -m eval.help.run_eval --limit 4     # smoke test
uv run python -m eval.help.run_eval --no-table    # summary only

Each run writes a timestamped results/help_result_<ts>.json (never overwritten, diffable across runs).

What it measures

Not accuracy — Help replies are free prose with no single correct wording. The metric is compliance: the % of cases whose reply obeys every rule asserted for it.

• language — the reply must match the user's language. This is the regression guard for the button-path bug (/tools/help passes message=None, and the reply used to default to English even for an Indonesian conversation).
• report_guard — never suggest generating a report when report_ready.ready=false; do suggest it when true. Since generate_report is the only gated action, this also serves as the "no action leakage" check.
• orientation — quality of the suggested starter questions. Manual review: these run but are excluded from the auto compliance rate. Read their output_text in the JSON.

Assertion types: language_match {expected}, must_not_contain_any {patterns}, must_contain_any {patterns}.

Held-out vs carried-over (why the summary splits them)

carried_over: true cases mirror an example in help.md — the case id is the prompt's <!-- id: ... -->. They are a regression guard: if the prompt is refactored, the demonstrated rule must still hold. What is mirrored is the input spec + the assertion, never the example's reply text (temperature > 0 makes exact match invalid).

Held-out cases (carried_over: false) are absent from the prompt; their compliance is the real generalization signal. If held-out compliance drops while carried-over stays at 100%, the prompt is overfitting to its own examples ("train on test set"). That's why the two are reported separately.

Sync rule (manual, like intent): if help.md's Examples change, keep the mirrored ids here in sync. Current mirrored ids: help_ex_orient, help_ex_guard_delta, help_ex_guard_ready.

Dataset

help_dataset.json — see the _about / _carried_over doc keys in the file. Language detection reuses help._detect_reply_language; report_ready.missing uses the codes analysis / delta mapped to the real is_report_ready strings in the runner.

Known limitations

• Compliance is approximate across runs. HelpAgent runs at temperature=0.3, so the reply varies; a borderline case can flip pass/fail between runs. Treat the rate as a signal, not a fixed number — re-run before trusting a single-point drop.
• language_match grades with the same detector the feature uses (_detect_reply_language over the reply). It verifies the model obeyed the [Reply language] directive, assuming the detector is correct — the detector itself is unit-tested separately in tests/unit/agents/handlers/test_help.py. It can also misfire on a reply that mixes languages (e.g. an Indonesian reply quoting an English business question).
• Errored cases (stream crash) count as failures, not rule violations. If astream raises (Azure down, timeout), the case is flagged errored and reported under a separate ERRORED line — assertions are NOT run on the error string (a crash must not trivially "pass" a must_not_contain_any). A run with errors is not a clean pass; re-run once the cause clears.