File size: 3,127 Bytes
3743cfe | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 | # Help-skill eval
Scores the **live** Help skill (`src/agents/handlers/help.HelpAgent`) — the guide that
tells a user where they are and what to do next. Each golden case declares an analysis
state + report-readiness + chat history; the runner streams `HelpAgent.astream` for real
and asserts the **rules** the reply must obey.
Unlike `eval/readiness` (deterministic, no LLM), this calls the model, so it needs a
working `.env` (Azure OpenAI) and spends tokens. Run it before a deploy that touches
`config/prompts/help.md` — not on every commit. The fast, no-LLM guard is
`tests/unit/agents/handlers/test_help.py` (fake chain); this is the end-to-end
"does the model actually obey the prompt" layer on top.
## Run
```bash
uv run python -m eval.help.run_eval
uv run python -m eval.help.run_eval --limit 4 # smoke test
uv run python -m eval.help.run_eval --no-table # summary only
```
Each run writes a timestamped `results/help_result_<ts>.json` (never overwritten,
diffable across runs).
## What it measures
Not accuracy — Help replies are free prose with no single correct wording. The metric is
**compliance**: the % of cases whose reply obeys every rule asserted for it.
- **`language`** — the reply must match the user's language. This is the regression guard
for the button-path bug (`/tools/help` passes `message=None`, and the reply used to
default to English even for an Indonesian conversation).
- **`report_guard`** — never suggest generating a report when `report_ready.ready=false`;
do suggest it when `true`. Since `generate_report` is the only gated action, this also
serves as the "no action leakage" check.
- **`orientation`** — quality of the suggested starter questions. **Manual review**: these
run but are excluded from the auto compliance rate. Read their `output_text` in the JSON.
Assertion types: `language_match {expected}`, `must_not_contain_any {patterns}`,
`must_contain_any {patterns}`.
## Held-out vs carried-over (why the summary splits them)
`carried_over: true` cases **mirror an example in `help.md`** — the case `id` *is* the
prompt's `<!-- id: ... -->`. They are a regression guard: if the prompt is refactored, the
demonstrated rule must still hold. What is mirrored is the **input spec + the assertion**,
never the example's reply text (temperature > 0 makes exact match invalid).
Held-out cases (`carried_over: false`) are **absent from the prompt**; their compliance is
the real generalization signal. If held-out compliance drops while carried-over stays at
100%, the prompt is overfitting to its own examples ("train on test set"). That's why the
two are reported separately.
**Sync rule (manual, like `intent`):** if `help.md`'s Examples change, keep the mirrored
`id`s here in sync. Current mirrored ids: `help_ex_orient`, `help_ex_guard_delta`,
`help_ex_guard_ready`.
## Dataset
`help_dataset.json` — see the `_about` / `_carried_over` doc keys in the file. Language
detection reuses `help._detect_reply_language`; `report_ready.missing` uses the codes
`analysis` / `delta` mapped to the real `is_report_ready` strings in the runner.
|