# Help-skill eval

Scores the **live** Help skill (`src/agents/handlers/help.HelpAgent`) — the guide that
tells a user where they are and what to do next. Each golden case declares an analysis
state + report-readiness + chat history; the runner streams `HelpAgent.astream` for real
and asserts the **rules** the reply must obey.

Unlike `eval/readiness` (deterministic, no LLM), this calls the model, so it needs a
working `.env` (Azure OpenAI) and spends tokens. Run it before a deploy that touches
`config/prompts/help.md` — not on every commit. The fast, no-LLM guard is
`tests/unit/agents/handlers/test_help.py` (fake chain); this is the end-to-end
"does the model actually obey the prompt" layer on top.

## Run

```bash
uv run python -m eval.help.run_eval
uv run python -m eval.help.run_eval --limit 4     # smoke test
uv run python -m eval.help.run_eval --no-table    # summary only
```

Each run writes a timestamped `results/help_result_<ts>.json` (never overwritten,
diffable across runs).

## What it measures

Not accuracy — Help replies are free prose with no single correct wording. The metric is
**compliance**: the % of cases whose reply obeys every rule asserted for it.

- **`language`** — the reply must match the user's language. This is the regression guard
  for the button-path bug (`/tools/help` passes `message=None`, and the reply used to
  default to English even for an Indonesian conversation).
- **`report_guard`** — never suggest generating a report when `report_ready.ready=false`;
  do suggest it when `true`. Since `generate_report` is the only gated action, this also
  serves as the "no action leakage" check.
- **`orientation`** — quality of the suggested starter questions. **Manual review**: these
  run but are excluded from the auto compliance rate. Read their `output_text` in the JSON.

Assertion types: `language_match {expected}`, `must_not_contain_any {patterns}`,
`must_contain_any {patterns}`.

## Held-out vs carried-over (why the summary splits them)

`carried_over: true` cases **mirror an example in `help.md`** — the case `id` *is* the
prompt's `<!-- id: ... -->`. They are a regression guard: if the prompt is refactored, the
demonstrated rule must still hold. What is mirrored is the **input spec + the assertion**,
never the example's reply text (temperature > 0 makes exact match invalid).

Held-out cases (`carried_over: false`) are **absent from the prompt**; their compliance is
the real generalization signal. If held-out compliance drops while carried-over stays at
100%, the prompt is overfitting to its own examples ("train on test set"). That's why the
two are reported separately.

**Sync rule (manual, like `intent`):** if `help.md`'s Examples change, keep the mirrored
`id`s here in sync. Current mirrored ids: `help_ex_orient`, `help_ex_guard_delta`,
`help_ex_guard_ready`.

## Dataset

`help_dataset.json` — see the `_about` / `_carried_over` doc keys in the file. Language
detection reuses `help._detect_reply_language`; `report_ready.missing` uses the codes
`analysis` / `delta` mapped to the real `is_report_ready` strings in the runner.