| # Report-readiness eval | |
| Scores the deterministic `is_report_ready` signal (`src/agents/report/readiness.py`) | |
| that the Help skill consumes to decide whether to nudge the user toward generating a | |
| report. No LLM, no DB — each golden case declares an analysis state + a set of | |
| persisted records/reports, and the runner feeds them through `is_report_ready` via | |
| injectable fake stores. | |
| ## Run | |
| ```bash | |
| uv run python -m eval.readiness.run_eval | |
| uv run python -m eval.readiness.run_eval --limit 5 # smoke test | |
| uv run python -m eval.readiness.run_eval --no-table # summary only | |
| ``` | |
| Each run writes a timestamped `results/readiness_result_<ts>.json` (never | |
| overwritten, diffable across runs). | |
| ## What it measures | |
| - **Floor correctness** — exact `ready` + `missing` for the deterministic floor | |
| (validated goal · ≥1 substantive record · delta-since-report). Should sit at ~100%; | |
| this is the regression guard as criteria evolve. | |
| - **Alignment gap** — `alignment` cases have substantive records (floor says | |
| `ready=true`) but `aligned=false`: the analyses don't address the problem statement. | |
| The floor can't see this. The gap count is the evidence for/against adding the | |
| deferred LLM-judge — "ship the floor, earn the judge." | |
| ## Dataset | |
| `readiness_dataset.json` — groups: `floor`, `delta`, `edge` (doc-only product | |
| question), `alignment`. See the `_about` / `_alignment` doc keys in the file. The | |
| `aligned` label is a semantic judgment; owner: Rifqi (report semantics) + Sofhia. | |