eval/readiness/README.md

Report-readiness eval

Scores the deterministic is_report_ready signal (src/agents/report/readiness.py) that the Help skill consumes to decide whether to nudge the user toward generating a report. No LLM, no DB — each golden case declares an analysis state + a set of persisted records/reports, and the runner feeds them through is_report_ready via injectable fake stores.

Run

uv run python -m eval.readiness.run_eval
uv run python -m eval.readiness.run_eval --limit 5     # smoke test
uv run python -m eval.readiness.run_eval --no-table    # summary only

Each run writes a timestamped results/readiness_result_<ts>.json (never overwritten, diffable across runs).

What it measures

• Floor correctness — exact ready + missing for the deterministic floor (validated goal · ≥1 substantive record · delta-since-report). Should sit at ~100%; this is the regression guard as criteria evolve.
• Alignment gap — alignment cases have substantive records (floor says ready=true) but aligned=false: the analyses don't address the problem statement. The floor can't see this. The gap count is the evidence for/against adding the deferred LLM-judge — "ship the floor, earn the judge."

Dataset

readiness_dataset.json — groups: floor, delta, edge (doc-only product question), alignment. See the _about / _alignment doc keys in the file. The aligned label is a semantic judgment; owner: Rifqi (report semantics) + Sofhia.