eval/planner/README.md

Planner eval (E-planner)

Scores the live planner (PlannerService.plan) on golden questions against a fixture catalog grounded in PA Data Dummy.xlsx. Purpose: a regression net for changes to planner.md / examples.py — before/after any prompt tweak, run this and confirm the target cases improve while carried_over cases stay green.

Why rule-compliance scoring (not exact-match)

A question has many valid IRs, so we don't compare IRs verbatim. Each case pins only the properties that matter (expect assertions): does a count question use a count aggregate? does entity ranking group_by the entity? does a fuzzy model filter use like instead of an enumerated in? See _expect_keys in planner_dataset.json for the full assertion vocabulary.

Run

uv run python -m eval.planner.run_eval            # full run (needs Azure creds)
uv run python -m eval.planner.run_eval --limit 6  # smoke test
uv run python -m eval.planner.run_eval --selfcheck  # test the scorer, no LLM

Each run writes results/planner_result_<timestamp>.json (never overwritten). id is stable per case, so runs diff case-by-case over time.

What's covered

category	targets
count	scalar count → count aggregate (shipped fix)
ranking	top/bottom-N entities → group_by + avg + order_by + limit (Bug 1)
fuzzy_filter	partial model ref → like, never enumerate from samples (Bug 2)
column_disambiguation	"trend PA" must select PA_Percent, NOT Plan_PA_Percent (a wrong-column pick hidden behind alias pa_percent)
chart	a trend chart must aggregate before render_chart (not feed it 9,729 raw rows) + pick the right column
aggregate, descriptive, correlation, trend, merge	believed-correct baselines
counter_raw_rows	"show N records" must stay raw rows (guards Bug 1 fix from over-aggregating)
counter_exact_filter	exact filters stay exact (guards Bug 2 fix from over-likeing)
infeasible	measures absent from the catalog → infeasible_reason

carried_over: true = behavior believed correct today (regression guard); false = the known bugs. Expected baseline (before the planner fixes): the ranking and fuzzy_filter (777) cases FAIL, everything else green — that gap is exactly what the planner fixes should close, without turning any carried_over case red.

Files

• planner_dataset.json — cases (question + expect assertions)
• catalog_fixture.py — the PA Data Dummy catalog the planner plans against
• run_eval.py — runner + deterministic scorer (--selfcheck)

Date columns are typed date in the fixture (the post-fix catalog). The live system currently mis-types Excel date serials as int — an ingest bug, not a planner one — so the fixture types them correctly to keep this eval about planner logic.