# Planner eval (E-planner)

Scores the **live planner** (`PlannerService.plan`) on golden questions against a
fixture catalog grounded in `PA Data Dummy.xlsx`. Purpose: a **regression net**
for changes to `planner.md` / `examples.py` — before/after any prompt tweak, run
this and confirm the target cases improve while `carried_over` cases stay green.

## Why rule-compliance scoring (not exact-match)

A question has **many valid IRs**, so we don't compare IRs verbatim. Each case
pins only the **properties that matter** (`expect` assertions): does a count
question use a `count` aggregate? does entity ranking `group_by` the entity? does
a fuzzy model filter use `like` instead of an enumerated `in`? See `_expect_keys`
in `planner_dataset.json` for the full assertion vocabulary.

## Run

```bash
uv run python -m eval.planner.run_eval            # full run (needs Azure creds)
uv run python -m eval.planner.run_eval --limit 6  # smoke test
uv run python -m eval.planner.run_eval --selfcheck  # test the scorer, no LLM
```

Each run writes `results/planner_result_<timestamp>.json` (never overwritten).
`id` is stable per case, so runs diff case-by-case over time.

## What's covered

| category | targets |
|---|---|
| `count` | scalar count → `count` aggregate (shipped fix) |
| `ranking` | top/bottom-N entities → `group_by` + `avg` + `order_by` + `limit` (**Bug 1**) |
| `fuzzy_filter` | partial model ref → `like`, never enumerate from samples (**Bug 2**) |
| `column_disambiguation` | "trend PA" must select `PA_Percent`, NOT `Plan_PA_Percent` (a wrong-column pick hidden behind alias `pa_percent`) |
| `chart` | a trend chart must aggregate before `render_chart` (not feed it 9,729 raw rows) + pick the right column |
| `aggregate`, `descriptive`, `correlation`, `trend`, `merge` | believed-correct baselines |
| `counter_raw_rows` | "show N records" must stay raw rows (guards Bug 1 fix from over-aggregating) |
| `counter_exact_filter` | exact filters stay exact (guards Bug 2 fix from over-`like`ing) |
| `infeasible` | measures absent from the catalog → `infeasible_reason` |

`carried_over: true` = behavior believed correct today (regression guard);
`false` = the known bugs. **Expected baseline (before the planner fixes):** the
`ranking` and `fuzzy_filter` (777) cases FAIL, everything else green — that gap is
exactly what the planner fixes should close, without turning any `carried_over`
case red.

## Files

- `planner_dataset.json` — cases (question + `expect` assertions)
- `catalog_fixture.py` — the `PA Data Dummy` catalog the planner plans against
- `run_eval.py` — runner + deterministic scorer (`--selfcheck`)

> Date columns are typed `date` in the fixture (the *post-fix* catalog). The live
> system currently mis-types Excel date serials as `int` — an **ingest** bug, not
> a planner one — so the fixture types them correctly to keep this eval about
> planner logic.