File size: 2,922 Bytes
f282b15 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 | # Planner eval (E-planner)
Scores the **live planner** (`PlannerService.plan`) on golden questions against a
fixture catalog grounded in `PA Data Dummy.xlsx`. Purpose: a **regression net**
for changes to `planner.md` / `examples.py` — before/after any prompt tweak, run
this and confirm the target cases improve while `carried_over` cases stay green.
## Why rule-compliance scoring (not exact-match)
A question has **many valid IRs**, so we don't compare IRs verbatim. Each case
pins only the **properties that matter** (`expect` assertions): does a count
question use a `count` aggregate? does entity ranking `group_by` the entity? does
a fuzzy model filter use `like` instead of an enumerated `in`? See `_expect_keys`
in `planner_dataset.json` for the full assertion vocabulary.
## Run
```bash
uv run python -m eval.planner.run_eval # full run (needs Azure creds)
uv run python -m eval.planner.run_eval --limit 6 # smoke test
uv run python -m eval.planner.run_eval --selfcheck # test the scorer, no LLM
```
Each run writes `results/planner_result_<timestamp>.json` (never overwritten).
`id` is stable per case, so runs diff case-by-case over time.
## What's covered
| category | targets |
|---|---|
| `count` | scalar count → `count` aggregate (shipped fix) |
| `ranking` | top/bottom-N entities → `group_by` + `avg` + `order_by` + `limit` (**Bug 1**) |
| `fuzzy_filter` | partial model ref → `like`, never enumerate from samples (**Bug 2**) |
| `column_disambiguation` | "trend PA" must select `PA_Percent`, NOT `Plan_PA_Percent` (a wrong-column pick hidden behind alias `pa_percent`) |
| `chart` | a trend chart must aggregate before `render_chart` (not feed it 9,729 raw rows) + pick the right column |
| `aggregate`, `descriptive`, `correlation`, `trend`, `merge` | believed-correct baselines |
| `counter_raw_rows` | "show N records" must stay raw rows (guards Bug 1 fix from over-aggregating) |
| `counter_exact_filter` | exact filters stay exact (guards Bug 2 fix from over-`like`ing) |
| `infeasible` | measures absent from the catalog → `infeasible_reason` |
`carried_over: true` = behavior believed correct today (regression guard);
`false` = the known bugs. **Expected baseline (before the planner fixes):** the
`ranking` and `fuzzy_filter` (777) cases FAIL, everything else green — that gap is
exactly what the planner fixes should close, without turning any `carried_over`
case red.
## Files
- `planner_dataset.json` — cases (question + `expect` assertions)
- `catalog_fixture.py` — the `PA Data Dummy` catalog the planner plans against
- `run_eval.py` — runner + deterministic scorer (`--selfcheck`)
> Date columns are typed `date` in the fixture (the *post-fix* catalog). The live
> system currently mis-types Excel date serials as `int` — an **ingest** bug, not
> a planner one — so the fixture types them correctly to keep this eval about
> planner logic.
|