eval/README.md

# Eval harness + goalpost

Measures any planner against a **held-out** synthetic gold set (seed differs from
training, and gold is filtered to oracle-solvable so the ceiling is a clean 1.0).

```bash
uv run eval/run_eval.py --n 300 --seed 4242
```

Adopts the researched tooling: `jsonschema` for plan validity; set-based micro-F1 for
operations and canonicalization mappings; the **executor itself** for end-to-end
cell-recovery (the Raha-style dirty→clean comparison). promptfoo + `llm-rubric` will
wrap the report-quality layer once a model exists.

## Metrics
- **json_valid** — plan conforms to the schema (`eval/metrics.py:PLAN_SCHEMA`).
- **op_f1 / op_r** — micro-F1 / recall over `(column, operation)` pairs vs gold.
- **canon_f1 / canon_r** — micro-F1 / recall over `(column, raw→canonical)` mapping
  pairs. *This is the fuzzy skill rules can't do — the whole reason for the model.*
- **recovery** — fraction of clean-reference cells recovered by executing the plan.

## Baseline (measured) and the goalpost

Two reference systems frame every run:
- **ORACLE** = the gold plan → the ceiling.
- **HEURISTIC** (`scrubdata.mock_plan`) = the rule-based baseline the model must beat.

Measured on the frozen 300-example gold set (`eval/gold.jsonl`, **value_counts/aggregation
format**):

| system | json_valid | op_f1 | canon_f1 | canon_r | recovery |
|---|---|---|---|---|---|
| ORACLE (gold) | 1.000 | 1.000 | 1.000 | 1.000 | **1.000** |
| HEURISTIC (baseline) | 1.000 | 0.932 | **0.189** | 0.129 | **0.637** |

**Reading:** with case-folding + typo-clustering the heuristic does the *easy*
canonicalization (collapse to most-frequent surface), but it's still ~blind to
**alias/semantic** canonicalization (`USA`→`United States`, `NYC`→`New York`) — canon_f1
0.19 vs the oracle's 1.0. That gap is the fine-tuned model's job. (Earlier, on the old
sample-rows format, a fine-tune reached canon_f1 0.86 vs a big vanilla model's 0.45 —
proving small-aligned > big-generic; the v4 retrain re-establishes this on the new format.)

### 🎯 Goalpost for the fine-tuned Qwen3-4B
| metric | baseline | **target** | ceiling |
|---|---|---|---|
| json_valid | 1.000 | **≥ 0.99** | 1.000 |
| op_f1 | 0.932 | **≥ 0.98** | 1.000 |
| canon_f1 | 0.189 | **≥ 0.85** | 1.000 |
| recovery | 0.637 | **≥ 0.95** | 1.000 |

A fine-tune that hits these clearly beats the (now stronger) heuristic and approaches the
oracle — the headline being **canon_f1 0.133 → ≥0.85** (alias-level canonicalization) and
**recovery 0.627 → ≥0.95**.

## Plugging in the model
`evaluate(planner, gold)` takes any `planner(dirty_df, gold_plan) -> plan dict`. For
the model, wrap inference (build prompt via `scrubdata.prompt`, parse JSON) and pass it
in alongside the two reference systems. Track the table every fine-tune iteration; the
per-metric delta vs baseline is the cheap regression signal.

## Layer 2 — real out-of-distribution data (`uv run eval/run_real.py`)

Raha `hospital` (1000×20, row-aligned dirty/clean). Errors are char-substitution typos
(`birminghxm`→`birmingham`) — only ~2.5% of cells. Scored with the Raha **repair**
protocol (the right metric when data is already mostly correct):

| system | recovery | repair_recall | repair_prec | broken |
|---|---|---|---|---|
| NO-OP (dirty as-is) | 0.975 | 0.000 | 0.000 | 0 |
| HEURISTIC (baseline) | 0.880 | **0.293** | 0.065 | 2041 |

(Typo-clustering now fixes ~29% of the real char-substitution errors — up from 0. The
model should push repair_recall higher and improve repair_prec.)

**Reading (honest + important):** the rule heuristic fixes **0** typos. Its 2021 changed
cells are **convention divergence, not errors** — our tool parses `100%`→`1.0` and
reformats phones, which this benchmark stores as raw text. That's product value, so raw
`recovery`/`broken` *understates* a standardizing tool on a foreign benchmark. The honest
metric here is **`repair_recall`** — did we fix the actual char-substitution typos
(`birminghxm`→`birmingham`)? The heuristic can't (scores 0); cluster-canonicalization is
the model's job. Two takeaways:
1. **The headline real-data metric is `repair_recall`** (error-fixing), not recovery.
2. **Product feature surfaced:** offer a "preserve original formats" toggle — some users
   want raw representation kept; standardizing is the default but should be reversible
   (matches PRODUCT.md's trust contract).

### 🎯 Real-data goalpost (fine-tuned model)
| metric | NO-OP | HEURISTIC | **target** | note |
|---|---|---|---|---|
| **repair_recall** | 0.000 | 0.000 | **≥ 0.30** | the real test — fix typos via clustering |
| repair_prec | 0.000 | 0.000 | **≥ 0.70** | of cells changed, fraction that fixed an error |
| recovery | 0.975 | 0.874 | report-only | convention-sensitive; not a pass/fail gate |

The model plugs into `_score(dirty, clean, model_output)` exactly like the heuristic.

> Data auto-fetched to `data/real/hospital/` (gitignored). Add Flights/Beers/CleanML the
> same way for breadth.

## Scale: aggregation + agentic batching (validated)

Cleaning *large* tables doesn't mean bigger prompts — it means reasoning over **patterns**:
- **Aggregation** — the profiler sends per-column `value_counts` (`[value, frequency]`), so
  the prompt size depends on DISTINCT values, not rows. Rare typos sit at the tail next to
  their dominant canonical (`birminghxm`:1 vs `birmingham`:312) — visible at any scale.
- **Column batching** — `scrubdata.model_planner.make_batched_planner` plans a wide table
  in small column-batches, so a 20-column table never blows one prompt.

**Validated** on the real Raha hospital table (1000×20) with a *vanilla* model (no retrain):
**repair_recall 0.509** (fixed 259/509 typos), vs **0.000** for the old one-shot+sample-rows
approach. The v4 fine-tune trains on this `value_counts` format.

---

## The wide suite (current north-star)

The single-dataset hospital metric was retired as north-star (biased: one table,
recall-only, convention-sensitive, abstain-blind). The current harness:

- **`run_real_multi.py`** — 65-dataset suite (5 Raha real-error benchmarks + seeded
  error injection over 15 harvested open-data domains), scored with a **churn-neutral**
  metric (pure case/whitespace rewrites that don't restore gold count as nothing) and
  aggregated as a **double macro** (error-type × domain, harmonic mean) so no single
  table or error type dominates. Reports REAL vs INJECTED slices separately — injected
  typos are in-distribution for frequency clustering by construction.
- **`ablations.py`** — removes one grounding component at a time (reference, abstain,
  ambiguity margin, case-match). Caught two metric artifacts (churn inflation,
  reference-unsafe traps) now fixed and documented in the paper.
- **`calibration.py`** — risk–coverage + ECE for the abstention confidence
  (AURC 0.120; 90% precision at the default threshold, ≥95% at 0.91).
- **`pii_leak.py`** — masking leak test: 0/360 residual detectable PII.
- **`pii_slice.py`** — OOD PII column typing on Gretel test: 5/5 types, 0/7 FP.
- **`inject.py`** — seeded, self-verifying error injectors (typo/OCR/case/whitespace)
  that turn any clean table into validation data.

Baselines include OpenRefine fingerprint + kNN clustering (`scrubdata/baselines.py`,
with blocking, as the real tool uses). Full results & discussion: `docs/paper/`.