docs/PAPER.md

> **SUPERSEDED SCAFFOLD (2026-06-12).** The paper was reframed; current title:
> "Verified Cleaning Plans: Plan-Level Selective Prediction Turns Local LLM
> Planners into Trustworthy Table Cleaners". This file is the original outline,
> kept for history. The live paper is docs/paper/main.tex.

# ScrubData — paper scaffold & related-work map

**Working title:** *Small fine-tuned planners with execution-verified data and calibrated
abstention match larger models on tabular canonicalization.*

**One-line claim (measured):** a ≤4B fine-tune that emits a *cleaning plan* (not edited cells)
reaches `canon_f1 0.86` on alias-level canonicalization vs `0.45` for a large generic model and
`0.13` for a rule heuristic — and, with reference grounding + calibrated abstention, beats the
tool people actually use (OpenRefine) on a wide validation suite at far lower damage.

## Contributions (the combination is the novelty — not "LLM cleans data")
1. **Planner/executor decomposition.** The model proposes a structured JSON plan; deterministic
   pandas executes it. Auditable, reversible, **no silent edits** (`observability.py`,
   `trace.py`). This is the trust/monitorability contract.
2. **Execution-self-verified synthetic SFT.** Every training example's plan is checked to
   actually recover the known-clean original by *running the executor* (`training/build_dataset.py`).
   A clean, citable data-generation method (drops non-recovering examples).
3. **Reference grounding + calibrated abstention.** Canonicalization is reconciled against a
   type-scoped taxonomy (GeoNames/pycountry; `reconcile.py`, `grounded.py`); the system ABSTAINS
   under ambiguity instead of hallucinating a canonical (`eval/calibration.py`: risk-coverage +
   ECE). Structural fix for the over-correction larger models also exhibit.
4. **Aggregation + column-batching.** Prompt size scales with *distinct values*, not rows
   (`profiler.py` value_counts + `model_planner.make_batched_planner`).

## Related work (position against — reviewers know this field)
- **Error detection/repair:** Raha & Baran (Mahdavi et al.), HoloClean (Rekatsinas et al. 2017,
  `arXiv 1702.00820`), GARF — we *use* their hospital/beers/flights/rayyan as OOD eval and cite
  GARF as the frequency-only baseline our grounding beats (it cannot supply a canonical for a lone
  column).
- **LLMs for data wrangling:** "Can Foundation Models Wrangle Your Data?" (Narayan et al. 2022),
  Jellyfish, Table-GPT/TableLlama (`2311.09206`), RetClean (`2303.16909`). We differ by being a
  *small fine-tuned planner* + grounding + abstain, not a large zero-shot value-editor.
- **Grounding / entity disambiguation:** RACOON (`2409.14556`), TURL (`2006.14806`), Belotti et al.
  table-EL (`2408.06423`), MTab — motivate retrieval-then-abstain and warn against memorizing
  canonicals into weights (TURL ~40% OOD collapse). See `taxonomy-grounding.md`.
- **The tool we beat:** **OpenRefine** clustering — fingerprint (key collision) + nearest-neighbor
  (kNN/edit-distance), reimplemented as `scrubdata/baselines.py` for head-to-head.
- **Selective prediction:** calibrated abstention / risk-coverage (El-Yaniv & Wiener; Geifman &
  El-Yaniv) — our ECE/AURC study; also the AI-safety monitorability framing.

## Experiments
- **Headline:** canon_f1 vs large-generic vs heuristic on frozen synthetic gold (Layer 1).
- **Wide north-star (`eval/run_real_multi.py`):** double-macro (error-type × domain) F1 + damage +
  abstain over Raha real-error sets **+ seeded error-injection** on 20+ harvested gov/GitHub clean
  domains (`eval/inject.py`); multi-seed 95% CIs. Hospital is 1 dataset of many.
- **Money result:** grounded vs OpenRefine fingerprint & kNN on the same suite (grounded wins F1 +
  damage; kNN over-merges — higher recall, low precision, high damage).
- **Calibration (`eval/calibration.py`):** risk-coverage, AURC, ECE; operating point for ≥95%
  precision via the abstain threshold.
- **Ablations to add:** −grounding, −abstain, −execution-verification, −aggregation.

## Honest limitations (the integrity reviewers reward)
- Reference *coverage* is the recall ceiling (Belotti) — uncovered entities abstain by design.
- Convention vs error: standardization (date→ISO, `%`→fraction) is product value, not damage —
  the metric is case/whitespace-normalized but a format-aware variant is future work.
- ECE shows mild over-confidence (difflib-ratio scores) — temperature/Platt scaling is future work.
- Some benchmark sources gated (CleanML/TableEG behind Dropbox/Drive; licenses noted).

## To-do before submission
multi-seed CIs (running) · −ablations · OpenRefine table with CIs · cs.DB endorser (primary cs.DB, cross-list cs.CL+cs.LG; endorser targets = the data-cleaning authors we cite) · selective-
prediction figure · keep the eval README's convention-vs-error honesty.