docs/FIELD_NOTES.md

Field notes — building ScrubData small, on purpose

Build Small Hackathon, June 2026. A ≤4B model, a Gradio Space, and two weeks of finding out what "small but honest" actually costs.

The bet

The person who most needs data cleaning — the ops coordinator with a messy CRM export and a Monday deadline — will never write a pandas script, and shouldn't have to ship her customer data to a frontier API either. The bet: a 4B model running locally is enough, if you stop asking it to edit data and start asking it to plan.

So the model never touches a cell. It reads an aggregated profile (per-value frequency counts — so the model sees a bounded, fixed-size summary whether the table has a hundred rows or a million) and emits a JSON plan; deterministic pandas executes it. Every change is named, reversible, and logged. Silent edits are impossible by construction. That decomposition turned out to be the whole project.

Things that broke, in order

The fine-tune that aced the test and failed the job. v4 hit canonicalization F1 0.90 on held-out synthetic data — and scored exactly 0.000 on real hospital typos. It had never seen a high-cardinality real column. Fix: derive training pairs from real dirty/clean benchmark tables by cell alignment, keeping only learnable canonicalizations (a surface form that's a string variant of its target and never a legitimate value elsewhere). Real repair recall: 0.00 → 0.42. Synthetic data teaches the format; real data teaches the job.

The GGUF that lobotomized the model. Same adapter, two exports: Q8_0 worked perfectly, Q4_K_M degenerated into <tool_call> loops. Hours of template debugging later: the quantization itself was corrupting the export. Then the bf16 path had its own version — training converged (loss 0.16) but free-running generation still emitted tool-call loops, because Qwen3's tool-calling prior dominates the first token. The fix is two tokens long: suppress_tokens=[151657, 151658].

The model that invented cities. Asked for canonical forms, a generative model generates — including guntxrsvillx → huntsville (wrong town). Frequency clustering can't fix this either: a lone column has no signal to vote against the error (GARF proves this structurally). The fix came from the literature: never free-generate a canonical. Retrieve candidates from a reference taxonomy (GeoNames, ISO), require a similarity threshold and an ambiguity margin, and abstain when unsure. boxz is equally close to Box and Boaz — so the system declines and asks. We measured the abstention: precision rises monotonically with the threshold (90% at the default, 95% at 0.91). Knowing when not to act turned out to be the most valuable feature.

The eval that graded itself too kindly — twice. Our own ablations caught two metric artifacts: (1) convention-tolerant scoring counted bulk case-rewrites as "good changes," inflating precision — removing case-matching gained +0.12 until we made the metric churn-neutral; (2) our adversarial traps included Boazz, which grounding correctly maps to the real city Boaz — the trap was punishing correct behavior. Both fixes are reported in the paper as results, because an eval you haven't tried to break is an eval you can't trust.

The honest negative result. On injected typos, classical frequency clustering remains a strong baseline — by construction: injection puts the canonical in the column, which is clustering's ideal regime. Grounding's edge is real errors, tail entities, and not wrong-merging. We report both slices separately rather than averaging the difference away.

The verifier that made the model shippable. The fine-tune's hospital numbers told an awkward story: recall 0.475 (best we'd measured for a local model) at precision 0.185 — it fixed errors and invented merges. Instead of retraining, we scored every proposed mapping with three deterministic gates distilled from its actual failures: a value occurring ≥3 times is data, not a typo (errors are rare); a repair target must dominate its source in frequency (no mapping one typo onto another); digit-bearing codes only repair when the letter part is near-identical (amix-2 → ami-2 yes, ak_ → al_ no). The gated model plan alone: 0.993 precision at 0.287 coverage — 146 of 147 changes correct. Union it with the grounded heuristic and you get 0.905 precision at 0.413 coverage on hospital's 509 real errors. Every dropped mapping becomes a review flag, not a silent skip. That composition — verify the model's output, never trust it — is what the app now ships as its default planner.

The PII turn

A friend pointed at the OpenMed project (small Apache-2.0 token classifiers; their paper is the sister result to our thesis — small specialized beats big generic). Their 44M PII model, trained on clinical sentences, turned out to transfer perfectly to bare CSV cells: 100% on names and addresses, no prompt template needed. We put it behind a sensitive-type allowlist and a column-level vote, added a deterministic checksum tier (Luhn, IBAN mod-97 — math, not vibes), and made masking an executor operation. Leak test: 0/360 residual detectable PII after masking. OOD type detection: 5/5 with 0/7 false positives. The privacy ribbon at the top of the app — "nothing leaves this machine" — now describes the PII handling too, not just the inference.

The word that broke the demo

We shipped the engine, then sent the live Space to people who actually have messy spreadsheets and aren't data people. The most useful feedback wasn't a bug report — it was that the word "cleaning" didn't mean anything to them. One tester read "clean my Excel" as deleting data: "¿Te refieres a que elimine algo de algún archivo?" ("you mean it removes something from the file?"). Another didn't know where to start: "¿eso del Excel te lo subimos ahí o cómo?" ("the Excel thing — do we upload it there, or how?"). The clearest explanation of the whole product turned out to be a sentence we typed by hand in a chat reply — "it fixes text errors: names, phones, emails, cities" — and that sentence was nowhere in the app.

The engine was fine. The framing was the failure. So we changed the product to show what cleaning is instead of naming it: the hero now opens with a literal before→after strip (nigeia → Nigeria, Calfornia → California) before any upload, the headline is the sentence that worked in chat ("Fix the messy text in your spreadsheet"), the copy says plainly "I never delete your data," jargon labels are gone ("with PII" → "with sensitive data"), and a one-click "watch it run on a sample" path removes the "where do I even start" wall. One honesty footnote from the rewrite: our first before→after example added a +52 country code to a phone number — which the executor doesn't actually do — so we cut it. The demo strip can only show what the engine truly does.

n was small and informal (~3 people we know), so this isn't a usability study. But you only need to watch one person mistake your tool for a delete button to learn the lesson: the people who most need the tool don't share your vocabulary, and the demo has to teach the concept before it can show the feature.

What we'd tell the next person

1. Planner/executor is the trust unlock. Auditability isn't a feature you add; it's a decomposition you choose.
2. Verify supervision by executing it. Every training example we kept provably recovers the clean table. Bad plans can't become labels.
3. Ground generation in references and budget for abstention. A small model that declines correctly beats a big model that guesses confidently.
4. Attack your own eval before reviewers do. Both of our metric bugs were found by ablations we almost didn't run.
5. Small models are enough more often than you think — and roughly $35 of GPU credit covers an embarrassing number of mistakes if each one teaches you something.
6. Test the framing on someone outside your vocabulary. The engine can be correct and the product still unusable if the first screen assumes a word — "cleaning" — that your user doesn't have. Show the concept before you name the feature.

— Built with a ≤4B planner, a 44M PII classifier, checksums, and a reference gazetteer. Total model weight: under 4.1B parameters. Total cloud spend: about $35.