docs/DATASETS.md

Dataset inventory — every source the system trains on, evaluates on, or must clean

Stage-3 consolidated registry (2026-06-11). Assignment discipline: a source is TRAIN, EVAL, or BENCH — never both sides of train/eval.

Paired dirty/clean (27 — eval/paired_bench.py → docs/PAIRED_BENCH.md)

source	origin	license	assignment	notes
hospital, beers, movies_1	Raha (BigDaMa)	Apache-2.0	TRAIN	champion mix since v6
flights, rayyan	Raha	Apache-2.0	EVAL (GEN)	held-out real errors
tax	Raha	Apache-2.0	unused	numeric-heavy, huge
ed2_restaurants	BigDaMa ED2	research	EVAL (GEN)	real NYC variants; errors past row 2k
fodors_zagats	Magellan EM	BSD-ish data	TRAIN	variant-masked EM table
dblp_acm, dblp_scholar	Magellan EM	research	BENCH only	out-of-regime (unique titles / convention-mismatch gold)
cleanml_company, cleanml_movie	CleanML	research	TRAIN	Company = org canon
gidcl_imdb	SICS-FRC GIDCL	none stated	TRAIN (v9+)	1M-row pair; 57k errors; subset 86k rows
zeroed_billionaire, zeroed_tax100k	WelkinNi/ZeroED	none stated	BENCH	injected; rich categoricals
dgov_* (5 tables)	LUH-DBS Matelda	Apache-2.0	BENCH	real data.gov tables, injected typos (6,692 more available)
tt_* (8 tables)	ToughTables 2T_WD	CC-BY-4.0	BENCH	gold-anchored entity misspellings, 370–33.5k corrections each

Wild messy tables (35 — eval/wild_bench.py → docs/WILD_BENCH.md)

24 portal tables (training/unpaired_sources.json cache: NYC/Chicago/SF/LA/Seattle/TX/WA portals, spotify, billboard, titanic, worldcities, airlines) + 12 stage-3 additions (training/harvest_wild.py): bx_books (mojibake), salary_survey, fec_indiv80 (PII, headerless), acnc_charities (AU), uk_price_paid (headerless UK), irs_eo1, glassdoor_jobs (multiline cells), paris_trees (FR), online_retail, bl_flickr_books, open_food_facts (211 cols), ct_real_estate. Backlog: CMS doctors (API 400), NHTSA FLAT_CMPL (multi-GB), Canada contracts (627MB).

Alias vocabularies (training generator material)

vocab	size	license	regime
toughtables_aliases	49,629	CC-BY-4.0	real entity misspellings (gold-anchored)
musicbrainz_hint_aliases	34,017	CC0	community-recorded artist misspellings
rxnorm_aliases	17,701	public domain	drug name synonyms
ror_aliases	73k orgs	CC0	research orgs
geonames_city_aliases	80k cities	CC-BY	city aliases
wikidata_company_aliases	10.2k	CC0	company aliases
onet_jobtitle_aliases	1,016	CC-BY-4.0	job titles
nickname_aliases	555	Apache-2.0	first names
openflights_airports	7,698	ODbL/DbCL	airports reference
libpostal_aliases	—	MIT	address abbreviations

Measured conclusions that govern future widening

1. Pre-paired corpus discovery is SATURATED (3 verified hunts) — synthesis from vocabularies is the widening path.
2. Pair volume / vocab training does NOT move held-out generalization (v7–v9, 4 retrains + tt-transfer test): the planner's value_counts cap (80) structurally hides high-cardinality dirty cells. The unlock is architectural: error-suspect / windowed profiling and cross-row entity voting.
3. The deterministic side (grounding + ops + verifier union) carries never-seen tables today; every op added from a measured regime (normalize_punctuation) moved GEN; convention/encoding ops are the cheapest remaining wins.