Spaces:
Running
Running
| # Dataset inventory — every source the system trains on, evaluates on, or must clean | |
| Stage-3 consolidated registry (2026-06-11). Assignment discipline: a source is | |
| TRAIN, EVAL, or BENCH — never both sides of train/eval. | |
| ## Paired dirty/clean (27 — eval/paired_bench.py → docs/PAIRED_BENCH.md) | |
| | source | origin | license | assignment | notes | | |
| |---|---|---|---|---| | |
| | hospital, beers, movies_1 | Raha (BigDaMa) | Apache-2.0 | TRAIN | champion mix since v6 | | |
| | flights, rayyan | Raha | Apache-2.0 | EVAL (GEN) | held-out real errors | | |
| | tax | Raha | Apache-2.0 | unused | numeric-heavy, huge | | |
| | ed2_restaurants | BigDaMa ED2 | research | EVAL (GEN) | real NYC variants; errors past row 2k | | |
| | fodors_zagats | Magellan EM | BSD-ish data | TRAIN | variant-masked EM table | | |
| | dblp_acm, dblp_scholar | Magellan EM | research | BENCH only | out-of-regime (unique titles / convention-mismatch gold) | | |
| | cleanml_company, cleanml_movie | CleanML | research | TRAIN | Company = org canon | | |
| | gidcl_imdb | SICS-FRC GIDCL | none stated | TRAIN (v9+) | 1M-row pair; 57k errors; subset 86k rows | | |
| | zeroed_billionaire, zeroed_tax100k | WelkinNi/ZeroED | none stated | BENCH | injected; rich categoricals | | |
| | dgov_* (5 tables) | LUH-DBS Matelda | Apache-2.0 | BENCH | real data.gov tables, injected typos (6,692 more available) | | |
| | tt_* (8 tables) | ToughTables 2T_WD | CC-BY-4.0 | BENCH | gold-anchored entity misspellings, 370–33.5k corrections each | | |
| ## Wild messy tables (35 — eval/wild_bench.py → docs/WILD_BENCH.md) | |
| 24 portal tables (training/unpaired_sources.json cache: NYC/Chicago/SF/LA/Seattle/TX/WA | |
| portals, spotify, billboard, titanic, worldcities, airlines) + 12 stage-3 additions | |
| (training/harvest_wild.py): bx_books (mojibake), salary_survey, fec_indiv80 (PII, | |
| headerless), acnc_charities (AU), uk_price_paid (headerless UK), irs_eo1, | |
| glassdoor_jobs (multiline cells), paris_trees (FR), online_retail, bl_flickr_books, | |
| open_food_facts (211 cols), ct_real_estate. Backlog: CMS doctors (API 400), NHTSA | |
| FLAT_CMPL (multi-GB), Canada contracts (627MB). | |
| ## Alias vocabularies (training generator material) | |
| | vocab | size | license | regime | | |
| |---|---|---|---| | |
| | toughtables_aliases | 49,629 | CC-BY-4.0 | real entity misspellings (gold-anchored) | | |
| | musicbrainz_hint_aliases | 34,017 | CC0 | community-recorded artist misspellings | | |
| | rxnorm_aliases | 17,701 | public domain | drug name synonyms | | |
| | ror_aliases | 73k orgs | CC0 | research orgs | | |
| | geonames_city_aliases | 80k cities | CC-BY | city aliases | | |
| | wikidata_company_aliases | 10.2k | CC0 | company aliases | | |
| | onet_jobtitle_aliases | 1,016 | CC-BY-4.0 | job titles | | |
| | nickname_aliases | 555 | Apache-2.0 | first names | | |
| | openflights_airports | 7,698 | ODbL/DbCL | airports reference | | |
| | libpostal_aliases | — | MIT | address abbreviations | | |
| ## Measured conclusions that govern future widening | |
| 1. Pre-paired corpus discovery is SATURATED (3 verified hunts) — synthesis from | |
| vocabularies is the widening path. | |
| 2. Pair volume / vocab training does NOT move held-out generalization (v7–v9, 4 | |
| retrains + tt-transfer test): the planner's value_counts cap (80) structurally | |
| hides high-cardinality dirty cells. The unlock is architectural: error-suspect / | |
| windowed profiling and cross-row entity voting. | |
| 3. The deterministic side (grounding + ops + verifier union) carries never-seen | |
| tables today; every op added from a measured regime (normalize_punctuation) | |
| moved GEN; convention/encoding ops are the cheapest remaining wins. | |