web_docs/docs/development/cleanup-roadmap.md

metadata

title: Cleanup Roadmap

Cleanup Roadmap — Manager's Memory

This is the Manager surface for the monorepo refactor: the high-level map of what's done, what's next, and how to route work. It pairs with two other persistent stores — keep all three in sync, don't duplicate:

• CLAUDE.md (repo root) — the standing rules every session/sub-agent must obey.
• Claude Code memory (.claude/.../memory/) — cross-session facts; the canonical, detailed refactor recipe + progress lives in project_core_lib_refactor.md.
• This file — the living backlog + routing table for the cleanup.

Broader, whole-repo view (duplicated assets, stray entry points, orphaned stacks, packaging sprawl, usability) lives in the Refactor & Tech-Debt Plan. This file stays focused on the scripts/ → packages/ port (that plan's Theme 6).

Goal

Eliminate the top-level scripts/ tree. Move everything into packages/ refactored correctly as libraries (not just relocated). Any leftover scripts/ module is a porting candidate, not a permanent home.

Sub-agent routing (the specialists)

The Manager (interactive Claude) holds this roadmap and routes scoped work to isolated specialists in .claude/agents/. Each runs in a clean, narrow context and returns only a summary — no raw file dumps bleed back.

Specialist	Owns	Spin up for
python-packages-specialist	the packages/ uv workspace (accessibility, agents, core, core-lib, datamodels, ingestion, llm, scrapers)	"where should this Python live", porting scripts/ → packages/, library refactors. Enforces prefer packages, never add to scripts/
data-dbt-specialist	dbt_project/, scripts/datasources/*, scripts/enrichment*, scripts/discovery/	dbt models, SQL/JSONB transformation logic, data semantics
api-specialist	api/ (app, routes, models, auth, errors, batch_jobs)	FastAPI routes, Pydantic schemas, API DB access, OTel
frontend-specialist	frontend/src/, website/	React/TS components, hooks, API client, Tailwind, Docusaurus

Overlap note: Python library structure / where code lives → python-packages-specialist; dbt/SQL transformation semantics → data-dbt-specialist. A datasource port touches both — lead with python-packages for the move, pull in data-dbt for the SQL details.

Route by file scope. A task that crosses layers gets split: the specialist flags out-of-scope work in its summary and hands it back to the Manager to re-route.

The established port recipe (summary — full version in memory)

1. Branch off main: feat/datasource-<source>-port.
2. Two commits, never one: first a pure git mv legacy_loader.py <name>_pipeline.py (so git blame --follow survives), then a second commit that refactors contents.
3. Per script: define <Name>Row(RawRow) pydantic schema (Field max_length = bronze widths); implement <Source>Pipeline(DataSourcePipeline[<Name>Row]) with extract() (async stream of validated dicts) + load_batch() (parameterized text() UPSERT, JSONB via CAST(:col AS jsonb)); replace psycopg2 / hardcoded DATABASE_URL with core_lib.db (async_session, get_async_engine). Preserve pure helpers and UPSERT ON CONFLICT semantics verbatim. Keep --file/--limit/--truncate.
4. Unit tests in tests/test_<source>_<name>_pipeline.py (helpers, schema +/-, metadata, synthetic-file extract, error paths).
5. After porting, grep the whole repo for the OLD import path — exporters/QA/frontend-prep scripts silently still import it. Update or list in the PR body.
6. New workspace member ⇒ uv sync (or .venv/bin/pip install -e packages/<new>).
7. Triage before porting: many "still references old table" scripts are dead/superseded by dbt — grep usages + check for a dbt replacement before assuming a port; archive dead ones to archive/datasources/<source>/ via git mv.

Status (as of 2026-05-30)

Done / merged to main: core-lib framework + 6 ports (census/states, fec/contributions, gsa/domains, hifld/locations, dot/events, uscm/mayors). 16 branches consolidated → just main. packages/llm extracted (gemini + enrichment subpackages). Migration-048 cleanup swept refs to the dropped public.jurisdiction table (now public.civic_jurisdiction).

scripts/colab/ eliminated → packages/llm/src/llm/governance/ (2026-05-30): 24 live modules + notebook + README + mount_drive.sh + 2 CLIs moved via git mv (blame preserved); flat Colab imports rewritten to package-relative (from .x import …); dead colab_public_data.py

• colab_notebook_ui.py (+ its test) deleted. Notebook bootstrap now adds packages/llm/src to sys.path and imports llm.governance.*; CLIs run via python -m llm.governance.<cli>. Tests (test_colab_bootstrap, test_colab_runtime_phases, test_meeting_consolidated_summary, test_pipeline_media_scope) repointed.

scripts/utils/gdrive_paths.py → core_lib.gdrive_paths (2026-06-10): the residual governance cross-dep is gone. git mv-d into packages/core-lib/src/core_lib/ (blame preserved), chosen over packages/llm because it's a pure-stdlib path util shared by both llm.governance.* and scrapers.wikidata.export_bronze_to_json (and several scripts/) — core-lib is already a dependency of both, so no package gains a heavy llm dep. All importers repointed to core_lib.gdrive_paths; colab_bootstrap + the notebook §1 bootstrap now add packages/core-lib/src to sys.path; legacy scripts/discovery/* + scripts/utils/log_sync.py bootstraps add the same. Unit tests under packages/core-lib/tests/test_gdrive_paths.py.

scripts/ dead-code sweep + 5 ports (2026-06-10, branch feat/scripts-refactor-cleanup):

• Deleted provably-dead/archived code (16 files, zero importers): datasources/localview/archive/*, discovery/archive/*, datasources/osf/load_osf_rds_to_bronze.R, datasources/hifld/download_and_load_hifld.sh, datasources/voter_data/*, the one-off state-naming migrations (migrations/migrate_state_naming.py, fix_persons_scraped_jurisdiction_ids.sql, top-level migrate_all_*state_naming.py), and personal-machine Cursor scripts.
• Ported (git mv + all importers re-pointed + tests):
  • nonprofit-990 enrichers → scrapers.irs.{enrich_nonprofits_gt990,enrich_nonprofits_bigquery} (heavy deps boto3/bigquery/xmltodict made lazy; fixed broken subprocess paths in manage_nonprofits.py).
  • jurisdiction_id.py → core_lib.jurisdictions.jurisdiction_id (foundational, ~20 importers incl. api/batch_jobs); resolved an upward layering bug by relocating slug_snake_case down to core_lib.text.
  • jurisdiction-mapping analysis → ingestion.jurisdictions.mapping.* (5 modules + CLI).
  • reusable discovery modules → scrapers.discovery.* (8 from discovery/ + crawl_llm_sidecar from scraping/; ~70 importers re-pointed).
  • clean leaf scrapers → scrapers.discovery.social_media_discovery, scrapers.youtube.{scrape_youtube_channels, youtube_channel_enrich} (removed real packages→scripts violations).
  • jurisdiction_pilot leaf sub-web → scrapers.discovery.{http_fetch,mayor_url_discovery, county_municipality_websites,website_youtube_search} (clean of the hub).
• Deferred (complex, needs scoping): the jurisdiction_pilot hub — scrape_priority_states (~3500L) top-level imports ~10 scripts/discovery persist/orchestrator KEEP modules (bronze_*_persist, contact_directory_heuristics, contact_profile_images, jurisdiction_contact_seed_urls, …) plus ma_pilot.mayor_boost, and is itself a CLI pipeline (run_scrape_priority_states_*.sh). Porting it cleanly requires first porting those scripts/discovery persist modules (DB-writing — route via the data/ingestion lens); website_elections is deferred too (→ needs election_extract_from_html). The two big orchestrators (jurisdiction_discovery_pipeline, comprehensive_discovery_pipeline_jurisdiction) stay in scripts/ for now. The remaining jurisdiction_pilot internal helpers (vendor_detection, legistar_scraper, google_civic_youtube, website_civicplus_meetings, load_ocd_jurisdictions, debug_youtube_discovery) are used only by the hub — port them together with it.

In flight: feat/llm-enrichment-extraction — enrichment subpackage port.

Backlog (prioritized):

• Small/clean ports: nccs, naco, ballotpedia (measures), nces.
• Medium (multi-loader): census (acs, municipalities…), parcels, jurisdictions, openstates.
• Complex (need scoping, don't fit DataSourcePipeline cleanly): irs/load_irs_bmf.py, ballotpedia_integration.py (1570L), google_civic (1147L), wikidata (18 files), youtube (29 files).
• HTTP downloaders (BaseAsyncClient migration, not DataSourcePipeline): download_gsa_domains, download_hifld, download_state_dot_public_pages, load_fec_bulk.
• Skip (not pipelines): one-off SQL fixes, demos, helper modules, READMEs.

Remaining scripts/ subdirs still to triage: data, database, datasources, deployment, discovery, eboard, enrichment, enrichment_ai, examples, frontend, huggingface, jurisdictions, localview, maintenance, mcp, media, migrations, scraping, utils, wikicommons, wikimedia. (colab ✅ done → packages/llm/src/llm/governance/.)

Context hygiene (native, not hand-rolled)

Claude Code handles compaction and tool-result lifecycle automatically — don't build a message-pruning wrapper. When a unit of work finishes: record durable facts in memory, update this roadmap's Status, and start a fresh session for the next module.