CLAUDE.md

Monorepo Architecture & Rules — Open Navigator

Tech Stack

• Backend: FastAPI (Python 3.11+)
• Data: dbt Core + SQL (PostgreSQL — local warehouse on localhost:5433)
• Frontend: React (Vite, TypeScript); Docusaurus for documentation
• Tooling: uv for Python dependency management & the workspace (NOT Poetry); Ruff for lint/format; Node.js for the frontend

Repository Layout

• api/: FastAPI application — entry points api/main.py / api/app.py, route handlers in api/routes/, Pydantic models in api/models.py.
• web_app/: React + Vite + TypeScript app (port 5173).
• web_docs/: Docusaurus documentation site (port 3000).
• dbt_project/: dbt models, macros, and schema.yml files. Standalone uv project (its protobuf/pathspec pins conflict with the main resolution). Medallion: bronze → staging → intermediate → marts.
• packages/: internal shared Python libraries — the uv workspace (packages/*): core, core-lib, datamodels, ingestion, scrapers, llm, agents, accessibility. This is the destination for the scripts/ → packages/ refactor.
• scripts/: LEGACY top-level scripts being ported into packages/. Do not add new code here — port instead (see Refactor Workflow).

Note: apps/ (FastAPI, web) and services/ are planned for a later migration phase per pyproject.toml; today the API lives in api/ and the web app in web_app/.

Where New Code Goes — scripts/ Is Being Retired (CRITICAL)

• scripts/ is frozen. We are actively retiring the top-level scripts/ tree into packages/. Treat it as legacy: read it, port from it, but never add to it.
• All new Python features go in packages/ as a proper importable library module (under the relevant packages/<lib>/src/<lib>/…), with a real module path, not a loose top-level script.
• New runnable entry points belong in a package as a CLI module invoked with python -m <lib>.<module> (argparse main() + if __name__ == "__main__"), not as a new file in scripts/.
• Do not even suggest creating a new scripts/ file or a "scripts/-style runner." If a one-off runner is needed, propose it as a package CLI module instead.
• When a task would naturally extend a scripts/ file, port the needed piece into the appropriate package first, then build on the package version. Route such work to the python-packages-specialist sub-agent, which enforces this rule.

Running Locally — Three Services

1. Documentation (Docusaurus) — port 3000
2. Main Application (React + Vite) — port 5173
3. API Backend (FastAPI) — port 8001

• Launch command: ./start-all.sh

Explicit Development Guidelines

• CRITICAL: Never refactor a shared Python library in packages/ without running its tests first. A library change can ripple into both the API (api/) and the ingestion/dbt-adjacent loaders — run pytest for the touched package and its dependents before committing.
• Do not read large raw or mock data files directly (e.g. analyze.log, parquet dumps, data/cache/ contents). Refer to schemas, the Pydantic models in packages/datamodels, or dbt schema.yml / TypeScript type definitions instead.
• When refactoring dbt models, always verify downstream dependencies via the dbt DAG (dbt ls --select <model>+, or the docs graph) before changing them.
• New Python belongs in packages/ as a proper library — never extend scripts/ in place.

Refactor Workflow

• Roadmap / Manager memory: web_docs/docs/development/cleanup-roadmap.md — the living backlog + status for the scripts/ → packages/ library refactor. Read it before starting cleanup work.
• Specialist sub-agents (in .claude/agents/): route scoped work to python-packages-specialist (Python libraries in packages/; enforces prefer-packages / never-add-to-scripts/), data-dbt-specialist (dbt/SQL), api-specialist (FastAPI), or frontend-specialist (React/Docusaurus). Cross-layer tasks get split across them.

Data Pipeline Standards (CRITICAL)

• Transformations: ALWAYS use dbt. No Python for SQL logic or JSONB extraction.
• Python: Use only for ingestion (API calls, scraping), ML, or orchestration.
• Naming:
  • state_code (2-letter) vs state (full name). Include BOTH.
  • website_url is the primary web column name.
  • Do NOT use dimensional dim_/fact_ (dimension/fact) names — do not recommend or apply star-schema dim/fact naming to models or tables. Name models by the entity they represent (e.g. jurisdictions, event_*).
• Keys: ALWAYS define an explicit primary key, and foreign keys for every relationship, on tables/models exposed in the public schema (declare via dbt constraints / schema.yml so they are enforced in Postgres).
• Scripts: Data loading scripts in scripts/datasources/ must start with load_.

No Fabricated Data (CRITICAL)

• NEVER display fabricated, dummy, placeholder, mocked, or hard-coded "example" numbers or data to the user — not in the UI, API responses, charts, docs, or summaries. Every figure shown must trace to a real value from the warehouse/source.
• This applies especially to financial and civic figures (budgets, dollar amounts, contributions, "Follow the money" / "Money Moves" lenses, vote counts, statistics). A made-up dollar amount is worse than showing nothing.
• If real data is missing, empty, or not yet ingested, show an explicit empty/unavailable state (e.g. "No data available", a disabled card, null/—) — do not invent stand-in numbers to fill the gap or make a demo "look complete."
• Do not seed components, fixtures-as-defaults, or fallback constants with realistic-looking numbers. Test fixtures stay in tests; never let them leak into a served code path.
• When unsure whether a value is real, treat it as unavailable and surface the gap rather than guessing.

Database Access

• Host: localhost:5433 (ALREADY RUNNING — do not suggest new Docker PG instances).
• Databases: open_navigator (primary) and openstates (source).
• API Access: Use the public schema in open_navigator. Avoid direct bronze access.
• Keys: Every table/model exposed in public MUST declare an explicit primary key and foreign keys for all relationships (enforced via dbt constraints / schema.yml).
• CAUTION: Never delete or suggest deleting data/cache/.

Documentation Rules (Docusaurus)

• MANDATORY: ALL docs go in web_docs/docs/ subdirectories.
• Formatting: kebab-case filenames, YAML frontmatter included, lowercase only.
• Root: No .md files in root except README, LICENSE, and CONTRIBUTING.

Frontend UX — Scope Label Must Match the Active Filter (MANDATORY, SITE-WIDE)

This rule governs every scoped/browse/list/search surface in web_app/ — not just Browse Causes.

• The visible scope label is the single source of truth for what the data is filtered to, and it MUST match the active filter exactly. Whatever filter is in effect, the label next to the page/section title states it precisely:
  • National / no geo filter → show National (or no place qualifier), never a stale city/state.
  • State filter → show the state (e.g. Browse Causes · Alabama / All of AL).
  • City / jurisdiction filter → show the city/jurisdiction (e.g. Browse Causes · Tuscaloosa).
  • Same idea for non-geo dimensions (topic, cause, date range, entity type, source): the label names the actual scope in effect.
• Label and data move together — atomically. Changing the filter must update the label, the underlying query, and the rendered results in lockstep. A label that says one scope while the data reflects another (or vice-versa) is a bug. The label is never decorative: it always corresponds to a real, applied filter (see No Fabricated Data — don't show a "Tuscaloosa" chip over national data).
• Filters carry over across navigation. A scope the user picks travels to every page where it applies (Browse Topics/Causes/Questions, Search, decision/meeting lists, maps…). Never silently drop or reset a filter on navigation; if a destination genuinely can't honor it (data isn't at that grain), surface that explicitly rather than pretending it applied.
• Carry scope via the URL (e.g. ?state=AL&city=Tuscaloosa, ?scope=national) so it survives refresh / deep-link / back-button and is the authority both the label and the API query read from. The entry point appends the params; the destination reads them (useSearchParams) and renders the matching label.
• Let the user see and change the active scope in place — e.g. the 📍 Tuscaloosa / All of AL toggle — and broaden up the hierarchy (city → state → national) without losing context.
• Canonical reference: web_app/src/pages/BrowseTopics.tsx and BrowseCauses.tsx — read ?state=&city=, render the scope label + a one-click broaden control, and forward the scope into DecisionCardList and the API. New scoped surfaces must follow this pattern.

Frontend UX — Show the Match Evidence on Filtered Tiles (MANDATORY, SITE-WIDE)

When a result tile/card is shown because of a topic, keyword, cause, question, or any other content filter, the tile MUST show the evidence for why it matched — a real quote/passage from the underlying record — not just a title, badge, and date. A user looking at "36 results for 'fluoride'" must be able to see where fluoride appears in each result without clicking in.

• Show the matched passage, quoted. Surface the actual text from the transcript, decision statement, summary, agenda/minutes, or bill that contains the filter term. It must be a verbatim excerpt of the real record — never paraphrased, summarized, or fabricated (see No Fabricated Data).
• Highlight the matched terms. The matched keyword(s) within the excerpt are visibly marked so the association is unmistakable (the convention here is server-side ts_headline(... StartSel=<mark>, StopSel=</mark>) → the highlightSnippet() helper renders <mark> segments React-escaped, never dangerouslySetInnerHTML).
• Backend produces the evidence; the tile renders it. The search/filter SQL emits the highlighted snippet (e.g. ts_headline over the searchable text), returned in the result's description. The card component renders it under the title (e.g. StoryCard's excerpt, clamped so it can't blow out tile height). Every result type the filter can return (meetings, transcripts/documents, decisions, bills, …) carries its own snippet.
• No filter term in the record ⇒ it should not be a result. If you cannot produce a real matched passage for a tile under an active content filter, that is a signal the match is spurious (or matched only on metadata) — surface that honestly; do not invent a quote to justify the tile.
• Plain browse (no content filter) is exempt. With no topic/keyword filter in effect there's nothing to "highlight"; a contextual lead-in (e.g. the start of the decision statement) is fine. The requirement applies whenever a content filter is what produced the result set.
• Canonical reference: api/routes/search_postgres.py (ts_headline snippet → SearchResult.description for the meeting, document, and decision legs) feeding web_app/src/pages/UnifiedSearch.tsx (toStoryCard / toTranscriptCard → StoryCard.excerpt via highlightSnippet). New filtered surfaces must follow this pattern.

Code Style

• Python: Type hints, PEP 8, pathlib.
• React: Functional components, TypeScript interfaces, Tailwind CSS.
• dbt: Use Medallion architecture (bronze -> staging -> intermediate -> marts).

Git Commit Standards (MANDATORY)

• ALWAYS use Conventional Commits for ALL commit messages.
• Format: <type>(<scope>): <description>
• Types: feat, fix, chore, docs, refactor, test, perf, ci, build, revert
• Examples:
  • feat(api): add jurisdiction search endpoint
  • fix(bronze): handle missing state_code in census loader
  • chore(deps): upgrade loguru to 0.7.3
  • docs(web_docs): add FastAPI deployment guide

Branch & PR Workflow (MANDATORY)

• NEVER push directly to main. main is branch-protected on GitHub (getcommunityone/open-navigator): direct pushes are rejected. All changes land via pull request.
• Every change goes through a PR. Branch off the latest main, commit there, push the branch, and open a PR against main:

  git checkout main && git pull
  git checkout -b <type>/<short-topic>        # e.g. feat/money-flow-sankey
  # ... commit work (Conventional Commits) ...
  git push -u origin <type>/<short-topic>
  gh pr create --base main
  

• A PR must be green before merge. Required CI checks (Frontend Build, Documentation Build, Backend Tests, API Types) must pass; resolve all conversations. The Docker Build Test is not required (it self-skips when no Docker files change).
• Do NOT merge your own work silently. Prefer review. The solo maintainer may self-merge as admin (gh pr merge <n> --squash --admin) only because GitHub forbids self-approval; this is a stopgap, not the norm — once a second reviewer exists, require the approval.
• Never rewrite or force-push shared history (main, or any branch with an open PR). A parallel session may be committing alongside you: stage only your own files, and verify your work landed via git log rather than amending.
• Agents/automation must follow the same flow — branch + PR, never a direct push to main.

Git Commit Standards (MANDATORY)

Simple Python Scripts & Packages → Loguru

Use loguru for all standalone scripts and simple Python packages:

from loguru import logger

logger.info("Loading data from {}", source)
logger.success("Loaded {:,} rows", count)
logger.warning("Missing field: {}", field)
logger.error("Failed to connect: {}", err)

• Import only from loguru import logger — no manual handler setup needed for scripts.
• Use logger.success() to signal a completed step.
• For scripts that write log files, follow the pattern in scripts/load_bronze.py (sink to timestamped file + scripts/utils/log_sync.py for upload).

FastAPI → OpenTelemetry

Use OpenTelemetry for all FastAPI services:

from opentelemetry import trace
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor

FastAPIInstrumentor.instrument_app(app)
tracer = trace.get_tracer(__name__)

with tracer.start_as_current_span("operation-name") as span:
    span.set_attribute("key", value)

• Instrument at app startup via FastAPIInstrumentor.
• Use spans for discrete operations (DB queries, external calls, enrichment steps).
• Export to OTLP collector; fall back to console exporter in development.

React → OpenTelemetry

Use OpenTelemetry for frontend observability:

import { trace } from '@opentelemetry/api';

const tracer = trace.getTracer('open-navigator-frontend');
const span = tracer.startSpan('fetch-jurisdictions');
// ... operation ...
span.end();

• Initialize the Web SDK once in src/instrumentation.ts, imported at the app entry point.
• Use @opentelemetry/sdk-trace-web + @opentelemetry/exporter-trace-otlp-http.
• Instrument route changes and key user interactions (search, filter, data load).

Calendar Years — Storage vs Serialization

The rule splits by layer; do not conflate them:

• SQL storage (columns): a bare calendar year is an integer (smallint is fine) — never text/varchar/double precision/numeric. Integer keeps range filters (WHERE year >= 2020) and numeric sort correct. Bronze may keep the source-native type for raw fidelity, but cast to integer by the staging layer so intermediate/marts/public are uniform.
• Real dates: when you have a full date, use a date/timestamp column — not a year column.
• JSON / API / manifests (the wire): serialize a bare year as a string (e.g. "year": "2026"), not a number — JSON numbers get locale-formatted (e.g. 2,024) by UI clients. Convert at the JSON boundary: str(y) in Python, ::text in SQL/jsonb_build_object. A real DATE/TIMESTAMP already serializes as an ISO string, so this exception does not apply to it.
• calendar_year_label() (scripts/utils/calendar_year_util.py) is the canonical Python helper for the wire/string form — it normalizes any value to a clean 4-digit string or None. Use it at serialization / serving-table boundaries, not to define an integer storage column. Bronze raw loaders that land source-native VARCHAR(4) years via this helper are fine (raw fidelity); staging still casts to integer.
• Internal paths may still use numeric years for folders; convert with str(y) at the JSON boundary.
• Migration: python scripts/discovery/fix_scraped_meetings_manifest_years.py (see --dry-run).