# Architecture

Snapshot of how the POTS Shutdown Tracker works end-to-end. Update this
file when behavior changes.

## High-level flow

The core path is crawl → parse → normalize → classify → review →
search → coverage → area-risk → UI. The standalone Mermaid source lives
in [docs/diagrams/data-flow.mmd](diagrams/data-flow.mmd).

```mermaid
flowchart LR
    source["Public source pages / PDFs"] --> crawler["connectors/<carrier>.py"]
    crawler --> raw["services/crawler.py<br/>fetch, dedupe, write RawNotice"]
    raw --> parse["services/parser.py + parsers/source_specific.py<br/>parse_notice()"]
    parse --> normalize["Normalize into RawNotice + NormalizedNotice graph"]
    normalize --> classify["services/review.py + services/att_mac_freeze.py<br/>classify and auto-dispose"]
    classify --> review["Review queue + admin actions"]
    review --> trust["Trust summary / queryable corpus gate"]
    trust --> search["services/search.py"]
    search --> coverage["services/coverage.py"]
    coverage --> area["services/area_risk.py"]
    area --> ui["API + UI"]
    normalize -. audit trail .-> audit["services/audit.py"]
    normalize -. embeddings .-> embeddings["services/embeddings.py"]
```

```mermaid
sequenceDiagram
    actor User
    participant UI as UI
    participant API as api/router.py
    participant Trust as services/trust.py
    participant Search as services/search.py

    User->>UI: Enter a search
    UI->>API: POST /search
    API->>Trust: require_queryable_corpus()
    Trust-->>API: is_queryable?
    alt corpus queryable
        API->>Search: execute search
        Search-->>API: matches + supporting notices
        API-->>UI: 200 results
    else corpus not queryable
        API-->>UI: 503 trust-gate refusal
    end
```

## Component responsibilities

### Ingestion

- **Connectors** (`connectors/<source>.py`) own source-specific URL
  discovery. They emit `DiscoveredDocument` objects with a URL, title
  hints, and optional pre-fetched raw text or bytes.
- **Crawler** (`services/crawler.py`) fetches with
  `services/connectors/common.py`, respects per-source rate budgets,
  and writes `RawNotice` rows. Content-hash dedupe: identical bytes are
  skipped but `last_seen_at` is advanced (see
  [IMPROVEMENT_PLAN.md](../IMPROVEMENT_PLAN.md) P2-T1).
- **Storage backend** (`services/storage.py`) is pluggable:
  filesystem (local/dev) or Hugging Face Dataset (hosted). Dataset repo
  must be private (enforced by P2-T2).

### Parsing and normalization

- **Rule parser** (`parsers/rule_parser.py`) is the source of truth for
  `notice_type`, `rule_family`, `restriction_types`, and default
  `signal_family="shutdown"`.
- **Source-specific overrides** (`parsers/source_specific.py`) apply per
  carrier. AT&T has the deepest overrides (DSA tracker, network
  disclosure, tariff/MAC Freeze paths). Overrides run before the
  generic parser and can be overridden back to `shutdown` only when the
  MAC-Freeze guard's dual-signal rule is satisfied (see P1-T5).
- **Active window policy** (`utils/policy.py`) decides `is_active` from
  `issue_date`, `revised_date`, `target_date`, `fetched_at`, and the
  configured lookback months. Default lookback is 60 months. Shutdown
  notices with `target_date <= today` stay active permanently; other
  signal families still respect the post-target grace window from P2-T5.

### User-facing APIs

- **Trust summary** (`services/trust.py`) reports corpus health,
  `is_queryable`, covered carriers, recently announced and updated
  notices. This is the single source of truth for whether the app
  should answer user queries.
- **Search** (`services/search.py`) mixes structured filters and vector
  ranking; respects `signal_family` (defaults to `shutdown`).
- **Coverage** (`services/coverage.py`) resolves city/state/ZIP against
  structured `ImpactedAddress` and `NoticeLocation`, then falls back to
  text-city geography with address/contact suppression heuristics.
  Priority: `impacted_address (4) > notice_location (3) >
  text_city_geography (2) > structured_state (1) > parsed_state_fallback (1)`.
- **Area risk** (`services/area_risk.py`) builds a grade bubble per
  area using a five-tier model (`green/yellow/blue/orange/red`), plus a
  per-carrier breakdown, caveats, and nearby-municipality context.
  Past-target shutdown notices are permanent red evidence; MAC Freeze
  alone is blue; scheduled shutdowns split at the 12-month boundary;
  nearby municipalities can nudge a weak direct result up to yellow but
  never above it. Historical context is now folded into the main
  supporting-notices list instead of a separate channel.
  State-level MAC Freeze tariff filings surface as blue for a city
  search when no shutdown evidence conflicts. Confidence is `low` and
  the caveat explicitly attributes the signal to the state-wide tariff
  rather than a city-level filing.
- **Matching** (`services/matching.py`) returns precise matches
  (exact_address, fuzzy_address, wire_center, CLLI) and, post P1-T6, a
  separate `area_awareness` list of state-level hits with no risk
  level.
- **MAC Freeze assessment** (`services/att_mac_freeze.py`) is a
  dedicated endpoint when `signal_family` is `att_mac_freeze` or `all`.
  It never reaches shutdown grade bubbles.
- **Signal links** (`services/signal_links.py`) crosses MAC Freeze to
  shutdown only by geography + carrier-root match.
- **Bulk lookup** (`services/bulk_lookup.py`) accepts xlsx uploads at
  `/bulk-lookup/jobs`, validates city/state rows, queues background
  processing through a `ThreadPoolExecutor` on `app.state`, calls
  `assess_area_risk` per valid row, and writes an enriched xlsx with a
  `Summary` sheet back to the shared storage backend. The job list,
  detail, and download endpoints are trust-gated with
  `require_queryable_corpus`; uploads are retained for 7 days by
  default and swept by the scheduler/CLI cleanup path.

### Admin workflows

- **Review queue** surfaces low-confidence or structurally incomplete
  notices. Auto-disposition is opt-in and dry-run by default.
- **Reparse / reapply-active-window / sanitize-notice-dates /
  rebuild-embeddings** are triggerable via `/admin/*` or CLI. All admin
  mutations require the `X-Admin-Key` header (P1-T3) and write to
  `AdminAuditEvent` (P3-T5).

## Frontend responsibilities

- **Audience split** is Vite-time (`VITE_POTS_TRACKER_AUDIENCE`).
  Consumer sees Search + trust gate. Internal sees Search, Dashboard,
  Match, Admin, Notice Detail. The audience split is UI-only; server
  auth is the real guardrail for admin.
- **Trust gate** blocks Search and (post P1-T4) Match when
  `is_queryable=false`. Bulk lookup uses the same trust gate for upload,
  job list/detail, and download.
- **Area-risk UI** clearly labels evidence scope (city vs state), match
  source (impacted address vs text geography), and caveats.
- **MAC Freeze** renders in its own panel with explicit "not by itself a
  shutdown" copy.
- **Bulk lookup UI** exposes `/bulk-lookup` for both internal and
  consumer audiences. It uploads xlsx files, polls queued/running jobs,
  renders color counts and top carriers, and links to the enriched
  workbook download when complete.

### Brand and visual system

- The SPA uses the Masters Telecom palette: Navy `#1A2B62`,
  Adriatic Blue `#4194BC`, Light Blue `#A0C6DB`, Grey `#808085`,
  and White `#FFFFFF`.
- Brand colors are chrome only: header, links, primary buttons, subtle
  info surfaces, and borders.
- Tier colors stay semantic and visually distinct from the brand blues:
  Red `#B42318`, Orange `#DC6803`, Yellow `#CA8504`, Violet
  `#6B5CA5`, and Green `#079455`.
- The backend `grade_bucket="blue"` contract is unchanged, but the UI
  renders that tier as Violet so MAC-Freeze does not read as generic
  info next to Navy/Adriatic chrome.
- User-facing blue-tier copy keeps the label "Future Shutdown Notice
  Anticipated"; only color-word references say Violet.

## Hosted deployment shape

- Docker image (see `Dockerfile`) builds the frontend, then copies the
  dist into a Python runtime that serves both the SPA and the API on
  port 7860.
- Hugging Face Space executes the image. Optional storage backend is a
  private HF Dataset repo.
- Neon Postgres holds the corpus. `vector` extension required.
- APScheduler runs weekly. In multi-instance deployments the scheduler
  acquires a PG advisory lock before firing (P2-T3).
- APScheduler also runs daily bulk-lookup retention cleanup at 03:00
  with `misfire_grace_time=300` and `coalesce=True`.

## Bulk lookup

Bulk lookup is a trust-gated user-facing workflow for operators who need
to enrich a spreadsheet of locations instead of checking one area at a
time.

- `POST /bulk-lookup/jobs` accepts one `.xlsx` file with city/state
  columns, validates headers and row limits synchronously, stores the
  input blob, creates a `BulkLookupJob`, and queues background
  processing.
- `GET /bulk-lookup/jobs` lists the 20 most recent jobs.
- `GET /bulk-lookup/jobs/{id}` returns status, row counts, color counts,
  top carriers, and download availability.
- `GET /bulk-lookup/jobs/{id}/download` streams the completed enriched
  workbook and returns `409` until processing is complete or `410` after
  expiry.

Processing uses `ThreadPoolExecutor(max_workers=settings.bulk_lookup_concurrent_workers)`
on `app.state`. Each worker opens its own SQLAlchemy session, reads the
input from the configured storage backend, calls `assess_area_risk` for
each valid row, flags invalid rows without failing the whole job, and
writes a Results sheet plus Summary sheet. The executor is shut down in
the FastAPI lifespan `finally` block.

Retention defaults to 7 days. The scheduler cleanup job and
`python app/cli.py cleanup-bulk-lookup-jobs` both call the same cleanup
service to expire rows and delete stored input/output blobs.

## Active window

A notice is `is_active=True` when:

- Its latest of `issue_date`, `revised_date`, `target_date`, or
  `fetched_at` falls within `POTS_TRACKER_LOOKBACK_MONTHS` (default 60).
- Non-shutdown notices still respect the post-target grace window from
  P2-T5: `target_date is None` or
  `target_date + active_window_post_target_grace_days >= today`.
- Shutdown notices with `target_date <= today` remain active
  permanently so they stay available as historical red evidence for
  area-risk grading.

Archived notices can still exist in the corpus, but they no longer flow
through a separate historical-context channel.

## Signal families

| Value            | Meaning                                         |
|------------------|-------------------------------------------------|
| `shutdown`       | Analog / POTS / copper retirement, discontinuance, switch decommissioning, TDM→IP transition, network disclosure. Default. |
| `att_mac_freeze` | AT&T commercial availability restriction (no new orders / moves / adds / changes, grandfathering). Not a shutdown by itself. |

Queries default to `shutdown`. The UI never cross-mingles the two.

## Storage model

See `backend/app/pots_shutdown_tracker/models/entities.py` for the full
schema. Key relationships:

- `RawNotice 1 ←→ 1 NormalizedNotice`
- `NormalizedNotice 1 ←→ N NoticeLocation`
- `NormalizedNotice 1 ←→ N ImpactedAddress`
- `NormalizedNotice 1 ←→ N NoticeChunk`
- `NormalizedNotice 1 ←→ N NoticeAuditEvent`
- `CustomerMatch` is a persisted record of matching results.
- `AIJob` records every AI prompt/response for audit.
- `BulkLookupJob` tracks uploaded xlsx jobs, blob references, row
  counts, color counts, top carriers, and retention expiry.

## See also

- [docs/ONBOARDING.md](ONBOARDING.md) — first-day setup.
- [docs/TESTING.md](TESTING.md) — test strategy and fixtures.
- [docs/SECURITY.md](SECURITY.md) — threat model and auth posture.
- [docs/diagrams/data-flow.mmd](diagrams/data-flow.mmd) — standalone Mermaid source for the architecture diagrams.
- [docs/RUNBOOK.md](RUNBOOK.md) — operational playbook (create per P4-T2).
- [IMPROVEMENT_PLAN.md](../IMPROVEMENT_PLAN.md) — prioritized work list.