# SWIA Commission Agreement Intake — Operations & Architecture Runbook

> Internal portal for capturing, approving, and finalizing producer commission
> agreements. FastAPI + SQLite + Jinja, deployed as a Docker Space on Hugging
> Face at **caf.swia.ai** (`pramodmisra/pif`). This doc is the single source of
> truth for how data persists, where each list comes from, and how to operate it.
> Last major update: 2026-06-19 (Rounds 6 & 7 + email-rule route-ordering fix).

---

## 1. Deployment & the two git remotes

The repo has **two remotes on `main`** — keep them in lockstep:

| Remote | URL | Role |
|--------|-----|------|
| `github` | github.com/pramodmisra/pif-app | Canonical source of truth (also runs the daily refresh Action) |
| `origin` | huggingface.co/spaces/pramodmisra/pif | The live Space; pushing here triggers a rebuild/deploy |

**Always `git push github main && git push origin main`.** They have silently
drifted before. If `origin` rejects (non-fast-forward), `git fetch origin`,
inspect `git log HEAD..origin/main`, and `git merge origin/main -X ours` rather
than force-pushing (force-push drops the Space's commit ancestry).

A push to `origin` triggers a **factory rebuild** (fresh container). A config
change (secrets/volumes) or `restart_space()` triggers a restart. A push to
`origin` while RUNNING rebuilds with **no downtime** (old container keeps serving
through `RUNNING_BUILDING` → `RUNNING_APP_STARTING` → `RUNNING`).

**Uptime self-heal (GitHub Action `uptime-check.yml`):** a cron every 15 min runs
`scripts/uptime_check.py`, which reads the Space runtime stage and issues a
**non-destructive plain restart** if it has fallen into a recoverable error
(`RUNTIME_ERROR`/`PAUSED`/`STOPPED`/`SLEEPING`). It does **not** restart on
`BUILD_ERROR`/`CONFIG_ERROR`/`NO_APP_FILE` (a code problem a restart can't fix) —
it fails the job so GitHub emails the owner instead of restart-looping. Reuses the
same `HF_TOKEN` secret as the daily refresh. Manual run: Actions tab → *Uptime
Self-Heal* → *Run workflow*.

---

## 2. Persistent storage (CRITICAL)

The DB is SQLite at `DATABASE_URL=sqlite:////data/app.db`. `/data` is **only
persistent because a Storage Bucket is mounted there**:

- Bucket **`pramodmisra/pif-db`** (private) is mounted **read-write at `/data`**.
- HF's old flat "$5/mo storage tier" API is retired (`request_space_storage` 404s).
  Current model = **buckets + volumes**:
  ```python
  api.create_bucket("pramodmisra/pif-db", private=True)
  api.set_space_volumes("pramodmisra/pif",
      volumes=[Volume(type="bucket", source="pramodmisra/pif-db",
                      mount_path="/data", read_only=False)])
  api.restart_space("pramodmisra/pif", factory_reboot=True)
  ```
- SQLite-on-bucket is supported by HF (same pattern as Label Studio Spaces).
- `huggingface_hub.SpaceRuntime.volumes` may show `None` even when mounted —
  confirm via raw API `GET /api/spaces/{id}?full=true` → `runtime.volumes`, or
  `api.list_bucket_tree("pramodmisra/pif-db")` (you'll see `app.db`).

**⚠️ NEVER detach this bucket / remove the volume — the entire user database
(logins, password resets, CE edits) lives there.** Without it, `/data` is
ephemeral and every rebuild wipes the DB back to the seed.

---

## 3. Data flow: what populates each list

```
Google Sheet (clients)  ─┐
                         ├─►  build_refresh_data.py  ─►  refresh_data.json  ─►  HF dataset pif-data
producers_seed.json  ────┘        (CI, daily 08:00 UTC)     (gitignored)         (private)
                                                                                      │
                                                                          start.py pulls at boot
                                                                                      ▼
                                                              Producer + Client tables in /data/app.db
```

### Producers — `producers_seed.json` (TRACKED) is the source of truth
- **Why a seed file:** `refresh_data.json` is **gitignored** (it carries 18,656
  client records). On a fresh CI checkout it's absent, so the daily job used to
  emit an **empty** producer list → every reboot collapsed producers to the 11
  seeded ones. Fixed 2026-05-29 by adding the tracked `producers_seed.json`.
- `build_refresh_data.py` → `load_existing_producers()` reads `producers_seed.json`
  first (always present in CI), falling back to `refresh_data.json`.
- Each row: `{code, prefix, suffix, name, email?}`. Built from
  `producers_codes_emails.csv` (the canonical roster, 35 people / 71 codes).
- **To change the producer roster:** edit `producers_seed.json` (or regenerate
  from the CSV), commit, push both remotes, then run the refresh (Section 6).
- **`Producer.is_active` is ADMIN-OWNED — the refresh no longer resets it.**
  Until 2026-06-23, both `start.py`'s boot import and `/api/data-refresh` did
  *"deactivate ALL producers, then re-activate every code in the refresh"* — so
  the nightly Space restart re-activated the whole 71-code roster and wiped any
  deactivations made in the admin panel (the "57 reset back to 74 overnight"
  bug). Now the refresh **upserts only**: existing producers get name/email/code
  updates but keep their current `is_active`; brand-new roster codes are added
  active (deactivate once and it sticks). Standing approvers (Bella `SANBE1` +
  the 12 locked-pair/Fitch codes) are still force-activated. To trim the visible
  producer list, **deactivate in the admin panel** — it now survives every
  refresh. Regression guard: `_smoke_producer_persist.py`.

### Producer emails — real values from the roster
- `start.py`'s import (and the admin `/api/data-refresh` upload) set
  `Producer.email` from the JSON when present. Real emails (e.g. Kathryn
  Faulkner = `kdye@`, Michael Banner = `mbenner@`, Banks Snellings =
  `banks.snellings@`) win over the auto-generated `first-initial+lastname@` rule.
- Producers with a blank email in the CSV (Richard S. Brewer, UAC Risk
  Management) fall back to the auto-generated address.

### Clients — Google Sheet (PowerApps-synced)
- Sheet `1RYmxN855…` tab `gid=1047572149`, columns `LookupCode, Name`.
- `build_refresh_data.py` fetches the public CSV export (guard: ≥15,000 rows),
  dedups by code → ~18,656 clients. Producers are **not** in the Sheet.

### Client Executives (CE) — separate `client_executives` table
- CEs are an **independent list** (NOT the Producer table). Seeded once from
  `ce_seed.json` (7 SWIA CEs) when the table is empty; managed via
  **Admin → Manage Client Executives** (add/edit/toggle/delete + bulk paste).
- The form's CE picker, name resolution, and approval email come from this table
  first, with a fallback to the Producer lookup for legacy/edge cases.

---

## 4. Authentication & user lifecycle

- **Self-signup** (`/signup`): any `@snellingswalters.com` email → active `user`,
  logged in immediately (no email verification, no admin approval).
- **Password reset**: self-service `/forgot-password` → emailed token (1 h);
  superadmin can also trigger a reset from the User Management panel.
- **Persistence**: users live in `/data/app.db` (persistent bucket). `start.py`
  skips seeding when any user exists, so registrations/edits survive rebuilds.
- **Seeded users**: only the two permanent superadmins (System Admin + Pramod)
  are in `seed_db.py`, so a fresh DB never resurrects deleted people.
- **Removal**: superadmin-only Delete — blocked while active and blocked if the
  user has submissions (deactivate-don't-delete preserves the audit trail).

---

## 5. The client lookup (form performance)

The intake form must **NOT** pre-render all 18,656 clients as `<option>`s — that
produced a 6.3 MB page that broke the dropdown. Current design (`form.html`):
- Both client dropdowns start **empty**; the **Filter by Code** and **Filter by
  Name** boxes fill them on type (client-side `startsWith` over the `clientData`
  array). Page is ~1.1 MB.
- `ensureSelected()` re-adds the chosen option for code↔name sync and draft restore.
- **Do not** re-add `{% for c in clients %}` loops to those selects. If the client
  list grows much larger, move to a server-side `/api/clients/search` endpoint
  instead of embedding `clientData`.

---

## 6. Common operations

**Refresh clients + producers now (manual):**
```bash
python scripts/build_refresh_data.py          # builds refresh_data.json
HF_TOKEN=<write-token> python scripts/upload_to_hf.py   # uploads to dataset + restarts Space
```
The daily GitHub Action `daily-client-refresh.yml` (08:00 UTC) does the same.

**Change the producer roster / emails:** edit `producers_seed.json` → commit →
push both remotes → run the refresh above (or wait for the daily Action) →
the Space re-imports on restart.

**Add / edit a Client Executive:** Admin → Manage Client Executives (persists in
the DB). To change the initial seed, edit `ce_seed.json` (only seeds an empty table).

**Check the live Space:**
```python
from huggingface_hub import HfApi
api = HfApi(token=TOKEN)
api.get_space_runtime("pramodmisra/pif").stage     # RUNNING / BUILDING / *_ERROR
list(api.list_bucket_tree("pramodmisra/pif-db"))   # confirms app.db in persistent storage
```

**Site is down / 503 / RUNTIME_ERROR ("Scheduling failure: unable to schedule"):**
This is an HF **infra-side** failure to place the container on free CPU hardware —
NOT a code/git/storage problem. The fix is a plain, non-destructive restart (the
bucket persists across both restart *and* factory reboot, so the DB is safe):
```python
api.restart_space("pramodmisra/pif")               # re-requests scheduling
# poll api.get_space_runtime(...).stage until RUNNING (usually ~30s)
```
The `uptime-check.yml` Action does this automatically within ~15 min. If a plain
restart still errors, escalate to `restart_space(..., factory_reboot=True)`. If
"Scheduling failure" persists after both, it's HF capacity — wait/retry, or move
off the free tier to paid `cpu-upgrade` (dedicated hardware; needs billing
approval). **First confirm storage is intact** (`app.db` ≈ 1.34 MB in
`list_bucket_tree`) — a byte-identical `app.db` proves no data/credentials/logins
were touched; the restart swaps the container, not the data.

---

## 7. Gotchas (hard-won)

- `refresh_data.json` is **gitignored** → never rely on it existing in CI; producers
  come from `producers_seed.json`.
- HF run-log fetch only returns the recent tail (uvicorn lines); `start.py`'s
  pre-uvicorn prints may not appear.
- `set_space_volumes` **replaces** all volumes — read current ones first.
- **`RUNTIME_ERROR` ≠ broken code.** On the free CPU tier "Scheduling failure:
  unable to schedule" is HF failing to place the container; a plain restart fixes
  it (see §6). Don't push code or touch volumes/secrets to "fix" it. Space
  secrets/variables survive every rebuild/restart, and the raw `?full=true` API
  returns them as an **empty** list on purpose (write-only) — that is NOT evidence
  of loss.
- Pushing to `origin` rebuilds the Space (fresh container); the bucket persists, so
  the DB survives. Don't expect `/data` content from a previous *ephemeral* era to
  carry over — enabling the bucket started `/data` fresh once (one-time re-seed).
- **FastAPI route ordering: declare static paths before bare dynamic ones.** A bare
  `@router.post("/api/x/{id}")` (single trailing segment, `id: int`) compiles to `[^/]+` and
  will capture a literal sibling like `/api/x/new`, routing the create call into the update
  handler → 422 (it does NOT fall through). So every resource's `/new` must be declared *above*
  its `/{id}` route. This silently broke the email-rules "Add" button until fixed (see §9).
- **The admin tier editor (`POST /admin/api/agreements/{id}/tiers`) can rename an
  agreement's internal `name` and rewrites all tiers from the submitted form** — a save
  that drops a tier row or renames the row is how canonical agreements drift (this caused
  the Round 6 bugs). Engine/form behavior that *must* survive such edits should match on
  `display_name`/a prefix, not the exact `name`, and `start.py` should self-heal canonical
  tiers. Hardening this endpoint is a tracked follow-up.

---

## 8. Round 5 (pre-launch, June 2026) changes

All Round 5 DB changes are **additive + idempotent**, applied by `start.py` on every
boot against the existing `/data/app.db`. None delete users, submissions, approvals,
producers, clients, or CEs. (Verified: the only migration that issues a DELETE — the
employee-list refresh — is scoped to `dropdown_options WHERE category='employee'` and
leaves every other table untouched.)

- **Agreement display names** — new `commission_agreements.display_name` column. The
  internal `name` is the stable key (engine + form JS match on it); `display_name` is
  the user-facing label (shown in the dropdown, summary, PDF/DOCX, dashboard). The
  Referral agreement now displays as **"… - Middle Market"** with `name` unchanged.
- **New agreement types** — `Referral / Origination Fee Agreement - EMB` (TypeCode
  REFERRAL-EMB; P1 Emerging-Markets 0%/100%, Originating Producer 10%/0% Y1&2, gone
  Y3+) and `Fitch Irick Agreement` (TypeCode FITC-IRICK; Wayne Dean DEAWA2 +
  John Streer STRJO2 both locked, 20%/100% & 20%/0%, **CE/CCP still available** — a
  `FIXED_PAIR_AGREEMENTS` mechanism distinct from the CE/CCP-disabling locked pairs).
  Both Fitch producers are added to the standing-producer re-activation list so the
  daily refresh never deactivates them.
- **Agreement is_active is ADMIN-OWNED (2026-06-23).** start.py no longer force-
  deactivates any agreement on boot. Removed: the R6.3 block that retired the
  standalone **"Emerging Markets Agreement"** and the legacy "Emerging Markets -
  House Standard" deactivation — both flipped agreements back to Inactive every
  restart, undoing admin reactivations overnight. The standalone Emerging Markets
  Agreement is now toggled from the admin panel like every other type and stays put.
  (EMB = `Referral / Origination Fee Agreement - EMB` remains a separate active
  agreement; the duplicate `… - Emerging Markets` row is left Inactive by choice.)
- **NOCEA1 "No Client Executive"** — CE-list placeholder (0% / 0%, no email). On **CL
  and Bond** accounts the form replaces the "Add CE" checkbox with a required CE
  dropdown that defaults to NOCEA1; on all other departments the checkbox stays
  optional and NOCEA1 is just a selectable option. NOCEA1 never deducts from
  producers, is recorded as an auto-approved participant (visible in tracking + PDF),
  and is **excluded** from approval emails and the final-PDF distribution.
- **Employee roster** — the "Employee as Originator" dropdown is the real 112-person
  SWIA list (`SWIA_EMPLOYEES` in `app/seed_db.py`, single source of truth for both the
  fresh-DB seed and the start.py refresh).
- **Email** — `email_service._send_via_resend` now retries (3 attempts, 1s/2s backoff).
  The final signed PDF is emailed to **every party**: CM (submitter), all producers,
  originator, CCP/Mike Parsa, a real CE (not NOCEA1), and payroll@ — built by unioning
  the routing rules with each submission's approver emails. Accounting is intentionally
  not wired yet (address unconfirmed; one routing-rule insert away — see code comment).
- **Admin-created agreement types** — creating a type with no tiers now auto-writes a
  default `Producer 1 · All Years (100%/100%)` rule, and the intake form falls back to
  a default visibility (`DEFAULT_VISIBILITY`) for any type not in the JS map — so a new
  admin-created type always renders producer fields. Tune real %s on the tier editor.

**Data-safety invariant going forward:** every future migration must be additive +
idempotent. Never call `seed(reset=True)` or `python -m app.seed_db --reset` against the
Space — those wipe all tables and exist for local dev only (now flagged with ⚠️ comments).

---

## 9. Rounds 6 & 7 (post-launch, June 2026) changes

Both shipped post-launch from Elliot Wallace's feedback. All additive/idempotent; no
record/credential deletions; verified by `_smoke_round6.py` / `_smoke_round7.py` plus the
`_smoke_round4.py` regression suite.

### Round 6 — feedback fixes (commit `88b25ae`)
Root cause for #1 and #3: behavior was keyed on the exact internal agreement `name`, which
the admin tier editor can rename/strip (see §7 gotcha).
- **Middle Market referral showed no Originating Producer field.** Detection is now
  name-agnostic: `rules_engine._is_referral_orig()` and `form.html getVisibility()` match on
  `name` OR `display_name` OR the `"Referral / Origination Fee Agreement"` prefix. Options
  carry `data-display`; the JS falls back to it. `is_p1_only_deduction` follows the same rule.
- **Fitch Irick showed only Wayne Dean, no John Streer.** The fixed-pair engine forces both
  codes, but Streer's row/approval only render with a P2 tier — an admin edit had dropped it.
- **Standalone "Emerging Markets Agreement" retired** (only applies inside the EMB referral
  now): deactivated server-side (kept for historical submissions), removed from the form map.
- **`start.py` self-healing migrations (idempotent, surgical):** restore Fitch's P2 (Streer)
  tier and the Middle Market originator (P2) tiers **only when a whole producer role is
  missing** (healthy rows with admin %-tweaks untouched); ensure the Middle Market
  `display_name`; deactivate Emerging Markets.

### Round 7 — CE layer on locked-pair agreements (commit `5937f90`)
- The 5 locked-pair agreements (**Mentor MENTO3-6 + Bond Partner Webb/Herron**) now carry the
  CE layer with the **mandatory "No Client Executive" (NOCEA1) dropdown**, same UX as CL/Bond.
- Engine: `include_ce` is no longer gated on `not is_locked_pair`; **CCP stays disabled** for
  locked pairs. NOCEA1 deducts nothing / no approval; a **real CE takes the standard 2%** split
  across the locked producers and becomes an approver — all reusing the Round 5 NOCEA1 path.
- Form: `ce: true` on the 5 locked-pair entries; `updateCEMode` / `getFormData` / draft-load
  route locked pairs through `ce_dept_required_mode` (defaults to NOCEA1).
- **No DB migration** — NOCEA1 already exists in the live `client_executives` table.

### Email-rule "Add" button fix (commit `c85ab21`)
- **Admin → Email Routing Rules → "+ Add Rule" did nothing.** `POST /api/email-rules/new` was
  shadowed by `POST /api/email-rules/{rule_id}` declared above it (the §7 route-ordering trap) →
  422, no rule created. Fix: moved the static `/new` handler above `/{rule_id}` in
  `admin_routes.py`. No logic/schema change; integer-id updates still match `/{rule_id}`.
  Verified by `_smoke_email_rule.py` (TestClient + dependency overrides).