Spaces:
Running
Running
| # SWIA Commission Agreement Intake — Operations & Architecture Runbook | |
| > Internal portal for capturing, approving, and finalizing producer commission | |
| > agreements. FastAPI + SQLite + Jinja, deployed as a Docker Space on Hugging | |
| > Face at **caf.swia.ai** (`pramodmisra/pif`). This doc is the single source of | |
| > truth for how data persists, where each list comes from, and how to operate it. | |
| > Last major update: 2026-06-19 (Rounds 6 & 7 + email-rule route-ordering fix). | |
| --- | |
| ## 1. Deployment & the two git remotes | |
| The repo has **two remotes on `main`** — keep them in lockstep: | |
| | Remote | URL | Role | | |
| |--------|-----|------| | |
| | `github` | github.com/pramodmisra/pif-app | Canonical source of truth (also runs the daily refresh Action) | | |
| | `origin` | huggingface.co/spaces/pramodmisra/pif | The live Space; pushing here triggers a rebuild/deploy | | |
| **Always `git push github main && git push origin main`.** They have silently | |
| drifted before. If `origin` rejects (non-fast-forward), `git fetch origin`, | |
| inspect `git log HEAD..origin/main`, and `git merge origin/main -X ours` rather | |
| than force-pushing (force-push drops the Space's commit ancestry). | |
| A push to `origin` triggers a **factory rebuild** (fresh container). A config | |
| change (secrets/volumes) or `restart_space()` triggers a restart. A push to | |
| `origin` while RUNNING rebuilds with **no downtime** (old container keeps serving | |
| through `RUNNING_BUILDING` → `RUNNING_APP_STARTING` → `RUNNING`). | |
| **Uptime self-heal (GitHub Action `uptime-check.yml`):** a cron every 15 min runs | |
| `scripts/uptime_check.py`, which reads the Space runtime stage and issues a | |
| **non-destructive plain restart** if it has fallen into a recoverable error | |
| (`RUNTIME_ERROR`/`PAUSED`/`STOPPED`/`SLEEPING`). It does **not** restart on | |
| `BUILD_ERROR`/`CONFIG_ERROR`/`NO_APP_FILE` (a code problem a restart can't fix) — | |
| it fails the job so GitHub emails the owner instead of restart-looping. Reuses the | |
| same `HF_TOKEN` secret as the daily refresh. Manual run: Actions tab → *Uptime | |
| Self-Heal* → *Run workflow*. | |
| --- | |
| ## 2. Persistent storage (CRITICAL) | |
| The DB is SQLite at `DATABASE_URL=sqlite:////data/app.db`. `/data` is **only | |
| persistent because a Storage Bucket is mounted there**: | |
| - Bucket **`pramodmisra/pif-db`** (private) is mounted **read-write at `/data`**. | |
| - HF's old flat "$5/mo storage tier" API is retired (`request_space_storage` 404s). | |
| Current model = **buckets + volumes**: | |
| ```python | |
| api.create_bucket("pramodmisra/pif-db", private=True) | |
| api.set_space_volumes("pramodmisra/pif", | |
| volumes=[Volume(type="bucket", source="pramodmisra/pif-db", | |
| mount_path="/data", read_only=False)]) | |
| api.restart_space("pramodmisra/pif", factory_reboot=True) | |
| ``` | |
| - SQLite-on-bucket is supported by HF (same pattern as Label Studio Spaces). | |
| - `huggingface_hub.SpaceRuntime.volumes` may show `None` even when mounted — | |
| confirm via raw API `GET /api/spaces/{id}?full=true` → `runtime.volumes`, or | |
| `api.list_bucket_tree("pramodmisra/pif-db")` (you'll see `app.db`). | |
| **⚠️ NEVER detach this bucket / remove the volume — the entire user database | |
| (logins, password resets, CE edits) lives there.** Without it, `/data` is | |
| ephemeral and every rebuild wipes the DB back to the seed. | |
| --- | |
| ## 3. Data flow: what populates each list | |
| ``` | |
| Google Sheet (clients) ─┐ | |
| ├─► build_refresh_data.py ─► refresh_data.json ─► HF dataset pif-data | |
| producers_seed.json ────┘ (CI, daily 08:00 UTC) (gitignored) (private) | |
| │ | |
| start.py pulls at boot | |
| ▼ | |
| Producer + Client tables in /data/app.db | |
| ``` | |
| ### Producers — `producers_seed.json` (TRACKED) is the source of truth | |
| - **Why a seed file:** `refresh_data.json` is **gitignored** (it carries 18,656 | |
| client records). On a fresh CI checkout it's absent, so the daily job used to | |
| emit an **empty** producer list → every reboot collapsed producers to the 11 | |
| seeded ones. Fixed 2026-05-29 by adding the tracked `producers_seed.json`. | |
| - `build_refresh_data.py` → `load_existing_producers()` reads `producers_seed.json` | |
| first (always present in CI), falling back to `refresh_data.json`. | |
| - Each row: `{code, prefix, suffix, name, email?}`. Built from | |
| `producers_codes_emails.csv` (the canonical roster, 35 people / 71 codes). | |
| - **To change the producer roster:** edit `producers_seed.json` (or regenerate | |
| from the CSV), commit, push both remotes, then run the refresh (Section 6). | |
| - **`Producer.is_active` is ADMIN-OWNED — the refresh no longer resets it.** | |
| Until 2026-06-23, both `start.py`'s boot import and `/api/data-refresh` did | |
| *"deactivate ALL producers, then re-activate every code in the refresh"* — so | |
| the nightly Space restart re-activated the whole 71-code roster and wiped any | |
| deactivations made in the admin panel (the "57 reset back to 74 overnight" | |
| bug). Now the refresh **upserts only**: existing producers get name/email/code | |
| updates but keep their current `is_active`; brand-new roster codes are added | |
| active (deactivate once and it sticks). Standing approvers (Bella `SANBE1` + | |
| the 12 locked-pair/Fitch codes) are still force-activated. To trim the visible | |
| producer list, **deactivate in the admin panel** — it now survives every | |
| refresh. Regression guard: `_smoke_producer_persist.py`. | |
| ### Producer emails — real values from the roster | |
| - `start.py`'s import (and the admin `/api/data-refresh` upload) set | |
| `Producer.email` from the JSON when present. Real emails (e.g. Kathryn | |
| Faulkner = `kdye@`, Michael Banner = `mbenner@`, Banks Snellings = | |
| `banks.snellings@`) win over the auto-generated `first-initial+lastname@` rule. | |
| - Producers with a blank email in the CSV (Richard S. Brewer, UAC Risk | |
| Management) fall back to the auto-generated address. | |
| ### Clients — Google Sheet (PowerApps-synced) | |
| - Sheet `1RYmxN855…` tab `gid=1047572149`, columns `LookupCode, Name`. | |
| - `build_refresh_data.py` fetches the public CSV export (guard: ≥15,000 rows), | |
| dedups by code → ~18,656 clients. Producers are **not** in the Sheet. | |
| ### Client Executives (CE) — separate `client_executives` table | |
| - CEs are an **independent list** (NOT the Producer table). Seeded once from | |
| `ce_seed.json` (7 SWIA CEs) when the table is empty; managed via | |
| **Admin → Manage Client Executives** (add/edit/toggle/delete + bulk paste). | |
| - The form's CE picker, name resolution, and approval email come from this table | |
| first, with a fallback to the Producer lookup for legacy/edge cases. | |
| --- | |
| ## 4. Authentication & user lifecycle | |
| - **Self-signup** (`/signup`): any `@snellingswalters.com` email → active `user`, | |
| logged in immediately (no email verification, no admin approval). | |
| - **Password reset**: self-service `/forgot-password` → emailed token (1 h); | |
| superadmin can also trigger a reset from the User Management panel. | |
| - **Persistence**: users live in `/data/app.db` (persistent bucket). `start.py` | |
| skips seeding when any user exists, so registrations/edits survive rebuilds. | |
| - **Seeded users**: only the two permanent superadmins (System Admin + Pramod) | |
| are in `seed_db.py`, so a fresh DB never resurrects deleted people. | |
| - **Removal**: superadmin-only Delete — blocked while active and blocked if the | |
| user has submissions (deactivate-don't-delete preserves the audit trail). | |
| --- | |
| ## 5. The client lookup (form performance) | |
| The intake form must **NOT** pre-render all 18,656 clients as `<option>`s — that | |
| produced a 6.3 MB page that broke the dropdown. Current design (`form.html`): | |
| - Both client dropdowns start **empty**; the **Filter by Code** and **Filter by | |
| Name** boxes fill them on type (client-side `startsWith` over the `clientData` | |
| array). Page is ~1.1 MB. | |
| - `ensureSelected()` re-adds the chosen option for code↔name sync and draft restore. | |
| - **Do not** re-add `{% for c in clients %}` loops to those selects. If the client | |
| list grows much larger, move to a server-side `/api/clients/search` endpoint | |
| instead of embedding `clientData`. | |
| --- | |
| ## 6. Common operations | |
| **Refresh clients + producers now (manual):** | |
| ```bash | |
| python scripts/build_refresh_data.py # builds refresh_data.json | |
| HF_TOKEN=<write-token> python scripts/upload_to_hf.py # uploads to dataset + restarts Space | |
| ``` | |
| The daily GitHub Action `daily-client-refresh.yml` (08:00 UTC) does the same. | |
| **Change the producer roster / emails:** edit `producers_seed.json` → commit → | |
| push both remotes → run the refresh above (or wait for the daily Action) → | |
| the Space re-imports on restart. | |
| **Add / edit a Client Executive:** Admin → Manage Client Executives (persists in | |
| the DB). To change the initial seed, edit `ce_seed.json` (only seeds an empty table). | |
| **Check the live Space:** | |
| ```python | |
| from huggingface_hub import HfApi | |
| api = HfApi(token=TOKEN) | |
| api.get_space_runtime("pramodmisra/pif").stage # RUNNING / BUILDING / *_ERROR | |
| list(api.list_bucket_tree("pramodmisra/pif-db")) # confirms app.db in persistent storage | |
| ``` | |
| **Site is down / 503 / RUNTIME_ERROR ("Scheduling failure: unable to schedule"):** | |
| This is an HF **infra-side** failure to place the container on free CPU hardware — | |
| NOT a code/git/storage problem. The fix is a plain, non-destructive restart (the | |
| bucket persists across both restart *and* factory reboot, so the DB is safe): | |
| ```python | |
| api.restart_space("pramodmisra/pif") # re-requests scheduling | |
| # poll api.get_space_runtime(...).stage until RUNNING (usually ~30s) | |
| ``` | |
| The `uptime-check.yml` Action does this automatically within ~15 min. If a plain | |
| restart still errors, escalate to `restart_space(..., factory_reboot=True)`. If | |
| "Scheduling failure" persists after both, it's HF capacity — wait/retry, or move | |
| off the free tier to paid `cpu-upgrade` (dedicated hardware; needs billing | |
| approval). **First confirm storage is intact** (`app.db` ≈ 1.34 MB in | |
| `list_bucket_tree`) — a byte-identical `app.db` proves no data/credentials/logins | |
| were touched; the restart swaps the container, not the data. | |
| --- | |
| ## 7. Gotchas (hard-won) | |
| - `refresh_data.json` is **gitignored** → never rely on it existing in CI; producers | |
| come from `producers_seed.json`. | |
| - HF run-log fetch only returns the recent tail (uvicorn lines); `start.py`'s | |
| pre-uvicorn prints may not appear. | |
| - `set_space_volumes` **replaces** all volumes — read current ones first. | |
| - **`RUNTIME_ERROR` ≠ broken code.** On the free CPU tier "Scheduling failure: | |
| unable to schedule" is HF failing to place the container; a plain restart fixes | |
| it (see §6). Don't push code or touch volumes/secrets to "fix" it. Space | |
| secrets/variables survive every rebuild/restart, and the raw `?full=true` API | |
| returns them as an **empty** list on purpose (write-only) — that is NOT evidence | |
| of loss. | |
| - Pushing to `origin` rebuilds the Space (fresh container); the bucket persists, so | |
| the DB survives. Don't expect `/data` content from a previous *ephemeral* era to | |
| carry over — enabling the bucket started `/data` fresh once (one-time re-seed). | |
| - **FastAPI route ordering: declare static paths before bare dynamic ones.** A bare | |
| `@router.post("/api/x/{id}")` (single trailing segment, `id: int`) compiles to `[^/]+` and | |
| will capture a literal sibling like `/api/x/new`, routing the create call into the update | |
| handler → 422 (it does NOT fall through). So every resource's `/new` must be declared *above* | |
| its `/{id}` route. This silently broke the email-rules "Add" button until fixed (see §9). | |
| - **The admin tier editor (`POST /admin/api/agreements/{id}/tiers`) can rename an | |
| agreement's internal `name` and rewrites all tiers from the submitted form** — a save | |
| that drops a tier row or renames the row is how canonical agreements drift (this caused | |
| the Round 6 bugs). Engine/form behavior that *must* survive such edits should match on | |
| `display_name`/a prefix, not the exact `name`, and `start.py` should self-heal canonical | |
| tiers. Hardening this endpoint is a tracked follow-up. | |
| --- | |
| ## 8. Round 5 (pre-launch, June 2026) changes | |
| All Round 5 DB changes are **additive + idempotent**, applied by `start.py` on every | |
| boot against the existing `/data/app.db`. None delete users, submissions, approvals, | |
| producers, clients, or CEs. (Verified: the only migration that issues a DELETE — the | |
| employee-list refresh — is scoped to `dropdown_options WHERE category='employee'` and | |
| leaves every other table untouched.) | |
| - **Agreement display names** — new `commission_agreements.display_name` column. The | |
| internal `name` is the stable key (engine + form JS match on it); `display_name` is | |
| the user-facing label (shown in the dropdown, summary, PDF/DOCX, dashboard). The | |
| Referral agreement now displays as **"… - Middle Market"** with `name` unchanged. | |
| - **New agreement types** — `Referral / Origination Fee Agreement - EMB` (TypeCode | |
| REFERRAL-EMB; P1 Emerging-Markets 0%/100%, Originating Producer 10%/0% Y1&2, gone | |
| Y3+) and `Fitch Irick Agreement` (TypeCode FITC-IRICK; Wayne Dean DEAWA2 + | |
| John Streer STRJO2 both locked, 20%/100% & 20%/0%, **CE/CCP still available** — a | |
| `FIXED_PAIR_AGREEMENTS` mechanism distinct from the CE/CCP-disabling locked pairs). | |
| Both Fitch producers are added to the standing-producer re-activation list so the | |
| daily refresh never deactivates them. | |
| - **Agreement is_active is ADMIN-OWNED (2026-06-23).** start.py no longer force- | |
| deactivates any agreement on boot. Removed: the R6.3 block that retired the | |
| standalone **"Emerging Markets Agreement"** and the legacy "Emerging Markets - | |
| House Standard" deactivation — both flipped agreements back to Inactive every | |
| restart, undoing admin reactivations overnight. The standalone Emerging Markets | |
| Agreement is now toggled from the admin panel like every other type and stays put. | |
| (EMB = `Referral / Origination Fee Agreement - EMB` remains a separate active | |
| agreement; the duplicate `… - Emerging Markets` row is left Inactive by choice.) | |
| - **NOCEA1 "No Client Executive"** — CE-list placeholder (0% / 0%, no email). On **CL | |
| and Bond** accounts the form replaces the "Add CE" checkbox with a required CE | |
| dropdown that defaults to NOCEA1; on all other departments the checkbox stays | |
| optional and NOCEA1 is just a selectable option. NOCEA1 never deducts from | |
| producers, is recorded as an auto-approved participant (visible in tracking + PDF), | |
| and is **excluded** from approval emails and the final-PDF distribution. | |
| - **Employee roster** — the "Employee as Originator" dropdown is the real 112-person | |
| SWIA list (`SWIA_EMPLOYEES` in `app/seed_db.py`, single source of truth for both the | |
| fresh-DB seed and the start.py refresh). | |
| - **Email** — `email_service._send_via_resend` now retries (3 attempts, 1s/2s backoff). | |
| The final signed PDF is emailed to **every party**: CM (submitter), all producers, | |
| originator, CCP/Mike Parsa, a real CE (not NOCEA1), and payroll@ — built by unioning | |
| the routing rules with each submission's approver emails. Accounting is intentionally | |
| not wired yet (address unconfirmed; one routing-rule insert away — see code comment). | |
| - **Admin-created agreement types** — creating a type with no tiers now auto-writes a | |
| default `Producer 1 · All Years (100%/100%)` rule, and the intake form falls back to | |
| a default visibility (`DEFAULT_VISIBILITY`) for any type not in the JS map — so a new | |
| admin-created type always renders producer fields. Tune real %s on the tier editor. | |
| **Data-safety invariant going forward:** every future migration must be additive + | |
| idempotent. Never call `seed(reset=True)` or `python -m app.seed_db --reset` against the | |
| Space — those wipe all tables and exist for local dev only (now flagged with ⚠️ comments). | |
| --- | |
| ## 9. Rounds 6 & 7 (post-launch, June 2026) changes | |
| Both shipped post-launch from Elliot Wallace's feedback. All additive/idempotent; no | |
| record/credential deletions; verified by `_smoke_round6.py` / `_smoke_round7.py` plus the | |
| `_smoke_round4.py` regression suite. | |
| ### Round 6 — feedback fixes (commit `88b25ae`) | |
| Root cause for #1 and #3: behavior was keyed on the exact internal agreement `name`, which | |
| the admin tier editor can rename/strip (see §7 gotcha). | |
| - **Middle Market referral showed no Originating Producer field.** Detection is now | |
| name-agnostic: `rules_engine._is_referral_orig()` and `form.html getVisibility()` match on | |
| `name` OR `display_name` OR the `"Referral / Origination Fee Agreement"` prefix. Options | |
| carry `data-display`; the JS falls back to it. `is_p1_only_deduction` follows the same rule. | |
| - **Fitch Irick showed only Wayne Dean, no John Streer.** The fixed-pair engine forces both | |
| codes, but Streer's row/approval only render with a P2 tier — an admin edit had dropped it. | |
| - **Standalone "Emerging Markets Agreement" retired** (only applies inside the EMB referral | |
| now): deactivated server-side (kept for historical submissions), removed from the form map. | |
| - **`start.py` self-healing migrations (idempotent, surgical):** restore Fitch's P2 (Streer) | |
| tier and the Middle Market originator (P2) tiers **only when a whole producer role is | |
| missing** (healthy rows with admin %-tweaks untouched); ensure the Middle Market | |
| `display_name`; deactivate Emerging Markets. | |
| ### Round 7 — CE layer on locked-pair agreements (commit `5937f90`) | |
| - The 5 locked-pair agreements (**Mentor MENTO3-6 + Bond Partner Webb/Herron**) now carry the | |
| CE layer with the **mandatory "No Client Executive" (NOCEA1) dropdown**, same UX as CL/Bond. | |
| - Engine: `include_ce` is no longer gated on `not is_locked_pair`; **CCP stays disabled** for | |
| locked pairs. NOCEA1 deducts nothing / no approval; a **real CE takes the standard 2%** split | |
| across the locked producers and becomes an approver — all reusing the Round 5 NOCEA1 path. | |
| - Form: `ce: true` on the 5 locked-pair entries; `updateCEMode` / `getFormData` / draft-load | |
| route locked pairs through `ce_dept_required_mode` (defaults to NOCEA1). | |
| - **No DB migration** — NOCEA1 already exists in the live `client_executives` table. | |
| ### Email-rule "Add" button fix (commit `c85ab21`) | |
| - **Admin → Email Routing Rules → "+ Add Rule" did nothing.** `POST /api/email-rules/new` was | |
| shadowed by `POST /api/email-rules/{rule_id}` declared above it (the §7 route-ordering trap) → | |
| 422, no rule created. Fix: moved the static `/new` handler above `/{rule_id}` in | |
| `admin_routes.py`. No logic/schema change; integer-id updates still match `/{rule_id}`. | |
| Verified by `_smoke_email_rule.py` (TestClient + dependency overrides). | |