Hugging Face's logo

Spaces:
pramodmisra
/
pif
Running

App Files Community
pif / OPERATIONS.md
pramodmisra's picture
pramodmisra
Docs: OPERATIONS — RUNTIME_ERROR recovery + uptime self-heal Action
50efa1f
preview code
|
Raw
History Blame Contribute Delete
18.9 kB

SWIA Commission Agreement Intake — Operations & Architecture Runbook

Internal portal for capturing, approving, and finalizing producer commission agreements. FastAPI + SQLite + Jinja, deployed as a Docker Space on Hugging Face at caf.swia.ai (pramodmisra/pif). This doc is the single source of truth for how data persists, where each list comes from, and how to operate it. Last major update: 2026-06-19 (Rounds 6 & 7 + email-rule route-ordering fix).

1. Deployment & the two git remotes

The repo has two remotes on main — keep them in lockstep:

Remote URL Role
github github.com/pramodmisra/pif-app Canonical source of truth (also runs the daily refresh Action)
origin huggingface.co/spaces/pramodmisra/pif The live Space; pushing here triggers a rebuild/deploy

Always git push github main && git push origin main. They have silently drifted before. If origin rejects (non-fast-forward), git fetch origin, inspect git log HEAD..origin/main, and git merge origin/main -X ours rather than force-pushing (force-push drops the Space's commit ancestry).

A push to origin triggers a factory rebuild (fresh container). A config change (secrets/volumes) or restart_space() triggers a restart. A push to origin while RUNNING rebuilds with no downtime (old container keeps serving through RUNNING_BUILDING → RUNNING_APP_STARTING → RUNNING).

Uptime self-heal (GitHub Action uptime-check.yml): a cron every 15 min runs scripts/uptime_check.py, which reads the Space runtime stage and issues a non-destructive plain restart if it has fallen into a recoverable error (RUNTIME_ERROR/PAUSED/STOPPED/SLEEPING). It does not restart on BUILD_ERROR/CONFIG_ERROR/NO_APP_FILE (a code problem a restart can't fix) — it fails the job so GitHub emails the owner instead of restart-looping. Reuses the same HF_TOKEN secret as the daily refresh. Manual run: Actions tab → Uptime Self-Heal → Run workflow.

2. Persistent storage (CRITICAL)

The DB is SQLite at DATABASE_URL=sqlite:////data/app.db. /data is only persistent because a Storage Bucket is mounted there:

  • Bucket pramodmisra/pif-db (private) is mounted read-write at /data.
  • HF's old flat "$5/mo storage tier" API is retired (request_space_storage 404s). Current model = buckets + volumes:
    api.create_bucket("pramodmisra/pif-db", private=True)
api.set_space_volumes("pramodmisra/pif",
    volumes=[Volume(type="bucket", source="pramodmisra/pif-db",
                    mount_path="/data", read_only=False)])
api.restart_space("pramodmisra/pif", factory_reboot=True)
  • SQLite-on-bucket is supported by HF (same pattern as Label Studio Spaces).
  • huggingface_hub.SpaceRuntime.volumes may show None even when mounted — confirm via raw API GET /api/spaces/{id}?full=true → runtime.volumes, or api.list_bucket_tree("pramodmisra/pif-db") (you'll see app.db).

⚠️ NEVER detach this bucket / remove the volume — the entire user database (logins, password resets, CE edits) lives there. Without it, /data is ephemeral and every rebuild wipes the DB back to the seed.

3. Data flow: what populates each list 

Google Sheet (clients)  ─┐
                         ├─►  build_refresh_data.py  ─►  refresh_data.json  ─►  HF dataset pif-data
producers_seed.json  ────┘        (CI, daily 08:00 UTC)     (gitignored)         (private)
                                                                                      │
                                                                          start.py pulls at boot
                                                                                      ▼
                                                              Producer + Client tables in /data/app.db

Producers — producers_seed.json (TRACKED) is the source of truth

  • Why a seed file: refresh_data.json is gitignored (it carries 18,656 client records). On a fresh CI checkout it's absent, so the daily job used to emit an empty producer list → every reboot collapsed producers to the 11 seeded ones. Fixed 2026-05-29 by adding the tracked producers_seed.json.
  • build_refresh_data.py → load_existing_producers() reads producers_seed.json first (always present in CI), falling back to refresh_data.json.
  • Each row: {code, prefix, suffix, name, email?}. Built from producers_codes_emails.csv (the canonical roster, 35 people / 71 codes).
  • To change the producer roster: edit producers_seed.json (or regenerate from the CSV), commit, push both remotes, then run the refresh (Section 6).
  • Producer.is_active is ADMIN-OWNED — the refresh no longer resets it. Until 2026-06-23, both start.py's boot import and /api/data-refresh did "deactivate ALL producers, then re-activate every code in the refresh" — so the nightly Space restart re-activated the whole 71-code roster and wiped any deactivations made in the admin panel (the "57 reset back to 74 overnight" bug). Now the refresh upserts only: existing producers get name/email/code updates but keep their current is_active; brand-new roster codes are added active (deactivate once and it sticks). Standing approvers (Bella SANBE1 + the 12 locked-pair/Fitch codes) are still force-activated. To trim the visible producer list, deactivate in the admin panel — it now survives every refresh. Regression guard: _smoke_producer_persist.py.

Producer emails — real values from the roster

  • start.py's import (and the admin /api/data-refresh upload) set Producer.email from the JSON when present. Real emails (e.g. Kathryn Faulkner = kdye@, Michael Banner = mbenner@, Banks Snellings = banks.snellings@) win over the auto-generated first-initial+lastname@ rule.
  • Producers with a blank email in the CSV (Richard S. Brewer, UAC Risk Management) fall back to the auto-generated address.

Clients — Google Sheet (PowerApps-synced)

  • Sheet 1RYmxN855… tab gid=1047572149, columns LookupCode, Name.
  • build_refresh_data.py fetches the public CSV export (guard: ≥15,000 rows), dedups by code → ~18,656 clients. Producers are not in the Sheet.

Client Executives (CE) — separate client_executives table

  • CEs are an independent list (NOT the Producer table). Seeded once from ce_seed.json (7 SWIA CEs) when the table is empty; managed via Admin → Manage Client Executives (add/edit/toggle/delete + bulk paste).
  • The form's CE picker, name resolution, and approval email come from this table first, with a fallback to the Producer lookup for legacy/edge cases.

4. Authentication & user lifecycle

  • Self-signup (/signup): any @snellingswalters.com email → active user, logged in immediately (no email verification, no admin approval).
  • Password reset: self-service /forgot-password → emailed token (1 h); superadmin can also trigger a reset from the User Management panel.
  • Persistence: users live in /data/app.db (persistent bucket). start.py skips seeding when any user exists, so registrations/edits survive rebuilds.
  • Seeded users: only the two permanent superadmins (System Admin + Pramod) are in seed_db.py, so a fresh DB never resurrects deleted people.
  • Removal: superadmin-only Delete — blocked while active and blocked if the user has submissions (deactivate-don't-delete preserves the audit trail).

5. The client lookup (form performance)

The intake form must NOT pre-render all 18,656 clients as <option>s — that produced a 6.3 MB page that broke the dropdown. Current design (form.html):

  • Both client dropdowns start empty; the Filter by Code and Filter by Name boxes fill them on type (client-side startsWith over the clientData array). Page is ~1.1 MB.
  • ensureSelected() re-adds the chosen option for code↔name sync and draft restore.
  • Do not re-add {% for c in clients %} loops to those selects. If the client list grows much larger, move to a server-side /api/clients/search endpoint instead of embedding clientData.

6. Common operations

Refresh clients + producers now (manual):

python scripts/build_refresh_data.py          # builds refresh_data.json
HF_TOKEN=<write-token> python scripts/upload_to_hf.py   # uploads to dataset + restarts Space

The daily GitHub Action daily-client-refresh.yml (08:00 UTC) does the same.

Change the producer roster / emails: edit producers_seed.json → commit → push both remotes → run the refresh above (or wait for the daily Action) → the Space re-imports on restart.

Add / edit a Client Executive: Admin → Manage Client Executives (persists in the DB). To change the initial seed, edit ce_seed.json (only seeds an empty table).

Check the live Space:

from huggingface_hub import HfApi
api = HfApi(token=TOKEN)
api.get_space_runtime("pramodmisra/pif").stage     # RUNNING / BUILDING / *_ERROR
list(api.list_bucket_tree("pramodmisra/pif-db"))   # confirms app.db in persistent storage

Site is down / 503 / RUNTIME_ERROR ("Scheduling failure: unable to schedule"): This is an HF infra-side failure to place the container on free CPU hardware — NOT a code/git/storage problem. The fix is a plain, non-destructive restart (the bucket persists across both restart and factory reboot, so the DB is safe):

api.restart_space("pramodmisra/pif")               # re-requests scheduling
# poll api.get_space_runtime(...).stage until RUNNING (usually ~30s)

The uptime-check.yml Action does this automatically within ~15 min. If a plain restart still errors, escalate to restart_space(..., factory_reboot=True). If "Scheduling failure" persists after both, it's HF capacity — wait/retry, or move off the free tier to paid cpu-upgrade (dedicated hardware; needs billing approval). First confirm storage is intact (app.db ≈ 1.34 MB in list_bucket_tree) — a byte-identical app.db proves no data/credentials/logins were touched; the restart swaps the container, not the data.

7. Gotchas (hard-won)

  • refresh_data.json is gitignored → never rely on it existing in CI; producers come from producers_seed.json.
  • HF run-log fetch only returns the recent tail (uvicorn lines); start.py's pre-uvicorn prints may not appear.
  • set_space_volumes replaces all volumes — read current ones first.
  • RUNTIME_ERROR ≠ broken code. On the free CPU tier "Scheduling failure: unable to schedule" is HF failing to place the container; a plain restart fixes it (see §6). Don't push code or touch volumes/secrets to "fix" it. Space secrets/variables survive every rebuild/restart, and the raw ?full=true API returns them as an empty list on purpose (write-only) — that is NOT evidence of loss.
  • Pushing to origin rebuilds the Space (fresh container); the bucket persists, so the DB survives. Don't expect /data content from a previous ephemeral era to carry over — enabling the bucket started /data fresh once (one-time re-seed).
  • FastAPI route ordering: declare static paths before bare dynamic ones. A bare @router.post("/api/x/{id}") (single trailing segment, id: int) compiles to [^/]+ and will capture a literal sibling like /api/x/new, routing the create call into the update handler → 422 (it does NOT fall through). So every resource's /new must be declared above its /{id} route. This silently broke the email-rules "Add" button until fixed (see §9).
  • The admin tier editor (POST /admin/api/agreements/{id}/tiers) can rename an agreement's internal name and rewrites all tiers from the submitted form — a save that drops a tier row or renames the row is how canonical agreements drift (this caused the Round 6 bugs). Engine/form behavior that must survive such edits should match on display_name/a prefix, not the exact name, and start.py should self-heal canonical tiers. Hardening this endpoint is a tracked follow-up.

8. Round 5 (pre-launch, June 2026) changes

All Round 5 DB changes are additive + idempotent, applied by start.py on every boot against the existing /data/app.db. None delete users, submissions, approvals, producers, clients, or CEs. (Verified: the only migration that issues a DELETE — the employee-list refresh — is scoped to dropdown_options WHERE category='employee' and leaves every other table untouched.)

  • Agreement display names — new commission_agreements.display_name column. The internal name is the stable key (engine + form JS match on it); display_name is the user-facing label (shown in the dropdown, summary, PDF/DOCX, dashboard). The Referral agreement now displays as "… - Middle Market" with name unchanged.
  • New agreement types — Referral / Origination Fee Agreement - EMB (TypeCode REFERRAL-EMB; P1 Emerging-Markets 0%/100%, Originating Producer 10%/0% Y1&2, gone Y3+) and Fitch Irick Agreement (TypeCode FITC-IRICK; Wayne Dean DEAWA2 + John Streer STRJO2 both locked, 20%/100% & 20%/0%, CE/CCP still available — a FIXED_PAIR_AGREEMENTS mechanism distinct from the CE/CCP-disabling locked pairs). Both Fitch producers are added to the standing-producer re-activation list so the daily refresh never deactivates them.
  • Agreement is_active is ADMIN-OWNED (2026-06-23). start.py no longer force- deactivates any agreement on boot. Removed: the R6.3 block that retired the standalone "Emerging Markets Agreement" and the legacy "Emerging Markets - House Standard" deactivation — both flipped agreements back to Inactive every restart, undoing admin reactivations overnight. The standalone Emerging Markets Agreement is now toggled from the admin panel like every other type and stays put. (EMB = Referral / Origination Fee Agreement - EMB remains a separate active agreement; the duplicate … - Emerging Markets row is left Inactive by choice.)
  • NOCEA1 "No Client Executive" — CE-list placeholder (0% / 0%, no email). On CL and Bond accounts the form replaces the "Add CE" checkbox with a required CE dropdown that defaults to NOCEA1; on all other departments the checkbox stays optional and NOCEA1 is just a selectable option. NOCEA1 never deducts from producers, is recorded as an auto-approved participant (visible in tracking + PDF), and is excluded from approval emails and the final-PDF distribution.
  • Employee roster — the "Employee as Originator" dropdown is the real 112-person SWIA list (SWIA_EMPLOYEES in app/seed_db.py, single source of truth for both the fresh-DB seed and the start.py refresh).
  • Email — email_service._send_via_resend now retries (3 attempts, 1s/2s backoff). The final signed PDF is emailed to every party: CM (submitter), all producers, originator, CCP/Mike Parsa, a real CE (not NOCEA1), and payroll@ — built by unioning the routing rules with each submission's approver emails. Accounting is intentionally not wired yet (address unconfirmed; one routing-rule insert away — see code comment).
  • Admin-created agreement types — creating a type with no tiers now auto-writes a default Producer 1 · All Years (100%/100%) rule, and the intake form falls back to a default visibility (DEFAULT_VISIBILITY) for any type not in the JS map — so a new admin-created type always renders producer fields. Tune real %s on the tier editor.

Data-safety invariant going forward: every future migration must be additive + idempotent. Never call seed(reset=True) or python -m app.seed_db --reset against the Space — those wipe all tables and exist for local dev only (now flagged with ⚠️ comments).

9. Rounds 6 & 7 (post-launch, June 2026) changes

Both shipped post-launch from Elliot Wallace's feedback. All additive/idempotent; no record/credential deletions; verified by _smoke_round6.py / _smoke_round7.py plus the _smoke_round4.py regression suite.

Round 6 — feedback fixes (commit 88b25ae)

Root cause for #1 and #3: behavior was keyed on the exact internal agreement name, which the admin tier editor can rename/strip (see §7 gotcha).

  • Middle Market referral showed no Originating Producer field. Detection is now name-agnostic: rules_engine._is_referral_orig() and form.html getVisibility() match on name OR display_name OR the "Referral / Origination Fee Agreement" prefix. Options carry data-display; the JS falls back to it. is_p1_only_deduction follows the same rule.
  • Fitch Irick showed only Wayne Dean, no John Streer. The fixed-pair engine forces both codes, but Streer's row/approval only render with a P2 tier — an admin edit had dropped it.
  • Standalone "Emerging Markets Agreement" retired (only applies inside the EMB referral now): deactivated server-side (kept for historical submissions), removed from the form map.
  • start.py self-healing migrations (idempotent, surgical): restore Fitch's P2 (Streer) tier and the Middle Market originator (P2) tiers only when a whole producer role is missing (healthy rows with admin %-tweaks untouched); ensure the Middle Market display_name; deactivate Emerging Markets.

Round 7 — CE layer on locked-pair agreements (commit 5937f90)

  • The 5 locked-pair agreements (Mentor MENTO3-6 + Bond Partner Webb/Herron) now carry the CE layer with the mandatory "No Client Executive" (NOCEA1) dropdown, same UX as CL/Bond.
  • Engine: include_ce is no longer gated on not is_locked_pair; CCP stays disabled for locked pairs. NOCEA1 deducts nothing / no approval; a real CE takes the standard 2% split across the locked producers and becomes an approver — all reusing the Round 5 NOCEA1 path.
  • Form: ce: true on the 5 locked-pair entries; updateCEMode / getFormData / draft-load route locked pairs through ce_dept_required_mode (defaults to NOCEA1).
  • No DB migration — NOCEA1 already exists in the live client_executives table.

Email-rule "Add" button fix (commit c85ab21)

  • Admin → Email Routing Rules → "+ Add Rule" did nothing. POST /api/email-rules/new was shadowed by POST /api/email-rules/{rule_id} declared above it (the §7 route-ordering trap) → 422, no rule created. Fix: moved the static /new handler above /{rule_id} in admin_routes.py. No logic/schema change; integer-id updates still match /{rule_id}. Verified by _smoke_email_rule.py (TestClient + dependency overrides).