reference/operations.mdEngineering, reference~2 min read

16. Observability, Operations & Configuration

16.1 Golden Signals & Alerts (page unless noted)

Metric Threshold Notes
API p95 latency (patient/clinician GETs) > 2 s for 10 min Matches NFR; per-route dashboards.
Suppression outbox lag > 45 s Guards the 60 s SLAs before breach.
Compile SLA (discharge→published) > 10 min p95 Ticket at 8 min (warn).
Gate pass-rate drop −15% vs 7-day baseline Model drift detector; blocks deploys.
Notification delivery failure > 2% / 15 min; any safety-tier failure Safety tier pages immediately.
Silent-deterioration detector health No run in 30 min Beat liveness.
Queue depth / age llm > 50 or age > 10 min Scale worker-llm; investigate.
ADT feed silence No A03 in 6 business hours Likely interface outage; contact partner.
Review queue > 20 items or oldest > 12 h Ticket clinical lead (no page).

16.2 Tracing, Dashboards, Runbooks

OpenTelemetry traces span API → Celery → Bedrock/Twilio (trace_id also in audit rows, enabling the “replay any instruction’s lineage” requirement). Dashboards-as-code in ops/: Activation Funnel, Episode Ops, Pipeline & Gates, Notifications, Agents & Kill State, Infra. Runbooks RB-01…RB-12 cover: ADT outage, Bedrock outage, Twilio failover, model rollback (RB-07), restore drill, break-glass, kill-switch use, DLQ drain, on-call handoff, incident comms, partner escalation, pilot safety event.

16.3 Analytics Denominators (ADR-016, ADM-004/007)

Population Definition (frozen)
Offered Episode reached offered (candidate × cohort criteria), the only valid outcome-rate denominator.
Activated Completed verification + consent (activation_at set).
Engaged ≥1 task response or med status event after activation.
Completed Episode closed_completed at day 30.

Every outcome report footer prints denominators, ADT/HIE coverage %, and the ADR-016 flag/suppress status. Execution report categories: required, attempted, completed, blocked, escalated, unresolved, mapped 1:1 from task states.

16.4 Tenant Config & Flags

tenant_config stores typed keys (defaults → tenant → cohort), versioned publishes, and kill-switch state; served via cached client (TTL 10 s) in api/workers. Key registry (non-exhaustive): identity.factors, provenance.hierarchy, ops.tier, safety.silent_hours_*, brief.lead_hours, notify.quiet_hours, analytics.coverage_*, int008.enabled, ehr.trigger_mode.

16.5 Scheduled Jobs (beat)

Job Cadence Purpose
fhir_document_poll 15 min ADR-007 backstop ingestion.
silent_deterioration_scan 15 min §13.3.
brief_precompute_scan 30 min Appointments entering T−24 h window.
episode_day_boundaries hourly Day-30 closing, reminder anchoring, tz-aware.
outbox_drain (worker-notify) 5 s 60 s suppression SLA.
audit_archive_export nightly Partition → S3 Object-Lock.
golden_benchmark nightly §17.2 model regression.
access_review_export weekly §9.4 evidence.