content/engineering/reference/operations.html

<article class="doc" id="doc-reference-operations" aria-hidden="true"><div class="strip"><span class="path">reference/operations.md</span><span class="tag">Engineering, reference</span><span class="meta">~2 min read</span></div>
<h1>16. Observability, Operations &amp; Configuration</h1>
<h2 id="doc-reference-operations--h1">16.1 Golden Signals &amp; Alerts (page unless noted)</h2>
<div class="tbl-wrap"><table class="col-table compact"><colgroup><col class="col-auto"/><col class="col-sm"/><col class="col-md"/></colgroup>
<thead>
<tr>
<th>Metric</th>
<th>Threshold</th>
<th>Notes</th>
</tr>
</thead>
<tbody>
<tr>
<td data-label="Metric">API p95 latency (patient/clinician GETs)</td>
<td data-label="Threshold">&gt; 2 s for 10 min</td>
<td data-label="Notes">Matches NFR; per-route dashboards.</td>
</tr>
<tr>
<td data-label="Metric">Suppression outbox lag</td>
<td data-label="Threshold">&gt; 45 s</td>
<td data-label="Notes">Guards the 60 s SLAs before breach.</td>
</tr>
<tr>
<td data-label="Metric">Compile SLA (discharge→published)</td>
<td data-label="Threshold">&gt; 10 min p95</td>
<td data-label="Notes">Ticket at 8 min (warn).</td>
</tr>
<tr>
<td data-label="Metric">Gate pass-rate drop</td>
<td data-label="Threshold">−15% vs 7-day baseline</td>
<td data-label="Notes">Model drift detector; blocks deploys.</td>
</tr>
<tr>
<td data-label="Metric">Notification delivery failure</td>
<td data-label="Threshold">&gt; 2% / 15 min; any safety-tier failure</td>
<td data-label="Notes">Safety tier pages immediately.</td>
</tr>
<tr>
<td data-label="Metric">Silent-deterioration detector health</td>
<td data-label="Threshold">No run in 30 min</td>
<td data-label="Notes">Beat liveness.</td>
</tr>
<tr>
<td data-label="Metric">Queue depth / age</td>
<td data-label="Threshold">llm &gt; 50 or age &gt; 10 min</td>
<td data-label="Notes">Scale worker-llm; investigate.</td>
</tr>
<tr>
<td data-label="Metric">ADT feed silence</td>
<td data-label="Threshold">No A03 in 6 business hours</td>
<td data-label="Notes">Likely interface outage; contact partner.</td>
</tr>
<tr>
<td data-label="Metric">Review queue</td>
<td data-label="Threshold">&gt; 20 items or oldest &gt; 12 h</td>
<td data-label="Notes">Ticket clinical lead (no page).</td>
</tr>
</tbody>
</table></div>
<h2 id="doc-reference-operations--h2">16.2 Tracing, Dashboards, Runbooks</h2>
<p>OpenTelemetry traces span API → Celery → Bedrock/Twilio (trace_id also in audit rows, enabling the “replay any instruction’s lineage” requirement). Dashboards-as-code in <code>ops/</code>: Activation Funnel, Episode Ops, Pipeline &amp; Gates, Notifications, Agents &amp; Kill State, Infra. Runbooks RB-01…RB-12 cover: ADT outage, Bedrock outage, Twilio failover, model rollback (RB-07), restore drill, break-glass, kill-switch use, DLQ drain, on-call handoff, incident comms, partner escalation, pilot safety event.</p>
<h2 id="doc-reference-operations--h3">16.3 Analytics Denominators (ADR-016, ADM-004/007)</h2>
<div class="tbl-wrap"><table class="col-table compact"><colgroup><col class="col-auto"/><col class="col-lg"/></colgroup>
<thead>
<tr>
<th>Population</th>
<th>Definition (frozen)</th>
</tr>
</thead>
<tbody>
<tr>
<td data-label="Population">Offered</td>
<td data-label="Definition (frozen)">Episode reached <code>offered</code> (candidate × cohort criteria), the only valid outcome-rate denominator.</td>
</tr>
<tr>
<td data-label="Population">Activated</td>
<td data-label="Definition (frozen)">Completed verification + consent (<code>activation_at</code> set).</td>
</tr>
<tr>
<td data-label="Population">Engaged</td>
<td data-label="Definition (frozen)">≥1 task response or med status event after activation.</td>
</tr>
<tr>
<td data-label="Population">Completed</td>
<td data-label="Definition (frozen)">Episode <code>closed_completed</code> at day 30.</td>
</tr>
</tbody>
</table></div>
<p>Every outcome report footer prints denominators, ADT/HIE coverage %, and the ADR-016 flag/suppress status. Execution report categories: required, attempted, completed, blocked, escalated, unresolved, mapped 1:1 from task states.</p>
<h2 id="doc-reference-operations--h4">16.4 Tenant Config &amp; Flags</h2>
<p><code>tenant_config</code> stores typed keys (defaults → tenant → cohort), versioned publishes, and kill-switch state; served via cached client (TTL 10 s) in api/workers. Key registry (non-exhaustive): <code>identity.factors</code>, <code>provenance.hierarchy</code>, <code>ops.tier</code>, <code>safety.silent_hours_*</code>, <code>brief.lead_hours</code>, <code>notify.quiet_hours</code>, <code>analytics.coverage_*</code>, <code>int008.enabled</code>, <code>ehr.trigger_mode</code>.</p>
<h2 id="doc-reference-operations--h5">16.5 Scheduled Jobs (beat)</h2>
<div class="tbl-wrap"><table class="col-table compact"><colgroup><col class="col-auto"/><col class="col-sm"/><col class="col-md"/></colgroup>
<thead>
<tr>
<th>Job</th>
<th>Cadence</th>
<th>Purpose</th>
</tr>
</thead>
<tbody>
<tr>
<td data-label="Job">fhir_document_poll</td>
<td data-label="Cadence">15 min</td>
<td data-label="Purpose">ADR-007 backstop ingestion.</td>
</tr>
<tr>
<td data-label="Job">silent_deterioration_scan</td>
<td data-label="Cadence">15 min</td>
<td data-label="Purpose">§13.3.</td>
</tr>
<tr>
<td data-label="Job">brief_precompute_scan</td>
<td data-label="Cadence">30 min</td>
<td data-label="Purpose">Appointments entering T−24 h window.</td>
</tr>
<tr>
<td data-label="Job">episode_day_boundaries</td>
<td data-label="Cadence">hourly</td>
<td data-label="Purpose">Day-30 closing, reminder anchoring, tz-aware.</td>
</tr>
<tr>
<td data-label="Job">outbox_drain (worker-notify)</td>
<td data-label="Cadence">5 s</td>
<td data-label="Purpose">60 s suppression SLA.</td>
</tr>
<tr>
<td data-label="Job">audit_archive_export</td>
<td data-label="Cadence">nightly</td>
<td data-label="Purpose">Partition → S3 Object-Lock.</td>
</tr>
<tr>
<td data-label="Job">golden_benchmark</td>
<td data-label="Cadence">nightly</td>
<td data-label="Purpose">§17.2 model regression.</td>
</tr>
<tr>
<td data-label="Job">access_review_export</td>
<td data-label="Cadence">weekly</td>
<td data-label="Purpose">§9.4 evidence.</td>
</tr>
</tbody>
</table></div>
<div class="pn"><a class="pn-prev" href="#"></a><a class="pn-next" href="#"></a></div></article>