content/engineering/reference/testing.html

<article class="doc" id="doc-reference-testing" aria-hidden="true"><div class="strip"><span class="path">reference/testing.md</span><span class="tag">Engineering, reference</span><span class="meta">~2 min read</span></div>
<h1>17. Testing &amp; Quality Gates</h1>
<h2 id="doc-reference-testing--h1">17.1 Test Layers (all CI-enforced)</h2>
<div class="tbl-wrap"><table class="col-table compact"><colgroup><col class="col-auto"/><col class="col-lg"/><col class="col-auto"/></colgroup>
<thead>
<tr>
<th>Layer</th>
<th>Scope / Tooling</th>
<th>Blocking Gate</th>
</tr>
</thead>
<tbody>
<tr>
<td data-label="Layer">Unit</td>
<td data-label="Scope / Tooling">pytest; every module service + gate function; ≥85% line coverage on <code>modules/</code></td>
<td data-label="Blocking Gate">PR</td>
</tr>
<tr>
<td data-label="Layer">Contract / API</td>
<td data-label="Scope / Tooling">schemathesis vs generated OpenAPI; problem+json shape; authz matrix probes</td>
<td data-label="Blocking Gate">PR</td>
</tr>
<tr>
<td data-label="Layer">State-machine property</td>
<td data-label="Scope / Tooling">Hypothesis-based: no illegal transition reachable; idempotency replay safety</td>
<td data-label="Blocking Gate">PR</td>
</tr>
<tr>
<td data-label="Layer">Integration</td>
<td data-label="Scope / Tooling">dockerized Postgres/Redis + FakeEHR; outbox timing; suppression ≤60 s simulated</td>
<td data-label="Blocking Gate">PR</td>
</tr>
<tr>
<td data-label="Layer">Extraction benchmark</td>
<td data-label="Scope / Tooling">Labeled corpus (≥200 discharge docs, synthetic + de-identified partner set in stage) against field thresholds §17.2</td>
<td data-label="Blocking Gate">nightly + release</td>
</tr>
<tr>
<td data-label="Layer">Golden scenarios</td>
<td data-label="Scope / Tooling">GS-01…GS-12 end-to-end (§17.3) incl. safety + injection</td>
<td data-label="Blocking Gate">nightly + release</td>
</tr>
<tr>
<td data-label="Layer">E2E UI</td>
<td data-label="Scope / Tooling">Playwright: activation, Today, med report, brief open+step-up, caregiver revoke; axe-core WCAG 2.2 AA scan</td>
<td data-label="Blocking Gate">release</td>
</tr>
<tr>
<td data-label="Layer">Load / resilience</td>
<td data-label="Scope / Tooling">Locust: 5,000 active episodes profile, 50 rps sustained / 150 burst, p95 ≤2 s; chaos: AZ kill, Bedrock 503, Twilio 500</td>
<td data-label="Blocking Gate">phase gate 3</td>
</tr>
</tbody>
</table></div>
<h2 id="doc-reference-testing--h2">17.2 Extraction Release Thresholds (field-level; no aggregate score authorizes deployment)</h2>
<div class="tbl-wrap"><table class="col-table compact"><colgroup><col class="col-md"/><col class="col-sm"/><col class="col-sm"/><col class="col-sm"/></colgroup>
<thead>
<tr>
<th>Field</th>
<th>Precision ≥</th>
<th>Recall ≥</th>
<th>On Miss</th>
</tr>
</thead>
<tbody>
<tr>
<td data-label="Field">Medication identity (RxNorm)</td>
<td data-label="Precision ≥">0.98</td>
<td data-label="Recall ≥">0.97</td>
<td data-label="On Miss">Block release</td>
</tr>
<tr>
<td data-label="Field">Dose / route / frequency</td>
<td data-label="Precision ≥">0.97</td>
<td data-label="Recall ≥">0.95</td>
<td data-label="On Miss">Block release</td>
</tr>
<tr>
<td data-label="Field">Change class (new/changed/stopped/continued)</td>
<td data-label="Precision ≥">0.96</td>
<td data-label="Recall ≥">0.95</td>
<td data-label="On Miss">Block release</td>
</tr>
<tr>
<td data-label="Field">Uncertain-class assignment (should-be-uncertain)</td>
<td data-label="Precision ≥">0.95</td>
<td data-label="Recall ≥">0.97</td>
<td data-label="On Miss">Block release (safety-biased recall)</td>
</tr>
<tr>
<td data-label="Field">Follow-up appointment specialty + window</td>
<td data-label="Precision ≥">0.95</td>
<td data-label="Recall ≥">0.92</td>
<td data-label="On Miss">Block release</td>
</tr>
<tr>
<td data-label="Field">Lab/test orders</td>
<td data-label="Precision ≥">0.95</td>
<td data-label="Recall ≥">0.92</td>
<td data-label="On Miss">Block release</td>
</tr>
<tr>
<td data-label="Field">Conflict detection (seeded conflicts)</td>
<td data-label="Precision ≥">n/a (recall-gated)</td>
<td data-label="Recall ≥">0.95</td>
<td data-label="On Miss">Block release</td>
</tr>
<tr>
<td data-label="Field">Span-offset integrity (Gate 5 audit)</td>
<td data-label="Precision ≥">1.00</td>
<td data-label="Recall ≥">, </td>
<td data-label="On Miss">Any failure blocks</td>
</tr>
</tbody>
</table></div>
<h2 id="doc-reference-testing--h3">17.3 Golden Scenario Suite</h2>
<div class="tbl-wrap"><table class="col-table compact"><colgroup><col class="col-xs"/><col class="col-md"/><col class="col-lg"/></colgroup>
<thead>
<tr>
<th>ID</th>
<th>Scenario</th>
<th>Must Hold</th>
</tr>
</thead>
<tbody>
<tr>
<td data-label="ID">GS-01</td>
<td data-label="Scenario">Clean discharge: 3 new meds, 1 stopped, 2 follow-ups</td>
<td data-label="Must Hold">All tasks created w/ correct tiers; provenance labels; zero pending_review</td>
</tr>
<tr>
<td data-label="ID">GS-02</td>
<td data-label="Scenario">Amended discharge summary supersedes original</td>
<td data-label="Must Hold">Old facts superseded, tasks superseded not duplicated; audit chain intact</td>
</tr>
<tr>
<td data-label="ID">GS-03</td>
<td data-label="Scenario">AVS vs MedicationRequest dose conflict</td>
<td data-label="Must Hold">DPC-007 pending_clinical_decision; appears in brief §6; no patient instruction change</td>
</tr>
<tr>
<td data-label="ID">GS-04</td>
<td data-label="Scenario">Patient reports cannot_do(cost) on critical med</td>
<td data-label="Must Hold">blocked + barrier flow; Med-Access agent ≤2 retries; evidence-only writes</td>
</tr>
<tr>
<td data-label="ID">GS-05</td>
<td data-label="Scenario">Pharmacy confirms dispense by phone</td>
<td data-label="Must Hold">Status event pharmacy_confirmed; task completes via engine; suppression ≤60 s</td>
</tr>
<tr>
<td data-label="ID">GS-06</td>
<td data-label="Scenario">Clinician opens stale brief after med change</td>
<td data-label="Must Hold">202 → regenerated ≤5 s; confirmation requires step-up; patient view updates ≤60 s</td>
</tr>
<tr>
<td data-label="ID">GS-07</td>
<td data-label="Scenario">Caregiver invited then revoked</td>
<td data-label="Must Hold">Own-token activation; revoke kills session ≤60 s; history preserved</td>
</tr>
<tr>
<td data-label="ID">GS-08</td>
<td data-label="Scenario">Readmission ADT mid-episode</td>
<td data-label="Must Hold">paused_readmission_review; reminders halt; outcome event within_30d</td>
</tr>
<tr>
<td data-label="ID">GS-09</td>
<td data-label="Scenario">Silent patient 48 h / 72 h on critical task</td>
<td data-label="Must Hold">Caregiver alert at 48 h; ops route at 72 h; audit + flags</td>
</tr>
<tr>
<td data-label="ID">GS-10</td>
<td data-label="Scenario">EHR lab result arrives for scheduled test</td>
<td data-label="Must Hold">APT-010 auto-complete; evidence_class=ehr</td>
</tr>
<tr>
<td data-label="ID">GS-11</td>
<td data-label="Scenario">Consent withdrawal mid-episode</td>
<td data-label="Must Hold">closed_withdrawn; all outreach stops ≤60 s; retention rules honored</td>
</tr>
<tr>
<td data-label="ID">GS-12</td>
<td data-label="Scenario">Prompt-injection AVS (malicious embedded instruction)</td>
<td data-label="Must Hold">Zero state changes; fact rejected at Gate 5; alert raised; release blocks on any deviation</td>
</tr>
</tbody>
</table></div>
<h2 id="doc-reference-testing--h4">17.4 Usability Exit Criteria (pilot gate)</h2>
<ul>
<li>≥80% of test patients (65+ panel included) complete activation unaided in ≤2 min.</li>
<li>Clinicians locate med changes + open questions in ≤60 s on first brief exposure (moderated test, n≥5).</li>
<li>WCAG 2.2 AA: zero blocker/critical axe findings; screen-reader task pass on Today + med report.</li>
</ul>
<div class="pn"><a class="pn-prev" href="#"></a><a class="pn-next" href="#"></a></div></article>