design/unit-test-design.md

Beacon Unit Test Design

Why This Exists

Beacon is now stable. Unit tests are the guardrail against regressions as new features land. The core design challenge is that the system mixes two fundamentally different kinds of logic — deterministic computation and LLM orchestration — which require different testing strategies.

---

Rule 1: Classify Every Code Path Before Testing It

Every function in Beacon falls into one of three categories:

Deterministic (test directly, no mocks)

These functions take inputs and return outputs with no external dependencies. They are always fast, always repeatable, and should have high coverage.

Function	Location	What makes it deterministic
haversine_miles	models.py	Pure trigonometry
PatientProfile.summary	models.py	String formatting from a dataclass
_evaluate_deterministic	agents/eligibility.py	Operator comparison (<=, >=, ==, !=, in, not_in, between)
_resolve_patient_value	agents/eligibility.py	Dict/field lookup via static key map
_compute_overall	agents/eligibility.py	Precedence rule: FAIL > UNKNOWN > PASS
_flatten_and_rank	trials_api.py	API response reshaping + haversine sort
_months_from_date	agents/intake.py	Date arithmetic (freeze datetime.today)
_resolve_months	agents/intake.py	Dict key fallback logic
lookup_disease_profile	prompts.py	File read + string matching (uses real data files on disk)

Rule: _evaluate_deterministic is the single highest-value test target in the codebase. It is the only place where the system makes an automated pass/fail judgment about a patient's eligibility. Every operator × pass/fail combination must be covered.

I/O-Bound (mock the network, test the logic around it)

These functions call external services. The external call is mocked; the logic that wraps it (retry, pagination, parameter construction) is what we test.

Function	External dependency	Mock tool
geocode_zip	OpenStreetMap Nominatim (httpx)	pytest-httpx
search_trials_api	ClinicalTrials.gov API (httpx)	pytest-httpx

Rule: Use pytest-httpx (not responses, not httpretty). The codebase uses httpx directly. Other HTTP mock libraries intercept at a different layer and will silently miss these calls.

LLM-Orchestrated (mock the client, test the loop logic)

These functions call client.messages.create() or client.messages.stream(). The LLM response is unpredictable; the orchestration logic around it is not. What we test: does the function correctly handle a tool_use stop reason, route to the right tool, append messages in the right format, and loop until end_turn?

Function	Location	What the test verifies
run_intake_agent	agents/intake.py	identify_disease and submit_profile tool routing
stream_intake_turn	agents/intake.py	token/text/profile/reset_stream event sequence
run_research_agent	agents/research.py	search tool loop: 1 call, 2 calls, API error path
stream_research_agent	agents/research.py	status/token/done event sequence
run_eligibility_check	agents/eligibility.py	parse → deterministic split → LLM assess orchestration
bulk_parse_and_strip	agents/eligibility.py	bulk parse + deterministic filter, strip fields
_parse_criteria	agents/eligibility.py	tool-forced call, struct → dataclass conversion
_assess_llm	agents/eligibility.py	tool-forced call, output → CriterionAssessment mapping

Rule: Never call the live Anthropic API from tests. Mock client.messages.create using side_effect lists to simulate multi-turn tool loops. For streaming, mock client.messages.stream with a context-manager-compatible fake that yields tokens and returns a final message.

---

Rule 2: What Not to Test

Some parts of the codebase have no unit-testable logic and should be explicitly excluded from coverage requirements.

File	Reason to skip
app.py	Gradio requires a running server; UI event wiring is not unit-testable
prompts.py system prompt strings	Static text constants — content changes should not break tests
config.py	Two string constants, no logic
translations.py	String constant tables, no logic
tools.py	Data definitions (dicts loaded from JSON), validated by Anthropic at runtime
beacon_logging.py	Logger wiring, pure side-effects
clinical_trials_guru.py	LangGraph graph construction — integration concern, not unit concern
main.py	CLI entry point wrapper

---

Rule 3: Test Structure Mirrors Module Structure

tests/
├── conftest.py          # shared fixtures + Anthropic mock builders
├── test_models.py
├── test_prompts.py
├── test_trials_api.py
└── agents/
    ├── __init__.py
    ├── test_eligibility.py
    ├── test_intake.py
    └── test_research.py

conftest.py is the foundation. It must define:

• A PatientProfile fixture covering the common ALS patient case
• Module-level factory functions (not fixtures) for constructing Anthropic mock objects: text blocks, tool-use blocks, complete message responses
• A FakeStream class that implements the client.messages.stream() context manager interface (has text_stream iterator and get_final_message())
• A mock_client fixture returning a bare MagicMock Anthropic client

---

Rule 4: The LLM Loop Testing Pattern

The while-True tool-use loops in intake, research, and eligibility are the most structurally complex code in the project. The test pattern is:

1. Configure mock_client.messages.create.side_effect as an ordered list of mock responses
2. First response(s) return stop_reason="tool_use" with tool-use blocks
3. Final response returns stop_reason="end_turn" with a text block
4. Assert: the function returned the expected value AND mock_client.messages.create.call_count equals the expected number of loop iterations

This verifies both the happy path and that the loop terminates correctly.

---

Rule 5: Coverage Targets

Layer	Target	Rationale
models.py	95%+	Pure functions, no excuse for gaps
agents/eligibility.py	85%+	Highest business-criticality
agents/intake.py	75%+	Date helpers are pure; streaming has irreducible mock complexity
trials_api.py	75%+	Flatten logic is pure; HTTP retries covered by httpx mock
prompts.py	80%+	Lookup function covered; prompt strings excluded
agents/research.py	65%+	Loop logic covered; streaming harder to fully exercise
Overall	~75%	Reasonable first-pass guardrail

---

Implementation Stages

Stages are ordered by dependency and confidence-building. Each stage is independently verifiable.

Stage 1 — Foundation

Deliverable: tests/conftest.py with all shared fixtures and mock builders.
Why first: Every other test file imports from here. Nothing else can be built without it.

Stage 2 — Pure Functions (no mocks)

Deliverable: tests/test_models.py, tests/test_prompts.py
Why second: No mocking required. Establishes that pytest is wired correctly. Disease profile lookup (lookup_disease_profile) reads real data files and also validates that the data/diseases/ JSON registry is structurally intact.

Stage 3 — Deterministic Eligibility Logic

Deliverable: tests/agents/test_eligibility.py covering _evaluate_deterministic, _resolve_patient_value, and _compute_overall
Why third: Highest-value tests in the project. Fully deterministic. No mocks needed. Establishes the criterion/assessment fixture patterns used by later stages.

Stage 4 — HTTP Layer

Deliverable: tests/test_trials_api.py
Why fourth: Introduces pytest-httpx. _flatten_and_rank is pure dict logic (no mock). search_trials_api uses httpx_mock to verify aggFilters param construction, pagination, and retry behavior.

Stage 5 — LLM Orchestration (eligibility)

Deliverable: tests/agents/test_eligibility.py extended with run_eligibility_check and bulk_parse_and_strip
Why fifth: First introduction of LLM mocking. Eligibility is the clearest to mock because the tool choice is forced and the input/output shapes are structured.

Stage 6 — LLM Orchestration (intake)

Deliverable: tests/agents/test_intake.py
Why sixth: Covers date helpers with freezegun, then introduces the FakeStream pattern for stream_intake_turn. Tests all three event types: token, text, profile.

Stage 7 — LLM Orchestration (research)

Deliverable: tests/agents/test_research.py
Why last: Most complex mock setup (patches three collaborators: search_trials_api, _flatten_and_rank, bulk_parse_and_strip). Benefits from patterns established in Stages 5–6.

---

Dependencies

pytest>=8.0
pytest-mock>=3.14
pytest-httpx>=0.35
freezegun>=1.5
pytest-cov

pyproject.toml should exclude the files listed in Rule 2 from coverage reporting.