design/unit-test-design.md

# Beacon Unit Test Design

## Why This Exists

Beacon is now stable. Unit tests are the guardrail against regressions as new features land. The core design challenge is that the system mixes two fundamentally different kinds of logic — deterministic computation and LLM orchestration — which require different testing strategies.

---

## Rule 1: Classify Every Code Path Before Testing It

Every function in Beacon falls into one of three categories:

### Deterministic (test directly, no mocks)

These functions take inputs and return outputs with no external dependencies. They are always fast, always repeatable, and should have high coverage.

| Function | Location | What makes it deterministic |
|----------|----------|----------------------------|
| `haversine_miles` | `models.py` | Pure trigonometry |
| `PatientProfile.summary` | `models.py` | String formatting from a dataclass |
| `_evaluate_deterministic` | `agents/eligibility.py` | Operator comparison (<=, >=, ==, !=, in, not_in, between) |
| `_resolve_patient_value` | `agents/eligibility.py` | Dict/field lookup via static key map |
| `_compute_overall` | `agents/eligibility.py` | Precedence rule: FAIL > UNKNOWN > PASS |
| `_flatten_and_rank` | `trials_api.py` | API response reshaping + haversine sort |
| `_months_from_date` | `agents/intake.py` | Date arithmetic (freeze `datetime.today`) |
| `_resolve_months` | `agents/intake.py` | Dict key fallback logic |
| `lookup_disease_profile` | `prompts.py` | File read + string matching (uses real data files on disk) |

> **Rule:** `_evaluate_deterministic` is the single highest-value test target in the codebase. It is the only place where the system makes an automated pass/fail judgment about a patient's eligibility. Every operator × pass/fail combination must be covered.

### I/O-Bound (mock the network, test the logic around it)

These functions call external services. The external call is mocked; the logic that wraps it (retry, pagination, parameter construction) is what we test.

| Function | External dependency | Mock tool |
|----------|-------------------|-----------|
| `geocode_zip` | OpenStreetMap Nominatim (httpx) | `pytest-httpx` |
| `search_trials_api` | ClinicalTrials.gov API (httpx) | `pytest-httpx` |

> **Rule:** Use `pytest-httpx` (not `responses`, not `httpretty`). The codebase uses `httpx` directly. Other HTTP mock libraries intercept at a different layer and will silently miss these calls.

### LLM-Orchestrated (mock the client, test the loop logic)

These functions call `client.messages.create()` or `client.messages.stream()`. The LLM response is unpredictable; the orchestration logic around it is not. What we test: does the function correctly handle a `tool_use` stop reason, route to the right tool, append messages in the right format, and loop until `end_turn`?

| Function | Location | What the test verifies |
|----------|----------|----------------------|
| `run_intake_agent` | `agents/intake.py` | identify_disease and submit_profile tool routing |
| `stream_intake_turn` | `agents/intake.py` | token/text/profile/reset_stream event sequence |
| `run_research_agent` | `agents/research.py` | search tool loop: 1 call, 2 calls, API error path |
| `stream_research_agent` | `agents/research.py` | status/token/done event sequence |
| `run_eligibility_check` | `agents/eligibility.py` | parse → deterministic split → LLM assess orchestration |
| `bulk_parse_and_strip` | `agents/eligibility.py` | bulk parse + deterministic filter, strip fields |
| `_parse_criteria` | `agents/eligibility.py` | tool-forced call, struct → dataclass conversion |
| `_assess_llm` | `agents/eligibility.py` | tool-forced call, output → CriterionAssessment mapping |

> **Rule:** Never call the live Anthropic API from tests. Mock `client.messages.create` using `side_effect` lists to simulate multi-turn tool loops. For streaming, mock `client.messages.stream` with a context-manager-compatible fake that yields tokens and returns a final message.

---

## Rule 2: What Not to Test

Some parts of the codebase have no unit-testable logic and should be explicitly excluded from coverage requirements.

| File | Reason to skip |
|------|---------------|
| `app.py` | Gradio requires a running server; UI event wiring is not unit-testable |
| `prompts.py` system prompt strings | Static text constants — content changes should not break tests |
| `config.py` | Two string constants, no logic |
| `translations.py` | String constant tables, no logic |
| `tools.py` | Data definitions (dicts loaded from JSON), validated by Anthropic at runtime |
| `beacon_logging.py` | Logger wiring, pure side-effects |
| `clinical_trials_guru.py` | LangGraph graph construction — integration concern, not unit concern |
| `main.py` | CLI entry point wrapper |

---

## Rule 3: Test Structure Mirrors Module Structure

```
tests/
├── conftest.py          # shared fixtures + Anthropic mock builders
├── test_models.py
├── test_prompts.py
├── test_trials_api.py
└── agents/
    ├── __init__.py
    ├── test_eligibility.py
    ├── test_intake.py
    └── test_research.py
```

`conftest.py` is the foundation. It must define:
- A `PatientProfile` fixture covering the common ALS patient case
- Module-level factory functions (not fixtures) for constructing Anthropic mock objects: text blocks, tool-use blocks, complete message responses
- A `FakeStream` class that implements the `client.messages.stream()` context manager interface (has `text_stream` iterator and `get_final_message()`)
- A `mock_client` fixture returning a bare `MagicMock` Anthropic client

---

## Rule 4: The LLM Loop Testing Pattern

The while-True tool-use loops in intake, research, and eligibility are the most structurally complex code in the project. The test pattern is:

1. Configure `mock_client.messages.create.side_effect` as an ordered list of mock responses
2. First response(s) return `stop_reason="tool_use"` with tool-use blocks
3. Final response returns `stop_reason="end_turn"` with a text block
4. Assert: the function returned the expected value AND `mock_client.messages.create.call_count` equals the expected number of loop iterations

This verifies both the happy path and that the loop terminates correctly.

---

## Rule 5: Coverage Targets

| Layer | Target | Rationale |
|-------|--------|-----------|
| `models.py` | 95%+ | Pure functions, no excuse for gaps |
| `agents/eligibility.py` | 85%+ | Highest business-criticality |
| `agents/intake.py` | 75%+ | Date helpers are pure; streaming has irreducible mock complexity |
| `trials_api.py` | 75%+ | Flatten logic is pure; HTTP retries covered by httpx mock |
| `prompts.py` | 80%+ | Lookup function covered; prompt strings excluded |
| `agents/research.py` | 65%+ | Loop logic covered; streaming harder to fully exercise |
| **Overall** | **~75%** | Reasonable first-pass guardrail |

---

## Implementation Stages

Stages are ordered by dependency and confidence-building. Each stage is independently verifiable.

### Stage 1 — Foundation
**Deliverable:** `tests/conftest.py` with all shared fixtures and mock builders.  
**Why first:** Every other test file imports from here. Nothing else can be built without it.

### Stage 2 — Pure Functions (no mocks)
**Deliverable:** `tests/test_models.py`, `tests/test_prompts.py`  
**Why second:** No mocking required. Establishes that pytest is wired correctly. Disease profile lookup (`lookup_disease_profile`) reads real data files and also validates that the `data/diseases/` JSON registry is structurally intact.

### Stage 3 — Deterministic Eligibility Logic
**Deliverable:** `tests/agents/test_eligibility.py` covering `_evaluate_deterministic`, `_resolve_patient_value`, and `_compute_overall`  
**Why third:** Highest-value tests in the project. Fully deterministic. No mocks needed. Establishes the criterion/assessment fixture patterns used by later stages.

### Stage 4 — HTTP Layer
**Deliverable:** `tests/test_trials_api.py`  
**Why fourth:** Introduces `pytest-httpx`. `_flatten_and_rank` is pure dict logic (no mock). `search_trials_api` uses `httpx_mock` to verify aggFilters param construction, pagination, and retry behavior.

### Stage 5 — LLM Orchestration (eligibility)
**Deliverable:** `tests/agents/test_eligibility.py` extended with `run_eligibility_check` and `bulk_parse_and_strip`  
**Why fifth:** First introduction of LLM mocking. Eligibility is the clearest to mock because the tool choice is forced and the input/output shapes are structured.

### Stage 6 — LLM Orchestration (intake)
**Deliverable:** `tests/agents/test_intake.py`  
**Why sixth:** Covers date helpers with `freezegun`, then introduces the `FakeStream` pattern for `stream_intake_turn`. Tests all three event types: token, text, profile.

### Stage 7 — LLM Orchestration (research)
**Deliverable:** `tests/agents/test_research.py`  
**Why last:** Most complex mock setup (patches three collaborators: `search_trials_api`, `_flatten_and_rank`, `bulk_parse_and_strip`). Benefits from patterns established in Stages 5–6.

---

## Dependencies

```
pytest>=8.0
pytest-mock>=3.14
pytest-httpx>=0.35
freezegun>=1.5
pytest-cov
```

`pyproject.toml` should exclude the files listed in Rule 2 from coverage reporting.