TRACKS.md

TRACKS.md — Experimental Track Registry

Single source of truth for all experimental tracks, their file ownership, tagging conventions, and isolation rules.
Referenced by CLAUDE.md. Read that file first for general project context.

---

Why Tracks?

The baseline pipeline (Track A) achieves 36% top-1 diagnostic accuracy on MedQA. To improve this, we are evaluating multiple independent strategies in parallel. Each strategy is an isolated "track" with its own code, configuration, and results — so we can compare them fairly without cross-contamination.

---

Track Registry

ID	Name	Directory	Strategy
A	Baseline	src/backend/app/	The production 6-step pipeline. No modifications for experiments.
B	RAG Variants	src/backend/tracks/rag_variants/	Test different chunking sizes, segment strategies, and embedding models to optimize guideline retrieval quality and downstream diagnostic accuracy.
C	Iterative Refinement	src/backend/tracks/iterative/	Run the diagnosis step in a serial loop — each iteration critiques and refines the previous output. Continue until the marginal improvement drops below a cost/benefit threshold. Produces a convergence chart.
D	Arbitrated Parallel	src/backend/tracks/arbitrated/	Run multiple specialist reasoning agents in parallel. An arbiter agent evaluates all outputs, tailors resubmission prompts for each specialist based on their strengths/weaknesses, and repeats until the cost/benefit ratio plateaus. Produces a cost/benefit chart.
E	Combined	src/backend/tracks/combined/	Compose per-axis winners from B/C/D/F/G/H. Tests 3 composition patterns (breadth-then-depth, depth-within-breadth, bookend). Phase 3 — build after Phase 1+2 data.
F	Prompt Architecture	src/backend/tracks/prompt_arch/	Test how reasoning prompt structure affects accuracy: structured template, few-shot, reverse reasoning, Bayesian framing. Phase 2.
G	Multi-Sample Voting	src/backend/tracks/voting/	Self-consistency via repeated sampling + majority/weighted vote. 1/3/5 samples at varying temperatures. Phase 2.
H	Evidence Verification	src/backend/tracks/verification/	Post-hoc grounding check: verify each diagnosis against patient evidence, re-rank by grounding score. Phase 2.
—	Shared	src/backend/tracks/shared/	Cross-track utilities: cost tracking, comparison harness, chart generation. Not a track itself.

---

File Tagging Convention

Every file owned by a track MUST carry a track tag on line 1. This makes ownership unambiguous when reading any file in isolation.

Format by file type

File Type	Tag Format	Example
Python (.py)	# [Track X: Name]	# [Track B: RAG Variants]
JSON (.json)	First key in object	{"_track": "Track B: RAG Variants", ...}
Markdown (.md)	HTML comment	<!-- [Track B: RAG Variants] -->
Config (.env, .yaml)	Comment	# [Track B: RAG Variants]

Track A exception

Track A files (src/backend/app/) were written before the track system existed. They are tagged with # [Track A: Baseline] on line 1, but their code is NOT modified for experimental purposes. Experiments extend or wrap Track A code from within their own track directory.

---

Isolation Rules

These rules prevent cross-contamination between experimental tracks:

1. File Ownership

• Each file belongs to exactly one track (identified by its line-1 tag and directory).
• Files in src/backend/app/ belong to Track A.
• Files in src/backend/tracks/<dir>/ belong to the corresponding track.
• Files in src/backend/tracks/shared/ are shared utilities, not owned by any single track.

2. No Cross-Modification

• Never modify a Track A file to serve an experiment. Instead, import and extend from your track's directory.
• Never modify a Track B file from Track C code, and so forth.
• If two tracks need the same utility, put it in shared/.

3. Import Direction

Track B/C/D code  →  may import from  →  Track A (app/) and shared/
Track A code      →  NEVER imports    →  Track B/C/D
shared/ code      →  may import from  →  Track A (app/) only

4. Results Isolation

• Each track stores results in src/backend/tracks/<dir>/results/.
• Result filenames include the track ID prefix (e.g., trackB_medqa_20260215.json).
• Cross-track comparison is done only via src/backend/tracks/shared/compare.py.

5. Configuration Isolation

• Track-specific parameters live in each track's own config or constants — not in app/config.py.
• The shared app/config.py provides only baseline/global settings (API keys, endpoints, etc.).

---

Track Details

Track A: Baseline

Purpose: The production-ready pipeline. The control group for all experiments.

Pipeline: Parse → Reason → Drug Check → Guideline Retrieval → Conflict Detection → Synthesis

Key parameters:

• Embedding: all-MiniLM-L6-v2 (384 dims)
• RAG top-k: 5
• No guideline chunking (each guideline = 1 document)
• Clinical reasoning temperature: 0.3
• Synthesis temperature: 0.2
• Single-pass reasoning (no iteration)

Baseline accuracy (50-case MedQA): 36% top-1, 38% mentioned

---

Track B: RAG Variants

Purpose: Determine whether retrieval quality improvements translate to better diagnostic accuracy.

Experiments:

1. Chunking strategies — Split each guideline into smaller segments (100-word chunks, 200-word chunks, sentence-level) with configurable overlap
2. Embedding models — Compare all-MiniLM-L6-v2 (384d) vs all-mpnet-base-v2 (768d) vs bge-base-en-v1.5 (768d) vs medcpt (medical-specific)
3. Top-k variation — Test k=3, k=5, k=8, k=10 to find optimal retrieval breadth
4. Re-ranking — Add a cross-encoder re-ranking step after initial retrieval

Measured outcomes:

• RAG retrieval accuracy (30-query test suite)
• MedQA diagnostic accuracy (same 50-case seed=42)
• Retrieval latency per query

Key files:

• src/backend/tracks/rag_variants/config.py — Variant definitions
• src/backend/tracks/rag_variants/chunker.py — Guideline chunking strategies
• src/backend/tracks/rag_variants/retriever.py — Modified retrieval with configurable embedding/chunking
• src/backend/tracks/rag_variants/run_variants.py — Runner that tests all configurations
• src/backend/tracks/rag_variants/results/ — Per-variant results

---

Track C: Iterative Refinement

Purpose: Determine whether repeated self-critique improves diagnostic accuracy, and find the point of diminishing returns.

Method:

1. Run baseline clinical reasoning (iteration 0)
2. Feed the output back along with the patient data and a critique prompt
3. The model reviews its own differential, identifies weaknesses, and produces a refined version
4. Repeat until: (a) max iterations reached, or (b) the differential stops changing meaningfully
5. Track accuracy and LLM cost at each iteration to produce a convergence/cost-benefit chart

Measured outcomes:

• Accuracy at each iteration (top-1, top-3, mentioned)
• LLM token cost at each iteration
• Convergence curve: accuracy vs. cumulative cost
• Iteration at which improvement drops below threshold

Key files:

• src/backend/tracks/iterative/config.py — Max iterations, convergence threshold
• src/backend/tracks/iterative/refiner.py — Iterative reasoning loop with self-critique
• src/backend/tracks/iterative/run_iterative.py — Runner with per-iteration scoring
• src/backend/tracks/iterative/results/ — Per-iteration results and charts

---

Track D: Arbitrated Parallel

Purpose: Determine whether multiple specialist agents, coordinated by an arbiter, outperform a single-pass generalist — and at what cost.

Method:

1. Run N specialist reasoning agents in parallel, each with a domain-specific system prompt (e.g., cardiologist, neurologist, infectious disease specialist)
2. An arbiter agent receives all N specialist outputs plus the patient data
3. The arbiter evaluates each specialist's differential, identifies agreements and disagreements
4. The arbiter generates tailored resubmission prompts for each specialist — telling the cardiologist "the neurologist raised X, reconsider Y" and vice versa
5. Specialists run again with the arbiter's feedback
6. Repeat until: (a) consensus reached, (b) max rounds, or (c) cost/benefit drops below threshold
7. The arbiter produces the final merged differential
8. Track accuracy and cost at each round to produce a cost/benefit chart

Measured outcomes:

• Accuracy at each arbitration round (top-1, top-3, mentioned)
• Per-specialist accuracy contribution
• LLM token cost per round (N specialists + 1 arbiter)
• Cost/benefit convergence chart
• Consensus rate across rounds

Key files:

• src/backend/tracks/arbitrated/config.py — Specialist definitions, max rounds, threshold
• src/backend/tracks/arbitrated/specialists.py — Domain-specific reasoning agents
• src/backend/tracks/arbitrated/arbiter.py — Arbiter agent that evaluates and coordinates
• src/backend/tracks/arbitrated/run_arbitrated.py — Runner with per-round scoring
• src/backend/tracks/arbitrated/results/ — Per-round results and charts

---

Adding a New Track

1. Choose an unused letter ID (E, F, ...).
2. Create src/backend/tracks/<dir_name>/ with __init__.py.
3. Add the track to the Track Registry table above.
4. Tag every new file on line 1 with # [Track X: Name].
5. Store results in src/backend/tracks/<dir_name>/results/.
6. Add a comparison entry in src/backend/tracks/shared/compare.py.
7. Never import from another track's directory — only from app/ and shared/.