docs/EVALUATION_PROGRESS.md

Evaluation Progress

System: hypothetico-deductive multi-agent v2
Last updated: 2026-05-09
Pipeline: BeliefState synthesis · Groq LLM debate · domain-aware ViT routing

---

Summary

#	Dataset	Type	Cases	Top-1	Top-3	Status
1	DDxPlus (eval_01)	Structured symptom (baseline)	500	1.4%	47.2%	✅ Done (pre-fix baseline)
2	DDxPlus (eval_02)	Structured symptom (fixed)	100	68.0%	86.0%	✅ Done
3	MedQA-USMLE	Real USMLE MCQ	200	0.0%	0.0%	✅ Done (task mismatch — see note)
4	Chest X-ray Pneumonia	Medical imaging (binary)	100	95.0%	95.0%	✅ Done
5	CheXpert Plus	Medical imaging (multi-label)	—	—	—	⏳ Pending
6	CUPCase	Real clinical case reports	—	—	—	⏳ Pending
7	MedXpertQA-MM	Multimodal clinical QA	—	—	—	⏳ Pending

---

Completed Evaluations

✅ Eval 1 — DDxPlus (pre-fix baseline)

• File: results/eval_01_ddxplus.json
• Script: evaluate.py
• Cases: 500 · Top-1: 1.4% · Top-3: 47.2%
• What went wrong: Three pipeline bugs: (a) DeepSeek JSON inside <think> stripped before parsing → empty llm_parsed; (b) empty hypothesis → lab gave −0.20 shift to correct disease; (c) lab NORMAL_RANGES category names ("Infection / Leukemia", "Diabetes / Hyperglycemia") dominated debate_result["ranked"], overriding clinical disease names in synthesis. Avg confidence reported as 100% (normalization artefact).
• Fixed in: v2 pipeline — see agents/orchestrator.py, app.py, agents/clinical_reasoner.py

---

✅ Eval 2 — DDxPlus (fixed pipeline)

• File: results/eval_02_ddxplus.json
• Script: evaluate.py
• Cases: 100 · Top-1: 68.0% · Top-3: 86.0% · Runtime: 49.4s (0.49s/case)
• Agents active: ClinicalReasonerAgent (keyword fallback), LabAgent (early return — no labs in DDxPlus), ImageAgent (no_imaging), HistoryAgent, DebateAgent (vote-only), SynthesisAgent (BeliefState)
• Best diseases (F1=1.0): Tuberculosis, Spontaneous pneumothorax, Scombroid food poisoning, Panic attack, PSVT, Allergic sinusitis, Anemia, Atrial fibrillation, Cluster headache, GERD, Localized edema, Acute pulmonary edema
• Worst diseases (F1=0.0): Acute rhinosinusitis, Bronchitis, Chronic rhinosinusitis, HIV (initial infection), Influenza
• Key finding: Respiratory diseases (14% category top-1) are the main weakness — Bronchitis/Pneumonia/Influenza share overlapping keyword sets and are misclassified as "COPD exacerbation"

---

✅ Eval 3 — MedQA-USMLE

• File: results/eval_02_medqa.json
• Script: eval_medqa.py
• Cases: 200 · Top-1: 0.0% · Top-3: 0.0% · Runtime: 403.9s (2.02s/case)
• Agents active: ClinicalReasonerAgent (keyword fallback), all other agents (early return), DebateAgent (vote-only)
• Why 0%: Structural task-format mismatch. Our pipeline outputs disease names; MedQA answer options are drugs ("Ketotifen eye drops"), mechanisms ("Cross-linking of DNA"), pathology descriptions ("Lactose-fermenting gram-negative rods"), and ethics choices — not disease names. Scoring 0% is expected and does not indicate a pipeline bug.
• Most common prediction: acute otitis media — its keywords (fever, cough, nasal congestion, ear) appear in many USMLE vignettes.
• Fix required: MCQ answer mapper layer post-pipeline (cosine similarity: disease → MCQ option)

---

✅ Eval 4 — Chest X-ray Pneumonia (ImageAgent isolated)

• File: results/eval_03_chestxray.json
• Script: eval_chestxray.py
• Cases: 100 (shuffled seed=42) · Top-1: 95.0% · Runtime: 39.8s (0.4s/case)
• Model: nickmuchi/vit-finetuned-chest-xray-pneumonia (2-class: NORMAL / PNEUMONIA)
• Dataset note: Originally requested NIH ChestX-ray14 (alkzar90/NIH-Chest-X-ray-dataset) — unavailable due to deprecated dataset loading script. Used hf-vision/chest-xray-pneumonia (same source images, Kaggle chest X-ray dataset).
• Per-class F1: PNEUMONIA 0.955 · NORMAL 0.944
• Errors: 5 — 3 false positives (NORMAL → PNEUMONIA), 2 false negatives (PNEUMONIA → NORMAL)
• Bug fixed during eval: _detect_domain() was routing nearly-square chest X-rays to the brain model (outputs "meningioma tumor", "no tumor"). Fixed with dark-border heuristic (border_mean < 30).

---

Pending Evaluations

⏳ Eval 5 — CheXpert Plus (multi-label chest X-ray)

• Dataset: stanfordmlgroup/CheXpert or compatible HuggingFace mirror
• Goal: Test ImageAgent with the 14-class DenseNet upgrade across multi-label pathologies
• Prerequisite: Replace _REGISTRY["chest"] with nickmuchi/densenet-finetuned-chest-xray-classification
• Expected metrics: Per-class AUC and F1 for 14 CheXpert labels (Atelectasis, Cardiomegaly, Consolidation, Edema, Pleural Effusion, etc.)
• Script to create: eval_chexpert.py

---

⏳ Eval 6 — CUPCase (real clinical case reports)

• Dataset: Real clinical case report corpus (narrative free-text, ground-truth diagnosis in report conclusion)
• Goal: Test full pipeline (LLM-enabled) on actual physician-written cases — the closest proxy to real-world deployment
• Prerequisite: Enable Groq LLM (_groq_enabled = True), add rate-limit retry wrapper
• Expected metrics: Top-1/3 accuracy; also measure reasoning_chain quality vs. ground truth diagnosis path
• Script to create: eval_cupcase.py

---

⏳ Eval 7 — MedXpertQA-MM (multimodal clinical QA)

• Dataset: MedXpertQA multimodal variant — clinical questions with both image and text
• Goal: Test the full multi-agent pipeline (text + imaging) end-to-end on cases where both modalities are required
• Prerequisites: Multi-class imaging model (Eval 5 prerequisite), MCQ answer mapper (fixes MedQA gap), Groq LLM enabled
• Expected metrics: Top-1 accuracy on multimodal questions; ablation study (text-only vs. text+image)
• Script to create: eval_medxpertqa_mm.py

---

Improvement Roadmap (by expected impact)

Priority	Change	Affected Evals	Expected Δ	Effort
🔴 1	Enable Groq LLM in eval scripts (add retry wrapper)	DDxPlus, MedQA	+15–25% DDxPlus; +20–35% MedQA	Low
🔴 2	Swap chest ViT → 14-class DenseNet-121	CheXpert Plus, NIH	Enables 12 new disease classes	Low
🟡 3	MCQ answer mapper (sentence-transformers cosine sim)	MedQA	+30–50% on diagnosis questions	Medium
🟡 4	Expand keyword vocabulary (rare diseases, autoimmune)	DDxPlus, CUPCase	+5–10% DDxPlus	Medium
🟢 5	Vignette lab extractor (regex + LLM pre-pass)	MedQA, CUPCase	Lab critique activates on prose values	Medium
🟢 6	Respiratory keyword disambiguation (Bronchitis vs COPD)	DDxPlus	Fixes 14% respiratory category	Low