feat(eval): add real lab-report eval data and KB references

Co-authored-by: Codex <chatgpt-codex-connector[bot]@users.noreply.github.com>

.gitignore

@@ -6,9 +6,10 @@ venv/
 .env
 .env.*
 
-# Generated training/eval data (reproducible from seed; don't commit)
+# Generated synthetic data (reproducible from seed; don't commit). Hand-collected real
+# reports under eval/data/real/ and KB references ARE committed.
 train/data/
-eval/data/
+eval/data/synth*
 runs/
 
 # Model artifacts (bundle the final GGUF via git-lfs deliberately, not by accident)

eval/data/real/README.md

@@ -0,0 +1,46 @@
+# Real lab-report eval set
+
+Hand-collected, publicly available **sample** lab reports (fake patients, no PHI), used to
+measure extraction accuracy on real-world formats — the credibility number for the OpenBMB
+before/after. These are for **evaluation** (and demo), not the primary training source: the
+model trains on the synthetic generator (`train/synth_reports.py`); these tell us how well it
+generalizes to messy real layouts.
+
+## Files (13 reports, varied labs/countries/panels)
+| File | Lab / format | Notes |
+|---|---|---|
+| 01_sterling_accuris.pdf | Sterling (IN), 19 pages | huge multi-panel; first pages = CBC |
+| 02_cbc_umc_johndoe.pdf | US CBC, 1 page | **labeled** ✓ (macrocytic-anemia picture) |
+| 03_lpl_wm17s.pdf | Dr Lal PathLabs (IN), 7p | multi-panel |
+| 04_pk0016_urine.pdf | urine + chem, 13p | |
+| 05_gribbles_cbm.pdf | Gribbles (AU/MY), 3p | |
+| 06_drlogy_cbc.pdf | Drlogy CBC, 1 page | **labeled** ✓ |
+| 07_sample_505271.pdf | US, 2p | |
+| 08_investigation_scanned.pdf | **image-only / scanned** | no text layer → good *vision*-modality test |
+| 09_pathkind_pl02.pdf | Pathkind (IN), 8p | |
+| 11_functionaldx.pdf | FunctionalDX, 24p | interpretive; heavy |
+| 12_lpl_k017.pdf | LPL (IN), 2p | |
+| 15_gribbles_crp.pdf | Gribbles, 9p | pediatric |
+| 16_zrt_female_hormones.pdf | ZRT hormones, 5p | estradiol/progesterone/cortisol etc. |
+
+Removed: a veterinary (dog) report — wrong domain. Reference tables (PSAP, NBME) moved to
+`kb/references/` (they're KB material, not reports — see below).
+
+## Labels
+`labels.jsonl` — one row per report: `{"image": "<file>.pdf", "tests": [...], "notes": []}`.
+The extractor reads PDFs directly (it renders pages to images), so `image` points at the PDF.
+**Currently labeled: 02 and 06** (full, verified). The rest are best labeled via bootstrap:
+
+```bash
+# 1) draft labels with the current extractor, then hand-correct into gold
+EXTRACTOR_BACKEND=api OPENBMB_API_KEY=... python eval/run_eval.py \
+    --labels eval/data/real/labels.jsonl --run    # prints metrics for labeled rows
+```
+To add a report: run the extractor on it, copy the predicted `tests` into a new `labels.jsonl`
+row, and correct any mistakes against the PDF. Faster and more accurate than typing from scratch.
+
+## Run the eval
+```bash
+python eval/run_eval.py --labels eval/data/real/labels.jsonl --run     # needs a model/API
+```
+Use it twice (base vs fine-tuned GGUF) for the before/after.

eval/data/real/labels.jsonl

@@ -0,0 +1,2 @@
+{"image": "06_drlogy_cbc.pdf", "tests": [{"marker": "Hemoglobin (Hb)", "value": "12.5", "unit": "g/dL", "reference_range": "13.0 - 17.0", "status": "low"}, {"marker": "Total RBC count", "value": "5.2", "unit": "mill/cumm", "reference_range": "4.5 - 5.5", "status": "normal"}, {"marker": "Packed Cell Volume (PCV)", "value": "57.5", "unit": "%", "reference_range": "40 - 50", "status": "high"}, {"marker": "MCV", "value": "87.75", "unit": "fL", "reference_range": "83 - 101", "status": "normal"}, {"marker": "MCH", "value": "27.2", "unit": "pg", "reference_range": "27 - 32", "status": "normal"}, {"marker": "MCHC", "value": "32.8", "unit": "g/dL", "reference_range": "32.5 - 34.5", "status": "normal"}, {"marker": "RDW", "value": "13.6", "unit": "%", "reference_range": "11.6 - 14.0", "status": "normal"}, {"marker": "Total WBC count", "value": "9000", "unit": "cumm", "reference_range": "4000-11000", "status": "normal"}, {"marker": "Neutrophils", "value": "60", "unit": "%", "reference_range": "50 - 62", "status": "normal"}, {"marker": "Lymphocytes", "value": "31", "unit": "%", "reference_range": "20 - 40", "status": "normal"}, {"marker": "Eosinophils", "value": "1", "unit": "%", "reference_range": "00 - 06", "status": "normal"}, {"marker": "Monocytes", "value": "7", "unit": "%", "reference_range": "00 - 10", "status": "normal"}, {"marker": "Basophils", "value": "1", "unit": "%", "reference_range": "00 - 02", "status": "normal"}, {"marker": "Platelet Count", "value": "150000", "unit": "cumm", "reference_range": "150000 - 410000", "status": "normal"}], "notes": []}
+{"image": "02_cbc_umc_johndoe.pdf", "tests": [{"marker": "White Blood Cell (WBC)", "value": "6.9", "unit": "K/mcL", "reference_range": "4.8-10.8", "status": "normal"}, {"marker": "Red Blood Cell (RBC)", "value": "1.8", "unit": "M/mcL", "reference_range": "4.7-6.1", "status": "low"}, {"marker": "Hemoglobin (HB/Hgb)", "value": "6.5", "unit": "g/dL", "reference_range": "14.0-18.0", "status": "low"}, {"marker": "Hematocrit (HCT)", "value": "19.5", "unit": "%", "reference_range": "42-52", "status": "low"}, {"marker": "Mean Cell Volume (MCV)", "value": "109.6", "unit": "fL", "reference_range": "80-100", "status": "high"}, {"marker": "Mean Cell Hemoglobin (MCH)", "value": "36.5", "unit": "pg", "reference_range": "27.0-32.0", "status": "high"}, {"marker": "Mean Cell Hb Conc (MCHC)", "value": "33.3", "unit": "g/dL", "reference_range": "32.0-36.0", "status": "normal"}, {"marker": "Red Cell Dist Width (RDW)", "value": "16.0", "unit": "%", "reference_range": "11.5-14.5", "status": "high"}, {"marker": "Platelet count", "value": "180", "unit": "K/mcL", "reference_range": "150-450", "status": "normal"}, {"marker": "Mean Platelet Volume", "value": "7.9", "unit": "fL", "reference_range": "7.5-11.0", "status": "normal"}, {"marker": "Neutrophil (Neut)", "value": "50", "unit": "%", "reference_range": "33-73", "status": "normal"}, {"marker": "Lymphocyte (Lymph)", "value": "36", "unit": "%", "reference_range": "13-52", "status": "normal"}, {"marker": "Monocyte (Mono)", "value": "8", "unit": "%", "reference_range": "0-10", "status": "normal"}, {"marker": "Eosinophil (Eos)", "value": "5", "unit": "%", "reference_range": "0-5", "status": "normal"}, {"marker": "Basophil (Baso)", "value": "1", "unit": "%", "reference_range": "0-2", "status": "normal"}, {"marker": "Neutrophil, Absolute", "value": "3.5", "unit": "K/mcL", "reference_range": "1.8-7.8", "status": "normal"}, {"marker": "Lymphocyte, Absolute", "value": "2.5", "unit": "K/mcL", "reference_range": "1.0-4.8", "status": "normal"}, {"marker": "Monocyte, Absolute", "value": "0.6", "unit": "K/mcL", "reference_range": "0-0.8", "status": "normal"}, {"marker": "Eosinophil, Absolute", "value": "0.4", "unit": "K/mcL", "reference_range": "0-0.45", "status": "normal"}, {"marker": "Basophil, Absolute", "value": "0.1", "unit": "K/mcL", "reference_range": "0-0.2", "status": "normal"}], "notes": []}