{ "compiled": "2026-11-20", "note": "Compiled from the project's plan (STROKE_COPILOT_PLAN.md), its changelog and its phase documents; every number is copied from those documents, and where they disagree the newest changelog entry is used and the correction is noted.", "groups": [ { "id": "data", "title": "Data and splits", "summary": "ISLES'24 supplies 149 stroke cases: 129 development cases in five folds and 20 held-out cases that stayed closed until Phase 4 (DEC-5). The RSNA haemorrhage scans are split by patient 80/10/10 into training, validation and test (DEC-22). The two runs here turned the finished vision results into the 'contract': one structured file of imaging findings per case, which is all the language models see of the scan.", "experiments": [ { "id": "EXP-33", "title": "Vision-output contracts for 129 development cases", "asked": "Can a valid vision-output file (the contract the language models read) be built for every development case?", "method": "No model is run: each field is read from a finished run and joined by case. Core from the nnU-Net fold runs, penumbra and occlusion evidence from EXP-1, the bleed flag from EXP-16's calibrated confidence at 0.175, confidence from EXP-32, ASPECTS left null.", "result": "All 129 development cases built and validated. Bleed flagged in 68; core under 1 mL in 30; occlusion site ICA 13, basilar 1, unknown 115. Median core 4.5 mL, median penumbra 143.1 mL.", "numbers": [ { "label": "Contracts built and validated", "value": "129" }, { "label": "Bleed flagged", "value": "68" }, { "label": "Core under 1 mL", "value": "30" }, { "label": "Occlusion site unknown", "value": "115" } ], "status": "done", "decisions": [ "DEC-7", "DEC-9", "DEC-25", "DEC-26", "DEC-27", "DEC-28" ], "note": "The 115 unknown sites follow the contract's site rule: the labels cannot separate M1 from M2, so real-case thrombectomy reasoning will often meet 'site not reported'.", "source": "CHANGELOG 2026-11-15 (Phase 3: … C7 contract build); plan §7 Phase 3, Vision A" }, { "id": "EXP-34", "title": "Ground-truth contracts for the comparison arm", "asked": "What would the contract look like if the vision models were perfect?", "method": "For each development case: the reference final infarct as the core, no bleed, the same penumbra and occlusion site as the predicted contract, confidence 1.0 and no warnings.", "result": "129 validated. 8 have a negative mismatch volume, where the final infarct exceeds the Tmax > 6 s area; the schema allows it. In the 20-case sample, thrombolysis is expected 'no' in 10 cases with predicted vision against 0 with ground truth (Vision B's false bleed flags); thrombectomy 'no' in 18 against 17.", "numbers": [ { "label": "Ground-truth contracts validated", "value": "129" }, { "label": "Negative mismatch volume", "value": "8" }, { "label": "Thrombolysis expected 'no', predicted vs ground truth", "value": "10 vs 0" } ], "status": "done", "decisions": [ "DEC-11" ], "source": "CHANGELOG 2026-11-16 (Phase 3: teammates' open tasks done from the Mac); plan §7 Phase 3, NLP A" } ] }, { "id": "vision-a", "title": "Vision A: outlining the core", "summary": "This strand asked whether a neural network (nnU-Net) can outline the dead tissue on ISLES'24 CT perfusion better than the clinical-standard threshold. Over five folds it overlaps the true infarct better (Dice 0.215 against 0.111) but its volume is not measurably closer, and both methods under-call large infarcts. The confidence attached to the volume is simply how far it sits from the nearest decision cut-off.", "experiments": [ { "id": "EXP-1", "title": "Threshold baseline for the core", "asked": "How well does the clinical-standard rule (core = blood flow under 30% of the healthy side) match the final infarct?", "method": "All 129 development cases on Colab. Core = rCBF < 30% inside the Tmax > 6 s area, penumbra = Tmax > 6 s; scored against the final infarct with Dice (overlap between the model's outline and the true one, 0 to 1), volume error and how often the volume lands on the wrong side of a decision cut-off.", "result": "Dice 0.111 ± 0.031 over folds (pooled median 0.027); absolute volume error 25.2 ± 5.0 mL. The predicted core falls on the other side of a cut-off from the reference in 35, 29, 23 and 17 of 129 cases at 21, 31, 51 and 70 mL.", "numbers": [ { "label": "Dice, mean ± std over folds", "value": "0.111 ± 0.031" }, { "label": "Absolute volume error", "value": "25.2 ± 5.0 mL" }, { "label": "Wrong side of 21/31/51/70 mL", "value": "35/29/23/17 of 129" }, { "label": "Occlusion site unknown at a 5 mm limit", "value": "115 of 129" } ], "status": "done", "decisions": [ "DEC-7" ], "note": "A Dice this low was checked for a bug before it was accepted: only half of the infarct lies inside Tmax > 6 s and its median rCBF is 0.65, so the low score comes from the data. The 30% cut-off is kept as the clinical standard.", "source": "CHANGELOG 2026-11-04 (Phase 2: EXP-1 results); plan §7 Phase 2" }, { "id": "EXP-2", "title": "nnU-Net dummy run on a Colab T4", "asked": "Does nnU-Net train, checkpoint and resume on Colab, and how long is an epoch?", "method": "A 3-epoch dummy on fold 0 on a Tesla T4 runtime with 2 CPUs and 12.7 GB RAM.", "result": "It ran to the end: 103 training and 26 validation cases, and resuming from the checkpoint worked. Epochs took 664, 509 and 570 s; at about 510 s, 250 epochs would take about 35 hours on a T4.", "numbers": [ { "label": "Epoch times", "value": "664, 509 and 570 s" } ], "status": "dummy run only", "decisions": [], "note": "Fold-0 Dice was 0.247 after 3 epochs; the changelog states it is not a result.", "source": "CHANGELOG 2026-11-04 (Phase 2, task 3: EXP-2 dummy result)" }, { "id": "EXP-3", "title": "Fold 0 at 150 epochs on Colab", "asked": "Planned as the first full nnU-Net fold-0 training run.", "method": "Sized at 150 epochs, about 5 hours of training on a Colab L4.", "result": "", "numbers": [], "status": "superseded", "decisions": [], "note": "The documents record it only as a stop record with no numbers (150 epochs, no final checkpoint). Fold 0 was trained instead as EXP-14, at 300 epochs on a rented GPU.", "source": "CHANGELOG 2026-11-04 (Phase 2, task 3: EXP-2 dummy result); CHANGELOG 2026-11-05 (run records); CHANGELOG 2026-11-19 (EXP-61's first try stopped)" }, { "id": "EXP-4", "title": "nnU-Net dummy run on a Colab L4", "asked": "Is the slow T4 epoch a CPU limit, and how fast is an L4?", "method": "The same 3-epoch dummy on an L4 with 12 CPUs, 53 GB RAM and 11 data workers.", "result": "Epochs took 191, 119 and 140 s, about 4.3 times faster than the T4, which fits the CPU explanation.", "numbers": [ { "label": "Epoch times", "value": "191, 119 and 140 s" }, { "label": "Speed-up over the T4", "value": "about 4.3 times" } ], "status": "dummy run only", "decisions": [], "note": "Fold-0 Dice was 0.224 after 3 epochs, not a result.", "source": "CHANGELOG 2026-11-04 (Phase 2, task 3: EXP-2 dummy result)" }, { "id": "EXP-13", "title": "nnU-Net dummy run on a rented RTX 4090", "asked": "Does the rented-GPU set-up work, and how fast is it?", "method": "A 3-epoch dummy on one rented RTX 4090.", "result": "Epochs took 37–44 s, against about 121 s on the Colab L4, so about 3× faster. GPU utilisation median 99%, with no worker bottleneck. Resume works.", "numbers": [ { "label": "Epoch time", "value": "37–44 s" }, { "label": "GPU utilisation, median", "value": "99%" } ], "status": "dummy run only", "decisions": [ "DEC-21" ], "source": "CHANGELOG 2026-11-05 (DEC-21 rented GPU, EXP-13 and EXP-14 …)" }, { "id": "EXP-14", "title": "nnU-Net fold 0, 300 epochs", "asked": "How well does nnU-Net outline the infarct on the first fold, against the threshold baseline?", "method": "nnU-Net 3d_fullres trained for 300 epochs on a rented RTX 4090, scored on the 26 development cases of fold 0.", "result": "Dice 0.218 ± 0.249 (median 0.111) and absolute volume error 19.6 ± 23.3 mL, against the threshold baseline's 0.123 and 20.5 mL on the same 26 cases. Wrong side of 21/31/51/70 mL: 5/5/3/3 of 26. 9 of 26 cases score 0.", "numbers": [ { "label": "Dice", "value": "0.218 ± 0.249" }, { "label": "Dice, median", "value": "0.111" }, { "label": "Absolute volume error", "value": "19.6 ± 23.3 mL" }, { "label": "Threshold baseline, same cases", "value": "Dice 0.123, 20.5 mL" }, { "label": "Wrong side of 21/31/51/70 mL", "value": "5/5/3/3 of 26" } ], "status": "done", "decisions": [ "DEC-21" ], "note": "Dice nearly doubles, but volume error, which matters more for the rules, barely moves. Training was interrupted when the rented host's contract ended and resumed from the epoch-269 checkpoint.", "source": "CHANGELOG 2026-11-05 (DEC-21 rented GPU, EXP-13 and EXP-14 …); plan §7 Phase 2" }, { "id": "EXP-15", "title": "Error analysis on fold 0", "asked": "Where and why does nnU-Net fail to find the infarct?", "method": "The 26 fold-0 cases of EXP-14, joined to the threshold baseline; for each case the infarct size, occlusion site, onset time and how much of the infarct showed as at risk (Tmax > 6 s) on the first scan.", "result": "9 of 26 cases are Dice-0 misses (the outline does not touch the infarct at all): in 4 of the 9 the model marked nothing, and one is an over-call, 32.6 mL marked for a 1.1 mL infarct. Small infarcts are missed more often: 7 of the 17 cases under 21 mL, against 2 of the 9 larger ones. Misses showed a median of 30% of the infarct as at risk, against 64% in the other cases, but the groups overlap (Mann-Whitney p ≈ 0.28).", "numbers": [ { "label": "Dice-0 misses", "value": "9 of 26" }, { "label": "Missed, under 21 mL", "value": "7 of 17" }, { "label": "Missed, larger", "value": "2 of 9" }, { "label": "Median share visible, misses vs others", "value": "30% vs 64%" } ], "status": "done", "decisions": [], "note": "One miss is a 101 mL infarct the model marked as empty. On all 9 misses the threshold baseline also scored under 0.05, which does not show the scan is the problem, because the baseline only looks inside the Tmax > 6 s area.", "source": "docs/ERROR_ANALYSIS_VISION_A.md (Summary; Results measured by code); CHANGELOG 2026-11-07 and 2026-11-13 (review of PR #1)" }, { "id": "EXP-18", "title": "nnU-Net fold 1, 300 epochs", "asked": "How well does nnU-Net do on fold 1?", "method": "nnU-Net 3d_fullres, 300 epochs on a rented RTX 4090, scored on the fold's 26 development cases.", "result": "Dice 0.211 ± 0.246; absolute volume error 22.0 ± 42.2 mL; wrong side of 21/31/51/70 mL in 3/4/4/3 of 26.", "numbers": [ { "label": "Dice", "value": "0.211 ± 0.246" }, { "label": "Absolute volume error", "value": "22.0 ± 42.2 mL" }, { "label": "Wrong side of 21/31/51/70 mL", "value": "3/4/4/3 of 26" } ], "status": "done", "decisions": [ "DEC-21" ], "source": "CHANGELOG 2026-11-13 (Phase 3: A1 done — nnU-Net folds 1–4 collected …)" }, { "id": "EXP-19", "title": "nnU-Net fold 2, 300 epochs", "asked": "How well does nnU-Net do on fold 2?", "method": "nnU-Net 3d_fullres, 300 epochs on a rented RTX 4090, scored on the fold's 26 development cases.", "result": "Dice 0.173 ± 0.165; absolute volume error 27.0 ± 46.1 mL; wrong side of 21/31/51/70 mL in 9/6/3/3 of 26.", "numbers": [ { "label": "Dice", "value": "0.173 ± 0.165" }, { "label": "Absolute volume error", "value": "27.0 ± 46.1 mL" }, { "label": "Wrong side of 21/31/51/70 mL", "value": "9/6/3/3 of 26" } ], "status": "done", "decisions": [ "DEC-21" ], "note": "Its first attempt landed on the same machine as EXP-18, where the two runs starved each other; it was destroyed and set up again on another host.", "source": "CHANGELOG 2026-11-13 (Phase 3: A1 done — nnU-Net folds 1–4 collected …)" }, { "id": "EXP-20", "title": "nnU-Net fold 3, 300 epochs", "asked": "How well does nnU-Net do on fold 3?", "method": "nnU-Net 3d_fullres, 300 epochs on a rented RTX 4090, scored on the fold's 26 development cases.", "result": "Dice 0.290 ± 0.281; absolute volume error 20.9 ± 27.2 mL; wrong side of 21/31/51/70 mL in 5/5/2/3 of 26.", "numbers": [ { "label": "Dice", "value": "0.290 ± 0.281" }, { "label": "Absolute volume error", "value": "20.9 ± 27.2 mL" }, { "label": "Wrong side of 21/31/51/70 mL", "value": "5/5/2/3 of 26" } ], "status": "done", "decisions": [ "DEC-21" ], "source": "CHANGELOG 2026-11-13 (Phase 3: A1 done — nnU-Net folds 1–4 collected …)" }, { "id": "EXP-21", "title": "nnU-Net fold 4, 300 epochs", "asked": "How well does nnU-Net do on fold 4?", "method": "nnU-Net 3d_fullres, 300 epochs on a rented RTX 4090, scored on the fold's 25 development cases.", "result": "Dice 0.184 ± 0.240 over 25 cases; absolute volume error 28.3 ± 39.9 mL; wrong side of 21/31/51/70 mL in 8/8/8/5.", "numbers": [ { "label": "Dice", "value": "0.184 ± 0.240" }, { "label": "Absolute volume error", "value": "28.3 ± 39.9 mL" }, { "label": "Wrong side of 21/31/51/70 mL", "value": "8/8/8/5" } ], "status": "done", "decisions": [ "DEC-21" ], "source": "CHANGELOG 2026-11-13 (Phase 3: A1 done — nnU-Net folds 1–4 collected …)" }, { "id": "EXP-22", "title": "Fold-0 outlines regenerated", "asked": "Can fold 0's predicted outlines be recovered for the error analysis?", "method": "EXP-14 kept no validation masks, so they were predicted again from its final checkpoint. A free dummy on 3 fold-0 cases first matched EXP-14's per-case Dice within 0.0001 and volume within 0.004 mL.", "result": "Fold 0 Dice 0.218 ± 0.249 over 26 cases, the same as EXP-14.", "numbers": [ { "label": "Dice", "value": "0.218 ± 0.249" } ], "status": "done", "decisions": [], "source": "CHANGELOG 2026-11-12 (Phase 3: fold jobs, offer watcher …)" }, { "id": "EXP-25", "title": "Five-fold table: nnU-Net against the threshold", "asked": "Over all five folds, is nnU-Net better than the threshold baseline on overlap, volume and cut-off crossings?", "method": "All 129 development cases, each scored by the fold model that did not train on it; paired comparison with a bootstrap interval over cases, a Wilcoxon signed-rank test and exact McNemar tests on the crossings.", "result": "Dice 0.215 ± 0.046 over 5 folds against the threshold's 0.111 ± 0.031 (paired +0.104, 95% CI +0.078 to +0.133; p < 0.0001). Absolute volume error 23.5 ± 3.9 mL against 25.2 ± 5.0 mL (paired −1.6 mL, CI −4.5 to +1.3; p 0.35): not a clear difference. Wrong side of 21/31/51/70 mL in 30/28/20/17 of 129 against 35/29/23/17; no crossing difference is significant.", "numbers": [ { "label": "Dice, nnU-Net", "value": "0.215 ± 0.046" }, { "label": "Dice, threshold", "value": "0.111 ± 0.031" }, { "label": "Paired Dice gain", "value": "+0.104 (95% CI +0.078 to +0.133)" }, { "label": "Absolute volume error", "value": "23.5 ± 3.9 mL against 25.2 ± 5.0 mL" }, { "label": "Paired volume difference", "value": "−1.6 mL (CI −4.5 to +1.3; p 0.35)" }, { "label": "Wrong side of 21/31/51/70 mL", "value": "30/28/20/17 against 35/29/23/17 of 129" } ], "status": "done", "decisions": [ "DEC-25" ], "note": "For lesions of 21 mL or more (45 cases) both methods are far off, mostly under-calling (nnU-Net 52.8 vs 55.7 mL absolute error). nnU-Net is clearly better at locating the infarct, but its volume is not measurably better.", "source": "CHANGELOG 2026-11-13 (Phase 3: A1 done … 5-fold table; DEC-25); plan §7 Phase 3; README" }, { "id": "EXP-26", "title": "Core volume checks and corrections", "asked": "Is the nnU-Net core consistent with the penumbra, and can its volume be corrected?", "method": "The 129 development cases, cross-fitted by fold (each correction fitted on four folds and tested on the fifth).", "result": "The nnU-Net core never exceeds the Tmax > 6 s volume (0 of 129), so no clipping is needed. Large infarcts are under-called: for infarcts ≥ 70 mL the medians are reference 118 mL, nnU-Net 35, threshold 25, Tmax > 6 s 188, and in 7 of 22 the final infarct exceeds the whole Tmax > 6 s region. Linear corrections made things worse (+2.4 to +2.6 mL; 54–61 cases on the wrong side of 21 mL against 30) and the best robust one gained 0.6 mL, with a CI crossing zero.", "numbers": [ { "label": "Core larger than the Tmax > 6 s volume", "value": "0 of 129" }, { "label": "Infarcts ≥ 70 mL: median reference", "value": "118 mL" }, { "label": "Same infarcts: median nnU-Net", "value": "35" }, { "label": "Best robust correction", "value": "gained 0.6 mL" } ], "status": "done", "decisions": [ "DEC-26" ], "note": "No volume calibration was adopted. The gap is reported as a limit: final infarct is not acute core, and every patient was reperfused.", "source": "CHANGELOG 2026-11-13 (B4 decided — DEC-26); plan §2 DEC-26" }, { "id": "EXP-27", "title": "Error analysis on all five folds", "asked": "Does the fold-0 failure pattern hold over all 129 development cases?", "method": "The five nnU-Net folds joined to the threshold baseline, each case scored by the fold model that did not train on it. An over-call or under-call is a volume on the wrong side of at least one cut-off (21, 31, 51 or 70 mL).", "result": "The model scores Dice 0 on 37 of the 127 cases with a final infarct: 32 of 82 under 21 mL, against 5 of 45 at 21 mL or more. Under-calls are much more common than over-calls: 34 against 9. The median visible share is 35% for misses against 62% for the other cases.", "numbers": [ { "label": "Dice-0 misses", "value": "37 of 127" }, { "label": "Missed, under 21 mL", "value": "32 of 82" }, { "label": "Missed, 21 mL and over", "value": "5 of 45" }, { "label": "Under-calls against over-calls", "value": "34 against 9" }, { "label": "Median share visible, misses vs others", "value": "35% vs 62%" } ], "status": "done", "decisions": [], "note": "Corrected at review: the run first counted 39 Dice-0 cases; 2 cases with no final infarct (0 mL) score Dice 0 but miss nothing and are now counted apart, giving 37. No significance test was run on these groups. The check against EXP-25 found 0 of 51 numbers different.", "source": "docs/ERROR_ANALYSIS_VISION_A.md (Five-fold follow-ups); CHANGELOG 2026-11-15 (review of PR #7) and 2026-11-16 (reruns)" }, { "id": "EXP-30", "title": "Core confidence, first version", "asked": "Can a confidence score for the core volume be built from signals available at inference time?", "method": "Three signals (missing perfusion data near the at-risk area, predicted core size, disagreement with the threshold core) mapped to a confidence by a small logistic regression, cross-fitted by fold on the 129 development cases and split into three equal bands.", "result": "The done check passed: wrong side of a cut-off, low band against high, 24 against 7; errors of 10 mL or more, 33 against 17; Spearman −0.345. Penumbra confidence: median 0.678, all 129 cases under 0.9.", "numbers": [ { "label": "Wrong side of a cut-off, low vs high band", "value": "24 against 7" }, { "label": "Errors of 10 mL or more, low vs high band", "value": "33 against 17" }, { "label": "Spearman", "value": "−0.345" } ], "status": "done", "decisions": [], "note": "Two problems were found: the missing-data share was inflated because the plain CT covers the whole head while the perfusion scan covers a slab, and two signals pointed against the design. The method was changed and rerun as EXP-31.", "source": "CHANGELOG 2026-11-16 (Vision A confidence: slab fix, checks, three versions); docs/CONFIDENCE_VISION_A.md" }, { "id": "EXP-31", "title": "Core confidence, three versions compared", "asked": "After the fix, which set of signals gives the most useful confidence?", "method": "Three versions cross-fitted on the same 129 development cases: v1 (the three first signals), v2 (v1 plus the distance to the nearest cut-off) and v3 (v2 plus the Tmax > 6 s volume).", "result": "All three versions pass the done check. Missing-data share: median 11.5% over the measurement area, against 32.2% over all plain-CT tissue (EXP-30). v2 was chosen: wrong side of a cut-off in the low band against the high band 26 / 7 (v1 24 / 7, v3 26 / 7).", "numbers": [ { "label": "Missing-data share, median", "value": "11.5%" }, { "label": "v2: wrong side, low / high band", "value": "26 / 7" }, { "label": "Penumbra confidence, median", "value": "0.885" }, { "label": "Penumbra confidence under 0.9", "value": "94 of 129" } ], "status": "done", "decisions": [ "DEC-27" ], "note": "The choice was made after seeing all three versions on the same cases, and v1 and v2 differ by 2 cases. In every version all 7 high-band cut-off errors are under-calls, so complete misses still get high confidence. DEC-27 later replaced v2 with the cut-off distance alone (EXP-32).", "source": "CHANGELOG 2026-11-16 (Vision A confidence: EXP-31 results, v2 chosen); docs/CONFIDENCE_VISION_A.md" }, { "id": "EXP-32", "title": "Core confidence from the cut-off distance alone", "asked": "Does one signal, the distance to the nearest decision cut-off, do as well as the four-signal version?", "method": "Cross-fitted by fold on the same 129 development cases, scored by AUC for landing on the wrong side of a cut-off (how well the score ranks wrong above right; 0.5 is chance) and by Brier score (squared error of the probability; lower is better).", "result": "AUC 0.736 for the distance alone against 0.722 for v2 (v2 minus distance −0.013, 95% interval −0.048 to +0.019); Brier 0.188 against 0.193 (no signal 0.222). The two are equivalent, and the distance needs one input, so it is used.", "numbers": [ { "label": "AUC, distance alone", "value": "0.736" }, { "label": "AUC, v2", "value": "0.722" }, { "label": "Brier, distance alone", "value": "0.188" }, { "label": "Brier, v2", "value": "0.193" }, { "label": "Brier, no signal", "value": "0.222" } ], "status": "done", "decisions": [ "DEC-27" ], "note": "Chosen for simplicity after seeing the results. A 0.755 quoted at review was the raw, unfitted ranking and is corrected to 0.736. 30 cases have a core under 1 mL: they get median confidence 0.90, but only 25 are right, so the contract carries a warning that a missed infarct cannot be ruled out.", "source": "CHANGELOG 2026-11-15 (DEC-27 confidence); docs/CONFIDENCE_VISION_A.md (Decision: DEC-27); plan §2 DEC-27" } ] }, { "id": "vision-b", "title": "Vision B: the bleed check", "summary": "This strand asked whether a ResNet-50 trained on RSNA scans can rule out a brain bleed, the one finding that stops treatment outright. On RSNA's test split, opened once, its sensitivity (share of bleeds caught) is 0.962 and its specificity (share of clean scans left alone) 0.885. On ISLES'24 stroke scans, presumed bleed-free, it flags 68 of 129: a domain shift traced to brain tissue rather than the skull base, which no simple rule and no general biomedical model fixed.", "experiments": [ { "id": "EXP-5", "title": "Bleed detector dummy run on Kaggle", "asked": "Does the RSNA split keep the natural bleed rate, and how fast does training run?", "method": "A dummy on a Kaggle T4: the patient-level split check, loader and GPU speed, 150 real training steps and scoring on a 300-scan validation sample.", "result": "The scan bleed rate is 0.4058 in validation and 0.4119 in test, against 0.4085 overall. Real training ran at 158 slices per second; a balanced epoch is 172,618 slices, about 18.2 minutes of training.", "numbers": [ { "label": "Bleed rate, validation", "value": "0.4058" }, { "label": "Bleed rate, test", "value": "0.4119" }, { "label": "Bleed rate, overall", "value": "0.4085" } ], "status": "dummy run only", "decisions": [ "DEC-22" ], "note": "Its AUCs (slice 0.805, scan 0.730) only show the pipeline runs; they are not results.", "source": "CHANGELOG 2026-11-04 (EXP-5 Vision B dummy; EXP-6 sized)" }, { "id": "EXP-6", "title": "ResNet-50 bleed detector on RSNA", "asked": "How well does a slice classifier detect any intracranial bleed, per scan, on RSNA validation?", "method": "A pretrained ResNet-50 trained for 10 epochs on a Kaggle T4 on all labelled RSNA data. A scan's score is the mean of its top two slices, calibrated with Platt scaling so the output is a probability.", "result": "Validation (2,166 scans): slice AUC 0.975 and scan AUC 0.977 (AUC: how well scores rank bleeds above non-bleeds; 1 is perfect). ECE (average gap between stated probability and observed frequency) 0.010. At the threshold 0.278 set for 95% sensitivity, sensitivity is 0.951 and specificity 0.917.", "numbers": [ { "label": "Scan AUC, validation", "value": "0.977" }, { "label": "Slice AUC", "value": "0.975" }, { "label": "ECE", "value": "0.010" }, { "label": "Sensitivity at 0.278", "value": "0.951" }, { "label": "Specificity at 0.278", "value": "0.917" } ], "status": "done", "decisions": [ "DEC-15", "DEC-16", "DEC-17", "DEC-22" ], "note": "The 0.278 threshold was set on the same split, so its specificity is optimistic; DEC-28 later replaced it with 0.175. The test split stayed closed until EXP-60.", "source": "CHANGELOG 2026-11-04 (EXP-6 Vision B; EXP-11 Haiku pass …); plan §7 Phase 2" }, { "id": "EXP-16", "title": "Bleed detector on ISLES'24: domain-shift check", "asked": "How often does the RSNA-trained detector flag a bleed on ISLES'24 scans, which should mostly have none?", "method": "All 129 development plain CTs, reformatted the way the RSNA data were stored (5 mm slabs, three CT windows) and scored with EXP-6.", "result": "At the provisional threshold 0.278, 45 of 129 were called positive on 5 mm slabs: a presumed false-positive rate of 34.9% (95% Wilson CI 27.2–43.4%). On native thin slices: 61 of 129.", "numbers": [ { "label": "Flagged at 0.278", "value": "45 of 129 (34.9%)" }, { "label": "95% Wilson CI", "value": "27.2–43.4%" }, { "label": "Flagged on native thin slices", "value": "61 of 129" }, { "label": "Flagged at 0.175", "value": "68 of 129 (52.7%)" } ], "status": "done", "decisions": [ "DEC-28" ], "note": "ISLES'24 has no bleed labels, so positives are presumed false positives. The 68 of 129 at 0.175 was counted later from the same per-case scores.", "source": "CHANGELOG 2026-11-12 (review of PR #3, EXP-16 recorded; review of PR #2); plan §7 Phase 3, Vision B" }, { "id": "EXP-17", "title": "Choosing the bleed threshold", "asked": "What threshold gives high sensitivity (bleeds caught) at an honest estimate of specificity (clean scans left alone)?", "method": "RSNA validation: 2,166 scans, 1,894 patients, 879 bleed scans. The threshold is chosen on one half of the patients and measured on the other, then swapped, over 20 seeds (cross-fitting), so it is not measured on the scans that set it.", "result": "Proposed 0.175 (96% target): sensitivity 0.961 (95% CI 0.946–0.972), specificity 0.878 (0.859–0.895). Cross-fitted at a 95% target: sensitivity 0.945, specificity 0.914, below 95% in 11 of 20 seeds.", "numbers": [ { "label": "Threshold", "value": "0.175" }, { "label": "Sensitivity", "value": "0.961 (95% CI 0.946–0.972)" }, { "label": "Specificity", "value": "0.878 (0.859–0.895)" }, { "label": "Missed bleeds per 100 bleeds", "value": "3.9" }, { "label": "False alarms per 100 non-bleeds", "value": "12.2" } ], "status": "done", "decisions": [ "DEC-28" ], "note": "On ISLES'24 the threshold would flag 68 of 129 presumed bleed-free scans (45 at 0.278, 87 at a stricter 0.125). DEC-28 adopted 0.175 because a false 'no bleed' is the worst failure.", "source": "CHANGELOG 2026-11-12 (review of PR #2, EXP-17 recorded); plan §2 DEC-28; plan §7 Phase 3, Vision B" }, { "id": "EXP-28", "title": "RSNA slice spacing and field of view", "asked": "Are RSNA scans really 5 mm thick, and how wide is their field of view?", "method": "The shard index of 17,398 RSNA training scans on Kaggle; no images read.", "result": "Slice spacing median 5.037 mm; 11,475 scans (66.0%) within 0.25 mm of the documented 5 mm. Field of view median 250.0 mm, p5–p95 216.0–253.3 mm.", "numbers": [ { "label": "RSNA training scans", "value": "17,398" }, { "label": "Slice spacing, median", "value": "5.037 mm" }, { "label": "Within 0.25 mm of 5 mm", "value": "66.0%" }, { "label": "Field of view, median", "value": "250.0 mm" } ], "status": "done", "decisions": [], "note": "The index holds the gap between slice centres, not the slice thickness, which RSNA's files do not carry.", "source": "CHANGELOG 2026-11-14 (Vision B: RSNA slice spacing and field of view)" }, { "id": "EXP-29", "title": "ISLES'24 rerun with the field of view matched", "asked": "Does a mismatch in image scale explain the false alarms on ISLES'24?", "method": "The 129 development scans scored twice, paired: as in EXP-16, and cropped or padded to RSNA's 250 mm field of view.", "result": "With the field of view matched: 42 of 129 (32.6%, 95% CI 25.1–41.0%) against 45 of 129, with EXP-16 reproduced exactly; exact McNemar p = 0.65. At 0.175: 68 of 129 (52.7%, 95% CI 44.1–61.1%) unmatched against 64 of 129 (49.6%, 41.1–58.1%) matched, p ≈ 0.54.", "numbers": [ { "label": "Flagged at 0.278, matched", "value": "42 of 129 (32.6%)" }, { "label": "Flagged at 0.278, as in EXP-16", "value": "45 of 129" }, { "label": "McNemar p at 0.278", "value": "0.65" }, { "label": "Flagged at 0.175, matched vs unmatched", "value": "64 of 129 vs 68 of 129" } ], "status": "done", "decisions": [], "note": "Matching the field of view barely changes the rate, so it does not explain it. It does move 19 of 129 calls (14.7%) one way or the other, so many ISLES'24 scores sit close to the threshold.", "source": "CHANGELOG 2026-11-14 (Vision B: RSNA slice spacing and field of view) and 2026-11-15 (review of PR #6)" }, { "id": "EXP-35", "title": "Crop check and a read of borderline flags", "asked": "Did the field-of-view crop cut any head, and what do the scans flagged only at the lower threshold show?", "method": "The 33 ISLES'24 scans over 250 mm, and the top two slabs of the 23 scans flagged at 0.175 but not at 0.278.", "result": "No head or brain voxel falls outside the centred 250 mm square on any of the 33 scans. Of the 23 borderline scans: skull-base slab 12, normal calcification 4, dense falx 3, dense vessels 1, nothing obvious 3; none was read as a bleed.", "numbers": [ { "label": "Large-field scans checked", "value": "33" }, { "label": "Borderline scans read", "value": "23" }, { "label": "Skull-base slab", "value": "12" }, { "label": "Read as a bleed", "value": "none" } ], "status": "done", "decisions": [], "note": "The slabs were read by Claude (an AI assistant), not a radiologist, at low resolution.", "source": "CHANGELOG 2026-11-16 (Phase 3: teammates' open tasks done from the Mac); plan §7 Phase 3, Vision B" }, { "id": "EXP-49", "title": "Item 2: slab rules tested on RSNA", "asked": "Can a simple rule on which slices count remove false alarms without losing bleeds?", "method": "Ten candidate scan-score rules fixed before any result (for example a 'gate' that leaves out slices showing little brain), selected on RSNA validation only by the same cross-fitting as EXP-17 (96% target, 20 seeds). Pre-registered in docs/ITEM2_FALSE_ALARMS.md.", "result": "All 10 candidates are eligible. The proposal is gate(0.55): specificity 0.8853 against 0.8812 with no rule, at sensitivity 0.9595 against 0.9597 (means over 20 seeds). Paired specificity gain on the main seed +0.0047 [+0.0015, +0.0086]: 6 fewer false alarms of 1,287 bleed-free scans.", "numbers": [ { "label": "Specificity, gate(0.55)", "value": "0.8853" }, { "label": "Specificity, no rule", "value": "0.8812" }, { "label": "Paired gain", "value": "+0.0047 [+0.0015, +0.0086]" }, { "label": "Fewer false alarms", "value": "6 of 1,287" } ], "status": "done", "decisions": [ "DEC-28" ], "note": "The gain is the best of 10 candidates on the same data, so slightly optimistic. Grad-CAM (a heat map of where the model looked) shows RSNA's false positives put 73% of their heat in brain tissue, as true positives do (85%).", "source": "CHANGELOG 2026-11-19 (Item 2: EXP-49's result); docs/ITEM2_FALSE_ALARMS.md (Results)" }, { "id": "EXP-50", "title": "Item 2: why ISLES'24 scans are flagged", "asked": "Does the chosen slab rule clear the false alarms on ISLES'24, and where in the head do they come from?", "method": "All 129 development scans scored under every candidate rule; Grad-CAM on the top two slabs of each flagged scan, read against density masks for brain, bone and other tissue.", "result": "With no rule the 68 flags of 129 (52.7%) are reproduced. gate(0.55) flags 64: it clears 4 and adds none (McNemar exact p = 0.125). The flagged scans' top slab is mostly in brain tissue (median relative brain fraction 0.87); Grad-CAM puts a median 76% of the heat in brain tissue, and only 2 of 136 slabs have more heat on bone than on brain.", "numbers": [ { "label": "Flags, no rule", "value": "68 of 129 (52.7%)" }, { "label": "Flags, gate(0.55)", "value": "64" }, { "label": "McNemar exact p", "value": "0.125" }, { "label": "Grad-CAM heat in brain, median", "value": "76%" } ], "status": "done", "decisions": [ "DEC-28" ], "note": "A negative result: skull-base slabs explain only a few of the false alarms, and no slab rule fixes them. DEC-28 stands and gate(0.55) was not adopted. These are density masks and Grad-CAM, read by Claude, not by a radiologist.", "source": "CHANGELOG 2026-11-19 (Item 2: EXP-51 and EXP-50, the ISLES'24 read-out); docs/ITEM2_FALSE_ALARMS.md (Results)" }, { "id": "EXP-51", "title": "Item 2: rerun with the threshold export", "asked": "Does a rerun of EXP-49 that also saves each rule's threshold give the same selection?", "method": "EXP-49 rerun on Kaggle with the threshold export added.", "result": "Its selection is identical field by field. The no-rule full-split threshold is 0.174984 on EXP-6's calibration, exactly DEC-28; gate(0.55) has the same raw cut (0.6268).", "numbers": [ { "label": "No-rule threshold", "value": "0.174984" }, { "label": "gate(0.55) raw cut", "value": "0.6268" } ], "status": "done", "decisions": [ "DEC-28" ], "source": "CHANGELOG 2026-11-19 (Item 2: EXP-51 and EXP-50, the ISLES'24 read-out)" }, { "id": "EXP-60", "title": "The RSNA test split, opened once", "asked": "Does the chosen bleed detector and threshold hold on patients never used for any choice?", "method": "EXP-6 with the 0.175 threshold on the RSNA test split: 2,180 scans of 1,893 patients (bleed rate 0.4119), opened once after a dry run on validation reproduced EXP-6. Intervals from a patient bootstrap.", "result": "Scan AUC 0.9805 [0.9743, 0.9859], sensitivity 0.962 [0.948, 0.975], specificity 0.885 [0.867, 0.904], ECE 0.015: 3.8 missed bleeds per 100 bleeds and 11.5 false alarms per 100 non-bleeds. By subtype: epidural 27/28, intraparenchymal 521/534, intraventricular 368/370, subarachnoid 341/353, subdural 400/409.", "numbers": [ { "label": "Scan AUC", "value": "0.9805 [0.9743, 0.9859]" }, { "label": "Sensitivity", "value": "0.962 [0.948, 0.975]" }, { "label": "Specificity", "value": "0.885 [0.867, 0.904]" }, { "label": "ECE", "value": "0.015" }, { "label": "Missed bleeds per 100 bleeds", "value": "3.8" }, { "label": "False alarms per 100 non-bleeds", "value": "11.5" } ], "status": "done", "decisions": [ "DEC-22", "DEC-28", "DEC-31" ], "note": "Agrees with EXP-17's cross-fitted validation estimate (0.961, 0.878). The test split is not to be opened again. Run as the first task of Phase 4.", "source": "CHANGELOG 2026-11-19 (Phase 4, Task 1 done: EXP-60 …); plan §7 Phase 4" }, { "id": "EXP-82", "title": "BiomedCLIP against the ResNet-50, and subtype probes", "asked": "Does a frozen general biomedical model (BiomedCLIP) detect bleeds as well as the ResNet-50 trained on RSNA, and do the ResNet-50's features hold the bleed subtype?", "method": "RSNA validation (2,166 scans; 879 bleeds) on Kaggle. BiomedCLIP as a linear probe on a seeded 60,000-slice training sample and zero-shot with two fixed prompts; every model scored with the same top-2 aggregation and calibration. The method was fixed before any score.", "result": "Scan AUC: ResNet-50 0.977, BiomedCLIP linear probe 0.899, zero-shot 0.753. Specificity at 96% sensitivity: 0.882, 0.416, 0.162. Per-subtype linear probes on the ResNet-50's frozen features: scan AUC 0.91 (epidural, 31 scans) to 0.98 (intraventricular), against 0.77 to 0.93 for the 'any bleed' score used as that subtype's detector.", "numbers": [ { "label": "Scan AUC, ResNet-50", "value": "0.977" }, { "label": "Scan AUC, BiomedCLIP probe", "value": "0.899" }, { "label": "Scan AUC, BiomedCLIP zero-shot", "value": "0.753" }, { "label": "Specificity at 96% sensitivity", "value": "0.882, 0.416, 0.162" }, { "label": "Per-subtype probe AUC", "value": "0.91 to 0.98" } ], "status": "done", "decisions": [ "DEC-33", "DEC-34" ], "note": "BiomedCLIP is not fine-tuned, so this compares representations, not best achievable models; the operating point is in-sample on validation for every model alike; the AUCs have no intervals; the RSNA test split was not read. Report only: nothing feeds the pipeline.", "source": "CHANGELOG 2026-11-20 (EXP-76, EXP-82 and EXP-83 done …); docs/PHASE5_HANDOVER.md §D" }, { "id": "EXP-83", "title": "BiomedCLIP flags on ISLES'24", "asked": "Does switching to a general biomedical model remove the false alarms on ISLES'24?", "method": "The 129 development scans (presumed bleed-free) scored with the BiomedCLIP probe and zero-shot at their operating points, compared with the ResNet-50 by McNemar's test.", "result": "Flags on the 129 development scans: ResNet-50 68, probe 117 (McNemar 54 against 5, p < 0.0001), zero-shot 94 (40 against 14, p 0.0005).", "numbers": [ { "label": "Flags, ResNet-50", "value": "68" }, { "label": "Flags, BiomedCLIP probe", "value": "117" }, { "label": "Flags, BiomedCLIP zero-shot", "value": "94" } ], "status": "done", "decisions": [ "DEC-33", "DEC-34" ], "note": "At its operating point BiomedCLIP also flags most clean RSNA scans (58% probe, 84% zero-shot), so its ISLES'24 flags mostly reflect that lower specificity.", "source": "CHANGELOG 2026-11-20 (EXP-76, EXP-82 and EXP-83 done …); docs/PHASE5_HANDOVER.md §D" } ] }, { "id": "rulebook", "title": "The rulebook and the rule engine", "summary": "The rules are a rulebook of 47 passages written in the project's own words from the 2026 AHA/ASA guideline and the DAWN and DEFUSE-3 trials (DEC-18), with the same rules also written as code to give the expected decision for each case. Two tests asked whether a search step could fetch the right passages for a case. The best method found 8 of 15 key rules, so the reasoner is given the whole rulebook, about 4,600 tokens (DEC-19).", "experiments": [ { "id": "EXP-7", "title": "Can retrieval find the rules a case needs?", "asked": "If the reasoner were given only retrieved passages, would it get the rules each test case rests on?", "method": "The embedding model BAAI/bge-base-en-v1.5 on the Mac CPU; two decision queries per fixture (the vision findings in words plus the note), top 8 passages each, checked against the 15 rules the five fixtures rest on.", "result": "9 of the 15 rules were found; 5 were never returned, including the main ICA/M1 thrombectomy rows for two fixtures. The best of 12 query and passage formats found 11; whole-section expansion found 13, but only by returning 26–35 of the 47 passages.", "numbers": [ { "label": "Key rules found", "value": "9 of the 15" }, { "label": "Best of 12 formats", "value": "11" }, { "label": "Whole-section expansion", "value": "13" } ], "status": "done", "decisions": [ "DEC-19" ], "note": "The embedding model ranks the right family of rules first but cannot order rows that differ only in their numbers. Retrieval stays only for the verifier's candidate passages.", "source": "CHANGELOG 2026-11-04 (corpus: retrieval test, DEC-19); plan §2 DEC-19" }, { "id": "EXP-79", "title": "Retrieval re-tested with four methods", "asked": "Do lexical, hybrid or reranking methods find the key rules that dense retrieval missed?", "method": "The same 15 key fixture rules; four methods fixed before any score (top 8 per decision query), run locally: bge-base, BM25 (word matching), a BM25 + bge hybrid by reciprocal rank fusion, and the hybrid reranked by bge-reranker-base.", "result": "Key rules found in the top 8 (of 15): bge 5, BM25 6, the hybrid 8, the reranked hybrid 3; median best rank 12, 12, 8 and 14. No method fixes the problem, and DEC-19 stands.", "numbers": [ { "label": "bge-base (dense)", "value": "5" }, { "label": "BM25", "value": "6" }, { "label": "BM25 + bge hybrid", "value": "8" }, { "label": "Hybrid reranked", "value": "3" } ], "status": "done", "decisions": [ "DEC-19", "DEC-37" ], "note": "EXP-7's own queries were not saved; the rebuilt ones find 5 with bge, not EXP-7's 9 of 15, so the counts compare methods on the same queries, not with EXP-7. 15 rules from five fixtures; no method or setting was tuned, by design.", "source": "CHANGELOG 2026-11-20 (Phase 5 tasks E, G, H done; I drafted); docs/PHASE5_HANDOVER.md §G" } ] }, { "id": "reasoner", "title": "The reasoner: language models deciding", "summary": "This strand asked how reliably language models recommend thrombolysis and thrombectomy when every claim they make is checked. With the rulebook in the prompt, Claude Sonnet 5.5 and Qwen3.5-27B had 4.0% and 5.0% of claims flagged, Gemma 3 27B and MedGemma 27B 17.1% and 17.7%, and the rulebook cut false-content claims by 8 to 15 percentage points for every model. A LoRA-tuned 4B model flagged far fewer claims than its untuned base but under-recommended treatment.", "experiments": [ { "id": "EXP-8", "title": "Reasoner dummy: one fixture twice", "asked": "Does the model API accept the output format, and does a reply validate?", "method": "Claude Sonnet 5.5 with prompt v1 and the whole rulebook, on one hand-written test case (fixture fx02), twice.", "result": "0 format failures, and both runs got the decisions right (thrombolysis no, thrombectomy yes). The system prompt is 10,559 tokens, against an estimate of about 6,000.", "numbers": [ { "label": "Format failures", "value": "0" }, { "label": "System prompt", "value": "10,559 tokens" } ], "status": "dummy run only", "decisions": [ "DEC-19" ], "source": "CHANGELOG 2026-11-04 (EXP-8 reasoner dummy)" }, { "id": "EXP-9", "title": "Reasoner on the five fixtures", "asked": "Does the reasoner reach the expected decisions on the five hand-written test cases?", "method": "Claude Sonnet 5.5, prompt v1 with the whole rulebook; five fixtures, twice each.", "result": "0 format failures, and all 18 judged decisions match the expected ones.", "numbers": [ { "label": "Format failures", "value": "0" }, { "label": "Judged decisions right", "value": "18/18" } ], "status": "done", "decisions": [ "DEC-19" ], "note": "One fixture's thrombectomy decision accepts either answer. A read of some outputs found a likely real hallucination: a rule cited for a patient whose ASPECTS is unknown.", "source": "CHANGELOG 2026-11-04 (EXP-9 reasoner on the fixtures); plan §7 Phase 2" }, { "id": "EXP-36", "title": "Test requests before the paid runs", "asked": "Do the model routes return valid answers before the main run is paid for?", "method": "One request per model on fixture fx02, prompt v2 with the rulebook: Claude Sonnet 5.5 live at the batch settings, and Qwen3.5-27B through an API.", "result": "Both returned valid recommendations: Sonnet 5.5 in 20 s with 3,042 output tokens, Qwen3.5-27B in 99 s with 9,098 (7,950 thinking).", "numbers": [ { "label": "Sonnet 5.5", "value": "20 s" }, { "label": "Qwen3.5-27B", "value": "99 s" } ], "status": "done", "decisions": [ "DEC-24" ], "note": "A test run only, not a result.", "source": "CHANGELOG 2026-11-16 (C8: three decisions, the without-rulebook arm, first test requests)" }, { "id": "EXP-37", "title": "The main model comparison runs", "asked": "How often do four language models make untraceable claims, with and without the rulebook, on the same cases?", "method": "Claude Sonnet 5.5, Qwen3.5-27B, Gemma 3 27B and MedGemma 27B, each on 7 arms (with the rulebook: 5 fixtures, 150 synthetic scenarios, and 20 real development cases with predicted and with ground-truth vision; without it: the 5 fixtures, 75 of the scenarios and the 20 real cases), plus two Sonnet repeats; prompt v3, every answer checked by the verifier. The hallucination rate is the share of an answer's claims the verifier cannot trace to the scan findings, the note or the rulebook.", "result": "Pooled verifier rates at the first scoring, with / without the rulebook: Sonnet 4.5% / 19.0%, Qwen 6.2% / 18.2%, Gemma 3 27B 18.4% / 29.7%, MedGemma 17.1% / 22.8%. Format failures (answers that did not fit the required structure): 1 / 295, 55 / 295, 5 / 295 and 35 / 295. Sonnet's three runs on the same 20 real cases: 4.8%, 7.6%, 4.9%.", "numbers": [ { "label": "Sonnet 5.5, with / without rulebook", "value": "4.5% / 19.0%" }, { "label": "Qwen3.5-27B", "value": "6.2% / 18.2%" }, { "label": "Gemma 3 27B", "value": "18.4% / 29.7%" }, { "label": "MedGemma 27B", "value": "17.1% / 22.8%" }, { "label": "Sonnet run to run", "value": "4.8%, 7.6%, 4.9%" } ], "status": "done", "decisions": [ "DEC-24", "DEC-29" ], "note": "These are pooled, unadjudicated numbers without intervals, from before the rescoring under metric points 19 and 20; the table with intervals is EXP-42. Qwen's first 295 answers were invalid (the provider had dropped the system prompt) and were set aside and redone. Without-rulebook rates count content only.", "source": "CHANGELOG 2026-11-16 (EXP-37 finished …; EXP-37, the main paid run: fixes made while it ran); plan §7 Phase 3, NLP A" }, { "id": "EXP-38", "title": "Reasoner prompt v1 against v2", "asked": "Does a prompt rewritten from the first verifier findings reduce errors?", "method": "The 5 fixtures with the rulebook, Claude Sonnet 5.5, verified with the paid runs' settings.", "result": "Both 10/10 decisions right, with no format failures or omissions. v2 writes 154 claims against v1's 339, with 13 flagged (8.4%) against 27 (8.0%); wrong numbers, invented findings and faulty derivations 0 against 9.", "numbers": [ { "label": "Decisions right, v1 and v2", "value": "10/10" }, { "label": "Claims, v1 / v2", "value": "339 / 154" }, { "label": "Flagged, v1 / v2", "value": "27 (8.0%) / 13 (8.4%)" }, { "label": "Critical, v1 / v2", "value": "8 (3 cases) / 9 (3 cases)" } ], "status": "done", "decisions": [], "note": "v2's 9 critical flags were checked afterwards: all 9 are verifier mis-scores, not reasoner errors.", "source": "CHANGELOG 2026-11-16 (EXP-38: reasoner prompt v1 vs v2; v2's critical flags checked)" }, { "id": "EXP-39", "title": "Reasoner prompt v3", "asked": "Does one changed sentence (cite only passages about the decision's own treatment) remove v2's citation flags?", "method": "v3 on the 5 fixtures with EXP-38's settings.", "result": "10/10 decisions, 142 claims, 5 flagged (3.5%) against v2's 13 (8.4%), 4 critical against 9, and no rule_ids citation flags against 4. No wrong numbers, invented findings or faulty derivations.", "numbers": [ { "label": "Decisions right", "value": "10/10" }, { "label": "Claims", "value": "142" }, { "label": "Flagged", "value": "5 (3.5%)" }, { "label": "Critical", "value": "4 (3 cases)" } ], "status": "done", "decisions": [], "note": "v3 is the prompt of the paid runs. On an unadjudicated reading none of its 4 critical flags is a reasoner error. Known limit: claims that chain two passages.", "source": "CHANGELOG 2026-11-16 (v2's critical flags checked; EXP-39: prompt v3); plan §7 Phase 3, NLP A" }, { "id": "EXP-40", "title": "Gold answers checked by the verifier", "asked": "Do the rule-derived model answers used for fine-tuning pass the full verifier?", "method": "30 gold answers built in code from the rule engine, one or more from each pair of decision routes, checked live by the verifier.", "result": "Round 1: 500 claims, 15 flagged (3.0%), 4 critical, all from three template faults. Round 2, after the fixes: 507 claims, 0 flagged, 45 of 45 decisions right.", "numbers": [ { "label": "Round 1 flagged", "value": "15 (3.0%)" }, { "label": "Round 2 flagged", "value": "0" }, { "label": "Decisions right", "value": "45 of 45" } ], "status": "done", "decisions": [ "DEC-30" ], "note": "Round 2 reused round 1's judge cache for unchanged claims, so it does not measure the judge's run-to-run variance.", "source": "CHANGELOG 2026-11-16 (EXP-37 finished; Item 4 steps 1–2: the gold-answer builder, EXP-40)" }, { "id": "EXP-42", "title": "The hallucination table (C9)", "asked": "With intervals and paired comparisons, how do the four models rank, and what does the rulebook change?", "method": "EXP-37's 30 cells rescored from saved traces with no model calls; 95% bootstrap intervals over cases (10,000 resamples). Headline: with the rulebook, predicted vision, each case once (5 fixtures + 150 scenarios + 20 real cases).", "result": "Hallucination rate, rescored under metric points 19 and 20: Sonnet 4.0% [3.3, 4.8], Qwen 5.0% [4.2, 5.8], Gemma 17.1% [15.7, 18.6], MedGemma 17.7% [15.9, 19.5]; Qwen minus Sonnet is +1.0 pts [−0.1, +2.1], so not clearly different. The rulebook lowers content errors (false content, not citation slips) for every model: Sonnet −14.9 pts [−16.6, −13.2], Qwen −12.9 [−14.7, −11.1], Gemma −13.0 [−15.3, −10.7], MedGemma −8.0 [−10.4, −5.4].", "numbers": [ { "label": "Sonnet 5.5", "value": "4.0% [3.3, 4.8]" }, { "label": "Qwen3.5-27B", "value": "5.0% [4.2, 5.8]" }, { "label": "Gemma 3 27B", "value": "17.1% [15.7, 18.6]" }, { "label": "MedGemma 27B", "value": "17.7% [15.9, 19.5]" }, { "label": "Qwen minus Sonnet", "value": "+1.0 pts [−0.1, +2.1]" }, { "label": "Rulebook effect, content rate", "value": "−8.0 to −14.9 pts" } ], "status": "done", "decisions": [ "DEC-11", "DEC-24", "DEC-29" ], "note": "First scoring: 4.4%, 6.2%, 18.6% and 18.3%; the rescoring changed only mis-attributed-source flags, and content and critical rates are unchanged. These are verifier rates, which the human check (EXP-41) shows are upper bounds. Medical training (MedGemma against its base, Gemma) shows no lower hallucination rate.", "source": "CHANGELOG 2026-11-17 (C9: the hallucination table, EXP-42) and 2026-11-19 (EXP-41: the four labellers' sheets … rescore); plan §7 Phase 3, NLP B" }, { "id": "EXP-43", "title": "Training set for the small model", "asked": "Can a fine-tuning set be built from synthetic cases without reusing any evaluation case?", "method": "Synthetic scenarios from a new seed and new value grids, each with a rule-derived gold answer; 4 of the 16 key decision routes left out of training so they can test generalisation. No real patient data.", "result": "1,200 examples, 1,080 train and 120 validation, every gold answer passing its checks. Thrombolysis yes / no / undetermined: 280 / 393 / 527. Thrombectomy: 246 / 570 / 384. The held-out routes: 0.", "numbers": [ { "label": "Examples", "value": "1,200" }, { "label": "Train / validation", "value": "1,080 / 120" }, { "label": "Thrombectomy yes / no / undetermined", "value": "246 / 570 / 384" } ], "status": "done", "decisions": [ "DEC-30" ], "source": "CHANGELOG 2026-11-17 (DEC-30 approved; Item 4 step 3 … EXP-43)" }, { "id": "EXP-44", "title": "Fine-tuning dummy run", "asked": "Does the fine-tuning code run on the GPU, and how long will the full run take?", "method": "30 steps on 20 examples on a Colab A100: LoRA (training a small add-on to the model rather than all its weights) on Gemma 3 4B.", "result": "30 steps on 20 examples in 156 s. Peak GPU 13.7 GB. Validation loss 0.200, 0.130, 0.119. Projected full run 3.06 h.", "numbers": [ { "label": "Peak GPU memory", "value": "13.7 GB" }, { "label": "Projected full run", "value": "3.06 h" } ], "status": "dummy run only", "decisions": [ "DEC-30" ], "note": "The generated answer started as valid structured output, then repeated one claim until the token cap: a model trained only 30 steps.", "source": "CHANGELOG 2026-11-18 (Item 4 step 5: the dummy run EXP-44 and the fine-tune EXP-45)" }, { "id": "EXP-45", "title": "Fine-tuning Gemma 3 4B", "asked": "Can a 4-billion-parameter model be tuned on gold answers to do the task?", "method": "LoRA fine-tune of Gemma 3 4B on EXP-43 on a Colab A100: 270 steps (1,080 examples × 2 epochs ÷ 8), about 3 h of training.", "result": "Validation loss fell at every evaluation, from 0.092 at step 25 to 0.0045 at step 270. Generation check: 3 of 3 answers stopped by themselves and validated; 2 of 3 gave the gold verdicts.", "numbers": [ { "label": "Training steps", "value": "270" }, { "label": "Validation loss, final", "value": "0.0045" }, { "label": "Generation check", "value": "3 of 3 validated; 2 of 3 gave the gold verdicts" } ], "status": "done", "decisions": [ "DEC-30" ], "note": "The gold answers are templated, so a low loss is expected and says little by itself. The real evaluation is EXP-47 and EXP-48.", "source": "CHANGELOG 2026-11-18 (Item 4 step 5: the dummy run EXP-44 and the fine-tune EXP-45)" }, { "id": "EXP-46", "title": "Serving test for the small model pair", "asked": "Do the prompted and the tuned Gemma 3 4B both serve and answer validly?", "method": "One request per model on fixture fx02, both served with identical settings on a Colab A100.", "result": "Both models valid: the base answered in 11.6 s with 695 output tokens, the tuned model in 8.8 s with 557 output tokens.", "numbers": [ { "label": "Base model", "value": "11.6 s" }, { "label": "Tuned model", "value": "8.8 s" } ], "status": "done", "decisions": [ "DEC-30" ], "note": "A test run only, not a result.", "source": "CHANGELOG 2026-11-18 (Item 4 step 6: serving the small reasoner pair, notebook 55)" }, { "id": "EXP-47", "title": "The small model pair on the evaluation cases", "asked": "How do the prompted and the tuned Gemma 3 4B answer the same cases as the larger models?", "method": "Both models with prompt v3 and the rulebook on 5 fixtures, 150 scenarios and 20 real development cases with predicted vision (175 requests per model), served on a Colab A100; answers verified as in EXP-37.", "result": "All 350 requests answered; 349 of 350 valid. One base-model fixture hit the 8,000-token limit and counts as a format failure; the tuned model: 175 of 175.", "numbers": [ { "label": "Requests answered", "value": "350" }, { "label": "Valid", "value": "349 of 350" }, { "label": "Tuned model valid", "value": "175 of 175" } ], "status": "done", "decisions": [ "DEC-30" ], "note": "The rates and comparisons from these answers are reported as EXP-48.", "source": "CHANGELOG 2026-11-19 (Item 4 step 7 started: EXP-47 …; EXP-47 scored, EXP-48 the DEC-30 report)" }, { "id": "EXP-48", "title": "Does fine-tuning a small model help?", "asked": "Does the LoRA-tuned Gemma 3 4B hallucinate less than the same model merely prompted, especially on decision routes it never saw in training?", "method": "Paired comparison on the same 175 cases (EXP-47), with the 51 scenarios that take a held-out route as the main result; rescored under metric points 19 and 20 with no model calls.", "result": "Tuned minus base, rescored: −27.3 pts [−29.6, −25.1] over all 175 cases and −23.5 pts [−27.4, −19.6] on the 51 held-out-route scenarios (first scoring: 14.3% against 41.6%, and 17.4% against 42.0%). But the tuned model under-recommends treatment: thrombectomy yes caught 10/35 (3/18 on held-out routes), thrombolysis yes 24/42. Its critical rate (errors that would change a decision if believed) is higher by +1.9 pts [+0.4, +3.4].", "numbers": [ { "label": "Tuned minus base, all 175", "value": "−27.3 pts [−29.6, −25.1]" }, { "label": "Tuned minus base, held-out routes", "value": "−23.5 pts [−27.4, −19.6]" }, { "label": "Thrombectomy yes caught, tuned", "value": "10/35" }, { "label": "Critical rate, tuned minus base", "value": "+1.9 pts [+0.4, +3.4]" }, { "label": "Decisions right, base / tuned", "value": "83 of 249 / 193 of 251" }, { "label": "Pooled rate, base / tuned (README)", "value": "40.6% / 13.3%" } ], "status": "done", "decisions": [ "DEC-30" ], "note": "Plan §7 still gives the first-scoring figure for held-out routes (−24.6 pts [−28.6, −20.7]); the rescored figures are from the newer changelog entry and docs/PHASE5_TTESTS.md. The treatment bias is reported as a finding, not tuned away; its likely cause, the training set's base rate (20% thrombectomy yes), is unverified. The later t-tests (EXP-69) found the higher critical rate fragile: it holds for the pooled rate, not the per-case mean.", "source": "CHANGELOG 2026-11-19 (EXP-47 scored, EXP-48 the DEC-30 report; and the rescore under points 19 and 20); docs/PHASE5_TTESTS.md §3 and §7; plan §7 Phase 3; README" } ] }, { "id": "verifier", "title": "The verifier and the hallucination metric", "summary": "The verifier splits each answer into single claims and checks each against the scan findings, the note and the rulebook; a claim it cannot trace counts as a hallucination, in seven sub-types. This strand asked whether the verifier itself can be trusted. It catches planted errors (recall at least 0.80 for every sub-type), but people confirmed only 18 of 54 of its flags, so its rates are upper bounds, and swapping the judge for a non-Claude model changed 3 of 100 verdicts.", "experiments": [ { "id": "EXP-10", "title": "Verifier dummy on one fixture", "asked": "Do the claim extractor and the judge run end to end, and do planted errors get caught?", "method": "One fixture's hand-written clean ('gold') answer and its 14 planted errors, with Claude Sonnet 5.5 extracting the claims and Claude Haiku 5.5 judging rule claims.", "result": "Strict recall was 1.0 for sub-types 1, 2, 3, 4, 6 and 7 (2 seeds each). For faulty derivation (5) it was 0 strict and 1.0 lenient. 4 of the 22 gold claims were flagged.", "numbers": [ { "label": "Strict recall, six sub-types", "value": "1.0" }, { "label": "Faulty derivation, strict / lenient", "value": "0 / 1.0" }, { "label": "Gold claims flagged", "value": "4 of the 22" } ], "status": "dummy run only", "decisions": [], "note": "The flags came from the verifier, not the gold answer; three verifier fixes followed before EXP-11.", "source": "CHANGELOG 2026-11-04 (EXP-10 verifier dummy; two verifier fixes)" }, { "id": "EXP-11", "title": "Planted errors: choosing the judge", "asked": "Does the verifier catch known fabrications of every kind, and which judge model should it use?", "method": "5 gold answers and 70 hand-written planted errors, 2 per sub-type per fixture; the same extracted claims judged by Claude Haiku 5.5 and Claude Opus 5.5. The bar: recall of at least 0.80 for every sub-type and false flags of at most 0.10.", "result": "With the final code: lowest strict recall 0.80 (uncited rule) for both judges, wrong rule 0.9, the other five sub-types 1.0; false flags on gold 0.010 for Haiku against 0.019 for Opus. Both judges miss the same three seeds, so the misses come from the verifier, not the judge model.", "numbers": [ { "label": "Lowest strict recall, both judges", "value": "0.80" }, { "label": "False flags, Haiku 5.5", "value": "0.010" }, { "label": "False flags, Opus 5.5", "value": "0.019" }, { "label": "Planted errors", "value": "70" } ], "status": "done", "decisions": [ "DEC-20" ], "note": "The first Haiku pass, before code fixes, had faulty-derivation recall 0.1 and false flags 0.095 (10 of 105 gold claims). The gold answers were written in the verifier's style, so the 0.010 false-flag rate does not carry over to real answers (EXP-12).", "source": "CHANGELOG 2026-11-04 (EXP-11 Opus pass; second rescore; Haiku pass); plan §2 DEC-20" }, { "id": "EXP-12", "title": "First end-to-end run on the fixtures", "asked": "Does the whole chain run from test case to reasoner to verifier, and what does the verifier flag on real model output?", "method": "The five fixtures through Claude Sonnet 5.5 (prompt v1) and the verifier; later rescored several times from saved outputs and caches, with no model calls, as verifier faults were fixed.", "result": "First scoring: 371 claims, hallucination rate 0.235 (content 0.148), decisions right 1.000, format failures 0. After the verifier fixes: 0.132, then 0.080 (28/351), then 19 of 350 flags (0.054) with the third judge prompt (EXP-23).", "numbers": [ { "label": "First scoring", "value": "0.235" }, { "label": "After the first rescore", "value": "0.132" }, { "label": "After five code fixes", "value": "0.080 (28/351)" }, { "label": "With judge_v3", "value": "19 / 350 (0.054)" }, { "label": "Critical rate, corrected", "value": "0.014 (5/350)" } ], "status": "done", "decisions": [ "DEC-23" ], "note": "The first rate was not a result: most flags were verifier issues on real output. A critical rate first reported as 0.185 used the wrong denominator and was corrected to 0.014. Two unadjudicated readings of the 28 remaining flags disagree: one counted about 9 real reasoner errors and 19 verifier errors, a team member's independent labels 22 real reasoner errors and 6 verifier mistakes.", "source": "CHANGELOG 2026-11-04 (EXP-12 end to end: acceptance run), 2026-11-05 (rescores) and 2026-11-12 (free EXP-12 rescore; EXP-23)" }, { "id": "EXP-23", "title": "Re-check of verifier fixes and judge prompt v3", "asked": "Do the fixes and the new judge prompt keep the planted-error bar while removing false flags?", "method": "EXP-11 and EXP-12 re-scored under judge prompts v2 and v3 on Colab.", "result": "The EXP-11 bar is unchanged under both prompts (min recall 0.80, false flags 0.010). EXP-12 flags fall from 28 / 351 to 23 / 350 with judge_v2 and 19 / 350 with judge_v3 (rate 0.054, critical 0.009), with all 9 reasoner errors of the earlier reading still flagged.", "numbers": [ { "label": "EXP-11 min recall", "value": "0.80" }, { "label": "EXP-11 false flags", "value": "0.010" }, { "label": "EXP-12 flags, before / v2 / v3", "value": "28 / 351, 23 / 350, 19 / 350" } ], "status": "done", "decisions": [ "DEC-23" ], "note": "judge_v3's examples come from EXP-12, so that gain is in-sample.", "source": "CHANGELOG 2026-11-12 (Phase 3: fold jobs … verifier fixes); plan §2 DEC-23" }, { "id": "EXP-24", "title": "A cheaper claim extractor", "asked": "Can Claude Haiku 5.5 replace Sonnet as the claim extractor and keep the planted-error bar?", "method": "EXP-11 and EXP-12 re-extracted with Haiku 5.5 on Colab.", "result": "Bar not met. Strict recall for uncited rules was 0.30; every other sub-type 1.0; false flags 0.010. On EXP-12 it flags 17 of 321 claims, but misses 3 of the 9 reasoner errors.", "numbers": [ { "label": "Uncited-rule strict recall", "value": "0.30" }, { "label": "False flags", "value": "0.010" }, { "label": "Reasoner errors missed", "value": "3 of the 9" } ], "status": "done", "decisions": [ "DEC-24" ], "note": "So the extractor stays Claude Sonnet 5.5, and the comparison matrix was shrunk to fit the budget.", "source": "CHANGELOG 2026-11-12 (Phase 3: fold jobs … EXP-24); plan §2 DEC-24" }, { "id": "EXP-41", "title": "Human check of the verifier", "asked": "Do people agree with the verifier's verdicts, and with each other?", "method": "100 claims sampled from EXP-37's rulebook arms, stratified by model and verifier verdict; four labellers in two mixed pairs, 50 claims each, every claim labelled twice, blind to the model and the verdict. A third rater settled the 28 disagreements.", "result": "Human–human κ 0.27 (pairs 0.05 and 0.33; κ is agreement beyond chance, 1 is perfect). Verifier–human agreement on all 100: 0.61 with the verifier rescored under metric points 19 and 20 (0.48 at the first scoring); weighted by the sampling strata, 0.87. The verifier caught 18 of the 21 human-judged errors, and humans confirmed 18 of its 54 flags (content 14/33, citation 4/21), so its rates are upper bounds.", "numbers": [ { "label": "Human–human κ", "value": "0.27" }, { "label": "Verifier–human agreement", "value": "0.61" }, { "label": "Weighted by strata", "value": "0.87" }, { "label": "Human-judged errors caught", "value": "18 of the 21" }, { "label": "Verifier flags confirmed", "value": "18 of its 54" }, { "label": "Verifier-clean claims agreed", "value": "43/46" } ], "status": "done", "decisions": [], "note": "Metric points 19 and 20 came from these same labels, so the 0.61 is in-sample. Most disagreements between labellers were true-but-uncited claims. Population human error rates per model are not estimated: about 8 clean claims per model give intervals too wide to use.", "source": "CHANGELOG 2026-11-19 (EXP-41 done; Phase 3 closed) and 2026-11-17 (EXP-41: the human-labelling sample); plan §7 Phase 3, NLP B" }, { "id": "EXP-81", "title": "A judge from outside the Claude family", "asked": "Does the verifier's limited agreement with people come from which model judges?", "method": "EXP-41's 100 human-labelled claims re-judged by GPT-6 Luna through an API, with the same judge prompt, threshold and reasoning effort, from an empty judge cache so no Haiku verdict is reused. 469 judge calls.", "result": "The two judges flag the same 97 of 100 claims (κ 0.94). Agreement with the humans: Haiku 0.61 (weighted 0.87), Luna 0.62 (0.87). The 3 disagreements are all on Sonnet's claims: Haiku flagged and Luna did not, and the humans sided with Luna on 2.", "numbers": [ { "label": "Same flag", "value": "97 of 100" }, { "label": "Judge–judge κ", "value": "0.94" }, { "label": "Haiku vs humans", "value": "0.61" }, { "label": "Luna vs humans", "value": "0.62" } ], "status": "done", "decisions": [ "DEC-38", "DEC-34" ], "note": "Metric points 19 and 20 were adjusted on these same labels; about 25 claims per model, so per-model differences are small-sample; the code checks are shared by both judges, so this tests the judge, not the whole verifier. 3 claims are too few to conclude anything about judge bias.", "source": "CHANGELOG 2026-11-20 (EXP-81 done: task F, the non-Claude judge); docs/PHASE5_HANDOVER.md §F" } ] }, { "id": "heldout", "title": "The held-out test (Phase 4)", "summary": "Twenty ISLES'24 cases were kept closed until every model, prompt and analysis was fixed in writing (DEC-31), then opened once. The larger models' hallucination rates held up on them (Sonnet 3.2% against 3.3% on development cases), and the one wrong decision by the best model came from a false bleed flag the verifier could not see. Correctness rests on only 23 scored decisions, and a 20% change in core volume flipped none of them.", "experiments": [ { "id": "EXP-61", "title": "Held-out path checked on development cases", "asked": "Does the code that will build held-out contracts reproduce the development results exactly?", "method": "Before unlocking, on a Colab A100: the five fold checkpoints compared, the pre-registered rebuild of the nnU-Net plans, 10 development cases re-predicted by their validating fold, and all 129 development contracts rebuilt.", "result": "7 of 7 checks passed. The 10 re-predicted cases reproduce the recorded masks (8 identical, at most 3 voxels different, Dice ≥ 0.9998, volume within 0.01 mL). The held-out code rebuilds all 129 EXP-33 contracts with 0 differences apart from confidence.core, and Vision B reproduces the 68 flags (largest confidence difference 0.0037).", "numbers": [ { "label": "Checks passed", "value": "7 of 7" }, { "label": "Masks identical", "value": "8" }, { "label": "Contracts rebuilt", "value": "129" }, { "label": "Ensemble speed", "value": "3.0 s per case" } ], "status": "done", "decisions": [ "DEC-31" ], "note": "confidence.core differs by design: the mapping fitted on all 129 is in-sample on development cases. A first try stopped before reading anything, because a loose folder pattern matched an older run; runs are now named exactly.", "source": "CHANGELOG 2026-11-19 (Phase 4, Task 2 done: EXP-61 …); plan §7 Phase 4" }, { "id": "EXP-62", "title": "Held-out cases unlocked", "asked": "What do the vision models produce on the 20 cases kept closed since day one?", "method": "The 20 held-out ISLES'24 cases, cores from the ensemble of the five fold models (their probabilities averaged), everything else as for development cases; ground-truth contracts and notes built the same way as before. Nothing is retrained after this point.", "result": "4 of 4 checks passed. Bleed flagged 10/20; occlusion site unknown in 20/20; core median 1.85 mL predicted against 11.0 mL reference; 7 near-empty cores.", "numbers": [ { "label": "Bleed flagged", "value": "10/20" }, { "label": "Occlusion site unknown", "value": "20/20" }, { "label": "Core, median predicted", "value": "1.85 mL" }, { "label": "Core, median reference", "value": "11.0 mL" }, { "label": "Near-empty cores", "value": "7" } ], "status": "done", "decisions": [ "DEC-31", "DEC-5" ], "source": "CHANGELOG 2026-11-19 (Phase 4, step 2: held-out cases unlocked, EXP-62); plan §7 Phase 4" }, { "id": "EXP-63", "title": "Analysis code checked on development cases", "asked": "Does the pre-registered analysis code reproduce known numbers, and can a 20% core error change a decision at all?", "method": "The held-out analyses run on the 20 development real cases and all 129 development contracts; a supplementary rule-level check with the core volume multiplied by 0.8 and 1.2 on the 150 synthetic scenarios, added before unlocking.", "result": "It reproduces the development numbers exactly (Dice 0.215, absolute volume error 23.5 mL, 68/129 bleed flags). Core ×0.8 and ×1.2 change 0 of 258 rule decisions across the 129 development contracts; on the scenarios they change 9 of 300 (3.0%) and 10 of 300 (3.3%). On development cases Sonnet, Qwen and the tuned 4B each make 1 wrong decision, each from a false bleed flag the verifier misses.", "numbers": [ { "label": "Rule decisions changed, development contracts", "value": "0 of 258" }, { "label": "Scenarios, core ×0.8", "value": "9 of 300 (3.0%)" }, { "label": "Scenarios, core ×1.2", "value": "10 of 300 (3.3%)" } ], "status": "done", "decisions": [ "DEC-31" ], "note": "A code check on development cases, not a result. The scenario share depends on the scenario grid, so it shows that a core error can decide a case, not how often in practice.", "source": "CHANGELOG 2026-11-19 (Phase 4, Task 3: the D3 analyses, code and development check); docs/PHASE4_PREREGISTRATION.md §4" }, { "id": "EXP-65", "title": "The six models on the held-out cases", "asked": "How do all six language models answer the 20 held-out cases, with predicted and with ground-truth vision?", "method": "14 cells: six models × two vision arms, plus Sonnet with the core multiplied by 0.8 and 1.2; prompt v3 with the rulebook; every answer verified with the frozen settings.", "result": "Hallucination rate, predicted / ground-truth vision: Sonnet 5.5 3.2% / 5.3%, Qwen3.5-27B 6.3% / 3.6%, Gemma 3 27B 11.6% / 11.3%, MedGemma 27B 10.0% / 9.0%. Format failures: Qwen 9 / 2, MedGemma 2 / 6, the other two 0.", "numbers": [ { "label": "Sonnet 5.5, predicted / ground truth", "value": "3.2% / 5.3%" }, { "label": "Qwen3.5-27B", "value": "6.3% / 3.6%" }, { "label": "Gemma 3 27B", "value": "11.6% / 11.3%" }, { "label": "MedGemma 27B", "value": "10.0% / 9.0%" }, { "label": "Sonnet, core ×0.8 / ×1.2", "value": "3.3% / 4.6%" } ], "status": "done", "decisions": [ "DEC-31", "DEC-24", "DEC-30" ], "note": "The two 4B models' held-out rates are given in EXP-66's comparison (40.4% and 21.5% with predicted vision). 20 cases, so the rates are rough.", "source": "CHANGELOG 2026-11-20 (Phase 4, steps 3 and 4 done: EXP-65 runs, EXP-66 D3 analyses)" }, { "id": "EXP-66", "title": "Held-out results: correctness, error sources, sensitivity", "asked": "On unseen cases, are the decisions right, where do wrong ones come from, and does a 20% core error flip any?", "method": "The pre-registered analyses on EXP-65's answers. A decision is right if it matches the rule engine applied to the ground-truth contract; cases the rules cannot determine are excluded. A wrong decision is a vision error if the rules on the predicted contract are already wrong, otherwise a reasoning error.", "result": "Decisions right with predicted vision, on 23 scored decisions: Sonnet 22/23, Qwen 13/13 (9 answers missing), Gemma 3 27B 22/23, MedGemma 21/21, Gemma 3 4B 3/23, tuned 4B 21/23. Sonnet's 1 wrong decision is a vision error (a false bleed flag) that the verifier missed; the base 4B made 20 reasoning errors, all caught. Core ×0.8 and ×1.2 change 0/40 rule decisions and 0/40 of Sonnet's own answers.", "numbers": [ { "label": "Sonnet 5.5, decisions right", "value": "22/23" }, { "label": "Gemma 3 4B, decisions right", "value": "3/23" }, { "label": "Decisions flipped by ±20% core", "value": "0/40" }, { "label": "Held-out Dice (ensemble)", "value": "0.182" }, { "label": "Hallucination, held-out vs development: Sonnet", "value": "3.2% vs 3.3%" }, { "label": "Same: tuned 4B", "value": "21.5% vs 12.3%" } ], "status": "done", "decisions": [ "DEC-31" ], "note": "Correctness rests on 23 scored decisions: thrombectomy truth is 'no' in all 20 cases and thrombolysis is undetermined in 17, so it says little; the tuned model's bias towards 'no' happens to match. Held-out against development hallucination rates (unpaired): Qwen 6.3% vs 3.3%, Gemma 11.6% vs 10.9%, MedGemma 10.0% vs 7.9%, 4B 40.4% vs 35.1%. The verifier cannot see a vision error whose claims faithfully restate the contract.", "source": "CHANGELOG 2026-11-20 (Phase 4, steps 3 and 4 done: EXP-65 runs, EXP-66 D3 analyses); plan §7 Phase 4; README" } ] }, { "id": "explain", "title": "Explaining the models (SHAP)", "summary": "After the results were frozen, Shapley values were used to ask what each part of the pipeline relies on. In the rules, the occlusion site and the clock decide most treatments and the core volume rarely does; nnU-Net leans most on the Tmax map; and the bleed detector's false alarms on ISLES'24 follow the same pattern as its false alarms on its own data. These values measure reliance on an input, not what would happen if a model were retrained without it.", "experiments": [ { "id": "EXP-68", "title": "Which inputs decide the rules?", "asked": "In the rulebook itself, which patient and imaging inputs actually move a treatment decision?", "method": "Exact Shapley values (each input's fair share of the outcome, averaged over every combination of known and unknown inputs) over 13 input groups, 8,192 combinations per scenario, on the 150 synthetic scenarios.", "result": "Thrombectomy turns mostly on the occlusion site (mean |value| 0.57); time, NIHSS and pre-stroke mRS give 0.14 each, ASPECTS 0.06, the core 0.007. Thrombolysis turns on time (0.41), then the occlusion site (0.14) and NIHSS (0.13); the core and Tmax > 6 s give 0.06 and 0.05. Where the rules say yes, vision inputs carry 28% of thrombolysis's attribution and 50% of thrombectomy's.", "numbers": [ { "label": "Thrombectomy: occlusion site", "value": "0.57" }, { "label": "Thrombectomy: core volume", "value": "0.007" }, { "label": "Thrombolysis: time", "value": "0.41" }, { "label": "Thrombolysis: core volume", "value": "0.06" }, { "label": "Vision share where rules say yes", "value": "28% and 50%" } ], "status": "done", "decisions": [ "DEC-35", "DEC-34" ], "note": "A synthetic scenario set with even route coverage, not real patients; it measures the rules, not any model. 'Unknown' is not neutral for these rules.", "source": "CHANGELOG 2026-11-20 (Phase 5 additions: owner, and SHAP row 8 done, EXP-68); docs/PHASE5_SHAP_SURVEY.md §5" }, { "id": "EXP-70", "title": "What drives the four-signal core confidence?", "asked": "Inside the four-signal confidence (v2), which signal carries the score?", "method": "Exact SHAP values of v2's logistic mapping, with all 129 development cases as background.", "result": "The distance to the nearest cut-off carries 73% of the attribution, core size 14%, missing Tmax 12%, disagreement 1%.", "numbers": [ { "label": "Cut-off distance", "value": "73%" }, { "label": "Core size", "value": "14%" }, { "label": "Missing Tmax", "value": "12%" }, { "label": "Disagreement", "value": "1%" } ], "status": "done", "decisions": [ "DEC-35", "DEC-27" ], "note": "In-sample, since v2 was fitted on the same cases. It backs DEC-27's choice of the distance alone.", "source": "CHANGELOG 2026-11-20 (Phase 5 task B: SHAP rows 3, 6, 10 and 11 done); docs/PHASE5_SHAP_SURVEY.md §5" }, { "id": "EXP-71", "title": "Does the retriever read the numbers?", "asked": "Did retrieval fail (EXP-7) because the embedding model ignores numbers?", "method": "Partition SHAP over the passage tokens of the bge similarity, for the 12 key fixture passages each paired with a gold claim that cites it.", "result": "Number tokens carry at least their share of the similarity in 10 of the 12 passages (median 13% of the attribution on 10% of the tokens). So the model does not ignore numbers; sibling rules share the same numbers in other combinations.", "numbers": [ { "label": "Passages where numbers carry their share", "value": "10 of the 12" }, { "label": "Median attribution on number tokens", "value": "13%" }, { "label": "Share of tokens that are numbers", "value": "10%" } ], "status": "done", "decisions": [ "DEC-35", "DEC-19" ], "note": "EXP-7's own queries were not saved; the gold claims stand in, so ranks differ from EXP-7's.", "source": "CHANGELOG 2026-11-20 (Phase 5 task B: SHAP rows 3, 6, 10 and 11 done); docs/PHASE5_SHAP_SURVEY.md §5" }, { "id": "EXP-72", "title": "Pixel attributions for the false alarms", "asked": "Does a second attribution method agree with Grad-CAM about where ISLES'24's false alarms come from?", "method": "Expected gradients (a SHAP method for images) on the top 2 slabs of the 68 flagged ISLES'24 development scans, beside Grad-CAM on the same slabs.", "result": "The run reproduces EXP-50 (68 flags; Grad-CAM 76% in brain). SHAP: median 50% of the attribution in brain tissue and 26% in bone (Grad-CAM 9%); brain over bone in 127 of 136 slabs; median Spearman with Grad-CAM 0.50.", "numbers": [ { "label": "SHAP attribution in brain, median", "value": "50%" }, { "label": "Grad-CAM heat in brain", "value": "76%" }, { "label": "Brain over bone", "value": "127 of 136" }, { "label": "Spearman with Grad-CAM, median", "value": "0.50" } ], "status": "done", "decisions": [ "DEC-35" ], "note": "A second method supports the reading that the false alarms come from brain tissue, not skull-base slabs. SHAP weights bone edges more than Grad-CAM does.", "source": "CHANGELOG 2026-11-20 (Phase 5 task B: SHAP rows 3, 6, 10 and 11 done); docs/PHASE5_SHAP_SURVEY.md §5" }, { "id": "EXP-73", "title": "What kind of claim gets flagged?", "asked": "Which features of a claim, known before it is checked, go with the verifier flagging it?", "method": "A surrogate model (gradient boosting) of the verifier's flags on 27,472 claims of the six models' rulebook arms, cross-validated by case, explained with SHAP.", "result": "Out-of-fold AUC 0.92 [0.92, 0.93]. Rule claims (44% flagged) and non-numeric comparisons (26%) push towards a flag; restated imaging values (2%) push away. Model and claim kind carry 60% of the attribution.", "numbers": [ { "label": "Surrogate AUC", "value": "0.92 [0.92, 0.93]" }, { "label": "Rule claims flagged", "value": "44%" }, { "label": "Non-numeric comparisons flagged", "value": "26%" }, { "label": "Imaging values flagged", "value": "2%" } ], "status": "done", "decisions": [ "DEC-35" ], "note": "It explains the surrogate, not the verifier itself.", "source": "CHANGELOG 2026-11-20 (Phase 5 task B: SHAP rows 3, 6, 10 and 11 done); docs/PHASE5_HANDOVER.md §B" }, { "id": "EXP-74", "title": "What moves the small models' answers?", "asked": "Do the prompted and the tuned Gemma 3 4B respond to their inputs the way the rules do?", "method": "Exact Shapley values of six input groups on 24 synthetic scenarios, 64 versions each (1,280 requests per model on a Colab A100), for the rule engine and both models; the bleed finding is kept fixed.", "result": "The base model recommends thrombolysis in every version of every scenario, so no input moves that answer. The tuned model responds to its inputs: for thrombolysis it ranks time first, 26% (rules 43%); for thrombectomy it leans on the occlusion site 28% (rules 48%) and as much on thrombolysis contraindications 28% (rules 0%).", "numbers": [ { "label": "Tuned, thrombolysis: time", "value": "26% (rules 43%)" }, { "label": "Tuned, thrombectomy: occlusion site", "value": "28% (rules 48%)" }, { "label": "Tuned, thrombectomy: contraindications", "value": "28% (rules 0%)" }, { "label": "Complete scenarios, base / tuned", "value": "21 of 24 / 18" } ], "status": "done", "decisions": [ "DEC-35", "DEC-30" ], "note": "24 synthetic scenarios; a scenario with a format failure is dropped. The values show what moves the answer, not whether it is right; a hidden input is rendered as 'not recorded'. The full-input versions reproduce EXP-47's answers (41 of 42 and 36 of 36 decisions).", "source": "CHANGELOG 2026-11-20 (EXP-74 done: SHAP row 9 …); docs/PHASE5_HANDOVER.md §B" }, { "id": "EXP-75", "title": "Which scan inputs does nnU-Net rely on?", "asked": "Of its five inputs (plain CT and four perfusion maps), which does the segmentation model lean on?", "method": "Exact Shapley values over all 32 combinations of the five inputs on the 129 development cases (4,128 predictions on a Colab A100); an absent input is filled with a neutral value.", "result": "Mean |Shapley value| for Dice (and volume): Tmax 0.096, 38% (12.2 mL, 35%); plain CT 0.059, 23% (8.6 mL, 24%); CBF 0.039, 16% (6.5 mL, 19%); MTT 0.032, 13% (4.1 mL, 12%); CBV 0.026, 10% (3.8 mL, 11%). With all inputs Dice is 0.215; with none, 0.008.", "numbers": [ { "label": "Tmax", "value": "38%" }, { "label": "Plain CT", "value": "23%" }, { "label": "CBF", "value": "16%" }, { "label": "MTT", "value": "13%" }, { "label": "CBV", "value": "10%" }, { "label": "Wrong side of a cut-off, all inputs vs none", "value": "33% vs 35%" } ], "status": "done", "decisions": [ "DEC-35" ], "note": "Reliance, not ablation: no model was trained without an input. The perfusion maps share information, which Shapley values split between them. The full combination reproduces every recorded volume within 0.02 mL.", "source": "CHANGELOG 2026-11-20 (EXP-75 done: SHAP row 1 …); docs/PHASE5_SHAP_SURVEY.md §5" }, { "id": "EXP-76", "title": "The bleed detector's three CT windows, on RSNA", "asked": "How much does each of the three CT windows (brain, subdural, bone) contribute to detecting a bleed?", "method": "Exact Shapley values over the 8 window combinations on every RSNA validation scan (2,166; 879 with a bleed), on Kaggle; an absent window is set to its RSNA training mean.", "result": "Of the scan AUC above chance (0.477): brain 0.320 (67%), subdural 0.108 (23%), bone 0.049 (10%). At the working threshold the brain window alone flags almost every scan (specificity 0.017), so the other two windows' job is to pull clean scans back down. On RSNA's own 152 false alarms they do not: mean values brain +0.44, subdural +0.06, bone −0.03.", "numbers": [ { "label": "Brain window", "value": "67%" }, { "label": "Subdural window", "value": "23%" }, { "label": "Bone window", "value": "10%" }, { "label": "Specificity, brain window alone", "value": "0.017" } ], "status": "done", "decisions": [ "DEC-35" ], "note": "This corrected EXP-77's first reading: 'the brain window alone flags all 129 ISLES'24 scans' is an effect of filling absent windows with their means, seen on RSNA too, not of ISLES'24. Reliance, not ablation; group means.", "source": "CHANGELOG 2026-11-20 (EXP-76, EXP-82 and EXP-83 done …); docs/PHASE5_SHAP_SURVEY.md §5" }, { "id": "EXP-77", "title": "The three CT windows, on ISLES'24", "asked": "Do ISLES'24's false alarms show the same window pattern as RSNA's?", "method": "The same Shapley analysis on the 129 ISLES'24 development scans, using the window means from EXP-76's sample.", "result": "All three windows reproduce the 68 flags. On the 68 flagged scans the brain window carries the attribution (+0.39) and subdural and bone net out near zero (+0.02, −0.02); on unflagged scans they pull the probability down (subdural −0.07, bone −0.10).", "numbers": [ { "label": "Flagged scans: brain window", "value": "+0.39" }, { "label": "Flagged scans: subdural, bone", "value": "+0.02, −0.02" }, { "label": "Unflagged scans: subdural, bone", "value": "−0.07, −0.10" } ], "status": "done", "decisions": [ "DEC-35" ], "note": "ISLES'24's false alarms look like RSNA's own false alarms: the domain shift changes how often this happens (68 of 129 against 152 of 1,287), not how. The count 'brain window alone flags 129' reflects how an absent window is filled (see EXP-76).", "source": "CHANGELOG 2026-11-20 (EXP-77 done: row 5, ISLES'24 half; and the correction under EXP-76); docs/PHASE5_SHAP_SURVEY.md §5" } ] }, { "id": "stats", "title": "Statistics, subgroups and reproducibility", "summary": "These report-only checks asked whether the headline findings survive classical tests, whether any patient group is served worse, and whether the work can be rerun. Of 35 t-tests, 12 hold after correction, confirming the rulebook effect, nnU-Net's overlap gain and the model ranking but not a volume gain or any subgroup difference. A 3.2 MB replay bundle rebuilds the hallucination table for the synthetic cases without any API key: all 20 cells match their reported rates exactly.", "experiments": [ { "id": "EXP-69", "title": "Student's t-tests beside the bootstrap results", "asked": "Do classical t-tests on per-case values agree with the intervals already reported?", "method": "35 tests fixed before any t statistic: paired where the same cases are on both sides, Welch otherwise, a rank test beside each, and Holm's correction (which guards against chance findings among many tests) over all 35 as one family.", "result": "12 hold after Holm: nnU-Net's Dice gain (T1, +0.104 [+0.077, +0.131]), the rulebook effect for all four models (−9 to −15 points), Gemma 3 27B, MedGemma, Gemma 3 4B and the tuned 4B above Sonnet, the tuned 4B below its base (−29 and −24 points), and BiomedCLIP's probe against the ResNet-50 (per-patient Brier score 0.116 against 0.044, +0.073 [+0.064, +0.081] over 1,894 patients). Not clear after Holm: the volume error (T2, −1.6 mL [−4.6, +1.4], p 0.29), every predicted-against-ground-truth vision comparison, Qwen against Sonnet, the tuned 4B's critical rate, and every held-out and subgroup comparison.", "numbers": [ { "label": "Tests run", "value": "35" }, { "label": "Clear after Holm", "value": "12" }, { "label": "T1, Dice gain", "value": "+0.104 [+0.077, +0.131]" }, { "label": "T2, volume error", "value": "−1.6 mL [−4.6, +1.4], p 0.29" }, { "label": "T9, Brier difference", "value": "+0.073 [+0.064, +0.081]" } ], "status": "done", "decisions": [ "DEC-36", "DEC-34" ], "note": "The data had already been seen before the tests were fixed, so this is not a pre-registration; the development-against-held-out tests (T7) are post hoc. The t-test and the bootstrap disagree in 2 of the 18 with a reported interval: Qwen minus Sonnet (t +1.3 points [+0.2, +2.4] against +1.0 [−0.1, +2.1]) and the tuned 4B's critical rate (t +0.9 points [−0.8, +2.5] against +1.9 [+0.4, +3.4]). An earlier changelog entry reported 34 of 35 tests with 11 clear; the 35th (T9) ran later the same day.", "source": "docs/PHASE5_TTESTS.md §7; CHANGELOG 2026-11-20 (EXP-76, EXP-82 and EXP-83 done; T9 run, all 35 t-tests) and (Phase 5 task C: EXP-69)" }, { "id": "EXP-78", "title": "The bleed detector exported to ONNX", "asked": "Does the bleed detector give the same answers in a portable format, and how fast is it on a laptop CPU?", "method": "EXP-6's ResNet-50 exported to ONNX (opset 17); parity checked on every 5 mm slab of 20 ISLES'24 development scans (601 slabs); latency on one laptop CPU.", "result": "94.0 MB against 94.3 MB for the PyTorch checkpoint. Largest probability difference 8.8 × 10⁻⁶; all 20 scans get the same call. About 25 ms per slab at batch 1 in either runtime; a batch of 32 takes 2.1 s with ONNX Runtime against 6.7 s with PyTorch.", "numbers": [ { "label": "Largest probability difference", "value": "8.8 × 10⁻⁶" }, { "label": "Same call", "value": "all 20 scans" }, { "label": "Batch of 32, ONNX Runtime", "value": "2.1 s" }, { "label": "Batch of 32, PyTorch", "value": "6.7 s" } ], "status": "done", "decisions": [ "DEC-37", "DEC-34" ], "note": "One machine; median of 30 runs. The weights stay private to the team under the data-use terms.", "source": "CHANGELOG 2026-11-20 (Phase 5 tasks E, G, H done; I drafted); docs/PHASE5_HANDOVER.md §H" }, { "id": "EXP-80", "title": "Subgroups: age, sex, centre, onset", "asked": "Do the models' errors differ by age band, sex, hospital centre or how the onset time is known?", "method": "From saved results only: Vision A error and Vision B false alarms on the 129 development cases; reasoner hallucination and correctness on the 150 scenarios and the 20 real development cases. Also two surrogate models of vision errors.", "result": "Vision A's Dice is lower at 80 and over (0.16 [0.10, 0.22] against 0.25 [0.19, 0.30]) and at centre 1 (0.18 against 0.29); its volume error is larger in women (31 [21, 42] against 16 [11, 23] mL). These are unadjusted and do not survive Holm's correction. Vision B flags 47–61% of every group, and no reasoner's rate differs by age band, sex or onset pattern.", "numbers": [ { "label": "Dice, 80 and over", "value": "0.16 [0.10, 0.22]" }, { "label": "Dice, under 80", "value": "0.25 [0.19, 0.30]" }, { "label": "Volume error, women", "value": "31 [21, 42] mL" }, { "label": "Volume error, men", "value": "16 [11, 23] mL" }, { "label": "Vision B flags per group", "value": "47–61%" } ], "status": "done", "decisions": [ "DEC-37", "DEC-34" ], "note": "129 development cases from two centres; no adjustment for infarct size; on the 20 real cases every group is too small to conclude anything. Surrogates: Vision A's wrong-side errors are predictable from what is known at presentation (AUC 0.73 [0.63, 0.82]); Vision B's false alarms are not (AUC 0.34 [0.24, 0.43]).", "source": "CHANGELOG 2026-11-20 (Phase 5 tasks E, G, H done; I drafted); docs/PHASE5_HANDOVER.md §E and §B" } ] }, { "id": "demo", "title": "The public demo", "summary": "The demo is a public web page (DEC-32) with the pipeline's story, a case explorer where every claim links to its source, the results as charts, and a scenario builder that runs the reasoner and verifier live. It serves 195 cases (5 fixtures, 150 scenarios, 20 development and 20 held-out cases, by alias only). In a load test with live runs switched off, 50 of 50 visitors got a result with 0 errors, and one live run on the deployed page took 51.6 s.", "experiments": [ { "id": "EXP-64", "title": "Live dummy for the demo", "asked": "How long does one live run take, with a cold and a warm prompt cache?", "method": "Five live scenario-builder runs covering five decision routes, with synthetic inputs: Claude Sonnet 5.5 with prompt v3 and the rulebook, then the verifier.", "result": "5 of 5 live runs succeeded. Times: 77 s, 45 s, 55 s, 68 s and 153 s. Sonnet agreed with the rules on every determined decision.", "numbers": [ { "label": "Live runs succeeded", "value": "5 of 5" }, { "label": "Fastest / slowest run", "value": "45 s / 153 s" } ], "status": "done", "decisions": [ "DEC-32" ], "note": "After it, the judge calls in live runs were made concurrent. No live answer is counted in any result.", "source": "CHANGELOG 2026-11-19 (Phase 4, Task 4: EXP-64, the Space's live dummy); CHANGELOG 2026-11-20 (the Space's key and daily cap; step 5)" } ] }, { "id": "clinician", "title": "Clinician review", "summary": "Two clinicians are to review the rulebook and test cases and give written opinions on nine of the system's answers, each working independently (DEC-34). The pack is fixed and approved but had not been sent when this record was compiled, so there are no answers. Whatever they say will be expert opinion from two people, never clinical validation.", "experiments": [ { "id": "EXP-67", "title": "Clinician review pack", "asked": "Do two clinicians agree that the rulebook reads the guideline correctly, and how do they judge the system's answers?", "method": "Two instruments answered independently: a 76-item rulebook and test-case review (69 scored items) and a written interview on nine cases chosen by a seeded rule over fixed strata (right, flagged, wrong-and-caught, wrong-and-missed). Agreement will be reported as percent agreement, Cohen's κ and PABAK.", "result": "", "numbers": [ { "label": "Review items", "value": "76" }, { "label": "Scored items", "value": "69" }, { "label": "Interview cases", "value": "nine" } ], "status": "planned", "decisions": [ "DEC-34" ], "note": "The pack is fixed and approved; not yet sent, no answers. Under the freeze nothing is re-run because of an answer: a rule error a clinician finds is counted and reported as a limitation, not fixed.", "source": "docs/PHASE5_HANDOVER.md §A; docs/CLINICIAN_PACK.md; CHANGELOG 2026-11-20 (Phase 5 additions, task A: the clinician pack)" } ] } ], "decisions": [ { "id": "DEC-1", "title": "The project", "text": "The group project is the Stroke Decision Copilot, replacing an earlier project called Bias Lens." }, { "id": "DEC-2", "title": "Datasets", "text": "ISLES'24 (Zenodo) for perfusion, infarct and clinical variables; RSNA Intracranial Haemorrhage (Kaggle) for bleed detection; the AHA/ASA guideline and the DAWN and DEFUSE-3 papers for the rules." }, { "id": "DEC-3", "title": "Compute and storage", "text": "Compute is Google Colab Pro with a shared Google Drive folder for storage. Training data is copied to Colab local disk as one tarball per session." }, { "id": "DEC-4", "title": "Contract first", "text": "The vision-output schema is frozen at the end of Phase 0. The language side builds against hand-written fixtures until Phase 3." }, { "id": "DEC-5", "title": "Evaluation split fixed on day one", "text": "5 folds + ~20 held-out cases. Held-out cases are untouched until Phase 4." }, { "id": "DEC-6", "title": "Team onboarding", "text": "The lead builds Phase 0–1 alone, then sends teammates instructions when assigning them work, with no fixed onboarding date." }, { "id": "DEC-7", "title": "What 'penumbra' means", "text": "The penumbra volume is the total Tmax > 6 s volume, core included; salvageable tissue is the mismatch volume." }, { "id": "DEC-8", "title": "Which guideline", "text": "The rule baseline is the 2026 AHA/ASA guideline plus DAWN and DEFUSE-3, not the 2018/2019 guideline." }, { "id": "DEC-9", "title": "ASPECTS field", "text": "The contract carries a nullable ASPECTS field. No ASPECTS model is built: it is null from the real pipeline and set only in fixtures and synthetic scenarios." }, { "id": "DEC-10", "title": "Claims with several sources", "text": "A rationale claim can cite several sources and be marked 'derived'. The hallucination definition changes to match." }, { "id": "DEC-11", "title": "Primary evaluation on synthetic scenarios", "text": "The primary hallucination evaluation uses 100–200 synthetic scenarios; held-out cases are the realism check. Added: seeded errors, a no-retrieval arm, and a ground-truth against predicted vision arm." }, { "id": "DEC-12", "title": "ISLES'24 version", "text": "ISLES'24 uses the latest Zenodo record (17652035), not 16748089, because the data dictionary was only added after that version." }, { "id": "DEC-13", "title": "Slim data format", "text": "Slim format v1 gains the native plain-CT and Circle of Willis volumes." }, { "id": "DEC-14", "title": "Rounding the mismatch ratio", "text": "The mismatch ratio is rounded to two decimals, from the rounded volumes." }, { "id": "DEC-15", "title": "Meaning of the bleed confidence", "text": "The haemorrhage confidence means the calibrated probability that a haemorrhage is present." }, { "id": "DEC-16", "title": "All RSNA data", "text": "Vision B uses all labelled RSNA data, with no subset." }, { "id": "DEC-17", "title": "RSNA stays on Kaggle", "text": "Vision B preprocesses once to 320 px JPEG tar shards on Kaggle, then trains there, as an exception to DEC-3; only checkpoints and metrics move to Drive." }, { "id": "DEC-18", "title": "A rulebook in our own words", "text": "The corpus is a rulebook in the project's own words, one passage per criterion, each pointing to its source, section, recommendation and page. It keeps the numeric criteria as facts and copies no wording, because the publishers allow no reproduction or AI use of their text." }, { "id": "DEC-19", "title": "The whole rulebook in the prompt", "text": "The reasoner gets the whole rulebook in its prompt, not retrieved passages; retrieval stays only for the verifier's candidate passages. Reason: EXP-7 found top-k retrieval missed 6 of 15 key fixture rules. The rulebook is about 4,600 tokens, so sending it whole is cheap." }, { "id": "DEC-20", "title": "The verifier's models", "text": "The verifier's judge is Claude Haiku 5.5 with τ = 0.6; the claim extractor is Claude Sonnet 5.5. Chosen by the EXP-11 bake-off: both judges reach the seeded-error bar with the same recall, Haiku with fewer false flags (0.010 vs 0.019)." }, { "id": "DEC-21", "title": "A rented GPU for the long runs", "text": "Vision A training (nnU-Net on ISLES'24) runs on a GPU pod rented per job, one RTX 4090, as an amendment to DEC-3. Every run ends by destroying its pod. Vision A trains 300 epochs per fold, all five folds, fold 0 first." }, { "id": "DEC-22", "title": "The RSNA split", "text": "The RSNA split is by patient, 80/10/10 train/validation/test, cut separately for patients with and without a bleed, so validation and test keep the natural bleed rate (EXP-6: 0.4058 and 0.4119 against 0.4085). The test split stays closed until the final Vision B model is chosen." }, { "id": "DEC-23", "title": "Judge prompt v3", "text": "The judge prompt is judge_v3, amending DEC-20 (model, effort and τ unchanged). EXP-23: the EXP-11 seeded-error bar is unchanged (min recall 0.80, false flags 0.010), and EXP-12 drops from 23 to 19 flags of 350 against judge_v2; its examples come from EXP-12, so that gain is in-sample." }, { "id": "DEC-24", "title": "The models compared", "text": "The reasoner models compared are Claude Sonnet 5.5 (frontier API), Qwen3.5-27B (open mid-size), MedGemma 27B text (small medical) and Gemma 3 27B, MedGemma's base, so medical training is compared in a pair. After EXP-24 (Haiku failed the uncited-rule bar, recall 0.30) the extractor stays Claude Sonnet 5.5 and the matrix shrinks: per model, the 5 fixtures with and without the rulebook, 150 scenarios with the rulebook and 75 without, and 20 real development cases." }, { "id": "DEC-25", "title": "nnU-Net feeds the core", "text": "The core that feeds the contract is the nnU-Net prediction, not the rCBF < 30% threshold; each development case uses the fold model that did not train on it. On EXP-25 nnU-Net is no worse on any contract measure and clearly better at locating the infarct, but its volume is not measurably better." }, { "id": "DEC-26", "title": "Two-stage Vision A", "text": "Vision A is two-stage: the core is the nnU-Net final-infarct prediction used as the core proxy, and the penumbra is the Tmax > 6 s rule. No core or penumbra heads, because ISLES'24 labels only the final infarct, and no volume calibration." }, { "id": "DEC-27", "title": "Core confidence is the cut-off distance", "text": "The core confidence is the distance from nnU-Net's core volume to the nearest decision cut-off (21, 31, 51, 70 mL), mapped to a confidence by a small logistic regression cross-fitted by fold. It matches the four-signal version on EXP-32 (AUC 0.736 vs 0.722), and near-empty predictions carry the warning 'Little or no core predicted: a missed infarct cannot be ruled out'." }, { "id": "DEC-28", "title": "Bleed threshold 0.175", "text": "The Vision B threshold for the contract is 0.175 on the calibrated scale, replacing EXP-6's provisional 0.278. EXP-17 chose it on RSNA validation: sensitivity 0.961 (95% CI 0.946–0.972), specificity 0.878 (0.859–0.895). On ISLES'24 it flags 68 of 129 scans, so the predicted-vision arm will refuse treatment in about half the real cases." }, { "id": "DEC-29", "title": "The without-rulebook arm", "text": "The without-rulebook arm gives the reasoner the same prompt with the rulebook removed; rules come from the model's own knowledge and are cited with a fixed placeholder source. The verifier scores content only in this arm. Limits: a true rule the 47-passage corpus does not hold counts as a wrong rule, and the arms' attribution numbers are not comparable." }, { "id": "DEC-30", "title": "A fine-tuned small reasoner", "text": "Two rows are added to the model list: Gemma 3 4B with prompt v3 and the rulebook, and the same model with a LoRA adapter trained on gold answers. Training scenarios are synthetic, with 4 of the 16 key routes left out of training; no real case and no held-out case is used. The held-out routes are the main result." }, { "id": "DEC-31", "title": "Phase 4 pre-registration", "text": "Fixed before any held-out case is opened: held-out cores come from the five fold models' nnU-Net ensemble; all six reasoner models run with prompt v3 and the rulebook on predicted and ground-truth vision arms, 240 requests; correctness, error propagation and core ×0.8 and ×1.2 are analysed as written. Nothing is retrained after unlocking." }, { "id": "DEC-32", "title": "A public demo", "text": "The demo is a public Hugging Face Space: a scrolling story of the pipeline, a case explorer where every claim is clickable and its source lights up, the results as interactive charts, and one live element, a scenario builder with structured inputs only. Cases appear by alias, never by ID, and no live answer is counted in any result. Amended on 20 Nov 2026: the same parts sit on three pages (the story, the live run and the record); the story follows a scripted case, with a real case by alias for the scan images only; and a stored case run unchanged falls back to its own precomputed answer." }, { "id": "DEC-33", "title": "BiomedCLIP comparison", "text": "Report only: BiomedCLIP zero-shot and as a linear probe against EXP-6's ResNet-50 on RSNA validation, with ISLES'24 development flag counts, and per-subtype linear probes on EXP-6's frozen features. The test split is never read." }, { "id": "DEC-34", "title": "Phase 5 additions are report only", "text": "Phase 5 additions are report only; DEC-31's freeze holds (nothing retrained, nothing feeds the contract, the pipeline or the Space). The first is the clinician review (EXP-67): two clinicians, two instruments answered independently. Their answers are expert opinion, never clinical validation." }, { "id": "DEC-35", "title": "The SHAP analyses", "text": "Report-only SHAP analyses of Vision A's input channels and confidence, Vision B's windows and pixels, the rule engine, the Gemma 3 4B pair, the retriever and three surrogates. The values measure reliance on an input, not an ablation, because no model was trained without it. No held-out case is used." }, { "id": "DEC-36", "title": "Student's t-tests", "text": "35 tests beside the existing results, which stay as they are. The case is the unit; tests are paired where the same cases are on both sides and Welch otherwise, with a rank test beside every one and Holm's correction over the whole list as one family. The data had already been seen; the list was fixed before any t statistic." }, { "id": "DEC-37", "title": "Subgroups, retrieval re-test, ONNX, reproducibility", "text": "Report only: subgroups from saved results by age band, sex, centre and onset pattern; EXP-7's retrieval check repeated with BM25, a hybrid and a reranker (DEC-19 stands whatever the result); EXP-6's ResNet-50 exported to ONNX. The replay bundle is a release asset and the code is MIT-licensed." }, { "id": "DEC-38", "title": "A non-Claude judge", "text": "Report only: EXP-41's 100 human-labelled claims re-judged by GPT-6 Luna (a judge from none of the six reasoners' families), with judge_v3, τ 0.6 and reasoning effort medium, from an empty judge cache, so no Haiku verdict is reused." } ], "limits": [ "Research prototype, not for clinical use.", "The 'correct' recommendation is derived mechanically from the ground-truth masks, the clinical record and the rulebook, and the rulebook is the project's own reading of the guideline.", "Final infarct stands in for the acute core, in a cohort where every patient was reperfused.", "Real cases have no ASPECTS or mass-effect input.", "A case whose rule depends on an unknown input is scored as 'undetermined' and excluded from correctness, not guessed.", "Every real case received thrombectomy, so rule-derived labels are not compared with what clinicians actually did.", "Synthetic scenarios are the primary evaluation; the real held-out cases are the realism check.", "The hallucination rates are the verifier's, and they are upper bounds: people confirmed 18 of its 54 flags, verifier–human agreement was 0.61, and the labellers agreed with each other only at κ 0.27.", "Two of the verifier's scoring rules (metric points 19 and 20) were adjusted on the same 100 human-labelled claims used to check it.", "The verifier cannot see a vision error whose claims faithfully restate the contract: the best model's one wrong held-out decision came from a false bleed flag it missed.", "Held-out correctness rests on 23 scored decisions: the thrombectomy truth is 'no' in all 20 cases and thrombolysis is undetermined in 17.", "The bleed detector flags 68 of 129 presumed bleed-free ISLES'24 development scans, a domain shift, so with predicted vision the rules refuse treatment in about half the real cases.", "ISLES'24 has no bleed labels, so its flagged scans are presumed, not proven, false alarms.", "The occlusion site is unknown in 115 of the 129 development cases and in all 20 held-out cases, because the vessel labels cannot separate M1 from M2.", "nnU-Net's volume is not measurably better than the threshold's, and both under-call large infarcts.", "Near-empty core predictions get high confidence, yet a missed infarct cannot be ruled out; the contract carries that warning.", "In the without-rulebook arm a true rule the 47-passage rulebook does not hold counts as a wrong rule, and only content is scored, so the two arms' citation numbers are not comparable.", "The fine-tuned small model was trained on templated gold answers, and it under-recommends treatment.", "The 129 development cases come from two centres; the sample is small, so subgroup differences can be neither confirmed nor ruled out.", "The SHAP values measure reliance on an input, not ablation: no model was trained without an input.", "The t-tests were fixed after the data had been seen, and the development-against-held-out tests are post hoc.", "BiomedCLIP was not fine-tuned, so that comparison is between representations, not best achievable models.", "Some scan readings (the borderline flags, Grad-CAM maps) were made by an AI assistant, not a radiologist.", "The clinician review had not been sent when this record was compiled; when answered, it will be expert opinion from two clinicians, never clinical validation." ] }