binesh commited on
Commit
5efcebc
·
verified ·
1 Parent(s): 4136cc2

Add the single-pass margin-gate flip detector and the route-flip null result

Browse files
Files changed (1) hide show
  1. README.md +25 -0
README.md CHANGED
@@ -25,6 +25,7 @@ sensitivity in medical vision-language models originates*.
25
 
26
  - Code and the full experiment write-up: **[github.com/thedatasense/babymedgemma](https://github.com/thedatasense/babymedgemma)**
27
  - Interactive write-up: **[bineshkumar.me/phd-thesis/causality](https://bineshkumar.me/phd-thesis/causality/)**
 
28
 
29
  > **Scope.** A research probe, **not a clinical model**. It answers binary presence
30
  > questions about 14 chest findings. Not a medical device; not for clinical use.
@@ -77,6 +78,30 @@ Accuracy alone cannot tell a seeing model from a blind one here: a blind model,
77
  model given a *shuffled* grounding token, and a genuinely seeing model all score
78
  ~0.50 accuracy while their AUCs are 0.500 / 0.506 / 0.604. Report AUC.
79
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
80
  ## Load
81
 
82
  ```python
 
25
 
26
  - Code and the full experiment write-up: **[github.com/thedatasense/babymedgemma](https://github.com/thedatasense/babymedgemma)**
27
  - Interactive write-up: **[bineshkumar.me/phd-thesis/causality](https://bineshkumar.me/phd-thesis/causality/)**
28
+ - Live single-pass flip-detector demo: **[bineshkumar.me/phd-thesis/explorer/#margin-demo](https://bineshkumar.me/phd-thesis/explorer/#margin-demo)**
29
 
30
  > **Scope.** A research probe, **not a clinical model**. It answers binary presence
31
  > questions about 14 chest findings. Not a medical device; not for clinical use.
 
78
  model given a *shuffled* grounding token, and a genuinely seeing model all score
79
  ~0.50 accuracy while their AUCs are 0.500 / 0.506 / 0.604. Report AUC.
80
 
81
+ ### The discarded margin is a single-pass flip detector
82
+
83
+ Because the answer is the sign of the yes-minus-no margin and a flip is a sign change,
84
+ a margin near zero marks a paraphrase-unstable answer. The absolute margin of one
85
+ forward pass therefore ranks flip-prone questions at no cost beyond the answer itself,
86
+ and it beats the detectors that cost more (`scripts/analysis/detect.py`):
87
+
88
+ | detector | passes | in-distribution | MIMIC | VinDr |
89
+ |---|---|---|---|---|
90
+ | **absolute margin** | **1** | **0.923** | **0.974** | **0.974** |
91
+ | paraphrase self-consistency | k | 0.709 | 0.786 | 0.624 |
92
+ | hidden-state probe | 1 + fit | 0.826 | 0.838 | 0.801 |
93
+
94
+ The other failure, an answer that ignores the image, has **no** single-pass signal: the
95
+ margin is at chance for it (0.470 to 0.519), because a confident blind answer looks
96
+ exactly like a confident grounded one. Only a second pass that swaps in another
97
+ patient's image detects it (0.827 to 0.907). A finer question, whether a stable answer
98
+ quietly changes how much it uses the image from one phrasing to another, is real at the
99
+ population level (cross-draw covariance 0.062, between-draw correlation 0.56, 82%
100
+ case-specific over three seeds) but **not identifiable per case** (zero grounded-to-unreliant
101
+ transitions over 345 patient-seed evaluations, patient-level upper bound near 3%). Image
102
+ reliance is a population property here, not a per-prediction label. Try it in the
103
+ **[live margin-gate demo](https://bineshkumar.me/phd-thesis/explorer/#margin-demo)**.
104
+
105
  ## Load
106
 
107
  ```python