binesh commited on
Commit
9356d18
·
verified ·
1 Parent(s): b96a912

Card: provenance at scale, held-out phrasing, threshold robustness

Browse files
Files changed (1) hide show
  1. README.md +30 -3
README.md CHANGED
@@ -39,9 +39,10 @@ sensitivity in medical vision-language models originates*.
39
  | Paraphrases | 48 templates across 4 phenomena | 4.8 generated per question |
40
  | Grounding token | yes (MedSigLIP pooled embedding) | no |
41
 
42
- **Use `probe-1841/` to reproduce the dissertation's Section 5.5 numbers** (augmentation
43
- lever 29.5% -> 9.5%, rank-1 causal patching). Those results are specific to that
44
- training distribution and do not transfer to the default weights.
 
45
 
46
  ## What the default model does
47
 
@@ -55,6 +56,32 @@ text-only model scores exactly 0.500 and all accuracy above that is visual.
55
  | **MIMIC (unseen hospital)** | 450 | 0.671 | **0.743** | 0.500 | 0.049 |
56
  | **VinDr (unseen hospital)** | 5,478 | 0.686 | **0.756** | 0.500 | 0.060 |
57
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
58
  For reference, MedSigLIP's own zero-shot binary performance on this task is AUC 0.734,
59
  and an earlier version of this probe scored **AUC 0.500 — indistinguishable from blind
60
  — on unseen data**. Per finding, pneumothorax transfers essentially unchanged (0.899
 
39
  | Paraphrases | 48 templates across 4 phenomena | 4.8 generated per question |
40
  | Grounding token | yes (MedSigLIP pooled embedding) | no |
41
 
42
+ The dissertation's Section 5.5 now describes the **default** model. `probe-1841/` is
43
+ kept for reproducing the earlier published figures (augmentation lever 29.5% -> 9.5%)
44
+ and the experiments not re-run at scale (divergence trajectory, decoder-depth sweep,
45
+ vision-dropout grounding sweep).
46
 
47
  ## What the default model does
48
 
 
56
  | **MIMIC (unseen hospital)** | 450 | 0.671 | **0.743** | 0.500 | 0.049 |
57
  | **VinDr (unseen hospital)** | 5,478 | 0.686 | **0.756** | 0.500 | 0.060 |
58
 
59
+ ### What it establishes about paraphrase sensitivity
60
+
61
+ Training the same architecture under three phrasing distributions, changing nothing
62
+ else (8 seeds each; text-only accuracy 50.0-50.3% in all three, so none can exploit
63
+ an answer prior):
64
+
65
+ | training regime | flip rate | scored on **unseen** phrasings | accuracy |
66
+ |---|---|---|---|
67
+ | every paraphrase | **4.8%** | **26.6%** | 75.3% |
68
+ | one fixed phrasing | 67.1% | 65.9% | 66.8% |
69
+ | register tied to the answer | 88.4% | 87.4% | 58.1% |
70
+
71
+ Broad coverage separates from both narrow regimes at Cliff's delta = 1.00
72
+ (p = 1.6e-4), and the two narrow regimes separate from each other (p = 3.1e-4,
73
+ delta = 0.97), so a phrasing shortcut is a distinct and larger harm than narrow
74
+ coverage alone. The second column trains on half the paraphrase bank and scores only
75
+ on the 24 phrasings withheld, separating invariance from familiarity with wording
76
+ already seen: **26.6% against 65.9% is the honest size of the lever.**
77
+
78
+ A rank-1 direction at the answer position restores flipped answers (net recovery
79
+ 0.98-1.00 across layers 0-4; random-direction control 0.00-0.02; non-flip disruption
80
+ 0.000-0.006), so the flip is a low-rank, language-side, readout-stage effect. It is
81
+ not an artifact of where the decision boundary sits: the ordering holds at all 17
82
+ offsets across +/-2 SD of the margin, and a threshold-free dispersion statistic
83
+ reproduces it (0.023 / 0.498 / 1.362, delta = -1.00).
84
+
85
  For reference, MedSigLIP's own zero-shot binary performance on this task is AUC 0.734,
86
  and an earlier version of this probe scored **AUC 0.500 — indistinguishable from blind
87
  — on unseen data**. Per finding, pneumothorax transfers essentially unchanged (0.899