Visual Question Answering
Transformers
Safetensors
baby_medgemma
feature-extraction
medical
chest-x-ray
vision-language-model
interpretability
mechanistic-interpretability
research
custom_code
Instructions to use saillab/FlipLens with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use saillab/FlipLens with Transformers:
# Use a pipeline as a high-level helper # Warning: Pipeline type "visual-question-answering" is no longer supported in transformers v5. # You must load the model directly (see below) or downgrade to v4.x with: # pip install "transformers<5.0.0" from transformers import pipeline pipe = pipeline("visual-question-answering", model="saillab/FlipLens", trust_remote_code=True)# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("saillab/FlipLens", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
Card: provenance at scale, held-out phrasing, threshold robustness
Browse files
README.md
CHANGED
|
@@ -39,9 +39,10 @@ sensitivity in medical vision-language models originates*.
|
|
| 39 |
| Paraphrases | 48 templates across 4 phenomena | 4.8 generated per question |
|
| 40 |
| Grounding token | yes (MedSigLIP pooled embedding) | no |
|
| 41 |
|
| 42 |
-
|
| 43 |
-
lever 29.5% -> 9.5%
|
| 44 |
-
|
|
|
|
| 45 |
|
| 46 |
## What the default model does
|
| 47 |
|
|
@@ -55,6 +56,32 @@ text-only model scores exactly 0.500 and all accuracy above that is visual.
|
|
| 55 |
| **MIMIC (unseen hospital)** | 450 | 0.671 | **0.743** | 0.500 | 0.049 |
|
| 56 |
| **VinDr (unseen hospital)** | 5,478 | 0.686 | **0.756** | 0.500 | 0.060 |
|
| 57 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 58 |
For reference, MedSigLIP's own zero-shot binary performance on this task is AUC 0.734,
|
| 59 |
and an earlier version of this probe scored **AUC 0.500 — indistinguishable from blind
|
| 60 |
— on unseen data**. Per finding, pneumothorax transfers essentially unchanged (0.899
|
|
|
|
| 39 |
| Paraphrases | 48 templates across 4 phenomena | 4.8 generated per question |
|
| 40 |
| Grounding token | yes (MedSigLIP pooled embedding) | no |
|
| 41 |
|
| 42 |
+
The dissertation's Section 5.5 now describes the **default** model. `probe-1841/` is
|
| 43 |
+
kept for reproducing the earlier published figures (augmentation lever 29.5% -> 9.5%)
|
| 44 |
+
and the experiments not re-run at scale (divergence trajectory, decoder-depth sweep,
|
| 45 |
+
vision-dropout grounding sweep).
|
| 46 |
|
| 47 |
## What the default model does
|
| 48 |
|
|
|
|
| 56 |
| **MIMIC (unseen hospital)** | 450 | 0.671 | **0.743** | 0.500 | 0.049 |
|
| 57 |
| **VinDr (unseen hospital)** | 5,478 | 0.686 | **0.756** | 0.500 | 0.060 |
|
| 58 |
|
| 59 |
+
### What it establishes about paraphrase sensitivity
|
| 60 |
+
|
| 61 |
+
Training the same architecture under three phrasing distributions, changing nothing
|
| 62 |
+
else (8 seeds each; text-only accuracy 50.0-50.3% in all three, so none can exploit
|
| 63 |
+
an answer prior):
|
| 64 |
+
|
| 65 |
+
| training regime | flip rate | scored on **unseen** phrasings | accuracy |
|
| 66 |
+
|---|---|---|---|
|
| 67 |
+
| every paraphrase | **4.8%** | **26.6%** | 75.3% |
|
| 68 |
+
| one fixed phrasing | 67.1% | 65.9% | 66.8% |
|
| 69 |
+
| register tied to the answer | 88.4% | 87.4% | 58.1% |
|
| 70 |
+
|
| 71 |
+
Broad coverage separates from both narrow regimes at Cliff's delta = 1.00
|
| 72 |
+
(p = 1.6e-4), and the two narrow regimes separate from each other (p = 3.1e-4,
|
| 73 |
+
delta = 0.97), so a phrasing shortcut is a distinct and larger harm than narrow
|
| 74 |
+
coverage alone. The second column trains on half the paraphrase bank and scores only
|
| 75 |
+
on the 24 phrasings withheld, separating invariance from familiarity with wording
|
| 76 |
+
already seen: **26.6% against 65.9% is the honest size of the lever.**
|
| 77 |
+
|
| 78 |
+
A rank-1 direction at the answer position restores flipped answers (net recovery
|
| 79 |
+
0.98-1.00 across layers 0-4; random-direction control 0.00-0.02; non-flip disruption
|
| 80 |
+
0.000-0.006), so the flip is a low-rank, language-side, readout-stage effect. It is
|
| 81 |
+
not an artifact of where the decision boundary sits: the ordering holds at all 17
|
| 82 |
+
offsets across +/-2 SD of the margin, and a threshold-free dispersion statistic
|
| 83 |
+
reproduces it (0.023 / 0.498 / 1.362, delta = -1.00).
|
| 84 |
+
|
| 85 |
For reference, MedSigLIP's own zero-shot binary performance on this task is AUC 0.734,
|
| 86 |
and an earlier version of this probe scored **AUC 0.500 — indistinguishable from blind
|
| 87 |
— on unseen data**. Per finding, pneumothorax transfers essentially unchanged (0.899
|