Visual Question Answering
Transformers
Safetensors
baby_medgemma
feature-extraction
medical
chest-x-ray
vision-language-model
interpretability
mechanistic-interpretability
research
custom_code
Instructions to use saillab/FlipLens with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use saillab/FlipLens with Transformers:
# Use a pipeline as a high-level helper # Warning: Pipeline type "visual-question-answering" is no longer supported in transformers v5. # You must load the model directly (see below) or downgrade to v4.x with: # pip install "transformers<5.0.0" from transformers import pipeline pipe = pipeline("visual-question-answering", model="saillab/FlipLens", trust_remote_code=True)# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("saillab/FlipLens", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
Add the single-pass margin-gate flip detector and the route-flip null result
Browse files
README.md
CHANGED
|
@@ -25,6 +25,7 @@ sensitivity in medical vision-language models originates*.
|
|
| 25 |
|
| 26 |
- Code and the full experiment write-up: **[github.com/thedatasense/babymedgemma](https://github.com/thedatasense/babymedgemma)**
|
| 27 |
- Interactive write-up: **[bineshkumar.me/phd-thesis/causality](https://bineshkumar.me/phd-thesis/causality/)**
|
|
|
|
| 28 |
|
| 29 |
> **Scope.** A research probe, **not a clinical model**. It answers binary presence
|
| 30 |
> questions about 14 chest findings. Not a medical device; not for clinical use.
|
|
@@ -77,6 +78,30 @@ Accuracy alone cannot tell a seeing model from a blind one here: a blind model,
|
|
| 77 |
model given a *shuffled* grounding token, and a genuinely seeing model all score
|
| 78 |
~0.50 accuracy while their AUCs are 0.500 / 0.506 / 0.604. Report AUC.
|
| 79 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 80 |
## Load
|
| 81 |
|
| 82 |
```python
|
|
|
|
| 25 |
|
| 26 |
- Code and the full experiment write-up: **[github.com/thedatasense/babymedgemma](https://github.com/thedatasense/babymedgemma)**
|
| 27 |
- Interactive write-up: **[bineshkumar.me/phd-thesis/causality](https://bineshkumar.me/phd-thesis/causality/)**
|
| 28 |
+
- Live single-pass flip-detector demo: **[bineshkumar.me/phd-thesis/explorer/#margin-demo](https://bineshkumar.me/phd-thesis/explorer/#margin-demo)**
|
| 29 |
|
| 30 |
> **Scope.** A research probe, **not a clinical model**. It answers binary presence
|
| 31 |
> questions about 14 chest findings. Not a medical device; not for clinical use.
|
|
|
|
| 78 |
model given a *shuffled* grounding token, and a genuinely seeing model all score
|
| 79 |
~0.50 accuracy while their AUCs are 0.500 / 0.506 / 0.604. Report AUC.
|
| 80 |
|
| 81 |
+
### The discarded margin is a single-pass flip detector
|
| 82 |
+
|
| 83 |
+
Because the answer is the sign of the yes-minus-no margin and a flip is a sign change,
|
| 84 |
+
a margin near zero marks a paraphrase-unstable answer. The absolute margin of one
|
| 85 |
+
forward pass therefore ranks flip-prone questions at no cost beyond the answer itself,
|
| 86 |
+
and it beats the detectors that cost more (`scripts/analysis/detect.py`):
|
| 87 |
+
|
| 88 |
+
| detector | passes | in-distribution | MIMIC | VinDr |
|
| 89 |
+
|---|---|---|---|---|
|
| 90 |
+
| **absolute margin** | **1** | **0.923** | **0.974** | **0.974** |
|
| 91 |
+
| paraphrase self-consistency | k | 0.709 | 0.786 | 0.624 |
|
| 92 |
+
| hidden-state probe | 1 + fit | 0.826 | 0.838 | 0.801 |
|
| 93 |
+
|
| 94 |
+
The other failure, an answer that ignores the image, has **no** single-pass signal: the
|
| 95 |
+
margin is at chance for it (0.470 to 0.519), because a confident blind answer looks
|
| 96 |
+
exactly like a confident grounded one. Only a second pass that swaps in another
|
| 97 |
+
patient's image detects it (0.827 to 0.907). A finer question, whether a stable answer
|
| 98 |
+
quietly changes how much it uses the image from one phrasing to another, is real at the
|
| 99 |
+
population level (cross-draw covariance 0.062, between-draw correlation 0.56, 82%
|
| 100 |
+
case-specific over three seeds) but **not identifiable per case** (zero grounded-to-unreliant
|
| 101 |
+
transitions over 345 patient-seed evaluations, patient-level upper bound near 3%). Image
|
| 102 |
+
reliance is a population property here, not a per-prediction label. Try it in the
|
| 103 |
+
**[live margin-gate demo](https://bineshkumar.me/phd-thesis/explorer/#margin-demo)**.
|
| 104 |
+
|
| 105 |
## Load
|
| 106 |
|
| 107 |
```python
|