Visual Question Answering
Transformers
Safetensors
baby_medgemma
feature-extraction
medical
chest-x-ray
vision-language-model
interpretability
mechanistic-interpretability
research
custom_code
Instructions to use saillab/FlipLens with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use saillab/FlipLens with Transformers:
# Use a pipeline as a high-level helper # Warning: Pipeline type "visual-question-answering" is no longer supported in transformers v5. # You must load the model directly (see below) or downgrade to v4.x with: # pip install "transformers<5.0.0" from transformers import pipeline pipe = pipeline("visual-question-answering", model="saillab/FlipLens", trust_remote_code=True)# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("saillab/FlipLens", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
Add model card (896 + LM-head faithful release)
Browse files
README.md
CHANGED
|
@@ -1,91 +1,84 @@
|
|
| 1 |
---
|
| 2 |
license: other
|
| 3 |
-
license_name:
|
|
|
|
|
|
|
| 4 |
tags:
|
| 5 |
-
- medical
|
| 6 |
- chest-x-ray
|
| 7 |
- vision-language-model
|
|
|
|
| 8 |
- mechanistic-interpretability
|
| 9 |
-
-
|
| 10 |
-
- gemma
|
| 11 |
-
- siglip
|
| 12 |
base_model:
|
| 13 |
- google/medsiglip-448
|
| 14 |
-
|
| 15 |
-
pipeline_tag:
|
| 16 |
-
pretty_name: baby-MedGemma
|
| 17 |
---
|
| 18 |
|
| 19 |
# baby-MedGemma
|
| 20 |
|
| 21 |
A small, **MedGemma-faithful probe** for causally isolating *where paraphrase
|
| 22 |
-
sensitivity in medical vision-language models originates*
|
| 23 |
-
`google/medsiglip-448` encoder
|
| 24 |
-
|
| 25 |
-
|
| 26 |
-
|
| 27 |
-
|
| 28 |
-
-
|
| 29 |
-
|
| 30 |
-
|
| 31 |
-
>
|
| 32 |
-
> chest-X-ray presence questions
|
| 33 |
-
>
|
| 34 |
-
>
|
| 35 |
-
>
|
| 36 |
-
|
| 37 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 38 |
|
| 39 |
```python
|
| 40 |
import torch
|
| 41 |
from transformers import AutoModel
|
|
|
|
| 42 |
|
| 43 |
model = AutoModel.from_pretrained("saillab/babymedgemma", trust_remote_code=True).eval()
|
| 44 |
-
|
| 45 |
-
# tokenize a presence question with the bundled vocabulary
|
| 46 |
input_ids, ans_pos = model.encode_question("is there cardiomegaly ?")
|
| 47 |
-
|
| 48 |
-
# frozen MedSigLIP-448 features for a chest radiograph (needs gated encoder access)
|
| 49 |
-
from PIL import Image
|
| 50 |
-
vision_features = model.encode_images([Image.open("cxr.png").convert("RGB")])
|
| 51 |
-
|
| 52 |
logits = model(input_ids=input_ids, vision_features=vision_features, ans_pos=ans_pos).logits
|
| 53 |
print(model.config.id2label[int(logits.argmax(-1))]) # "yes" or "no"
|
| 54 |
```
|
| 55 |
|
| 56 |
-
|
| 57 |
-
|
| 58 |
-
|
| 59 |
-
|
| 60 |
-
## Model
|
| 61 |
-
|
| 62 |
-
The 256 image tokens are projected and prepended inline and read by the same
|
| 63 |
-
causally-masked Gemma-3 decoder (13.9M trainable parameters), mirroring how
|
| 64 |
-
MedGemma fuses vision and text. Because the encoder is frozen and identical for
|
| 65 |
-
every paraphrase of a question, any answer that changes across paraphrases is
|
| 66 |
-
language-side by construction. The probe reaches 88.8% accuracy and is
|
| 67 |
-
text-reliant like MedGemma (75% of answers unchanged when the image is removed).
|
| 68 |
-
|
| 69 |
-
## Key result
|
| 70 |
-
|
| 71 |
-
Varying only the training-phrasing distribution sets the flip rate: augmented
|
| 72 |
-
8.4% vs canonical 30.3% vs adversarial 30.4% over 16 seeds (augmented vs each
|
| 73 |
-
narrow regime: Mann-Whitney U p = 1.4e-6, Cliff's delta = 1.00). A rank-1
|
| 74 |
-
difference-direction patch restores flipped answers at the early layers with
|
| 75 |
-
clean controls, so the flip is a low-rank, language-side, readout-stage
|
| 76 |
-
direction. The origin is the training data; the fix is paraphrase augmentation.
|
| 77 |
-
The full A-to-E experiments, figures, and what each does and does not prove are
|
| 78 |
-
in the [repository README](https://github.com/thedatasense/babymedgemma#the-framework-five-experiments-a-to-e).
|
| 79 |
-
|
| 80 |
-
## Training data and license
|
| 81 |
-
|
| 82 |
-
Trained on binary presence questions over MIMIC-CXR and PadChest chest
|
| 83 |
-
radiographs with register-tagged paraphrases. The features and checkpoints are
|
| 84 |
-
derived from those datasets, so their use inherits the MIMIC-CXR and PadChest
|
| 85 |
-
data-use agreements. Released for research use only.
|
| 86 |
|
| 87 |
## Citation
|
| 88 |
|
| 89 |
> Sadanandan, B. (2026). *Paraphrase Sensitivity in Medical Vision-Language
|
| 90 |
-
> Models: Measurement, Mechanisms, Mitigation, and Deployment Safety.*
|
| 91 |
-
>
|
|
|
|
| 1 |
---
|
| 2 |
license: other
|
| 3 |
+
license_name: health-ai-developer-foundations
|
| 4 |
+
license_link: https://developers.google.com/health-ai-developer-foundations/terms
|
| 5 |
+
library_name: transformers
|
| 6 |
tags:
|
| 7 |
+
- medical
|
| 8 |
- chest-x-ray
|
| 9 |
- vision-language-model
|
| 10 |
+
- interpretability
|
| 11 |
- mechanistic-interpretability
|
| 12 |
+
- research
|
|
|
|
|
|
|
| 13 |
base_model:
|
| 14 |
- google/medsiglip-448
|
| 15 |
+
- google/gemma-3-4b-it
|
| 16 |
+
pipeline_tag: visual-question-answering
|
|
|
|
| 17 |
---
|
| 18 |
|
| 19 |
# baby-MedGemma
|
| 20 |
|
| 21 |
A small, **MedGemma-faithful probe** for causally isolating *where paraphrase
|
| 22 |
+
sensitivity in medical vision-language models originates*. It pairs a frozen
|
| 23 |
+
`google/medsiglip-448` encoder (run at 896 pixels, the MedGemma resolution) with a
|
| 24 |
+
small trained **Gemma-3** decoder, so a mechanism found here transfers to
|
| 25 |
+
MedGemma-4B, while letting the one experiment the frozen deployed model cannot
|
| 26 |
+
support run: **varying the training-phrasing distribution one factor at a time.**
|
| 27 |
+
|
| 28 |
+
- Code, five-experiment write-up, and figures: **[github.com/thedatasense/babymedgemma](https://github.com/thedatasense/babymedgemma)**
|
| 29 |
+
- Interactive write-up: **[bineshkumar.me/phd-thesis/causality](https://bineshkumar.me/phd-thesis/causality/)**
|
| 30 |
+
|
| 31 |
+
> **Scope.** A controlled research probe, **not a general medical VQA model**. It
|
| 32 |
+
> has a 734-word domain vocabulary (chest-X-ray presence questions plus the yes/no
|
| 33 |
+
> answer tokens) and grounds weakly, so it is loadable and runnable for inspection
|
| 34 |
+
> and reproduction only. It supports a *sufficiency* claim about the cause of
|
| 35 |
+
> paraphrase sensitivity, not proof that the deployed MedGemma-4B has this exact
|
| 36 |
+
> origin. Not a medical device; not for clinical use.
|
| 37 |
+
|
| 38 |
+
## Fidelity to MedGemma-1.5
|
| 39 |
+
|
| 40 |
+
Matched to the deployment stack ([arXiv:2604.05081](https://arxiv.org/pdf/2604.05081)):
|
| 41 |
+
frozen MedSigLIP encoder, 896-pixel input, 256 image tokens, image tokens
|
| 42 |
+
projected and prepended inline, a Gemma-3 decoder (rotary position embeddings,
|
| 43 |
+
RMSNorm, grouped-query attention, GeGLU, QK-norm), and the yes/no decision read
|
| 44 |
+
from the tied Gemma-3 LM head. Simplified on purpose for the experiment: 13.9M
|
| 45 |
+
decoder parameters trained from scratch, supervised training, and a 734-word
|
| 46 |
+
closed vocabulary, so the training-phrasing distribution becomes a controllable
|
| 47 |
+
input rather than a frozen prior.
|
| 48 |
+
|
| 49 |
+
## What it establishes
|
| 50 |
+
|
| 51 |
+
| | Result |
|
| 52 |
+
|---|---|
|
| 53 |
+
| Cause (data provenance) | Paraphrase augmentation cuts the flip rate from 29.5% (one fixed phrasing) to 9.5% (every paraphrase), Cliff's delta = 1.00, 16 seeds |
|
| 54 |
+
| Mechanism (causal patching) | A single rank-1, language-side direction in the early layers restores flipped answers (net recovery near 1.0); random and non-flip controls do nothing |
|
| 55 |
+
| Corroboration | An unsupervised sparse-autoencoder feature recovers that direction (\|cosine\| 0.74, ahead of principal-component analysis at 0.52); a Jacobian lens separates flipping from stable paraphrases (8.9x, from layer 0) |
|
| 56 |
+
| Ruled out | Decoder depth (flip rate 9.1% to 10.2% across depths 2 to 8) and weak grounding (inconclusive) |
|
| 57 |
+
|
| 58 |
+
The bundled weights are the **augmented seed-0** checkpoint (88.5% native accuracy,
|
| 59 |
+
8.2% flip rate).
|
| 60 |
+
|
| 61 |
+
## Load
|
| 62 |
|
| 63 |
```python
|
| 64 |
import torch
|
| 65 |
from transformers import AutoModel
|
| 66 |
+
from PIL import Image
|
| 67 |
|
| 68 |
model = AutoModel.from_pretrained("saillab/babymedgemma", trust_remote_code=True).eval()
|
|
|
|
|
|
|
| 69 |
input_ids, ans_pos = model.encode_question("is there cardiomegaly ?")
|
| 70 |
+
vision_features = model.encode_images([Image.open("cxr.png").convert("RGB")]) # frozen MedSigLIP at 896px
|
|
|
|
|
|
|
|
|
|
|
|
|
| 71 |
logits = model(input_ids=input_ids, vision_features=vision_features, ans_pos=ans_pos).logits
|
| 72 |
print(model.config.id2label[int(logits.argmax(-1))]) # "yes" or "no"
|
| 73 |
```
|
| 74 |
|
| 75 |
+
`encode_images` needs gated access to `google/medsiglip-448`. The precomputed
|
| 76 |
+
896-pixel feature cache (`feature_cache/medsiglip_feats.pt`) and the per-run
|
| 77 |
+
`BabyGemmaVLM` checkpoints (`checkpoints/`) are in this repo; the code to
|
| 78 |
+
reproduce every experiment is on GitHub.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 79 |
|
| 80 |
## Citation
|
| 81 |
|
| 82 |
> Sadanandan, B. (2026). *Paraphrase Sensitivity in Medical Vision-Language
|
| 83 |
+
> Models: Measurement, Mechanisms, Mitigation, and Deployment Safety.* PhD
|
| 84 |
+
> dissertation, University of New Haven.
|