Visual Question Answering
Transformers
Safetensors
baby_medgemma
feature-extraction
medical
chest-x-ray
vision-language-model
interpretability
mechanistic-interpretability
research
custom_code
Instructions to use saillab/FlipLens with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use saillab/FlipLens with Transformers:
# Use a pipeline as a high-level helper # Warning: Pipeline type "visual-question-answering" is no longer supported in transformers v5. # You must load the model directly (see below) or downgrade to v4.x with: # pip install "transformers<5.0.0" from transformers import pipeline pipe = pipeline("visual-question-answering", model="saillab/FlipLens", trust_remote_code=True)# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("saillab/FlipLens", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
Model card: transformers load instructions
Browse files
README.md
CHANGED
|
@@ -11,31 +11,49 @@ tags:
|
|
| 11 |
- siglip
|
| 12 |
base_model:
|
| 13 |
- google/medsiglip-448
|
|
|
|
|
|
|
| 14 |
pretty_name: baby-MedGemma
|
| 15 |
---
|
| 16 |
|
| 17 |
-
# baby-MedGemma
|
| 18 |
|
| 19 |
-
|
| 20 |
-
|
| 21 |
-
|
|
|
|
|
|
|
| 22 |
|
| 23 |
- **Code, docs, and results:** https://github.com/thedatasense/babymedgemma
|
| 24 |
- **Interactive write-up:** https://bineshkumar.me/phd-thesis/causality/
|
| 25 |
|
| 26 |
-
##
|
| 27 |
|
| 28 |
-
|
| 29 |
-
|
| 30 |
-
|
| 31 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 32 |
|
| 33 |
## Model
|
| 34 |
|
| 35 |
-
The
|
| 36 |
-
|
| 37 |
-
RMSNorm, grouped-query attention). The 256 image tokens are projected and
|
| 38 |
-
prepended inline and read by the same causally-masked decoder, mirroring how
|
| 39 |
MedGemma fuses vision and text. Because the encoder is frozen and identical for
|
| 40 |
every paraphrase of a question, any answer that changes across paraphrases is
|
| 41 |
language-side by construction. The probe reaches 88.8% accuracy and is
|
|
@@ -50,27 +68,12 @@ difference-direction patch restores flipped answers at the early layers with
|
|
| 50 |
clean controls, so the flip is a low-rank, language-side, readout-stage
|
| 51 |
direction. The origin is the training data; the fix is paraphrase augmentation.
|
| 52 |
|
| 53 |
-
## How to use
|
| 54 |
-
|
| 55 |
-
```bash
|
| 56 |
-
git clone https://github.com/thedatasense/babymedgemma && cd babymedgemma
|
| 57 |
-
huggingface-cli download saillab/babymedgemma feature_cache/medsiglip_feats.pt --local-dir .
|
| 58 |
-
mv feature_cache/medsiglip_feats.pt cache/
|
| 59 |
-
python train.py --arch gemma --regime augmented --seed 0
|
| 60 |
-
```
|
| 61 |
-
|
| 62 |
-
Each checkpoint is a `state_dict` for the `BabyGemmaVLM` module in
|
| 63 |
-
`gemma_model.py`; load it with `model.load_state_dict(torch.load(path))` after
|
| 64 |
-
building the model with the matching vocab and dimensions (see `train.py`).
|
| 65 |
-
|
| 66 |
## Intended use and limitations
|
| 67 |
|
| 68 |
Research artifact for interpretability of paraphrase sensitivity in medical VLMs.
|
| 69 |
It is a controlled probe, not the deployed MedGemma-4B, trained on ~1,800
|
| 70 |
-
questions; it grounds only weakly and
|
| 71 |
-
|
| 72 |
-
image use or pretraining provenance. **Not a medical device; not for clinical
|
| 73 |
-
use.**
|
| 74 |
|
| 75 |
## Training data and license
|
| 76 |
|
|
|
|
| 11 |
- siglip
|
| 12 |
base_model:
|
| 13 |
- google/medsiglip-448
|
| 14 |
+
library_name: transformers
|
| 15 |
+
pipeline_tag: image-text-to-text
|
| 16 |
pretty_name: baby-MedGemma
|
| 17 |
---
|
| 18 |
|
| 19 |
+
# baby-MedGemma
|
| 20 |
|
| 21 |
+
A small, **MedGemma-faithful probe** for causally isolating *where paraphrase
|
| 22 |
+
sensitivity in medical vision-language models originates*: a frozen
|
| 23 |
+
`google/medsiglip-448` encoder with a small **Gemma-3 decoder** (rotary position
|
| 24 |
+
embeddings, RMSNorm, grouped-query attention: 6 query / 2 key-value heads),
|
| 25 |
+
image tokens prepended inline. Loadable directly with the `transformers` API.
|
| 26 |
|
| 27 |
- **Code, docs, and results:** https://github.com/thedatasense/babymedgemma
|
| 28 |
- **Interactive write-up:** https://bineshkumar.me/phd-thesis/causality/
|
| 29 |
|
| 30 |
+
## Load and run
|
| 31 |
|
| 32 |
+
```python
|
| 33 |
+
import torch
|
| 34 |
+
from transformers import AutoModel
|
| 35 |
+
|
| 36 |
+
model = AutoModel.from_pretrained("saillab/babymedgemma", trust_remote_code=True).eval()
|
| 37 |
+
|
| 38 |
+
# tokenize a presence question with the bundled vocabulary
|
| 39 |
+
input_ids, ans_pos = model.encode_question("is there cardiomegaly ?")
|
| 40 |
+
|
| 41 |
+
# frozen MedSigLIP-448 features for a chest radiograph (needs gated encoder access)
|
| 42 |
+
from PIL import Image
|
| 43 |
+
vision_features = model.encode_images([Image.open("cxr.png").convert("RGB")])
|
| 44 |
+
|
| 45 |
+
logits = model(input_ids=input_ids, vision_features=vision_features, ans_pos=ans_pos).logits
|
| 46 |
+
print(model.config.id2label[int(logits.argmax(-1))]) # "yes" or "no"
|
| 47 |
+
```
|
| 48 |
+
|
| 49 |
+
The bundled `model.safetensors` is the augmented-regime seed-0 checkpoint. The
|
| 50 |
+
full set of 85 per-run checkpoints is under `checkpoints/`, and the frozen
|
| 51 |
+
MedSigLIP feature cache is at `feature_cache/medsiglip_feats.pt` (~3 GB).
|
| 52 |
|
| 53 |
## Model
|
| 54 |
|
| 55 |
+
The 256 image tokens are projected and prepended inline and read by the same
|
| 56 |
+
causally-masked Gemma-3 decoder (13.9M trainable parameters), mirroring how
|
|
|
|
|
|
|
| 57 |
MedGemma fuses vision and text. Because the encoder is frozen and identical for
|
| 58 |
every paraphrase of a question, any answer that changes across paraphrases is
|
| 59 |
language-side by construction. The probe reaches 88.8% accuracy and is
|
|
|
|
| 68 |
clean controls, so the flip is a low-rank, language-side, readout-stage
|
| 69 |
direction. The origin is the training data; the fix is paraphrase augmentation.
|
| 70 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 71 |
## Intended use and limitations
|
| 72 |
|
| 73 |
Research artifact for interpretability of paraphrase sensitivity in medical VLMs.
|
| 74 |
It is a controlled probe, not the deployed MedGemma-4B, trained on ~1,800
|
| 75 |
+
questions over a small word-level vocabulary; it grounds only weakly and
|
| 76 |
+
manipulates adaptation-stage data. **Not a medical device; not for clinical use.**
|
|
|
|
|
|
|
| 77 |
|
| 78 |
## Training data and license
|
| 79 |
|