binesh commited on
Commit
712de7a
·
verified ·
1 Parent(s): 54e9276

Add model card (896 + LM-head faithful release)

Browse files
Files changed (1) hide show
  1. README.md +56 -63
README.md CHANGED
@@ -1,91 +1,84 @@
1
  ---
2
  license: other
3
- license_name: research-use-inherits-source-dua
 
 
4
  tags:
5
- - medical-imaging
6
  - chest-x-ray
7
  - vision-language-model
 
8
  - mechanistic-interpretability
9
- - paraphrase-robustness
10
- - gemma
11
- - siglip
12
  base_model:
13
  - google/medsiglip-448
14
- library_name: transformers
15
- pipeline_tag: image-text-to-text
16
- pretty_name: baby-MedGemma
17
  ---
18
 
19
  # baby-MedGemma
20
 
21
  A small, **MedGemma-faithful probe** for causally isolating *where paraphrase
22
- sensitivity in medical vision-language models originates*: a frozen
23
- `google/medsiglip-448` encoder with a small **Gemma-3 decoder** (rotary position
24
- embeddings, RMSNorm, grouped-query attention: 6 query / 2 key-value heads), image
25
- tokens prepended inline. Loadable directly with the `transformers` API.
26
-
27
- - **Code, full experiments, and results:** https://github.com/thedatasense/babymedgemma
28
- - **Interactive write-up:** https://bineshkumar.me/phd-thesis/causality/
29
-
30
- > **Scope.** This is a controlled research probe, **not a general medical VQA
31
- > model**. It has a small domain-specific vocabulary (733 words from the
32
- > chest-X-ray presence questions) and grounds weakly, so it is loadable and
33
- > runnable for inspection and reproduction only. It supports a *sufficiency*
34
- > claim about the cause of paraphrase sensitivity, not proof that the deployed
35
- > MedGemma-4B has this exact origin. Not a medical device; not for clinical use.
36
-
37
- ## Load and run
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
38
 
39
  ```python
40
  import torch
41
  from transformers import AutoModel
 
42
 
43
  model = AutoModel.from_pretrained("saillab/babymedgemma", trust_remote_code=True).eval()
44
-
45
- # tokenize a presence question with the bundled vocabulary
46
  input_ids, ans_pos = model.encode_question("is there cardiomegaly ?")
47
-
48
- # frozen MedSigLIP-448 features for a chest radiograph (needs gated encoder access)
49
- from PIL import Image
50
- vision_features = model.encode_images([Image.open("cxr.png").convert("RGB")])
51
-
52
  logits = model(input_ids=input_ids, vision_features=vision_features, ans_pos=ans_pos).logits
53
  print(model.config.id2label[int(logits.argmax(-1))]) # "yes" or "no"
54
  ```
55
 
56
- The bundled `model.safetensors` is the augmented-regime seed-0 checkpoint. The
57
- full set of 85 per-run checkpoints is under `checkpoints/`, and the frozen
58
- MedSigLIP feature cache is at `feature_cache/medsiglip_feats.pt` (~3 GB).
59
-
60
- ## Model
61
-
62
- The 256 image tokens are projected and prepended inline and read by the same
63
- causally-masked Gemma-3 decoder (13.9M trainable parameters), mirroring how
64
- MedGemma fuses vision and text. Because the encoder is frozen and identical for
65
- every paraphrase of a question, any answer that changes across paraphrases is
66
- language-side by construction. The probe reaches 88.8% accuracy and is
67
- text-reliant like MedGemma (75% of answers unchanged when the image is removed).
68
-
69
- ## Key result
70
-
71
- Varying only the training-phrasing distribution sets the flip rate: augmented
72
- 8.4% vs canonical 30.3% vs adversarial 30.4% over 16 seeds (augmented vs each
73
- narrow regime: Mann-Whitney U p = 1.4e-6, Cliff's delta = 1.00). A rank-1
74
- difference-direction patch restores flipped answers at the early layers with
75
- clean controls, so the flip is a low-rank, language-side, readout-stage
76
- direction. The origin is the training data; the fix is paraphrase augmentation.
77
- The full A-to-E experiments, figures, and what each does and does not prove are
78
- in the [repository README](https://github.com/thedatasense/babymedgemma#the-framework-five-experiments-a-to-e).
79
-
80
- ## Training data and license
81
-
82
- Trained on binary presence questions over MIMIC-CXR and PadChest chest
83
- radiographs with register-tagged paraphrases. The features and checkpoints are
84
- derived from those datasets, so their use inherits the MIMIC-CXR and PadChest
85
- data-use agreements. Released for research use only.
86
 
87
  ## Citation
88
 
89
  > Sadanandan, B. (2026). *Paraphrase Sensitivity in Medical Vision-Language
90
- > Models: Measurement, Mechanisms, Mitigation, and Deployment Safety.*
91
- > Unpublished doctoral dissertation, University of New Haven.
 
1
  ---
2
  license: other
3
+ license_name: health-ai-developer-foundations
4
+ license_link: https://developers.google.com/health-ai-developer-foundations/terms
5
+ library_name: transformers
6
  tags:
7
+ - medical
8
  - chest-x-ray
9
  - vision-language-model
10
+ - interpretability
11
  - mechanistic-interpretability
12
+ - research
 
 
13
  base_model:
14
  - google/medsiglip-448
15
+ - google/gemma-3-4b-it
16
+ pipeline_tag: visual-question-answering
 
17
  ---
18
 
19
  # baby-MedGemma
20
 
21
  A small, **MedGemma-faithful probe** for causally isolating *where paraphrase
22
+ sensitivity in medical vision-language models originates*. It pairs a frozen
23
+ `google/medsiglip-448` encoder (run at 896 pixels, the MedGemma resolution) with a
24
+ small trained **Gemma-3** decoder, so a mechanism found here transfers to
25
+ MedGemma-4B, while letting the one experiment the frozen deployed model cannot
26
+ support run: **varying the training-phrasing distribution one factor at a time.**
27
+
28
+ - Code, five-experiment write-up, and figures: **[github.com/thedatasense/babymedgemma](https://github.com/thedatasense/babymedgemma)**
29
+ - Interactive write-up: **[bineshkumar.me/phd-thesis/causality](https://bineshkumar.me/phd-thesis/causality/)**
30
+
31
+ > **Scope.** A controlled research probe, **not a general medical VQA model**. It
32
+ > has a 734-word domain vocabulary (chest-X-ray presence questions plus the yes/no
33
+ > answer tokens) and grounds weakly, so it is loadable and runnable for inspection
34
+ > and reproduction only. It supports a *sufficiency* claim about the cause of
35
+ > paraphrase sensitivity, not proof that the deployed MedGemma-4B has this exact
36
+ > origin. Not a medical device; not for clinical use.
37
+
38
+ ## Fidelity to MedGemma-1.5
39
+
40
+ Matched to the deployment stack ([arXiv:2604.05081](https://arxiv.org/pdf/2604.05081)):
41
+ frozen MedSigLIP encoder, 896-pixel input, 256 image tokens, image tokens
42
+ projected and prepended inline, a Gemma-3 decoder (rotary position embeddings,
43
+ RMSNorm, grouped-query attention, GeGLU, QK-norm), and the yes/no decision read
44
+ from the tied Gemma-3 LM head. Simplified on purpose for the experiment: 13.9M
45
+ decoder parameters trained from scratch, supervised training, and a 734-word
46
+ closed vocabulary, so the training-phrasing distribution becomes a controllable
47
+ input rather than a frozen prior.
48
+
49
+ ## What it establishes
50
+
51
+ | | Result |
52
+ |---|---|
53
+ | Cause (data provenance) | Paraphrase augmentation cuts the flip rate from 29.5% (one fixed phrasing) to 9.5% (every paraphrase), Cliff's delta = 1.00, 16 seeds |
54
+ | Mechanism (causal patching) | A single rank-1, language-side direction in the early layers restores flipped answers (net recovery near 1.0); random and non-flip controls do nothing |
55
+ | Corroboration | An unsupervised sparse-autoencoder feature recovers that direction (\|cosine\| 0.74, ahead of principal-component analysis at 0.52); a Jacobian lens separates flipping from stable paraphrases (8.9x, from layer 0) |
56
+ | Ruled out | Decoder depth (flip rate 9.1% to 10.2% across depths 2 to 8) and weak grounding (inconclusive) |
57
+
58
+ The bundled weights are the **augmented seed-0** checkpoint (88.5% native accuracy,
59
+ 8.2% flip rate).
60
+
61
+ ## Load
62
 
63
  ```python
64
  import torch
65
  from transformers import AutoModel
66
+ from PIL import Image
67
 
68
  model = AutoModel.from_pretrained("saillab/babymedgemma", trust_remote_code=True).eval()
 
 
69
  input_ids, ans_pos = model.encode_question("is there cardiomegaly ?")
70
+ vision_features = model.encode_images([Image.open("cxr.png").convert("RGB")]) # frozen MedSigLIP at 896px
 
 
 
 
71
  logits = model(input_ids=input_ids, vision_features=vision_features, ans_pos=ans_pos).logits
72
  print(model.config.id2label[int(logits.argmax(-1))]) # "yes" or "no"
73
  ```
74
 
75
+ `encode_images` needs gated access to `google/medsiglip-448`. The precomputed
76
+ 896-pixel feature cache (`feature_cache/medsiglip_feats.pt`) and the per-run
77
+ `BabyGemmaVLM` checkpoints (`checkpoints/`) are in this repo; the code to
78
+ reproduce every experiment is on GitHub.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
79
 
80
  ## Citation
81
 
82
  > Sadanandan, B. (2026). *Paraphrase Sensitivity in Medical Vision-Language
83
+ > Models: Measurement, Mechanisms, Mitigation, and Deployment Safety.* PhD
84
+ > dissertation, University of New Haven.