binesh commited on
Commit
06b4d96
·
verified ·
1 Parent(s): 7afb903

Model card: transformers load instructions

Browse files
Files changed (1) hide show
  1. README.md +33 -30
README.md CHANGED
@@ -11,31 +11,49 @@ tags:
11
  - siglip
12
  base_model:
13
  - google/medsiglip-448
 
 
14
  pretty_name: baby-MedGemma
15
  ---
16
 
17
- # baby-MedGemma weights
18
 
19
- Checkpoints and the frozen MedSigLIP-448 feature cache for **baby-MedGemma**, a
20
- small MedGemma-faithful probe used to causally isolate *where paraphrase
21
- sensitivity in medical vision-language models originates*.
 
 
22
 
23
  - **Code, docs, and results:** https://github.com/thedatasense/babymedgemma
24
  - **Interactive write-up:** https://bineshkumar.me/phd-thesis/causality/
25
 
26
- ## What's in this repo
27
 
28
- | Path | Contents |
29
- |---|---|
30
- | `feature_cache/medsiglip_feats.pt` | Frozen MedSigLIP-448 features (256 pooled tokens per image) for the 1,775 chest radiographs, ~3 GB. Lets you train the probe without re-running the encoder. |
31
- | `checkpoints/` | 85 baby-Gemma per-run checkpoints (`model.pt`), mirroring the code repo's `results_gemma/` layout across the B, C, and D experiments. |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
32
 
33
  ## Model
34
 
35
- The probe pairs a frozen `google/medsiglip-448` vision encoder with a small
36
- **Gemma-3 decoder** (13.9M trainable parameters: rotary position embeddings,
37
- RMSNorm, grouped-query attention). The 256 image tokens are projected and
38
- prepended inline and read by the same causally-masked decoder, mirroring how
39
  MedGemma fuses vision and text. Because the encoder is frozen and identical for
40
  every paraphrase of a question, any answer that changes across paraphrases is
41
  language-side by construction. The probe reaches 88.8% accuracy and is
@@ -50,27 +68,12 @@ difference-direction patch restores flipped answers at the early layers with
50
  clean controls, so the flip is a low-rank, language-side, readout-stage
51
  direction. The origin is the training data; the fix is paraphrase augmentation.
52
 
53
- ## How to use
54
-
55
- ```bash
56
- git clone https://github.com/thedatasense/babymedgemma && cd babymedgemma
57
- huggingface-cli download saillab/babymedgemma feature_cache/medsiglip_feats.pt --local-dir .
58
- mv feature_cache/medsiglip_feats.pt cache/
59
- python train.py --arch gemma --regime augmented --seed 0
60
- ```
61
-
62
- Each checkpoint is a `state_dict` for the `BabyGemmaVLM` module in
63
- `gemma_model.py`; load it with `model.load_state_dict(torch.load(path))` after
64
- building the model with the matching vocab and dimensions (see `train.py`).
65
-
66
  ## Intended use and limitations
67
 
68
  Research artifact for interpretability of paraphrase sensitivity in medical VLMs.
69
  It is a controlled probe, not the deployed MedGemma-4B, trained on ~1,800
70
- questions; it grounds only weakly and manipulates adaptation-stage data, so it
71
- speaks to where a paraphrase flip is decided and why it is learnable, not to
72
- image use or pretraining provenance. **Not a medical device; not for clinical
73
- use.**
74
 
75
  ## Training data and license
76
 
 
11
  - siglip
12
  base_model:
13
  - google/medsiglip-448
14
+ library_name: transformers
15
+ pipeline_tag: image-text-to-text
16
  pretty_name: baby-MedGemma
17
  ---
18
 
19
+ # baby-MedGemma
20
 
21
+ A small, **MedGemma-faithful probe** for causally isolating *where paraphrase
22
+ sensitivity in medical vision-language models originates*: a frozen
23
+ `google/medsiglip-448` encoder with a small **Gemma-3 decoder** (rotary position
24
+ embeddings, RMSNorm, grouped-query attention: 6 query / 2 key-value heads),
25
+ image tokens prepended inline. Loadable directly with the `transformers` API.
26
 
27
  - **Code, docs, and results:** https://github.com/thedatasense/babymedgemma
28
  - **Interactive write-up:** https://bineshkumar.me/phd-thesis/causality/
29
 
30
+ ## Load and run
31
 
32
+ ```python
33
+ import torch
34
+ from transformers import AutoModel
35
+
36
+ model = AutoModel.from_pretrained("saillab/babymedgemma", trust_remote_code=True).eval()
37
+
38
+ # tokenize a presence question with the bundled vocabulary
39
+ input_ids, ans_pos = model.encode_question("is there cardiomegaly ?")
40
+
41
+ # frozen MedSigLIP-448 features for a chest radiograph (needs gated encoder access)
42
+ from PIL import Image
43
+ vision_features = model.encode_images([Image.open("cxr.png").convert("RGB")])
44
+
45
+ logits = model(input_ids=input_ids, vision_features=vision_features, ans_pos=ans_pos).logits
46
+ print(model.config.id2label[int(logits.argmax(-1))]) # "yes" or "no"
47
+ ```
48
+
49
+ The bundled `model.safetensors` is the augmented-regime seed-0 checkpoint. The
50
+ full set of 85 per-run checkpoints is under `checkpoints/`, and the frozen
51
+ MedSigLIP feature cache is at `feature_cache/medsiglip_feats.pt` (~3 GB).
52
 
53
  ## Model
54
 
55
+ The 256 image tokens are projected and prepended inline and read by the same
56
+ causally-masked Gemma-3 decoder (13.9M trainable parameters), mirroring how
 
 
57
  MedGemma fuses vision and text. Because the encoder is frozen and identical for
58
  every paraphrase of a question, any answer that changes across paraphrases is
59
  language-side by construction. The probe reaches 88.8% accuracy and is
 
68
  clean controls, so the flip is a low-rank, language-side, readout-stage
69
  direction. The origin is the training data; the fix is paraphrase augmentation.
70
 
 
 
 
 
 
 
 
 
 
 
 
 
 
71
  ## Intended use and limitations
72
 
73
  Research artifact for interpretability of paraphrase sensitivity in medical VLMs.
74
  It is a controlled probe, not the deployed MedGemma-4B, trained on ~1,800
75
+ questions over a small word-level vocabulary; it grounds only weakly and
76
+ manipulates adaptation-stage data. **Not a medical device; not for clinical use.**
 
 
77
 
78
  ## Training data and license
79