Visual Question Answering
Transformers
Safetensors
baby_medgemma
feature-extraction
medical
chest-x-ray
vision-language-model
interpretability
mechanistic-interpretability
research
custom_code
Instructions to use saillab/FlipLens with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use saillab/FlipLens with Transformers:
# Use a pipeline as a high-level helper # Warning: Pipeline type "visual-question-answering" is no longer supported in transformers v5. # You must load the model directly (see below) or downgrade to v4.x with: # pip install "transformers<5.0.0" from transformers import pipeline pipe = pipeline("visual-question-answering", model="saillab/FlipLens", trust_remote_code=True)# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("saillab/FlipLens", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
Card: single model
Browse files
README.md
CHANGED
|
@@ -29,22 +29,7 @@ sensitivity in medical vision-language models originates*.
|
|
| 29 |
> **Scope.** A research probe, **not a clinical model**. It answers binary presence
|
| 30 |
> questions about 14 chest findings. Not a medical device; not for clinical use.
|
| 31 |
|
| 32 |
-
##
|
| 33 |
-
|
| 34 |
-
| | default (this repo root) | `probe-1841/` |
|
| 35 |
-
|---|---|---|
|
| 36 |
-
| Purpose | grounded model that **transfers to unseen hospitals** | the **controlled probe** behind the dissertation's causal experiments |
|
| 37 |
-
| Training data | 107k per-finding-balanced questions (NIH + PadChest) | 1,841 questions (MIMIC-CXR + PadChest) |
|
| 38 |
-
| Tokenizer | MedGemma SentencePiece, pruned to 141 pieces | word-level, 734 words |
|
| 39 |
-
| Paraphrases | 48 templates across 4 phenomena | 4.8 generated per question |
|
| 40 |
-
| Grounding token | yes (MedSigLIP pooled embedding) | no |
|
| 41 |
-
|
| 42 |
-
The dissertation's Section 5.5 now describes the **default** model. `probe-1841/` is
|
| 43 |
-
kept for reproducing the earlier published figures (augmentation lever 29.5% -> 9.5%)
|
| 44 |
-
and the experiments not re-run at scale (divergence trajectory, decoder-depth sweep,
|
| 45 |
-
vision-dropout grounding sweep).
|
| 46 |
-
|
| 47 |
-
## What the default model does
|
| 48 |
|
| 49 |
Trained on NIH + PadChest, evaluated on **MIMIC and VinDr held out entirely** (no
|
| 50 |
image, and no hospital, seen in training). Every split is balanced per finding, so a
|
|
@@ -108,22 +93,16 @@ logits = model(input_ids=input_ids, vision_features=vision_features,
|
|
| 108 |
print(model.config.id2label[int(logits.argmax(-1))]) # "yes" or "no"
|
| 109 |
```
|
| 110 |
|
| 111 |
-
|
| 112 |
-
`ground` argument). `encode_images` needs gated access to `google/medsiglip-448`.
|
| 113 |
-
`feature_cache/` and `checkpoints/` in this repo belong to the **probe** variant.
|
| 114 |
|
| 115 |
## Repository structure
|
| 116 |
|
| 117 |
```
|
| 118 |
-
config.json model.safetensors modeling_babymedgemma.py
|
| 119 |
-
probe-1841/ the 1,841-question dissertation probe
|
| 120 |
-
feature_cache/medsiglip_feats.pt 896-pixel MedSigLIP cache (probe variant)
|
| 121 |
-
checkpoints/{B,C,D}/ per-run BabyGemmaVLM state_dicts (probe variant)
|
| 122 |
```
|
| 123 |
|
| 124 |
Training and analysis code, and the exact scripts that produced every number here, are on
|
| 125 |
-
GitHub: [thedatasense/babymedgemma](https://github.com/thedatasense/babymedgemma).
|
| 126 |
-
either model with the same call; the probe needs `subfolder="probe-1841"`.
|
| 127 |
|
| 128 |
## Citation
|
| 129 |
|
|
|
|
| 29 |
> **Scope.** A research probe, **not a clinical model**. It answers binary presence
|
| 30 |
> questions about 14 chest findings. Not a medical device; not for clinical use.
|
| 31 |
|
| 32 |
+
## What it does
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 33 |
|
| 34 |
Trained on NIH + PadChest, evaluated on **MIMIC and VinDr held out entirely** (no
|
| 35 |
image, and no hospital, seen in training). Every split is balanced per finding, so a
|
|
|
|
| 93 |
print(model.config.id2label[int(logits.argmax(-1))]) # "yes" or "no"
|
| 94 |
```
|
| 95 |
|
| 96 |
+
`encode_images` needs gated access to `google/medsiglip-448`.
|
|
|
|
|
|
|
| 97 |
|
| 98 |
## Repository structure
|
| 99 |
|
| 100 |
```
|
| 101 |
+
config.json model.safetensors modeling_babymedgemma.py the model + self-contained wrapper
|
|
|
|
|
|
|
|
|
|
| 102 |
```
|
| 103 |
|
| 104 |
Training and analysis code, and the exact scripts that produced every number here, are on
|
| 105 |
+
GitHub: [thedatasense/babymedgemma](https://github.com/thedatasense/babymedgemma).
|
|
|
|
| 106 |
|
| 107 |
## Citation
|
| 108 |
|