license: other
license_name: physionet-dua
license_link: https://physionet.org/about/licenses/
library_name: pytorch
tags:
- chest-xray
- radiology
- vision-encoder
- vit
- medical-imaging
pipeline_tag: image-feature-extraction
extra_gated_heading: Access requires PhysioNet credentialing
extra_gated_prompt: >-
These weights are derived from MIMIC-CXR and Chest ImaGenome, distributed
under the PhysioNet Credentialed Health Data Use Agreement. By requesting
access you confirm you hold the required PhysioNet credentialed access for
BOTH source datasets and will comply with the PhysioNet DUA. Access is granted
manually by the repository owner.
extra_gated_fields:
Full name: text
Affiliation: text
Intended use: text
I hold PhysioNet credentialed access to MIMIC-CXR and Chest ImaGenome: checkbox
I will comply with the PhysioNet Data Use Agreement and not redistribute these weights: checkbox
extra_gated_button_content: Request access
CXR ViT-L/14 — captioning
ViT-L/14 chest-radiograph vision encoder, pretrained from scratch with the objective: Global image -> full radiology report captioning (20 epochs).
Part of a controlled study comparing pretraining objectives for chest-X-ray vision encoders (all share the same ViT-L/14 backbone, data, and budget).
Files
| File | What it is |
|---|---|
encoder_final.pt |
Vision encoder — ViT-L/14 trunk only (state_dict under key encoder_state, 294 tensors). Use this to extract frozen image features. |
model_best.pt |
Full pretraining model — ViT-L/14 + GPT-2-style causal captioning decoder (state_dict under key model_state). |
Architecture & training
- ViT-L/14, 384x384 input -> 729 patch tokens (27x27 grid), trained from scratch in bf16. Text side (where present) uses the GPT-2 tokenizer (vocab 50257).
- Training data: MIMIC-CXR (reports) and Chest ImaGenome (region-phrase / anatomy boxes).
- Objective: Global image -> full radiology report captioning (20 epochs).
Downstream results (frozen-feature transfer)
MIMIC RRG RadGraph-s 0.197 (best of from-scratch); PG-box mIoU 0.237; VQA BLEU-4 0.187.
Full per-task comparison across all 9 from-scratch encoders is in the project logbook.
Usage (vision encoder)
import torch
ckpt = torch.load("encoder_final.pt", map_location="cpu", weights_only=False)
vision_state = ckpt["encoder_state"] # 294 tensors, ViT-L/14
# load into your ViT-L/14 implementation, then forward 384x384 images -> [B, 729, 1024]
Intended use & limitations
- Research use only. Frozen feature extraction / fine-tuning for chest-X-ray tasks.
- NOT a diagnostic device. No clinical or patient-facing use.
- Trained only on adult frontal/lateral CXR distributions of the source datasets; may not generalize to other modalities, body regions, populations, or acquisition settings.
⚠️ License / Data Use Agreement
These weights are derived from MIMIC-CXR and Chest ImaGenome, which are distributed under the PhysioNet Credentialed Health Data Use Agreement. Redistribution and use of models derived from these data are subject to that DUA — you must hold the appropriate credentialed access and comply with its terms. Do not make this repository public or share access without confirming your DUA permits sharing derived model weights.
Citation
If you use this encoder, please cite the source datasets (MIMIC-CXR; Chest ImaGenome) and this project.