| --- |
| license: other |
| license_name: physionet-dua |
| license_link: https://physionet.org/about/licenses/ |
| library_name: pytorch |
| tags: [chest-xray, radiology, vision-encoder, vit, medical-imaging] |
| pipeline_tag: image-feature-extraction |
| extra_gated_heading: "Access requires PhysioNet credentialing" |
| extra_gated_prompt: >- |
| These weights are derived from MIMIC-CXR and Chest ImaGenome, distributed under the |
| PhysioNet Credentialed Health Data Use Agreement. By requesting access you confirm you |
| hold the required PhysioNet credentialed access for BOTH source datasets and will comply |
| with the PhysioNet DUA. Access is granted manually by the repository owner. |
| extra_gated_fields: |
| Full name: text |
| Affiliation: text |
| Intended use: text |
| I hold PhysioNet credentialed access to MIMIC-CXR and Chest ImaGenome: checkbox |
| I will comply with the PhysioNet Data Use Agreement and not redistribute these weights: checkbox |
| extra_gated_button_content: "Request access" |
| --- |
| |
| # CXR ViT-L/14 — captioning |
|
|
| ViT-L/14 chest-radiograph vision encoder, **pretrained from scratch** with the objective: |
| **Global image -> full radiology report captioning (20 epochs).** |
|
|
| Part of a controlled study comparing pretraining objectives for chest-X-ray vision encoders |
| (all share the same ViT-L/14 backbone, data, and budget). |
|
|
| ## Files |
| | File | What it is | |
| |---|---| |
| | `encoder_final.pt` | **Vision encoder** — ViT-L/14 trunk only (`state_dict` under key `encoder_state`, 294 tensors). Use this to extract frozen image features. | |
| | `model_best.pt` | **Full pretraining model** — ViT-L/14 + GPT-2-style causal captioning decoder (`state_dict` under key `model_state`). | |
|
|
| ## Architecture & training |
| - ViT-L/14, 384x384 input -> 729 patch tokens (27x27 grid), trained **from scratch** in bf16. Text side (where present) uses the GPT-2 tokenizer (vocab 50257). |
| - **Training data:** MIMIC-CXR (reports) and Chest ImaGenome (region-phrase / anatomy boxes). |
| - **Objective:** Global image -> full radiology report captioning (20 epochs). |
|
|
| ## Downstream results (frozen-feature transfer) |
| MIMIC RRG RadGraph-s 0.197 (best of from-scratch); PG-box mIoU 0.237; VQA BLEU-4 0.187. |
|
|
| Full per-task comparison across all 9 from-scratch encoders is in the project logbook. |
|
|
| ## Usage (vision encoder) |
| ```python |
| import torch |
| ckpt = torch.load("encoder_final.pt", map_location="cpu", weights_only=False) |
| vision_state = ckpt["encoder_state"] # 294 tensors, ViT-L/14 |
| # load into your ViT-L/14 implementation, then forward 384x384 images -> [B, 729, 1024] |
| ``` |
|
|
| ## Intended use & limitations |
| - **Research use only.** Frozen feature extraction / fine-tuning for chest-X-ray tasks. |
| - **NOT a diagnostic device.** No clinical or patient-facing use. |
| - Trained only on adult frontal/lateral CXR distributions of the source datasets; may not |
| generalize to other modalities, body regions, populations, or acquisition settings. |
|
|
| ## ⚠️ License / Data Use Agreement |
| These weights are derived from **MIMIC-CXR** and **Chest ImaGenome**, which are distributed |
| under the **PhysioNet Credentialed Health Data Use Agreement**. Redistribution and use of |
| models derived from these data are subject to that DUA — you must hold the appropriate |
| credentialed access and comply with its terms. Do **not** make this repository public or |
| share access without confirming your DUA permits sharing derived model weights. |
|
|
| ## Citation |
| If you use this encoder, please cite the source datasets (MIMIC-CXR; Chest ImaGenome) and |
| this project. |
|
|
