Upload MIMIC test evaluation results

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+assets/AnatomicalAttention.gif filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,197 @@
+---
+license: mit
+library_name: transformers
+pipeline_tag: image-to-text
+tags:
+  - medical-ai
+  - radiology
+  - chest-xray
+  - report-generation
+  - segmentation
+  - anatomical-attention
+metrics:
+  - BLEU
+  - METEOR
+  - ROUGE
+  - CIDEr
+---
+
+# LAnA
+
+**Layer-Wise Anatomical Attention model**
+
+[![ArXiv](https://img.shields.io/badge/ArXiv-2512.16841-B31B1B?logo=arxiv&logoColor=white)](https://arxiv.org/abs/2512.16841)
+[![LinkedIn](https://img.shields.io/badge/LinkedIn-devmuniz-0A66C2?logo=linkedin&logoColor=white)](https://www.linkedin.com/in/devmuniz)
+[![GitHub Profile](https://img.shields.io/badge/GitHub-devMuniz02-181717?logo=github&logoColor=white)](https://github.com/devMuniz02)
+[![Portfolio](https://img.shields.io/badge/Portfolio-devmuniz02.github.io-0F172A?logo=googlechrome&logoColor=white)](https://devmuniz02.github.io/)
+[![GitHub Repo](https://img.shields.io/badge/Repository-layer--wise--anatomical--attention-181717?logo=github&logoColor=white)](https://github.com/devMuniz02/layer-wise-anatomical-attention)
+[![Hugging Face](https://img.shields.io/badge/Hugging%20Face-manu02-FFD21E?logoColor=black)](https://huggingface.co/manu02)
+
+![Layer-Wise Anatomical Attention](assets/AnatomicalAttention.gif)
+
+## Overview
+
+LAnA is a medical report-generation project for chest X-ray images. The completed project is intended to generate radiology reports with a vision-language model guided by layer-wise anatomical attention built from predicted anatomical masks.
+
+The architecture combines a DINOv3 vision encoder, lung and heart segmentation heads, and a GPT-2 decoder modified so each transformer layer receives a different anatomical attention bias derived from the segmentation mask.
+
+## How to Run
+
+Standard `AutoModel.from_pretrained(..., trust_remote_code=True)` loading is currently blocked for this repo because the custom model constructor performs nested pretrained submodel loads.
+Use the verified manual load path below instead: download the HF repo snapshot, import the downloaded package, and load the exported `model.safetensors` directly.
+You must set an `HF_TOKEN` environment variable with permission to access the DINOv3 model repositories used by this project, otherwise the required vision backbones cannot be downloaded.
+
+```python
+from pathlib import Path
+import sys
+
+import numpy as np
+import torch
+from PIL import Image
+from huggingface_hub import snapshot_download
+from safetensors.torch import load_file
+from transformers import AutoTokenizer
+
+repo_dir = Path(snapshot_download("manu02/LAnA"))
+sys.path.insert(0, str(repo_dir))
+
+from lana_radgen import LanaConfig, LanaForConditionalGeneration
+
+config = LanaConfig.from_pretrained(repo_dir)
+config.lung_segmenter_checkpoint = str(repo_dir / "segmenters" / "lung_segmenter_dinounet_finetuned.pth")
+config.heart_segmenter_checkpoint = str(repo_dir / "segmenters" / "heart_segmenter_dinounet_best.pth")
+
+device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
+
+model = LanaForConditionalGeneration(config)
+state_dict = load_file(str(repo_dir / "model.safetensors"))
+missing, unexpected = model.load_state_dict(state_dict, strict=True)
+assert not missing and not unexpected
+
+model.tokenizer = AutoTokenizer.from_pretrained(repo_dir, trust_remote_code=True)
+model.move_non_quantized_modules(device)
+model.eval()
+
+image_path = Path("example.png")
+image = Image.open(image_path).convert("RGB")
+image = image.resize((512, 512), resample=Image.BICUBIC)
+array = np.asarray(image, dtype=np.float32) / 255.0
+pixel_values = torch.from_numpy(array).permute(2, 0, 1)
+mean = torch.tensor([0.485, 0.456, 0.406]).view(3, 1, 1)
+std = torch.tensor([0.229, 0.224, 0.225]).view(3, 1, 1)
+pixel_values = ((pixel_values - mean) / std).unsqueeze(0).to(device)
+
+with torch.no_grad():
+    generated = model.generate(pixel_values=pixel_values, max_new_tokens=128)
+
+report = model.tokenizer.batch_decode(generated, skip_special_tokens=True)[0]
+print(report)
+```
+
+## Intended Use
+
+- Input: a chest X-ray image resized to `512x512` and normalized with ImageNet mean/std.
+- Output: a generated radiology report.
+- Best fit: research use, report-generation experiments, and anatomical-attention ablations.
+
+## MIMIC Test Results
+
+Frontal-only evaluation using `PA/AP` studies only.
+
+### Current Checkpoint Results
+
+### All Frontal Test Studies
+
+| Metric | Value |
+| --- | --- |
+| Number of studies | `3041` |
+| ROUGE-L | `0.1641` |
+| BLEU-1 | `0.2243` |
+| BLEU-4 | `0.0383` |
+| METEOR | `0.2005` |
+| RadGraph F1 | `0.0941` |
+| RadGraph entity F1 | `0.1819` |
+| RadGraph relation F1 | `0.1652` |
+| CheXpert F1 14-micro | `0.1245` |
+| CheXpert F1 5-micro | `0.2190` |
+| CheXpert F1 14-macro | `0.0443` |
+| CheXpert F1 5-macro | `0.0991` |
+
+### Findings-Only Frontal Test Studies
+
+| Metric | Value |
+| --- | --- |
+| Number of studies | `2210` |
+| ROUGE-L | `0.1721` |
+| BLEU-1 | `0.2310` |
+| BLEU-4 | `0.0429` |
+| METEOR | `0.2125` |
+| RadGraph F1 | `0.1017` |
+| RadGraph entity F1 | `0.1922` |
+| RadGraph relation F1 | `0.1741` |
+| CheXpert F1 14-micro | `0.1166` |
+| CheXpert F1 5-micro | `0.2071` |
+| CheXpert F1 14-macro | `0.0406` |
+| CheXpert F1 5-macro | `0.0920` |
+
+### Final Completed Training Results
+
+The final table will be populated when the planned training run is completed. Until then, final-report metrics remain `TBD`.
+
+| Metric | Value |
+| --- | --- |
+| Number of studies | TBD |
+| RadGraph F1 | TBD |
+| RadGraph entity F1 | TBD |
+| RadGraph relation F1 | TBD |
+| CheXpert F1 14-micro | TBD |
+| CheXpert F1 5-micro | TBD |
+| CheXpert F1 14-macro | TBD |
+| CheXpert F1 5-macro | TBD |
+
+## Data
+
+- Full project datasets: CheXpert and MIMIC-CXR.
+- Intended project scope: train on curated chest X-ray/report data from both datasets and evaluate on MIMIC-CXR test studies.
+- Current released checkpoint datasets: `MIMIC-CXR (findings-only)` for training and `MIMIC-CXR (findings-only)` for validation.
+- Current published evaluation: MIMIC-CXR test split, `frontal-only (PA/AP)` studies.
+
+## Evaluation
+
+- Medical report metrics implemented in the repository include RadGraph F1 and CheXpert F1 (`14-micro`, `5-micro`, `14-macro`, `5-macro`).
+
+## Training Snapshot
+
+- Run: `mimic only`
+- This section describes the current public checkpoint, not the final completed project.
+- Method: `lora_adamw`
+- Vision encoder: `facebook/dinov3-vits16-pretrain-lvd1689m`
+- Text decoder: `gpt2`
+- Segmentation encoder: `facebook/dinov3-convnext-small-pretrain-lvd1689m`
+- Image size: `512`
+- Local batch size: `1`
+- Effective global batch size: `8`
+- Scheduler: `cosine`
+- Warmup steps: `2636`
+- Weight decay: `0.01`
+- Steps completed: `6692`
+- Planned total steps: `52716`
+- Images seen: `53540`
+- Total training time: `1.0000` hours
+- Hardware: `NVIDIA GeForce RTX 5070`
+- Final train loss: `0.6296`
+- Validation loss: `2.3133`
+
+## Status
+
+- Project status: `Training in progress`
+- Release status: `Research preview checkpoint`
+- Current checkpoint status: `Not final`
+- Training completion toward planned run: `12.70%` (`0` / `3` epochs)
+- Current published metrics are intermediate and will change as training continues.
+
+## Notes
+
+- Set `HF_TOKEN` with permission to access the DINOv3 repositories required by this model before downloading or running inference.
+- `segmenters/` contains the lung and heart segmentation checkpoints used to build anatomical attention masks.
+- `evaluations/mimic_test_metrics.json` contains the latest saved MIMIC test metrics.

benchmark_results.json

@@ -0,0 +1,391 @@
+{
+  "results": [
+    {
+      "method": "qlora_paged_adamw8bit",
+      "local_batch_size": 1,
+      "global_batch_size_requested": 1,
+      "status": "failed",
+      "error": "element 0 of tensors does not require grad and does not have a grad_fn"
+    },
+    {
+      "method": "qlora_paged_adamw8bit",
+      "local_batch_size": 1,
+      "global_batch_size_requested": 8,
+      "status": "failed",
+      "error": "element 0 of tensors does not require grad and does not have a grad_fn"
+    },
+    {
+      "method": "qlora_paged_adamw8bit",
+      "local_batch_size": 1,
+      "global_batch_size_requested": 16,
+      "status": "failed",
+      "error": "element 0 of tensors does not require grad and does not have a grad_fn"
+    },
+    {
+      "method": "qlora_paged_adamw8bit",
+      "local_batch_size": 2,
+      "global_batch_size_requested": 2,
+      "status": "failed",
+      "error": "element 0 of tensors does not require grad and does not have a grad_fn"
+    },
+    {
+      "method": "qlora_paged_adamw8bit",
+      "local_batch_size": 2,
+      "global_batch_size_requested": 8,
+      "status": "failed",
+      "error": "element 0 of tensors does not require grad and does not have a grad_fn"
+    },
+    {
+      "method": "qlora_paged_adamw8bit",
+      "local_batch_size": 2,
+      "global_batch_size_requested": 16,
+      "status": "failed",
+      "error": "element 0 of tensors does not require grad and does not have a grad_fn"
+    },
+    {
+      "method": "qlora_paged_adamw8bit",
+      "local_batch_size": 4,
+      "global_batch_size_requested": 4,
+      "status": "failed",
+      "error": "element 0 of tensors does not require grad and does not have a grad_fn"
+    },
+    {
+      "method": "qlora_paged_adamw8bit",
+      "local_batch_size": 4,
+      "global_batch_size_requested": 8,
+      "status": "failed",
+      "error": "element 0 of tensors does not require grad and does not have a grad_fn"
+    },
+    {
+      "method": "qlora_paged_adamw8bit",
+      "local_batch_size": 4,
+      "global_batch_size_requested": 16,
+      "status": "failed",
+      "error": "element 0 of tensors does not require grad and does not have a grad_fn"
+    },
+    {
+      "method": "lora_adamw",
+      "local_batch_size": 1,
+      "global_batch_size_requested": 1,
+      "status": "ok",
+      "effective_global_batch_size": 1,
+      "gradient_accumulation_steps": 1,
+      "optimizer_step_time_sec": 0.12944729999981064,
+      "images_per_sec": 7.7251514709187665,
+      "mean_loss": 9.920842170715332,
+      "trainable_params": 1106688
+    },
+    {
+      "method": "lora_adamw",
+      "local_batch_size": 1,
+      "global_batch_size_requested": 8,
+      "status": "ok",
+      "effective_global_batch_size": 8,
+      "gradient_accumulation_steps": 8,
+      "optimizer_step_time_sec": 0.792737899999338,
+      "images_per_sec": 10.091607831550228,
+      "mean_loss": 8.131502032279968,
+      "trainable_params": 1106688
+    },
+    {
+      "method": "lora_adamw",
+      "local_batch_size": 1,
+      "global_batch_size_requested": 16,
+      "status": "ok",
+      "effective_global_batch_size": 16,
+      "gradient_accumulation_steps": 16,
+      "optimizer_step_time_sec": 1.6773667999987083,
+      "images_per_sec": 9.538760395169572,
+      "mean_loss": 8.80642619729042,
+      "trainable_params": 1106688
+    },
+    {
+      "method": "lora_adamw",
+      "local_batch_size": 2,
+      "global_batch_size_requested": 2,
+      "status": "ok",
+      "effective_global_batch_size": 2,
+      "gradient_accumulation_steps": 1,
+      "optimizer_step_time_sec": 0.20009290000052715,
+      "images_per_sec": 9.995357156574427,
+      "mean_loss": 9.088608741760254,
+      "trainable_params": 1106688
+    },
+    {
+      "method": "lora_adamw",
+      "local_batch_size": 2,
+      "global_batch_size_requested": 8,
+      "status": "ok",
+      "effective_global_batch_size": 8,
+      "gradient_accumulation_steps": 4,
+      "optimizer_step_time_sec": 0.8304937000011705,
+      "images_per_sec": 9.63282442719159,
+      "mean_loss": 8.245712995529175,
+      "trainable_params": 1106688
+    },
+    {
+      "method": "lora_adamw",
+      "local_batch_size": 2,
+      "global_batch_size_requested": 16,
+      "status": "ok",
+      "effective_global_batch_size": 16,
+      "gradient_accumulation_steps": 8,
+      "optimizer_step_time_sec": 1.6668036999981268,
+      "images_per_sec": 9.599210752902685,
+      "mean_loss": 9.106984257698059,
+      "trainable_params": 1106688
+    },
+    {
+      "method": "lora_adamw",
+      "local_batch_size": 4,
+      "global_batch_size_requested": 4,
+      "status": "ok",
+      "effective_global_batch_size": 4,
+      "gradient_accumulation_steps": 1,
+      "optimizer_step_time_sec": 0.4656030999994982,
+      "images_per_sec": 8.591008092524106,
+      "mean_loss": 8.862140655517578,
+      "trainable_params": 1106688
+    },
+    {
+      "method": "lora_adamw",
+      "local_batch_size": 4,
+      "global_batch_size_requested": 8,
+      "status": "ok",
+      "effective_global_batch_size": 8,
+      "gradient_accumulation_steps": 2,
+      "optimizer_step_time_sec": 2.6093234999989363,
+      "images_per_sec": 3.0659287742601715,
+      "mean_loss": 8.241507053375244,
+      "trainable_params": 1106688
+    },
+    {
+      "method": "lora_adamw",
+      "local_batch_size": 4,
+      "global_batch_size_requested": 16,
+      "status": "ok",
+      "effective_global_batch_size": 16,
+      "gradient_accumulation_steps": 4,
+      "optimizer_step_time_sec": 18.058491499999946,
+      "images_per_sec": 0.8860097755119827,
+      "mean_loss": 8.916554927825928,
+      "trainable_params": 1106688
+    },
+    {
+      "method": "full_adam",
+      "local_batch_size": 1,
+      "global_batch_size_requested": 1,
+      "status": "ok",
+      "effective_global_batch_size": 1,
+      "gradient_accumulation_steps": 1,
+      "optimizer_step_time_sec": 1.4309436000003188,
+      "images_per_sec": 0.6988395629288094,
+      "mean_loss": 8.042855262756348,
+      "trainable_params": 125521920
+    },
+    {
+      "method": "full_adam",
+      "local_batch_size": 1,
+      "global_batch_size_requested": 8,
+      "status": "ok",
+      "effective_global_batch_size": 8,
+      "gradient_accumulation_steps": 8,
+      "optimizer_step_time_sec": 2.7121656999988772,
+      "images_per_sec": 2.9496722858796245,
+      "mean_loss": 7.829526960849762,
+      "trainable_params": 125521920
+    },
+    {
+      "method": "full_adam",
+      "local_batch_size": 1,
+      "global_batch_size_requested": 16,
+      "status": "ok",
+      "effective_global_batch_size": 16,
+      "gradient_accumulation_steps": 16,
+      "optimizer_step_time_sec": 1.8378386999993381,
+      "images_per_sec": 8.705878268863183,
+      "mean_loss": 9.189274996519089,
+      "trainable_params": 125521920
+    },
+    {
+      "method": "full_adam",
+      "local_batch_size": 2,
+      "global_batch_size_requested": 2,
+      "status": "ok",
+      "effective_global_batch_size": 2,
+      "gradient_accumulation_steps": 1,
+      "optimizer_step_time_sec": 0.23647629999868514,
+      "images_per_sec": 8.457507158269646,
+      "mean_loss": 9.128178596496582,
+      "trainable_params": 125521920
+    },
+    {
+      "method": "full_adam",
+      "local_batch_size": 2,
+      "global_batch_size_requested": 8,
+      "status": "ok",
+      "effective_global_batch_size": 8,
+      "gradient_accumulation_steps": 4,
+      "optimizer_step_time_sec": 0.8083188999989943,
+      "images_per_sec": 9.897083935572896,
+      "mean_loss": 8.64337944984436,
+      "trainable_params": 125521920
+    },
+    {
+      "method": "full_adam",
+      "local_batch_size": 2,
+      "global_batch_size_requested": 16,
+      "status": "ok",
+      "effective_global_batch_size": 16,
+      "gradient_accumulation_steps": 8,
+      "optimizer_step_time_sec": 1.8274533999974665,
+      "images_per_sec": 8.755353214490823,
+      "mean_loss": 8.331470370292664,
+      "trainable_params": 125521920
+    },
+    {
+      "method": "full_adam",
+      "local_batch_size": 4,
+      "global_batch_size_requested": 4,
+      "status": "ok",
+      "effective_global_batch_size": 4,
+      "gradient_accumulation_steps": 1,
+      "optimizer_step_time_sec": 0.511095199999545,
+      "images_per_sec": 7.826330593602838,
+      "mean_loss": 8.954268455505371,
+      "trainable_params": 125521920
+    },
+    {
+      "method": "full_adam",
+      "local_batch_size": 4,
+      "global_batch_size_requested": 8,
+      "status": "ok",
+      "effective_global_batch_size": 8,
+      "gradient_accumulation_steps": 2,
+      "optimizer_step_time_sec": 2.2738564999981463,
+      "images_per_sec": 3.518251921353226,
+      "mean_loss": 9.192809581756592,
+      "trainable_params": 125521920
+    },
+    {
+      "method": "full_adam",
+      "local_batch_size": 4,
+      "global_batch_size_requested": 16,
+      "status": "ok",
+      "effective_global_batch_size": 16,
+      "gradient_accumulation_steps": 4,
+      "optimizer_step_time_sec": 18.631701800000883,
+      "images_per_sec": 0.8587513997244869,
+      "mean_loss": 8.159156560897827,
+      "trainable_params": 125521920
+    },
+    {
+      "method": "full_adam8bit",
+      "local_batch_size": 1,
+      "global_batch_size_requested": 1,
+      "status": "ok",
+      "effective_global_batch_size": 1,
+      "gradient_accumulation_steps": 1,
+      "optimizer_step_time_sec": 0.13992360000156623,
+      "images_per_sec": 7.146757230294293,
+      "mean_loss": 9.259998321533203,
+      "trainable_params": 125521920
+    },
+    {
+      "method": "full_adam8bit",
+      "local_batch_size": 1,
+      "global_batch_size_requested": 8,
+      "status": "ok",
+      "effective_global_batch_size": 8,
+      "gradient_accumulation_steps": 8,
+      "optimizer_step_time_sec": 0.8451360999988538,
+      "images_per_sec": 9.465930990299492,
+      "mean_loss": 8.10985803604126,
+      "trainable_params": 125521920
+    },
+    {
+      "method": "full_adam8bit",
+      "local_batch_size": 1,
+      "global_batch_size_requested": 16,
+      "status": "ok",
+      "effective_global_batch_size": 16,
+      "gradient_accumulation_steps": 16,
+      "optimizer_step_time_sec": 1.8945816999930685,
+      "images_per_sec": 8.445135936897595,
+      "mean_loss": 8.591163873672485,
+      "trainable_params": 125521920
+    },
+    {
+      "method": "full_adam8bit",
+      "local_batch_size": 2,
+      "global_batch_size_requested": 2,
+      "status": "ok",
+      "effective_global_batch_size": 2,
+      "gradient_accumulation_steps": 1,
+      "optimizer_step_time_sec": 0.23971350000101666,
+      "images_per_sec": 8.343293139483249,
+      "mean_loss": 9.75894832611084,
+      "trainable_params": 125521920
+    },
+    {
+      "method": "full_adam8bit",
+      "local_batch_size": 2,
+      "global_batch_size_requested": 8,
+      "status": "ok",
+      "effective_global_batch_size": 8,
+      "gradient_accumulation_steps": 4,
+      "optimizer_step_time_sec": 0.9259438999997656,
+      "images_per_sec": 8.6398322835779,
+      "mean_loss": 8.462790489196777,
+      "trainable_params": 125521920
+    },
+    {
+      "method": "full_adam8bit",
+      "local_batch_size": 2,
+      "global_batch_size_requested": 16,
+      "status": "ok",
+      "effective_global_batch_size": 16,
+      "gradient_accumulation_steps": 8,
+      "optimizer_step_time_sec": 1.8237968999983423,
+      "images_per_sec": 8.772906676184471,
+      "mean_loss": 10.191668510437012,
+      "trainable_params": 125521920
+    },
+    {
+      "method": "full_adam8bit",
+      "local_batch_size": 4,
+      "global_batch_size_requested": 4,
+      "status": "ok",
+      "effective_global_batch_size": 4,
+      "gradient_accumulation_steps": 1,
+      "optimizer_step_time_sec": 0.5224713000006886,
+      "images_per_sec": 7.655922918626779,
+      "mean_loss": 8.14057445526123,
+      "trainable_params": 125521920
+    },
+    {
+      "method": "full_adam8bit",
+      "local_batch_size": 4,
+      "global_batch_size_requested": 8,
+      "status": "ok",
+      "effective_global_batch_size": 8,
+      "gradient_accumulation_steps": 2,
+      "optimizer_step_time_sec": 3.7809107000011863,
+      "images_per_sec": 2.1158923430795364,
+      "mean_loss": 8.521550178527832,
+      "trainable_params": 125521920
+    },
+    {
+      "method": "full_adam8bit",
+      "local_batch_size": 4,
+      "global_batch_size_requested": 16,
+      "status": "ok",
+      "effective_global_batch_size": 16,
+      "gradient_accumulation_steps": 4,
+      "optimizer_step_time_sec": 27.688971800002037,
+      "images_per_sec": 0.5778473868790903,
+      "mean_loss": 9.247632026672363,
+      "trainable_params": 125521920
+    }
+  ]
+}

config.json

@@ -0,0 +1,33 @@
+{
+  "anatomical_attention_bias": 2.0,
+  "architectures": [
+    "LanaForConditionalGeneration"
+  ],
+  "decoder_compute_dtype": "bfloat16",
+  "decoder_load_in_4bit": false,
+  "dtype": "float32",
+  "freeze_segmenter": true,
+  "heart_segmenter_checkpoint": "segmenters/heart_segmenter_dinounet_best.pth",
+  "image_size": 512,
+  "layer_mask_base_kernel_size": 3,
+  "layer_mask_kernel_growth": 2,
+  "lung_segmenter_checkpoint": "segmenters/lung_segmenter_dinounet_finetuned.pth",
+  "mask_size": 32,
+  "max_position_embeddings": 2048,
+  "model_type": "lana_radgen",
+  "num_attention_layers": 12,
+  "segmentation_attention_implementation": "sdpa",
+  "segmentation_model_name": "facebook/dinov3-convnext-small-pretrain-lvd1689m",
+  "text_hidden_size": 768,
+  "text_model_name": "gpt2",
+  "transformers_version": "5.3.0",
+  "use_cache": true,
+  "use_segmentation_mask": true,
+  "vision_model_name": "facebook/dinov3-vits16-pretrain-lvd1689m",
+  "visual_feature_dim": 384,
+  "vocab_size": 50257,
+  "auto_map": {
+    "AutoConfig": "configuration_lana.LanaConfig",
+    "AutoModel": "modeling_lana.LanaForConditionalGeneration"
+  }
+}

configuration_lana.py

@@ -0,0 +1,3 @@
+from lana_radgen.configuration_lana import LanaConfig
+
+__all__ = ["LanaConfig"]

evaluations/mimic_test_findings_only_metrics.json

@@ -0,0 +1,38 @@
+{
+  "split": "test",
+  "subset": "findings-only frontal studies",
+  "dataset": "mimic-cxr",
+  "view_filter": "frontal-only (PA/AP), structured Findings section only",
+  "num_examples": 2210,
+  "bleu_1": 0.23099023872215996,
+  "bleu_4": 0.0429479479188206,
+  "meteor": 0.21248313160360002,
+  "rouge_l": 0.17210734193417726,
+  "chexpert_f1_14_micro": 0.11655011655011654,
+  "chexpert_f1_5_micro": 0.20709914320685435,
+  "chexpert_f1_14_macro": 0.04057070914402376,
+  "chexpert_f1_5_macro": 0.09202593660588262,
+  "chexpert_f1_micro": 0.11655011655011654,
+  "chexpert_f1_macro": 0.04057070914402376,
+  "chexpert_per_label_f1": {
+    "Enlarged Cardiomediastinum": 0.0,
+    "Cardiomegaly": 0.0,
+    "Lung Opacity": 0.0,
+    "Lung Lesion": 0.0,
+    "Edema": 0.022471910112359553,
+    "Consolidation": 0.05797101449275362,
+    "Pneumonia": 0.01673640167364017,
+    "Atelectasis": 0.0,
+    "Pneumothorax": 0.05716798592788039,
+    "Pleural Effusion": 0.3796867584243,
+    "Pleural Other": 0.0,
+    "Fracture": 0.0,
+    "Support Devices": 0.03395585738539898,
+    "No Finding": 0.0
+  },
+  "radgraph_f1": 0.10172866854646034,
+  "radgraph_f1_entity": 0.19217701907879298,
+  "radgraph_f1_relation": 0.17414731467894073,
+  "radgraph_available": true,
+  "radgraph_error": null
+}

evaluations/mimic_test_metrics.json

@@ -0,0 +1,115 @@
+{
+  "split": "test",
+  "subset": "all frontal studies",
+  "dataset": "mimic-cxr",
+  "view_filter": "frontal-only (PA/AP)",
+  "num_examples": 3041,
+  "bleu_1": 0.224283665754139,
+  "bleu_4": 0.038250603325884945,
+  "meteor": 0.20053919101008125,
+  "rouge_l": 0.16408262967411588,
+  "chexpert_f1_14_micro": 0.1244686823321838,
+  "chexpert_f1_5_micro": 0.21896350608231963,
+  "chexpert_f1_14_macro": 0.04429260458577706,
+  "chexpert_f1_5_macro": 0.09914404243362482,
+  "chexpert_f1_micro": 0.1244686823321838,
+  "chexpert_f1_macro": 0.04429260458577706,
+  "chexpert_per_label_f1": {
+    "Enlarged Cardiomediastinum": 0.0,
+    "Cardiomegaly": 0.0,
+    "Lung Opacity": 0.0,
+    "Lung Lesion": 0.0,
+    "Edema": 0.023121387283236997,
+    "Consolidation": 0.056790123456790124,
+    "Pneumonia": 0.02762430939226519,
+    "Atelectasis": 0.0,
+    "Pneumothorax": 0.059987236758136574,
+    "Pleural Effusion": 0.415808701428097,
+    "Pleural Other": 0.0,
+    "Fracture": 0.0,
+    "Support Devices": 0.036764705882352935,
+    "No Finding": 0.0
+  },
+  "radgraph_f1": 0.0941067057393548,
+  "radgraph_f1_entity": 0.18191243977753782,
+  "radgraph_f1_relation": 0.1652384677607375,
+  "radgraph_available": true,
+  "radgraph_error": null,
+  "evaluation_suite": "mimic_test_dual",
+  "all_test": {
+    "split": "test",
+    "subset": "all frontal studies",
+    "dataset": "mimic-cxr",
+    "view_filter": "frontal-only (PA/AP)",
+    "num_examples": 3041,
+    "bleu_1": 0.224283665754139,
+    "bleu_4": 0.038250603325884945,
+    "meteor": 0.20053919101008125,
+    "rouge_l": 0.16408262967411588,
+    "chexpert_f1_14_micro": 0.1244686823321838,
+    "chexpert_f1_5_micro": 0.21896350608231963,
+    "chexpert_f1_14_macro": 0.04429260458577706,
+    "chexpert_f1_5_macro": 0.09914404243362482,
+    "chexpert_f1_micro": 0.1244686823321838,
+    "chexpert_f1_macro": 0.04429260458577706,
+    "chexpert_per_label_f1": {
+      "Enlarged Cardiomediastinum": 0.0,
+      "Cardiomegaly": 0.0,
+      "Lung Opacity": 0.0,
+      "Lung Lesion": 0.0,
+      "Edema": 0.023121387283236997,
+      "Consolidation": 0.056790123456790124,
+      "Pneumonia": 0.02762430939226519,
+      "Atelectasis": 0.0,
+      "Pneumothorax": 0.059987236758136574,
+      "Pleural Effusion": 0.415808701428097,
+      "Pleural Other": 0.0,
+      "Fracture": 0.0,
+      "Support Devices": 0.036764705882352935,
+      "No Finding": 0.0
+    },
+    "radgraph_f1": 0.0941067057393548,
+    "radgraph_f1_entity": 0.18191243977753782,
+    "radgraph_f1_relation": 0.1652384677607375,
+    "radgraph_available": true,
+    "radgraph_error": null
+  },
+  "findings_only_test": {
+    "split": "test",
+    "subset": "findings-only frontal studies",
+    "dataset": "mimic-cxr",
+    "view_filter": "frontal-only (PA/AP), structured Findings section only",
+    "num_examples": 2210,
+    "bleu_1": 0.23099023872215996,
+    "bleu_4": 0.0429479479188206,
+    "meteor": 0.21248313160360002,
+    "rouge_l": 0.17210734193417726,
+    "chexpert_f1_14_micro": 0.11655011655011654,
+    "chexpert_f1_5_micro": 0.20709914320685435,
+    "chexpert_f1_14_macro": 0.04057070914402376,
+    "chexpert_f1_5_macro": 0.09202593660588262,
+    "chexpert_f1_micro": 0.11655011655011654,
+    "chexpert_f1_macro": 0.04057070914402376,
+    "chexpert_per_label_f1": {
+      "Enlarged Cardiomediastinum": 0.0,
+      "Cardiomegaly": 0.0,
+      "Lung Opacity": 0.0,
+      "Lung Lesion": 0.0,
+      "Edema": 0.022471910112359553,
+      "Consolidation": 0.05797101449275362,
+      "Pneumonia": 0.01673640167364017,
+      "Atelectasis": 0.0,
+      "Pneumothorax": 0.05716798592788039,
+      "Pleural Effusion": 0.3796867584243,
+      "Pleural Other": 0.0,
+      "Fracture": 0.0,
+      "Support Devices": 0.03395585738539898,
+      "No Finding": 0.0
+    },
+    "radgraph_f1": 0.10172866854646034,
+    "radgraph_f1_entity": 0.19217701907879298,
+    "radgraph_f1_relation": 0.17414731467894073,
+    "radgraph_available": true,
+    "radgraph_error": null
+  }
+}

lana_radgen/__init__.py

@@ -0,0 +1,9 @@
+from .configuration_lana import LanaConfig
+from .modeling_lana import LanaForConditionalGeneration
+from .modeling_outputs import LanaModelOutput
+
+__all__ = [
+    "LanaConfig",
+    "LanaForConditionalGeneration",
+    "LanaModelOutput",
+]

lana_radgen/attention/__init__.py

@@ -0,0 +1,3 @@
+from .layerwise_anatomical_attention import build_layerwise_attention_bias
+
+__all__ = ["build_layerwise_attention_bias"]

lana_radgen/attention/layerwise_anatomical_attention.py

@@ -0,0 +1,62 @@
+import torch
+import torch.nn.functional as F
+
+
+def _gaussian_kernel_1d(kernel_size: int, sigma: float, device: torch.device, dtype: torch.dtype) -> torch.Tensor:
+    radius = kernel_size // 2
+    x = torch.arange(-radius, radius + 1, device=device, dtype=dtype)
+    kernel = torch.exp(-(x * x) / (2.0 * sigma * sigma))
+    return kernel / kernel.sum()
+
+
+@torch.no_grad()
+def build_layerwise_attention_bias(
+    masks: torch.Tensor,
+    num_layers: int,
+    target_tokens: int,
+    base_kernel_size: int = 3,
+    kernel_growth: int = 2,
+    strength: float = 2.0,
+    eps: float = 1e-8,
+) -> torch.Tensor:
+    if masks.ndim == 3:
+        masks = masks.unsqueeze(1)
+    if masks.ndim != 4 or masks.shape[1] != 1:
+        raise ValueError(f"Expected masks shaped (B,1,H,W) or (B,H,W), got {tuple(masks.shape)}")
+
+    masks = masks.float()
+    batch_size = masks.shape[0]
+    resized = F.interpolate(masks, size=(32, 32), mode="bilinear", align_corners=False).clamp(0.0, 1.0)
+
+    max_kernel = base_kernel_size + max(num_layers, 0) * kernel_growth
+    if max_kernel % 2 == 0:
+        max_kernel += 1
+    pad = max_kernel // 2
+
+    weight_h = torch.zeros((num_layers, 1, 1, max_kernel), device=resized.device, dtype=resized.dtype)
+    weight_v = torch.zeros((num_layers, 1, max_kernel, 1), device=resized.device, dtype=resized.dtype)
+
+    for layer_idx in range(num_layers):
+        kernel_size = base_kernel_size + (num_layers - layer_idx) * kernel_growth
+        if kernel_size % 2 == 0:
+            kernel_size += 1
+        sigma = max((kernel_size - 1) / 6.0, 1e-3)
+        kernel = _gaussian_kernel_1d(kernel_size, sigma, resized.device, resized.dtype)
+        start = (max_kernel - kernel_size) // 2
+        end = start + kernel_size
+        weight_h[layer_idx, 0, 0, start:end] = kernel
+        weight_v[layer_idx, 0, start:end, 0] = kernel
+
+    repeated = resized.expand(batch_size, num_layers, 32, 32).contiguous()
+    horizontal = F.conv2d(F.pad(repeated, (pad, pad, 0, 0), mode="reflect"), weight_h, groups=num_layers)
+    vertical = F.conv2d(F.pad(horizontal, (0, 0, pad, pad), mode="reflect"), weight_v, groups=num_layers)
+
+    min_vals = vertical.amin(dim=(2, 3), keepdim=True)
+    max_vals = vertical.amax(dim=(2, 3), keepdim=True)
+    normalized = (vertical - min_vals) / (max_vals - min_vals).clamp_min(eps)
+
+    flat = normalized.view(batch_size, num_layers, -1)
+    if flat.shape[-1] != target_tokens:
+        flat = F.interpolate(flat, size=target_tokens, mode="linear", align_corners=False)
+    layerwise_bias = flat.unsqueeze(-2).expand(-1, -1, target_tokens, -1)
+    return torch.tril(layerwise_bias) * strength

lana_radgen/configuration_lana.py

@@ -0,0 +1,53 @@
+from transformers import PretrainedConfig
+
+
+class LanaConfig(PretrainedConfig):
+    model_type = "lana_radgen"
+
+    def __init__(
+        self,
+        vision_model_name: str = "facebook/dinov3-vits16-pretrain-lvd1689m",
+        text_model_name: str = "gpt2",
+        image_size: int = 512,
+        mask_size: int = 32,
+        num_attention_layers: int = 12,
+        max_position_embeddings: int = 2048,
+        visual_feature_dim: int = 384,
+        text_hidden_size: int = 768,
+        vocab_size: int = 50257,
+        layer_mask_base_kernel_size: int = 3,
+        layer_mask_kernel_growth: int = 2,
+        anatomical_attention_bias: float = 2.0,
+        use_segmentation_mask: bool = True,
+        segmentation_model_name: str = "facebook/dinov3-convnext-small-pretrain-lvd1689m",
+        segmentation_attention_implementation: str = "sdpa",
+        freeze_segmenter: bool = True,
+        lung_segmenter_checkpoint: str = "",
+        heart_segmenter_checkpoint: str = "",
+        use_cache: bool = True,
+        decoder_load_in_4bit: bool = False,
+        decoder_compute_dtype: str = "float16",
+        **kwargs,
+    ):
+        self.vision_model_name = vision_model_name
+        self.text_model_name = text_model_name
+        self.image_size = image_size
+        self.mask_size = mask_size
+        self.num_attention_layers = num_attention_layers
+        self.max_position_embeddings = max_position_embeddings
+        self.visual_feature_dim = visual_feature_dim
+        self.text_hidden_size = text_hidden_size
+        self.vocab_size = vocab_size
+        self.layer_mask_base_kernel_size = layer_mask_base_kernel_size
+        self.layer_mask_kernel_growth = layer_mask_kernel_growth
+        self.anatomical_attention_bias = anatomical_attention_bias
+        self.use_segmentation_mask = use_segmentation_mask
+        self.segmentation_model_name = segmentation_model_name
+        self.segmentation_attention_implementation = segmentation_attention_implementation
+        self.freeze_segmenter = freeze_segmenter
+        self.lung_segmenter_checkpoint = lung_segmenter_checkpoint
+        self.heart_segmenter_checkpoint = heart_segmenter_checkpoint
+        self.use_cache = use_cache
+        self.decoder_load_in_4bit = decoder_load_in_4bit
+        self.decoder_compute_dtype = decoder_compute_dtype
+        super().__init__(**kwargs)

lana_radgen/gpt2_modified.py

@@ -0,0 +1,379 @@
+from typing import Optional, Union
+
+import torch
+import torch.nn.functional as F
+from torch import nn
+from transformers import GPT2Config, GPT2LMHeadModel, GPT2Model
+from transformers.cache_utils import Cache, DynamicCache, EncoderDecoderCache
+from transformers.masking_utils import create_causal_mask
+from transformers.modeling_attn_mask_utils import _prepare_4d_attention_mask_for_sdpa
+from transformers.modeling_outputs import BaseModelOutputWithPastAndCrossAttentions, CausalLMOutputWithCrossAttentions
+from transformers.modeling_utils import ALL_ATTENTION_FUNCTIONS
+from transformers.models.gpt2.modeling_gpt2 import GPT2Attention, GPT2Block, eager_attention_forward
+
+
+class GPT2AttentionModified(GPT2Attention):
+    def forward(
+        self,
+        hidden_states: Optional[tuple[torch.FloatTensor]],
+        past_key_values: Optional[Cache] = None,
+        cache_position: Optional[torch.LongTensor] = None,
+        attention_mask: Optional[torch.FloatTensor] = None,
+        head_mask: Optional[torch.FloatTensor] = None,
+        encoder_hidden_states: Optional[torch.Tensor] = None,
+        encoder_attention_mask: Optional[torch.FloatTensor] = None,
+        output_attentions: Optional[bool] = False,
+        **kwargs,
+    ):
+        is_cross_attention = encoder_hidden_states is not None
+        if past_key_values is not None:
+            if isinstance(past_key_values, EncoderDecoderCache):
+                is_updated = past_key_values.is_updated.get(self.layer_idx)
+                curr_past_key_value = past_key_values.cross_attention_cache if is_cross_attention else past_key_values.self_attention_cache
+            else:
+                curr_past_key_value = past_key_values
+
+        if is_cross_attention:
+            if not hasattr(self, "q_attn"):
+                raise ValueError("Cross-attention requires q_attn to be defined.")
+            query_states = self.q_attn(hidden_states)
+            attention_mask = encoder_attention_mask
+            if past_key_values is not None and is_updated:
+                key_states = curr_past_key_value.layers[self.layer_idx].keys
+                value_states = curr_past_key_value.layers[self.layer_idx].values
+            else:
+                key_states, value_states = self.c_attn(encoder_hidden_states).split(self.split_size, dim=2)
+                shape_kv = (*key_states.shape[:-1], -1, self.head_dim)
+                key_states = key_states.view(shape_kv).transpose(1, 2)
+                value_states = value_states.view(shape_kv).transpose(1, 2)
+        else:
+            query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2)
+            shape_kv = (*key_states.shape[:-1], -1, self.head_dim)
+            key_states = key_states.view(shape_kv).transpose(1, 2)
+            value_states = value_states.view(shape_kv).transpose(1, 2)
+
+        shape_q = (*query_states.shape[:-1], -1, self.head_dim)
+        query_states = query_states.view(shape_q).transpose(1, 2)
+
+        if (past_key_values is not None and not is_cross_attention) or (
+            past_key_values is not None and is_cross_attention and not is_updated
+        ):
+            cache_position = cache_position if not is_cross_attention else None
+            key_states, value_states = curr_past_key_value.update(
+                key_states, value_states, self.layer_idx, {"cache_position": cache_position}
+            )
+            if is_cross_attention:
+                past_key_values.is_updated[self.layer_idx] = True
+
+        is_causal = attention_mask is None and query_states.shape[-2] > 1 and not is_cross_attention
+        attention_interface = eager_attention_forward
+        if self.config._attn_implementation != "eager":
+            attention_interface = ALL_ATTENTION_FUNCTIONS[self.config._attn_implementation]
+
+        attn_output, attn_weights = attention_interface(
+            self,
+            query_states,
+            key_states,
+            value_states,
+            attention_mask,
+            head_mask=head_mask,
+            dropout=self.attn_dropout.p if self.training else 0.0,
+            is_causal=is_causal,
+            **kwargs,
+        )
+
+        attn_output = attn_output.reshape(*attn_output.shape[:-2], -1).contiguous()
+        attn_output = self.c_proj(attn_output)
+        attn_output = self.resid_dropout(attn_output)
+        return attn_output, attn_weights
+
+
+class GPT2BlockModified(GPT2Block):
+    def __init__(self, config, layer_idx=None):
+        super().__init__(config=config, layer_idx=layer_idx)
+        self.attn = GPT2AttentionModified(config=config, layer_idx=layer_idx)
+
+
+class GPT2ModelModified(GPT2Model):
+    def __init__(self, config):
+        super().__init__(config)
+        self.config_causal = config
+        self.config_causal._attn_implementation = "eager"
+        self.h = nn.ModuleList([GPT2BlockModified(config, layer_idx=i) for i in range(config.num_hidden_layers)])
+
+    def forward(
+        self,
+        input_ids: Optional[torch.LongTensor] = None,
+        past_key_values: Optional[Union[tuple[tuple[torch.Tensor]], Cache]] = None,
+        cache_position: Optional[torch.LongTensor] = None,
+        attention_mask: Optional[torch.FloatTensor] = None,
+        token_type_ids: Optional[torch.LongTensor] = None,
+        position_ids: Optional[torch.LongTensor] = None,
+        head_mask: Optional[torch.FloatTensor] = None,
+        inputs_embeds: Optional[torch.FloatTensor] = None,
+        encoder_hidden_states: Optional[torch.Tensor] = None,
+        encoder_attention_mask: Optional[torch.FloatTensor] = None,
+        use_cache: Optional[bool] = None,
+        output_attentions: Optional[bool] = None,
+        output_hidden_states: Optional[bool] = None,
+        return_dict: Optional[bool] = None,
+        segmentation_mask: Optional[torch.FloatTensor] = None,
+        **kwargs,
+    ) -> Union[tuple, BaseModelOutputWithPastAndCrossAttentions]:
+        output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
+        output_hidden_states = output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
+        use_cache = use_cache if use_cache is not None else self.config.use_cache
+        return_dict = return_dict if return_dict is not None else self.config.use_return_dict
+
+        if input_ids is not None and inputs_embeds is not None:
+            raise ValueError("You cannot specify both input_ids and inputs_embeds at the same time")
+        if input_ids is not None:
+            self.warn_if_padding_and_no_attention_mask(input_ids, attention_mask)
+            input_shape = input_ids.size()
+            input_ids = input_ids.view(-1, input_shape[-1])
+            batch_size = input_ids.shape[0]
+        elif inputs_embeds is not None:
+            input_shape = inputs_embeds.size()[:-1]
+            batch_size = inputs_embeds.shape[0]
+        else:
+            raise ValueError("You have to specify either input_ids or inputs_embeds")
+
+        device = input_ids.device if input_ids is not None else inputs_embeds.device
+
+        if token_type_ids is not None:
+            token_type_ids = token_type_ids.view(-1, input_shape[-1])
+
+        if self.gradient_checkpointing and self.training and use_cache:
+            use_cache = False
+
+        if use_cache:
+            if past_key_values is None:
+                past_key_values = DynamicCache()
+            elif isinstance(past_key_values, tuple):
+                past_key_values = DynamicCache.from_legacy_cache(past_key_values)
+            if self.config.add_cross_attention and not isinstance(past_key_values, EncoderDecoderCache):
+                past_key_values = EncoderDecoderCache(past_key_values, DynamicCache())
+
+        if inputs_embeds is None:
+            inputs_embeds = self.wte(input_ids)
+
+        if cache_position is None:
+            past_seen_tokens = past_key_values.get_seq_length() if past_key_values is not None else 0
+            cache_position = torch.arange(past_seen_tokens, past_seen_tokens + inputs_embeds.shape[1], device=inputs_embeds.device)
+        if position_ids is None:
+            position_ids = cache_position.unsqueeze(0)
+
+        position_embeds = self.wpe(position_ids)
+        hidden_states = inputs_embeds + position_embeds.to(inputs_embeds.device)
+
+        if attention_mask is not None and attention_mask.ndim < 4:
+            attention_mask = attention_mask.view(batch_size, -1)
+
+        causal_mask = create_causal_mask(
+            config=self.config_causal,
+            input_embeds=inputs_embeds,
+            attention_mask=attention_mask,
+            cache_position=cache_position,
+            past_key_values=past_key_values,
+            position_ids=position_ids,
+        )
+
+        _use_sdpa = self._attn_implementation == "sdpa" and output_attentions is False and head_mask is None
+        if self.config.add_cross_attention and encoder_hidden_states is not None:
+            encoder_batch_size, encoder_sequence_length, _ = encoder_hidden_states.size()
+            encoder_hidden_shape = (encoder_batch_size, encoder_sequence_length)
+            if encoder_attention_mask is None:
+                encoder_attention_mask = torch.ones(encoder_hidden_shape, device=device)
+            if _use_sdpa:
+                encoder_attention_mask = _prepare_4d_attention_mask_for_sdpa(
+                    mask=encoder_attention_mask, dtype=inputs_embeds.dtype, tgt_len=input_shape[-1]
+                )
+            elif self._attn_implementation != "flash_attention_2":
+                encoder_attention_mask = self.invert_attention_mask(encoder_attention_mask)
+        else:
+            encoder_attention_mask = None
+
+        if head_mask is None:
+            head_mask = [None] * self.config.n_layer
+
+        if token_type_ids is not None:
+            hidden_states = hidden_states + self.wte(token_type_ids)
+
+        hidden_states = self.drop(hidden_states)
+        output_shape = (-1,) + input_shape[1:] + (hidden_states.size(-1),)
+        all_self_attentions = () if output_attentions else None
+        all_cross_attentions = () if output_attentions and self.config.add_cross_attention else None
+        all_hidden_states = () if output_hidden_states else None
+
+        for i, block in enumerate(self.h):
+            if output_hidden_states:
+                all_hidden_states = all_hidden_states + (hidden_states,)
+
+            block_mask = causal_mask
+            if segmentation_mask is not None and causal_mask is not None:
+                block_mask = causal_mask.clone()
+                seq_len = input_shape[-1]
+                if block_mask.shape[2] != seq_len or block_mask.shape[3] != seq_len:
+                    block_mask = block_mask[:, :, :seq_len, :seq_len]
+                layer_bias = segmentation_mask[:, i, : block_mask.shape[2], : block_mask.shape[3]].unsqueeze(1)
+                block_mask = block_mask + layer_bias.to(dtype=block_mask.dtype, device=block_mask.device)
+
+            outputs = block(
+                hidden_states=hidden_states,
+                past_key_values=past_key_values if not (self.gradient_checkpointing and self.training) else None,
+                cache_position=cache_position,
+                attention_mask=block_mask,
+                encoder_hidden_states=encoder_hidden_states,
+                encoder_attention_mask=encoder_attention_mask,
+                use_cache=use_cache,
+                output_attentions=output_attentions,
+                head_mask=head_mask[i],
+                **kwargs,
+            )
+            if isinstance(outputs, tuple):
+                hidden_states = outputs[0]
+                if output_attentions and len(outputs) > 1:
+                    all_self_attentions = all_self_attentions + (outputs[1],)
+                    if self.config.add_cross_attention and len(outputs) > 2:
+                        all_cross_attentions = all_cross_attentions + (outputs[2],)
+            else:
+                hidden_states = outputs
+
+        hidden_states = self.ln_f(hidden_states)
+        hidden_states = hidden_states.view(output_shape)
+        if output_hidden_states:
+            all_hidden_states = all_hidden_states + (hidden_states,)
+
+        past_key_values = past_key_values if use_cache else None
+        if not return_dict:
+            return tuple(v for v in [hidden_states, past_key_values, all_hidden_states, all_self_attentions, all_cross_attentions] if v is not None)
+
+        return BaseModelOutputWithPastAndCrossAttentions(
+            last_hidden_state=hidden_states,
+            past_key_values=past_key_values,
+            hidden_states=all_hidden_states,
+            attentions=all_self_attentions,
+            cross_attentions=all_cross_attentions,
+        )
+
+
+class GPT2LMHeadModelModified(GPT2LMHeadModel):
+    def __init__(self, config):
+        super().__init__(config)
+        self.transformer = GPT2ModelModified(config)
+        self.post_init()
+
+    def forward(
+        self,
+        input_ids: Optional[torch.LongTensor] = None,
+        past_key_values: Optional[tuple[tuple[torch.Tensor]]] = None,
+        cache_position: Optional[torch.LongTensor] = None,
+        attention_mask: Optional[torch.FloatTensor] = None,
+        token_type_ids: Optional[torch.LongTensor] = None,
+        position_ids: Optional[torch.LongTensor] = None,
+        head_mask: Optional[torch.FloatTensor] = None,
+        inputs_embeds: Optional[torch.FloatTensor] = None,
+        encoder_hidden_states: Optional[torch.Tensor] = None,
+        encoder_attention_mask: Optional[torch.FloatTensor] = None,
+        labels: Optional[torch.LongTensor] = None,
+        use_cache: Optional[bool] = None,
+        output_attentions: Optional[bool] = None,
+        output_hidden_states: Optional[bool] = None,
+        return_dict: Optional[bool] = None,
+        logits_to_keep: Union[int, torch.Tensor] = 0,
+        segmentation_mask: Optional[torch.FloatTensor] = None,
+        **kwargs,
+    ) -> Union[tuple, CausalLMOutputWithCrossAttentions]:
+        return_dict = return_dict if return_dict is not None else self.config.use_return_dict
+        transformer_outputs = self.transformer(
+            input_ids,
+            past_key_values=past_key_values,
+            attention_mask=attention_mask,
+            cache_position=cache_position,
+            token_type_ids=token_type_ids,
+            position_ids=position_ids,
+            head_mask=head_mask,
+            inputs_embeds=inputs_embeds,
+            encoder_hidden_states=encoder_hidden_states,
+            encoder_attention_mask=encoder_attention_mask,
+            use_cache=use_cache,
+            output_attentions=output_attentions,
+            output_hidden_states=output_hidden_states,
+            return_dict=return_dict,
+            segmentation_mask=segmentation_mask,
+            **kwargs,
+        )
+        hidden_states = transformer_outputs[0]
+        slice_indices = slice(-logits_to_keep, None) if isinstance(logits_to_keep, int) and logits_to_keep > 0 else slice(None)
+        logits = self.lm_head(hidden_states[:, slice_indices, :])
+
+        loss = None
+        if labels is not None:
+            loss = self.loss_function(logits, labels, vocab_size=self.config.vocab_size, **kwargs)
+
+        if not return_dict:
+            output = (logits,) + transformer_outputs[1:]
+            return ((loss,) + output) if loss is not None else output
+
+        return CausalLMOutputWithCrossAttentions(
+            loss=loss,
+            logits=logits,
+            past_key_values=transformer_outputs.past_key_values,
+            hidden_states=transformer_outputs.hidden_states,
+            attentions=transformer_outputs.attentions,
+            cross_attentions=transformer_outputs.cross_attentions,
+        )
+
+
+@torch.no_grad()
+def expand_gpt2_positional_embeddings(
+    model: torch.nn.Module,
+    new_max_positions: int,
+    mode: str = "linear",
+    align_corners: bool = True,
+):
+    if hasattr(model, "transformer") and hasattr(model.transformer, "wpe"):
+        model_for_wpe = model.transformer
+    elif hasattr(model, "wpe"):
+        model_for_wpe = model
+    else:
+        raise ValueError("Model does not expose GPT-2 positional embeddings.")
+
+    wpe = model_for_wpe.wpe
+    old_n, d = wpe.weight.shape
+    if new_max_positions == old_n:
+        return model
+
+    device = wpe.weight.device
+    dtype = wpe.weight.dtype
+    if new_max_positions < old_n:
+        new_weight = wpe.weight[:new_max_positions].clone()
+    else:
+        if mode != "linear":
+            raise ValueError(f"Unsupported positional expansion mode: {mode}")
+        w = wpe.weight.transpose(0, 1).unsqueeze(0)
+        w_new = F.interpolate(w, size=new_max_positions, mode="linear", align_corners=align_corners)
+        new_weight = w_new.squeeze(0).transpose(0, 1).contiguous()
+
+    new_wpe = torch.nn.Embedding(new_max_positions, d, device=device, dtype=dtype)
+    new_wpe.weight.copy_(new_weight)
+    if hasattr(model, "transformer") and hasattr(model.transformer, "wpe"):
+        model.transformer.wpe = new_wpe
+    else:
+        model.wpe = new_wpe
+    if hasattr(model.config, "n_positions"):
+        model.config.n_positions = new_max_positions
+    if hasattr(model.config, "n_ctx"):
+        model.config.n_ctx = new_max_positions
+    return model
+
+
+def create_decoder(text_model_name: str, attention_implementation: str, max_position_embeddings: int, **decoder_kwargs):
+    config = GPT2Config.from_pretrained(text_model_name)
+    config._attn_implementation = attention_implementation
+    config.n_positions = max_position_embeddings
+    config.n_ctx = max_position_embeddings
+    config.use_cache = decoder_kwargs.pop("use_cache", True)
+    decoder = GPT2LMHeadModelModified.from_pretrained(text_model_name, config=config, **decoder_kwargs)
+    decoder.config._attn_implementation = attention_implementation
+    return expand_gpt2_positional_embeddings(decoder, new_max_positions=max_position_embeddings, mode="linear")

lana_radgen/modeling_lana.py

@@ -0,0 +1,214 @@
+import logging
+from typing import Optional
+
+import torch
+import torch.nn as nn
+from transformers import AutoConfig, AutoModel, AutoTokenizer, BitsAndBytesConfig, PreTrainedModel
+
+from .attention import build_layerwise_attention_bias
+from .configuration_lana import LanaConfig
+from .gpt2_modified import create_decoder
+from .modeling_outputs import LanaModelOutput
+from .segmenters import AnatomicalSegmenter
+
+logger = logging.getLogger(__name__)
+
+
+class LanaForConditionalGeneration(PreTrainedModel):
+    config_class = LanaConfig
+    base_model_prefix = "lana"
+    supports_gradient_checkpointing = True
+
+    def __init__(self, config: LanaConfig):
+        super().__init__(config)
+        vision_config = AutoConfig.from_pretrained(config.vision_model_name, trust_remote_code=True)
+        if getattr(vision_config, "hidden_size", None) is not None:
+            config.visual_feature_dim = vision_config.hidden_size
+
+        self.vision_encoder = AutoModel.from_pretrained(config.vision_model_name, trust_remote_code=True)
+        decoder_kwargs = {
+            "ignore_mismatched_sizes": True,
+            "use_cache": config.use_cache,
+        }
+        if config.decoder_load_in_4bit:
+            compute_dtype = getattr(torch, config.decoder_compute_dtype, torch.float16)
+            decoder_kwargs["quantization_config"] = BitsAndBytesConfig(
+                load_in_4bit=True,
+                bnb_4bit_quant_type="nf4",
+                bnb_4bit_use_double_quant=True,
+                bnb_4bit_compute_dtype=compute_dtype,
+            )
+            decoder_kwargs["device_map"] = {"": 0}
+        self.text_decoder = create_decoder(
+            text_model_name=config.text_model_name,
+            attention_implementation=config.segmentation_attention_implementation,
+            max_position_embeddings=config.max_position_embeddings,
+            **decoder_kwargs,
+        )
+        self.tokenizer = AutoTokenizer.from_pretrained(config.text_model_name)
+        if self.tokenizer.pad_token_id is None:
+            self.tokenizer.pad_token = self.tokenizer.eos_token
+
+        config.vocab_size = self.text_decoder.config.vocab_size
+        config.text_hidden_size = self.text_decoder.config.hidden_size
+        config.num_attention_layers = self.text_decoder.config.n_layer
+
+        self.visual_projection = nn.Sequential(
+            nn.Linear(config.visual_feature_dim, config.text_hidden_size),
+            nn.GELU(),
+            nn.Linear(config.text_hidden_size, config.text_hidden_size),
+            nn.GELU(),
+            nn.Linear(config.text_hidden_size, config.text_hidden_size),
+            nn.GELU(),
+            nn.Linear(config.text_hidden_size, config.text_hidden_size),
+        )
+        self.segmenter = None
+        if config.use_segmentation_mask:
+            self.segmenter = AnatomicalSegmenter(
+                model_name=config.segmentation_model_name,
+                freeze=config.freeze_segmenter,
+                lung_checkpoint=config.lung_segmenter_checkpoint,
+                heart_checkpoint=config.heart_segmenter_checkpoint,
+            )
+        self.post_init()
+
+    def move_non_quantized_modules(self, device: torch.device) -> None:
+        self.vision_encoder.to(device)
+        self.visual_projection.to(device)
+        if self.segmenter is not None:
+            self.segmenter.to(device)
+        if not getattr(self.config, "decoder_load_in_4bit", False):
+            self.text_decoder.to(device)
+
+    def _encode_images(self, pixel_values: torch.Tensor) -> torch.Tensor:
+        if any(param.requires_grad for param in self.vision_encoder.parameters()):
+            outputs = self.vision_encoder(pixel_values=pixel_values)
+        else:
+            with torch.no_grad():
+                outputs = self.vision_encoder(pixel_values=pixel_values)
+        hidden = outputs.last_hidden_state
+        if hidden.shape[1] > 1:
+            hidden = hidden[:, 1:, :]
+        return self.visual_projection(hidden)
+
+    def _build_layerwise_bias(self, anatomical_masks: Optional[torch.Tensor], total_sequence_length: int) -> Optional[torch.Tensor]:
+        if anatomical_masks is None:
+            return None
+        return build_layerwise_attention_bias(
+            masks=anatomical_masks,
+            num_layers=self.config.num_attention_layers,
+            target_tokens=total_sequence_length,
+            base_kernel_size=self.config.layer_mask_base_kernel_size,
+            kernel_growth=self.config.layer_mask_kernel_growth,
+            strength=self.config.anatomical_attention_bias,
+        )
+
+    def _resolve_attention_bias(self, pixel_values: torch.Tensor, anatomical_masks: Optional[torch.Tensor], total_sequence_length: int):
+        if anatomical_masks is not None:
+            return self._build_layerwise_bias(anatomical_masks, total_sequence_length=total_sequence_length)
+        if self.segmenter is None:
+            return None
+        layerwise_bias = self.segmenter(
+            pixel_values,
+            num_layers=self.config.num_attention_layers,
+            target_tokens=total_sequence_length,
+            strength=self.config.anatomical_attention_bias,
+        )
+        if layerwise_bias is None:
+            logger.warning("Segmentation attention is enabled but no segmenter checkpoints were loaded; continuing without anatomical attention.")
+        return layerwise_bias
+
+    def forward(
+        self,
+        pixel_values: torch.Tensor,
+        input_ids: Optional[torch.LongTensor] = None,
+        attention_mask: Optional[torch.Tensor] = None,
+        anatomical_masks: Optional[torch.Tensor] = None,
+        labels: Optional[torch.LongTensor] = None,
+        output_attentions: Optional[bool] = None,
+        output_hidden_states: Optional[bool] = None,
+        return_dict: Optional[bool] = True,
+        **kwargs,
+    ) -> LanaModelOutput:
+        vision_features = self._encode_images(pixel_values)
+        batch_size, prefix_length, _ = vision_features.shape
+
+        if input_ids is None:
+            bos = self.tokenizer.bos_token_id or self.tokenizer.eos_token_id
+            input_ids = torch.full((batch_size, 1), bos, device=vision_features.device, dtype=torch.long)
+            attention_mask = torch.ones_like(input_ids)
+        elif attention_mask is None:
+            attention_mask = torch.ones_like(input_ids)
+
+        text_embeds = self.text_decoder.transformer.wte(input_ids)
+        inputs_embeds = torch.cat([vision_features, text_embeds], dim=1)
+        merged_attention_mask = torch.cat(
+            [
+                torch.ones((batch_size, prefix_length), device=attention_mask.device, dtype=attention_mask.dtype),
+                attention_mask,
+            ],
+            dim=1,
+        )
+
+        merged_labels = None
+        if labels is not None:
+            ignore_prefix = torch.full((batch_size, prefix_length), -100, device=labels.device, dtype=labels.dtype)
+            merged_labels = torch.cat([ignore_prefix, labels], dim=1)
+
+        layerwise_bias = self._resolve_attention_bias(
+            pixel_values=pixel_values,
+            anatomical_masks=anatomical_masks,
+            total_sequence_length=inputs_embeds.shape[1],
+        )
+        decoder_outputs = self.text_decoder(
+            inputs_embeds=inputs_embeds,
+            attention_mask=merged_attention_mask,
+            labels=merged_labels,
+            segmentation_mask=layerwise_bias,
+            use_cache=False,
+            output_attentions=output_attentions,
+            output_hidden_states=output_hidden_states,
+            return_dict=True,
+            **kwargs,
+        )
+
+        return LanaModelOutput(
+            loss=decoder_outputs.loss,
+            logits=decoder_outputs.logits,
+            attentions=decoder_outputs.attentions,
+            layerwise_attentions=layerwise_bias,
+            hidden_states=decoder_outputs.hidden_states,
+            vision_features=vision_features,
+        )
+
+    @torch.inference_mode()
+    def generate(
+        self,
+        pixel_values: torch.Tensor,
+        anatomical_masks: Optional[torch.Tensor] = None,
+        max_new_tokens: int = 128,
+        **kwargs,
+    ):
+        vision_features = self._encode_images(pixel_values)
+        batch_size = pixel_values.shape[0]
+        bos = self.tokenizer.bos_token_id or self.tokenizer.eos_token_id
+        start_tokens = torch.full((batch_size, 1), bos, device=pixel_values.device, dtype=torch.long)
+        text_embeds = self.text_decoder.transformer.wte(start_tokens)
+        inputs_embeds = torch.cat([vision_features, text_embeds], dim=1)
+        attention_mask = torch.ones(inputs_embeds.shape[:2], device=pixel_values.device, dtype=torch.long)
+
+        layerwise_bias = self._resolve_attention_bias(
+            pixel_values=pixel_values,
+            anatomical_masks=anatomical_masks,
+            total_sequence_length=inputs_embeds.shape[1] + max_new_tokens,
+        )
+        return self.text_decoder.generate(
+            inputs_embeds=inputs_embeds,
+            attention_mask=attention_mask,
+            max_new_tokens=max_new_tokens,
+            pad_token_id=self.tokenizer.pad_token_id,
+            eos_token_id=self.tokenizer.eos_token_id,
+            segmentation_mask=layerwise_bias,
+            use_cache=True,
+            **kwargs,
+        )

lana_radgen/modeling_outputs.py

@@ -0,0 +1,15 @@
+from dataclasses import dataclass
+from typing import Optional, Tuple
+
+import torch
+from transformers.utils import ModelOutput
+
+
+@dataclass
+class LanaModelOutput(ModelOutput):
+    loss: Optional[torch.FloatTensor] = None
+    logits: Optional[torch.FloatTensor] = None
+    attentions: Optional[Tuple[torch.FloatTensor, ...]] = None
+    layerwise_attentions: Optional[torch.FloatTensor] = None
+    hidden_states: Optional[Tuple[torch.FloatTensor, ...]] = None
+    vision_features: Optional[torch.FloatTensor] = None

lana_radgen/segmenters.py

@@ -0,0 +1,123 @@
+import logging
+from pathlib import Path
+
+import torch
+import torch.nn as nn
+from transformers import AutoModel
+
+from .attention.layerwise_anatomical_attention import build_layerwise_attention_bias
+
+LOGGER = logging.getLogger(__name__)
+
+
+def _freeze_module(module: nn.Module) -> None:
+    for param in module.parameters():
+        param.requires_grad = False
+
+
+class _DinoUNetLung(nn.Module):
+    def __init__(self, model_name: str, freeze: bool = True):
+        super().__init__()
+        self.encoder = AutoModel.from_pretrained(model_name, trust_remote_code=True)
+        self.channel_adapter = nn.Conv2d(768, 512, kernel_size=1)
+        self.decoder = nn.Sequential(
+            nn.Conv2d(512, 256, 3, padding=1),
+            nn.ReLU(inplace=True),
+            nn.ConvTranspose2d(256, 128, 2, stride=2),
+            nn.ReLU(inplace=True),
+            nn.ConvTranspose2d(128, 64, 2, stride=2),
+            nn.ReLU(inplace=True),
+            nn.Conv2d(64, 1, 1),
+        )
+        if freeze:
+            _freeze_module(self)
+
+    @torch.no_grad()
+    def forward(self, x: torch.Tensor) -> torch.Tensor:
+        enc_feats = self.encoder(x, output_hidden_states=True, return_dict=True)
+        feats = next(h for h in reversed(enc_feats.hidden_states) if isinstance(h, torch.Tensor) and h.ndim == 4)
+        feats = self.channel_adapter(feats)
+        pred = self.decoder(feats)
+        return (torch.sigmoid(pred) > 0.5).float()
+
+
+class _DinoUNetHeart(nn.Module):
+    def __init__(self, model_name: str, freeze: bool = True):
+        super().__init__()
+        self.encoder = AutoModel.from_pretrained(model_name, trust_remote_code=True)
+        self.adapter = nn.Conv2d(768, 512, 1)
+        self.decoder = nn.Sequential(
+            nn.Conv2d(512, 256, 3, padding=1),
+            nn.ReLU(True),
+            nn.ConvTranspose2d(256, 128, 2, 2),
+            nn.ReLU(True),
+            nn.ConvTranspose2d(128, 64, 2, 2),
+            nn.ReLU(True),
+            nn.Conv2d(64, 3, 1),
+        )
+        if freeze:
+            _freeze_module(self)
+
+    @torch.no_grad()
+    def forward(self, x: torch.Tensor) -> torch.Tensor:
+        enc = self.encoder(x, output_hidden_states=True, return_dict=True)
+        feat = next(h for h in reversed(enc.hidden_states) if isinstance(h, torch.Tensor) and h.ndim == 4)
+        feat = self.adapter(feat)
+        logits = self.decoder(feat)
+        pred = torch.argmax(logits, dim=1)
+        return (pred == 2).unsqueeze(1).float()
+
+
+class AnatomicalSegmenter(nn.Module):
+    def __init__(
+        self,
+        model_name: str,
+        freeze: bool = True,
+        lung_checkpoint: str = "",
+        heart_checkpoint: str = "",
+    ):
+        super().__init__()
+        self.lung_model = _DinoUNetLung(model_name=model_name, freeze=freeze)
+        self.heart_model = _DinoUNetHeart(model_name=model_name, freeze=freeze)
+        self.loaded_lung_checkpoint = self._load_submodule(self.lung_model, lung_checkpoint, "lung")
+        self.loaded_heart_checkpoint = self._load_submodule(self.heart_model, heart_checkpoint, "heart")
+
+    @staticmethod
+    def _load_submodule(module: nn.Module, checkpoint_path: str, label: str) -> bool:
+        if not checkpoint_path:
+            return False
+        path = Path(checkpoint_path)
+        if not path.exists():
+            LOGGER.warning("Requested %s segmenter checkpoint does not exist: %s", label, path)
+            return False
+        state = torch.load(path, map_location="cpu", weights_only=False)
+        if isinstance(state, dict) and "state_dict" in state:
+            state = state["state_dict"]
+        module.load_state_dict(state, strict=False)
+        LOGGER.info("Loaded %s segmenter checkpoint from %s", label, path)
+        return True
+
+    @property
+    def has_any_checkpoint(self) -> bool:
+        return self.loaded_lung_checkpoint or self.loaded_heart_checkpoint
+
+    @torch.no_grad()
+    def forward(self, pixel_values: torch.Tensor, num_layers: int, target_tokens: int, strength: float) -> torch.Tensor | None:
+        if not self.has_any_checkpoint:
+            return None
+
+        masks = []
+        if self.loaded_heart_checkpoint:
+            masks.append(self.heart_model(pixel_values))
+        if self.loaded_lung_checkpoint:
+            masks.append(self.lung_model(pixel_values))
+        if not masks:
+            return None
+
+        combined_mask = torch.clamp(sum(masks), 0.0, 1.0)
+        return build_layerwise_attention_bias(
+            masks=combined_mask,
+            num_layers=num_layers,
+            target_tokens=target_tokens,
+            strength=strength,
+        )

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a5b80f4c62ee4863205f671c7d8670ebdd8e119bb449194633b15fa80b6479a7
+size 1159628024

modeling_lana.py

@@ -0,0 +1,3 @@
+from lana_radgen.modeling_lana import LanaForConditionalGeneration
+
+__all__ = ["LanaForConditionalGeneration"]

pipeline_autotune.json

@@ -0,0 +1,111 @@
+{
+  "train": {
+    "method": "lora_adamw",
+    "batch_size": 1,
+    "global_batch_size": 8,
+    "candidate_dir": "C:\\Users\\emman\\Desktop\\PROYECTOS_VS_CODE\\PRUEBAS_DE_PYTHON\\Chest-X-ray-Diagnosis-Automated-Reporting-using-CNNs-and-LLMs---UDEM-PEF-Thesis-Fall-2025\\artifacts\\full_3_epoch_mask_run\\_autotune\\train\\candidate_0_lora_adamw_b1_g8",
+    "status": "ok",
+    "elapsed_seconds": 111.93326840000373,
+    "images_per_second": 9.704605089103115,
+    "steps": 16,
+    "train_loss_last": 7.840930461883545
+  },
+  "eval": {
+    "batch_size": 8,
+    "status": "ok",
+    "elapsed_seconds": 37.971331600005215,
+    "examples_per_second": 1.685482107243013
+  },
+  "benchmarks": {
+    "train": [
+      {
+        "method": "lora_adamw",
+        "batch_size": 1,
+        "global_batch_size": 8,
+        "candidate_dir": "C:\\Users\\emman\\Desktop\\PROYECTOS_VS_CODE\\PRUEBAS_DE_PYTHON\\Chest-X-ray-Diagnosis-Automated-Reporting-using-CNNs-and-LLMs---UDEM-PEF-Thesis-Fall-2025\\artifacts\\full_3_epoch_mask_run\\_autotune\\train\\candidate_0_lora_adamw_b1_g8",
+        "status": "ok",
+        "elapsed_seconds": 111.93326840000373,
+        "images_per_second": 9.704605089103115,
+        "steps": 16,
+        "train_loss_last": 7.840930461883545
+      },
+      {
+        "method": "lora_adamw",
+        "batch_size": 2,
+        "global_batch_size": 8,
+        "candidate_dir": "C:\\Users\\emman\\Desktop\\PROYECTOS_VS_CODE\\PRUEBAS_DE_PYTHON\\Chest-X-ray-Diagnosis-Automated-Reporting-using-CNNs-and-LLMs---UDEM-PEF-Thesis-Fall-2025\\artifacts\\full_3_epoch_mask_run\\_autotune\\train\\candidate_1_lora_adamw_b2_g8",
+        "status": "ok",
+        "elapsed_seconds": 114.21902520000003,
+        "images_per_second": 8.770855046196548,
+        "steps": 16,
+        "train_loss_last": 7.009981155395508
+      },
+      {
+        "method": "lora_adamw",
+        "batch_size": 2,
+        "global_batch_size": 4,
+        "candidate_dir": "C:\\Users\\emman\\Desktop\\PROYECTOS_VS_CODE\\PRUEBAS_DE_PYTHON\\Chest-X-ray-Diagnosis-Automated-Reporting-using-CNNs-and-LLMs---UDEM-PEF-Thesis-Fall-2025\\artifacts\\full_3_epoch_mask_run\\_autotune\\train\\candidate_2_lora_adamw_b2_g4",
+        "status": "ok",
+        "elapsed_seconds": 116.37137660000008,
+        "images_per_second": 8.927318141708337,
+        "steps": 16,
+        "train_loss_last": 7.958044052124023
+      },
+      {
+        "method": "full_adam8bit",
+        "batch_size": 1,
+        "global_batch_size": 8,
+        "candidate_dir": "C:\\Users\\emman\\Desktop\\PROYECTOS_VS_CODE\\PRUEBAS_DE_PYTHON\\Chest-X-ray-Diagnosis-Automated-Reporting-using-CNNs-and-LLMs---UDEM-PEF-Thesis-Fall-2025\\artifacts\\full_3_epoch_mask_run\\_autotune\\train\\candidate_3_full_adam8bit_b1_g8",
+        "status": "ok",
+        "elapsed_seconds": 121.47497799999837,
+        "images_per_second": 8.749786245453723,
+        "steps": 16,
+        "train_loss_last": 6.867033958435059
+      },
+      {
+        "method": "full_adamw",
+        "batch_size": 1,
+        "global_batch_size": 8,
+        "candidate_dir": "C:\\Users\\emman\\Desktop\\PROYECTOS_VS_CODE\\PRUEBAS_DE_PYTHON\\Chest-X-ray-Diagnosis-Automated-Reporting-using-CNNs-and-LLMs---UDEM-PEF-Thesis-Fall-2025\\artifacts\\full_3_epoch_mask_run\\_autotune\\train\\candidate_4_full_adamw_b1_g8",
+        "status": "ok",
+        "elapsed_seconds": 120.72124660000554,
+        "images_per_second": 8.777123798241934,
+        "steps": 16,
+        "train_loss_last": 6.959325313568115
+      },
+      {
+        "method": "qlora_paged_adamw8bit",
+        "batch_size": 1,
+        "global_batch_size": 8,
+        "status": "failed",
+        "error": "Command '['C:\\\\Users\\\\emman\\\\Desktop\\\\PROYECTOS_VS_CODE\\\\PRUEBAS_DE_PYTHON\\\\Chest-X-ray-Diagnosis-Automated-Reporting-using-CNNs-and-LLMs---UDEM-PEF-Thesis-Fall-2025\\\\venv310\\\\Scripts\\\\python.exe', 'scripts/train.py', '--run-name', 'autotune_train_5', '--dataset', 'combined', '--epochs', '1', '--batch-size', '1', '--global-batch-size', '8', '--eval-batch-size', '1', '--image-size', '512', '--device', 'cuda', '--output-dir', 'C:\\\\Users\\\\emman\\\\Desktop\\\\PROYECTOS_VS_CODE\\\\PRUEBAS_DE_PYTHON\\\\Chest-X-ray-Diagnosis-Automated-Reporting-using-CNNs-and-LLMs---UDEM-PEF-Thesis-Fall-2025\\\\artifacts\\\\full_3_epoch_mask_run\\\\_autotune\\\\train\\\\candidate_5_qlora_paged_adamw8bit_b1_g8', '--method', 'qlora_paged_adamw8bit', '--max-train-steps', '16', '--save-every-n-steps', '1000', '--log-level', 'INFO', '--disable-wandb']' returned non-zero exit status 1."
+      }
+    ],
+    "eval": [
+      {
+        "batch_size": 1,
+        "status": "ok",
+        "elapsed_seconds": 105.64014480000333,
+        "examples_per_second": 0.6058302941667114
+      },
+      {
+        "batch_size": 2,
+        "status": "ok",
+        "elapsed_seconds": 70.24700000000303,
+        "examples_per_second": 0.9110709354135728
+      },
+      {
+        "batch_size": 4,
+        "status": "ok",
+        "elapsed_seconds": 46.93239729999914,
+        "examples_per_second": 1.3636635603952234
+      },
+      {
+        "batch_size": 8,
+        "status": "ok",
+        "elapsed_seconds": 37.971331600005215,
+        "examples_per_second": 1.685482107243013
+      }
+    ]
+  }
+}

run_summary.json

@@ -0,0 +1,162 @@
+{
+  "method": "lora_adamw",
+  "run_name": "mimic only",
+  "steps": 6692,
+  "epochs_completed": 0,
+  "epoch_index": 0,
+  "target_epochs": 3,
+  "progress_epochs": 0.38088056399348363,
+  "training_completion_percent": 12.696018799782788,
+  "elapsed_seconds": 3600.0493718999996,
+  "images_seen": 53540,
+  "train_loss_last": 0.6295745968818665,
+  "train_loss_mean": 2.933644461443075,
+  "val_loss": 2.31326277256012,
+  "images_per_second": 14.872018261167115,
+  "trainable_params": 2878464,
+  "vision_model_name": "facebook/dinov3-vits16-pretrain-lvd1689m",
+  "text_model_name": "gpt2",
+  "segmentation_model_name": "facebook/dinov3-convnext-small-pretrain-lvd1689m",
+  "lung_segmenter_checkpoint": "models/lung_segmenter_dinounet_finetuned.pth",
+  "heart_segmenter_checkpoint": "models/heart_segmenter_dinounet_best.pth",
+  "image_size": 512,
+  "batch_size": 1,
+  "global_batch_size": 8,
+  "gradient_accumulation_steps": 8,
+  "steps_per_epoch": 17572,
+  "planned_total_steps": 52716,
+  "scheduler": "cosine",
+  "warmup_steps": 2636,
+  "warmup_ratio": 0.05,
+  "weight_decay": 0.01,
+  "precision": "bf16",
+  "torch_compile": false,
+  "torch_compile_mode": "default",
+  "hardware": "NVIDIA GeForce RTX 5070",
+  "seed": 42,
+  "resume_supported": true,
+  "checkpoint_every_n_steps": 1000,
+  "cumulative_loss_sum": 157067.32446566224,
+  "cumulative_loss_count": 53540,
+  "completed": false,
+  "target_duration_seconds": 3600,
+  "target_duration_mode": "per_invocation",
+  "train_datasets": "MIMIC-CXR (findings-only)",
+  "validation_datasets": "MIMIC-CXR (findings-only)",
+  "latest_evaluation": {
+    "split": "test",
+    "subset": "all frontal studies",
+    "dataset": "mimic-cxr",
+    "view_filter": "frontal-only (PA/AP)",
+    "num_examples": 3041,
+    "bleu_1": 0.224283665754139,
+    "bleu_4": 0.038250603325884945,
+    "meteor": 0.20053919101008125,
+    "rouge_l": 0.16408262967411588,
+    "chexpert_f1_14_micro": 0.1244686823321838,
+    "chexpert_f1_5_micro": 0.21896350608231963,
+    "chexpert_f1_14_macro": 0.04429260458577706,
+    "chexpert_f1_5_macro": 0.09914404243362482,
+    "chexpert_f1_micro": 0.1244686823321838,
+    "chexpert_f1_macro": 0.04429260458577706,
+    "chexpert_per_label_f1": {
+      "Enlarged Cardiomediastinum": 0.0,
+      "Cardiomegaly": 0.0,
+      "Lung Opacity": 0.0,
+      "Lung Lesion": 0.0,
+      "Edema": 0.023121387283236997,
+      "Consolidation": 0.056790123456790124,
+      "Pneumonia": 0.02762430939226519,
+      "Atelectasis": 0.0,
+      "Pneumothorax": 0.059987236758136574,
+      "Pleural Effusion": 0.415808701428097,
+      "Pleural Other": 0.0,
+      "Fracture": 0.0,
+      "Support Devices": 0.036764705882352935,
+      "No Finding": 0.0
+    },
+    "radgraph_f1": 0.0941067057393548,
+    "radgraph_f1_entity": 0.18191243977753782,
+    "radgraph_f1_relation": 0.1652384677607375,
+    "radgraph_available": true,
+    "radgraph_error": null
+  },
+  "latest_evaluations": {
+    "all_test": {
+      "split": "test",
+      "subset": "all frontal studies",
+      "dataset": "mimic-cxr",
+      "view_filter": "frontal-only (PA/AP)",
+      "num_examples": 3041,
+      "bleu_1": 0.224283665754139,
+      "bleu_4": 0.038250603325884945,
+      "meteor": 0.20053919101008125,
+      "rouge_l": 0.16408262967411588,
+      "chexpert_f1_14_micro": 0.1244686823321838,
+      "chexpert_f1_5_micro": 0.21896350608231963,
+      "chexpert_f1_14_macro": 0.04429260458577706,
+      "chexpert_f1_5_macro": 0.09914404243362482,
+      "chexpert_f1_micro": 0.1244686823321838,
+      "chexpert_f1_macro": 0.04429260458577706,
+      "chexpert_per_label_f1": {
+        "Enlarged Cardiomediastinum": 0.0,
+        "Cardiomegaly": 0.0,
+        "Lung Opacity": 0.0,
+        "Lung Lesion": 0.0,
+        "Edema": 0.023121387283236997,
+        "Consolidation": 0.056790123456790124,
+        "Pneumonia": 0.02762430939226519,
+        "Atelectasis": 0.0,
+        "Pneumothorax": 0.059987236758136574,
+        "Pleural Effusion": 0.415808701428097,
+        "Pleural Other": 0.0,
+        "Fracture": 0.0,
+        "Support Devices": 0.036764705882352935,
+        "No Finding": 0.0
+      },
+      "radgraph_f1": 0.0941067057393548,
+      "radgraph_f1_entity": 0.18191243977753782,
+      "radgraph_f1_relation": 0.1652384677607375,
+      "radgraph_available": true,
+      "radgraph_error": null
+    },
+    "findings_only_test": {
+      "split": "test",
+      "subset": "findings-only frontal studies",
+      "dataset": "mimic-cxr",
+      "view_filter": "frontal-only (PA/AP), structured Findings section only",
+      "num_examples": 2210,
+      "bleu_1": 0.23099023872215996,
+      "bleu_4": 0.0429479479188206,
+      "meteor": 0.21248313160360002,
+      "rouge_l": 0.17210734193417726,
+      "chexpert_f1_14_micro": 0.11655011655011654,
+      "chexpert_f1_5_micro": 0.20709914320685435,
+      "chexpert_f1_14_macro": 0.04057070914402376,
+      "chexpert_f1_5_macro": 0.09202593660588262,
+      "chexpert_f1_micro": 0.11655011655011654,
+      "chexpert_f1_macro": 0.04057070914402376,
+      "chexpert_per_label_f1": {
+        "Enlarged Cardiomediastinum": 0.0,
+        "Cardiomegaly": 0.0,
+        "Lung Opacity": 0.0,
+        "Lung Lesion": 0.0,
+        "Edema": 0.022471910112359553,
+        "Consolidation": 0.05797101449275362,
+        "Pneumonia": 0.01673640167364017,
+        "Atelectasis": 0.0,
+        "Pneumothorax": 0.05716798592788039,
+        "Pleural Effusion": 0.3796867584243,
+        "Pleural Other": 0.0,
+        "Fracture": 0.0,
+        "Support Devices": 0.03395585738539898,
+        "No Finding": 0.0
+      },
+      "radgraph_f1": 0.10172866854646034,
+      "radgraph_f1_entity": 0.19217701907879298,
+      "radgraph_f1_relation": 0.17414731467894073,
+      "radgraph_available": true,
+      "radgraph_error": null
+    }
+  }
+}

segmenters/heart_segmenter_dinounet_best.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e7f17093041df317bdd22440789ce3aed407a8bda9d7527751d23e8c106fb59b
+size 204910713

segmenters/lung_segmenter_dinounet_finetuned.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:086027098b3e2243dd56e5ef3b7a248a0532c3ae401da27091d94617d41b7403
+size 204911991

tokenizer_config.json

@@ -0,0 +1,12 @@
+{
+  "add_prefix_space": false,
+  "backend": "tokenizers",
+  "bos_token": "<|endoftext|>",
+  "eos_token": "<|endoftext|>",
+  "errors": "replace",
+  "is_local": false,
+  "model_max_length": 1024,
+  "pad_token": "<|endoftext|>",
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "<|endoftext|>"
+}