Upload MIMIC test evaluation results

README.md

@@ -98,34 +98,43 @@ print(report)
 
 Frontal-only evaluation using `PA/AP` studies only.
 
-### Current Checkpoint Results
-
-| Metric | Value |
-| --- | --- |
-| Number of studies | TBD |
-| RadGraph F1 | TBD |
-| RadGraph entity F1 | TBD |
-| RadGraph relation F1 | TBD |
-| CheXpert F1 14-micro | TBD |
-| CheXpert F1 5-micro | TBD |
-| CheXpert F1 14-macro | TBD |
-| CheXpert F1 5-macro | TBD |
-
-### Final Completed Training Results
-
-The final table will be populated when the planned training run is completed. Until then, final-report metrics remain `TBD`.
-
-| Metric | Value |
-| --- | --- |
-| Number of studies | TBD |
-| RadGraph F1 | TBD |
-| RadGraph entity F1 | TBD |
-| RadGraph relation F1 | TBD |
-| CheXpert F1 14-micro | TBD |
-| CheXpert F1 5-micro | TBD |
-| CheXpert F1 14-macro | TBD |
-| CheXpert F1 5-macro | TBD |
-
+These comparison tables are refreshed across the full LAnA collection whenever any collection model is evaluated.
+
+### Cross-Model Comparison: All Frontal Test Studies
+
+| Metric | LAnA-MIMIC-CHEXPERT | LAnA-MIMIC | LAnA | LAnA-v2 | LAnA-v3 | LAnA-v4 (Model still training) |
+| --- | --- | --- | --- | --- | --- | --- |
+| Run status | `Completed` | `Completed` | `Completed` | `Completed` | `Completed` | `Model still training` |
+| Number of studies | `3041` | `3041` | `3041` | `3041` | `3041` | `3041` |
+| ROUGE-L | `0.1513` | `0.1653` | `0.1686` | `0.1670` | `0.1745` | `0.1693` |
+| BLEU-1 | `0.1707` | `0.1916` | `0.2091` | `0.2174` | `0.2346` | `0.2266` |
+| BLEU-4 | `0.0357` | `0.0386` | `0.0417` | `0.0417` | `0.0484` | `0.0441` |
+| METEOR | `0.2079` | `0.2202` | `0.2298` | `0.2063` | `0.2129` | `0.2017` |
+| RadGraph F1 | `0.0918` | `0.0921` | `0.1024` | `0.1057` | `0.0939` | `0.0814` |
+| RadGraph entity F1 | `0.1399` | `0.1459` | `0.1587` | `0.1569` | `0.1441` | `0.1453` |
+| RadGraph relation F1 | `0.1246` | `0.1322` | `0.1443` | `0.1474` | `0.1280` | `0.1311` |
+| CheXpert F1 14-micro | `0.1829` | `0.1565` | `0.2116` | `0.1401` | `0.3116` | `0.2211` |
+| CheXpert F1 5-micro | `0.2183` | `0.1530` | `0.2512` | `0.2506` | `0.2486` | `0.0592` |
+| CheXpert F1 14-macro | `0.1095` | `0.0713` | `0.1095` | `0.0401` | `0.1363` | `0.0760` |
+| CheXpert F1 5-macro | `0.1634` | `0.1007` | `0.1644` | `0.1004` | `0.1686` | `0.0354` |
+
+### Cross-Model Comparison: Findings-Only Frontal Test Studies
+
+| Metric | LAnA-MIMIC-CHEXPERT | LAnA-MIMIC | LAnA | LAnA-v2 | LAnA-v3 | LAnA-v4 (Model still training) |
+| --- | --- | --- | --- | --- | --- | --- |
+| Run status | `Completed` | `Completed` | `Completed` | `Completed` | `Completed` | `Model still training` |
+| Number of studies | `2210` | `2210` | `2210` | `2210` | `2210` | `2210` |
+| ROUGE-L | `0.1576` | `0.1720` | `0.1771` | `0.1771` | `0.1848` | `0.1776` |
+| BLEU-1 | `0.1754` | `0.2003` | `0.2177` | `0.2263` | `0.2480` | `0.2365` |
+| BLEU-4 | `0.0405` | `0.0449` | `0.0484` | `0.0487` | `0.0573` | `0.0509` |
+| METEOR | `0.2207` | `0.2347` | `0.2466` | `0.2240` | `0.2310` | `0.2158` |
+| RadGraph F1 | `0.1010` | `0.1000` | `0.1119` | `0.1181` | `0.1046` | `0.0926` |
+| RadGraph entity F1 | `0.1517` | `0.1577` | `0.1713` | `0.1739` | `0.1584` | `0.1583` |
+| RadGraph relation F1 | `0.1347` | `0.1413` | `0.1549` | `0.1628` | `0.1405` | `0.1428` |
+| CheXpert F1 14-micro | `0.1651` | `0.1442` | `0.1907` | `0.1365` | `0.2921` | `0.2221` |
+| CheXpert F1 5-micro | `0.2152` | `0.1716` | `0.2415` | `0.2455` | `0.2394` | `0.0649` |
+| CheXpert F1 14-macro | `0.1047` | `0.0700` | `0.1039` | `0.0381` | `0.1326` | `0.0758` |
+| CheXpert F1 5-macro | `0.1611` | `0.1112` | `0.1578` | `0.0952` | `0.1636` | `0.0382` |
 
 ## Data
 
@@ -138,6 +147,15 @@ The final table will be populated when the planned training run is completed. Un
 
 - Medical report metrics implemented in the repository include RadGraph F1 and CheXpert F1 (`14-micro`, `5-micro`, `14-macro`, `5-macro`).
 
+## Experiment Model Descriptions
+
+- `LAnA-MIMIC-CHEXPERT`: This variant was trained on a combined dataset of `CheXpert` and `MIMIC-CXR` using LoRA fine-tuning with the `AdamW` optimizer.
+- `LAnA-MIMIC`: This model was trained on the `MIMIC-CXR (findings-only)` dataset using LoRA fine-tuning with the `AdamW` optimizer.
+- `LAnA`: This model was trained on the `MIMIC-CXR (findings-only)` dataset using full-model optimization with `AdamW` instead of LoRA.
+- `LAnA-v2`: This version keeps the same training setup as `LAnA`, but increases the effective global batch size from `16` to `128`.
+- `LAnA-v3`: This version keeps the same training setup as `LAnA`, including the effective global batch size of `16`, but changes how EOS is handled so training and generation follow the same behavior. The model no longer uses the EOS token during training, and generation remained greedy without stopping when an EOS token was produced. In the previous setup, decoding was also greedy, stopped at EOS, and used a maximum of `128` new tokens.
+- `LAnA-v4`: This version keeps the same decoding behavior as `LAnA-v3`, but increases the effective global batch size from `16` to `128`.
+
 ## Training Snapshot
 
 - Run: `LAnA-v4`
@@ -173,4 +191,4 @@ The final table will be populated when the planned training run is completed. Un
 
 - Set `HF_TOKEN` with permission to access the DINOv3 repositories required by this model before downloading or running inference.
 - `segmenters/` contains the lung and heart segmentation checkpoints used to build anatomical attention masks.
-- `evaluations/mimic_test_metrics.json` contains the latest saved MIMIC test metrics.
+- `evaluations/mimic_test_metrics.json` contains the latest saved MIMIC test metrics.

evaluations/mimic_test_findings_only_metrics.json

@@ -4,19 +4,19 @@
   "dataset": "mimic-cxr",
   "view_filter": "frontal-only (PA/AP), structured Findings section only",
   "num_examples": 2210,
-  "bleu_1": 0.23179623497255453,
-  "bleu_4": 0.04902797277049711,
-  "meteor": 0.2125228998859286,
-  "rouge_l": 0.17742184702691816,
-  "chexpert_f1_14_micro": 0.21896792189679218,
-  "chexpert_f1_5_micro": 0.05121638924455826,
-  "chexpert_f1_14_macro": 0.07088977497872763,
-  "chexpert_f1_5_macro": 0.03252032520325203,
-  "chexpert_f1_micro": 0.21896792189679218,
-  "chexpert_f1_macro": 0.07088977497872763,
+  "bleu_1": 0.2365366145096764,
+  "bleu_4": 0.05092983875760064,
+  "meteor": 0.21583177087647024,
+  "rouge_l": 0.17757622652257565,
+  "chexpert_f1_14_micro": 0.22206742825299527,
+  "chexpert_f1_5_micro": 0.06491372226787182,
+  "chexpert_f1_14_macro": 0.07578923935276514,
+  "chexpert_f1_5_macro": 0.03816425120772947,
+  "chexpert_f1_micro": 0.22206742825299527,
+  "chexpert_f1_macro": 0.07578923935276514,
   "chexpert_per_label_f1": {
-    "Enlarged Cardiomediastinum": 0.0,
-    "Cardiomegaly": 0.16260162601626016,
+    "Enlarged Cardiomediastinum": 0.028169014084507043,
+    "Cardiomegaly": 0.19082125603864733,
     "Lung Opacity": 0.0,
     "Lung Lesion": 0.0,
     "Edema": 0.0,
@@ -27,12 +27,12 @@
     "Pleural Effusion": 0.0,
     "Pleural Other": 0.0,
     "Fracture": 0.0,
-    "Support Devices": 0.48493543758967,
-    "No Finding": 0.3449197860962567
+    "Support Devices": 0.4929681717246484,
+    "No Finding": 0.3490909090909091
   },
-  "radgraph_f1": 0.088204837339789,
-  "radgraph_f1_entity": 0.15309947777272206,
-  "radgraph_f1_relation": 0.1377130122662026,
+  "radgraph_f1": 0.09257418507561303,
+  "radgraph_f1_entity": 0.15825019075973226,
+  "radgraph_f1_relation": 0.1427606782830153,
   "radgraph_available": true,
   "radgraph_error": null
 }

evaluations/mimic_test_metrics.json

@@ -4,19 +4,19 @@
   "dataset": "mimic-cxr",
   "view_filter": "frontal-only (PA/AP)",
   "num_examples": 3041,
-  "bleu_1": 0.2227295189394753,
-  "bleu_4": 0.042573846171241554,
-  "meteor": 0.19888441013163247,
-  "rouge_l": 0.16924689298814102,
-  "chexpert_f1_14_micro": 0.21826654240447343,
-  "chexpert_f1_5_micro": 0.047995636760294516,
-  "chexpert_f1_14_macro": 0.07155355415264575,
-  "chexpert_f1_5_macro": 0.030688753269398433,
-  "chexpert_f1_micro": 0.21826654240447343,
-  "chexpert_f1_macro": 0.07155355415264575,
+  "bleu_1": 0.22660013916205363,
+  "bleu_4": 0.044065714234343675,
+  "meteor": 0.20172974056236193,
+  "rouge_l": 0.16930790037433466,
+  "chexpert_f1_14_micro": 0.22114848541163354,
+  "chexpert_f1_5_micro": 0.05919661733615222,
+  "chexpert_f1_14_macro": 0.0760104965288089,
+  "chexpert_f1_5_macro": 0.035443037974683546,
+  "chexpert_f1_micro": 0.22114848541163354,
+  "chexpert_f1_macro": 0.0760104965288089,
   "chexpert_per_label_f1": {
-    "Enlarged Cardiomediastinum": 0.0,
-    "Cardiomegaly": 0.15344376634699217,
+    "Enlarged Cardiomediastinum": 0.0273972602739726,
+    "Cardiomegaly": 0.17721518987341772,
     "Lung Opacity": 0.0,
     "Lung Lesion": 0.0,
     "Edema": 0.0,
@@ -27,12 +27,12 @@
     "Pleural Effusion": 0.0,
     "Pleural Other": 0.0,
     "Fracture": 0.0,
-    "Support Devices": 0.5560807907176623,
-    "No Finding": 0.29222520107238603
+    "Support Devices": 0.5660377358490566,
+    "No Finding": 0.29349676540687775
   },
-  "radgraph_f1": 0.07782635321975587,
-  "radgraph_f1_entity": 0.1406092548436011,
-  "radgraph_f1_relation": 0.1263392751416026,
+  "radgraph_f1": 0.0813727768314203,
+  "radgraph_f1_entity": 0.1452824713307004,
+  "radgraph_f1_relation": 0.13105287501866947,
   "radgraph_available": true,
   "radgraph_error": null,
   "evaluation_suite": "mimic_test_dual",
@@ -42,19 +42,19 @@
     "dataset": "mimic-cxr",
     "view_filter": "frontal-only (PA/AP)",
     "num_examples": 3041,
-    "bleu_1": 0.2227295189394753,
-    "bleu_4": 0.042573846171241554,
-    "meteor": 0.19888441013163247,
-    "rouge_l": 0.16924689298814102,
-    "chexpert_f1_14_micro": 0.21826654240447343,
-    "chexpert_f1_5_micro": 0.047995636760294516,
-    "chexpert_f1_14_macro": 0.07155355415264575,
-    "chexpert_f1_5_macro": 0.030688753269398433,
-    "chexpert_f1_micro": 0.21826654240447343,
-    "chexpert_f1_macro": 0.07155355415264575,
+    "bleu_1": 0.22660013916205363,
+    "bleu_4": 0.044065714234343675,
+    "meteor": 0.20172974056236193,
+    "rouge_l": 0.16930790037433466,
+    "chexpert_f1_14_micro": 0.22114848541163354,
+    "chexpert_f1_5_micro": 0.05919661733615222,
+    "chexpert_f1_14_macro": 0.0760104965288089,
+    "chexpert_f1_5_macro": 0.035443037974683546,
+    "chexpert_f1_micro": 0.22114848541163354,
+    "chexpert_f1_macro": 0.0760104965288089,
     "chexpert_per_label_f1": {
-      "Enlarged Cardiomediastinum": 0.0,
-      "Cardiomegaly": 0.15344376634699217,
+      "Enlarged Cardiomediastinum": 0.0273972602739726,
+      "Cardiomegaly": 0.17721518987341772,
       "Lung Opacity": 0.0,
       "Lung Lesion": 0.0,
       "Edema": 0.0,
@@ -65,12 +65,12 @@
       "Pleural Effusion": 0.0,
       "Pleural Other": 0.0,
       "Fracture": 0.0,
-      "Support Devices": 0.5560807907176623,
-      "No Finding": 0.29222520107238603
+      "Support Devices": 0.5660377358490566,
+      "No Finding": 0.29349676540687775
     },
-    "radgraph_f1": 0.07782635321975587,
-    "radgraph_f1_entity": 0.1406092548436011,
-    "radgraph_f1_relation": 0.1263392751416026,
+    "radgraph_f1": 0.0813727768314203,
+    "radgraph_f1_entity": 0.1452824713307004,
+    "radgraph_f1_relation": 0.13105287501866947,
     "radgraph_available": true,
     "radgraph_error": null
   },
@@ -80,19 +80,19 @@
     "dataset": "mimic-cxr",
     "view_filter": "frontal-only (PA/AP), structured Findings section only",
     "num_examples": 2210,
-    "bleu_1": 0.23179623497255453,
-    "bleu_4": 0.04902797277049711,
-    "meteor": 0.2125228998859286,
-    "rouge_l": 0.17742184702691816,
-    "chexpert_f1_14_micro": 0.21896792189679218,
-    "chexpert_f1_5_micro": 0.05121638924455826,
-    "chexpert_f1_14_macro": 0.07088977497872763,
-    "chexpert_f1_5_macro": 0.03252032520325203,
-    "chexpert_f1_micro": 0.21896792189679218,
-    "chexpert_f1_macro": 0.07088977497872763,
+    "bleu_1": 0.2365366145096764,
+    "bleu_4": 0.05092983875760064,
+    "meteor": 0.21583177087647024,
+    "rouge_l": 0.17757622652257565,
+    "chexpert_f1_14_micro": 0.22206742825299527,
+    "chexpert_f1_5_micro": 0.06491372226787182,
+    "chexpert_f1_14_macro": 0.07578923935276514,
+    "chexpert_f1_5_macro": 0.03816425120772947,
+    "chexpert_f1_micro": 0.22206742825299527,
+    "chexpert_f1_macro": 0.07578923935276514,
     "chexpert_per_label_f1": {
-      "Enlarged Cardiomediastinum": 0.0,
-      "Cardiomegaly": 0.16260162601626016,
+      "Enlarged Cardiomediastinum": 0.028169014084507043,
+      "Cardiomegaly": 0.19082125603864733,
       "Lung Opacity": 0.0,
       "Lung Lesion": 0.0,
       "Edema": 0.0,
@@ -103,12 +103,12 @@
       "Pleural Effusion": 0.0,
       "Pleural Other": 0.0,
       "Fracture": 0.0,
-      "Support Devices": 0.48493543758967,
-      "No Finding": 0.3449197860962567
+      "Support Devices": 0.4929681717246484,
+      "No Finding": 0.3490909090909091
     },
-    "radgraph_f1": 0.088204837339789,
-    "radgraph_f1_entity": 0.15309947777272206,
-    "radgraph_f1_relation": 0.1377130122662026,
+    "radgraph_f1": 0.09257418507561303,
+    "radgraph_f1_entity": 0.15825019075973226,
+    "radgraph_f1_relation": 0.1427606782830153,
     "radgraph_available": true,
     "radgraph_error": null
   }

run_summary.json

@@ -44,5 +44,122 @@
   "target_duration_mode": "per_invocation",
   "repo_id": "manu02/LAnA-v4",
   "train_datasets": "MIMIC-CXR (findings-only)",
-  "validation_datasets": "MIMIC-CXR (findings-only)"
+  "validation_datasets": "MIMIC-CXR (findings-only)",
+  "repo_url": "https://huggingface.co/manu02/LAnA-v4",
+  "latest_evaluation": {
+    "split": "test",
+    "subset": "all frontal studies",
+    "dataset": "mimic-cxr",
+    "view_filter": "frontal-only (PA/AP)",
+    "num_examples": 3041,
+    "bleu_1": 0.22660013916205363,
+    "bleu_4": 0.044065714234343675,
+    "meteor": 0.20172974056236193,
+    "rouge_l": 0.16930790037433466,
+    "chexpert_f1_14_micro": 0.22114848541163354,
+    "chexpert_f1_5_micro": 0.05919661733615222,
+    "chexpert_f1_14_macro": 0.0760104965288089,
+    "chexpert_f1_5_macro": 0.035443037974683546,
+    "chexpert_f1_micro": 0.22114848541163354,
+    "chexpert_f1_macro": 0.0760104965288089,
+    "chexpert_per_label_f1": {
+      "Enlarged Cardiomediastinum": 0.0273972602739726,
+      "Cardiomegaly": 0.17721518987341772,
+      "Lung Opacity": 0.0,
+      "Lung Lesion": 0.0,
+      "Edema": 0.0,
+      "Consolidation": 0.0,
+      "Pneumonia": 0.0,
+      "Atelectasis": 0.0,
+      "Pneumothorax": 0.0,
+      "Pleural Effusion": 0.0,
+      "Pleural Other": 0.0,
+      "Fracture": 0.0,
+      "Support Devices": 0.5660377358490566,
+      "No Finding": 0.29349676540687775
+    },
+    "radgraph_f1": 0.0813727768314203,
+    "radgraph_f1_entity": 0.1452824713307004,
+    "radgraph_f1_relation": 0.13105287501866947,
+    "radgraph_available": true,
+    "radgraph_error": null
+  },
+  "latest_evaluations": {
+    "all_test": {
+      "split": "test",
+      "subset": "all frontal studies",
+      "dataset": "mimic-cxr",
+      "view_filter": "frontal-only (PA/AP)",
+      "num_examples": 3041,
+      "bleu_1": 0.22660013916205363,
+      "bleu_4": 0.044065714234343675,
+      "meteor": 0.20172974056236193,
+      "rouge_l": 0.16930790037433466,
+      "chexpert_f1_14_micro": 0.22114848541163354,
+      "chexpert_f1_5_micro": 0.05919661733615222,
+      "chexpert_f1_14_macro": 0.0760104965288089,
+      "chexpert_f1_5_macro": 0.035443037974683546,
+      "chexpert_f1_micro": 0.22114848541163354,
+      "chexpert_f1_macro": 0.0760104965288089,
+      "chexpert_per_label_f1": {
+        "Enlarged Cardiomediastinum": 0.0273972602739726,
+        "Cardiomegaly": 0.17721518987341772,
+        "Lung Opacity": 0.0,
+        "Lung Lesion": 0.0,
+        "Edema": 0.0,
+        "Consolidation": 0.0,
+        "Pneumonia": 0.0,
+        "Atelectasis": 0.0,
+        "Pneumothorax": 0.0,
+        "Pleural Effusion": 0.0,
+        "Pleural Other": 0.0,
+        "Fracture": 0.0,
+        "Support Devices": 0.5660377358490566,
+        "No Finding": 0.29349676540687775
+      },
+      "radgraph_f1": 0.0813727768314203,
+      "radgraph_f1_entity": 0.1452824713307004,
+      "radgraph_f1_relation": 0.13105287501866947,
+      "radgraph_available": true,
+      "radgraph_error": null
+    },
+    "findings_only_test": {
+      "split": "test",
+      "subset": "findings-only frontal studies",
+      "dataset": "mimic-cxr",
+      "view_filter": "frontal-only (PA/AP), structured Findings section only",
+      "num_examples": 2210,
+      "bleu_1": 0.2365366145096764,
+      "bleu_4": 0.05092983875760064,
+      "meteor": 0.21583177087647024,
+      "rouge_l": 0.17757622652257565,
+      "chexpert_f1_14_micro": 0.22206742825299527,
+      "chexpert_f1_5_micro": 0.06491372226787182,
+      "chexpert_f1_14_macro": 0.07578923935276514,
+      "chexpert_f1_5_macro": 0.03816425120772947,
+      "chexpert_f1_micro": 0.22206742825299527,
+      "chexpert_f1_macro": 0.07578923935276514,
+      "chexpert_per_label_f1": {
+        "Enlarged Cardiomediastinum": 0.028169014084507043,
+        "Cardiomegaly": 0.19082125603864733,
+        "Lung Opacity": 0.0,
+        "Lung Lesion": 0.0,
+        "Edema": 0.0,
+        "Consolidation": 0.0,
+        "Pneumonia": 0.0,
+        "Atelectasis": 0.0,
+        "Pneumothorax": 0.0,
+        "Pleural Effusion": 0.0,
+        "Pleural Other": 0.0,
+        "Fracture": 0.0,
+        "Support Devices": 0.4929681717246484,
+        "No Finding": 0.3490909090909091
+      },
+      "radgraph_f1": 0.09257418507561303,
+      "radgraph_f1_entity": 0.15825019075973226,
+      "radgraph_f1_relation": 0.1427606782830153,
+      "radgraph_available": true,
+      "radgraph_error": null
+    }
+  }
 }