docs/report_part4_5.md

---

# 4. Experiments and Results

## 4.1 Experiments

### 4.1.1 Dataset and configuration

All experiments use the 50,000-study subset described in Section 3.3: 40,000 train / 5,000 validation / 5,000 test, patient-disjoint, one frontal image per study, with the impression task conditioned on the ground-truth findings. Findings and impression targets come from the MIMIC-CXR reports; VQA samples come from MIMIC-Ext-CXR-VQA, attached by study. The reported run is evaluated on the held-out test split with predicted PNU guidance — the abnormality string comes from the frozen CheXpert classifier rather than from ground-truth labels, which is the realistic inference setting. The VQA data contain 127,010 question–answer items in total (102,941 train / 12,189 validation / 11,880 test); there are far more VQA samples than studies because a single image can be paired with several questions, whereas the findings and impression tasks contribute one sample per study. All metrics below are on the 11,880-item test set.

### 4.1.2 Implementation and hyperparameters

The model is implemented in PyTorch with HuggingFace `transformers`, `peft` (LoRA), and `bitsandbytes` (4-bit NF4 quantisation). Training runs on a single GPU through cloud notebooks, with the per-device batch size and gradient accumulation auto-tuned to the GPU so that the effective batch size stays fixed. The image encoder and the base LLM are frozen throughout; only the CheXpert head (Stage 0), the projection (Stages 1–2), the LoRA adapters (Stage 2), and the ITC head (Stage 1) ever receive gradients.

**Stage 0 (CheXpert classifier).** The abnormality head is a small MLP (768 → 256 → 14 × 3) trained on top of pre-computed, frozen RAD-DINO `[CLS]` embeddings, so the encoder is run only once and the head trains in minutes on a T4. To handle the heavy class imbalance it is optimised with an asymmetric focal cross-entropy (γ = 0 for the rare positive/uncertain states, γ = 4 for the dominant negative state) rather than inverse-frequency weighting, and the best epoch is selected by the macro-F1 of the positive class. Table 4.1 lists the hyperparameters of all three stages.

**Table 4.1.** Training hyperparameters for the three stages.

| Hyperparameter | Stage 0 (classifier) | Stage 1 (ITC) | Stage 2 |
|---|---|---|---|
| Trainable modules | CheXpert MLP head | Projection + ITC head | Projection + LoRA |
| Frozen modules | Encoder; LLM not loaded | Encoder, classifier; LLM not loaded | Encoder, classifier, base LLM |
| Loss | Asymmetric focal CE (γ 0 / 4) | Symmetric InfoNCE | Causal CE |
| Temperature τ | — | 0.07 | — |
| Epochs | 8 | 2 | 10 |
| Peak learning rate | 1e-3 | 1e-3 | 2e-4 |
| LR schedule | cosine | cosine, 5% warm-up | cosine, 5% warm-up |
| Weight decay | 1e-4 | 0.01 | 0.01 |
| Dropout | 0.2 | — | — |
| Effective batch size | 256 | 64–96 | 16 |
| Mixed precision | FP32 head (cached embeds) | BF16 (Ampere+) / FP16 (T4) | BF16 / FP16 |
| LLM quantisation | — | — | 4-bit NF4, double-quant |
| LoRA (r, α, dropout) | — | — | (16, 32, 0.05) |
| LoRA modules | — | — | {q, k, v, o}_proj |
| Cutoff length | — | — | 512 tokens |
| Optimiser | AdamW | AdamW | AdamW |

### 4.1.3 Evaluation protocol

Metrics follow Section 3.6. Generation uses greedy decoding with the canonical instruction variant; the maximum number of new tokens is 300 (findings), 200 (impression), and 64 (VQA). For report generation we report BLEU-1/4, ROUGE-1/2/L, METEOR, and BERTScore-F1; for VQA we report exact match, token-F1, micro-F1, BLEU-1, METEOR, and BERTScore-F1. Clinical correctness is assessed through the dedicated CheXpert abnormality classifier (Section 4.2.2), which predicts the 14 pathologies directly from the image and is scored with per-pathology precision, recall, and F1 (macro-averaged).

> **Note on cross-paper comparison.** The baseline numbers in Table 4.2 are quoted as compiled in the META-CXR paper (Edirisinghe et al., 2025, Table 2), which is the closest prior work to ours. NLG scores (BLEU, METEOR, ROUGE, BERTScore) depend on the tokeniser, smoothing, and the specific BERTScore backbone/baseline, so absolute values are only approximately comparable across papers; the ordering matters more than the exact digits. In particular our BERTScore-F1 (DistilBERT, no baseline rescaling) is not comparable to META-CXR's rescaled BERTScore of 0.426 and is therefore omitted from the comparison table.

## 4.2 Results

### 4.2.1 Report generation: findings and impression

Table 4.2 compares CXR-HIEU findings generation with representative MIMIC-CXR systems. The model is competitive across all standard metrics and sits in the mid-range of the comparison: BLEU-1 (0.364) is close to RaDialog (0.360) and KiUT (0.362); BLEU-4 (0.103) is on par with META-CXR (0.102) and above RaDialog (0.095); METEOR (0.161) is in line with KiUT and R2GenGPT (0.160) and just below META-CXR (0.173); and ROUGE-L (0.292) matches R2GenGPT (0.297) and exceeds RaDialog (0.271) and META-CXR (0.280). Only the largest systems (KARGEN, MAIRA-2) lead clearly. This places CXR-HIEU among recent LLM-based generators despite its much smaller, frozen-backbone design.

**Table 4.2.** Findings generation on MIMIC-CXR test. Baseline values as compiled in Edirisinghe et al. (2025), Table 2; "—" = not reported.

| Model | BLEU-1 | BLEU-4 | METEOR | ROUGE-L | CIDEr |
|---|---|---|---|---|---|
| METransformer | 0.379 | 0.124 | 0.152 | 0.291 | 0.362 |
| KiUT | 0.362 | 0.113 | 0.160 | 0.285 | — |
| R2GenGPT | 0.401 | 0.134 | 0.160 | 0.297 | 0.269 |
| RaDialog | 0.360 | 0.095 | 0.140 | 0.271 | — |
| KARGEN | 0.417 | 0.140 | 0.165 | 0.305 | 0.289 |
| MAIRA-2 | 0.460 | 0.231 | — | 0.384 | — |
| META-CXR | 0.390 | 0.102 | 0.173 | 0.280 | 0.291 |
| **CXR-HIEU (ours)** | **0.364** | **0.103** | **0.161** | **0.292** | **—** |

The full breakdown for both report tasks is given in Table 4.3. Impression scores are markedly lower than findings on every metric, which is discussed in Section 4.3.

**Table 4.3.** CXR-HIEU report-generation results (test). BERTScore-F1 is DistilBERT-based (not comparable to other papers; see note above).

| Task | BLEU-1 | BLEU-4 | ROUGE-1 | ROUGE-2 | ROUGE-L | METEOR | BERTScore-F1 |
|---|---|---|---|---|---|---|---|
| Findings | 0.364 | 0.103 | 0.403 | 0.174 | 0.292 | 0.161 | 0.803 |
| Impression | 0.100 | 0.013 | 0.136 | 0.036 | 0.118 | 0.104 | 0.720 |

### 4.2.2 Clinical correctness: abnormality classification

Clinical correctness is measured by the CheXpert abnormality classifier (Stage 0), which predicts the 14 pathologies — with an explicit *uncertain* state — directly from the image. This is the same role that META-CXR's MHCAC classifier plays. Table 4.4 gives our per-pathology results. Performance is strongest on common, visually salient categories (No Finding F1 0.785, Pleural Effusion 0.622, Support Devices 0.552, Edema 0.508) and collapses to zero on rare categories with few positives (Enlarged Cardiomediastinum, Pleural Other, Fracture), reflecting the strong class imbalance of MIMIC-CXR. The macro averages over the 14 labels are precision 0.319, recall 0.377, F1 0.328; over the 11 labels the model actually predicts (excluding the three degenerate classes) the macro-F1 rises to about 0.42.

**Table 4.4.** CheXpert classifier per-pathology results (test split).

| Pathology | Precision | Recall | F1 |
|---|---|---|---|
| No Finding | 0.679 | 0.930 | 0.785 |
| Enlarged Cardiomediastinum | 0.000 | 0.000 | 0.000 |
| Cardiomegaly | 0.345 | 0.462 | 0.395 |
| Lung Opacity | 0.317 | 0.648 | 0.425 |
| Lung Lesion | 0.345 | 0.110 | 0.167 |
| Edema | 0.447 | 0.587 | 0.508 |
| Consolidation | 0.276 | 0.088 | 0.133 |
| Pneumonia | 0.282 | 0.158 | 0.203 |
| Atelectasis | 0.340 | 0.456 | 0.390 |
| Pneumothorax | 0.438 | 0.382 | 0.408 |
| Pleural Effusion | 0.545 | 0.724 | 0.622 |
| Pleural Other | 0.000 | 0.000 | 0.000 |
| Fracture | 0.000 | 0.000 | 0.000 |
| Support Devices | 0.445 | 0.727 | 0.552 |
| **Macro average (14)** | **0.319** | **0.377** | **0.328** |

Beyond the per-class precision/recall/F1, the *uncertain* state is almost never recovered: the classifier's accuracy on uncertain-labelled cases is close to zero for most pathologies (non-zero only for Pneumonia ≈ 0.21, Edema ≈ 0.08, Atelectasis ≈ 0.02), so in practice it collapses the three-class problem onto positive vs negative. We do not benchmark these numbers against report-generation systems: most of the literature reports a *report-derived* clinical-efficacy F1 — running a labeler on the generated report text — which measures a different quantity from our image-level classifier and is therefore not directly comparable.

### 4.2.3 Visual question answering

Table 4.5 gives the overall VQA results on the 11,880-item test set. Tables 4.6 and 4.7 break the score down by answer type and by question semantic/content type. The model is clearly stronger on closed-ended questions (exact match 0.385) than on open-ended ones (0.204), and within content types it does best on size and plane questions and worst on anatomy and attribute questions.

**Table 4.5.** Overall VQA results (test, 11,880 items).

| Metric | Score |
|---|---|
| Set micro-F1 | 0.424 |
| Exact match | 0.306 |
| Token-F1 | 0.321 |
| BLEU-1 | 0.101 |
| METEOR | 0.179 |
| BERTScore-F1 | 0.435 |

**Table 4.5.** VQA by answer type and semantic type.

| Subset | n | Micro-F1 | Exact match | Token-F1 |
|---|---|---|---|---|
| Closed | 6,672 | 0.500 | 0.385 | 0.385 |
| Open | 5,208 | 0.359 | 0.204 | 0.239 |
| Verify | 6,672 | 0.500 | 0.385 | 0.385 |
| Choose | 1,171 | 0.353 | 0.214 | 0.242 |
| Query | 4,037 | 0.360 | 0.201 | 0.238 |

**Table 4.5.** VQA by content type.

| Content type | n | Micro-F1 | Exact match | Token-F1 |
|---|---|---|---|---|
| Presence | 4,514 | 0.486 | 0.367 | 0.367 |
| Abnormality | 2,445 | 0.380 | 0.288 | 0.310 |
| Attribute | 2,003 | 0.347 | 0.194 | 0.229 |
| Anatomy | 1,574 | 0.386 | 0.189 | 0.227 |
| Size | 659 | 0.569 | 0.516 | 0.516 |
| Gender | 359 | 0.411 | 0.284 | 0.284 |
| Plane | 326 | 0.601 | 0.439 | 0.439 |

## 4.3 Discussion

**Report generation.** On findings, CXR-HIEU is competitive with established systems across all standard metrics — BLEU-1 0.364, BLEU-4 0.103, METEOR 0.161, ROUGE-L 0.292 — sitting in the mid-range of Table 4.2, close to RaDialog, KiUT, and R2GenGPT, and behind only the largest models (KARGEN, MAIRA-2). This is a strong result given that only a small projection and LoRA adapters are trained on top of a frozen encoder and a frozen 7B LLM. We avoid any state-of-the-art claim on absolute scores, since BLEU/METEOR/BERTScore are sensitive to tokenisation and metric implementation and are only approximately comparable across papers.

**Impression weakness.** Impression scores are much lower than findings on every metric, even though the impression prompt is given the ground-truth findings and should make the task easier. Likely factors: impressions are short and abstractive (harsh for exact-overlap metrics), the impression task gets a smaller training share, and the model may carry findings-style phrasing into the impression. This is the clearest area for improvement (Section 5).

**Clinical correctness.** The Stage-0 CheXpert classifier — our clinical metric — reaches macro-F1 0.328 over 14 labels (≈ 0.42 over the 11 labels it can predict). The gap to a heavier system such as META-CXR's multi-encoder classifier (weighted F1 0.73) is consistent with their design — three fused encoders (CNN, ViT, Swin), expert-token cross-attention, and class-balanced/contrastive losses — versus our single-encoder MLP head on top of frozen RAD-DINO features. Because the test-time PNU string is produced by this classifier (predicted, not oracle), its errors propagate into the prompt, so improving it is the highest-leverage next step.

**VQA.** The model handles closed-ended verification and presence questions far better than open-ended query and attribute questions, mirroring the general difficulty ordering in medical VQA. The strong results on size and plane questions suggest the visual tokens retain coarse geometric information well.

**Comparison with prior work.** CXR-HIEU's findings metrics fall within the range reported by recent LLM-based chest-X-ray systems (e.g. RaDialog, R2GenGPT, KiUT, META-CXR), despite training only a small projection and LoRA adapters on a frozen backbone. The abnormality classifier is the weaker component: a single-encoder MLP head cannot match a heavier multi-encoder classifier such as META-CXR's, whose authors themselves flag its inference-time complexity as a drawback. CXR-HIEU instead trades classifier accuracy for a far smaller and simpler model, consistent with the limited-compute goal of this thesis.

**Limitations.** Several limitations qualify these results. (i) Evaluation uses a single corpus (MIMIC-CXR) from one institution, so generalisation to other scanners and patient populations is untested. (ii) The abnormality classifier is the weakest link. Three rare labels — Enlarged Cardiomediastinum, Pleural Other, and Fracture — score an exact zero on precision, recall, and F1: the head almost never fires on them because there are too few positive examples. The accuracy on the *uncertain* state is likewise very low, so the model effectively reduces the three-class problem to positive vs negative. Part of this is intrinsic rather than a modelling failure: a finding is labelled *uncertain* precisely because the reporting radiologist was not sure, and such judgements often rely on clinical context, prior studies, or experience — an expert may suspect a condition by inference — rather than on cues that are visible in the raw image alone, so an image-only classifier has little signal to learn from. (iii) Because the PNU string is predicted by this classifier, its errors propagate into generation and VQA. (iv) Like any LLM-based generator, the model can hallucinate fluent but unsupported statements that lexical metrics do not penalise. (v) Impression generation clearly underperforms findings. (vi) Compute constraints fix a small effective batch, a 7B LLM, and a single-view, single-image setup, with no multi-view or longitudinal (prior-study) context. Cross-paper metric comparison is also only approximate, given differing tokenisers and metric implementations.

---

# 5. Conclusion and future work

This thesis presented CXR-HIEU, a unified vision–language model that performs chest-X-ray findings generation, impression generation, and visual question answering with a single RAD-DINO + projection + Vicuna-7B (LoRA) backbone, guided by an explicit Positive/Negative/Uncertain abnormality signal and trained with a parameter-efficient two-stage schedule. On the MIMIC-CXR test split the model is competitive with established systems on the standard report-generation metrics (BLEU-1 0.364, BLEU-4 0.103, METEOR 0.161, ROUGE-L 0.292), its abnormality classifier reaches a macro-F1 of 0.328 (≈ 0.42 on the labels it can predict), and it achieves a VQA exact match of 0.306, markedly stronger on closed-ended questions. These results show that a frozen-encoder, frozen-LLM design adapted only through a small projection and LoRA adapters can drive three chest-X-ray tasks at once on modest hardware.

Several directions would strengthen the work:

- **Stronger abnormality classifier.** The clearest lever: adopt a META-CXR-style design (multi-encoder fusion, class-balancing or focal loss, contrastive/uncertainty objectives) so the predicted PNU string is more reliable and propagates fewer errors into generation and VQA.
- **Clinical validation.** Assess the generated reports with expert radiologists for factual correctness, beyond the lexical and classifier metrics used here.
- **Impression generation.** Investigate and close the gap on impression (dedicated decoding budget, task-specific tuning, or a true end-to-end findings→impression cascade).
- **Scale and backbones.** Train on more data and views (multi-image studies) and evaluate stronger LLMs (e.g. Llama-3).
- **Cross-dataset validation.** Evaluate on IU X-ray and CheXpert to measure generalisation beyond MIMIC-CXR.

---

# References

1. E. Çallı, E. Sogancioglu, B. van Ginneken, K. G. van Leeuwen, K. Murphy. *Deep Learning for Chest X-ray Analysis: A Survey.* Medical Image Analysis, vol. 72, 2021.

2. D. Edirisinghe, W. Nimalsiri, M. Hennayake, D. Meedeniya, G. Lim. *Chest X-Ray Report Generation Using Abnormality Guided Vision Language Model (META-CXR).* IEEE Access, vol. 13, 2025.
3. C. Pellegrini, E. Özsoy, B. Busam, N. Navab, M. Keicher. *RaDialog: A Large Vision-Language Model for Radiology Report Generation and Conversational Assistance.* arXiv:2311.18681, 2023.
4. Z. Wang, L. Liu, L. Wang, L. Zhou. *R2GenGPT: Radiology Report Generation with Frozen LLMs.* Meta-Radiology, 2023.
5. Y. Li, Z. Wang, Y. Liu, L. Wang, L. Liu, L. Zhou. *KARGEN: Knowledge-enhanced Automated Radiology Report Generation using Large Language Models.* arXiv:2409.05370, 2024.
6. S. Bannur et al. *MAIRA-2: Grounded Radiology Report Generation.* arXiv:2406.04449, 2024.
7. Z. Wang, L. Liu, L. Wang, L. Zhou. *METransformer: Radiology Report Generation by Transformer with Multiple Learnable Expert Tokens.* CVPR, 2023.
8. Z. Huang, X. Zhang, S. Zhang. *KiUT: Knowledge-injected U-transformer for Radiology Report Generation.* CVPR, 2023.
9. F. Pérez-García et al. *RAD-DINO: Exploring Scalable Medical Image Encoders Beyond Text Supervision.* arXiv:2401.10815, 2024.
10. J. Li, D. Li, S. Savarese, S. Hoi. *BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models.* ICML, 2023.
11. W.-L. Chiang et al. *Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90% ChatGPT Quality.* 2023.
12. E. J. Hu et al. *LoRA: Low-Rank Adaptation of Large Language Models.* ICLR, 2022.
13. T. Dettmers, A. Pagnoni, A. Holtzman, L. Zettlemoyer. *QLoRA: Efficient Finetuning of Quantized LLMs.* NeurIPS, 2023.
14. A. Smit et al. *CheXbert: Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERT.* EMNLP, 2020.
15. J. Irvin et al. *CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison.* AAAI, 2019.
16. A. E. W. Johnson et al. *MIMIC-CXR-JPG, a Large Publicly Available Database of Labeled Chest Radiographs.* arXiv:1901.07042, 2019.
17. T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, Y. Artzi. *BERTScore: Evaluating Text Generation with BERT.* ICLR, 2020.
18. S. Banerjee, A. Lavie. *METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments.* ACL Workshop, 2005.
19. J. J. Lau, S. Gayen, A. Ben Abacha, D. Demner-Fushman. *A Dataset of Clinically Generated Visual Questions and Answers about Radiology Images (VQA-RAD).* Scientific Data, 2018.
20. B. Liu, L.-M. Zhan, L. Xu, L. Ma, Y. Yang, X.-M. Wu. *SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answering.* IEEE ISBI, 2021.
21. X. Hu et al. *Expert Knowledge-Aware Image Difference Graph Representation Learning for Difference-Aware Medical Visual Question Answering (Medical-Diff-VQA / MIMIC-Diff-VQA).* ACM SIGKDD (KDD), 2023.
22. S. Bae et al. *EHRXQA: A Multi-Modal Question Answering Dataset for Electronic Health Records with Chest X-ray Images (incl. MIMIC-CXR-VQA).* NeurIPS Datasets & Benchmarks, 2023.