Acknowledgements
This thesis would not have been possible without the help of many people. My deepest thanks go to my supervisor, Dr. Nghiem Thi Phuong, whose questions, patience, and detailed feedback shaped both the direction of this project and the way I approached it. I am grateful to the lecturers of the Department of Information and Communication Technology at the University of Science and Technology of Hanoi for the foundation they gave me over the past years. Finally, I thank my family and friends, who kept me motivated through the long nights of training runs and debugging that this work demanded.
Abstract
This thesis presents CXR-HIEU, a vision–language model for chest X-ray interpretation that performs findings generation, impression generation, and visual question answering with a single shared backbone. The model couples a frozen RAD-DINO image encoder with a trainable projection and a Vicuna-7B language model adapted through low-rank (LoRA) adapters, and is explicitly guided by a 14-pathology Positive / Negative / Uncertain (PNU) abnormality signal produced by a CheXpert-style classifier. It is trained with a parameter-efficient two-stage schedule — an image–text contrastive alignment followed by instruction tuning — on a 50,000-study subset of MIMIC-CXR, chosen so that training remains feasible on a limited cloud-compute budget while preserving the pathology distribution of the full dataset. On the held-out test split the model is competitive with established systems on semantic report-generation metrics (ROUGE-L 0.292 and a high METEOR and BERTScore for findings), reaches a visual-question-answering exact match of 0.306 (stronger on closed-ended questions), and its abnormality classifier attains a macro F1 of 0.328. These results show that a frozen-encoder, frozen-LLM design adapted only through a small projection and LoRA adapters can drive three chest-X-ray tasks at once on modest hardware, and point to abnormality classification and clinical-efficacy evaluation as the main directions for further improvement.
List of Abbreviations
| Abbreviation | Meaning |
|---|---|
| AI | Artificial Intelligence |
| AP | Anteroposterior (X-ray view) |
| BERTScore | BERT-based similarity Score |
| BLEU | Bilingual Evaluation Understudy |
| CE | Clinical Efficacy (F1) |
| CheXbert | BERT-based CheXpert report labeler |
| CheXpert | Chest eXpert — 14-label chest X-ray labeler |
| CXR | Chest X-Ray |
| DINO | self-Distillation with NO labels (DINOv2 self-supervision) |
| EHR | Electronic Health Record |
| F1 | F1-score (harmonic mean of precision and recall) |
| GPU | Graphics Processing Unit |
| InfoNCE | Info Noise-Contrastive Estimation (loss) |
| IQR | Interquartile Range |
| ITC | Image–Text Contrastive (learning) |
| LLM | Large Language Model |
| LoRA | Low-Rank Adaptation |
| METEOR | Metric for Evaluation of Translation with Explicit ORdering |
| MIMIC-CXR | Medical Information Mart for Intensive Care — Chest X-Ray |
| MLP | Multi-Layer Perceptron |
| NF4 | 4-bit NormalFloat quantisation |
| NLG | Natural Language Generation |
| PA | Posteroanterior (X-ray view) |
| PNU | Positive / Negative / Uncertain |
| QLoRA | Quantised Low-Rank Adaptation |
| RAD-DINO | Chest-X-ray ViT image encoder (DINOv2-based) |
| ROUGE | Recall-Oriented Understudy for Gisting Evaluation |
| RRG | Radiology Report Generation |
| ViT | Vision Transformer |
| VLM | Vision–Language Model |
| VQA | Visual Question Answering |