docs/_dump.txt

[TOCHeading] Table of Contents
[TOC1] Acknowledgements2
[TOC1] Abstract2
[TOC1] List of Abbreviations3
[TOC1] 1. Introduction4
[TOC2] 1.1 Context and Motivation4
[TOC2] 1.2 Internship objectives5
[TOC2] 1.3 Report structure5
[TOC1] 2. Related works6
[TOC2] 2.1 Medical report generation6
[TOC2] 2.2 Visual question answering6
[TOC1] 3. Materials and Methods7
[TOC2] 3.1 Overview7
[TOC2] 3.2 Key Concepts and Definitions9
[TOC2] 3.3 Data Preparation9
[TOC3] 3.3.1 Sources and dataset overview10
[TOC3] 3.3.2 Selecting the working subset13
[TOC3] 3.3.3 Storage and Consumption Layout16
[TOC3] 3.3.4 Training data format16
[TOC2] 3.4 Model Architecture17
[TOC3] 3.4.1 Image Encoder18
[TOC3] 3.4.2 MLP Projection19
[TOC3] 3.4.3 CheXpert Abnormality Classifier20
[TOC3] 3.4.4 Language Model and Parameter-Efficient Adaptation21
[TOC3] 3.4.5 Prompt Assembly21
[TOC2] 3.5 Training Strategy22
[TOC3] 3.5.1 Stage 0 — CheXpert Classifier Head23
[TOC3] 3.5.2 Stage 1 — Contrastive Alignment23
[TOC3] 3.5.3 Stage 2 — Instruction Tuning24
[TOC3] 3.5.4 Loss Masking and Image-Token Accounting24
[TOC2] 3.6 Evaluation Protocol25
[TOC3] 3.6.1 Tasks and Held-Out Data25
[TOC3] 3.6.2 Natural Language Generation Metrics26
[TOC3] 3.6.3 Clinical Correctness: Abnormality Classification26
[TOC3] 3.6.4 Visual Question Answering Metrics26
[TOC3] 3.6.5 Model Selection and Reporting27
[TOC1] 4. Experiments and Results27
[TOC2] 4.1 Experiments27
[TOC3] 4.1.1 Dataset and configuration27
[TOC3] 4.1.2 Implementation and hyperparameters27
[TOC3] 4.1.3 Evaluation protocol28
[TOC2] 4.2 Results29
[TOC3] 4.2.1 Report generation: findings and impression29
[TOC3] 4.2.2 Clinical correctness: abnormality classification30
[TOC3] 4.2.3 Visual question answering31
[TOC2] 4.3 Discussion32
[TOC1] 5. Conclusion and future work33
[TOC1] References34
[Heading1] Acknowledgements
[FirstParagraph] This thesis would not have been possible without the help of many people. My deepest thanks go to my supervisor, Dr. Nghiem Thi Phuong, whose questions, patience, and detailed feedback shaped both the direction of this project and the way I approached it. I am grateful to the lecturers of the Department of Information and Communication Technology at the University of Science and Technology of Hanoi for the foundation they gave me over the past years. Finally, I thank my family and friends, who kept me motivated through the long nights of training runs and debugging that this work demanded.
[Heading1] Abstract
[FirstParagraph] This thesis presents CXR-HIEU, a vision–language model for chest X-ray interpretation that performs findings generation, impression generation, and visual question answering with a single shared backbone. The model couples a frozen RAD-DINO image encoder with a trainable projection and a Vicuna-7B language model adapted through low-rank (LoRA) adapters, and is explicitly guided by a 14-pathology Positive / Negative / Uncertain (PNU) abnormality signal produced by a CheXpert-style classifier. It is trained with a parameter-efficient two-stage schedule — an image–text contrastive alignment followed by instruction tuning — on a 50,000-study subset of MIMIC-CXR, chosen so that training remains feasible on a limited cloud-compute budget while preserving the pathology distribution of the full dataset. On the held-out test split the model is competitive with established systems on semantic report-generation metrics (ROUGE-L 0.292 and a high METEOR and BERTScore for findings), reaches a visual-question-answering exact match of 0.306 (stronger on closed-ended questions), and its abnormality classifier attains a macro F1 of 0.328. These results show that a frozen-encoder, frozen-LLM design adapted only through a small projection and LoRA adapters can drive three chest-X-ray tasks at once on modest hardware, and point to abnormality classification and clinical-efficacy evaluation as the main directions for further improvement.
[Heading1] List of Abbreviations
[Compact] Abbreviation
[Compact] Meaning
[Compact] AI
[Compact] Artificial Intelligence
[Compact] AP
[Compact] Anteroposterior (X-ray view)
[Compact] BERTScore
[Compact] BERT-based similarity Score
[Compact] BLEU
[Compact] Bilingual Evaluation Understudy
[Compact] CE
[Compact] Clinical Efficacy (F1)
[Compact] CheXbert
[Compact] BERT-based CheXpert report labeler
[Compact] CheXpert
[Compact] Chest eXpert — 14-label chest X-ray labeler
[Compact] CXR
[Compact] Chest X-Ray
[Compact] DINO
[Compact] self-Distillation with NO labels (DINOv2 self-supervision)
[Compact] EHR
[Compact] Electronic Health Record
[Compact] F1
[Compact] F1-score (harmonic mean of precision and recall)
[Compact] GPU
[Compact] Graphics Processing Unit
[Compact] InfoNCE
[Compact] Info Noise-Contrastive Estimation (loss)
[Compact] IQR
[Compact] Interquartile Range
[Compact] ITC
[Compact] Image–Text Contrastive (learning)
[Compact] LLM
[Compact] Large Language Model
[Compact] LoRA
[Compact] Low-Rank Adaptation
[Compact] METEOR
[Compact] Metric for Evaluation of Translation with Explicit ORdering
[Compact] MIMIC-CXR
[Compact] Medical Information Mart for Intensive Care — Chest X-Ray
[Compact] MLP
[Compact] Multi-Layer Perceptron
[Compact] NF4
[Compact] 4-bit NormalFloat quantisation
[Compact] NLG
[Compact] Natural Language Generation
[Compact] PA
[Compact] Posteroanterior (X-ray view)
[Compact] PNU
[Compact] Positive / Negative / Uncertain
[Compact] QLoRA
[Compact] Quantised Low-Rank Adaptation
[Compact] RAD-DINO
[Compact] Chest-X-ray ViT image encoder (DINOv2-based)
[Compact] ROUGE
[Compact] Recall-Oriented Understudy for Gisting Evaluation
[Compact] RRG
[Compact] Radiology Report Generation
[Compact] ViT
[Compact] Vision Transformer
[Compact] VLM
[Compact] Vision–Language Model
[Compact] VQA
[Compact] Visual Question Answering
[Heading1] 1. Introduction
[Heading2] 1.1 Context and Motivation
[FirstParagraph] The chest X-ray (CXR) is the most frequently performed medical imaging examination in the world, with over two billion procedures carried out every year for the screening and diagnosis of cardiothoracic conditions [1]. Each examination must be read by a radiologist, who then writes a structured free-text report — typically a detailed Findings section followed by a concise Impression. As imaging volume keeps growing while the number of trained radiologists remains limited, report turnaround time, reader fatigue, and inter-reader variability have become real bottlenecks in clinical workflows.
[BodyText] Automatic radiology report generation (RRG) aims to ease this burden by drafting a report directly from the image, which the radiologist can then verify and edit. The field has evolved quickly: from early CNN–RNN captioning models, to transformer-based architectures, and most recently to large vision–language models (VLMs) such as LLaVA and BLIP-2, whose medical adaptations (RaDialog, LLaVA-Med, CheXagent, MAIRA-2) couple a frozen image encoder with a large language model (LLM) to produce fluent, instruction-following text. In parallel, clinicians often need to ask focused questions about an image — “is there a pleural effusion?”, “what is the size of the cardiac silhouette?” — which motivates visual question answering (VQA) on chest X-rays.
[BodyText] A recurring difficulty is that fluent text is not necessarily clinically correct: a model can produce a well-formed report that misses or flips a finding. A natural way to mitigate this is to feed the language model explicit pathology labels, so that generation stays anchored to detected abnormalities rather than to language priors alone. Motivated by this idea, this thesis builds a single VLM that performs findings generation, impression generation, and VQA, guided by an explicit 14-pathology Positive / Negative / Uncertain (PNU) signal.
[Heading2] 1.2 Internship objectives
[FirstParagraph] The objective of this internship is to design, implement, and evaluate CXR-HIEU, a unified vision–language model for chest X-ray interpretation built on a shared RAD-DINO + projection + Vicuna-7B (LoRA) backbone. Concretely, the work targets the following goals:
[Compact] Data pipeline. Build a reproducible pipeline that filters the MIMIC-CXR corpus and serialises it into a unified instruction format covering findings, impression, and VQA samples.
[Compact] Abnormality-guided prompting. Train a CheXpert-style classifier that emits a PNU abnormality string, and inject it into the language-model prompt as clinical guidance.
[Compact] Parameter-efficient two-stage training. Pre-align the visual projection with a contrastive objective, then instruction-tune the projection together with low-rank (QLoRA) adapters on the LLM, keeping the encoder and base LLM frozen so the model fits on a single consumer/cloud GPU.
[Compact] Evaluation. Measure the model with standard natural-language-generation, clinical-accuracy, and VQA metrics on a held-out, patient-disjoint test split, and position the results against representative prior work.
[Heading2] 1.3 Report structure
[FirstParagraph] The remainder of this report is organised as follows:
[Compact] Section 2 — Related works reviews prior work on medical report generation and on medical visual question answering, and situates the proposed model.
[Compact] Section 3 — Materials and Methods details the data preparation, the four model modules and the prompt format, the two-stage training schedule, and the evaluation protocol.
[Compact] Section 4 — Experiments and Results describes the experimental setup and hyperparameters, reports the results on report generation, VQA, and the abnormality classifier, compares them with the literature, and discusses them.
[Compact] Section 5 — Conclusion and future work summarises the contributions, limitations, and directions for further improvement.
[Heading1] 2. Related works
[Heading2] 2.1 Medical report generation
[FirstParagraph] Early radiology report generation systems adapted natural-image captioning architectures, pairing a CNN image encoder with an RNN/LSTM decoder. These models produced fluent sentences but struggled with the long, multi-sentence structure of radiology reports and with rare but clinically important findings. The introduction of R2Gen (Chen et al., 2020) replaced the recurrent decoder with a memory-driven transformer that records report patterns across studies, and R2GenCMN (Chen et al., 2021) added a shared cross-modal memory to better align visual and textual features; both became standard baselines on the MIMIC-CXR benchmark. A parallel line of work injected medical prior knowledge through knowledge graphs (e.g. PPKED, KGAE) or retrieval, and emphasised clinical-efficacy metrics — comparing the pathology labels extracted from generated and reference reports with the CheXbert labeler — rather than n-gram overlap alone.
[BodyText] More recently, the field has shifted to LLM-based generators. RaDialog (Pellegrini et al., 2023) instruction-tunes a Vicuna LLM on chest X-rays, conditions it on a structured list of CheXpert findings, and adapts it with low-rank (LoRA) adapters, enabling both report generation and conversational interaction. Related foundation-style efforts include CheXagent, MAIRA-2, LLaVA-Med, and XrayGPT. To improve clinical grounding, META-CXR introduces a U-MultiClass formulation that preserves three states per pathology — positive, negative, and uncertain — instead of collapsing them to a binary present/absent label.
[BodyText] Taken together, these advances — abnormality-aware generation, LLM-based decoding, and parameter-efficient adaptation — motivate the approach taken in this thesis: a single vision–language model that generates findings and impression and answers questions about a chest X-ray, explicitly guided by detected abnormalities. The components and training of this model are described in Section 3.
[Heading2] 2.2 Visual question answering
[FirstParagraph] Visual question answering combines image understanding with natural-language reasoning to answer a question about an image. In the medical domain, general benchmarks span radiology and pathology: VQA-RAD (Lau et al., Scientific Data, 2018) and SLAKE (Liu et al., IEEE ISBI, 2021) cover radiology images of several modalities and body regions — including, but not limited to, the chest — while PathVQA targets pathology microscopy rather than radiographs. Early methods on these benchmarks fuse a CNN image embedding with a question embedding and classify over a fixed answer vocabulary, often with co-attention or meta-learning of the visual features (e.g. MEVF, MICCAI 2019; MMQ, MICCAI 2021).
[BodyText] Dedicated chest-X-ray VQA resources are more recent. Medical-Diff-VQA / MIMIC-Diff-VQA (Hu et al., ACM KDD, 2023) poses difference questions over pairs of MIMIC-CXR images; MIMIC-CXR-VQA, introduced together with EHRXQA (Bae et al., NeurIPS Datasets & Benchmarks, 2023), pairs chest X-rays with structured electronic-health-record question answering; and the large-scale MIMIC-Ext-CXR-VQA used in this work organises questions by semantic type (verify, choose, query) and content type (presence, anatomy, attribute, size, and others).
[BodyText] Compared with chest-X-ray report generation, dedicated VQA for chest X-rays is a younger and less standardised area: most existing work contributes datasets or specialised settings (difference questions, EHR-linked QA) and still treats VQA as a standalone closed-set classification problem. This motivates the approach taken here: rather than building a separate VQA classifier, VQA is handled by the same encoder, projection, and language model used for report generation, with the question taking the place of the instruction and the same PNU abnormality guidance supplied in the prompt. A single set of weights therefore serves findings, impression, and VQA, and the abnormality signal benefits short-answer questions as well as long-form generation.
[Heading1] 3. Materials and Methods
[FirstParagraph] This chapter describes how the proposed Vision–Language Model (CXR-HIEU) is built and trained. It is organised in six parts. Section 3.1 gives an end-to-end overview. Section 3.2 defines the key concepts used throughout the chapter. Section 3.3 documents how the MIMIC-CXR corpus is analysed, filtered, and serialised into the training format. Section 3.4 describes the four modules of the model and the prompt that ties them together. Section 3.5 specifies the two-stage training schedule. Section 3.6 defines the evaluation protocol.
[Heading2] 3.1 Overview
[FirstParagraph] CXR-HIEU is a single vision–language network that solves three downstream tasks — findings generation, impression generation, and visual question answering (VQA) — through one shared backbone. It couples a frozen image encoder, a trainable projection, and a LoRA-adapted language model, with two design choices aimed at clinical reliability and at fitting a limited compute budget:
[Compact] a frozen 14-pathology CheXpert-style classifier whose predictions are serialised into the prompt as a Positive / Negative / Uncertain (PNU) string, and
[Compact] an optional contrastive Stage 1 that pre-aligns the projection in a joint image–text space without ever loading the language model.
[FirstParagraph] The full forward path can be summarised as:
[FirstParagraph] Only the MLP projection, the LoRA adapters on Vicuna, and (when enabled) the ITC head are trained. The image encoder, the CheXpert classifier, and the Vicuna base weights are kept frozen. In total, fewer than 0.3% of the parameters are trainable (≈ 21.5 M of ≈ 7.1 B).
[CaptionedFigure] [IMG] 
[ImageCaption] Figure 3.1. Overall workflow of CXR-HIEU: from data, through two-stage training, to inference and evaluation.
[Heading2] 3.2 Key Concepts and Definitions
[FirstParagraph] This section defines the core concepts that the rest of the chapter relies on.
Vision–Language Model (VLM). A model that takes an image together with a text instruction and produces a text response, by mapping visual features into the embedding space of a language model.
Chest X-ray report. A radiology report has two main free-text sections. Findings is the detailed, observation-by-observation description of the image; Impression is the short clinical summary written after the findings. We treat their generation as two separate tasks.
Visual Question Answering (VQA). Given an image and a natural-language question, the model returns a short answer (often a single word or phrase).
Large Language Model (LLM). A transformer trained on large text corpora to generate human-like text. We use Vicuna-7B, a LLaMA-derived instruction-tuned chat model, as the decoder.
Vision Transformer (ViT) and self-supervision. A ViT splits an image into fixed-size patches and processes them as a token sequence. RAD-DINO is a ViT-B/14 trained with DINOv2 self-supervision on chest X-rays; it requires no text labels.
Parameter-efficient fine-tuning (LoRA / QLoRA). Instead of updating all weights of the LLM, LoRA inserts small trainable low-rank matrices into selected layers and freezes the rest. QLoRA additionally keeps the base weights in 4-bit quantisation, which drastically reduces memory.
Image–Text Contrastive learning (ITC / InfoNCE). A training objective that pulls the embedding of an image and its matching text together while pushing non-matching pairs apart, using the symmetric InfoNCE loss. It is the mechanism behind Stage 1 alignment.
CheXpert labels and U-MultiClass. CheXpert defines 14 pathology categories. Rather than a binary present/absent label, U-MultiClass keeps three states per pathology — Positive, Negative, Uncertain — preserving the clinically important difference between a confident negative and a hedged one.
[Heading2] 3.3 Data Preparation
[FirstParagraph] The data pipeline runs in three phases, each placed where it is most efficient. (i) Selection runs locally on the MIMIC-CXR CSV metadata to decide which studies to keep. (ii) Image download runs on a cloud virtual machine (Google Colab): the images chosen in the manifest are downloaded from PhysioNet onto the VM’s disk and then pushed to a Hugging Face repository for reuse. This step is done in the cloud rather than on a local machine because the selected image set is too large for the local disk, and the cloud VM also has a faster, more stable connection to PhysioNet. (iii) Resize and re-shard runs once on a GPU host so that every training run consumes minimal-size JPEGs. The same subset is reused across all experiments, so the pipeline is run once — changing prompt templates or task weights only rebuilds the JSON, not the images.
[CaptionedFigure] [IMG] 
[ImageCaption] Figure 3.2. Three-phase data pipeline: selection (local) → image download (cloud VM → Hugging Face) → resize/shard → unified JSON.
[Heading3] 3.3.1 Sources and dataset overview
[FirstParagraph] All data come from the public PhysioNet distribution under credentialed access; no manual annotation is performed.
[Compact] Source
[Compact] Used for
[Compact] MIMIC-CXR Database
[Compact] Radiology report .txt files
[Compact] MIMIC-CXR-JPG
[Compact] Pre-converted JPEG images
[Compact] MIMIC-Ext-MIMIC-CXR-VQA
[Compact] (image, question, answer) triples
[Compact] mimic-cxr-2.0.0-split.csv
[Compact] Official patient-disjoint train/validate/test split
[Compact] mimic-cxr-2.0.0-metadata.csv
[Compact] ViewPosition (used for frontal selection)
[Compact] mimic-cxr-2.0.0-chexpert.csv
[Compact] 14 pathology labels per study
[BodyText] An exploratory analysis of the full corpus motivates the design choices below. MIMIC-CXR contains 377,110 images from 227,835 studies of 65,379 patients (on average 1.66 images per study and 3.48 studies per patient).
[Compact] Label imbalance (Figure 3.3). The 14 CheXpert labels are highly imbalanced: No Finding (33% positive), Support Devices (29%), Pleural Effusion (24%), and Lung Opacity (23%) dominate, while Fracture (1.9%) and Pleural Other (0.9%) are rare — which directly affects the abnormality classifier in Section 4.2.2.
[Compact] Views (Figure 3.4). Each study can contain several views. Frontal projections — AP (147,173) and PA (96,161) — together make up the majority; the rest are lateral (LATERAL 82,853, LL 35,133) or have no recorded ViewPosition (15,769).
[Compact] Images per study (Figure 3.5). Many studies contain two or more images (mean 1.66, up to 11), even though they share a single report.
[Compact] Report sections (Figure 3.6). Parsing all reports, a clean Findings section is present in 149,060 studies (65%) and an Impression in 186,865 (82%); many reports therefore lack one section. Findings are longer (median ≈ 45 words) than Impression (median ≈ 16 words).
[Compact] VQA (Figure 3.7). MIMIC-Ext-CXR-VQA provides 377,391 (image, question, answer) triples over these images, organised by semantic type (verify / choose / query) and content type (presence, anatomy, attribute, size, …).
[CaptionedFigure] [IMG] 
[ImageCaption] Figure 3.3. Distribution of the 14 CheXpert labels (Positive / Uncertain / Negative) over the full MIMIC-CXR — heavily imbalanced: No Finding and Support Devices dominate, while Fracture and Pleural Other are rare.
[CaptionedFigure] [IMG] 
[ImageCaption] Figure 3.4. View-position distribution: the two frontal views (AP, PA) form the majority; the rest are lateral (LATERAL, LL).
[CaptionedFigure] [IMG] 
[ImageCaption] Figure 3.5. Images per study (mean 1.66) — many studies have two or more images despite sharing a single report.
[CaptionedFigure] [IMG] 
[ImageCaption] Figure 3.6. Length (in words) of Findings vs Impression: the Impression is much shorter (median ≈ 16 words vs ≈ 45 for Findings).
[CaptionedFigure] [IMG] 
[ImageCaption] Figure 3.7. Distribution of VQA questions by semantic type and content type.
[Heading3] 3.3.2 Selecting the working subset
[FirstParagraph] We do not train on the full corpus. With no dedicated GPU available, training is done in the cloud, where compute is billed by the hour; repeatedly processing all ≈ 227k studies (≈ 377k images) would be both slow and expensive. A subset of 50,000 studies (40,000 train / 5,000 validation / 5,000 test) is large enough to train and evaluate the model while keeping cloud time and cost manageable, and — as Table 3.1 shows — it preserves the pathology distribution of the full dataset, so results on it remain representative.
[BodyText] A four-stage filter chain produces this subset:
[Compact] (a) One frontal image per study. Each DICOM is joined with the metadata CSV and only frontal views (ViewPosition ∈ {PA, AP}) are kept; if a study has several, it is collapsed to one image, preferring PA over AP. We use a single frontal image per study for two reasons. First, the frontal projection carries the most diagnostic information, and PA is the standard reference view (AP is reserved for bedside/portable exams), so PA is preferred when both exist. Second, we deliberately avoid multi-view training: besides its higher per-study compute, it is noisy here — a study has a single report that does not state which finding belongs to which view, so pairing several images with one report can mislead the model. Multi-view training is left as future work (Section 5).
[Compact] (b) Both report sections present. A strict regex accepts a section only if its header is exactly FINDINGS or IMPRESSION; synonyms (CONCLUSION, WET READ, composite headers) are not merged. A study survives only if both sections are present and non-empty, because the impression task later conditions on the ground-truth findings (Section 3.4.5).
[Compact] (c) Length-based outlier removal. Per-section word counts are computed; studies above Q3 + 1.5·IQR (multi-paragraph teaching reports) or below a small floor are dropped, trimming the long tail without shifting the median.
[Compact] (d) Stratified patient-disjoint sampling. Each study is assigned a stratum equal to its rarest positive CheXpert label; target counts are allocated per stratum proportionally to prevalence. The validation/test pools are filled first from the official split, any overflow drawn from train with the affected subjects removed from train, so the three sets are patient-disjoint.
[FirstParagraph] After steps (a)–(c) the eligible pool is 108,783 studies (from the 227,835 total), and step (d) samples the final 50,000.
[CaptionedFigure] [IMG] 
[ImageCaption] Figure 3.8. Four-stage filter chain reducing the MIMIC-CXR corpus to 50,000 studies.
[BodyText] Distribution preserved after selection. To confirm the subset is representative, Table 3.1 and Figure 3.9 compare the per-pathology positive rate of the full dataset, the eligible pool, and the final subset. The subset tracks the eligible pool almost exactly (largest gap 1.25 pp, on No Finding). The eligible pool itself differs from the raw corpus — mainly a higher No Finding rate and lower Support Devices / Pleural Effusion — which is expected, since requiring a clean Findings + Impression section and a frontal view removes many device-heavy ICU portables.
[BodyText] Table 3.1. CheXpert positive rate (%) at three stages of selection (largest 8 labels shown; |Δ| = subset − eligible).
[Compact] Pathology
[Compact] Full (%)
[Compact] Eligible (%)
[Compact] Subset (%)
[Compact] |Δ|
[Compact] No Finding
[Compact] 33.1
[Compact] 51.6
[Compact] 52.8
[Compact] 1.25
[Compact] Support Devices
[Compact] 29.2
[Compact] 8.7
[Compact] 8.5
[Compact] 0.24
[Compact] Pleural Effusion
[Compact] 23.8
[Compact] 12.3
[Compact] 11.6
[Compact] 0.65
[Compact] Lung Opacity
[Compact] 22.6
[Compact] 14.9
[Compact] 14.5
[Compact] 0.34
[Compact] Atelectasis
[Compact] 20.1
[Compact] 11.0
[Compact] 10.7
[Compact] 0.29
[Compact] Cardiomegaly
[Compact] 19.7
[Compact] 6.6
[Compact] 6.3
[Compact] 0.30
[Compact] Edema
[Compact] 11.9
[Compact] 8.0
[Compact] 7.6
[Compact] 0.46
[Compact] Pneumonia
[Compact] 7.3
[Compact] 5.6
[Compact] 5.6
[Compact] 0.01
[CaptionedFigure] [IMG] 
[ImageCaption] Figure 3.9. Positive rate of the 14 CheXpert labels at three stages — full dataset, eligible pool, and the 50k subset — the subset closely follows the original distribution.
[BodyText] The VQA distribution is likewise preserved: Figure 3.10 shows that the shares of question semantic types, content types, and answer types in the subset closely match the full dataset, so the VQA evaluation on the subset is not biased towards any question category.
[CaptionedFigure] [IMG] 
[ImageCaption] Figure 3.10. Distribution of VQA questions (semantic type, content type, answer type) — full dataset vs the 50k subset: the subset closely follows the original distribution.
[Heading3] 3.3.3 Storage and Consumption Layout
[FirstParagraph] For each split a manifest is emitted as JSON and CSV. Every row captures one image plus the standard PhysioNet identifiers, the 14 chex_* U-MultiClass label columns, and a has_vqa flag. The PhysioNet directory layout (files/pXX/pSUBJ/sSTUDY/<dicom>.jpg) is preserved so a partial download can be checked against the manifest by path lookup.
[BodyText] Because all training hosts consume the same images, we resize once offline. RAD-DINO centre-crops to 518×518 regardless of input, so the full-resolution JPEGs (~2–3 MP each, ~100 GB) are wasteful. Each image is resized so the shortest edge is 518, saved as JPEG quality 90, and packed into ~2 GB tar shards; the 50k subset compresses to ~5 GB.
[Heading3] 3.3.4 Training data format
[FirstParagraph] On the training host, a single builder turns the resized images, the manifests, and the VQA files into one JSON file — one entry per training sample — with a fixed schema:
[SourceCode] { image_path, task, target, question, structured_findings, split, study_id, subject_id }
[FirstParagraph] The identifier and path fields (image_path, study_id, subject_id) are self-explanatory; the remaining fields are:
[Compact] task — which task this sample belongs to: findings, impression, or vqa.
[Compact] target — the ground-truth text the model must produce (the Findings paragraph, the Impression, or the VQA answer).
[Compact] question — the natural-language question, for VQA samples only (null for findings/impression).
[Compact] structured_findings — the abnormality guidance prepended to the prompt: the PNU string for findings/VQA, or the ground-truth Findings text for impression samples.
[Compact] split — which partition the sample belongs to: train, validate, or test.
[FirstParagraph] Each selected study (one frontal image) yields up to three kinds of sample:
[Compact] a findings sample — target is the Findings paragraph, generated from the image and the abnormality (PNU) string;
[Compact] an impression sample — target is the Impression, generated from the image plus the study’s ground-truth Findings as context. The impression is thus produced as a short summary of the findings (a findings → impression cascade), which mirrors clinical practice, where the impression is written after the findings; this is also why step (b) requires a clean Findings section;
[Compact] VQA samples (when the study has associated questions) — target is the answer, with the question itself acting as the instruction.
[FirstParagraph] The 14 CheXpert labels are formatted into the PNU string (Section 3.4.3) and written into the structured_findings field at build time, so the trainer loads no labels at runtime. For the impression sample, this same field instead carries the ground-truth Findings text.
[BodyText] Because there is exactly one frontal image per study, the findings and impression tasks each contribute one sample per study (about 50,000 samples each). VQA, however, is larger — 127,010 question–answer samples in total (102,941 train / 12,189 validation / 11,880 test) — because a single image can be paired with several different questions in MIMIC-Ext-CXR-VQA.
[CaptionedFigure] [IMG] 
[ImageCaption] Figure 3.11. How one study (one frontal image) produces training samples: findings (image → findings), impression (image + GT findings → impression), and VQA.
[Heading2] 3.4 Model Architecture
[FirstParagraph] The model has four modules — image encoder, MLP projection, CheXpert classifier, and language model with LoRA — plus the prompt format that connects them. Each module is described below by its objective, inputs, outputs, and core logic.
[CaptionedFigure] [IMG] 
[ImageCaption] Figure 3.12. Overall architecture of CXR-HIEU (blue = frozen, orange = trainable).
[Heading3] 3.4.1 Image Encoder
[Compact] Objective. Turn a chest X-ray into a dense grid of patch features for the projection.
[Compact] Input. A 518×518 image (RAD-DINO’s native resolution, matching the offline resize target).
[Compact] Output. Patch features P ∈ R^{1369×768}; the [CLS] token is discarded.
[Compact] Core logic. The backbone is Microsoft’s RAD-DINO (ViT-B/14), self-supervised on ≈ 840k chest X-rays. It is loaded from the HuggingFace hub and kept entirely frozen. RAD-DINO is chosen because it ships as a standard HuggingFace model, is trained on a large amount of chest-X-ray data, and produces a patch grid dense enough to capture both global pathology and small focal abnormalities. Freezing is chosen for stability and memory: with the LLM held in 4-bit, gradient flow through an 86 M-parameter ViT would exhaust the activation budget of the GPUs available for this work.
[CaptionedFigure] [IMG] 
[ImageCaption] Figure 3.13. RAD-DINO architecture (ViT-B/14): a 518×518 image → 1369 patches → patch + position embeddings → 12 frozen Transformer Encoder blocks → [CLS] + patch tokens (768-d).
[Heading3] 3.4.2 MLP Projection
[Compact] Objective. Pool a long, variable-length patch sequence into a fixed visual-token budget, and bridge the vision space (768-d) to the LLM space (4096-d).
[Compact] Input. Patch features P ∈ R^{1369×768}.
[Compact] Output. 32 visual tokens V ∈ R^{32×4096}; a 1024-d intermediate is also exposed as a tap point for the ITC head.
[Compact] Core logic. A perceiver-style block with 32 learnable query tokens. The forward pass is:
[FirstParagraph] with Q₀ a learnable parameter and CrossAttn an 8-head attention block. We use 32 visual tokens: enough to retain spatial detail on small pathologies without inflating the LLM’s sequence length. The 1024-d intermediate H⁽¹⁾ is the grounding signal of the ITC head; it sits after the GELU so the contrastive objective sees an already-nonlinear representation.
[CaptionedFigure] [IMG] 
[ImageCaption] Figure 3.14. MLP Projection module: 32 queries → cross-attention → MLP (768 → 1024 → 4096), with the 1024-d branch feeding the ITC head.
[Heading3] 3.4.3 CheXpert Abnormality Classifier
[Compact] Objective. Provide explicit abnormality cues to the prompt as a readable 3-class string instead of a logit vector.
[Compact] Input. The global [CLS] embedding of RAD-DINO.
[Compact] Output. A 14×3 logit tensor → one of {Positive, Negative, Uncertain} per pathology → the PNU string.
[Compact] Core logic. A small MLP head on the frozen [CLS] embedding, predicting three states (positive / negative / uncertain) per pathology. The string injected into the prompt looks like:
[SourceCode] Positive Abnormalities: Cardiomegaly, Pleural EffusionNegative Abnormalities: No Finding, Edema, Pneumothorax, ...Uncertain Abnormalities: Atelectasis
[FirstParagraph] Three properties motivate this design. (1) The three-state form preserves the negative-vs-uncertain distinction, which binary CheXpert mappings destroy. (2) Expressing labels as text needs no architectural change when labels are missing — the field simply becomes empty and the prompt degrades gracefully. (3) Placing the PNU string between the visual tokens and the instruction lets self-attention route freely between text and image. The classifier is trained separately in Stage 0 and then frozen. During VLM training the ground-truth CSV labels populate the PNU string (oracle setting); at evaluation the classifier predicts its own PNU.
[CaptionedFigure] [IMG] 
[ImageCaption] Figure 3.15. From the RAD-DINO [CLS] embedding to the PNU string injected into the prompt.
[Heading3] 3.4.4 Language Model and Parameter-Efficient Adaptation
[Compact] Objective. Generate the findings / impression / answer text conditioned on visual tokens and the prompt.
[Compact] Input. The assembled token sequence with the <image> placeholder replaced by 32 visual tokens.
[Compact] Output. The autoregressive text response.
[Compact] Core logic. The decoder is Vicuna-7B v1.3, chosen for its clean USER: … ASSISTANT: … chat template (which simplifies label masking) and for offering strong instruction-following at a 7B size that fits the available hardware budget. It is loaded in 4-bit NF4 quantisation (double-quant, compute dtype BF16 on Ampere+ / FP16 on Turing), bringing the resident footprint from ≈ 14 GB down to ≈ 4 GB. Adaptation uses LoRA: rank-16 adapters on the four attention projections (q_proj, k_proj, v_proj, o_proj) of every block; the feed-forward sublayers are left untouched. With lora_alpha = 32 and lora_dropout = 0.05, the effective scaling is α/r = 2.
[CaptionedFigure] [IMG] 
[ImageCaption] Figure 3.16. Vicuna-7B architecture (32 decoder blocks, frozen 4-bit): masked self-attention + feed-forward MLP; LoRA (A→B, r=16) on q/k/v/o is the only trainable part.
[Heading3] 3.4.5 Prompt Assembly
[FirstParagraph] All three tasks share one prompt skeleton, following Vicuna’s v1.1 chat template:
[SourceCode] {SYSTEM_PROMPT} USER: <image>{PNU structured findings}{task-specific context block}{instruction} ASSISTANT: {target}
[FirstParagraph] The <image> placeholder is a special token (id 32000). At forward time the model finds this single token, replaces its embedding with the 32 visual tokens, and expands the attention mask, position ids, and label tensor by 31 positions so the causal mask stays consistent. Visual-token positions in the label tensor are set to −100 so they are excluded from the loss.
[BodyText] The task-specific context block differs per task:
[Compact] Findings — block empty; the findings are produced from the image plus the PNU labels.
[Compact] Impression — block is the literal Findings: <ground-truth findings>; the model conditions on the findings and summarises them.
[Compact] VQA — block empty; the question itself becomes the instruction.
[FirstParagraph] Findings and impression each have ten hand-written instruction paraphrases, sampled at training time; at evaluation the first variant is used deterministically. A sample is tokenised with cutoff_len = 512 and right-truncation (the response sits at the right end; left-truncation would destroy the system prompt and PNU block). The label tensor is masked with −100 on every prompt, padding, and visual token, so loss is computed strictly on the assistant response.
[CaptionedFigure] [IMG] 
[ImageCaption] Figure 3.17. Prompt structure and the mechanism that expands the <image> token into 32 visual tokens.
[Heading2] 3.5 Training Strategy
[FirstParagraph] Training uses a two-stage curriculum. Stage 1 first aligns the projection with the report text through explicit image–text contrastive learning; Stage 2 then instruction-tunes the projection together with the LLM’s LoRA adapters. The split follows a representation-then-instruction logic: it is wasteful to drive the LoRA adapters while the projection still emits ill-conditioned visual tokens, and the projection cannot be trained efficiently against the language-modelling loss without paying for a full Vicuna forward at every step. A Stage 0 classifier training precedes both.
[CaptionedFigure] [IMG] 
[ImageCaption] Figure 3.18. Two-stage training schedule (preceded by the Stage 0 classifier).
[Heading3] 3.5.1 Stage 0 — CheXpert Classifier Head
[FirstParagraph] The PNU classifier is fitted before Stages 1 and 2. It is a small MLP (768 → 256 → 14×3) on the frozen RAD-DINO [CLS] embedding, with a 3-class softmax (negative / positive / uncertain) per pathology. To counter the heavy class imbalance it is optimised with an asymmetric focal cross-entropy — no focal decay on the rare positive and uncertain states, strong decay (γ = 4) on the dominant negative — and the best epoch is selected by the macro-F1 of the positive class. Training takes minutes once the embeddings are cached; the checkpoint is then loaded read-only by both later stages.
[Heading3] 3.5.2 Stage 1 — Contrastive Alignment
[FirstParagraph] The goal is to specialise the projection (and only the projection) so its visual tokens are linearly aligned with the text representation of the matching report, before any language modelling.
[Compact] Image side. The 32 intermediate 1024-d tokens are mean-pooled, projected to 128-d, and L2-normalised by the ITC head.
[Compact] Text side. The canonical reference sentence per study (findings, falling back to impression) is encoded once, offline with microsoft/BiomedVLP-CXR-BERT-specialized into a 128-d L2-normalised vector. These are cached as {study_id → tensor[128]} and published to the data repo so any host can pull them in seconds.
[FirstParagraph] Stage 1 minimises the symmetric InfoNCE loss:
[FirstParagraph] Where: vᵢ is the image embedding from projection + ITC head, tᵢ the cached text embedding for the same study, and τ = 0.07 the temperature.
[BodyText] The dataset is de-duplicated to one image per study_id (the text embedding is study-level). Crucially, Stage 1 loads the model with load_llm = False — Vicuna is simply not instantiated. Freeing the ≈ 13 GB of Vicuna weights lifts the per-device batch from 8 (Stage 2 budget) to 64–96, which directly enlarges the InfoNCE negative pool. Stage 1 runs for 2 epochs at peak LR 1e-3 with a 5% cosine warm-up; the saved checkpoint is the projection-only state dict (the ITC head is discarded, as it has no role at generation time).
[CaptionedFigure] [IMG] 
[ImageCaption] Figure 3.19. Image–text alignment in Stage 1 (text embeddings precomputed offline, InfoNCE loss).
[Heading3] 3.5.3 Stage 2 — Instruction Tuning
[FirstParagraph] Stage 2 rebuilds the full model with load_llm = True, loads the Stage 1 projection weights, and switches to the autoregressive objective. The dataset returns mixed batches by task weight (findings 30% / impression 20% / VQA 50% for MIMIC-CXR; weights renormalise when VQA is absent). The loss is the standard causal cross-entropy:
[FirstParagraph] Where: T is the set of positions where the label is not −100 (assistant response only), V the 32 visual tokens, and c the textual prompt context.
[BodyText] Trainable parameters are the projection’s MLP and the LoRA adapters; the encoder and classifier stay frozen. Stage 2 runs for 10 epochs at peak LR 2e-4 with a 5% cosine warm-up; the effective batch size is fixed at 16 across hardware by trading per-device batch against gradient accumulation. The optimiser is AdamW with FP32 master weights for the projection and LoRA, the 4-bit base Vicuna acting as a quantised constant.
[Heading3] 3.5.4 Loss Masking and Image-Token Accounting
[FirstParagraph] The bookkeeping around the <image> placeholder deserves explicit mention. The tokenised prompt contains exactly one <image> token, replaced by 32 visual tokens at forward time. To keep the attention mask, position ids, and labels consistent, the forward pass expands all three by 31 entries at the placeholder: each visual-token mask entry is set to 1, position ids are made contiguous, and the visual span in the labels is filled with −100. The same expansion is applied at inference. This is the most error-prone part of the pipeline — an off-by-one silently shifts the labels and produces a degenerate loss curve — so an integration test asserts that the count of non-−100 label entries is preserved before and after expansion.
[CaptionedFigure] [IMG] 
[ImageCaption] Figure 3.20. Label masking — only the response span contributes to the loss.
[Heading2] 3.6 Evaluation Protocol
[FirstParagraph] Evaluation reflects the three downstream tasks. For findings and impression we report NLG metrics (lexical, fluency, semantic). Clinical correctness is measured separately, on the abnormality classifier, with per-pathology precision, recall, and F1. For VQA we report a short-answer suite.
[Compact] Family
[Compact] Metric
[Compact] Tasks
[Compact] Lexical n-gram
[Compact] BLEU-1, BLEU-4
[Compact] findings, impression, VQA
[Compact] ROUGE-1/2/L
[Compact] findings, impression
[Compact] Fluency / synonym
[Compact] METEOR
[Compact] findings, impression, VQA
[Compact] Semantic embedding
[Compact] BERTScore F1
[Compact] findings, impression, VQA
[Compact] Abnormality classification
[Compact] CheXpert classifier P / R / F1 (per pathology)
[Compact] image (14 labels)
[Compact] Exact answer
[Compact] Exact match, token F1
[Compact] VQA
[Heading3] 3.6.1 Tasks and Held-Out Data
[FirstParagraph] All metrics are computed on the patient-disjoint test split (5,000 studies). Inference uses greedy decoding (do_sample=False, num_beams=1) and the canonical instruction variant (index 0). For findings and impression the model receives the test image plus the PNU string predicted by the frozen classifier; the impression prompt additionally receives the ground-truth findings, matching the training setup. For VQA the question replaces the instruction slot. Maximum new tokens: 300 (findings), 200 (impression), 64 (VQA), matched to the 99th-percentile reference length.
[CaptionedFigure] [IMG] 
[ImageCaption] Figure 3.21. Per-task evaluation flow.
[Heading3] 3.6.2 Natural Language Generation Metrics
[Compact] BLEU. Corpus-level BLEU-1 and BLEU-4 with NLTK smoothing method 1. Reported for comparability with the RRG literature, but treated as a fluency floor — it correlates weakly with clinical correctness.
[Compact] ROUGE. ROUGE-1/2/L F-measures with Porter stemming. ROUGE-L is the most commonly reported single number for this task.
[Compact] METEOR. A weighted token-level F-measure that credits stems and WordNet synonyms (e.g. cardiomegaly ↔ enlarged heart), with a fragmentation penalty. Of the n-gram-style metrics it correlates best with human judgement on radiology.
[Compact] BERTScore. Greedy-aligned cosine similarity between contextual embeddings, aggregated to F1. Captures semantic equivalence the n-gram metrics miss, but has no notion of clinical correctness — it can reward a paraphrase that flips a finding’s polarity.
[Heading3] 3.6.3 Clinical Correctness: Abnormality Classification
[FirstParagraph] Clinical correctness is assessed through the CheXpert abnormality classifier (Stage 0), which predicts the 14 pathologies directly from the image. We report per-pathology precision, recall, and F1 for the positive class, together with their macro averages over the 14 labels, and the accuracy on the uncertain state. This measures how well the model detects abnormalities from the image itself; it is distinct from the report-derived clinical-efficacy F1 used by some report-generation work (running a labeler on the generated text), which we do not compute.
[Heading3] 3.6.4 Visual Question Answering Metrics
[FirstParagraph] VQA targets are short, so the suite differs from generation.
[Compact] Exact match. Lower-cased, punctuation-stripped, whitespace-collapsed string equality — the lower bound on correctness; harsh on phrasing but rewards the closed-form yes/no and quantitative questions that dominate the dataset.
[Compact] Token F1. F1 between the bags of normalised tokens — the most diagnostic single number for short-answer correctness.
[Compact] BLEU-1, METEOR, BERTScore. Reported for symmetry with the generation tasks. BLEU-4 and ROUGE-L are omitted (rarely meaningful / subsumed by token F1).
[Heading3] 3.6.5 Model Selection and Reporting
[FirstParagraph] Model selection uses eval_loss on the validation split (the same causal cross-entropy minimised in training); the best checkpoint by this criterion is evaluated on the test split. We deliberately do not select on downstream metrics, to avoid the optimistic bias of optimising the same signal that is later reported. Each run writes per-task predictions to results/{run_id}/predictions_{task}.json and an aggregated summary to results/{run_id}/metrics_summary.json.
[Heading1] 4. Experiments and Results
[Heading2] 4.1 Experiments
[Heading3] 4.1.1 Dataset and configuration
[FirstParagraph] All experiments use the 50,000-study subset described in Section 3.3: 40,000 train / 5,000 validation / 5,000 test, patient-disjoint, one frontal image per study, with the impression task conditioned on the ground-truth findings. Findings and impression targets come from the MIMIC-CXR reports; VQA samples come from MIMIC-Ext-CXR-VQA, attached by study. The reported run is evaluated on the held-out test split with predicted PNU guidance — the abnormality string comes from the frozen CheXpert classifier rather than from ground-truth labels, which is the realistic inference setting. The VQA data contain 127,010 question–answer items in total (102,941 train / 12,189 validation / 11,880 test); there are far more VQA samples than studies because a single image can be paired with several questions, whereas the findings and impression tasks contribute one sample per study. All metrics below are on the 11,880-item test set.
[Heading3] 4.1.2 Implementation and hyperparameters
[FirstParagraph] The model is implemented in PyTorch with HuggingFace transformers, peft (LoRA), and bitsandbytes (4-bit NF4 quantisation). Training runs on a single GPU through cloud notebooks, with the per-device batch size and gradient accumulation auto-tuned to the GPU so that the effective batch size stays fixed. The image encoder and the base LLM are frozen throughout; only the CheXpert head (Stage 0), the projection (Stages 1–2), the LoRA adapters (Stage 2), and the ITC head (Stage 1) ever receive gradients.
[BodyText] Stage 0 (CheXpert classifier). The abnormality head is a small MLP (768 → 256 → 14 × 3) trained on top of pre-computed, frozen RAD-DINO [CLS] embeddings, so the encoder is run only once and the head trains in minutes on a T4. To handle the heavy class imbalance it is optimised with an asymmetric focal cross-entropy (γ = 0 for the rare positive/uncertain states, γ = 4 for the dominant negative state) rather than inverse-frequency weighting, and the best epoch is selected by the macro-F1 of the positive class. Table 4.1 lists the hyperparameters of all three stages.
[BodyText] Table 4.1. Training hyperparameters for the three stages.
[Compact] Hyperparameter
[Compact] Stage 0 (classifier)
[Compact] Stage 1 (ITC)
[Compact] Stage 2
[Compact] Trainable modules
[Compact] CheXpert MLP head
[Compact] Projection + ITC head
[Compact] Projection + LoRA
[Compact] Frozen modules
[Compact] Encoder; LLM not loaded
[Compact] Encoder, classifier; LLM not loaded
[Compact] Encoder, classifier, base LLM
[Compact] Loss
[Compact] Asymmetric focal CE (γ 0 / 4)
[Compact] Symmetric InfoNCE
[Compact] Causal CE
[Compact] Epochs
[Compact] 10
[Compact] 3
[Compact] 5
[Compact] Peak learning rate
[Compact] 1e-3
[Compact] 1e-3
[Compact] 2e-4
[Compact] LR schedule
[Compact] cosine
[Compact] cosine, 5% warm-up
[Compact] cosine, 5% warm-up
[Compact] Weight decay
[Compact] 1e-4
[Compact] 0.01
[Compact] 0.01
[Compact] Mixed precision
[Compact] FP32 head (cached embeds)
[Compact] BF16 (Ampere+) / FP16 (T4)
[Compact] BF16 / FP16
[Compact] LLM quantisation
[Compact] —
[Compact] —
[Compact] 4-bit NF4, double-quant
[Compact] LoRA (r, α, dropout)
[Compact] —
[Compact] —
[Compact] (16, 32, 0.05)
[Compact] LoRA modules
[Compact] —
[Compact] —
[Compact] {q, k, v, o}_proj
[Compact] Optimiser
[Compact] AdamW
[Compact] AdamW
[Compact] AdamW
[Heading3] 4.1.3 Evaluation protocol
[FirstParagraph] Metrics follow Section 3.6. Generation uses greedy decoding with the canonical instruction variant; the maximum number of new tokens is 300 (findings), 200 (impression), and 64 (VQA). For report generation we report BLEU-1/4, ROUGE-1/2/L, METEOR, and BERTScore-F1; for VQA we report exact match, token-F1, micro-F1, BLEU-1, METEOR, and BERTScore-F1. Clinical correctness is assessed through the dedicated CheXpert abnormality classifier (Section 4.2.2), which predicts the 14 pathologies directly from the image and is scored with per-pathology precision, recall, and F1 (macro-averaged).
[Heading2] 4.2 Results
[Heading3] 4.2.1 Report generation: findings and impression
[FirstParagraph] Table 4.2 compares CXR-HIEU findings generation with representative MIMIC-CXR systems. The model is competitive across all standard metrics and sits in the mid-range of the comparison: BLEU-1 (0.364) is close to RaDialog (0.360) and KiUT (0.362); BLEU-4 (0.103) is on par with META-CXR (0.102) and above RaDialog (0.095); METEOR (0.161) is in line with KiUT and R2GenGPT (0.160) and just below META-CXR (0.173); and ROUGE-L (0.292) matches R2GenGPT (0.297) and exceeds RaDialog (0.271) and META-CXR (0.280). Only the largest systems (KARGEN, MAIRA-2) lead clearly. This places CXR-HIEU among recent LLM-based generators despite its much smaller, frozen-backbone design.
[BodyText] Table 4.2. Findings generation on MIMIC-CXR test. Baseline values as compiled in Edirisinghe et al. (2025), Table 2; “—” = not reported.
[Compact] Model
[Compact] BLEU-1
[Compact] BLEU-4
[Compact] METEOR
[Compact] ROUGE-L
[Compact] METransformer
[Compact] 0.379
[Compact] 0.124
[Compact] 0.152
[Compact] 0.291
[Compact] KiUT
[Compact] 0.362
[Compact] 0.113
[Compact] 0.160
[Compact] 0.285
[Compact] R2GenGPT
[Compact] 0.401
[Compact] 0.134
[Compact] 0.160
[Compact] 0.297
[Compact] RaDialog
[Compact] 0.360
[Compact] 0.095
[Compact] 0.140
[Compact] 0.271
[Compact] KARGEN
[Compact] 0.417
[Compact] 0.140
[Compact] 0.165
[Compact] 0.305
[Compact] MAIRA-2
[Compact] 0.460
[Compact] 0.231
[Compact] —
[Compact] 0.384
[Compact] META-CXR
[Compact] 0.390
[Compact] 0.102
[Compact] 0.173
[Compact] 0.280
[Compact] CXR-HIEU (ours)
[Compact] 0.364
[Compact] 0.103
[Compact] 0.161
[Compact] 0.292
[BodyText] The full breakdown for both report tasks is given in Table 4.3. Impression scores are markedly lower than findings on every metric, which is discussed in Section 4.3.
[BodyText] Table 4.3. CXR-HIEU report-generation results (test). BERTScore-F1 is DistilBERT-based (not comparable to other papers; see note above).
[Compact] Task
[Compact] BLEU-1
[Compact] BLEU-4
[Compact] ROUGE-1
[Compact] ROUGE-2
[Compact] ROUGE-L
[Compact] METEOR
[Compact] BERTScore-F1
[Compact] Findings
[Compact] 0.364
[Compact] 0.103
[Compact] 0.403
[Compact] 0.174
[Compact] 0.292
[Compact] 0.161
[Compact] 0.803
[Compact] Impression
[Compact] 0.100
[Compact] 0.013
[Compact] 0.136
[Compact] 0.036
[Compact] 0.118
[Compact] 0.104
[Compact] 0.720
[Heading3] 4.2.2 Clinical correctness: abnormality classification
[FirstParagraph] Clinical correctness is measured by the CheXpert abnormality classifier (Stage 0), which predicts the 14 pathologies — with an explicit uncertain state — directly from the image. This is the same role that META-CXR’s MHCAC classifier plays. Table 4.4 gives our per-pathology results. Performance is strongest on common, visually salient categories (No Finding F1 0.785, Pleural Effusion 0.622, Support Devices 0.552, Edema 0.508) and collapses to zero on rare categories with few positives (Enlarged Cardiomediastinum, Pleural Other, Fracture), reflecting the strong class imbalance of MIMIC-CXR. The macro averages over the 14 labels are precision 0.319, recall 0.377, F1 0.328; over the 11 labels the model actually predicts (excluding the three degenerate classes) the macro-F1 rises to about 0.42.
[BodyText] Table 4.4. CheXpert classifier per-pathology results (test split).
[Compact] Pathology
[Compact] Precision
[Compact] Recall
[Compact] F1
[Compact] No Finding
[Compact] 0.679
[Compact] 0.930
[Compact] 0.785
[Compact] Enlarged Cardiomediastinum
[Compact] 0.000
[Compact] 0.000
[Compact] 0.000
[Compact] Cardiomegaly
[Compact] 0.345
[Compact] 0.462
[Compact] 0.395
[Compact] Lung Opacity
[Compact] 0.317
[Compact] 0.648
[Compact] 0.425
[Compact] Lung Lesion
[Compact] 0.345
[Compact] 0.110
[Compact] 0.167
[Compact] Edema
[Compact] 0.447
[Compact] 0.587
[Compact] 0.508
[Compact] Consolidation
[Compact] 0.276
[Compact] 0.088
[Compact] 0.133
[Compact] Pneumonia
[Compact] 0.282
[Compact] 0.158
[Compact] 0.203
[Compact] Atelectasis
[Compact] 0.340
[Compact] 0.456
[Compact] 0.390
[Compact] Pneumothorax
[Compact] 0.438
[Compact] 0.382
[Compact] 0.408
[Compact] Pleural Effusion
[Compact] 0.545
[Compact] 0.724
[Compact] 0.622
[Compact] Pleural Other
[Compact] 0.000
[Compact] 0.000
[Compact] 0.000
[Compact] Fracture
[Compact] 0.000
[Compact] 0.000
[Compact] 0.000
[Compact] Support Devices
[Compact] 0.445
[Compact] 0.727
[Compact] 0.552
[Compact] Macro average (14)
[Compact] 0.319
[Compact] 0.377
[Compact] 0.328
[BodyText] Beyond the per-class precision/recall/F1, the uncertain state is almost never recovered: the classifier’s accuracy on uncertain-labelled cases is close to zero for most pathologies (non-zero only for Pneumonia ≈ 0.21, Edema ≈ 0.08, Atelectasis ≈ 0.02), so in practice it collapses the three-class problem onto positive vs negative. We do not benchmark these numbers against report-generation systems: most of the literature reports a report-derived clinical-efficacy F1 — running a labeler on the generated report text — which measures a different quantity from our image-level classifier and is therefore not directly comparable.
[Heading3] 4.2.3 Visual question answering
[FirstParagraph] Table 4.5 gives the overall VQA results on the 11,880-item test set. Tables 4.6 and 4.7 break the score down by answer type and by question semantic/content type. The model is clearly stronger on closed-ended questions (exact match 0.385) than on open-ended ones (0.204), and within content types it does best on size and plane questions and worst on anatomy and attribute questions.
[BodyText] Table 4.5. Overall VQA results (test, 11,880 items).
[Compact] Metric
[Compact] Score
[Compact] Set micro-F1
[Compact] 0.424
[Compact] Exact match
[Compact] 0.306
[Compact] Token-F1
[Compact] 0.321
[Compact] BLEU-1
[Compact] 0.101
[Compact] METEOR
[Compact] 0.179
[Compact] BERTScore-F1
[Compact] 0.435
[BodyText] Table 4.5. VQA by answer type and semantic type.
[Compact] Subset
[Compact] n
[Compact] Micro-F1
[Compact] Exact match
[Compact] Token-F1
[Compact] Closed
[Compact] 6,672
[Compact] 0.500
[Compact] 0.385
[Compact] 0.385
[Compact] Open
[Compact] 5,208
[Compact] 0.359
[Compact] 0.204
[Compact] 0.239
[Compact] Verify
[Compact] 6,672
[Compact] 0.500
[Compact] 0.385
[Compact] 0.385
[Compact] Choose
[Compact] 1,171
[Compact] 0.353
[Compact] 0.214
[Compact] 0.242
[Compact] Query
[Compact] 4,037
[Compact] 0.360
[Compact] 0.201
[Compact] 0.238
[BodyText] Table 4.5. VQA by content type.
[Compact] Content type
[Compact] n
[Compact] Micro-F1
[Compact] Exact match
[Compact] Token-F1
[Compact] Presence
[Compact] 4,514
[Compact] 0.486
[Compact] 0.367
[Compact] 0.367
[Compact] Abnormality
[Compact] 2,445
[Compact] 0.380
[Compact] 0.288
[Compact] 0.310
[Compact] Attribute
[Compact] 2,003
[Compact] 0.347
[Compact] 0.194
[Compact] 0.229
[Compact] Anatomy
[Compact] 1,574
[Compact] 0.386
[Compact] 0.189
[Compact] 0.227
[Compact] Size
[Compact] 659
[Compact] 0.569
[Compact] 0.516
[Compact] 0.516
[Compact] Gender
[Compact] 359
[Compact] 0.411
[Compact] 0.284
[Compact] 0.284
[Compact] Plane
[Compact] 326
[Compact] 0.601
[Compact] 0.439
[Compact] 0.439
[Heading2] 4.3 Discussion
[FirstParagraph] Report generation. On findings, CXR-HIEU is competitive with established systems across all standard metrics — BLEU-1 0.364, BLEU-4 0.103, METEOR 0.161, ROUGE-L 0.292 — sitting in the mid-range of Table 4.2, close to RaDialog, KiUT, and R2GenGPT, and behind only the largest models (KARGEN, MAIRA-2). This is a strong result given that only a small projection and LoRA adapters are trained on top of a frozen encoder and a frozen 7B LLM. We avoid any state-of-the-art claim on absolute scores, since BLEU/METEOR/BERTScore are sensitive to tokenisation and metric implementation and are only approximately comparable across papers.
[BodyText] Impression weakness. Impression scores are much lower than findings on every metric, even though the impression prompt is given the ground-truth findings and should make the task easier. Likely factors: impressions are short and abstractive (harsh for exact-overlap metrics), the impression task gets a smaller training share, and the model may carry findings-style phrasing into the impression. This is the clearest area for improvement (Section 5).
[BodyText] Clinical correctness. The Stage-0 CheXpert classifier — our clinical metric — reaches macro-F1 0.328 over 14 labels (≈ 0.42 over the 11 labels it can predict). The gap to a heavier system such as META-CXR’s multi-encoder classifier (weighted F1 0.73) is consistent with their design — three fused encoders (CNN, ViT, Swin), expert-token cross-attention, and class-balanced/contrastive losses — versus our single-encoder MLP head on top of frozen RAD-DINO features. Because the test-time PNU string is produced by this classifier (predicted, not oracle), its errors propagate into the prompt, so improving it is the highest-leverage next step.
[BodyText] VQA. The model handles closed-ended verification and presence questions far better than open-ended query and attribute questions, mirroring the general difficulty ordering in medical VQA. The strong results on size and plane questions suggest the visual tokens retain coarse geometric information well.
[BodyText] Comparison with prior work. CXR-HIEU’s findings metrics fall within the range reported by recent LLM-based chest-X-ray systems (e.g. RaDialog, R2GenGPT, KiUT, META-CXR), despite training only a small projection and LoRA adapters on a frozen backbone. The abnormality classifier is the weaker component: a single-encoder MLP head cannot match a heavier multi-encoder classifier such as META-CXR’s, whose authors themselves flag its inference-time complexity as a drawback. CXR-HIEU instead trades classifier accuracy for a far smaller and simpler model, consistent with the limited-compute goal of this thesis.
[BodyText] Limitations. Several limitations qualify these results. (i) The abnormality classifier is the weakest link. Three rare labels — Enlarged Cardiomediastinum, Pleural Other, and Fracture — score an exact zero on precision, recall, and F1: the head almost never fires on them because there are too few positive examples. The accuracy on the uncertain state is likewise very low, so the model effectively reduces the three-class problem to positive vs negative. Part of this is intrinsic rather than a modelling failure: a finding is labelled uncertain precisely because the reporting radiologist was not sure, and such judgements often rely on clinical context, prior studies, or experience — an expert may suspect a condition by inference — rather than on cues that are visible in the raw image alone, so an image-only classifier has little signal to learn from. (iii) Because the PNU string is predicted by this classifier, its errors propagate into generation and VQA. (iv) Like any LLM-based generator, the model can hallucinate fluent but unsupported statements that lexical metrics do not penalise. (v) Impression generation clearly underperforms findings. (vi) Compute constraints fix a small effective batch, a 7B LLM, and a single-view, single-image setup, with no multi-view or longitudinal (prior-study) context. Cross-paper metric comparison is also only approximate, given differing tokenisers and metric implementations.
[Heading1] 5. Conclusion and future work
[FirstParagraph] This thesis presented CXR-HIEU, a unified vision–language model that performs chest-X-ray findings generation, impression generation, and visual question answering with a single RAD-DINO + projection + Vicuna-7B (LoRA) backbone, guided by an explicit Positive/Negative/Uncertain abnormality signal and trained with a parameter-efficient two-stage schedule. On the MIMIC-CXR test split the model is competitive with established systems on the standard report-generation metrics (BLEU-1 0.364, BLEU-4 0.103, METEOR 0.161, ROUGE-L 0.292), its abnormality classifier reaches a macro-F1 of 0.328 (≈ 0.42 on the labels it can predict), and it achieves a VQA exact match of 0.306, markedly stronger on closed-ended questions. These results show that a frozen-encoder, frozen-LLM design adapted only through a small projection and LoRA adapters can drive three chest-X-ray tasks at once on modest hardware.
[BodyText] Several directions would strengthen the work:
[Compact] Stronger abnormality classifier. The clearest lever: adopt a META-CXR-style design (multi-encoder fusion, class-balancing or focal loss, contrastive/uncertainty objectives) so the predicted PNU string is more reliable and propagates fewer errors into generation and VQA.
[Compact] Clinical validation. Assess the generated reports with expert radiologists for factual correctness, beyond the lexical and classifier metrics used here.
[Compact] Impression generation. Investigate and close the gap on impression (dedicated decoding budget, task-specific tuning, or a true end-to-end findings→impression cascade).
[Compact] Scale and backbones. Train on more data and views (multi-image studies) 
[Heading1] References
E. Çallı, E. Sogancioglu, B. van Ginneken, K. G. van Leeuwen, K. Murphy. Deep Learning for Chest X-ray Analysis: A Survey. Medical Image Analysis, vol. 72, 2021.
D. Edirisinghe, W. Nimalsiri, M. Hennayake, D. Meedeniya, G. Lim. Chest X-Ray Report Generation Using Abnormality Guided Vision Language Model (META-CXR). IEEE Access, vol. 13, 2025.
C. Pellegrini, E. Özsoy, B. Busam, N. Navab, M. Keicher. RaDialog: A Large Vision-Language Model for Radiology Report Generation and Conversational Assistance. arXiv:2311.18681, 2023.
Z. Wang, L. Liu, L. Wang, L. Zhou. R2GenGPT: Radiology Report Generation with Frozen LLMs. Meta-Radiology, 2023.
Y. Li, Z. Wang, Y. Liu, L. Wang, L. Liu, L. Zhou. KARGEN: Knowledge-enhanced Automated Radiology Report Generation using Large Language Models. arXiv:2409.05370, 2024.
S. Bannur et al. MAIRA-2: Grounded Radiology Report Generation. arXiv:2406.04449, 2024.
Z. Wang, L. Liu, L. Wang, L. Zhou. METransformer: Radiology Report Generation by Transformer with Multiple Learnable Expert Tokens. CVPR, 2023.
Z. Huang, X. Zhang, S. Zhang. KiUT: Knowledge-injected U-transformer for Radiology Report Generation. CVPR, 2023.
F. Pérez-García et al. RAD-DINO: Exploring Scalable Medical Image Encoders Beyond Text Supervision. arXiv:2401.10815, 2024.
J. Li, D. Li, S. Savarese, S. Hoi. BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models. ICML, 2023.
W.-L. Chiang et al. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90% ChatGPT Quality. 2023.
E. J. Hu et al. LoRA: Low-Rank Adaptation of Large Language Models. ICLR, 2022.
T. Dettmers, A. Pagnoni, A. Holtzman, L. Zettlemoyer. QLoRA: Efficient Finetuning of Quantized LLMs. NeurIPS, 2023.
A. Smit et al. CheXbert: Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERT. EMNLP, 2020.
J. Irvin et al. CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison. AAAI, 2019.
A. E. W. Johnson et al. MIMIC-CXR-JPG, a Large Publicly Available Database of Labeled Chest Radiographs. arXiv:1901.07042, 2019.
T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, Y. Artzi. BERTScore: Evaluating Text Generation with BERT. ICLR, 2020.
S. Banerjee, A. Lavie. METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments. ACL Workshop, 2005.
J. J. Lau, S. Gayen, A. Ben Abacha, D. Demner-Fushman. A Dataset of Clinically Generated Visual Questions and Answers about Radiology Images (VQA-RAD). Scientific Data, 2018.
B. Liu, L.-M. Zhan, L. Xu, L. Ma, Y. Yang, X.-M. Wu. SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answering. IEEE ISBI, 2021.
X. Hu et al. Expert Knowledge-Aware Image Difference Graph Representation Learning for Difference-Aware Medical Visual Question Answering (Medical-Diff-VQA / MIMIC-Diff-VQA). ACM SIGKDD (KDD), 2023.
S. Bae et al. EHRXQA: A Multi-Modal Question Answering Dataset for Electronic Health Records with Chest X-ray Images (incl. MIMIC-CXR-VQA). NeurIPS Datasets & Benchmarks, 2023.