Hugging Face's logo

hieu3636
/
cxr-vlm-code

Model card Files
xet
 Community
cxr-vlm-code / docs /report_part1.md
convitom
Stop tracking docs
a20ed62
preview code
|
Raw
History Blame Contribute Delete
7.95 kB

1. Introduction

1.1 Context and Motivation

The chest X-ray (CXR) is the most frequently performed medical imaging examination in the world, with over two billion procedures carried out every year for the screening and diagnosis of cardiothoracic conditions [1]. Each examination must be read by a radiologist, who then writes a structured free-text report — typically a detailed Findings section followed by a concise Impression. As imaging volume keeps growing while the number of trained radiologists remains limited, report turnaround time, reader fatigue, and inter-reader variability have become real bottlenecks in clinical workflows.

Automatic radiology report generation (RRG) aims to ease this burden by drafting a report directly from the image, which the radiologist can then verify and edit. The field has evolved quickly: from early CNN–RNN captioning models, to transformer-based architectures, and most recently to large vision–language models (VLMs) such as LLaVA and BLIP-2, whose medical adaptations (RaDialog, LLaVA-Med, CheXagent, MAIRA-2) couple a frozen image encoder with a large language model (LLM) to produce fluent, instruction-following text. In parallel, clinicians often need to ask focused questions about an image — "is there a pleural effusion?", "what is the size of the cardiac silhouette?" — which motivates visual question answering (VQA) on chest X-rays.

A recurring difficulty is that fluent text is not necessarily clinically correct: a model can produce a well-formed report that misses or flips a finding. A natural way to mitigate this is to feed the language model explicit pathology labels, so that generation stays anchored to detected abnormalities rather than to language priors alone. Motivated by this idea, this thesis builds a single VLM that performs findings generation, impression generation, and VQA, guided by an explicit 14-pathology Positive / Negative / Uncertain (PNU) signal.

1.2 Internship objectives

The objective of this internship is to design, implement, and evaluate CXR-HIEU, a unified vision–language model for chest X-ray interpretation built on a shared RAD-DINO + projection + Vicuna-7B (LoRA) backbone. Concretely, the work targets the following goals:

  • Data pipeline. Build a reproducible pipeline that filters the MIMIC-CXR corpus and serialises it into a unified instruction format covering findings, impression, and VQA samples.
  • Abnormality-guided prompting. Train a CheXpert-style classifier that emits a PNU abnormality string, and inject it into the language-model prompt as clinical guidance.
  • Parameter-efficient two-stage training. Pre-align the visual projection with a contrastive objective, then instruction-tune the projection together with low-rank (QLoRA) adapters on the LLM, keeping the encoder and base LLM frozen so the model fits on a single consumer/cloud GPU.
  • Evaluation. Measure the model with standard natural-language-generation, clinical-accuracy, and VQA metrics on a held-out, patient-disjoint test split, and position the results against representative prior work.

1.3 Report structure

The remainder of this report is organised as follows:

  • Section 2 — Related works reviews prior work on medical report generation and on medical visual question answering, and situates the proposed model.
  • Section 3 — Materials and Methods details the data preparation, the four model modules and the prompt format, the two-stage training schedule, and the evaluation protocol.
  • Section 4 — Experiments and Results describes the experimental setup and hyperparameters, reports the results on report generation, VQA, and the abnormality classifier, compares them with the literature, and discusses them.
  • Section 5 — Conclusion and future work summarises the contributions, limitations, and directions for further improvement.

2. Related works

2.1 Medical report generation

Early radiology report generation systems adapted natural-image captioning architectures, pairing a CNN image encoder with an RNN/LSTM decoder. These models produced fluent sentences but struggled with the long, multi-sentence structure of radiology reports and with rare but clinically important findings. The introduction of R2Gen (Chen et al., 2020) replaced the recurrent decoder with a memory-driven transformer that records report patterns across studies, and R2GenCMN (Chen et al., 2021) added a shared cross-modal memory to better align visual and textual features; both became standard baselines on the MIMIC-CXR benchmark. A parallel line of work injected medical prior knowledge through knowledge graphs (e.g. PPKED, KGAE) or retrieval, and emphasised clinical-efficacy metrics — comparing the pathology labels extracted from generated and reference reports with the CheXbert labeler — rather than n-gram overlap alone.

More recently, the field has shifted to LLM-based generators. RaDialog (Pellegrini et al., 2023) instruction-tunes a Vicuna LLM on chest X-rays, conditions it on a structured list of CheXpert findings, and adapts it with low-rank (LoRA) adapters, enabling both report generation and conversational interaction. Related foundation-style efforts include CheXagent, MAIRA-2, LLaVA-Med, and XrayGPT. To improve clinical grounding, META-CXR introduces a U-MultiClass formulation that preserves three states per pathology — positive, negative, and uncertain — instead of collapsing them to a binary present/absent label.

Taken together, these advances — abnormality-aware generation, LLM-based decoding, and parameter-efficient adaptation — motivate the approach taken in this thesis: a single vision–language model that generates findings and impression and answers questions about a chest X-ray, explicitly guided by detected abnormalities. The components and training of this model are described in Section 3.

2.2 Visual question answering

Visual question answering combines image understanding with natural-language reasoning to answer a question about an image. In the medical domain, general benchmarks span radiology and pathology: VQA-RAD (Lau et al., Scientific Data, 2018) and SLAKE (Liu et al., IEEE ISBI, 2021) cover radiology images of several modalities and body regions — including, but not limited to, the chest — while PathVQA targets pathology microscopy rather than radiographs. Early methods on these benchmarks fuse a CNN image embedding with a question embedding and classify over a fixed answer vocabulary, often with co-attention or meta-learning of the visual features (e.g. MEVF, MICCAI 2019; MMQ, MICCAI 2021).

Dedicated chest-X-ray VQA resources are more recent. Medical-Diff-VQA / MIMIC-Diff-VQA (Hu et al., ACM KDD, 2023) poses difference questions over pairs of MIMIC-CXR images; MIMIC-CXR-VQA, introduced together with EHRXQA (Bae et al., NeurIPS Datasets & Benchmarks, 2023), pairs chest X-rays with structured electronic-health-record question answering; and the large-scale MIMIC-Ext-CXR-VQA used in this work organises questions by semantic type (verify, choose, query) and content type (presence, anatomy, attribute, size, and others).

Compared with chest-X-ray report generation, dedicated VQA for chest X-rays is a younger and less standardised area: most existing work contributes datasets or specialised settings (difference questions, EHR-linked QA) and still treats VQA as a standalone closed-set classification problem. This motivates the approach taken here: rather than building a separate VQA classifier, VQA is handled by the same encoder, projection, and language model used for report generation, with the question taking the place of the instruction and the same PNU abnormality guidance supplied in the prompt. A single set of weights therefore serves findings, impression, and VQA, and the abnormality signal benefits short-answer questions as well as long-form generation.