docs/methodology_rewrite.md

# 3. Materials and Methods

This chapter describes how the proposed Vision–Language Model (CXR-HIEU) is built and trained. It is organised in six parts. Section 3.1 gives an end-to-end overview. Section 3.2 defines the key concepts used throughout the chapter. Section 3.3 documents how the MIMIC-CXR corpus is analysed, filtered, and serialised into the training format. Section 3.4 describes the four modules of the model and the prompt that ties them together. Section 3.5 specifies the two-stage training schedule. Section 3.6 defines the evaluation protocol.

---

## 3.1 Overview

CXR-HIEU is a single vision–language network that solves three downstream tasks — findings generation, impression generation, and visual question answering (VQA) — through one shared backbone. It couples a frozen image encoder, a trainable projection, and a LoRA-adapted language model, with two design choices aimed at clinical reliability and at fitting a limited compute budget:

- a frozen 14-pathology CheXpert-style classifier whose predictions are serialised into the prompt as a Positive / Negative / Uncertain (PNU) string, and
- an optional contrastive Stage 1 that pre-aligns the projection in a joint image–text space without ever loading the language model.

The full forward path can be summarised as:

$$
\mathbf{x}_{518\times518} \xrightarrow{\text{RAD-DINO}} \mathbf{P} \in \mathbb{R}^{B\times 1369\times 768} \xrightarrow{\text{MLP-Proj}} \mathbf{V} \in \mathbb{R}^{B\times 32\times 4096} \xrightarrow{\text{Vicuna-7B + LoRA}} \hat{\mathbf{y}}
$$

Only the MLP projection, the LoRA adapters on Vicuna, and (when enabled) the ITC head are trained. The image encoder, the CheXpert classifier, and the Vicuna base weights are kept frozen. In total, fewer than 0.3% of the parameters are trainable (≈ 21.5 M of ≈ 7.1 B).

![Figure 3.1. Overall workflow of CXR-HIEU: from data, through two-stage training, to inference and evaluation.](figures/fig15_workflow.png)

---

## 3.2 Key Concepts and Definitions

This section defines the core concepts that the rest of the chapter relies on.

- **Vision–Language Model (VLM).** A model that takes an image together with a text instruction and produces a text response, by mapping visual features into the embedding space of a language model.

- **Chest X-ray report.** A radiology report has two main free-text sections. Findings is the detailed, observation-by-observation description of the image; Impression is the short clinical summary written *after* the findings. We treat their generation as two separate tasks.

- **Visual Question Answering (VQA).** Given an image and a natural-language question, the model returns a short answer (often a single word or phrase).

- **Large Language Model (LLM).** A transformer trained on large text corpora to generate human-like text. We use Vicuna-7B, a LLaMA-derived instruction-tuned chat model, as the decoder.

- **Vision Transformer (ViT) and self-supervision.** A ViT splits an image into fixed-size patches and processes them as a token sequence. RAD-DINO is a ViT-B/14 trained with DINOv2 self-supervision on chest X-rays; it requires no text labels.

- **Parameter-efficient fine-tuning (LoRA / QLoRA).** Instead of updating all weights of the LLM, LoRA inserts small trainable low-rank matrices into selected layers and freezes the rest. QLoRA additionally keeps the base weights in 4-bit quantisation, which drastically reduces memory.

- **Image–Text Contrastive learning (ITC / InfoNCE).** A training objective that pulls the embedding of an image and its matching text together while pushing non-matching pairs apart, using the symmetric InfoNCE loss. It is the mechanism behind Stage 1 alignment.

- **CheXpert labels and U-MultiClass.** CheXpert defines 14 pathology categories. Rather than a binary present/absent label, U-MultiClass keeps three states per pathology — Positive, Negative, Uncertain — preserving the clinically important difference between a confident negative and a hedged one.

---

## 3.3 Data Preparation

The data pipeline runs in three phases, each placed where it is most efficient. (i) Selection runs locally on the MIMIC-CXR CSV metadata to decide which studies to keep. (ii) Image download runs on a cloud virtual machine (Google Colab): the images chosen in the manifest are downloaded from PhysioNet onto the VM's disk and then pushed to a Hugging Face repository for reuse. This step is done in the cloud rather than on a local machine because the selected image set is too large for the local disk, and the cloud VM also has a faster, more stable connection to PhysioNet. (iii) Resize and re-shard runs once on a GPU host so that every training run consumes minimal-size JPEGs. The same subset is reused across all experiments, so the pipeline is run once — changing prompt templates or task weights only rebuilds the JSON, not the images.

![Figure 3.2. Three-phase data pipeline: selection (local) → image download (cloud VM → Hugging Face) → resize/shard → unified JSON.](figures/fig02_data_pipeline.png)

### 3.3.1 Sources and dataset overview

All data come from the public PhysioNet distribution under credentialed access; no manual annotation is performed.

| Source | Version | Used for |
|---|---|---|
| MIMIC-CXR | 2.1.0 | Radiology report `.txt` files |
| MIMIC-CXR-JPG | 2.1.0 | Pre-converted JPEG images |
| MIMIC-Ext-CXR-VQA | 1.0.0 | (image, question, answer) triples |
| `mimic-cxr-2.0.0-split` | — | Official patient-disjoint train/validate/test split |
| `mimic-cxr-2.0.0-metadata` | — | `ViewPosition` (used for frontal selection) |
| `mimic-cxr-2.0.0-chexpert` | — | 14 pathology labels per study |

An exploratory analysis of the full corpus motivates the design choices below. MIMIC-CXR contains 377,110 images from 227,835 studies of 65,379 patients (on average 1.66 images per study and 3.48 studies per patient).

- **Label imbalance (Figure 3.3).** The 14 CheXpert labels are highly imbalanced: *No Finding* (33% positive), *Support Devices* (29%), *Pleural Effusion* (24%), and *Lung Opacity* (23%) dominate, while *Fracture* (1.9%) and *Pleural Other* (0.9%) are rare — which directly affects the abnormality classifier in Section 4.2.2.
- **Views (Figure 3.4).** Each study can contain several views. Frontal projections — AP (147,173) and PA (96,161) — together make up the majority; the rest are lateral (LATERAL 82,853, LL 35,133) or have no recorded `ViewPosition` (15,769).
- **Images per study (Figure 3.5).** Many studies contain two or more images (mean 1.66, up to 11), even though they share a single report.
- **Report sections (Figure 3.6).** Parsing all reports, a clean Findings section is present in 149,060 studies (65%) and an Impression in 186,865 (82%); many reports therefore lack one section. Findings are longer (median ≈ 45 words) than Impression (median ≈ 16 words).
- **VQA (Figure 3.7).** MIMIC-Ext-CXR-VQA provides 377,391 (image, question, answer) triples over these images, organised by semantic type (verify / choose / query) and content type (presence, anatomy, attribute, size, …).

![Figure 3.3. Distribution of the 14 CheXpert labels (Positive / Uncertain / Negative) over the full MIMIC-CXR — heavily imbalanced: No Finding and Support Devices dominate, while Fracture and Pleural Other are rare.](figures/eda_chexpert_labels.png)

![Figure 3.4. View-position distribution: the two frontal views (AP, PA) form the majority; the rest are lateral (LATERAL, LL).](figures/eda_views.png)

![Figure 3.5. Images per study (mean 1.66) — many studies have two or more images despite sharing a single report.](figures/eda_imgs_per_study.png)

![Figure 3.6. Length (in words) of Findings vs Impression: the Impression is much shorter (median ≈ 16 words vs ≈ 45 for Findings).](figures/eda_report_length.png)

![Figure 3.7. Distribution of VQA questions by semantic type and content type.](figures/eda_vqa_types.png)

### 3.3.2 Selecting the working subset

We do not train on the full corpus. With no dedicated GPU available, training is done in the cloud, where compute is billed by the hour; repeatedly processing all ≈ 227k studies (≈ 377k images) would be both slow and expensive. A subset of 50,000 studies (40,000 train / 5,000 validation / 5,000 test) is large enough to train and evaluate the model while keeping cloud time and cost manageable, and — as Table 3.1 shows — it preserves the pathology distribution of the full dataset, so results on it remain representative.

A four-stage filter chain produces this subset:

- **(a) One frontal image per study.** Each DICOM is joined with the metadata CSV and only frontal views (`ViewPosition ∈ {PA, AP}`) are kept; if a study has several, it is collapsed to one image, preferring PA over AP. We use a single frontal image per study for two reasons. First, the frontal projection carries the most diagnostic information, and PA is the standard reference view (AP is reserved for bedside/portable exams), so PA is preferred when both exist. Second, we deliberately avoid multi-view training: besides its higher per-study compute, it is noisy here — a study has a single report that does not state which finding belongs to which view, so pairing several images with one report can mislead the model. Multi-view training is left as future work (Section 5).
- **(b) Both report sections present.** A strict regex accepts a section only if its header is exactly `FINDINGS` or `IMPRESSION`; synonyms (`CONCLUSION`, `WET READ`, composite headers) are not merged. A study survives only if both sections are present and non-empty, because the impression task later conditions on the ground-truth findings (Section 3.4.5).
- **(c) Length-based outlier removal.** Per-section word counts are computed; studies above `Q3 + 1.5·IQR` (multi-paragraph teaching reports) or below a small floor are dropped, trimming the long tail without shifting the median.
- **(d) Stratified patient-disjoint sampling.** Each study is assigned a stratum equal to its rarest positive CheXpert label; target counts are allocated per stratum proportionally to prevalence. The validation/test pools are filled first from the official split, any overflow drawn from train with the affected subjects removed from train, so the three sets are patient-disjoint.

After steps (a)–(c) the eligible pool is 108,783 studies (from the 227,835 total), and step (d) samples the final 50,000.

![Figure 3.8. Four-stage filter chain reducing the MIMIC-CXR corpus to 50,000 studies.](figures/fig03_selection_funnel.png)

**Distribution preserved after selection.** To confirm the subset is representative, Table 3.1 and Figure 3.9 compare the per-pathology positive rate of the full dataset, the eligible pool, and the final subset. The subset tracks the eligible pool almost exactly (largest gap 1.25 pp, on *No Finding*). The eligible pool itself differs from the raw corpus — mainly a higher *No Finding* rate and lower *Support Devices* / *Pleural Effusion* — which is expected, since requiring a clean Findings + Impression section and a frontal view removes many device-heavy ICU portables.

**Table 3.1.** CheXpert positive rate (%) at three stages of selection (largest 8 labels shown; |Δ| = subset − eligible).

| Pathology | Full (%) | Eligible (%) | Subset (%) | \|Δ\| |
|---|---|---|---|---|
| No Finding | 33.1 | 51.6 | 52.8 | 1.25 |
| Support Devices | 29.2 | 8.7 | 8.5 | 0.24 |
| Pleural Effusion | 23.8 | 12.3 | 11.6 | 0.65 |
| Lung Opacity | 22.6 | 14.9 | 14.5 | 0.34 |
| Atelectasis | 20.1 | 11.0 | 10.7 | 0.29 |
| Cardiomegaly | 19.7 | 6.6 | 6.3 | 0.30 |
| Edema | 11.9 | 8.0 | 7.6 | 0.46 |
| Pneumonia | 7.3 | 5.6 | 5.6 | 0.01 |

![Figure 3.9. Positive rate of the 14 CheXpert labels at three stages — full dataset, eligible pool, and the 50k subset — the subset closely follows the original distribution.](figures/eda_prevalence_compare.png)

The VQA distribution is likewise preserved: Figure 3.10 shows that the shares of question semantic types, content types, and answer types in the subset closely match the full dataset, so the VQA evaluation on the subset is not biased towards any question category.

![Figure 3.10. Distribution of VQA questions (semantic type, content type, answer type) — full dataset vs the 50k subset: the subset closely follows the original distribution.](figures/eda_vqa_compare.png)

### 3.3.3 Storage and Consumption Layout

For each split a manifest is emitted as JSON and CSV. Every row captures one image plus the standard PhysioNet identifiers, the 14 `chex_*` U-MultiClass label columns, and a `has_vqa` flag. The PhysioNet directory layout (`files/pXX/pSUBJ/sSTUDY/<dicom>.jpg`) is preserved so a partial download can be checked against the manifest by path lookup.

Because all training hosts consume the same images, we resize once offline. RAD-DINO centre-crops to 518×518 regardless of input, so the full-resolution JPEGs (~2–3 MP each, ~100 GB) are wasteful. Each image is resized so the shortest edge is 518, saved as JPEG quality 90, and packed into ~2 GB tar shards; the 50k subset compresses to ~5–8 GB.

### 3.3.4 Training data format

On the training host, a single builder turns the resized images, the manifests, and the VQA files into one JSON file — one entry per training sample — with a fixed schema:

```
{ image_path, task, target, question, structured_findings, split, study_id, subject_id }
```

The identifier and path fields (`image_path`, `study_id`, `subject_id`) are self-explanatory; the remaining fields are:

- `task` — which task this sample belongs to: `findings`, `impression`, or `vqa`.
- `target` — the ground-truth text the model must produce (the Findings paragraph, the Impression, or the VQA answer).
- `question` — the natural-language question, for VQA samples only (`null` for findings/impression).
- `structured_findings` — the abnormality guidance prepended to the prompt: the PNU string for findings/VQA, or the ground-truth Findings text for impression samples.
- `split` — which partition the sample belongs to: `train`, `validate`, or `test`.

Each selected study (one frontal image) yields up to three kinds of sample:

- a findings sample — target is the Findings paragraph, generated from the image and the abnormality (PNU) string;
- an impression sample — target is the Impression, generated from the image plus the study's ground-truth Findings as context. The impression is thus produced as a short summary of the findings (a findings → impression cascade), which mirrors clinical practice, where the impression is written after the findings; this is also why step (b) requires a clean Findings section;
- **VQA** samples (when the study has associated questions) — target is the answer, with the question itself acting as the instruction.

The 14 CheXpert labels are formatted into the PNU string (Section 3.4.3) and written into the `structured_findings` field at build time, so the trainer loads no labels at runtime. For the impression sample, this same field instead carries the ground-truth Findings text.

Because there is exactly one frontal image per study, the findings and impression tasks each contribute one sample per study (about 50,000 samples each). VQA, however, is larger — 127,010 question–answer samples in total (102,941 train / 12,189 validation / 11,880 test) — because a single image can be paired with several different questions in MIMIC-Ext-CXR-VQA.

![Figure 3.11. How one study (one frontal image) produces training samples: findings (image → findings), impression (image + GT findings → impression), and VQA.](figures/fig04_sample_creation.png)

---

## 3.4 Model Architecture

The model has four modules — image encoder, MLP projection, CheXpert classifier, and language model with LoRA — plus the prompt format that connects them. Each module is described below by its objective, inputs, outputs, and core logic.

![Figure 3.12. Overall architecture of CXR-HIEU (blue = frozen, orange = trainable).](figures/fig01_pipeline.png)

### 3.4.1 Image Encoder

- **Objective.** Turn a chest X-ray into a dense grid of patch features for the projection.
- **Input.** A 518×518 image (RAD-DINO's native resolution, matching the offline resize target).
- **Output.** Patch features `P ∈ R^{1369×768}`; the `[CLS]` token is discarded.
- **Core logic.** The backbone is Microsoft's RAD-DINO (ViT-B/14), self-supervised on ≈ 840k chest X-rays. It is loaded from the HuggingFace hub and kept entirely frozen. RAD-DINO is chosen because it ships as a standard HuggingFace model, is trained on a large amount of chest-X-ray data, and produces a patch grid dense enough to capture both global pathology and small focal abnormalities. Freezing is chosen for stability and memory: with the LLM held in 4-bit, gradient flow through an 86 M-parameter ViT would exhaust the activation budget of the GPUs available for this work.

![Figure 3.13. RAD-DINO architecture (ViT-B/14): a 518×518 image → 1369 patches → patch + position embeddings → 12 frozen Transformer Encoder blocks → [CLS] + patch tokens (768-d).](figures/fig05_patchify.png)

### 3.4.2 MLP Projection

- **Objective.** Pool a long, variable-length patch sequence into a fixed visual-token budget, and bridge the vision space (768-d) to the LLM space (4096-d).
- **Input.** Patch features `P ∈ R^{1369×768}`.
- **Output.** 32 visual tokens `V ∈ R^{32×4096}`; a 1024-d intermediate is also exposed as a tap point for the ITC head.
- **Core logic.** A perceiver-style block with 32 learnable query tokens. The forward pass is:

$$
\mathbf{H}^{(0)} = \operatorname{CrossAttn}(\mathbf{Q}_0, \mathbf{P}, \mathbf{P}) \in \mathbb{R}^{32\times 768}
$$
$$
\mathbf{H}^{(1)} = \operatorname{Dropout}(\operatorname{GELU}(\mathbf{W}_1 \mathbf{H}^{(0)})) \in \mathbb{R}^{32\times 1024}
$$
$$
\mathbf{V} = \mathbf{W}_2 \mathbf{H}^{(1)} \in \mathbb{R}^{32\times 4096}
$$

with `Q₀` a learnable parameter and `CrossAttn` an 8-head attention block. We use 32 visual tokens: enough to retain spatial detail on small pathologies without inflating the LLM's sequence length. The 1024-d intermediate `H⁽¹⁾` is the grounding signal of the ITC head; it sits *after* the GELU so the contrastive objective sees an already-nonlinear representation.

![Figure 3.14. MLP Projection module: 32 queries → cross-attention → MLP (768 → 1024 → 4096), with the 1024-d branch feeding the ITC head.](figures/fig06_projection.png)

### 3.4.3 CheXpert Abnormality Classifier

- **Objective.** Provide explicit abnormality cues to the prompt as a readable 3-class string instead of a logit vector.
- **Input.** The global `[CLS]` embedding of RAD-DINO.
- **Output.** A 14×3 logit tensor → one of {Positive, Negative, Uncertain} per pathology → the PNU string.
- **Core logic.** A small MLP head on the frozen `[CLS]` embedding, predicting three states (positive / negative / uncertain) per pathology. The string injected into the prompt looks like:

```
Positive Abnormalities: Cardiomegaly, Pleural Effusion
Negative Abnormalities: No Finding, Edema, Pneumothorax, ...
Uncertain Abnormalities: Atelectasis
```

Three properties motivate this design. (1) The three-state form preserves the negative-vs-uncertain distinction, which binary CheXpert mappings destroy. (2) Expressing labels as text needs no architectural change when labels are missing — the field simply becomes empty and the prompt degrades gracefully. (3) Placing the PNU string between the visual tokens and the instruction lets self-attention route freely between text and image. The classifier is trained separately in Stage 0 and then frozen. During VLM training the ground-truth CSV labels populate the PNU string (oracle setting); at evaluation the classifier predicts its own PNU.

![Figure 3.15. From the RAD-DINO [CLS] embedding to the PNU string injected into the prompt.](figures/fig07_chexpert_pnu.png)

### 3.4.4 Language Model and Parameter-Efficient Adaptation

- **Objective.** Generate the findings / impression / answer text conditioned on visual tokens and the prompt.
- **Input.** The assembled token sequence with the `<image>` placeholder replaced by 32 visual tokens.
- **Output.** The autoregressive text response.
- **Core logic.** The decoder is Vicuna-7B v1.3, chosen for its clean `USER: … ASSISTANT: …` chat template (which simplifies label masking) and for offering strong instruction-following at a 7B size that fits the available hardware budget. It is loaded in 4-bit NF4 quantisation (double-quant, compute dtype BF16 on Ampere+ / FP16 on Turing), bringing the resident footprint from ≈ 14 GB down to ≈ 4 GB. Adaptation uses LoRA: rank-16 adapters on the four attention projections (`q_proj`, `k_proj`, `v_proj`, `o_proj`) of every block; the feed-forward sublayers are left untouched. With `lora_alpha = 32` and `lora_dropout = 0.05`, the effective scaling is `α/r = 2`.

![Figure 3.16. Vicuna-7B architecture (32 decoder blocks, frozen 4-bit): masked self-attention + feed-forward MLP; LoRA (A→B, r=16) on q/k/v/o is the only trainable part.](figures/fig08_lora.png)

### 3.4.5 Prompt Assembly

All three tasks share one prompt skeleton, following Vicuna's v1.1 chat template:

```
{SYSTEM_PROMPT} USER: <image>
{PNU structured findings}
{task-specific context block}
{instruction} ASSISTANT: {target}
```

The `<image>` placeholder is a special token (id 32000). At forward time the model finds this single token, replaces its embedding with the 32 visual tokens, and expands the attention mask, position ids, and label tensor by 31 positions so the causal mask stays consistent. Visual-token positions in the label tensor are set to −100 so they are excluded from the loss.

The task-specific context block differs per task:

- **Findings** — block empty; the findings are produced from the image plus the PNU labels.
- **Impression** — block is the literal `Findings: <ground-truth findings>`; the model conditions on the findings and summarises them.
- **VQA** — block empty; the question itself becomes the instruction.

Findings and impression each have ten hand-written instruction paraphrases, sampled at training time; at evaluation the first variant is used deterministically. A sample is tokenised with `cutoff_len = 512` and right-truncation (the response sits at the right end; left-truncation would destroy the system prompt and PNU block). The label tensor is masked with −100 on every prompt, padding, and visual token, so loss is computed strictly on the assistant response.

![Figure 3.17. Prompt structure and the mechanism that expands the `<image>` token into 32 visual tokens.](figures/fig09_prompt.png)

---

## 3.5 Training Strategy

Training uses a two-stage curriculum. Stage 1 first aligns the projection with the report text through explicit image–text contrastive learning; Stage 2 then instruction-tunes the projection together with the LLM's LoRA adapters. The split follows a representation-then-instruction logic: it is wasteful to drive the LoRA adapters while the projection still emits ill-conditioned visual tokens, and the projection cannot be trained efficiently against the language-modelling loss without paying for a full Vicuna forward at every step. A Stage 0 classifier training precedes both.

![Figure 3.18. Two-stage training schedule (preceded by the Stage 0 classifier).](figures/fig10_curriculum.png)

### 3.5.1 Stage 0 — CheXpert Classifier Head

The PNU classifier is fitted before Stages 1 and 2. It is a small MLP (768 → 256 → 14×3) on the frozen RAD-DINO `[CLS]` embedding, with a 3-class softmax (negative / positive / uncertain) per pathology. To counter the heavy class imbalance it is optimised with an asymmetric focal cross-entropy — no focal decay on the rare positive and uncertain states, strong decay (γ = 4) on the dominant negative — and the best epoch is selected by the macro-F1 of the positive class. Training takes minutes once the embeddings are cached; the checkpoint is then loaded read-only by both later stages.

### 3.5.2 Stage 1 — Contrastive Alignment

The goal is to specialise the projection (and only the projection) so its visual tokens are linearly aligned with the text representation of the matching report, before any language modelling.

- **Image side.** The 32 intermediate 1024-d tokens are mean-pooled, projected to 128-d, and L2-normalised by the ITC head.
- **Text side.** The canonical reference sentence per study (findings, falling back to impression) is encoded once, offline with `microsoft/BiomedVLP-CXR-BERT-specialized` into a 128-d L2-normalised vector. These are cached as `{study_id → tensor[128]}` and published to the data repo so any host can pull them in seconds.

Stage 1 minimises the symmetric InfoNCE loss:

$$
\mathcal{L}_{\text{ITC}} = -\tfrac{1}{2}\Big[ \sum_{i}\log\frac{\exp(\mathbf{v}_i^\top\mathbf{t}_i/\tau)}{\sum_j \exp(\mathbf{v}_i^\top\mathbf{t}_j/\tau)} + \sum_{i}\log\frac{\exp(\mathbf{t}_i^\top\mathbf{v}_i/\tau)}{\sum_j \exp(\mathbf{t}_i^\top\mathbf{v}_j/\tau)} \Big]
$$

**Where:** `vᵢ` is the image embedding from projection + ITC head, `tᵢ` the cached text embedding for the same study, and `τ = 0.07` the temperature.

The dataset is de-duplicated to one image per `study_id` (the text embedding is study-level). Crucially, Stage 1 loads the model with `load_llm = False` — Vicuna is simply not instantiated. Freeing the ≈ 13 GB of Vicuna weights lifts the per-device batch from 8 (Stage 2 budget) to 64–96, which directly enlarges the InfoNCE negative pool. Stage 1 runs for 2 epochs at peak LR `1e-3` with a 5% cosine warm-up; the saved checkpoint is the projection-only state dict (the ITC head is discarded, as it has no role at generation time).

![Figure 3.19. Image–text alignment in Stage 1 (text embeddings precomputed offline, InfoNCE loss).](figures/fig11_contrastive.png)

### 3.5.3 Stage 2 — Instruction Tuning

Stage 2 rebuilds the full model with `load_llm = True`, loads the Stage 1 projection weights, and switches to the autoregressive objective. The dataset returns mixed batches by task weight (findings 30% / impression 20% / VQA 50% for MIMIC-CXR; weights renormalise when VQA is absent). The loss is the standard causal cross-entropy:

$$
\mathcal{L}_{\text{LM}} = -\frac{1}{|\mathcal{T}|}\sum_{(t,y)\in\mathcal{T}} \log p_\theta(y_t \mid y_{<t}, \mathbf{V}, \mathbf{c})
$$

**Where:** `T` is the set of positions where the label is not −100 (assistant response only), `V` the 32 visual tokens, and `c` the textual prompt context.

Trainable parameters are the projection's MLP and the LoRA adapters; the encoder and classifier stay frozen. Stage 2 runs for 10 epochs at peak LR `2e-4` with a 5% cosine warm-up; the effective batch size is fixed at 16 across hardware by trading per-device batch against gradient accumulation. The optimiser is AdamW with FP32 master weights for the projection and LoRA, the 4-bit base Vicuna acting as a quantised constant.

### 3.5.4 Loss Masking and Image-Token Accounting

The bookkeeping around the `<image>` placeholder deserves explicit mention. The tokenised prompt contains exactly one `<image>` token, replaced by 32 visual tokens at forward time. To keep the attention mask, position ids, and labels consistent, the forward pass expands all three by 31 entries at the placeholder: each visual-token mask entry is set to 1, position ids are made contiguous, and the visual span in the labels is filled with −100. The same expansion is applied at inference. This is the most error-prone part of the pipeline — an off-by-one silently shifts the labels and produces a degenerate loss curve — so an integration test asserts that the count of non-−100 label entries is preserved before and after expansion.

![Figure 3.20. Label masking — only the response span contributes to the loss.](figures/fig12_loss_mask.png)

---

## 3.6 Evaluation Protocol

Evaluation reflects the three downstream tasks. For findings and impression we report NLG metrics (lexical, fluency, semantic). Clinical correctness is measured separately, on the abnormality classifier, with per-pathology precision, recall, and F1. For VQA we report a short-answer suite.

| Family | Metric | Tasks |
|---|---|---|
| Lexical n-gram | BLEU-1, BLEU-4 | findings, impression, VQA |
| | ROUGE-1/2/L | findings, impression |
| Fluency / synonym | METEOR | findings, impression, VQA |
| Semantic embedding | BERTScore F1 | findings, impression, VQA |
| Abnormality classification | CheXpert classifier P / R / F1 (per pathology) | image (14 labels) |
| Exact answer | Exact match, token F1 | VQA |

### 3.6.1 Tasks and Held-Out Data

All metrics are computed on the patient-disjoint test split (5,000 studies). Inference uses greedy decoding (`do_sample=False`, `num_beams=1`) and the canonical instruction variant (index 0). For findings and impression the model receives the test image plus the PNU string predicted by the frozen classifier; the impression prompt additionally receives the ground-truth findings, matching the training setup. For VQA the question replaces the instruction slot. Maximum new tokens: 300 (findings), 200 (impression), 64 (VQA), matched to the 99th-percentile reference length.

![Figure 3.21. Per-task evaluation flow.](figures/fig13_eval_flow.png)

### 3.6.2 Natural Language Generation Metrics

- **BLEU.** Corpus-level BLEU-1 and BLEU-4 with NLTK smoothing method 1. Reported for comparability with the RRG literature, but treated as a fluency floor — it correlates weakly with clinical correctness.
- **ROUGE.** ROUGE-1/2/L F-measures with Porter stemming. ROUGE-L is the most commonly reported single number for this task.
- **METEOR.** A weighted token-level F-measure that credits stems and WordNet synonyms (e.g. *cardiomegaly* ↔ *enlarged heart*), with a fragmentation penalty. Of the n-gram-style metrics it correlates best with human judgement on radiology.
- **BERTScore.** Greedy-aligned cosine similarity between contextual embeddings, aggregated to F1. Captures semantic equivalence the n-gram metrics miss, but has no notion of clinical correctness — it can reward a paraphrase that flips a finding's polarity.

### 3.6.3 Clinical Correctness: Abnormality Classification

Clinical correctness is assessed through the CheXpert abnormality classifier (Stage 0), which predicts the 14 pathologies directly from the image. We report per-pathology precision, recall, and F1 for the positive class, together with their macro averages over the 14 labels, and the accuracy on the *uncertain* state. This measures how well the model detects abnormalities from the image itself; it is distinct from the report-derived clinical-efficacy F1 used by some report-generation work (running a labeler on the generated text), which we do not compute.

### 3.6.4 Visual Question Answering Metrics

VQA targets are short, so the suite differs from generation.

- **Exact match.** Lower-cased, punctuation-stripped, whitespace-collapsed string equality — the lower bound on correctness; harsh on phrasing but rewards the closed-form yes/no and quantitative questions that dominate the dataset.
- **Token F1.** F1 between the bags of normalised tokens — the most diagnostic single number for short-answer correctness.
- **BLEU-1, METEOR, BERTScore.** Reported for symmetry with the generation tasks. BLEU-4 and ROUGE-L are omitted (rarely meaningful / subsumed by token F1).

### 3.6.5 Model Selection and Reporting

Model selection uses `eval_loss` on the validation split (the same causal cross-entropy minimised in training); the best checkpoint by this criterion is evaluated on the test split. We deliberately do not select on downstream metrics, to avoid the optimistic bias of optimising the same signal that is later reported. Each run writes per-task predictions to `results/{run_id}/predictions_{task}.json` and an aggregated summary to `results/{run_id}/metrics_summary.json`.