README.md

---
license: other
task_categories:
  - image-to-text
language:
  - en
tags:
  - medical
  - radiology
  - chest-x-ray
  - report-generation
  - reasoning
  - mimic-cxr
pretty_name: VReason MIMIC-CXR
size_categories:
  - 100K<n<1M
configs:
- config_name: default
  data_files:
  - split: train
    path: data/default/train-*.parquet
  - split: validation
    path: data/default/validation-*.parquet
  - split: test
    path: data/default/test-*.parquet
- config_name: train
  data_files:
  - split: train
    path: data/default/train-*.parquet
- config_name: validation
  data_files:
  - split: train
    path: data/default/validation-*.parquet
- config_name: test
  data_files:
  - split: train
    path: data/default/test-*.parquet
---

# VReason MIMIC-CXR

A chest radiograph report-generation dataset augmented with structured
visual reasoning traces and region-of-interest (ROI) crops.
Each example walks through the radiologist's interpretation workflow
section by section before producing the final report.

## Dataset at a glance

| Split | Examples |
|-------|--------:|
| train | 100,750 |
| validation | 777 |
| test | 1,138 |

## Source

Derived from [MIMIC-CXR](https://physionet.org/content/mimic-cxr/) (Johnson
et al., 2019). Anatomical and pathological ROI crops were automatically
generated from the accompanying radiology reports using a region-proposal
pipeline. Structured reasoning traces were synthesised to mirror the
step-by-step interpretation workflow of board-certified radiologists.

> **Access requirement:** MIMIC-CXR is a credentialed dataset on PhysioNet.
> You must complete the required training and sign the data-use agreement
> before using this dataset.

---

## Dataset structure

### Fields

| Field | Type | Description |
|-------|------|-------------|
| `messages` | `list[dict]` | Conversation turns: `[{"role": "user"/"assistant", "content": str}]`. `<image>` tokens in the content index into `images` in order across all turns. |
| `images` | `list[str]` | `data:image/jpeg;base64,...` encoded images. Order: full-size frontal CXR → lateral CXR (if present) → resized versions → anatomical-ROI crops → pathological-ROI crops. |
| `solution` | `str` | Full ground-truth assistant response (identical to the assistant message content). |

### Conversation format

**User turn** — contains `<image>` tokens for the input radiograph(s):

```
<image><image>Based on the provided chest radiographs, explain your
diagnosis procedure and write a report.
```

**Assistant turn** — structured reasoning chain followed by the report:

```
<interpret>
  Reviewing <anatomical section>...
  <tool type="anatomical_roi" label=[...]><image>
  Inspecting <finding region>...
  <tool type="pathological_roi" label=[...]><image>  <observation text>
  ...
</interpret>
<finding>
  - **Section**: <finding text>
  ...
</finding>
<impression>
  - <one-line clinical summary>
</impression>
<report>
  <free-text radiology report>
</report>
```

Each `<tool ...><image>` tag in the assistant turn consumes the next entry
from `images`, continuing the sequence after the user-turn images.

---

## Loading the dataset

```python
from datasets import load_dataset

ds = load_dataset("EvidenceAIResearch/MIMIC-CXR-VReason")

# Access splits
train = ds["train"]
val   = ds["validation"]
test  = ds["test"]
```

### Decoding images

Images are stored as `data:image/jpeg;base64,...` strings and can be decoded
with standard Python:

```python
import base64, io
from PIL import Image

example = train[0]
for img_b64 in example["images"]:
    _, data = img_b64.split(",", 1)
    img = Image.open(io.BytesIO(base64.b64decode(data)))
    img.show()
```

### Inspecting a single example

```python
example = train[0]

# The conversation (user prompt + structured assistant response)
for turn in example["messages"]:
    print(f"[{turn['role']}]")
    print(turn["content"][:200], "...")
    print()

# Number of images attached (full CXRs + all ROI crops)
print(f"Images: {len(example['images'])}")
```

---

## Citation

```bibtex
@unpublished{ye2026visual,
  title={Visual Reasoning Enables Evidence-Grounded Radiology {AI}},
  author={Ye, Shuchang and Robertson, Harry and Moghadam, Alireza
          and Shu, Matthew and Harb, Nathan and Li, Jennifer
          and Mogdil, Aadhar and Raythatha, Jineel and Shen, Yujia
          and Song, Xinyun and Tan, Xinchen and Fu, Xiaolong
          and Meng, Mingyuan and Bi, Lei and Yang, Jean YH
          and Kim, Jinman},
  year={2026},
}
```