README.md

---
language:
- en
license: apache-2.0
library_name: transformers
tags:
- chest X-ray report generation
- radiology report generation
- image captioning
- chest X-ray
- X-ray
- radiology
- cxrmate
- cxrmate-2
- cxrmate-ed
- cxrmate-rrg24
- report
- radiology report
- multimodal
- patient data
- mimic-cxr
---

# CXRMate-2 — Chest X-ray Radiology Report Generation

*From:* **Toward Clinically Acceptable Chest X-ray Report Generation: A Retrospective Pilot Study**

<!-- CXRMate-2 is presented in the following Medical Image Analysis paper: -->
<!-- ```
``` -->
The pre-print for CXRMate-2 is available at:
```
Coming soon...
```

Abstract from the paper:
```
Coming soon...
```


## Load CXRMate-2:
```python
alias = 'aehrc/cxrmate-2'

model = transformers.AutoModelForCausalLM.from_pretrained(alias, trust_remote_code=True).to(device='cuda')
model.eval()

generation_config = transformers.GenerationConfig.from_pretrained(alias, trust_remote_code=True)

processor = transformers.AutoProcessor.from_pretrained(alias, trust_remote_code=True)
```

## Generate from URL:
```python
url = 'https://prod-images-static.radiopaedia.org/images/220869/76052f7902246ff862f52f5d3cd9cd_big_gallery.jpg'

display(Image(url=url))

processed = processor(images=url)
processed = processed.to(device='cuda')

generated_ids = model.generate(**processed, generation_config=generation_config)

findings, impression = processor.split_and_decode_sections(generated_ids) 
print(f'Findings:\t{findings[0]}\nImpression:\t{impression[0]}')
```

## Generate from study DICOMs:
```python
dcm_path = [
    './physionet.org/files/mimic-cxr/2.0.0/files/p12/p12000264/s55271473/522f9570-7cb12ecb-6327c8b8-b248b4be-58bb3dfd.dcm',
    './/datasets/work/hb-mlaifsp-mm/work/repositories/25_cxrmate2/work/data/physionet.org/files/mimic-cxr/2.0.0/files/p12/p12000264/s55271473/d0b61aff-f64c4ecf-85fae310-43668cf1-0c1d4c2d.dcm',
]

processed = processor(images=dcm_path)
processed = processed.to(device='cuda')

generated_ids = model.generate(**processed, generation_config=generation_config)

findings, impression = processor.split_and_decode_sections(generated_ids) 
print(f'Findings:\t{findings[0]}\nImpression:\t{impression[0]}')
```

## Generate from ReXgradient HuggingFace dataset:
**Scripts to create the ReXgradient, CheXpert Plus, and MIMC-CXR Hugging Face datasets will be released soon**
```python
# Run prepare_chexpert_plus.py or prepare_mimic_cxr_jpg.py or prepare_rexgradient.py to create dataset:
test_set = datasets.load_from_disk('/scratch3/nic261/database/cxrmate2/rexgradient_160k_dataset')['test']

# Wrap dataset to get priors:
test_set = processor.wrap_dataset(test_set)

random_idx = random.randint(0, len(test_set) - 1)
example = test_set[random_idx]

processed = processor(
    images=example['images'],  # This includes both the current and prior images.
    image_datetime=example['image_datetime'],  # This includes the datetimes for both the current and prior images.
    views=example['views'],  # This includes both the current and prior views.
    indication=example.get('indication', None),
    history=example.get('history', None),
    comparison=example.get('comparison', None),
    technique=example.get('technique', None),
    study_datetime=example.get('study_datetime', None),
    prior_findings=example.get('prior_findings', None),
    prior_impression=example.get('prior_impression', None),
    prior_study_datetime=example.get('prior_study_datetime', None),
)
processed = processed.to(device='cuda')

generated_ids = model.generate(**processed, generation_config=generation_config)

findings, impression = processor.split_and_decode_sections(generated_ids)
print(f'Findings:\t{findings[0]}\nImpression:\t{impression[0]}')
```


# Environment requirements:

Coming soon...
<!-- Environment requirements can be found here: https://github.com/aehrc/cxrmate-ed/blob/main/requirements.txt. -->


# Training:

Coming soon...

<!-- The training pipeline for CXRMate-2, is available at: https://github.com/aehrc/cxrmate-2. -->