README.md

---
library_name: peft
base_model: microsoft/llava-rad
pipeline_tag: image-text-to-text
tags:
- radiology
- chest-x-ray
- llava
- lora
- contrastive-decoding
- medical-report-generation
---

# 🩻 [MIDL 2026 Oral] CWCD Category-Specific LoRA Adapters

[Shantam Srivastava](https://scholar.google.com/citations?user=UXG7qiYAAAAJ&hl=en), [Mahesh Bhosale](https://bhosalems.github.io/), [David Doermann](https://scholar.google.com/citations?user=RoGOW9AAAAAJ&hl=en), [Mingchen Gao](https://cse.buffalo.edu/~mgao8/)

**University at Buffalo, The State University of New York**

[![Conference](https://img.shields.io/badge/MIDL%202026-Oral%20Presentation-1f6feb)](https://2026.midl.io/scientific-program)
[![Oral Selection](https://img.shields.io/badge/Oral%20Selection-%3C8%25-success)](https://2026.midl.io/scientific-program)
[![Paper](https://img.shields.io/badge/Paper-arXiv%3A2604.10410-b31b1b?logo=arxiv&logoColor=white)](https://arxiv.org/abs/2604.10410)
[![Oral Presentation](https://img.shields.io/badge/Oral%20Presentation-YouTube-ff0000?logo=youtube&logoColor=white)](https://youtu.be/kXXQ5yVJcTI?t=2582)
[![Press Release](https://img.shields.io/badge/Press%20Release-UB%20News-005bbb)](https://www.buffalo.edu/news.host.html/content/shared/university/news/news-center-releases/2026/03/AI-system-curbs-AI-hallucinations-automated-chest-X-rays.detail.html)
[![Code](https://img.shields.io/badge/Code-GitHub-181717?logo=github&logoColor=white)](https://github.com/shantamsrivastava/CWCD)

CWCD was accepted as a full paper at **Medical Imaging with Deep Learning (MIDL) 2026** and selected for an **oral presentation** (oral selection rate below 8%).

## 📖 Model Description

This repository contains the eight category-specific rank-1 LoRA adapters released with **CWCD: Category-Wise Contrastive Decoding for Structured Medical Report Generation**. CWCD generates a structured findings report through independent forward passes under eight anatomical categories.

At inference time, each adapter is used for both the normal X-ray and its corresponding category-masked X-ray. CWCD contrasts the resulting base and masked log-probabilities, applies Plausibility-Based Vocabulary Subselection, and greedily selects the next token. The eight category-wise findings are combined into the final structured findings report.

The complete inference implementation and usage instructions are available in the [CWCD GitHub repository](https://github.com/shantamsrivastava/CWCD).

## 🤗 Available Adapters

| Anatomical category | Adapter directory | LoRA rank | Target modules |
|---|---|---:|---|
| Lungs and Airways | `adapters/lungs` | 1 | `q_proj`, `v_proj` |
| Musculoskeletal and Chest Wall | `adapters/wall` | 1 | `q_proj`, `v_proj` |
| Cardiovascular | `adapters/cardiovascular` | 1 | `q_proj`, `v_proj` |
| Pleura | `adapters/pleura` | 1 | `q_proj`, `v_proj` |
| Hila and Mediastinum | `adapters/hila` | 1 | `q_proj`, `v_proj` |
| Tubes, Catheters, and Support Devices | `adapters/tubes` | 1 | `q_proj`, `v_proj` |
| Abdominal | `adapters/abdominal` | 1 | `q_proj`, `v_proj` |
| Other | `adapters/other` | 1 | `q_proj`, `v_proj` |

Each directory contains:

```text
adapter_config.json
adapter_model.bin
```

## 🧠 CWCD Inference

For each anatomical category, the GitHub inference pipeline:

1. loads the corresponding category-specific adapter;
2. obtains the base distribution from the normal X-ray;
3. obtains the masked distribution after blacking out that category's bounding boxes;
4. contrasts the two distributions in log-probability space;
5. applies Plausibility-Based Vocabulary Subselection;
6. generates the category-wise findings and combines all eight sections.

The contrastive score is:

```text
score(token) = (1 + alpha) * log P_base(token) - alpha * log P_masked(token)
```

The released defaults are `alpha=1.0` and `beta=0.5`.

## 🚀 Using the Adapters

The adapters are downloaded automatically by [`inference.py`](https://github.com/shantamsrivastava/CWCD/blob/main/inference.py) through `huggingface_hub.snapshot_download()`. Only the adapter configuration and weight files are retrieved.

```bash
git clone https://github.com/shantamsrivastava/CWCD.git
cd CWCD

python -m pip install -r requirements.txt
python -m pip install git+https://github.com/microsoft/LLaVA-Rad.git

python inference.py \
  --image chest_xray.png \
  --boxes boxes.example.json \
  --alpha 1.0 \
  --beta 0.5 \
  --output report.json
```

See the [GitHub README](https://github.com/shantamsrivastava/CWCD#readme) for environment setup, input format, standard category-wise inference, prompts, and all CLI options.

## 📦 Repository Structure

```text
adapters/
├── abdominal/
├── cardiovascular/
├── hila/
├── lungs/
├── other/
├── pleura/
├── tubes/
└── wall/
```

## ⚠️ Intended Use and Limitations

These adapters are intended strictly for research and educational use. CWCD has not been approved or validated for clinical or diagnostic deployment and must not be used to make medical decisions or inform patient care.

Generated findings can contain omissions, incorrect statements, or hallucinations. Performance depends on the base model, input image quality, category-specific bounding boxes, and the domain represented by the evaluation data. Users are responsible for protecting patient privacy and following all applicable data-use agreements.

## 🤝 Acknowledgements

CWCD builds on [LLaVA-Rad](https://github.com/microsoft/LLaVA-Rad), [LLaVA](https://github.com/haotian-liu/LLaVA), [Vicuna](https://github.com/lm-sys/FastChat), and [PEFT](https://github.com/huggingface/peft).

## 📑 Citation

```bibtex
@inproceedings{srivastava2026cwcd,
  title={CWCD: Category-Wise Contrastive Decoding for Structured Medical Report Generation},
  author={Srivastava, Shantam and Bhosale, Mahesh and Doermann, David and Gao, Mingchen},
  booktitle={Medical Imaging with Deep Learning},
  series={Proceedings of Machine Learning Research},
  volume={315},
  pages={868--893},
  year={2026},
  publisher={PMLR}
}
```