README.md

---
license: creativeml-openrail-m
language:
  - en
tags:
  - histopathology
  - diffusion
  - image-generation
  - medical-imaging
  - latent-diffusion
  - semantic-synthesis
  - tissue-synthesis
  - computational-pathology
  - stable-diffusion
datasets:
  - Camelyon16
  - PANDA
  - TCGA
pipeline_tag: image-to-image
library_name: diffusers
model_name: HeteroTissueDiffuse
arxiv: 2509.17847
---

# HeteroTissueDiffuse

**Semantic and Visual Crop-Guided Diffusion Models for Heterogeneous Tissue Synthesis in Histopathology**

*NeurIPS 2025*

[Saghir Alfasly](https://saghiralfasly.github.io/) · [Wataru Uegami](https://www.linkedin.com/in/wataru-uegami-8b106920a/) · [MD Enamul Hoq](https://www.linkedin.com/in/mhoq89/) · [Ghazal Alabtah](https://www.linkedin.com/in/ghazal-alabtah-00/) · [H.R. Tizhoosh](https://tizhoosh.com/)

KIMIA Lab, Department of AI & Informatics, Mayo Clinic, Rochester, MN, USA

[![Paper](https://img.shields.io/badge/arXiv-2509.17847-b31b1b.svg)](https://arxiv.org/abs/2509.17847)
[![Project Page](https://img.shields.io/badge/Project-Page-blue)](https://kimialabmayo.github.io/hetero_tissue_diffuse_page/)
[![GitHub](https://img.shields.io/badge/GitHub-Code-black?logo=github)](https://github.com/KimiaLabMayo/hetero_tissue_diffuse)

---

## Model Description

**HeteroTissueDiffuse** is a latent diffusion model (LDM) that synthesizes heterogeneous histopathology images by conditioning on both a **binary semantic map** and **raw tissue crop exemplars**. Unlike text- or embedding-guided approaches, it injects actual tissue appearance directly into the diffusion process, preserving staining characteristics, nuclear morphology, and cellular texture.

The model addresses a fundamental limitation of prior generative methods in histopathology: the tendency to produce homogeneous (single-tissue-type) images. By providing spatially-registered visual prompts for each tissue class, the model generates realistic heterogeneous slides that accurately reflect real-world tissue organization.

### Architecture

- **Base**: CompVis Latent Diffusion Model with VQ-regularized autoencoder
- **First stage**: `VQModelInterface` (3-channel latent, 8192 codebook)
- **Conditioning encoder**: `SpatialRescaler` with `in_channels=8` (replaces ADE20K default of 182)
- **U-Net**: 128 base channels, attention at resolutions 32/16/8
- **Image size**: 256×256 pixels
- **Sampling**: DDIM, 200 steps, η=1

### 8-Channel Conditioning Tensor

```
Channel 0:    normal onehot mask         (1 where segmentation == 0)
Channels 1–3: normal tissue crop RGB     (float32, normalized to [-1,1])
Channel 4:    tumor onehot mask          (1 where segmentation == 1)
Channels 5–7: tumor tissue crop RGB      (float32, normalized to [-1,1])
```

The tissue crops are small patches (typically 30–60px) extracted from a reference slide and pasted spatially within the corresponding mask region. This lets users control staining appearance at inference time without any fine-tuning.

---

## Available Checkpoints

| File | Dataset | Description |
|------|---------|-------------|
| `camelyon16/epoch=000064.ckpt` | Camelyon16 | Binary tumor/normal masks, 256×256, 64 epochs |
| `panda/last.ckpt` | PANDA | Gleason tissue regions, 256×256 |
| `tcga/last.ckpt` | TCGA (self-supervised) | 100 pseudo-phenotype clusters, 256×256, 232 epochs |

---

## Quick Start

### 1. Clone the inference code

```bash
git clone https://github.com/CompVis/stable-diffusion.git
cd stable-diffusion

# Apply the 2 required patches (see GitHub README for details)
# Then copy our inference script
wget https://raw.githubusercontent.com/Saghir/HeteroTissueDiffuse/main/inference_heteroTissueDiffuse_camelyon.py
```

### 2. Download the checkpoint

```bash
pip install huggingface_hub

python - <<'EOF'
from huggingface_hub import hf_hub_download
# Camelyon16
hf_hub_download(repo_id="Saghir/HeteroTissueDiffuse",
                filename="camelyon16/epoch=000064.ckpt", local_dir="inference/")
# PANDA
hf_hub_download(repo_id="Saghir/HeteroTissueDiffuse",
                filename="panda/last.ckpt", local_dir="inference/")
# TCGA
hf_hub_download(repo_id="Saghir/HeteroTissueDiffuse",
                filename="tcga/last.ckpt", local_dir="inference/")
EOF
```

Or via CLI (download one dataset at a time):
```bash
# Camelyon16
huggingface-cli download Saghir/HeteroTissueDiffuse \
    camelyon16/epoch=000064.ckpt --local-dir inference/

# PANDA
huggingface-cli download Saghir/HeteroTissueDiffuse \
    panda/last.ckpt --local-dir inference/

# TCGA
huggingface-cli download Saghir/HeteroTissueDiffuse \
    tcga/last.ckpt --local-dir inference/
```

### 3. Run inference

```bash
conda activate diff   # PyTorch 2.0.1 + CUDA 11+

python inference_heteroTissueDiffuse_camelyon.py \
    --normal_prompt  inference/promptNormal2.png \
    --tumor_prompt   inference/promptTumor2.png \
    --segmentation_root inference/masks \
    --ckpt_path      inference/epoch=000064.ckpt \
    --output_dir     outputs/inference_results
```

**Inputs:**
- `--normal_prompt` / `--tumor_prompt`: small PNG crops of representative tissue regions (provided as examples in the [GitHub repo](https://github.com/Saghir/HeteroTissueDiffuse/tree/main/inference))
- `--segmentation_root`: folder of `.npy` binary masks (512×512, dtype bool, 0=normal, 1=tumor)

**Outputs** (per mask):
- `frame_XXX.png` — generated histopathology image
- `prompt_frame_XXX.png` — visualization of the conditioning (mask + overlaid crops)

---

## Performance

### Downstream Segmentation (IoU)

| Training data | Camelyon16 | PANDA |
|---------------|-----------|-------|
| Real images | 0.72 | 0.96 |
| **Synthetic (ours)** | **0.71** | **0.95** |
| Synthetic (no conditioning) | 0.51 | 0.82 |

### Pathologist Assessment

A certified pathologist evaluated 120 images in a blinded study. Synthetic images conditioned with visual prompts received quality scores **indistinguishable from real images**:

> *"The generated images tended to have equal or higher quality than the real images."*

---

## Intended Use

- **Research**: generating large annotated synthetic histopathology datasets for downstream model training
- **Augmentation**: expanding small annotated datasets with realistic diverse tissue variations
- **Privacy-preserving data sharing**: synthetic data as a substitute for patient slides
- **Education**: illustrating tissue morphology variations

---

## Training Details

### Camelyon16 Checkpoint

- **Dataset**: Camelyon16 (lymph node whole-slide images, binary tumor/normal segmentation)
- **Patch size**: 256×256 pixels at 0.5 µm/px
- **Training steps**: 64 epochs
- **Optimizer**: Adam, lr=1e-6
- **Hardware**: A100 GPU
- **Framework**: PyTorch 2.0.1 + pytorch-lightning 1.4.2

### Self-Supervised Extension (TCGA)

Patches from 11,765 TCGA whole-slide images were embedded using a histopathology foundation model (PathDino), then clustered into 100 tissue phenotypes via k-means. These clusters form pseudo-semantic maps for training without manual annotation.

---

## Citation

```bibtex
@InProceedings{Alfasly2025HeteroTissueDiffuse,
    author    = {Alfasly, Saghir and Uegami, Wataru and Hoq, MD Enamul and Alabtah, Ghazal and Tizhoosh, H.R.},
    title     = {Semantic and Visual Crop-Guided Diffusion Models for Heterogeneous Tissue Synthesis in Histopathology},
    booktitle = {Neural Information Processing Systems (NeurIPS)},
    month     = {December},
    year      = {2025}
}
```

---

## License

This model is released under the **CreativeML Open RAIL-M** license, inherited from [CompVis/stable-diffusion](https://github.com/CompVis/stable-diffusion). This license permits research and commercial use but prohibits use cases that cause harm (e.g., generating deceptive or malicious content). See the full license [here](https://huggingface.co/spaces/CompVis/stable-diffusion-license).