--- license: creativeml-openrail-m language: - en tags: - histopathology - diffusion - image-generation - medical-imaging - latent-diffusion - semantic-synthesis - tissue-synthesis - computational-pathology - stable-diffusion datasets: - Camelyon16 - PANDA - TCGA pipeline_tag: image-to-image library_name: diffusers model_name: HeteroTissueDiffuse arxiv: 2509.17847 --- # HeteroTissueDiffuse **Semantic and Visual Crop-Guided Diffusion Models for Heterogeneous Tissue Synthesis in Histopathology** *NeurIPS 2025* [Saghir Alfasly](https://saghiralfasly.github.io/) · [Wataru Uegami](https://www.linkedin.com/in/wataru-uegami-8b106920a/) · [MD Enamul Hoq](https://www.linkedin.com/in/mhoq89/) · [Ghazal Alabtah](https://www.linkedin.com/in/ghazal-alabtah-00/) · [H.R. Tizhoosh](https://tizhoosh.com/) KIMIA Lab, Department of AI & Informatics, Mayo Clinic, Rochester, MN, USA [![Paper](https://img.shields.io/badge/arXiv-2509.17847-b31b1b.svg)](https://arxiv.org/abs/2509.17847) [![Project Page](https://img.shields.io/badge/Project-Page-blue)](https://kimialabmayo.github.io/hetero_tissue_diffuse_page/) [![GitHub](https://img.shields.io/badge/GitHub-Code-black?logo=github)](https://github.com/KimiaLabMayo/hetero_tissue_diffuse) --- ## Model Description **HeteroTissueDiffuse** is a latent diffusion model (LDM) that synthesizes heterogeneous histopathology images by conditioning on both a **binary semantic map** and **raw tissue crop exemplars**. Unlike text- or embedding-guided approaches, it injects actual tissue appearance directly into the diffusion process, preserving staining characteristics, nuclear morphology, and cellular texture. The model addresses a fundamental limitation of prior generative methods in histopathology: the tendency to produce homogeneous (single-tissue-type) images. By providing spatially-registered visual prompts for each tissue class, the model generates realistic heterogeneous slides that accurately reflect real-world tissue organization. ### Architecture - **Base**: CompVis Latent Diffusion Model with VQ-regularized autoencoder - **First stage**: `VQModelInterface` (3-channel latent, 8192 codebook) - **Conditioning encoder**: `SpatialRescaler` with `in_channels=8` (replaces ADE20K default of 182) - **U-Net**: 128 base channels, attention at resolutions 32/16/8 - **Image size**: 256×256 pixels - **Sampling**: DDIM, 200 steps, η=1 ### 8-Channel Conditioning Tensor ``` Channel 0: normal onehot mask (1 where segmentation == 0) Channels 1–3: normal tissue crop RGB (float32, normalized to [-1,1]) Channel 4: tumor onehot mask (1 where segmentation == 1) Channels 5–7: tumor tissue crop RGB (float32, normalized to [-1,1]) ``` The tissue crops are small patches (typically 30–60px) extracted from a reference slide and pasted spatially within the corresponding mask region. This lets users control staining appearance at inference time without any fine-tuning. --- ## Available Checkpoints | File | Dataset | Description | |------|---------|-------------| | `camelyon16/epoch=000064.ckpt` | Camelyon16 | Binary tumor/normal masks, 256×256, 64 epochs | | `panda/last.ckpt` | PANDA | Gleason tissue regions, 256×256 | | `tcga/last.ckpt` | TCGA (self-supervised) | 100 pseudo-phenotype clusters, 256×256, 232 epochs | --- ## Quick Start ### 1. Clone the inference code ```bash git clone https://github.com/CompVis/stable-diffusion.git cd stable-diffusion # Apply the 2 required patches (see GitHub README for details) # Then copy our inference script wget https://raw.githubusercontent.com/Saghir/HeteroTissueDiffuse/main/inference_heteroTissueDiffuse_camelyon.py ``` ### 2. Download the checkpoint ```bash pip install huggingface_hub python - <<'EOF' from huggingface_hub import hf_hub_download # Camelyon16 hf_hub_download(repo_id="Saghir/HeteroTissueDiffuse", filename="camelyon16/epoch=000064.ckpt", local_dir="inference/") # PANDA hf_hub_download(repo_id="Saghir/HeteroTissueDiffuse", filename="panda/last.ckpt", local_dir="inference/") # TCGA hf_hub_download(repo_id="Saghir/HeteroTissueDiffuse", filename="tcga/last.ckpt", local_dir="inference/") EOF ``` Or via CLI (download one dataset at a time): ```bash # Camelyon16 huggingface-cli download Saghir/HeteroTissueDiffuse \ camelyon16/epoch=000064.ckpt --local-dir inference/ # PANDA huggingface-cli download Saghir/HeteroTissueDiffuse \ panda/last.ckpt --local-dir inference/ # TCGA huggingface-cli download Saghir/HeteroTissueDiffuse \ tcga/last.ckpt --local-dir inference/ ``` ### 3. Run inference ```bash conda activate diff # PyTorch 2.0.1 + CUDA 11+ python inference_heteroTissueDiffuse_camelyon.py \ --normal_prompt inference/promptNormal2.png \ --tumor_prompt inference/promptTumor2.png \ --segmentation_root inference/masks \ --ckpt_path inference/epoch=000064.ckpt \ --output_dir outputs/inference_results ``` **Inputs:** - `--normal_prompt` / `--tumor_prompt`: small PNG crops of representative tissue regions (provided as examples in the [GitHub repo](https://github.com/Saghir/HeteroTissueDiffuse/tree/main/inference)) - `--segmentation_root`: folder of `.npy` binary masks (512×512, dtype bool, 0=normal, 1=tumor) **Outputs** (per mask): - `frame_XXX.png` — generated histopathology image - `prompt_frame_XXX.png` — visualization of the conditioning (mask + overlaid crops) --- ## Performance ### Downstream Segmentation (IoU) | Training data | Camelyon16 | PANDA | |---------------|-----------|-------| | Real images | 0.72 | 0.96 | | **Synthetic (ours)** | **0.71** | **0.95** | | Synthetic (no conditioning) | 0.51 | 0.82 | ### Pathologist Assessment A certified pathologist evaluated 120 images in a blinded study. Synthetic images conditioned with visual prompts received quality scores **indistinguishable from real images**: > *"The generated images tended to have equal or higher quality than the real images."* --- ## Intended Use - **Research**: generating large annotated synthetic histopathology datasets for downstream model training - **Augmentation**: expanding small annotated datasets with realistic diverse tissue variations - **Privacy-preserving data sharing**: synthetic data as a substitute for patient slides - **Education**: illustrating tissue morphology variations --- ## Training Details ### Camelyon16 Checkpoint - **Dataset**: Camelyon16 (lymph node whole-slide images, binary tumor/normal segmentation) - **Patch size**: 256×256 pixels at 0.5 µm/px - **Training steps**: 64 epochs - **Optimizer**: Adam, lr=1e-6 - **Hardware**: A100 GPU - **Framework**: PyTorch 2.0.1 + pytorch-lightning 1.4.2 ### Self-Supervised Extension (TCGA) Patches from 11,765 TCGA whole-slide images were embedded using a histopathology foundation model (PathDino), then clustered into 100 tissue phenotypes via k-means. These clusters form pseudo-semantic maps for training without manual annotation. --- ## Citation ```bibtex @InProceedings{Alfasly2025HeteroTissueDiffuse, author = {Alfasly, Saghir and Uegami, Wataru and Hoq, MD Enamul and Alabtah, Ghazal and Tizhoosh, H.R.}, title = {Semantic and Visual Crop-Guided Diffusion Models for Heterogeneous Tissue Synthesis in Histopathology}, booktitle = {Neural Information Processing Systems (NeurIPS)}, month = {December}, year = {2025} } ``` --- ## License This model is released under the **CreativeML Open RAIL-M** license, inherited from [CompVis/stable-diffusion](https://github.com/CompVis/stable-diffusion). This license permits research and commercial use but prohibits use cases that cause harm (e.g., generating deceptive or malicious content). See the full license [here](https://huggingface.co/spaces/CompVis/stable-diffusion-license).