| | --- |
| | license: apache-2.0 |
| | library_name: diffusers |
| | pipeline_tag: image-to-image |
| | tags: |
| | - zoomldm |
| | - histopathology |
| | - brca |
| | - latent-diffusion |
| | - custom-pipeline |
| | - arxiv:2411.16969 |
| | widget: |
| | - src: demo_images/input.jpeg |
| | prompt: BRCA sample conditioned on demo SSL feature (mag=0) |
| | output: |
| | url: demo_images/output.jpeg |
| | --- |
| | |
| | > [!WARNING] we do not have a full checkpoint conversion validation, if you encounter pipeline loading failure and unsidered output, please contact me via bili_sakura@zju.edu.cn |
| | |
| | # BiliSakura/ZoomLDM-brca |
| | |
| | Diffusers-format **BRCA** variant of ZoomLDM with a bundled custom pipeline and local `ldm` modules. |
| | |
| | ## Model Description |
| | |
| | - **Architecture:** ZoomLDM latent diffusion pipeline (`UNet + VAE + conditioning encoder`) |
| | - **Domain:** Histopathology (BRCA) |
| | - **Conditioning:** UNI-style SSL feature maps + magnification level (`0..4`) |
| | - **Format:** Self-contained local folder for `DiffusionPipeline.from_pretrained(...)` |
| |
|
| | ## Intended Use |
| |
|
| | Use this model for conditional multi-scale BRCA patch generation when you have compatible pre-extracted SSL features. |
| |
|
| | ## Out-of-Scope Use |
| |
|
| | - Not intended for diagnosis, treatment planning, or other clinical decisions. |
| | - Not a general-purpose text-to-image model. |
| | - Not validated for data outside the expected acquisition/distribution range. |
| |
|
| | ## Files |
| |
|
| | - `unet/`, `vae/`, `conditioning_encoder/`, `scheduler/` |
| | - `model_index.json` |
| | - `pipeline_zoomldm.py` |
| | - `ldm/` (bundled dependency modules) |
| |
|
| | ## Usage |
| |
|
| | ```python |
| | import torch |
| | from diffusers import DiffusionPipeline |
| | |
| | pipe = DiffusionPipeline.from_pretrained( |
| | "BiliSakura/ZoomLDM-brca", |
| | custom_pipeline="pipeline_zoomldm.py", |
| | trust_remote_code=True, |
| | ).to("cuda") |
| | |
| | out = pipe( |
| | ssl_features=ssl_feat_tensor.to("cuda"), # BRCA UNI-style SSL embeddings |
| | magnification=torch.tensor([0]).to("cuda"), # 0..4 |
| | num_inference_steps=50, |
| | guidance_scale=2.0, |
| | ) |
| | images = out.images |
| | ``` |
| |
|
| | ## Demo Generation (dataset-backed) |
| |
|
| | This repo includes `run_demo_inference.py`, which uses local repo assets only: |
| |
|
| | - image: `demo_images/input.jpeg` |
| | - SSL feature: `demo_data/0_ssl_feat.npy` |
| | - magnification label: `0` |
| |
|
| | Run: |
| |
|
| | ```bash |
| | python run_demo_inference.py |
| | ``` |
| |
|
| | ## Limitations |
| |
|
| | - Requires correctly precomputed BRCA conditioning features. |
| | - Magnification conditioning must match expected integer codes. |
| | - Generated content may reflect biases and artifacts from training data. |
| |
|
| | ## Citation |
| |
|
| | ```bibtex |
| | @InProceedings{Yellapragada_2025_CVPR, |
| | author = {Yellapragada, Srikar and Graikos, Alexandros and Triaridis, Kostas and Prasanna, Prateek and Gupta, Rajarsi and Saltz, Joel and Samaras, Dimitris}, |
| | title = {ZoomLDM: Latent Diffusion Model for Multi-scale Image Generation}, |
| | booktitle = {Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR)}, |
| | month = {June}, |
| | year = {2025}, |
| | pages = {23453-23463} |
| | } |
| | ``` |
| |
|
