README.md

metadata

license: other
license_name: cc-by-nc-nd-4.0
license_link: https://github.com/xiangjx/MuPaD/blob/main/LICENSE.md
tags:
  - histology
  - pathology
  - pytorch
  - vision
  - diffusion
  - flow
  - vit
language:
  - en
metrics:
  - fid
  - cosine similarity
extra_gated_prompt: >-
  - This model and associated code are released under the [CC-BY-NC-ND 4.0
  license](https://creativecommons.org/licenses/by-nc-nd/4.0/) and may only be
  used for non-commercial, academic research purposes with proper attribution. 

  - Any commercial use, sale, or other monetization of the MuPaD model and its
  derivatives, which include models trained on outputs from the MuPaD model or
  datasets created from the MuPaD model, is prohibited and requires prior
  approval. 

  - By downloading the model, you attest that all information (affiliation,
  research use) is correct and up-to-date. Downloading the model requires prior
  registration on Hugging Face and agreeing to the terms of use. By downloading
  this model, you agree not to distribute, publish or reproduce a copy of the
  model. If another user within your organization wishes to use the CytoSyn
  model, they must register as an individual user and agree to comply with the
  terms of use. Users may not attempt to re-identify the deidentified data used
  to develop the underlying model. 

  - This model is provided “as-is” without warranties of any kind, express or
  implied. This model has not been reviewed, certified, or approved by any
  regulatory body, including but not limited to the FDA (U.S.), EMA (Europe),
  MHRA (UK), or other medical device authorities. Any application of this model
  in healthcare or biomedical settings must comply with relevant regulatory
  requirements and undergo independent validation. Users assume full
  responsibility for how they use this model and any resulting consequences. The
  authors, contributors, and distributors disclaim any liability for damages,
  direct or indirect, resulting from model use. Users are responsible for
  ensuring compliance with data protection regulations (e.g., GDPR, HIPAA) when
  using it in research that involves patient data.
extra_gated_fields:
  Full Name (first and last): text
  Type of Affiliation:
    type: select
    options:
      - Industry
      - Academia
      - Other
  Current Affiliation (no abbreviations): text
  Current and Official Institutional Email: text
  Main use-case:
    type: select
    options:
      - Models Benchmarking
      - Data Augmentation
      - Models Interpretation
      - Other
  Please add information on your intended research use: text
  I agree to use this model for non-commercial, academic purposes only: checkbox
  I agree not to distribute the model: checkbox
  I am interested by receiving updates from Bioptimus:
    type: checkbox
    optional: true

Usage

Please run demo.py for full demonstrations. This repo generates 256x256 images.

Load Model

from diffusers import DiffusionPipeline
import torch

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

pipeline = DiffusionPipeline.from_pretrained(
    "xiangjx/MuPaD-256",
    custom_pipeline="xiangjx/MuPaD-256",
    trust_remote_code=True,
)
pipeline.to(device)

Text-to-Image Generation

Generate histopathology images from a text prompt.

# Text-to-Image generation
prompt = "lung adenocarcinoma"

output_t2i = pipeline(
    prompt=prompt,
    modality="text",
    num_images_per_prompt=4,
    num_inference_steps=250,
    guidance_scale=2.5,
    guidance_high=0.75,
    guidance_low=0.0,
    mode="sde",
    path_type="linear",
    seed=42
)

for i, img in enumerate(output_t2i["images"]):
    img.save(f"text2image_{i}.png")

Image-to-Image Generation

Generate images conditioned on a reference image.

from PIL import Image

# Load reference image
# Ensure you have a reference image path
raw_image = Image.open("test_image.png").convert("RGB")

output_i2i = pipeline(
    image=raw_image,
    modality="image",
    num_images_per_prompt=4,
    num_inference_steps=250,
    guidance_scale=2.5,
    guidance_high=0.75,
    guidance_low=0.0,
    mode="sde",
    path_type="linear",
    seed=42
)

for i, img in enumerate(output_i2i["images"]):
    img.save(f"image2image_{i}.png")

Software Dependencies

• torch>=2.0.0
• diffusers>=0.35.1
• timm>=0.9.0
• pillow
• huggingface-hub
• dictdot
• einops
• fairscale
• transformers==4.57.3
• sentencepiece