Hugging Face's logo

paige-ai
/
Prism2

Feature Extraction
Transformers
Safetensors
prism2
nlp
vision
PyTorch
custom_code
Model card Files
xet
 Community

You need to agree to share your contact information to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

This model and associated code are released under the CC-BY-NC-ND 4.0 license and may only be used for non-commercial, academic research purposes with proper attribution. Any commercial use, sale, or other monetization of the Prism2 Model and its derivatives, which include models trained on outputs from the Prism2 Model or datasets created from the Prism2 Model, is prohibited and requires prior approval. Please note that the primary email used to sign up for your Hugging Face account must match your institutional email to receive approval. By downloading the Prism2 Model, you attest that all information (affiliation, research use) is correct and up-to-date. Downloading the Prism2 Model requires prior registration on Hugging Face and agreeing to the terms of use. By downloading the Prism2 model, you agree not to distribute, publish or reproduce a copy of the Prism2 Model. If another user within your organization wishes to use the Prism2 Model, they must register as an individual user and agree to comply with the terms of use. Requests from commercial entities will be rejected by default if no statement of use received separately. If you are a commercial entity, please contact the corresponding author of this paper.

Further, by downloading the Prism2 model, you agree you will only use the Prism2 model for academic research purposes and will not use, or allow others to use, the Prism2 model to:

  1. Diagnose, cure, mitigate, treat, or prevent disease or any other conditions, including for Investigational Use Only (“IUO”), Research Use Only (“RUO”), commercial, clinical or other similar use, and including as a substitute for professional medical advice, a healthcare opinion, a diagnosis, treatment, or the clinical judgment of a healthcare professional, as no license or right is granted for any such purposes.

  2. Re-identify the deidentified data used to develop the Prism2 Model;

  3. Violate the law or others’ rights, including to:

    a. Engage in, promote, generate, contribute to, encourage, plan, incite, or further illegal or unlawful activity or content;

    b. Engage in, promote, incite, or facilitate the harassment, abuse, threatening, or bullying of individuals or groups of individuals;

    c. Engage in, promote, incite, or facilitate discrimination or other unlawful or harmful conduct in the provision of employment, employment benefits, credit, housing, other economic benefits, or other essential goods and services;

    d. Engage in the unauthorized or unlicensed practice of any profession including, but not limited to, financial, legal, medical/health, or related professional practices;

    e. Collect, process, disclose, generate, or infer the identity of individuals or the health, demographic, or other sensitive personal or private information about individuals without rights and consents required by applicable laws;

    f. Engage in or facilitate any action or generate any content that infringes, misappropriates, or otherwise violates any third-party rights, including the outputs or results of any products or services using the Prism2 Model or any related materials; and

    g. Create, generate, or facilitate the creation of malicious code, malware, computer viruses or do anything else that could disable, overburden, interfere with or impair the proper working, integrity, operation or appearance of a website or computer system.

  4. Engage in, promote, incite, facilitate, or assist in the planning or development of activities that present a risk of death or bodily harm to individuals, including the use of the Prism2 Model as a medical device, clinical support, diagnostic tool, or other technology intended to be used in the diagnosis, cure, mitigation, treatment, or prevention of disease or other conditions, including for Investigational Use Only (“IUO”), Research Use Only (“RUO”), commercial, clinical or similar use; and

  5. Intentionally deceive or mislead others, including representing that the use of the Prism2 Model or its outputs is human-generated.

Further, you agree that you will appropriately disclose to end users any known dangers of your AI system.

Log in or Sign Up to review the conditions and access this model content.

Model card for PRISM2

PRISM2 is a multi-modal generative foundation model for slide-level analysis of H&E-stained histopathology images. PRISM2 supports several modes of use:

  • slide-level "base" embedding extraction
  • slide-level "diagnostic" embedding extraction
  • free-text single-turn response generation (reports/question answering)
  • binary classification via prompting with a Yes/No question

This model is available solely for non-commercial research and evaluation purposes. It is not designed to be used as a multi-turn conversational chatbot or agent. Textual responses generated by PRISM2 are intended for assessing the model's quality and may contain errors. Using the generated responses in clinical settings is strictly prohibited.

Model Usage

Requirements

Using Python 3.10 or newer and a CUDA-capable GPU:

torch>=2.3
transformers>=4.51
safetensors
einops
flash-attn>=2.6.3

flash-attn is required — the perceiver's cross-attention uses flash_attn_varlen_func to match the training-time kernel.

Get the model

After gaining access to the model here, you will need to login to HuggingFace in the environment you wish to use the model. This can be done from the command line:

python -m pip install huggingface_hub
huggingface-cli login

or in your Python code:

from huggingface_hub import login

login()

Please refer to official HuggingFace documentation for more details.

Test model access:

from transformers import AutoModel
AutoModel.from_pretrained("paige-ai/Prism2", trust_remote_code=True)

Inference modes

PRISM2 requires Virchow2 tile embeddings with the following constraints:

  • Class token only (not Class + Mean)
  • 20x magnification (0.5 mpp)
  • 224x224 tile size
  • Foreground tissue tiles selected (with background/glass tiles removed)

See https://huggingface.co/paige-ai/Virchow2 for tile embedding.

PRISM2 takes pre-extracted tile embeddings, not raw slides. Per slide, you should provide a (N, 1280) tensor produced by Virchow2 where N is that slide's tile count. The processor pads + masks across a batch of slides with different N.

import torch
from transformers import AutoModel, AutoProcessor


model = AutoModel.from_pretrained(
    "paige-ai/Prism2", trust_remote_code=True, torch_dtype="auto",
).cuda().eval()
processor = AutoProcessor.from_pretrained("paige-ai/Prism2", trust_remote_code=True)

# Each slide is its own (N_i, 1280) tensor of Virchow2 tile embeddings.
slides = [tile_emb_slide_1, tile_emb_slide_2]   # list of (N_i, 1280) tensors

batch = processor(tile_embeddings=slides).to("cuda")

Creating a multi-slide embedding: PRISM2 can embed both single-slide and multi-slide (specimen-level) inputs. If you want to produce a single embedding for multiple whole slide images (eg. a patient embedding), you can simply concatenate all tile embeddings across the slides. Note that PRISM2 was trained on the part-specimen level which typically contains multiple whole slide images all sharing the same tissue type.

Base embedding

Slide-level base embedding (output of perceiver pooling).

with torch.autocast("cuda", torch.bfloat16):
    base = model.get_base_embedding(**batch)
print(base.shape)                                # torch.Size([B, 2560])

Diagnostic embedding

Slide-level diagnostic embedding (Phi-3 hidden state at the <|assistant|> position with a fixed image-only prompt).

with torch.autocast("cuda", torch.bfloat16):
    diag = model.get_diagnostic_embedding(**batch)
print(diag.shape)                                # torch.Size([B, 3072])

Yes/No question scoring

Scores a single yes/no question against every slide in the batch. Returns P(Yes | answer ∈ {Yes, No}) per slide, suitable for direct use with ROC / AUC.

with torch.autocast("cuda", torch.bfloat16):
    scores = model.yes_no_score(
        tile_embeddings=batch["tile_embeddings"],
        attention_mask=batch["attention_mask"],
        question="Is invasive carcinoma present?",
    )
print(scores)                                    # torch.Size([B])

Report generation

.get_response() can be used for single turn text responses from PRISM2, given tile embeddings and a prompt. For report generation it is recommended to use the prompt "Write a report".

with torch.autocast("cuda", torch.bfloat16):
    reports = model.get_response(**batch, prompt="Write a report", max_new_tokens=200)
print(reports)

Open-ended question answering 

with torch.autocast("cuda", torch.bfloat16):
    answers = model.get_response(
        **batch,
        prompt="What type of cancer is this?",
        max_new_tokens=100,
    )
print(answers)

Multiple-choice question answering

For multiple-choice questions it is highly recommended that the question follow the exact formatting below, but can be extended to up to 10 options.

prompt = (
    "What is the histologic type?\n"
    "Options:\n"
    "A. Invasive ductal carcinoma\n"
    "B. Invasive lobular carcinoma\n"
    "C. Tubular carcinoma\n"
    "D. Mucinous carcinoma"
)
with torch.autocast("cuda", torch.bfloat16):
    answers = model.get_response(**batch, prompt=prompt, max_new_tokens=20)
print(answers)

Validation

To validate the extracted base/diagnostic embeddings, you can use the resources provided in the accompanying Zenodo page to run linear probe evaluations:

https://zenodo.org/records/19056118

Note that bitwise reproducibilty of embeddings and linear probe results is not expected due to differences in foreground filtering, floating point error accumulation, and hardware/software.

Model Details

  • Developed by: Paige, NYC, USA and Microsoft Research, Cambridge, MA USA
  • Model Type: Vision-Language Encoder-Decoder
  • Model Stats:
    • WSI Encoder params (B): 0.6
    • Decoder params (B): 3.8
    • Params (B): 4.4
  • Model Architecture:
  • Training Details:
    • Stage 1 — contrastive + caption pretraining
    • Stage 2 — multi-task clincal dialogue training (the released checkpoint)
    • Precision: Mixed precision (bf16)
  • Paper:
  • Pretraining Dataset: Internal dataset of 2.3 million whole slide images from 685,507 specimens (200,692 patients) and 14 million question-answer pairs derived from clinical reports at Memorial Sloan Kettering Cancer Center.
  • License: CC-BY-NC-ND-4.0

A separate survival-finetuned variant of the perceiver is published at paige-ai/Prism2-survival for direct survival modeling.

Terms of use

This model and associated code are released under the CC-BY-NC-ND 4.0 license and may only be used for non-commercial, academic research purposes with proper attribution. Any commercial use, sale, or other monetization of the Prism2 Model and its derivatives, which include models trained on outputs from the Prism2 Model or datasets created from the Prism2 Model, is prohibited and requires prior approval. Please note that the primary email used to sign up for your Hugging Face account must match your institutional email to receive approval. By downloading the Prism2 Model, you attest that all information (affiliation, research use) is correct and up-to-date. Downloading the Prism2 Model requires prior registration on Hugging Face and agreeing to the terms of use. By downloading the Prism2 model, you agree not to distribute, publish or reproduce a copy of the Prism2 Model. If another user within your organization wishes to use the Prism2 Model, they must register as an individual user and agree to comply with the terms of use. If you are a commercial entity, please contact the corresponding author.

Further, by downloading the Prism2 model, you agree you will only use the Prism2 model for academic research purposes and will not use, or allow others to use, the Prism2 model to:

  1. Diagnose, cure, mitigate, treat, or prevent disease or any other conditions, including for Investigational Use Only ("IUO"), Research Use Only ("RUO"), commercial, clinical or other similar use, and including as a substitute for professional medical advice, a healthcare opinion, a diagnosis, treatment, or the clinical judgment of a healthcare professional, as no license or right is granted for any such purposes.

  2. Re-identify the deidentified data used to develop the Prism2 Model;

  3. Violate the law or others' rights, including to:

    a. Engage in, promote, generate, contribute to, encourage, plan, incite, or further illegal or unlawful activity or content;

    b. Engage in, promote, incite, or facilitate the harassment, abuse, threatening, or bullying of individuals or groups of individuals;

    c. Engage in, promote, incite, or facilitate discrimination or other unlawful or harmful conduct in the provision of employment, employment benefits, credit, housing, other economic benefits, or other essential goods and services;

    d. Engage in the unauthorized or unlicensed practice of any profession including, but not limited to, financial, legal, medical/health, or related professional practices;

    e. Collect, process, disclose, generate, or infer the identity of individuals or the health, demographic, or other sensitive personal or private information about individuals without rights and consents required by applicable laws;

    f. Engage in or facilitate any action or generate any content that infringes, misappropriates, or otherwise violates any third-party rights, including the outputs or results of any products or services using the Prism2 Model or any related materials; and

    g. Create, generate, or facilitate the creation of malicious code, malware, computer viruses or do anything else that could disable, overburden, interfere with or impair the proper working, integrity, operation or appearance of a website or computer system.

  4. Engage in, promote, incite, facilitate, or assist in the planning or development of activities that present a risk of death or bodily harm to individuals, including the use of the Prism2 Model as a medical device, clinical support, diagnostic tool, or other technology intended to be used in the diagnosis, cure, mitigation, treatment, or prevention of disease or other conditions, including for Investigational Use Only ("IUO"), Research Use Only ("RUO"), commercial, clinical or similar use; and

  5. Intentionally deceive or mislead others, including representing that the use of the Prism2 Model or its outputs is human-generated.

Further, you agree that you will appropriately disclose to end users any known dangers of your AI system.

Citation 

@article{vorontsov2025prism2,
  title={PRISM2: Unlocking Multi-Modal General Pathology AI with Clinical Dialogue},
  author={Vorontsov, Eugene and Shaikovski, George and Casson, Adam and Viret, Julian and Zimmermann, Eric and Tenenholtz, Neil and Wang, Yi Kan and Bernhard, Jan H. and Godrich, Ran A. and Retamero, Juan A. and Shia, Jinru and Gonen, Mithat and Weiser, Martin R. and Klimstra, David S. and Yousfi, Razik and Fusi, Nicolo and Fuchs, Thomas J. and Severson, Kristen and Liu, Siqi},
  journal={arXiv preprint arXiv:2506.13063},
  year={2025}
}

Disclaimer

PRISM2 has been developed for research purposes and is not intended for diagnosis of real patients or projection/prediction of future disease possibilities.

Fairness evaluation cannot be completed due to limitations in the metadata. Underlying biases of the training datasets may not be well characterized and may not be representative of all demographics.

Downloads last month
6
Safetensors
Model size
4B params
Tensor type
F32
·
Inference Providers NEW
Feature Extraction
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Paper for paige-ai/Prism2