README.md

---
license: cc-by-nc-sa-4.0
library_name: deepspotm
pipeline_tag: image-feature-extraction
base_model: kaiko-ai/midnight
base_model_relation: adapter
language:
- en
tags:
- biology
- medical
- histology
- pathology
- spatial-transcriptomics
- gene-expression
- lora
- computational-pathology
- foundation-model
- multimodal
- virtual-spatial-transcriptomics
- whole-slide-imaging
- oncology
- transcriptomics
- deep-learning
- cancer
extra_gated_heading: "Acknowledge license to accept the repository"
extra_gated_prompt: >-
  DeepSpotM is released for non-commercial academic research only, under
  CC-BY-NC-SA-4.0.
  Requests with vague or insufficient descriptions of intended use will be declined.
extra_gated_fields:
  Full name (first and last): text
  Current affiliation (no abbreviations): text
  Type of Affiliation: text
  Current and official institutional email:
    type: text
    help: "This must match your primary email in your Hugging Face account. Emails from gmail, hotmail, and qq domains will be denied."
  Please describe in detail how you intend to use this model:
    type: text
    help: "Be specific. Name your institution/lab, the broader research project or area, and concretely how it will be used (e.g. model training, benchmarking, validation). Generic answers such as 'for research purposes' are not sufficient and will be declined."
  I confirm that I have read and understood the model license terms: checkbox
  I agree to retain attribution and license any derivatives under CC-BY-NC-SA-4.0:
    type: checkbox
    help: "Redistribution and derivatives are permitted under the same CC-BY-NC-SA-4.0 license, with attribution and non-commercial use."
  I agree to receive news and updates about this technology: checkbox
extra_gated_button_content: "Request access"
---

# DeepSpot-M

**DeepSpot-M: a multimodal foundation model for transcriptome-wide virtual spatial transcriptomics from histology.**

DeepSpot-M is a multimodal foundation model that maps a histology image tile to
spatial gene expression. It tokenises a 224x224 H&E tile with a LoRA-adapted
pathology foundation backbone ([Midnight](https://huggingface.co/kaiko-ai/midnight))
and lets each gene query attend to the patch tokens through a cross-attention gene
decoder. A gene router hypernetwork generates gene-specific output projections from
frozen biological embeddings drawn from DNA, RNA, protein, single-cell and text
foundation models (Evo 2, Orthrus, ProtT5, scGPT, Apertus). Because genes are
represented as queryable embeddings rather than fixed outputs, one model predicts
transcriptome-wide expression and genes it never saw during training.

Code is available on [GitHub](https://github.com/ratschlab/DeepSpotM).

![DeepSpot-M model overview](https://huggingface.co/ratschlab/DeepSpotM/resolve/main/model_overview.png)

**Fig. DeepSpot-M predicts transcriptome-wide spatial gene expression from histology.**
A 224x224 H&E tile is tokenised into spatial patch embeddings by a LoRA-adapted
pathology foundation model. A cross-attention gene decoder lets each gene query
independently attend to patch tokens via multi-head attention, and a gene router
hypernetwork generates gene-specific output projections from frozen biological
embeddings drawn from DNA, RNA, protein, single-cell and text foundation models.
This design enables zero-shot prediction of genes at inference time.

> ⚠️ **Research use only. Not for clinical or diagnostic use.**

## Model description

DeepSpot-M adapts the Midnight pathology backbone with LoRA and feeds its patch
tokens to a cross-attention gene decoder conditioned on biological gene embeddings.
It takes 224x224 H&E tiles as input and outputs expression over the ~19k-gene panel
in `tokens.csv`. Five embedding sources are available, namely `evo2`, `orthrus`,
`prott5`, `scgpt` and `apertus`, selected at inference with `source=`.

## Usage

```python
from deepspotm import DeepSpotM   # pip install git+https://github.com/ratschlab/DeepSpotM.git

model, image_processor = DeepSpotM.from_pretrained(
    "ratschlab/DeepSpotM",
    source="scgpt",   # one of evo2, orthrus, prott5, scgpt, apertus
)

import torch
tile = image_processor(my_pil_tile).unsqueeze(0)   # 224x224 H&E tile
with torch.no_grad():
    expression, _, _ = model(tile)                 # (1, 19338)

# Output column i corresponds to model.gene_names[i].
preds = dict(zip(model.gene_names, expression.squeeze(0).tolist()))
print(preds["EPCAM"])
```

The predicted vector is ordered by `model.gene_names`, the genes in `tokens.csv`, so
`model.gene_names[i]` is the symbol for output column `i`.

### Predict only specific genes (faster)

You don't have to predict all ~19k genes. Pass a gene or a list and only those are
computed, because the cross-attention runs over just the requested gene queries.

```python
vals = model.predict_genes(tile, ["EPCAM", "CD3D", "PTPRC"])   # (1, 3)
vals = model.predict_genes(tile, "EPCAM")                       # (1, 1)
```

Output columns follow the requested order. Unknown symbols raise `KeyError`.

The vision backbone is built offline from a bundled config and its weights are baked
into `model.safetensors`, so loading needs no network access to the upstream backbone
repo.

## Tutorial

[`examples/predict_tcga_skcm.ipynb`](https://github.com/ratschlab/DeepSpotM/blob/main/examples/predict_tcga_skcm.ipynb)
runs DeepSpot-M end to end on a whole-slide TCGA-SKCM H&E image. It tiles the slide,
predicts BRAF, CD37 and COL1A1, and overlays the predictions on the tissue.

## Resources

- Code, [github.com/ratschlab/DeepSpotM](https://github.com/ratschlab/DeepSpotM)
- TCGA virtual spatial transcriptomics atlas of 28,664 slides across 32 cancers, [ratschlab/TCGA_virtual_spatial_transcriptomics_atlas](https://huggingface.co/datasets/ratschlab/TCGA_virtual_spatial_transcriptomics_atlas)
- HEST-1K virtual single-cell Xenium profiles for 59 samples, [ratschlab/HEST_Xenium_virtual_spatial_transcriptomics](https://huggingface.co/datasets/ratschlab/HEST_Xenium_virtual_spatial_transcriptomics)

## Limitations and biases

- Trained on a finite set of cancer indications. Performance on unseen tissue types,
  stains, scanners or resolutions may degrade.
- Predicts relative expression rather than absolute counts. Under-sequenced genes are
  predicted less reliably.
- Trained on oncology cohorts, so it is not representative of healthy tissue or
  non-oncology contexts. Not for clinical or diagnostic use.

## License

- Weights, CC-BY-NC-SA-4.0. Non-commercial, ShareAlike, with attribution.
- Code, [github.com/ratschlab/DeepSpotM](https://github.com/ratschlab/DeepSpotM), under PolyForm Noncommercial 1.0.0.

See `WEIGHTS_LICENSE.md` and `THIRD_PARTY_LICENSES.md`.

## Citation

**Paper**: [DeepSpot-M: a multimodal foundation model for transcriptome-wide virtual spatial transcriptomics from histology](https://www.medrxiv.org/content/10.64898/2026.06.19.26356060v1) (medRxiv, 2026).

```bibtex
@article{nonchev2026deepspotm,
  title   = {DeepSpot-M: a multimodal foundation model for transcriptome-wide virtual spatial transcriptomics from histology},
  author  = {Nonchev, Kalin and Dawo, Sebastian and Silina, Karina and Koelzer, Viktor H. and Raetsch, Gunnar},
  journal = {medRxiv},
  year    = {2026},
  doi     = {10.64898/2026.06.19.26356060},
  url     = {https://www.medrxiv.org/content/10.64898/2026.06.19.26356060v1}
}
```

See also `CITATION.cff`.