README.md

---
tags:
  - model_hub_mixin
  - pytorch_model_hub_mixin
---

# sCellTransformer

sCellTransformer (sCT) is a long-range foundation model designed for zero-shot
prediction tasks in single-cell RNA-seq and spatial transcriptomics data. It processes
raw gene expression profiles across multiple cells to predict discretized gene
expression levels for unseen cells without retraining. The model can handle up to 20,000
protein-coding genes and a bag of 50 cells in the same sample. This ability
(around a million-gene expressions tokens) allows it to learn cross-cell
relationships and capture long-range dependencies in gene expression data,
and to mitigate the sparsity typical in single-cell datasets.

sCT is trained on a large dataset of single-cell RNA-seq and finetuned on spatial
transcriptomics data. Evaluation tasks include zero-shot imputation of masked gene
expression, and zero-shot prediction of cell types.

**Developed by:** [InstaDeep](https://huggingface.co/InstaDeepAI)

### Model Sources

<!-- Provide the basic links for the model. -->

- **Repository:
  ** [Nucleotide Transformer](https://github.com/instadeepai/nucleotide-transformer)
- **Paper:
  ** [A long range foundation model for zero-shot predictions in single-cell and spatial transcriptomics data](https://openreview.net/pdf?id=VdX9tL3VXH)

### How to use

Until its next release, the transformers library needs to be installed from source with
the following command in order to use the models.
PyTorch should also be installed.

```
pip install --upgrade git+https://github.com/huggingface/transformers.git
pip install torch
```

A small snippet of code is given here in order to infer with the model from random
input.

```
import torch
from transformers import AutoModel

model = AutoModel.from_pretrained(
    "InstaDeepAI/sCellTransformer",
    trust_remote_code=True,
)
num_cells = model.config.num_cells
dummy_gene_expressions = torch.randint(0, 5, (1, 19968 * num_cells))
torch_output = model(dummy_gene_expressions)
```

A more concrete example is provided in the notebook example on one of the downstream
evaluation dataset.

#### Training data

The model was trained following a two-step procedure:
pre-training on single-cell data, then finetuning on spatial transcriptomics data.
The single-cell data used for pre-training, comes from the
[Cellxgene Census collection datasets](https://cellxgene.cziscience.com/)
used to train the scGPT models. It consists of around 50 millions
cells and approximately 60,000 genes. The spatial data comes from both the [human
breast cell atlas](https://cellxgene.cziscience.com/collections/4195ab4c-20bd-4cd3-8b3d-65601277e731)
and [the human heart atlas](https://www.heartcellatlas.org/).

#### Training procedure

As detailed in the paper, the gene expressions are first binned into a pre-defined
number of bins. This allows the model to better learn the distribution of the gene
expressions through sparsity mitigation, noise reduction, and extreme-values handling.
Then, the training objective is to predict the masked gene expressions in a cell,
following a BERT-like style training.

### BibTeX entry and citation info

```
@misc{joshi2025a,
title={A long range foundation model for zero-shot predictions in single-cell and
spatial transcriptomics data},
author={Ameya Joshi and Raphael Boige and Lee Zamparo and Ugo Tanielian and Juan Jose
Garau-Luis and Michail Chatzianastasis and Priyanka Pandey and Janik Sielemann and
Alexander Seifert and Martin Brand and Maren Lang and Karim Beguir and Thomas PIERROT},
year={2025},
url={https://openreview.net/forum?id=VdX9tL3VXH}
}
```