scCAFM: a causality-aware single-cell foundation model

Building a causality-aware single-cell RNA-seq foundation model via context-specific causal regulation modeling
Nature Biomedical Engineering, in press

Kaichen Xu*, Yuxuan Shen*, Maocheng Zhu, Zhenke Duan, Jin Liu†, Hao Wu†, Xiaobo Sun†
* Equal contribution  ·  † Corresponding authors

Paper Code Tutorial data License

scCAFM is a single-cell Causality-Aware Foundation Model pretrained on 49.7 million human and mouse cells to infer cell-specific gene regulatory networks (csGRNs) and learn transferable, contextual gene and cell embeddings. Its Structure Foundation Module (SFM) models causal regulatory relationships in a mixture-of-experts-mediated latent factor space, enabling scalable csGRN inference. Its Embedding Foundation Module (EFM) uses the inferred cell-specific causal gene orderings to learn causality-aware representations.

scCAFM is pretrained on human and mouse cells; SFM infers cell-specific GRNs and EFM learns cell and gene embeddings for downstream tasks

Model details

SFM EFM
Output Cell-specific GRNs Cell and contextual gene embeddings
Backbone 12-layer bidirectional Transformer 12-layer causal-masked Transformer with rotary position embeddings
Width 768 dimensions, 12 heads 768 dimensions, 12 heads
Latent factors 256, each gene routed to at most 32 –
Parameters 135.4 M 149.0 M
Weights models/sfm/sfm_model.safetensors models/efm/efm_model.safetensors
  • Pretraining data: 49.7 million cells, 31.5 million human and 18.2 million mouse.
  • Vocabulary: 20,275 gene tokens shared by both modules (15.6 M embedding parameters).
  • Total size: 300 M parameters, stored as float32 Safetensors.
  • Species: Homo sapiens and Mus musculus.

Architecture of the Structure Foundation Module and the Embedding Foundation Module

SFM tokenizes the genes of a cell, encodes them with a bidirectional Transformer, and routes transcription factors and target genes into a shared latent factor space; the product of the two factor matrices is the cell-specific GRN. It is trained with SEM-based self-reconstruction, a DAG constraint, and an OmniPath-derived GRN prior. The DAG constraint is evaluated in the latent factor space, which lowers its cost from cubic in the number of genes to cubic in the number of factors. EFM sorts the genes of each cell topologically along its inferred GRN and encodes the ordered tokens with a causal-masked Transformer, producing a cell embedding and contextual gene embeddings.

How to use

Install the sccafm package from the GitHub repository, which also documents the FlashAttention setup, then download this repository:

hf download kaichenxu/scCAFM --local-dir assets

Infer one gene regulatory network per cell:

import pandas as pd
import scanpy as sc
from sccafm import GRNInferencer, ScPreprocessor, load_vocab_json, resolve_model_assets

assets = resolve_model_assets("assets")
token_dict = load_vocab_json(assets.vocab)
human_tfs = pd.read_csv(assets.human_tfs)["TF"].astype(str).tolist()

adata = sc.read_h5ad("my_cells.h5ad")  # raw counts, gene symbols in var_names
adata.obs["species"] = "human"

# Keep every transcription factor as a regulator and add 500 variable target genes.
adata = ScPreprocessor(
    log1p=False,
    n_top_genes=500,
    remove_ribo_genes=True,
    remove_hb_genes=True,
    token_dict=token_dict,
    preserve_gene_names=human_tfs,
    hvg_exclude_gene_names=human_tfs,
    hvg_exclude_preserved_genes=True,
)(adata)

inferencer = GRNInferencer.from_pretrained(
    "assets", device="cuda:0", attention_backend="fa4", max_length=2048
)

# Cell-specific GRNs are computed lazily, batch by batch.
cell_grns = inferencer.infer_cell_specific(adata, batch_size=8, score_threshold=0.1)
for cell_ids, scores in cell_grns:
    ...  # scores: (cells, TFs, target genes)

# For a homogeneous population, average the networks into one pooled GRN.
pooled_grn = inferencer.infer_pooled(adata, batch_size=8)
edges = pooled_grn.to_edge_table()  # columns: Gene1 (TF), Gene2 (target), score

The same assets drive the EFM workflows, CellTypeAnnotator and GenePerturbationPredictor, shown in the tutorials below.

Tutorials

The datasets used by the notebooks are in kaichenxu/scCAFM-data.

Files

Path Description
models/sfm/ SFM configuration and pretrained weights
models/efm/ EFM configuration and pretrained weights
tokenizer/vocab.json, tokenizer/vocab.safetensors Shared gene vocabulary and its embedding tensor
tokenizer/cond_dict.json Condition vocabulary
resources/human_tfs.csv, resources/mouse_tfs.csv Human and mouse transcription-factor catalogs
resources/OmniPath.csv Prior-knowledge interactions derived from OmniPath
resources/homologous.csv Human–mouse gene homology
release.json Manifest that the sccafm package reads to locate the files above
config.json Summary of the release: modules, sizes, and file locations
model.safetensors.index.json Root Safetensors index mapping tensors to the SFM, EFM, and vocabulary files

Intended use and limitations

scCAFM is a research model for single-cell transcriptomics: cell-specific and pooled GRN inference, gene and cell representation learning, and downstream tasks such as perturbation prediction and cell type annotation in human and mouse.

  • It is not intended for clinical diagnosis or treatment decisions.
  • Inferred regulatory edges are computational hypotheses; validate them with experimental or orthogonal evidence.
  • Performance can differ for tissues, technologies, and preprocessing pipelines that are far from the pretraining data.

Citation

@article{xu2026sccafm,
  title   = {Building a causality-aware single-cell RNA-seq foundation model via context-specific causal regulation modeling},
  author  = {Xu, Kaichen and Shen, Yuxuan and Zhu, Maocheng and Duan, Zhenke and Liu, Jin and Wu, Hao and Sun, Xiaobo},
  journal = {Nature Biomedical Engineering},
  year    = {2026},
  note    = {In press}
}

License and support

The model is released under the GPL-3.0 license. Questions and bug reports are welcome through GitHub Issues.

Downloads last month
23
Safetensors
Model size
0.3B params
Tensor type
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support