Building a causality-aware single-cell RNA-seq foundation model via context-specific causal regulation modeling
Nature Biomedical Engineering, in press
Kaichen Xu*, Yuxuan Shen*, Maocheng Zhu, Zhenke Duan, Jin Liu†, Hao Wu†, Xiaobo Sun†
* Equal contribution · † Corresponding authors
scCAFM is a single-cell Causality-Aware Foundation Model pretrained on 49.7 million human and mouse cells to infer cell-specific gene regulatory networks (csGRNs) and learn transferable, contextual gene and cell embeddings. Its Structure Foundation Module (SFM) models causal regulatory relationships in a mixture-of-experts-mediated latent factor space, enabling scalable csGRN inference. Its Embedding Foundation Module (EFM) uses the inferred cell-specific causal gene orderings to learn causality-aware representations.
Model details
| SFM | EFM | |
|---|---|---|
| Output | Cell-specific GRNs | Cell and contextual gene embeddings |
| Backbone | 12-layer bidirectional Transformer | 12-layer causal-masked Transformer with rotary position embeddings |
| Width | 768 dimensions, 12 heads | 768 dimensions, 12 heads |
| Latent factors | 256, each gene routed to at most 32 | – |
| Parameters | 135.4 M | 149.0 M |
| Weights | models/sfm/sfm_model.safetensors |
models/efm/efm_model.safetensors |
- Pretraining data: 49.7 million cells, 31.5 million human and 18.2 million mouse.
- Vocabulary: 20,275 gene tokens shared by both modules (15.6 M embedding parameters).
- Total size: 300 M parameters, stored as float32 Safetensors.
- Species: Homo sapiens and Mus musculus.
SFM tokenizes the genes of a cell, encodes them with a bidirectional Transformer, and routes transcription factors and target genes into a shared latent factor space; the product of the two factor matrices is the cell-specific GRN. It is trained with SEM-based self-reconstruction, a DAG constraint, and an OmniPath-derived GRN prior. The DAG constraint is evaluated in the latent factor space, which lowers its cost from cubic in the number of genes to cubic in the number of factors. EFM sorts the genes of each cell topologically along its inferred GRN and encodes the ordered tokens with a causal-masked Transformer, producing a cell embedding and contextual gene embeddings.
How to use
Install the sccafm package from the GitHub repository, which also documents the FlashAttention setup, then download this repository:
hf download kaichenxu/scCAFM --local-dir assets
Infer one gene regulatory network per cell:
import pandas as pd
import scanpy as sc
from sccafm import GRNInferencer, ScPreprocessor, load_vocab_json, resolve_model_assets
assets = resolve_model_assets("assets")
token_dict = load_vocab_json(assets.vocab)
human_tfs = pd.read_csv(assets.human_tfs)["TF"].astype(str).tolist()
adata = sc.read_h5ad("my_cells.h5ad") # raw counts, gene symbols in var_names
adata.obs["species"] = "human"
# Keep every transcription factor as a regulator and add 500 variable target genes.
adata = ScPreprocessor(
log1p=False,
n_top_genes=500,
remove_ribo_genes=True,
remove_hb_genes=True,
token_dict=token_dict,
preserve_gene_names=human_tfs,
hvg_exclude_gene_names=human_tfs,
hvg_exclude_preserved_genes=True,
)(adata)
inferencer = GRNInferencer.from_pretrained(
"assets", device="cuda:0", attention_backend="fa4", max_length=2048
)
# Cell-specific GRNs are computed lazily, batch by batch.
cell_grns = inferencer.infer_cell_specific(adata, batch_size=8, score_threshold=0.1)
for cell_ids, scores in cell_grns:
... # scores: (cells, TFs, target genes)
# For a homogeneous population, average the networks into one pooled GRN.
pooled_grn = inferencer.infer_pooled(adata, batch_size=8)
edges = pooled_grn.to_edge_table() # columns: Gene1 (TF), Gene2 (target), score
The same assets drive the EFM workflows, CellTypeAnnotator and GenePerturbationPredictor, shown in the tutorials below.
Tutorials
| Tutorial | Data | Modules |
|---|---|---|
| Pooled GRNs with ChIP-seq benchmarking | hESC, mESC | SFM |
| Cell-specific GRNs in heterogeneous tissue | Mouse pancreas | SFM |
| Pooled GRNs with Perturb-seq validation | K562 | SFM |
| Cell type annotation with supervised fine-tuning | PBMC | SFM + EFM |
| Gene perturbation prediction | Norman | SFM + EFM |
The datasets used by the notebooks are in kaichenxu/scCAFM-data.
Files
| Path | Description |
|---|---|
models/sfm/ |
SFM configuration and pretrained weights |
models/efm/ |
EFM configuration and pretrained weights |
tokenizer/vocab.json, tokenizer/vocab.safetensors |
Shared gene vocabulary and its embedding tensor |
tokenizer/cond_dict.json |
Condition vocabulary |
resources/human_tfs.csv, resources/mouse_tfs.csv |
Human and mouse transcription-factor catalogs |
resources/OmniPath.csv |
Prior-knowledge interactions derived from OmniPath |
resources/homologous.csv |
Human–mouse gene homology |
release.json |
Manifest that the sccafm package reads to locate the files above |
config.json |
Summary of the release: modules, sizes, and file locations |
model.safetensors.index.json |
Root Safetensors index mapping tensors to the SFM, EFM, and vocabulary files |
Intended use and limitations
scCAFM is a research model for single-cell transcriptomics: cell-specific and pooled GRN inference, gene and cell representation learning, and downstream tasks such as perturbation prediction and cell type annotation in human and mouse.
- It is not intended for clinical diagnosis or treatment decisions.
- Inferred regulatory edges are computational hypotheses; validate them with experimental or orthogonal evidence.
- Performance can differ for tissues, technologies, and preprocessing pipelines that are far from the pretraining data.
Citation
@article{xu2026sccafm,
title = {Building a causality-aware single-cell RNA-seq foundation model via context-specific causal regulation modeling},
author = {Xu, Kaichen and Shen, Yuxuan and Zhu, Maocheng and Duan, Zhenke and Liu, Jin and Wu, Hao and Sun, Xiaobo},
journal = {Nature Biomedical Engineering},
year = {2026},
note = {In press}
}
License and support
The model is released under the GPL-3.0 license. Questions and bug reports are welcome through GitHub Issues.
- Downloads last month
- 23