|
|
--- |
|
|
license: apache-2.0 |
|
|
datasets: |
|
|
- JFLa/GF-CAB_Datasets |
|
|
language: |
|
|
- en |
|
|
metrics: |
|
|
- accuracy |
|
|
base_model: |
|
|
- ctheodoris/Geneformer |
|
|
pipeline_tag: token-classification |
|
|
library_name: transformers |
|
|
tags: |
|
|
- biology |
|
|
- single-cell |
|
|
- transcriptomics |
|
|
--- |
|
|
|
|
|
# 🧬 Geneformer-CAB: Benchmarking Scale and Architecture in Foundation Models for Single-Cell Transcriptomics |
|
|
|
|
|
## Model Overview |
|
|
|
|
|
**Geneformer-CAB (Cumulative-Assignment-Blocking)** is a benchmarked variant of the Geneformer architecture for modeling single-cell transcriptomic data. |
|
|
Rather than introducing an entirely new model, Geneformer-CAB systematically evaluates how **data scale** and **architectural refinements** interact to influence model generalization, predictive diversity, and robustness to batch effects. |
|
|
|
|
|
This model integrates two architectural enhancements: |
|
|
- **Cumulative probability recalibration**, which adjusts token-level prediction dynamics to reduce overconfident, frequency-driven outputs. |
|
|
- **Similarity-based regularization**, which penalizes redundant token predictions to promote diversity and alignment with rank-ordered gene expression profiles. |
|
|
|
|
|
Together, these mechanisms provide insight into the **limits of scale** in single-cell foundation models — revealing that scaling up pretraining data does not always yield superior downstream performance. |
|
|
|
|
|
--- |
|
|
|
|
|
## Key Results |
|
|
|
|
|
| Task Type | Comparison | Key Finding | |
|
|
|------------|-------------|-------------| |
|
|
| **Pretraining Objectives** | GF-CAB vs. Geneformer | Higher masked prediction accuracy and diversity across scales | |
|
|
| **Classification Tasks** | GF-CAB-1M vs. Geneformer-1M | Comparable or improved accuracy, narrowing the scale gap | |
|
|
| **Zero-shot Batch Mitigation** | GF-CAB vs. Geneformer | Stronger generalization across datasets, less scale-dependent | |
|
|
|
|
|
> Scaling pretraining data from 1M to 30M profiles improved discriminative tasks but reduced cross-dataset robustness — while architectural calibration in GF-CAB balanced both. |
|
|
|
|
|
--- |
|
|
|
|
|
## Model Architecture |
|
|
|
|
|
- **Base architecture:** Transformer encoder (BERT-style masked modeling) |
|
|
- **Input representation:** Ranked gene expression profiles per cell |
|
|
- **Masking objective:** Predict masked gene ranks, excluding unmasked regions |
|
|
- **Innovations:** |
|
|
- Cumulative probability recalibration (adjusted decoding dynamics) |
|
|
- Similarity-based penalty loss (reduces redundancy in token predictions) |
|
|
|
|
|
--- |
|
|
|
|
|
## Pretraining Data |
|
|
|
|
|
| Dataset | Description | Size | |
|
|
|----------|--------------|------| |
|
|
| **Genecorpus-1M** | Random subset of ranked single-cell profiles from public scRNA-seq datasets | 1 million profiles | |
|
|
| **Genecorpus-30M** | Large-scale extension incorporating additional datasets and donors | 30 million profiles | |
|
|
|
|
|
--- |
|
|
|
|
|
## Downstream Evaluation |
|
|
|
|
|
1. **Cell-type classification** (3 benchmark tasks) |
|
|
2. **Zero-shot batch-effect mitigation** (4 public datasets) |
|
|
|
|
|
Evaluation followed standardized pipelines based on Theodoris et al. (for classification) and Kedzierska et al. (for zero-shot robustness). |
|
|
|
|
|
--- |
|
|
|
|
|
## Intended Use |
|
|
|
|
|
This model is designed for: |
|
|
- Benchmarking **foundation models** on single-cell gene expression tasks |
|
|
- Studying **scaling effects** in biological pretraining |
|
|
- Investigating **rank-based profile modeling** and representation diversity |
|
|
|
|
|
--- |
