Instructions to use ctheodoris/Geneformer with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use ctheodoris/Geneformer with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("fill-mask", model="ctheodoris/Geneformer")# Load model directly from transformers import AutoTokenizer, AutoModelForMaskedLM tokenizer = AutoTokenizer.from_pretrained("ctheodoris/Geneformer") model = AutoModelForMaskedLM.from_pretrained("ctheodoris/Geneformer") - Inference
- Notebooks
- Google Colab
- Kaggle
Add nano-Geneformer as a community reference implementation
Browse files## What is nano-Geneformer?
I recently built **nano-Geneformer**, a lightweight and faithful reimplementation of Geneformer designed to make the core implementation easier to read, reproduce, benchmark, and extend while preserving the original architecture and inference behavior.
Repository:
https://github.com/huynguyen250896/nano-Geneformer
### Highlights
* Supports **all official Geneformer checkpoints** (V1, V2-104M, V2-104M_CLcancer, and V2-316M)
* Faithfully reproduces the original Geneformer architecture and inference pipeline
* Cleaner, modern PyTorch implementation with simplified installation and dependency management
* Suitable for learning, benchmarking, experimentation, fine-tuning, and future training from scratch
### Validation
I carefully benchmarked nano-Geneformer against the official implementation to ensure it can serve as a practical community reference implementation.
Compared with the official implementation, nano-Geneformer:
* reduces **peak GPU memory by up to 56.8%** for the largest Geneformer model (V2-316M)
* achieves **1.06–1.15× faster inference**
* reproduces cell embeddings with **mean cosine similarity ≈ 1.000000**
* preserves local/global representation geometry and pairwise distance structure across all official checkpoints
The full benchmark notebook is available in the repository.
### Why this PR?
The goal of nano-Geneformer is **not to replace the official implementation**, but to provide a lightweight community resource for users who want a smaller, easier-to-read implementation for learning, reproducibility, benchmarking, and research.
This PR only adds a link under **Community Projects**. It does **not** modify any code, pretrained models, datasets, checkpoints, or model behavior.
Thank you for your consideration.
|
@@ -10,7 +10,15 @@ Geneformer is a foundational transformer model pretrained on a large-scale corpu
|
|
| 10 |
|
| 11 |
- See [our manuscript](https://rdcu.be/ddrx0) for details of the original model trained on ~30 million transcriptomes in June 2021 and the initial report of our in silico perturbation and cell and gene classification strategies.
|
| 12 |
- See [our manuscript](https://rdcu.be/famFk) for details of the expanded model, now trained on ~104 million transcriptomes, and our quantization implementation for resource-efficient predictions.
|
| 13 |
-
- See [our preprint](https://www.biorxiv.org/content/10.1101/2024.08.16.608180v1.full.pdf
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 14 |
- See [geneformer.readthedocs.io](https://geneformer.readthedocs.io) for documentation.
|
| 15 |
|
| 16 |
# Model Description
|
|
@@ -60,6 +68,9 @@ Example applications demonstrated in [our manuscript](https://rdcu.be/ddrx0) inc
|
|
| 60 |
- in silico perturbation to determine transcription factor targets
|
| 61 |
- in silico perturbation to determine transcription factor cooperativity
|
| 62 |
|
|
|
|
|
|
|
|
|
|
| 63 |
# Installation
|
| 64 |
In addition to the pretrained model, contained herein are functions for tokenizing and collating data specific to single cell transcriptomics, pretraining the model, fine-tuning the model, extracting and plotting cell embeddings, and performing in silico pertrubation with either the pretrained or fine-tuned models. To install (~20s):
|
| 65 |
|
|
|
|
| 10 |
|
| 11 |
- See [our manuscript](https://rdcu.be/ddrx0) for details of the original model trained on ~30 million transcriptomes in June 2021 and the initial report of our in silico perturbation and cell and gene classification strategies.
|
| 12 |
- See [our manuscript](https://rdcu.be/famFk) for details of the expanded model, now trained on ~104 million transcriptomes, and our quantization implementation for resource-efficient predictions.
|
| 13 |
+
- See [our preprint](https://www.biorxiv.org/content/10.1101/2024.08.16.608180v1.full.pdf
|
| 14 |
+
|
| 15 |
+
|
| 16 |
+
|
| 17 |
+
|
| 18 |
+
|
| 19 |
+
|
| 20 |
+
|
| 21 |
+
) for details of our continual and multitask learning strategies.
|
| 22 |
- See [geneformer.readthedocs.io](https://geneformer.readthedocs.io) for documentation.
|
| 23 |
|
| 24 |
# Model Description
|
|
|
|
| 68 |
- in silico perturbation to determine transcription factor targets
|
| 69 |
- in silico perturbation to determine transcription factor cooperativity
|
| 70 |
|
| 71 |
+
# Community Projects
|
| 72 |
+
**[nano-Geneformer](https://github.com/huynguyen250896/nano-Geneformer)** — A lightweight, faithful, and easy-to-read reimplementation of Geneformer for learning, benchmarking, rapid experimentation, and extension.
|
| 73 |
+
|
| 74 |
# Installation
|
| 75 |
In addition to the pretrained model, contained herein are functions for tokenizing and collating data specific to single cell transcriptomics, pretraining the model, fine-tuning the model, extracting and plotting cell embeddings, and performing in silico pertrubation with either the pretrained or fine-tuned models. To install (~20s):
|
| 76 |
|