nqhuya commited on
Commit
252a7bb
·
verified ·
1 Parent(s): 04c2b2e

Add nano-Geneformer as a community reference implementation

Browse files

## What is nano-Geneformer?

I recently built **nano-Geneformer**, a lightweight and faithful reimplementation of Geneformer designed to make the core implementation easier to read, reproduce, benchmark, and extend while preserving the original architecture and inference behavior.

Repository:
https://github.com/huynguyen250896/nano-Geneformer

### Highlights

* Supports **all official Geneformer checkpoints** (V1, V2-104M, V2-104M_CLcancer, and V2-316M)
* Faithfully reproduces the original Geneformer architecture and inference pipeline
* Cleaner, modern PyTorch implementation with simplified installation and dependency management
* Suitable for learning, benchmarking, experimentation, fine-tuning, and future training from scratch

### Validation

I carefully benchmarked nano-Geneformer against the official implementation to ensure it can serve as a practical community reference implementation.

Compared with the official implementation, nano-Geneformer:

* reduces **peak GPU memory by up to 56.8%** for the largest Geneformer model (V2-316M)
* achieves **1.06–1.15× faster inference**
* reproduces cell embeddings with **mean cosine similarity ≈ 1.000000**
* preserves local/global representation geometry and pairwise distance structure across all official checkpoints

The full benchmark notebook is available in the repository.

### Why this PR?

The goal of nano-Geneformer is **not to replace the official implementation**, but to provide a lightweight community resource for users who want a smaller, easier-to-read implementation for learning, reproducibility, benchmarking, and research.

This PR only adds a link under **Community Projects**. It does **not** modify any code, pretrained models, datasets, checkpoints, or model behavior.

Thank you for your consideration.

Files changed (1) hide show
  1. README.md +12 -1
README.md CHANGED
@@ -10,7 +10,15 @@ Geneformer is a foundational transformer model pretrained on a large-scale corpu
10
 
11
  - See [our manuscript](https://rdcu.be/ddrx0) for details of the original model trained on ~30 million transcriptomes in June 2021 and the initial report of our in silico perturbation and cell and gene classification strategies.
12
  - See [our manuscript](https://rdcu.be/famFk) for details of the expanded model, now trained on ~104 million transcriptomes, and our quantization implementation for resource-efficient predictions.
13
- - See [our preprint](https://www.biorxiv.org/content/10.1101/2024.08.16.608180v1.full.pdf) for details of our continual and multitask learning strategies.
 
 
 
 
 
 
 
 
14
  - See [geneformer.readthedocs.io](https://geneformer.readthedocs.io) for documentation.
15
 
16
  # Model Description
@@ -60,6 +68,9 @@ Example applications demonstrated in [our manuscript](https://rdcu.be/ddrx0) inc
60
  - in silico perturbation to determine transcription factor targets
61
  - in silico perturbation to determine transcription factor cooperativity
62
 
 
 
 
63
  # Installation
64
  In addition to the pretrained model, contained herein are functions for tokenizing and collating data specific to single cell transcriptomics, pretraining the model, fine-tuning the model, extracting and plotting cell embeddings, and performing in silico pertrubation with either the pretrained or fine-tuned models. To install (~20s):
65
 
 
10
 
11
  - See [our manuscript](https://rdcu.be/ddrx0) for details of the original model trained on ~30 million transcriptomes in June 2021 and the initial report of our in silico perturbation and cell and gene classification strategies.
12
  - See [our manuscript](https://rdcu.be/famFk) for details of the expanded model, now trained on ~104 million transcriptomes, and our quantization implementation for resource-efficient predictions.
13
+ - See [our preprint](https://www.biorxiv.org/content/10.1101/2024.08.16.608180v1.full.pdf
14
+
15
+
16
+
17
+
18
+
19
+
20
+
21
+ ) for details of our continual and multitask learning strategies.
22
  - See [geneformer.readthedocs.io](https://geneformer.readthedocs.io) for documentation.
23
 
24
  # Model Description
 
68
  - in silico perturbation to determine transcription factor targets
69
  - in silico perturbation to determine transcription factor cooperativity
70
 
71
+ # Community Projects
72
+ **[nano-Geneformer](https://github.com/huynguyen250896/nano-Geneformer)** — A lightweight, faithful, and easy-to-read reimplementation of Geneformer for learning, benchmarking, rapid experimentation, and extension.
73
+
74
  # Installation
75
  In addition to the pretrained model, contained herein are functions for tokenizing and collating data specific to single cell transcriptomics, pretraining the model, fine-tuning the model, extracting and plotting cell embeddings, and performing in silico pertrubation with either the pretrained or fine-tuned models. To install (~20s):
76