Upload 5 files

README.md

@@ -1,3 +1,66 @@
----
-license: mit
----
+---
+license: mit
+tags:
+- single-cell
+- biology
+base_model:
+- MohamedMabrouk/scGPT
+---
+# scGPT
+scGPT is A foundation model for single-cell biology based on a generative pre trained transformer across a repository of over 33 million cells.
+
+# Abstract
+Generative pretrained models have achieved remarkable success in various domains such as language and computer vision. Specifically, the combination of large-scale diverse datasets and pretrained transformers has emerged as a promising approach for developing foundation models. Drawing parallels between language and cellular biology (in which texts comprise words; similarly, cells are defined by genes), our study probes the applicability of foundation models to advance cellular biology and genetic research. Using burgeoning single-cell sequencing data, we have constructed a foundation model for single-cell biology, scGPT, based on a generative pretrained transformer across a repository of over 33 million cells. Our findings illustrate that scGPT effectively distills critical biological insights concerning genes and cells. Through further adaptation of transfer learning, scGPT can be optimized to achieve superior performance across diverse downstream applications. This includes tasks such as cell type annotation, multi-batch integration, multi-omic integration, perturbation response prediction and gene network inference.
+
+# Code
+
+```python
+from tdc.multi_pred.anndata_dataset import DataLoader
+from tdc import tdc_hf_interface
+from tdc.model_server.tokenizers.scgpt import scGPTTokenizer
+import torch
+
+# an example dataset
+adata = DataLoader("cellxgene_sample_small",
+                   "./data",
+                   dataset_names=["cellxgene_sample_small"],
+                   no_convert=True).adata
+
+# code for loading the model and performing inference
+scgpt = tdc_hf_interface("scGPT")
+model = scgpt.load()  # This line can cause segmentation fault on inappropriate setup
+tokenizer = scGPTTokenizer()
+gene_ids = adata.var["feature_name"].to_numpy(
+)  # Convert to numpy array
+tokenized_data = tokenizer.tokenize_cell_vectors(
+    adata.X.toarray(), gene_ids)
+mask = torch.tensor([x != 0 for x in tokenized_data[0][1]],
+                    dtype=torch.bool)
+
+# Extract first embedding
+first_embed = model(tokenized_data[0][0],
+                    tokenized_data[0][1],
+                    attention_mask=mask)
+```
+
+# TDC.scGPT Source Code
+https://github.com/mims-harvard/TDC/blob/main/tdc/model_server/models/scgpt.py
+* hf migration code available upon request
+* weights extracted from base model
+
+# TDC Citation
+```
+@inproceedings{
+velez-arce2024signals,
+title={Signals in the Cells: Multimodal and Contextualized Machine Learning Foundations for Therapeutics},
+author={Alejandro Velez-Arce and Xiang Lin and Kexin Huang and Michelle M Li and Wenhao Gao and Bradley Pentelute and Tianfan Fu and Manolis Kellis and Marinka Zitnik},
+booktitle={NeurIPS 2024 Workshop on AI for New Drug Modalities},
+year={2024},
+url={https://openreview.net/forum?id=kL8dlYp6IM}
+}
+```
+# Additional Citations
+- Cui, H., Wang, C., Maan, H. et al. scGPT: toward building a foundation model for single-cell multi-omics using generative AI. Nat Methods 21, 1470–1480 (2024). https://doi.org/10.1038/s41592-024-02201-0
+
+# Model Github
+https://github.com/bowang-lab/scGPT

config.json

@@ -0,0 +1,22 @@
+{
+  "architectures": [
+    "ScGPTModel"
+  ],
+  "cell_emb_style": "cls",
+  "d_hid": 512,
+  "dropout": 0.0,
+  "embsize": 512,
+  "explicit_zero_prob": false,
+  "input_emb_style": "continuous",
+  "max_seq_len": 1536,
+  "model_type": "scgpt",
+  "nhead": 8,
+  "nlayers": 12,
+  "norm_scheme": "post",
+  "pad_token_id": 0,
+  "torch_dtype": "float32",
+  "transformers_version": "4.43.4",
+  "use_fast_transformer": true,
+  "use_flash_attention": false,
+  "vocab_size": 60697
+}

gitattributes

@@ -0,0 +1,38 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+<<<<<<< HEAD
+*.tar filter=lfs diff=lfs merge=lfs -text
+=======
+>>>>>>> 23519966c9f2ad55dae4b35e50ce64a1aa17a70b
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cabd40e6e22514865825975940f2c61ac1395f3317bba9f773858cd72914064c
+size 203233980

scgpt_gh_repo_original_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ad0252a1971e0cd619b7116dbab3177432236c4537225d54280a2aa7e5fe402a
+size 207861754