Upload 10 files

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+banner.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,3 +1,147 @@
 ---
 license: mit
+tags:
+  - single-cell
+  - transcriptomics
+  - biology
+  - pytorch
 ---
+
+<p align="center">
+  <a href="https://valegiunchiglia.github.io/cascade-website/">
+    <img src="banner.png" alt="CASCADE — Cross-System, Multi-Scale Single-Cell Foundation Model with Clinical Applications" width="100%">
+  </a>
+</p>
+
+<h1 align="center">CASCADE: context-aware single-cell modelling links cellular programmes to patient-level disease phenotypes</h1>
+
+<table align="center" border="0" cellspacing="0" cellpadding="4" style="border-collapse: collapse; border: none; background: transparent; margin-left: auto; margin-right: auto;">
+  <tr>
+    <td style="border: none; background: transparent;"><a href="https://valegiunchiglia.github.io/cascade-website/"><img src="https://img.shields.io/badge/Website-4CAF50?logo=googlechrome&logoColor=white" alt="Website"></a></td>
+    <td style="border: none; background: transparent;"><a href="https://github.com/mims-harvard/CASCADE"><img src="https://img.shields.io/badge/Code-181717?logo=github&logoColor=white" alt="Code"></a></td>
+    <td style="border: none; background: transparent;"><img src="https://img.shields.io/badge/Paper-coming%20soon-b31b1b?logo=arxiv&logoColor=white" alt="Paper (coming soon)"></td>
+    <td style="border: none; background: transparent;"><img src="https://img.shields.io/badge/%F0%9F%A4%97%20Dataset-HLCA-FFD21E" alt="Dataset: HLCA"></td>
+  </tr>
+</table>
+
+## Model Card
+
+This repository contains a CASCADE checkpoint pre-trained on the Human Lung Cell Atlas (HLCA) cohort.
+
+## Introduction
+
+CASCADE integrates contextual information into both input representation and pre-training
+objectives, allowing the same cell to be interpreted through multiple biologically meaningful
+axes and enabling patient-level phenotype prediction from single-cell profiles.
+
+**1. Context-aware tokenisation.** Each cell is encoded as context-dependent up- and
+down-regulated genes relative to a biologically defined reference group, producing multiple
+representations per cell across disease, tissue, cell type, and treatment contexts.
+
+**2. Context-specific representation learning.** Shared cell embeddings are projected through
+separate context-specific projectors (disease, tissue, cell type, treatment), learning how
+molecular programmes vary across biologically meaningful contexts via contrastive objectives.
+
+**3. Patient representation & explainability.** Cell-level embeddings are aggregated across all
+cells from a donor to produce a patient-level representation for multiscale phenotype
+prediction. CASCADE-Explainer identifies the cell types and genes most responsible for each
+prediction.
+
+- Code: https://github.com/mims-harvard/CASCADE
+- Project page: https://valegiunchiglia.github.io/cascade-website/
+- Source dataset: https://cellxgene.cziscience.com/collections/edb893ee-4066-4128-9aec-5eb2b03f8287
+
+## Training Data
+
+- **Disease states** (16): COVID-19, chronic obstructive pulmonary disease, chronic rhinitis, cystic fibrosis, hypersensitivity pneumonitis, interstitial lung disease, lung adenocarcinoma, lung large cell carcinoma, lymphangioleiomyomatosis, non-specific interstitial pneumonia, normal, pleomorphic carcinoma, pneumonia, pulmonary fibrosis, pulmonary sarcoidosis, squamous cell lung carcinoma
+- **Tissues** (4): lung, lung parenchyma, nose, respiratory airway
+- **Cell types** (51): identified by Cell Ontology (CL) ID — CL:0000037, CL:0000057, CL:0000077, CL:0000084, CL:0000097, CL:0000158, CL:0000186, CL:0000192, CL:0000236, CL:0000313, .... Look up terms at https://www.ebi.ac.uk/ols4/ontologies/cl.
+
+## Model Architecture
+
+CASCADE's encoder (`TransformerGenerator`) is a shared transformer over context-aware gene
+token sequences, followed by context-specific projection heads (one per context in the table
+below) trained with a context-specific contrastive objective.
+
+### Model Hyperparameters
+
+| Hyperparameter | Value |
+|---|---|
+| Embedding dim (`d_model`) | 384 |
+| Attention heads (`nhead`) | 6 |
+| Transformer layers (`nlayers`) | 12 |
+| Feedforward dim (`dim_embedding`) | 384 |
+| Dropout | 0.1 |
+| Vocabulary size | 18711 |
+| Cell embedding style | `avg-pool` |
+| Contexts | disease, cell_type, tissue |
+| Context-specific projections | True |
+| Domain adaptation (Sinkhorn) | True |
+
+### Files Included
+
+| File | Purpose |
+|---|---|
+| `model.safetensors` | Model weights only (stripped of optimizer/scheduler/scaler state) |
+| `config.json` | Architecture hyperparameters needed to reconstruct `TransformerGenerator` |
+| `tokenizer_dictionary_HLCA.pkl` | Gene/context vocabulary (18711 tokens) used by the context-aware tokenizer |
+| `metadata_dictionary_HLCA.pkl` | Obs-column metadata mapping preserved from the source AnnData |
+| `median_genes_*_all_HLCA.pkl` | Per-context median expression reference used to derive up-/down-regulated gene tokens at tokenization time |
+
+## Usage Instructions
+
+The model architecture (`TransformerGenerator`) is not a standard `transformers` class, so
+loading it requires the `cascade` package from the GitHub repo rather than `AutoModel`:
+
+```bash
+pip install git+https://github.com/mims-harvard/CASCADE
+```
+
+```python
+import json, pickle
+from huggingface_hub import hf_hub_download
+from safetensors.torch import load_file
+from cascade.model.cascade_model import TransformerGenerator
+
+repo_id = "<your-org>/CASCADE-HLCA"
+
+config = json.load(open(hf_hub_download(repo_id, "config.json")))
+vocab = pickle.load(open(hf_hub_download(repo_id, f"tokenizer_dictionary_HLCA.pkl"), "rb"))
+weights = load_file(hf_hub_download(repo_id, "model.safetensors"))
+
+model = TransformerGenerator(
+    d_model=config["d_model"],
+    nhead=config["nhead"],
+    ntoken=config["vocab_size"],
+    dim_embedding=config["dim_embedding"],
+    nlayers=config["nlayers"],
+    vocab=vocab,
+    nclass=config["nclass"],
+    dropout=config["dropout"],
+    pad_token=config["pad_token"],
+    cell_emb_style=config["cell_emb_style"],
+    context_specific_projections=config["context_specific_projections"],
+    constant_ctx=config["constant_ctx"],
+    only_contrastive=config["only_contrastive"],
+    DA=config["DA"],
+    lambda_sinkhorn=config["lambda_sinkhorn"],
+    merged_contexts=config["merged_contexts"],
+)
+model.load_state_dict(weights)
+model.eval()
+```
+
+To tokenize new raw data for this model, see `cascade/data/tokenizer.py` in the GitHub repo,
+using the `tokenizer_dictionary_HLCA.pkl` and `median_genes_*_all_HLCA.pkl` files
+from this repo as the vocab and per-context median reference respectively.
+
+## Citation
+
+Paper coming soon — see the [project page](https://valegiunchiglia.github.io/cascade-website/)
+for updates.
+
+## Contact
+
+For any questions or feedback, please open an issue in the [GitHub repository](https://github.com/mims-harvard/CASCADE)
+or contact [Valentina Giunchiglia](mailto:v.giunchiglia20@imperial.ac.uk) and
+[Marinka Zitnik](mailto:marinka@hms.harvard.edu).

config.json

@@ -0,0 +1,109 @@
+{
+  "dataset": "HLCA",
+  "architecture": "TransformerGenerator",
+  "d_model": 384,
+  "nhead": 6,
+  "nlayers": 12,
+  "dim_embedding": 384,
+  "dropout": 0.1,
+  "nclass": 2,
+  "pad_token": "<pad>",
+  "cell_emb_style": "avg-pool",
+  "context_specific_projections": true,
+  "constant_ctx": false,
+  "only_contrastive": true,
+  "DA": true,
+  "lambda_sinkhorn": 0.005,
+  "contexts": [
+    "disease",
+    "cell_type",
+    "tissue"
+  ],
+  "merged_contexts": "disease_cell_type_tissue",
+  "vocab_size": 18711,
+  "max_length_up": 1024,
+  "max_length_down": 1024,
+  "temperature": 0.1,
+  "seed": 20,
+  "training_epoch": 2,
+  "training_global_step": 3682000,
+  "data_composition": {
+    "diseases": [
+      "COVID-19",
+      "chronic obstructive pulmonary disease",
+      "chronic rhinitis",
+      "cystic fibrosis",
+      "hypersensitivity pneumonitis",
+      "interstitial lung disease",
+      "lung adenocarcinoma",
+      "lung large cell carcinoma",
+      "lymphangioleiomyomatosis",
+      "non-specific interstitial pneumonia",
+      "normal",
+      "pleomorphic carcinoma",
+      "pneumonia",
+      "pulmonary fibrosis",
+      "pulmonary sarcoidosis",
+      "squamous cell lung carcinoma"
+    ],
+    "tissues": [
+      "lung",
+      "lung parenchyma",
+      "nose",
+      "respiratory airway"
+    ],
+    "cell_type_ontology_ids": [
+      "CL:0000037",
+      "CL:0000057",
+      "CL:0000077",
+      "CL:0000084",
+      "CL:0000097",
+      "CL:0000158",
+      "CL:0000186",
+      "CL:0000192",
+      "CL:0000236",
+      "CL:0000313",
+      "CL:0000319",
+      "CL:0000451",
+      "CL:0000499",
+      "CL:0000583",
+      "CL:0000622",
+      "CL:0000623",
+      "CL:0000624",
+      "CL:0000625",
+      "CL:0000784",
+      "CL:0000786",
+      "CL:0000860",
+      "CL:0000861",
+      "CL:0000875",
+      "CL:0000990",
+      "CL:0002062",
+      "CL:0002063",
+      "CL:0002075",
+      "CL:0002138",
+      "CL:0002144",
+      "CL:0002145",
+      "CL:0002399",
+      "CL:0002480",
+      "CL:0002543",
+      "CL:0002632",
+      "CL:0002633",
+      "CL:0005006",
+      "CL:0005012",
+      "CL:0009089",
+      "CL:0010003",
+      "CL:0019001",
+      "CL:0019003",
+      "CL:0019019",
+      "CL:1000223",
+      "CL:1000312",
+      "CL:1001568",
+      "CL:1001603",
+      "CL:2000093",
+      "CL:4028004",
+      "CL:4028006",
+      "CL:4030023",
+      "unknown"
+    ]
+  }
+}

median_genes_all_all_HLCA.pkl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:15ad9a57844d311bcb025aa4ed5b87e4ca7518b45fcd33308b1bea934ed911be
+size 617483

median_genes_cells_all_HLCA.pkl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0a9f4b20cda2ba2cd0edfbf6756d92f27005c66acd3cdae29384a99e3e46621f
+size 19318975

median_genes_disease_all_HLCA.pkl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9f3f3d6074a8ac46c4c4a377baf371d0a6d7f920d8c332bb0743183a755e4f90
+size 6228132

median_genes_tissue_all_HLCA.pkl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c1ae8c02e822e46f8839b818e426795a24ef67c7234fedfc2548976eece46b72
+size 1739601

metadata_dictionary_HLCA.pkl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0dfce23239217b47b1151bc441ac71a6c9e9191e43cdf12f720b662b403a6639
+size 1368

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:81116c256f93ba8fd4d4e5018f0354674ed6640545148af8dec9111f308d8872
+size 90805088

tokenizer_dictionary_HLCA.pkl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fb57eebe837569c0189b992bc2f7465b85a51f3475c3ede97e80bad6141fd0a9
+size 392712