Add single-cell modality and COVID severity task

README.md

@@ -17,8 +17,8 @@ thumbnail: https://huggingface.co/spaces/ScientaLab/primo-eval/resolve/main/asse
 **A blind benchmark for omics foundation models.**
 
 PRIMO grades how well a model turns a patient's omics data into a useful
-**patient embedding**. You embed every dataset and upload **one** file; a fixed
-fixed probe scores each hidden task, and the results roll up into a blind,
+**patient embedding**. You select one modality, embed its datasets, and upload
+**one** file; a fixed probe scores each hidden task, and the results roll up into a blind,
 per-category leaderboard. The datasets are opaque (`d001`, `d002`…) and you never
 see the disease, tissue, or target, which leaves you grading the *embedding*
 itself with no room for per-task tuning.
@@ -42,20 +42,22 @@ that does not exist.
 
 ## What's in the data
 
-PRIMO benchmarks any omics modality. Today's datasets are all **bulk RNA**,
-covering **immune-mediated inflammatory diseases (IMIDs)** with real clinical
-labels or treatment-induced expression responses from published cohorts:
+PRIMO benchmarks any omics modality. It includes **bulk RNA** tasks across
+immune-mediated inflammatory diseases and a **single-cell RNA** COVID-19 PBMC
+severity task, all with real clinical labels or treatment-induced expression
+responses from published cohorts:
 
 - **Gastroenterology**: Crohn's disease, ulcerative colitis (anti-TNF response, severity scores)
 - **Dermatology**: atopic dermatitis, psoriasis (severity scores)
 - **Rheumatology**: rheumatoid arthritis (joint counts, molecular endotype)
+- **Infectious diseases**: COVID-19 severity from single-cell PBMC expression
 - **Perturbation response**: adalimumab transfer across inflammatory skin
   diseases, rituximab response in Sjögren salivary gland, and mouse intestinal
   anti-TNF response
 
 ## Submission format
 
-One file, one row per (`dataset_id`, `sample_id`), spanning all datasets:
+One file, one row per (`dataset_id`, `sample_id`), spanning one modality:
 
 - **CSV / TSV / Parquet**: a `dataset_id` column, a `sample_id` column, and one
   numeric column per embedding dimension. Embedding dim may differ per dataset
@@ -68,6 +70,12 @@ listed in the public `datasets.yaml` manifest. Alignment is by join, so row
 order does not matter; every labelled sample of a task must be present with no
 NaN/inf, or that task is skipped.
 
+Bulk H5AD files have one row per required submission sample. Single-cell H5AD
+files have one sparse raw-count row per cell; opaque cell ids are in
+`obs_names`, and the only public cell metadata is `obs["sample_id"]`, which maps
+each cell to its opaque collection sample. Submissions remain sample-level: emit
+exactly one embedding for every unique `sample_id`, not one embedding per cell.
+
 ## How it works
 
 Each dataset is embedded once and scored on every hidden task defined for it. Per
@@ -108,14 +116,16 @@ in each board's **per-task** table even when they are not ranked.
 
 ## Make a submission
 
-`quickstart.py` is the shortest path: it downloads every dataset, embeds each one
-(log2(CPM+1) → PCA) and writes the file the Submit tab wants. Swap its `embed`
+`quickstart.py` is the shortest path: it downloads one modality and embeds its
+datasets
+(log2(CPM+1) → PCA for bulk; per-cell log2(CP10K+1) → sample mean → PCA for
+single-cell) and writes the file the Submit tab wants. Swap its `embed`
 function for your encoder and nothing else changes. `example_submission.csv`
 shows the expected shape in four lines.
 
 ```bash
 pip install anndata scikit-learn pandas pyyaml huggingface_hub
-python quickstart.py --out submission.parquet
+python quickstart.py --modality bulk-rna --out submission.parquet
 ```
 
 ## Run the scorer locally
@@ -123,7 +133,7 @@ python quickstart.py --out submission.parquet
 ```bash
 pip install -r requirements.txt
 export HF_TOKEN=...   # read access to the PRIMO datasets
-python evaluator.py --submission my_embeddings.parquet
+python evaluator.py --modality bulk-rna --submission my_embeddings.parquet
 ```
 
 ## Baselines

app.py

@@ -7,7 +7,7 @@ do I enter?"), Contribute ("what is missing, and how do I add it?"), Method
 board and ``?tab=contribute`` opens a tab -- which is what the rail links and the
 open cards use. The tab strip is hidden in CSS; the rail is the navigation.
 
-Upload one embedding file spanning every dataset (rows keyed by ``dataset_id``
+Choose one modality and upload its embedding file (rows keyed by ``dataset_id``
 + ``sample_id``); a fixed task probe scores each task (a dataset may carry
 several hidden targets). Results roll up into boards -- the whole modality, one
 therapeutic area, one task family -- and each board ranks the models that
@@ -94,10 +94,15 @@ THEME = gr.themes.Base(
 
 SUBMIT_HEAD = (
     '<div class="pm-head"><div><h1>Submit a model</h1>'
-    "<p>One embedding file. Partial coverage is fine. You are ranked on every "
-    "board you cover in full.</p></div></div>"
+    "<p>One model, one modality. Partial coverage is fine within that modality. "
+    "You are ranked on every board you cover in full.</p></div></div>"
 )
 
+MODALITY_CHOICES = [
+    ("Bulk RNAseq", "bulk-rna"),
+    ("Single-cell RNAseq", "single-cell-rna"),
+]
+
 TABLE_SORT_JS = """
 () => {
   if (window.pmTableSortBound) return;
@@ -180,6 +185,16 @@ def _page_text(name: str) -> str:
     return (PAGES_DIR / f"{name}.md").read_text()
 
 
+def _download_help(modality: str | None) -> str:
+    """Show the quickstart command for the selected modality."""
+    if not modality:
+        return "Select a modality to get its download command."
+    return (
+        f"```bash\npython quickstart.py --modality {modality} "
+        "--out submission.parquet\n```"
+    )
+
+
 def _registry_by_id() -> dict[str, dict]:
     """Scoreable tasks keyed by task_id (dataset present in the public manifest).
 
@@ -270,6 +285,7 @@ def _rendered_board(slug: str | None) -> str:
 
 def evaluate(
     submission_path: str,
+    modality: str | None,
     model_name: str,
     institution: str,
     is_author_submission: bool,
@@ -285,6 +301,8 @@ def evaluate(
 
     if profile is None:
         return _refuse("Please sign in with Hugging Face to submit.")
+    if not modality:
+        return _refuse("Please select a modality.")
     if not submission_path:
         return _refuse("Please upload a submission file.")
     if not model_name or not model_name.strip():
@@ -309,7 +327,7 @@ def evaluate(
             "board keeps each name's latest submission. Please pick another name."
         )
     try:
-        result = score_all(submission_path, TOKEN)
+        result = score_all(submission_path, TOKEN, modality=modality)
     except SubmissionError as error:
         return _refuse(f"❌ {error}")
     except EvaluatorError as error:
@@ -441,6 +459,12 @@ def build_demo() -> gr.Blocks:
                                     gr.Markdown(_page_text("submit"))
                                 with gr.Column(scale=2, elem_id="pm-form"):
                                     gr.LoginButton()
+                                    modality_in = gr.Radio(
+                                        MODALITY_CHOICES,
+                                        label="Modality",
+                                        info="One model submission covers one modality.",
+                                    )
+                                    download_md = gr.Markdown(_download_help(None))
                                     model_tb = gr.Textbox(
                                         label="Model name",
                                         placeholder="e.g. eva-rna-v1",
@@ -508,6 +532,7 @@ def build_demo() -> gr.Blocks:
             evaluate,
             [
                 file_in,
+                modality_in,
                 model_tb,
                 institution_tb,
                 author_submission_cb,
@@ -525,6 +550,7 @@ def build_demo() -> gr.Blocks:
             status_html,
             show_progress="hidden",
         )
+        modality_in.change(_download_help, modality_in, download_md)
         demo.load(
             _init,
             None,

boards.py

@@ -39,7 +39,7 @@ METRIC_LABEL = {
 }
 MODALITY_LABEL = {
     "bulk RNA": "Bulk RNAseq",
-    "single-cell RNA": "Single-cell RNAseq",
+    "Single Cell RNA": "Single-cell RNAseq",
 }
 
 MODALITY_GROUP = "Per modality"
@@ -165,8 +165,8 @@ def _cohort_stats(tasks: list[dict]) -> tuple[int, int, int]:
     diseases: set[str] = set()
     for task in tasks:
         cohort = str(task.get("cohort_id") or _norm_id(task.get("dataset_id", "")))
-        n_samples = int(task.get("n_samples") or 0)
-        patients_by_cohort[cohort] = max(patients_by_cohort.get(cohort, 0), n_samples)
+        n_subjects = int(task.get("n_subjects") or task.get("n_samples") or 0)
+        patients_by_cohort[cohort] = max(patients_by_cohort.get(cohort, 0), n_subjects)
         diseases.update(str(d) for d in task.get("diseases") or [])
     return (
         len(patients_by_cohort),

evaluator.py

@@ -1,6 +1,6 @@
 """Standalone probe for the PRIMO public benchmark.
 
-Loads one embedding submission that spans every dataset (rows keyed by
+Loads one embedding submission for one modality (rows keyed by
 ``dataset_id`` + ``sample_id``), then scores each TASK = (dataset, target): a
 dataset is embedded once and reused across all its tasks. Per task it fits a
 fixed probe—linear for scalar targets and multi-output ridge for response
@@ -71,6 +71,7 @@ MANIFEST_FILENAME = "datasets.yaml"
 TASKS_FILENAME = "tasks.yaml"
 LABELS_FILENAME = "labels.csv"
 TARGETS_FILENAME = "targets.npz"
+MODALITIES = {"bulk-rna": "bulk RNA", "single-cell-rna": "Single Cell RNA"}
 
 RIDGE_ALPHAS = np.logspace(-3.0, 6.0, 19)
 PERTURBATION_TEST_Z_CLIP = 20.0
@@ -652,9 +653,13 @@ def fetch_manifest(token: str | None = None) -> list[dict]:
     return data or []
 
 
-def manifest_ids(manifest: list[dict]) -> set[str]:
-    """The canonical set of valid dataset ids from the manifest."""
-    return {_norm_id(entry["id"]) for entry in manifest}
+def manifest_ids(manifest: list[dict], modality: str | None = None) -> set[str]:
+    """Return manifest dataset ids, optionally for one modality."""
+    return {
+        _norm_id(entry["id"])
+        for entry in manifest
+        if modality is None or entry.get("modality") == modality
+    }
 
 
 def scoreable_tasks(tasks: list[dict], valid_ids: set[str]) -> list[dict]:
@@ -765,6 +770,7 @@ def score_all(
     path: str | Path,
     token: str | None = None,
     *,
+    modality: str | None = None,
     datasets: list[dict] | None = None,
     tasks: list[dict] | None = None,
     fetch_labels=None,
@@ -784,11 +790,21 @@ def score_all(
             raise EvaluatorError(
                 f"could not load the dataset manifest: {error}"
             ) from error
-    valid = manifest_ids(datasets)
-
-    unknown = sorted(set(blocks) - valid)
+    if modality is not None and modality not in MODALITIES:
+        raise SubmissionError(f"unknown or unavailable modality: {modality}")
+    all_ids = manifest_ids(datasets)
+    valid = manifest_ids(datasets, MODALITIES.get(modality))
+    if modality is not None and not valid:
+        raise SubmissionError(f"unknown or unavailable modality: {modality}")
+
+    unknown = sorted(set(blocks) - all_ids)
     if unknown:
         raise SubmissionError(f"unknown dataset_id(s) not in the benchmark: {unknown}")
+    wrong_modality = sorted(set(blocks) - valid)
+    if wrong_modality:
+        raise SubmissionError(
+            f"dataset_id(s) outside the selected modality: {wrong_modality}"
+        )
 
     if tasks is None:
         try:
@@ -816,7 +832,8 @@ def score_all(
             fetch_targets,
             token,
         )
-        for task in scoreable_tasks(tasks, valid)
+        for task in scoreable_tasks(tasks, all_ids)
+        if _norm_id(task[DATASET_ID]) in valid
     ]
     return _summarize(outcomes)
 
@@ -868,11 +885,12 @@ def _dataset_status(outcomes: list[TaskOutcome]) -> dict[str, str]:
 def _cli() -> None:
     parser = argparse.ArgumentParser(description="Score a PRIMO submission locally.")
     parser.add_argument("--submission", required=True, help="CSV/TSV/Parquet/NPZ file")
+    parser.add_argument("--modality", required=True, choices=MODALITIES)
     parser.add_argument("--token", default=None, help="HF token (else env HF_TOKEN)")
     args = parser.parse_args()
 
     token = args.token or os.environ.get("HF_TOKEN")
-    result = score_all(args.submission, token)
+    result = score_all(args.submission, token, modality=args.modality)
     print(
         f"scored     : {result['n_datasets_scored']}/{result['n_datasets_total']} "
         f"datasets, {result['n_scored']}/{result['n_total']} tasks  "

leaderboard.py

@@ -505,7 +505,7 @@ def tasks_table(tasks: list[dict]) -> pd.DataFrame:
                 "Area": str(task.get("therapeutic_area", "")),
                 "Disease": ", ".join(str(d) for d in task.get("diseases") or []),
                 "Tissue": str(task.get("tissue", "")),
-                "Patients": task.get("n_samples"),
+                "Patients": task.get("n_subjects") or task.get("n_samples"),
                 "Metric": metric_label(str(task.get("metric", ""))),
             }
         )

pages/submit.md

@@ -1,22 +1,24 @@
 **The fastest way in.** Two files, both in this [Space's repo](https://huggingface.co/spaces/ScientaLab/primo-eval/blob/main/quickstart.py):
 
 - 📥 [`quickstart.py`](https://huggingface.co/spaces/ScientaLab/primo-eval/blob/main/quickstart.py):
-  downloads every dataset, embeds them, writes a valid submission. Swap its
-  `embed` function for your model and you are done.
+  downloads the selected modality, embeds it, and writes a valid submission.
+  Swap its `embed` function for your model and you are done.
 - 📄 [`example_submission.csv`](https://huggingface.co/spaces/ScientaLab/primo-eval/blob/main/example_submission.csv):
   four lines, fake numbers, the exact shape we expect.
 
 ```bash
 pip install anndata scikit-learn pandas pyyaml huggingface_hub
-python quickstart.py --out submission.parquet
+python quickstart.py --modality bulk-rna --out submission.parquet
+# or: --modality single-cell-rna
 ```
 
 ---
 
 Or do it by hand, in **three steps**:
 
-1. **Get the data** → download the datasets from [ScientaLab/primo](https://huggingface.co/datasets/ScientaLab/primo) (start with its `datasets.yaml`).
-2. **Embed every dataset** → build **one** file: `dataset_id`, `sample_id`, then one column per embedding dim (`e0`, `e1`, …). CSV / TSV / Parquet, or NPZ.
+1. **Choose one modality** → use the selector in the form. The quickstart
+   downloads only its datasets from [ScientaLab/primo](https://huggingface.co/datasets/ScientaLab/primo).
+2. **Embed that modality** → build **one** file: `dataset_id`, `sample_id`, then one column per embedding dim (`e0`, `e1`, …). CSV / TSV / Parquet, or NPZ.
 3. **Sign in, fill the form, and hit Evaluate.** Add an institution for group submissions, check **Submitted by the model's authors** when applicable, and provide a paper link to make the model name clickable. A fixed task probe scores each hidden task (AUROC, Pearson or centered Spearman), reported per task category in its native metric.
 
 **Example file**
@@ -27,7 +29,14 @@ d001,S1,0.12,-0.44,0.98
 d002,S1,0.31,0.02,-0.15
 ```
 
-**Partial submissions are welcome.** Cover fewer datasets and you are still
+For bulk datasets, each H5AD row is one submission sample. For single-cell
+datasets, H5AD rows are cells and `obs["sample_id"]` maps them to opaque
+collection samples. Aggregate the cells however your model requires and submit
+exactly one embedding per unique `sample_id`; the submission schema is unchanged.
+
+Files containing dataset IDs from another modality are rejected. Reusing a model
+name for another modality replaces its previous leaderboard entry. **Partial
+submissions are welcome.** Cover fewer datasets within the selected modality and you are still
 scored: you get ranked on every **board** whose scored tasks you covered in
 full, and your numbers still show up in each board's **per-task** table, so
 nothing you send is thrown away.

quickstart.py

@@ -1,12 +1,13 @@
 """Produce a valid PRIMO submission in one command, then swap in your own model.
 
-Downloads every public dataset, embeds each one, and writes the single file the
-Submit tab expects. The embedding here is deliberately dumb -- log2(CPM+1) then
-PCA -- because the point is the plumbing, not the score: replace ``embed`` with
-your encoder and nothing else changes.
+Downloads one modality's public datasets and writes the file the Submit tab
+expects. The embedding here is deliberately dumb: bulk data uses
+log2(CPM+1); single-cell data uses per-cell log2(CP10K+1), mean pooling by
+collection sample, then PCA. Replace ``embed`` with your encoder and nothing else
+changes.
 
     pip install anndata scikit-learn pandas pyyaml huggingface_hub
-    python quickstart.py --out submission.parquet
+    python quickstart.py --modality bulk-rna --out submission.parquet
 
 Standalone on purpose: no import from this Space and none from our monorepo, so
 it keeps working if you copy the file into your own project.
@@ -19,64 +20,106 @@ import anndata as ad
 import numpy as np
 import pandas as pd
 import yaml
-from huggingface_hub import snapshot_download
+from huggingface_hub import hf_hub_download, snapshot_download
 from sklearn.decomposition import PCA
 
 PUBLIC_REPO = "ScientaLab/primo"
 MANIFEST_FILENAME = "datasets.yaml"
+MODALITIES = {"bulk-rna": "bulk RNA", "single-cell-rna": "Single Cell RNA"}
 
 DATASET_ID = "dataset_id"
 SAMPLE_ID = "sample_id"
 
 TARGET_SUM = 1_000_000
+SINGLE_CELL_TARGET_SUM = 10_000
 N_COMPONENTS = 50
 RANDOM_STATE = 0
 
 
+def prediction_ids(adata: ad.AnnData) -> np.ndarray:
+    """Opaque collection-sample ids required in the submission."""
+    if SAMPLE_ID in adata.obs:
+        return pd.unique(adata.obs[SAMPLE_ID].astype(str)).astype(str)
+    return adata.obs_names.astype(str).to_numpy()
+
+
+def _log_normalize(x, target_sum: int) -> np.ndarray:
+    """Return dense log2 counts-per-target expression."""
+    x = x.toarray() if hasattr(x, "toarray") else np.asarray(x)
+    totals = x.sum(axis=1, keepdims=True)
+    return np.log2(x / np.where(totals == 0, 1, totals) * target_sum + 1)
+
+
+def _sample_expression(adata: ad.AnnData) -> tuple[np.ndarray, np.ndarray]:
+    """Normalize expression and mean-pool cells into collection samples."""
+    ids = prediction_ids(adata)
+    if SAMPLE_ID not in adata.obs:
+        return ids, _log_normalize(adata.X, TARGET_SUM)
+    samples = adata.obs[SAMPLE_ID].astype(str).to_numpy()
+    pooled = [
+        _log_normalize(adata.X[samples == sample_id], SINGLE_CELL_TARGET_SUM).mean(0)
+        for sample_id in ids
+    ]
+    return ids, np.asarray(pooled)
+
+
 def embed(adata: ad.AnnData) -> np.ndarray:
-    """One dataset's raw counts -> one vector per patient. Replace me.
+    """One dataset's raw counts -> one vector per collection sample. Replace me.
 
-    Whatever you return, the contract is the same: one row per sample, in
-    ``adata.obs_names`` order, all finite. The embedding width is yours to pick
-    and may differ from one dataset to the next.
+    Whatever you return, the contract is one finite row per id returned by
+    ``prediction_ids``. The embedding width may differ between datasets.
     """
-    x = adata.X
-    x = x.toarray() if hasattr(x, "toarray") else np.asarray(x)
-    x = x.astype(float)
-    counts = x.sum(axis=1, keepdims=True)
-    x = np.log2(x / np.where(counts == 0, 1, counts) * TARGET_SUM + 1)
+    _, x = _sample_expression(adata)
     k = min(N_COMPONENTS, x.shape[0] - 1, x.shape[1])
     return PCA(n_components=k, random_state=RANDOM_STATE).fit_transform(x)
 
 
-def download(token: str | None) -> Path:
-    """Pull the public benchmark (manifest + every ``expression.h5ad``)."""
-    return Path(snapshot_download(PUBLIC_REPO, repo_type="dataset", token=token))
+def download(modality: str, token: str | None) -> Path:
+    """Download the manifest and expression files for one modality."""
+    manifest = Path(
+        hf_hub_download(
+            PUBLIC_REPO, MANIFEST_FILENAME, repo_type="dataset", token=token
+        )
+    )
+    paths = [
+        MANIFEST_FILENAME,
+        *[str(entry["path"]) for entry in datasets(manifest, modality)],
+    ]
+    return Path(
+        snapshot_download(
+            PUBLIC_REPO, repo_type="dataset", token=token, allow_patterns=paths
+        )
+    )
 
 
-def dataset_ids(root: Path) -> list[str]:
-    """The opaque ids to embed, read off the public manifest."""
-    manifest = yaml.safe_load((root / MANIFEST_FILENAME).read_text())
+def datasets(manifest_path: Path, modality: str) -> list[dict]:
+    """Manifest entries for one modality."""
+    manifest = yaml.safe_load(manifest_path.read_text())
     entries = manifest.get("datasets", []) if isinstance(manifest, dict) else manifest
-    return [str(entry["id"]) for entry in entries]
+    return [entry for entry in entries if entry["modality"] == modality]
 
 
-def build(root: Path) -> pd.DataFrame:
-    """Embed every dataset into the one frame the Submit tab expects.
+def build(root: Path, modality: str) -> pd.DataFrame:
+    """Embed one modality into the frame the Submit tab expects.
 
     Datasets of different widths stack into one table; the extra columns of a
     narrower dataset stay empty and the evaluator drops them per dataset, so each
     dataset keeps its own embedding size.
     """
     blocks = []
-    for dataset_id in dataset_ids(root):
-        adata = ad.read_h5ad(root / dataset_id / "expression.h5ad")
+    for entry in datasets(root / MANIFEST_FILENAME, modality):
+        dataset_id = str(entry["id"])
+        adata = ad.read_h5ad(root / entry["path"])
+        sample_ids = prediction_ids(adata)
         vectors = embed(adata)
-        print(f"{dataset_id}: {adata.n_obs} samples -> {vectors.shape[1]} dims")
+        print(
+            f"{dataset_id}: {adata.n_obs} observations, {len(sample_ids)} samples "
+            f"-> {vectors.shape[1]} dims"
+        )
         block = pd.DataFrame(
             vectors, columns=[f"e{i}" for i in range(vectors.shape[1])]
         )
-        block.insert(0, SAMPLE_ID, adata.obs_names.to_numpy())
+        block.insert(0, SAMPLE_ID, sample_ids)
         block.insert(0, DATASET_ID, dataset_id)
         blocks.append(block)
     return pd.concat(blocks, ignore_index=True)
@@ -84,11 +127,13 @@ def build(root: Path) -> pd.DataFrame:
 
 def main() -> None:
     parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
+    parser.add_argument("--modality", required=True, choices=MODALITIES)
     parser.add_argument("--out", type=Path, default=Path("submission.parquet"))
     parser.add_argument("--token", default=None, help="HF token, if you need one.")
     args = parser.parse_args()
 
-    submission = build(download(args.token))
+    modality = MODALITIES[args.modality]
+    submission = build(download(modality, args.token), modality)
     if args.out.suffix == ".csv":
         submission.to_csv(args.out, index=False)
     else:

render.py

@@ -343,12 +343,19 @@ def _task_tooltip(task: dict) -> str:
     diseases = (
         ", ".join(str(d) for d in task.get("diseases") or []) or "the listed cohort"
     )
-    patients = task.get("n_samples")
+    patients = task.get("n_subjects") or task.get("n_samples")
     patient_text = (
         f"{patients:,} patients"
         if isinstance(patients, int)
         else "an unspecified number of patients"
     )
+    n_samples = task.get("n_samples")
+    if (
+        isinstance(n_samples, int)
+        and isinstance(patients, int)
+        and n_samples != patients
+    ):
+        patient_text += f" ({n_samples:,} collection samples)"
     modality = str(task.get("modality") or "omics")
     tissue = str(task.get("tissue") or "unspecified tissue")
     target = str(task.get("target") or task.get("title") or "the task target")