README.md

---
title: PRIMO Benchmark
emoji: 🧬
colorFrom: indigo
colorTo: blue
sdk: gradio
sdk_version: 5.50.0
python_version: "3.10"
app_file: app.py
pinned: false
hf_oauth: true
thumbnail: https://huggingface.co/spaces/PRIMOmics/primo-eval/resolve/main/assets/primo-social-preview.png
---

# PRIMO: Patient Representations in Multi-Omics

**A blind benchmark for omics foundation models.**

PRIMO grades how well a model turns a patient's omics data into a useful
**patient embedding**. You select one modality, embed its datasets, and upload
**one** file; a fixed probe scores each hidden task, and the results roll up into a blind,
per-category leaderboard. The datasets are opaque (`d001`, `d002`…) and you never
see the disease, tissue, or target, which leaves you grading the *embedding*
itself with no room for per-task tuning.

A left **rail** navigates six pages: **Boards** (a grid of every board),
**Board** (one leaderboard at a time), **Tasks**, a **Submit** form (sign in with
Hugging Face), **Contribute**, and **Method**. Every board has its own URL,
`?board=rheumatology-bulk-rna`, and every page too, as `?tab=contribute`, so the
rail links and the open cards are shareable deep links.

Boards also shows **open boards**: greyed-out cards for the omics layers and the
therapeutic areas PRIMO does not cover yet, each linking to Contribute. They are
declared in `boards.py` (`OPEN_BOARDS`) and drop out on their own once the
registry covers that slice. Task categories deliberately get none, because a
category is pinned to a single metric, so an open one would advertise a probe
that does not exist.

🌐 Website: http://primomics.org/ ·
📄 Paper: https://openreview.net/forum?id=v2SA8gHwqo ·
📦 Data: https://huggingface.co/datasets/PRIMOmics/primo

## What's in the data

PRIMO benchmarks any omics modality. It includes **bulk RNA** tasks across
immune-mediated inflammatory diseases and a **single-cell RNA** COVID-19 PBMC
severity task, all with real clinical labels or treatment-induced expression
responses from published cohorts:

- **Gastroenterology**: Crohn's disease, ulcerative colitis (anti-TNF response, severity scores)
- **Dermatology**: atopic dermatitis, psoriasis (severity scores)
- **Rheumatology**: rheumatoid arthritis (joint counts, molecular endotype)
- **Infectious diseases**: COVID-19 severity from single-cell PBMC expression
- **Perturbation response**: adalimumab transfer across inflammatory skin
  diseases, rituximab response in Sjögren salivary gland, and mouse intestinal
  anti-TNF response

## Submission format

One file, one row per (`dataset_id`, `sample_id`), spanning one modality:

- **CSV / TSV / Parquet**: a `dataset_id` column, a `sample_id` column, and one
  numeric column per embedding dimension. Embedding dim may differ per dataset
  (pad short datasets with blank columns; blank/NaN padding columns are dropped
  per dataset).
- **NPZ**: `dataset_ids`, `sample_ids`, and a 2-D `embeddings` array.

The valid `dataset_id`s and how to download each dataset's `expression.h5ad` are
listed in the public `datasets.yaml` manifest. Alignment is by join, so row
order does not matter; every labelled sample of a task must be present with no
NaN/inf, or that task is skipped.

Bulk H5AD files have one row per required submission sample. Single-cell H5AD
files have one sparse raw-count row per cell; opaque cell ids are in
`obs_names`, and the only public cell metadata is `obs["sample_id"]`, which maps
each cell to its opaque collection sample. Submissions remain sample-level: emit
exactly one embedding for every unique `sample_id`, not one embedding per cell.

## How it works

Each dataset is embedded once and scored on every hidden task defined for it.
Rows are scaled to unit length, then per task: standardise on the training folds,
fit `LogisticRegressionCV` (classification) or `RidgeCV` (regression) with the
penalty chosen by repeated inner cross-validation, and predict each held-out
fold. We calculate the native metric per fold and average it over frozen,
subject-grouped partitions: **AUROC** (classification) or **Pearson r**
(regression). A few tasks instead use a fixed train/test split: the probe is fit
once on the training portion and scored on the held-out portion.

Tasks are grouped into **four families**, each reported in its own native
metric. Two families are never merged into one column:

- **Treatment outcome**: response to anti-TNF therapy (**AUROC**)
- **Clinical scores**: disease-severity regression (**Pearson r**)
- **Endotype**: molecular-subtype classification (**AUROC**)
- **Perturbation response**: hidden post-minus-pre DEG decoding (**centered
  Spearman**), where 1 is perfect, 0 is uninformative, and -1 is an inverse gene
  ranking.

A leaderboard shows one column per family. A board holding more than one family
also shows **Mean**, the average of those columns. It is what orders the rows,
but it does mix AUROC, Pearson and centered Spearman, so treat it as a tie-break and
compare models on the family columns.

## Boards

Results are shown as **boards**, self-contained leaderboards over a slice of the
registry: the whole modality, one therapeutic area, one task family. A board
ranks only the models that covered **all of its tasks**, so a submission that
skipped Dermatology is still ranked on Rheumatology. Modality is where we draw
the line: an area or category board never spans two modalities, because an AUROC
on bulk RNA and an AUROC on single-cell are not measuring the same thing.

Partial and failed submissions still get feedback, and their scores always appear
in each board's **per-task** table even when they are not ranked.

## Make a submission

`quickstart.py` is the shortest path: it downloads one modality and embeds its
datasets
(log2(CPM+1) → PCA for bulk; per-cell log2(CP10K+1) → sample mean → PCA for
single-cell) and writes the file the Submit tab wants. Swap its `embed`
function for your encoder and nothing else changes. `example_submission.csv`
shows the expected shape in four lines.

```bash
pip install anndata scikit-learn pandas pyyaml huggingface_hub
python quickstart.py --modality bulk-rna --out submission.parquet
```

## Run the scorer locally

```bash
pip install -r requirements.txt
export HF_TOKEN=...   # read access to the PRIMO datasets
python evaluator.py --modality bulk-rna --submission my_embeddings.parquet
```

## Baselines

`task_results.csv` carries an `is_baseline` flag. Reference submissions we
produce ourselves are published with it set, rendered as `name (baseline)`, and
**ranked in place**. A foundation model losing to raw log-CPM expression is
exactly the result worth publishing, so we keep it in the table rather than
tucked underneath. The published `HVG-1200-genes` baseline uses the 1,200
highest-variance genes after log2(CPM+1), without gene scaling. Its selection is
label-blind but fit over every sample of a dataset.

## Space configuration

- **`hf_oauth: true`** (set above) turns on the Submit tab's *Sign in with
  Hugging Face* button; submitting requires a logged-in HF account.
- Set an **`HF_TOKEN`** Space secret (fine-grained) with: **read** on
  `PRIMOmics/primo` (the public `datasets.yaml` manifest) and
  `PRIMOmics/primo-labels` (the private `tasks.yaml` registry +
  `<task_id>/labels.csv` and perturbation `<task_id>/targets.npz`), and **read + write** on
  `PRIMOmics/primo-results` (the persisted leaderboard).
- Results persist as one normalized `task_results.csv` (`model_name, task_id,
  score, repeat_scores, diagnostics, submitted_at, is_baseline, hf_username`) in the results dataset; the
  leaderboard is recomputed from it by joining the registry, so it survives
  Space restarts.
- A board keeps the **latest result for each `(model name, is_baseline, task)`**,
  so submissions accumulate: a later one fills in the tasks it covers and leaves
  the rest standing, and resubmitting a task replaces just that result. Because a
  task belongs to exactly one modality, one name can hold bulk and single-cell
  results at once without either displacing the other.
- A model name is owned by the account that first submitted it: `hf_username`
  locks it, and Submit refuses a name somebody else holds. It is never rendered
  anywhere: it works as a lock rather than as a credit. Names colliding with a
  per-task column (`Task`, `Family`, `Area`, `Modality`, `Metric`, `Best`) are
  refused too, since model names become column headers.
- The boards show a **Submissions count** per model — the most times any one of
  its task results was resubmitted — so a task tuned against by repeated
  resubmission is visible. Nothing is ever blocked.
- Submitter contact metadata (HF username, email, paper / model links, notes)
  persists to a separate `submissions.csv` in the same **private** results
  dataset, and never reaches the public leaderboard.

## Moving to another Hugging Face org

The three dataset repos are derived from one constant, `ORG` in `evaluator.py`.
The rest of the org name is spelled out and has to be changed by hand:

- `SPACE_REPO` in `deploy_space.py`
- `PUBLIC_REPO` in `quickstart.py` (standalone, so it cannot import `ORG`)
- `SOCIAL_PREVIEW_URL` in `app.py`
- the links in this file — including the YAML `thumbnail:` above — and in
  `pages/*.md`, `../public_dataset_README.md` (the dataset card) and
  `../HF_SETUP.md`

That list is exhaustive as of the `ScientaLab` → `PRIMOmics` move; `app.py` and
the dataset card were the two it missed the first time.

The theme follows the PRIMO charter: Funnel Display for headings, Funnel Sans for
everything else, Scienta Navy `#080F5F` / PRIMO Cyan `#16B3C0` on Paper
`#F3F8F8`. `colorFrom`/`colorTo` above stay `indigo`/`blue` because Hugging Face
only accepts eight named colours and none of them is cyan.