--- title: PRIMO Benchmark emoji: 🧬 colorFrom: indigo colorTo: blue sdk: gradio sdk_version: 5.50.0 python_version: "3.10" app_file: app.py pinned: false hf_oauth: true thumbnail: https://huggingface.co/spaces/PRIMOmics/primo-eval/resolve/main/assets/primo-social-preview.png --- # PRIMO: Patient Representations in Multi-Omics **A blind benchmark for omics foundation models.** PRIMO grades how well a model turns a patient's omics data into a useful **patient embedding**. You select one modality, embed its datasets, and upload **one** file; a fixed probe scores each hidden task, and the results roll up into a blind, per-category leaderboard. The datasets are opaque (`d001`, `d002`…) and you never see the disease, tissue, or target, which leaves you grading the *embedding* itself with no room for per-task tuning. A left **rail** navigates six pages: **Boards** (a grid of every board), **Board** (one leaderboard at a time), **Tasks**, a **Submit** form (sign in with Hugging Face), **Contribute**, and **Method**. Every board has its own URL, `?board=rheumatology-bulk-rna`, and every page too, as `?tab=contribute`, so the rail links and the open cards are shareable deep links. Boards also shows **open boards**: greyed-out cards for the omics layers and the therapeutic areas PRIMO does not cover yet, each linking to Contribute. They are declared in `boards.py` (`OPEN_BOARDS`) and drop out on their own once the registry covers that slice. Task categories deliberately get none, because a category is pinned to a single metric, so an open one would advertise a probe that does not exist. 🌐 Website: http://primomics.org/ Β· πŸ“„ Paper: https://openreview.net/forum?id=v2SA8gHwqo Β· πŸ“¦ Data: https://huggingface.co/datasets/PRIMOmics/primo ## What's in the data PRIMO benchmarks any omics modality. It includes **bulk RNA** tasks across immune-mediated inflammatory diseases and a **single-cell RNA** COVID-19 PBMC severity task, all with real clinical labels or treatment-induced expression responses from published cohorts: - **Gastroenterology**: Crohn's disease, ulcerative colitis (anti-TNF response, severity scores) - **Dermatology**: atopic dermatitis, psoriasis (severity scores) - **Rheumatology**: rheumatoid arthritis (joint counts, molecular endotype) - **Infectious diseases**: COVID-19 severity from single-cell PBMC expression - **Perturbation response**: adalimumab transfer across inflammatory skin diseases, rituximab response in SjΓΆgren salivary gland, and mouse intestinal anti-TNF response ## Submission format One file, one row per (`dataset_id`, `sample_id`), spanning one modality: - **CSV / TSV / Parquet**: a `dataset_id` column, a `sample_id` column, and one numeric column per embedding dimension. Embedding dim may differ per dataset (pad short datasets with blank columns; blank/NaN padding columns are dropped per dataset). - **NPZ**: `dataset_ids`, `sample_ids`, and a 2-D `embeddings` array. The valid `dataset_id`s and how to download each dataset's `expression.h5ad` are listed in the public `datasets.yaml` manifest. Alignment is by join, so row order does not matter; every labelled sample of a task must be present with no NaN/inf, or that task is skipped. Bulk H5AD files have one row per required submission sample. Single-cell H5AD files have one sparse raw-count row per cell; opaque cell ids are in `obs_names`, and the only public cell metadata is `obs["sample_id"]`, which maps each cell to its opaque collection sample. Submissions remain sample-level: emit exactly one embedding for every unique `sample_id`, not one embedding per cell. ## How it works Each dataset is embedded once and scored on every hidden task defined for it. Rows are scaled to unit length, then per task: standardise on the training folds, fit `LogisticRegressionCV` (classification) or `RidgeCV` (regression) with the penalty chosen by repeated inner cross-validation, and predict each held-out fold. We calculate the native metric per fold and average it over frozen, subject-grouped partitions: **AUROC** (classification) or **Pearson r** (regression). A few tasks instead use a fixed train/test split: the probe is fit once on the training portion and scored on the held-out portion. Tasks are grouped into **four families**, each reported in its own native metric. Two families are never merged into one column: - **Treatment outcome**: response to anti-TNF therapy (**AUROC**) - **Clinical scores**: disease-severity regression (**Pearson r**) - **Endotype**: molecular-subtype classification (**AUROC**) - **Perturbation response**: hidden post-minus-pre DEG decoding (**centered Spearman**), where 1 is perfect, 0 is uninformative, and -1 is an inverse gene ranking. A leaderboard shows one column per family. A board holding more than one family also shows **Mean**, the average of those columns. It is what orders the rows, but it does mix AUROC, Pearson and centered Spearman, so treat it as a tie-break and compare models on the family columns. ## Boards Results are shown as **boards**, self-contained leaderboards over a slice of the registry: the whole modality, one therapeutic area, one task family. A board ranks only the models that covered **all of its tasks**, so a submission that skipped Dermatology is still ranked on Rheumatology. Modality is where we draw the line: an area or category board never spans two modalities, because an AUROC on bulk RNA and an AUROC on single-cell are not measuring the same thing. Partial and failed submissions still get feedback, and their scores always appear in each board's **per-task** table even when they are not ranked. ## Make a submission `quickstart.py` is the shortest path: it downloads one modality and embeds its datasets (log2(CPM+1) β†’ PCA for bulk; per-cell log2(CP10K+1) β†’ sample mean β†’ PCA for single-cell) and writes the file the Submit tab wants. Swap its `embed` function for your encoder and nothing else changes. `example_submission.csv` shows the expected shape in four lines. ```bash pip install anndata scikit-learn pandas pyyaml huggingface_hub python quickstart.py --modality bulk-rna --out submission.parquet ``` ## Run the scorer locally ```bash pip install -r requirements.txt export HF_TOKEN=... # read access to the PRIMO datasets python evaluator.py --modality bulk-rna --submission my_embeddings.parquet ``` ## Baselines `task_results.csv` carries an `is_baseline` flag. Reference submissions we produce ourselves are published with it set, rendered as `name (baseline)`, and **ranked in place**. A foundation model losing to raw log-CPM expression is exactly the result worth publishing, so we keep it in the table rather than tucked underneath. The published `HVG-1200-genes` baseline uses the 1,200 highest-variance genes after log2(CPM+1), without gene scaling. Its selection is label-blind but fit over every sample of a dataset. ## Space configuration - **`hf_oauth: true`** (set above) turns on the Submit tab's *Sign in with Hugging Face* button; submitting requires a logged-in HF account. - Set an **`HF_TOKEN`** Space secret (fine-grained) with: **read** on `PRIMOmics/primo` (the public `datasets.yaml` manifest) and `PRIMOmics/primo-labels` (the private `tasks.yaml` registry + `/labels.csv` and perturbation `/targets.npz`), and **read + write** on `PRIMOmics/primo-results` (the persisted leaderboard). - Results persist as one normalized `task_results.csv` (`model_name, task_id, score, repeat_scores, diagnostics, submitted_at, is_baseline, hf_username`) in the results dataset; the leaderboard is recomputed from it by joining the registry, so it survives Space restarts. - A board keeps the **latest result for each `(model name, is_baseline, task)`**, so submissions accumulate: a later one fills in the tasks it covers and leaves the rest standing, and resubmitting a task replaces just that result. Because a task belongs to exactly one modality, one name can hold bulk and single-cell results at once without either displacing the other. - A model name is owned by the account that first submitted it: `hf_username` locks it, and Submit refuses a name somebody else holds. It is never rendered anywhere: it works as a lock rather than as a credit. Names colliding with a per-task column (`Task`, `Family`, `Area`, `Modality`, `Metric`, `Best`) are refused too, since model names become column headers. - The boards show a **Submissions count** per model β€” the most times any one of its task results was resubmitted β€” so a task tuned against by repeated resubmission is visible. Nothing is ever blocked. - Submitter contact metadata (HF username, email, paper / model links, notes) persists to a separate `submissions.csv` in the same **private** results dataset, and never reaches the public leaderboard. ## Moving to another Hugging Face org The three dataset repos are derived from one constant, `ORG` in `evaluator.py`. The rest of the org name is spelled out and has to be changed by hand: - `SPACE_REPO` in `deploy_space.py` - `PUBLIC_REPO` in `quickstart.py` (standalone, so it cannot import `ORG`) - `SOCIAL_PREVIEW_URL` in `app.py` - the links in this file β€” including the YAML `thumbnail:` above β€” and in `pages/*.md`, `../public_dataset_README.md` (the dataset card) and `../HF_SETUP.md` That list is exhaustive as of the `ScientaLab` β†’ `PRIMOmics` move; `app.py` and the dataset card were the two it missed the first time. The theme follows the PRIMO charter: Funnel Display for headings, Funnel Sans for everything else, Scienta Navy `#080F5F` / PRIMO Cyan `#16B3C0` on Paper `#F3F8F8`. `colorFrom`/`colorTo` above stay `indigo`/`blue` because Hugging Face only accepts eight named colours and none of them is cyan.