pages/about.md

What PRIMO measures

Can a foundation model turn a patient's omics data into an embedding that predicts how their disease behaves?

You embed each patient. We fit a linear probe on top and check whether that embedding predicts a real clinical outcome. You never see the labels, so you cannot tune to them, and when a score comes out high the credit belongs to the embedding.

Plenty of benchmarks already score reconstruction. Far fewer ask whether the representation is any use in the clinic, and that is the gap PRIMO tries to fill.

Today every cohort is bulk RNA-seq, in immune-mediated inflammatory diseases, and we ask the same three questions of each patient:

Task family	The question	Metric
Treatment outcome	Will this patient respond to anti-TNF therapy?	AUROC
Clinical scores	How severe is this patient's disease?	Pearson r
Endotype	Which molecular subtype is this patient?	AUROC

How a submission is scored

1. You send embeddings, never predictions: one vector per sample.
2. We fit the probe: ridge for regression, logistic for classification, regularisation picked by inner cross-validation.
3. Scores are out-of-fold. Transfer tasks use a fixed split instead: train on one drug's patients, test on the other's.
4. Each task family keeps its own metric. We never average an AUROC with a Pearson r inside one task score, and the two never share a column.

The probe is identical for everyone, which is what makes the numbers comparable: what changes between two rows is the representation behind them.

A board covering more than one family also shows a Mean of the family columns. It is there to give the table an order, but it does average AUROC with Pearson, so use it as a tie-break and do the real comparison on the family columns.

How the results are organised

A board is a self-contained leaderboard over a slice of the benchmark: the whole modality, one therapeutic area, or one task family. Pick one from the rail, or on the Boards page.

A board ranks only the models that covered all of its tasks. Skipping Dermatology therefore costs you nothing on the Rheumatology board.

Boards never cross modalities. An AUROC on bulk RNA and an AUROC on single-cell are not measuring the same thing, so we keep them apart.

What stays hidden, and why

The evaluation only stays honest while the labels stay ours. We still publish the biology, since you need it to judge whether a task is relevant to you, but we keep the provenance to ourselves.

Disclosure	What it covers
Per task	disease, tissue, therapeutic area, what is predicted, class names, patient count, metric
In aggregate	the public archives the cohorts sit in
Never	study accessions, which cohort is behind which dataset_id, and the labels

Every cohort comes from a published study deposited in {repositories}. We name the archive but never the accession: publishing the study id behind a dataset_id would put every label one download away.

On reuse. Both archives allow redistribution and place no restrictions of their own, but neither grants permission on a depositor's behalf. Terms are set study by study, and we have not audited all of them one by one. If you hold rights over a cohort you recognise here, tell us and we will take it down.

Cite PRIMO

Cite the benchmark paper: https://openreview.net/forum?id=v2SA8gHwqo

Want to add a cohort, propose a task on the cohorts already here, open a new modality or therapeutic area, or work with us on the methodology? The Contribute tab explains how.

Links

• 🌐 Website: primomics.org
• 📦 Data: PRIMOmics/primo
• 📄 Paper: OpenReview
• 💬 Questions: karim.elkanbi@scientalab.com