## What PRIMO measures

**Can a foundation model turn a patient's omics data into an embedding that
predicts how their disease behaves?**

You embed each patient. We fit a linear probe on top and check whether that
embedding predicts a real clinical outcome. You never see the labels, so you
cannot tune to them, and when a score comes out high the credit belongs to the
*embedding*.

Plenty of benchmarks already score reconstruction. Far fewer ask whether the
representation is any use in the clinic, and that is the gap PRIMO tries to fill.

Today every cohort is **bulk RNA-seq**, in **immune-mediated inflammatory
diseases**, and we ask the same three questions of each patient:

| Task family | The question | Metric |
|---|---|---|
| **Treatment outcome** | Will this patient respond to anti-TNF therapy? | AUROC |
| **Clinical scores** | How severe is this patient's disease? | Pearson r |
| **Endotype** | Which molecular subtype is this patient? | AUROC |

## How a submission is scored

1. **You send embeddings, never predictions**: one vector per sample.
2. **We fit the probe**: ridge for regression, logistic for classification,
   regularisation picked by inner cross-validation.
3. **Scores are out-of-fold.** Transfer tasks use a fixed split instead: train
   on one drug's patients, test on the other's.
4. **Each task family keeps its own metric.** We never average an AUROC with a
   Pearson r inside one task score, and the two never share a column.

The probe is identical for everyone, which is what makes the numbers comparable:
what changes between two rows is the representation behind them.

A board covering more than one family also shows a **Mean** of the family
columns. It is there to give the table an order, but it does average AUROC with
Pearson, so use it as a tie-break and do the real comparison on the family
columns.

## How the results are organised

A **board** is a self-contained leaderboard over a slice of the benchmark: the
whole modality, one therapeutic area, or one task family. Pick one from the rail,
or on the **Boards** page.

A board ranks only the models that covered **all of its tasks**. Skipping
Dermatology therefore costs you nothing on the Rheumatology board.

**Boards never cross modalities.** An AUROC on bulk RNA and an AUROC on
single-cell are not measuring the same thing, so we keep them apart.

## What stays hidden, and why

The evaluation only stays honest while the labels stay ours. We still publish
the biology, since you need it to judge whether a task is relevant to you, but
we keep the provenance to ourselves.

| Disclosure | What it covers |
|---|---|
| **Per task** | disease, tissue, therapeutic area, what is predicted, class names, patient count, metric |
| **In aggregate** | the public archives the cohorts sit in |
| **Never** | study accessions, which cohort is behind which `dataset_id`, and the labels |

Every cohort comes from a published study deposited in {repositories}. We name
the archive but never the accession: publishing the study id behind a
`dataset_id` would put every label one download away.

**On reuse.** Both archives allow redistribution and place no restrictions of
their own, but neither grants permission on a depositor's behalf. Terms are set
study by study, and we have not audited all of them one by one. If you hold
rights over a cohort you recognise here,
[tell us](mailto:karim.elkanbi@scientalab.com?subject=PRIMO%20data%20rights) and
we will take it down.

## Cite PRIMO

Cite the benchmark paper: <https://openreview.net/forum?id=v2SA8gHwqo>

Want to add a cohort, propose a task on the cohorts already here, open a new
modality or therapeutic area, or work with us on the methodology? The
**Contribute** tab explains how.

## Links

- 🌐 **Website**: [primomics.org](http://primomics.org/)
- 📦 **Data**: [PRIMOmics/primo](https://huggingface.co/datasets/PRIMOmics/primo)
- 📄 **Paper**: [OpenReview](https://openreview.net/forum?id=v2SA8gHwqo)
- 💬 **Questions**: [karim.elkanbi@scientalab.com](mailto:karim.elkanbi@scientalab.com)