primo-eval / pages /submit.md
HungryTorch's picture
Ship redesigned PRIMO Space (rail nav, primo.css) reading PRIMOmics data
c9d0ebc verified
|
Raw
History Blame Contribute Delete
1.68 kB

A newer version of the Gradio SDK is available: 6.22.0

Upgrade

The fastest way in. Two files, both in this Space's repo:

  • 📥 quickstart.py: downloads every dataset, embeds them, writes a valid submission. Swap its embed function for your model and you are done.
  • 📄 example_submission.csv: four lines, fake numbers, the exact shape we expect.
pip install anndata scikit-learn pandas pyyaml huggingface_hub
python quickstart.py --out submission.parquet

Or do it by hand, in three steps:

  1. Get the data → download the datasets from PRIMOmics/primo (start with its datasets.yaml).
  2. Embed every dataset → build one file: dataset_id, sample_id, then one column per embedding dim (e0, e1, …). CSV / TSV / Parquet, or NPZ.
  3. Sign in, fill the form, and hit Evaluate. A fixed linear probe scores each hidden task (AUROC or Pearson), reported per task category in its native metric.

Example file

dataset_id,sample_id,e0,e1,e2
d001,S1,0.12,-0.44,0.98
d002,S1,0.31,0.02,-0.15

Partial submissions are welcome. Cover fewer datasets and you are still scored: you get ranked on every board whose tasks you covered in full, and your numbers still show up in each board's per-task table, so nothing you send is thrown away.

Your first target is the baselines. We run our own reference submissions, linear probes on the most variable genes, and they sit on the boards labelled (baseline). Beating them is the bar to clear.