primo-eval / pages /submit.md
HungryTorch's picture
Ship redesigned PRIMO Space (rail nav, primo.css) reading PRIMOmics data
c9d0ebc verified
|
Raw
History Blame Contribute Delete
1.68 kB
**The fastest way in.** Two files, both in this Space's repo:
- πŸ“₯ [`quickstart.py`](https://huggingface.co/spaces/ScientaLab/primo-eval/blob/main/quickstart.py):
downloads every dataset, embeds them, writes a valid submission. Swap its
`embed` function for your model and you are done.
- πŸ“„ [`example_submission.csv`](https://huggingface.co/spaces/ScientaLab/primo-eval/blob/main/example_submission.csv):
four lines, fake numbers, the exact shape we expect.
```bash
pip install anndata scikit-learn pandas pyyaml huggingface_hub
python quickstart.py --out submission.parquet
```
---
Or do it by hand, in **three steps**:
1. **Get the data** β†’ download the datasets from [PRIMOmics/primo](https://huggingface.co/datasets/PRIMOmics/primo) (start with its `datasets.yaml`).
2. **Embed every dataset** β†’ build **one** file: `dataset_id`, `sample_id`, then one column per embedding dim (`e0`, `e1`, …). CSV / TSV / Parquet, or NPZ.
3. **Sign in, fill the form, and hit Evaluate.** A fixed linear probe scores each hidden task (AUROC or Pearson), reported per task category in its native metric.
**Example file**
```
dataset_id,sample_id,e0,e1,e2
d001,S1,0.12,-0.44,0.98
d002,S1,0.31,0.02,-0.15
```
**Partial submissions are welcome.** Cover fewer datasets and you are still
scored: you get ranked on every **board** whose tasks you covered in full, and
your numbers still show up in each board's **per-task** table, so nothing you
send is thrown away.
**Your first target is the baselines.** We run our own reference submissions,
linear probes on the most variable genes, and they sit on the boards labelled
`(baseline)`. Beating them is the bar to clear.