Spaces:
Running
Running
File size: 1,677 Bytes
75e6a8b d2058b2 75e6a8b d2058b2 75e6a8b d2058b2 3879ed2 c9d0ebc 3879ed2 48fcbed 75e6a8b d2058b2 75e6a8b | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 | **The fastest way in.** Two files, both in this Space's repo:
- 📥 [`quickstart.py`](https://huggingface.co/spaces/ScientaLab/primo-eval/blob/main/quickstart.py):
downloads every dataset, embeds them, writes a valid submission. Swap its
`embed` function for your model and you are done.
- 📄 [`example_submission.csv`](https://huggingface.co/spaces/ScientaLab/primo-eval/blob/main/example_submission.csv):
four lines, fake numbers, the exact shape we expect.
```bash
pip install anndata scikit-learn pandas pyyaml huggingface_hub
python quickstart.py --out submission.parquet
```
---
Or do it by hand, in **three steps**:
1. **Get the data** → download the datasets from [PRIMOmics/primo](https://huggingface.co/datasets/PRIMOmics/primo) (start with its `datasets.yaml`).
2. **Embed every dataset** → build **one** file: `dataset_id`, `sample_id`, then one column per embedding dim (`e0`, `e1`, …). CSV / TSV / Parquet, or NPZ.
3. **Sign in, fill the form, and hit Evaluate.** A fixed linear probe scores each hidden task (AUROC or Pearson), reported per task category in its native metric.
**Example file**
```
dataset_id,sample_id,e0,e1,e2
d001,S1,0.12,-0.44,0.98
d002,S1,0.31,0.02,-0.15
```
**Partial submissions are welcome.** Cover fewer datasets and you are still
scored: you get ranked on every **board** whose tasks you covered in full, and
your numbers still show up in each board's **per-task** table, so nothing you
send is thrown away.
**Your first target is the baselines.** We run our own reference submissions,
linear probes on the most variable genes, and they sit on the boards labelled
`(baseline)`. Beating them is the bar to clear.
|