Spaces:
Running
Running
| **The fastest way in.** Two files, both in this Space's repo: | |
| - π₯ [`quickstart.py`](https://huggingface.co/spaces/ScientaLab/primo-eval/blob/main/quickstart.py): | |
| downloads every dataset, embeds them, writes a valid submission. Swap its | |
| `embed` function for your model and you are done. | |
| - π [`example_submission.csv`](https://huggingface.co/spaces/ScientaLab/primo-eval/blob/main/example_submission.csv): | |
| four lines, fake numbers, the exact shape we expect. | |
| ```bash | |
| pip install anndata scikit-learn pandas pyyaml huggingface_hub | |
| python quickstart.py --out submission.parquet | |
| ``` | |
| --- | |
| Or do it by hand, in **three steps**: | |
| 1. **Get the data** β download the datasets from [PRIMOmics/primo](https://huggingface.co/datasets/PRIMOmics/primo) (start with its `datasets.yaml`). | |
| 2. **Embed every dataset** β build **one** file: `dataset_id`, `sample_id`, then one column per embedding dim (`e0`, `e1`, β¦). CSV / TSV / Parquet, or NPZ. | |
| 3. **Sign in, fill the form, and hit Evaluate.** A fixed linear probe scores each hidden task (AUROC or Pearson), reported per task category in its native metric. | |
| **Example file** | |
| ``` | |
| dataset_id,sample_id,e0,e1,e2 | |
| d001,S1,0.12,-0.44,0.98 | |
| d002,S1,0.31,0.02,-0.15 | |
| ``` | |
| **Partial submissions are welcome.** Cover fewer datasets and you are still | |
| scored: you get ranked on every **board** whose tasks you covered in full, and | |
| your numbers still show up in each board's **per-task** table, so nothing you | |
| send is thrown away. | |
| **Your first target is the baselines.** We run our own reference submissions, | |
| linear probes on the most variable genes, and they sit on the boards labelled | |
| `(baseline)`. Beating them is the bar to clear. | |