fix(primo): expose branded social preview metadata

README.md

@@ -17,8 +17,8 @@ thumbnail: https://huggingface.co/spaces/ScientaLab/primo-eval/resolve/main/asse
 **A blind benchmark for omics foundation models.**
 
 PRIMO grades how well a model turns a patient's omics data into a useful
-**patient embedding**. You select one modality, embed its datasets, and upload
-**one** file; a fixed probe scores each hidden task, and the results roll up into a blind,
+**patient embedding**. You embed every dataset and upload **one** file; a fixed
+fixed probe scores each hidden task, and the results roll up into a blind,
 per-category leaderboard. The datasets are opaque (`d001`, `d002`…) and you never
 see the disease, tissue, or target, which leaves you grading the *embedding*
 itself with no room for per-task tuning.
@@ -42,22 +42,20 @@ that does not exist.
 
 ## What's in the data
 
-PRIMO benchmarks any omics modality. It includes **bulk RNA** tasks across
-immune-mediated inflammatory diseases and a **single-cell RNA** COVID-19 PBMC
-severity task, all with real clinical labels or treatment-induced expression
-responses from published cohorts:
+PRIMO benchmarks any omics modality. Today's datasets are all **bulk RNA**,
+covering **immune-mediated inflammatory diseases (IMIDs)** with real clinical
+labels or treatment-induced expression responses from published cohorts:
 
 - **Gastroenterology**: Crohn's disease, ulcerative colitis (anti-TNF response, severity scores)
 - **Dermatology**: atopic dermatitis, psoriasis (severity scores)
 - **Rheumatology**: rheumatoid arthritis (joint counts, molecular endotype)
-- **Infectious diseases**: COVID-19 severity from single-cell PBMC expression
 - **Perturbation response**: adalimumab transfer across inflammatory skin
   diseases, rituximab response in Sjögren salivary gland, and mouse intestinal
   anti-TNF response
 
 ## Submission format
 
-One file, one row per (`dataset_id`, `sample_id`), spanning one modality:
+One file, one row per (`dataset_id`, `sample_id`), spanning all datasets:
 
 - **CSV / TSV / Parquet**: a `dataset_id` column, a `sample_id` column, and one
   numeric column per embedding dimension. Embedding dim may differ per dataset
@@ -70,12 +68,6 @@ listed in the public `datasets.yaml` manifest. Alignment is by join, so row
 order does not matter; every labelled sample of a task must be present with no
 NaN/inf, or that task is skipped.
 
-Bulk H5AD files have one row per required submission sample. Single-cell H5AD
-files have one sparse raw-count row per cell; opaque cell ids are in
-`obs_names`, and the only public cell metadata is `obs["sample_id"]`, which maps
-each cell to its opaque collection sample. Submissions remain sample-level: emit
-exactly one embedding for every unique `sample_id`, not one embedding per cell.
-
 ## How it works
 
 Each dataset is embedded once and scored on every hidden task defined for it. Per
@@ -116,16 +108,14 @@ in each board's **per-task** table even when they are not ranked.
 
 ## Make a submission
 
-`quickstart.py` is the shortest path: it downloads one modality and embeds its
-datasets
-(log2(CPM+1) → PCA for bulk; per-cell log2(CP10K+1) → sample mean → PCA for
-single-cell) and writes the file the Submit tab wants. Swap its `embed`
+`quickstart.py` is the shortest path: it downloads every dataset, embeds each one
+(log2(CPM+1) → PCA) and writes the file the Submit tab wants. Swap its `embed`
 function for your encoder and nothing else changes. `example_submission.csv`
 shows the expected shape in four lines.
 
 ```bash
 pip install anndata scikit-learn pandas pyyaml huggingface_hub
-python quickstart.py --modality bulk-rna --out submission.parquet
+python quickstart.py --out submission.parquet
 ```
 
 ## Run the scorer locally
@@ -133,7 +123,7 @@ python quickstart.py --modality bulk-rna --out submission.parquet
 ```bash
 pip install -r requirements.txt
 export HF_TOKEN=...   # read access to the PRIMO datasets
-python evaluator.py --modality bulk-rna --submission my_embeddings.parquet
+python evaluator.py --submission my_embeddings.parquet
 ```
 
 ## Baselines

app.py

@@ -7,7 +7,7 @@ do I enter?"), Contribute ("what is missing, and how do I add it?"), Method
 board and ``?tab=contribute`` opens a tab -- which is what the rail links and the
 open cards use. The tab strip is hidden in CSS; the rail is the navigation.
 
-Choose one modality and upload its embedding file (rows keyed by ``dataset_id``
+Upload one embedding file spanning every dataset (rows keyed by ``dataset_id``
 + ``sample_id``); a fixed task probe scores each task (a dataset may carry
 several hidden targets). Results roll up into boards -- the whole modality, one
 therapeutic area, one task family -- and each board ranks the models that
@@ -56,14 +56,7 @@ from evaluator import (
     scoreable_tasks,
 )
 from leaderboard import RESERVED_COLUMNS, source_repositories
-from render import (
-    evaluation_confirmation,
-    evaluation_status,
-    rail_html,
-    render_board,
-    render_boards,
-    render_tasks,
-)
+from render import rail_html, render_board, render_boards, render_tasks
 from results import (
     BASELINE_TAG,
     INSTITUTION,
@@ -83,6 +76,19 @@ HERE = Path(__file__).parent
 PAGES_DIR = HERE / "pages"
 CSS_PATH = str(HERE / "primo.css")
 ALLOWED_PATHS = [str(HERE / "assets"), str(HERE / "fonts")]
+SOCIAL_PREVIEW_URL = (
+    "https://huggingface.co/spaces/ScientaLab/primo-eval/resolve/main/"
+    "assets/primo-social-preview.png"
+)
+PAGE_HEAD = f"""
+<meta property="og:title" content="PRIMO Benchmark" />
+<meta property="og:description" content="A blind benchmark for omics foundation models." />
+<meta property="og:image" content="{SOCIAL_PREVIEW_URL}" />
+<meta name="twitter:card" content="summary_large_image" />
+<meta name="twitter:title" content="PRIMO Benchmark" />
+<meta name="twitter:description" content="A blind benchmark for omics foundation models." />
+<meta name="twitter:image" content="{SOCIAL_PREVIEW_URL}" />
+"""
 
 TOKEN = os.environ.get("HF_TOKEN")
 TAB_IDS = ("boards", "board", "tasks", "submit", "contribute", "method")
@@ -94,15 +100,10 @@ THEME = gr.themes.Base(
 
 SUBMIT_HEAD = (
     '<div class="pm-head"><div><h1>Submit a model</h1>'
-    "<p>One model, one modality. Partial coverage is fine within that modality. "
-    "You are ranked on every board you cover in full.</p></div></div>"
+    "<p>One embedding file. Partial coverage is fine. You are ranked on every "
+    "board you cover in full.</p></div></div>"
 )
 
-MODALITY_CHOICES = [
-    ("Bulk RNAseq", "bulk-rna"),
-    ("Single-cell RNAseq", "single-cell-rna"),
-]
-
 TABLE_SORT_JS = """
 () => {
   if (window.pmTableSortBound) return;
@@ -143,28 +144,6 @@ TABLE_SORT_JS = """
     const header = event.target.closest("th.pm-sort");
     if (header) sort(header);
   });
-  document.addEventListener("click", (event) => {
-    const evaluateButton = event.target.closest(
-      "#pm-evaluate button, button#pm-evaluate"
-    );
-    if (!evaluateButton) return;
-    if (evaluateButton.dataset.confirmed === "true") {
-      delete evaluateButton.dataset.confirmed;
-      return;
-    }
-
-    event.preventDefault();
-    event.stopImmediatePropagation();
-    const dialog = document.querySelector("#pm-evaluation-confirmation");
-    if (!dialog) return;
-    dialog.returnValue = "";
-    dialog.addEventListener("close", () => {
-      if (dialog.returnValue !== "confirm") return;
-      evaluateButton.dataset.confirmed = "true";
-      evaluateButton.click();
-    }, { once: true });
-    dialog.showModal();
-  }, true);
   document.addEventListener("keydown", (event) => {
     if (event.key !== "Enter" && event.key !== " ") return;
     const header = event.target.closest("th.pm-sort");
@@ -185,16 +164,6 @@ def _page_text(name: str) -> str:
     return (PAGES_DIR / f"{name}.md").read_text()
 
 
-def _download_help(modality: str | None) -> str:
-    """Show the quickstart command for the selected modality."""
-    if not modality:
-        return "Select a modality to get its download command."
-    return (
-        f"```bash\npython quickstart.py --modality {modality} "
-        "--out submission.parquet\n```"
-    )
-
-
 def _registry_by_id() -> dict[str, dict]:
     """Scoreable tasks keyed by task_id (dataset present in the public manifest).
 
@@ -285,7 +254,6 @@ def _rendered_board(slug: str | None) -> str:
 
 def evaluate(
     submission_path: str,
-    modality: str | None,
     model_name: str,
     institution: str,
     is_author_submission: bool,
@@ -301,8 +269,6 @@ def evaluate(
 
     if profile is None:
         return _refuse("Please sign in with Hugging Face to submit.")
-    if not modality:
-        return _refuse("Please select a modality.")
     if not submission_path:
         return _refuse("Please upload a submission file.")
     if not model_name or not model_name.strip():
@@ -327,7 +293,7 @@ def evaluate(
             "board keeps each name's latest submission. Please pick another name."
         )
     try:
-        result = score_all(submission_path, TOKEN, modality=modality)
+        result = score_all(submission_path, TOKEN)
     except SubmissionError as error:
         return _refuse(f"❌ {error}")
     except EvaluatorError as error:
@@ -365,13 +331,6 @@ def evaluate(
         }
         for task in result["per_task"]
     ]
-    if not rows:
-        summary += (
-            "\n\n⚠️ No valid scores were produced, so the model was not added to "
-            "any board."
-        )
-        return summary, _rendered_board(slug)
-
     meta = {
         "model_name": model,
         "submitted_at": submitted_at,
@@ -386,17 +345,10 @@ def evaluate(
     }
     try:
         append_results(rows, TOKEN)
-    except Exception as error:  # noqa: BLE001
-        traceback.print_exc()
-        summary += f"\n\n⚠️ scored, but the leaderboard was not saved: {error}"
-        return summary, _rendered_board(slug)
-
-    summary += f"\n\n✅ **{model} is now shown on the relevant boards.**"
-    try:
         append_submission(meta, TOKEN)
     except Exception as error:  # noqa: BLE001
         traceback.print_exc()
-        summary += f"\n\n⚠️ Contact metadata was not saved: {error}"
+        summary += f"\n\n⚠️ scored, but the leaderboard was not saved: {error}"
     return summary, _rendered_board(slug)
 
 
@@ -435,6 +387,7 @@ def build_demo() -> gr.Blocks:
     with gr.Blocks(
         title="PRIMO Benchmark",
         theme=THEME,
+        head=PAGE_HEAD,
         css_paths=[CSS_PATH],
         js=TABLE_SORT_JS,
         fill_width=True,
@@ -459,12 +412,6 @@ def build_demo() -> gr.Blocks:
                                     gr.Markdown(_page_text("submit"))
                                 with gr.Column(scale=2, elem_id="pm-form"):
                                     gr.LoginButton()
-                                    modality_in = gr.Radio(
-                                        MODALITY_CHOICES,
-                                        label="Modality",
-                                        info="One model submission covers one modality.",
-                                    )
-                                    download_md = gr.Markdown(_download_help(None))
                                     model_tb = gr.Textbox(
                                         label="Model name",
                                         placeholder="e.g. eva-rna-v1",
@@ -504,17 +451,12 @@ def build_demo() -> gr.Blocks:
                                     )
                                     run_btn = gr.Button(
                                         "Evaluate",
-                                        elem_id="pm-evaluate",
                                         elem_classes=[
                                             "pm-btn",
                                             "pm-btn--primary",
                                             "pm-btn--block",
                                         ],
                                     )
-                                    status_html = gr.HTML(
-                                        evaluation_status(), visible=False
-                                    )
-                                    gr.HTML(evaluation_confirmation())
                             result_md = gr.Markdown()
                     with gr.Tab("Contribute", id="contribute"):
                         with gr.Column(elem_classes=["pm-body", "pm-prose"]):
@@ -523,16 +465,10 @@ def build_demo() -> gr.Blocks:
                         with gr.Column(elem_classes=["pm-body", "pm-prose"]):
                             about_md = gr.Markdown()
 
-        evaluation_event = run_btn.click(
-            lambda: gr.update(visible=True),
-            None,
-            status_html,
-            show_progress="hidden",
-        ).then(
+        run_btn.click(
             evaluate,
             [
                 file_in,
-                modality_in,
                 model_tb,
                 institution_tb,
                 author_submission_cb,
@@ -544,13 +480,6 @@ def build_demo() -> gr.Blocks:
             ],
             [result_md, board_html],
         )
-        evaluation_event.then(
-            lambda: gr.update(visible=False),
-            None,
-            status_html,
-            show_progress="hidden",
-        )
-        modality_in.change(_download_help, modality_in, download_md)
         demo.load(
             _init,
             None,

boards.py

@@ -38,8 +38,8 @@ METRIC_LABEL = {
     "centered_spearman": "Centered Spearman score",
 }
 MODALITY_LABEL = {
-    "bulk RNA": "Bulk RNAseq",
-    "Single Cell RNA": "Single-cell RNAseq",
+    "bulk RNA": "bulk RNAseq",
+    "single-cell RNA": "single-cell RNAseq",
 }
 
 MODALITY_GROUP = "Per modality"
@@ -165,8 +165,8 @@ def _cohort_stats(tasks: list[dict]) -> tuple[int, int, int]:
     diseases: set[str] = set()
     for task in tasks:
         cohort = str(task.get("cohort_id") or _norm_id(task.get("dataset_id", "")))
-        n_subjects = int(task.get("n_subjects") or task.get("n_samples") or 0)
-        patients_by_cohort[cohort] = max(patients_by_cohort.get(cohort, 0), n_subjects)
+        n_samples = int(task.get("n_samples") or 0)
+        patients_by_cohort[cohort] = max(patients_by_cohort.get(cohort, 0), n_samples)
         diseases.update(str(d) for d in task.get("diseases") or [])
     return (
         len(patients_by_cohort),
@@ -177,8 +177,7 @@ def _cohort_stats(tasks: list[dict]) -> tuple[int, int, int]:
 
 def _blurb(group: str, name: str, tasks: list[dict]) -> str:
     if group == MODALITY_GROUP:
-        display = modality_label(name)
-        return f"Every PRIMO task, scored from {display[:1].lower()}{display[1:]} profiles."
+        return f"Every PRIMO task, scored from {modality_label(name)} profiles."
     if group == CATEGORY_GROUP:
         return CATEGORY_BLURB.get(name, label(name))
     families = [label(c) for c in distinct(tasks, "category")]
@@ -241,17 +240,17 @@ class OpenBoard:
 OPEN_BOARDS: tuple[OpenBoard, ...] = (
     OpenBoard(
         MODALITY_GROUP,
-        "Single-cell RNAseq",
+        "single-cell RNAseq",
         "Dissociated tissue, labelled at the patient level. No cohort yet.",
     ),
     OpenBoard(
         MODALITY_GROUP,
-        "Proteomics",
+        "proteomics",
         "Plasma or tissue proteins paired with clinical follow-up. No cohort yet.",
     ),
     OpenBoard(
         MODALITY_GROUP,
-        "Spatial transcriptomics",
+        "spatial transcriptomics",
         "Expression kept in place in the tissue, with patient outcomes. No cohort yet.",
     ),
     OpenBoard(

evaluator.py

@@ -1,6 +1,6 @@
 """Standalone probe for the PRIMO public benchmark.
 
-Loads one embedding submission for one modality (rows keyed by
+Loads one embedding submission that spans every dataset (rows keyed by
 ``dataset_id`` + ``sample_id``), then scores each TASK = (dataset, target): a
 dataset is embedded once and reused across all its tasks. Per task it fits a
 fixed probe—linear for scalar targets and multi-output ridge for response
@@ -71,7 +71,6 @@ MANIFEST_FILENAME = "datasets.yaml"
 TASKS_FILENAME = "tasks.yaml"
 LABELS_FILENAME = "labels.csv"
 TARGETS_FILENAME = "targets.npz"
-MODALITIES = {"bulk-rna": "bulk RNA", "single-cell-rna": "Single Cell RNA"}
 
 RIDGE_ALPHAS = np.logspace(-3.0, 6.0, 19)
 PERTURBATION_TEST_Z_CLIP = 20.0
@@ -653,13 +652,9 @@ def fetch_manifest(token: str | None = None) -> list[dict]:
     return data or []
 
 
-def manifest_ids(manifest: list[dict], modality: str | None = None) -> set[str]:
-    """Return manifest dataset ids, optionally for one modality."""
-    return {
-        _norm_id(entry["id"])
-        for entry in manifest
-        if modality is None or entry.get("modality") == modality
-    }
+def manifest_ids(manifest: list[dict]) -> set[str]:
+    """The canonical set of valid dataset ids from the manifest."""
+    return {_norm_id(entry["id"]) for entry in manifest}
 
 
 def scoreable_tasks(tasks: list[dict], valid_ids: set[str]) -> list[dict]:
@@ -770,7 +765,6 @@ def score_all(
     path: str | Path,
     token: str | None = None,
     *,
-    modality: str | None = None,
     datasets: list[dict] | None = None,
     tasks: list[dict] | None = None,
     fetch_labels=None,
@@ -790,21 +784,11 @@ def score_all(
             raise EvaluatorError(
                 f"could not load the dataset manifest: {error}"
             ) from error
-    if modality is not None and modality not in MODALITIES:
-        raise SubmissionError(f"unknown or unavailable modality: {modality}")
-    all_ids = manifest_ids(datasets)
-    valid = manifest_ids(datasets, MODALITIES.get(modality))
-    if modality is not None and not valid:
-        raise SubmissionError(f"unknown or unavailable modality: {modality}")
-
-    unknown = sorted(set(blocks) - all_ids)
+    valid = manifest_ids(datasets)
+
+    unknown = sorted(set(blocks) - valid)
     if unknown:
         raise SubmissionError(f"unknown dataset_id(s) not in the benchmark: {unknown}")
-    wrong_modality = sorted(set(blocks) - valid)
-    if wrong_modality:
-        raise SubmissionError(
-            f"dataset_id(s) outside the selected modality: {wrong_modality}"
-        )
 
     if tasks is None:
         try:
@@ -832,8 +816,7 @@ def score_all(
             fetch_targets,
             token,
         )
-        for task in scoreable_tasks(tasks, all_ids)
-        if _norm_id(task[DATASET_ID]) in valid
+        for task in scoreable_tasks(tasks, valid)
     ]
     return _summarize(outcomes)
 
@@ -885,12 +868,11 @@ def _dataset_status(outcomes: list[TaskOutcome]) -> dict[str, str]:
 def _cli() -> None:
     parser = argparse.ArgumentParser(description="Score a PRIMO submission locally.")
     parser.add_argument("--submission", required=True, help="CSV/TSV/Parquet/NPZ file")
-    parser.add_argument("--modality", required=True, choices=MODALITIES)
     parser.add_argument("--token", default=None, help="HF token (else env HF_TOKEN)")
     args = parser.parse_args()
 
     token = args.token or os.environ.get("HF_TOKEN")
-    result = score_all(args.submission, token, modality=args.modality)
+    result = score_all(args.submission, token)
     print(
         f"scored     : {result['n_datasets_scored']}/{result['n_datasets_total']} "
         f"datasets, {result['n_scored']}/{result['n_total']} tasks  "

leaderboard.py

@@ -505,7 +505,7 @@ def tasks_table(tasks: list[dict]) -> pd.DataFrame:
                 "Area": str(task.get("therapeutic_area", "")),
                 "Disease": ", ".join(str(d) for d in task.get("diseases") or []),
                 "Tissue": str(task.get("tissue", "")),
-                "Patients": task.get("n_subjects") or task.get("n_samples"),
+                "Patients": task.get("n_samples"),
                 "Metric": metric_label(str(task.get("metric", ""))),
             }
         )

pages/submit.md

@@ -1,24 +1,22 @@
 **The fastest way in.** Two files, both in this [Space's repo](https://huggingface.co/spaces/ScientaLab/primo-eval/blob/main/quickstart.py):
 
 - 📥 [`quickstart.py`](https://huggingface.co/spaces/ScientaLab/primo-eval/blob/main/quickstart.py):
-  downloads the selected modality, embeds it, and writes a valid submission.
-  Swap its `embed` function for your model and you are done.
+  downloads every dataset, embeds them, writes a valid submission. Swap its
+  `embed` function for your model and you are done.
 - 📄 [`example_submission.csv`](https://huggingface.co/spaces/ScientaLab/primo-eval/blob/main/example_submission.csv):
   four lines, fake numbers, the exact shape we expect.
 
 ```bash
 pip install anndata scikit-learn pandas pyyaml huggingface_hub
-python quickstart.py --modality bulk-rna --out submission.parquet
-# or: --modality single-cell-rna
+python quickstart.py --out submission.parquet
 ```
 
 ---
 
 Or do it by hand, in **three steps**:
 
-1. **Choose one modality** → use the selector in the form. The quickstart
-   downloads only its datasets from [ScientaLab/primo](https://huggingface.co/datasets/ScientaLab/primo).
-2. **Embed that modality** → build **one** file: `dataset_id`, `sample_id`, then one column per embedding dim (`e0`, `e1`, …). CSV / TSV / Parquet, or NPZ.
+1. **Get the data** → download the datasets from [ScientaLab/primo](https://huggingface.co/datasets/ScientaLab/primo) (start with its `datasets.yaml`).
+2. **Embed every dataset** → build **one** file: `dataset_id`, `sample_id`, then one column per embedding dim (`e0`, `e1`, …). CSV / TSV / Parquet, or NPZ.
 3. **Sign in, fill the form, and hit Evaluate.** Add an institution for group submissions, check **Submitted by the model's authors** when applicable, and provide a paper link to make the model name clickable. A fixed task probe scores each hidden task (AUROC, Pearson or centered Spearman), reported per task category in its native metric.
 
 **Example file**
@@ -29,14 +27,7 @@ d001,S1,0.12,-0.44,0.98
 d002,S1,0.31,0.02,-0.15
 ```
 
-For bulk datasets, each H5AD row is one submission sample. For single-cell
-datasets, H5AD rows are cells and `obs["sample_id"]` maps them to opaque
-collection samples. Aggregate the cells however your model requires and submit
-exactly one embedding per unique `sample_id`; the submission schema is unchanged.
-
-Files containing dataset IDs from another modality are rejected. Reusing a model
-name for another modality replaces its previous leaderboard entry. **Partial
-submissions are welcome.** Cover fewer datasets within the selected modality and you are still
+**Partial submissions are welcome.** Cover fewer datasets and you are still
 scored: you get ranked on every **board** whose scored tasks you covered in
 full, and your numbers still show up in each board's **per-task** table, so
 nothing you send is thrown away.

primo.css

@@ -98,7 +98,7 @@ footer { display: none !important; }
 .pm-rail-summary::-webkit-details-marker { display: none; }
 .pm-rail-summary:hover { background: var(--sidebar-highlight); }
 .pm-rail-summary.pm-active { background: var(--sidebar-accent); color: #fff; }
-.pm-rail-summary .pm-chevron { flex: none; color: #fff; transition: transform .15s ease; }
+.pm-rail-summary .pm-chevron { flex: none; color: var(--sidebar-sub-fg); transition: transform .15s ease; }
 .pm-rail-section[open] > .pm-rail-summary .pm-chevron { transform: rotate(90deg); }
 
 .pm-rail-group { padding: 8px 16px 0; }
@@ -264,22 +264,6 @@ code, .pm-inline-code { font-family: var(--font-mono); font-size: .95em; }
 #pm-form .upload-container, #pm-form .file-preview { border: 1px dashed var(--marine-200) !important;
   border-radius: var(--radius-md) !important; background: var(--surface-raised, #fff) !important; }
 
-/* ---------- evaluation confirmation ---------- */
-.pm-dialog { width: min(440px, calc(100vw - 32px)); padding: 0; border: 1px solid var(--border);
-  border-radius: var(--radius-lg); background: #fff; color: var(--text-body); box-shadow: 0 24px 64px rgb(17 27 68 / .24); }
-.pm-dialog::backdrop { background: rgb(17 27 68 / .55); }
-.pm-dialog-surface { padding: 24px; }
-.pm-dialog h2 { margin: 8px 0 0; font: 600 1.25rem/1.2 var(--font-title); letter-spacing: -.03em; color: var(--marine-700); }
-.pm-dialog-surface > p:not(.pm-over) { margin: 12px 0 0; font-size: 14px; line-height: 1.6; color: var(--text-secondary); }
-.pm-dialog-actions { display: flex; justify-content: flex-end; gap: 8px; margin-top: 24px; }
-.pm-evaluation-status { display: flex; align-items: center; gap: 10px; font-size: 13px; line-height: 1.45; color: var(--text-secondary); }
-.pm-spinner { width: 18px; height: 18px; flex: none; animation: pm-spin .8s linear infinite; }
-.pm-spinner circle, .pm-spinner path { fill: none; stroke-width: 3; }
-.pm-spinner circle { stroke: var(--marine-100); }
-.pm-spinner path { stroke: var(--marine-500); stroke-linecap: round; }
-@keyframes pm-spin { to { transform: rotate(360deg); } }
-@media (prefers-reduced-motion: reduce) { .pm-spinner { animation-duration: 1.6s; } }
-
 /* ---------- routes (Contribute) ---------- */
 .pm-route { display: block; padding: 20px; margin-bottom: 12px; border: 1px solid var(--border);
   border-radius: var(--radius-xl); box-shadow: var(--shadow-xs); text-decoration: none; transition: border-color .15s ease, box-shadow .15s ease; }

quickstart.py

@@ -1,13 +1,12 @@
 """Produce a valid PRIMO submission in one command, then swap in your own model.
 
-Downloads one modality's public datasets and writes the file the Submit tab
-expects. The embedding here is deliberately dumb: bulk data uses
-log2(CPM+1); single-cell data uses per-cell log2(CP10K+1), mean pooling by
-collection sample, then PCA. Replace ``embed`` with your encoder and nothing else
-changes.
+Downloads every public dataset, embeds each one, and writes the single file the
+Submit tab expects. The embedding here is deliberately dumb -- log2(CPM+1) then
+PCA -- because the point is the plumbing, not the score: replace ``embed`` with
+your encoder and nothing else changes.
 
     pip install anndata scikit-learn pandas pyyaml huggingface_hub
-    python quickstart.py --modality bulk-rna --out submission.parquet
+    python quickstart.py --out submission.parquet
 
 Standalone on purpose: no import from this Space and none from our monorepo, so
 it keeps working if you copy the file into your own project.
@@ -20,106 +19,64 @@ import anndata as ad
 import numpy as np
 import pandas as pd
 import yaml
-from huggingface_hub import hf_hub_download, snapshot_download
+from huggingface_hub import snapshot_download
 from sklearn.decomposition import PCA
 
 PUBLIC_REPO = "ScientaLab/primo"
 MANIFEST_FILENAME = "datasets.yaml"
-MODALITIES = {"bulk-rna": "bulk RNA", "single-cell-rna": "Single Cell RNA"}
 
 DATASET_ID = "dataset_id"
 SAMPLE_ID = "sample_id"
 
 TARGET_SUM = 1_000_000
-SINGLE_CELL_TARGET_SUM = 10_000
 N_COMPONENTS = 50
 RANDOM_STATE = 0
 
 
-def prediction_ids(adata: ad.AnnData) -> np.ndarray:
-    """Opaque collection-sample ids required in the submission."""
-    if SAMPLE_ID in adata.obs:
-        return pd.unique(adata.obs[SAMPLE_ID].astype(str)).astype(str)
-    return adata.obs_names.astype(str).to_numpy()
-
-
-def _log_normalize(x, target_sum: int) -> np.ndarray:
-    """Return dense log2 counts-per-target expression."""
-    x = x.toarray() if hasattr(x, "toarray") else np.asarray(x)
-    totals = x.sum(axis=1, keepdims=True)
-    return np.log2(x / np.where(totals == 0, 1, totals) * target_sum + 1)
-
-
-def _sample_expression(adata: ad.AnnData) -> tuple[np.ndarray, np.ndarray]:
-    """Normalize expression and mean-pool cells into collection samples."""
-    ids = prediction_ids(adata)
-    if SAMPLE_ID not in adata.obs:
-        return ids, _log_normalize(adata.X, TARGET_SUM)
-    samples = adata.obs[SAMPLE_ID].astype(str).to_numpy()
-    pooled = [
-        _log_normalize(adata.X[samples == sample_id], SINGLE_CELL_TARGET_SUM).mean(0)
-        for sample_id in ids
-    ]
-    return ids, np.asarray(pooled)
-
-
 def embed(adata: ad.AnnData) -> np.ndarray:
-    """One dataset's raw counts -> one vector per collection sample. Replace me.
+    """One dataset's raw counts -> one vector per patient. Replace me.
 
-    Whatever you return, the contract is one finite row per id returned by
-    ``prediction_ids``. The embedding width may differ between datasets.
+    Whatever you return, the contract is the same: one row per sample, in
+    ``adata.obs_names`` order, all finite. The embedding width is yours to pick
+    and may differ from one dataset to the next.
     """
-    _, x = _sample_expression(adata)
+    x = adata.X
+    x = x.toarray() if hasattr(x, "toarray") else np.asarray(x)
+    x = x.astype(float)
+    counts = x.sum(axis=1, keepdims=True)
+    x = np.log2(x / np.where(counts == 0, 1, counts) * TARGET_SUM + 1)
     k = min(N_COMPONENTS, x.shape[0] - 1, x.shape[1])
     return PCA(n_components=k, random_state=RANDOM_STATE).fit_transform(x)
 
 
-def download(modality: str, token: str | None) -> Path:
-    """Download the manifest and expression files for one modality."""
-    manifest = Path(
-        hf_hub_download(
-            PUBLIC_REPO, MANIFEST_FILENAME, repo_type="dataset", token=token
-        )
-    )
-    paths = [
-        MANIFEST_FILENAME,
-        *[str(entry["path"]) for entry in datasets(manifest, modality)],
-    ]
-    return Path(
-        snapshot_download(
-            PUBLIC_REPO, repo_type="dataset", token=token, allow_patterns=paths
-        )
-    )
+def download(token: str | None) -> Path:
+    """Pull the public benchmark (manifest + every ``expression.h5ad``)."""
+    return Path(snapshot_download(PUBLIC_REPO, repo_type="dataset", token=token))
 
 
-def datasets(manifest_path: Path, modality: str) -> list[dict]:
-    """Manifest entries for one modality."""
-    manifest = yaml.safe_load(manifest_path.read_text())
+def dataset_ids(root: Path) -> list[str]:
+    """The opaque ids to embed, read off the public manifest."""
+    manifest = yaml.safe_load((root / MANIFEST_FILENAME).read_text())
     entries = manifest.get("datasets", []) if isinstance(manifest, dict) else manifest
-    return [entry for entry in entries if entry["modality"] == modality]
+    return [str(entry["id"]) for entry in entries]
 
 
-def build(root: Path, modality: str) -> pd.DataFrame:
-    """Embed one modality into the frame the Submit tab expects.
+def build(root: Path) -> pd.DataFrame:
+    """Embed every dataset into the one frame the Submit tab expects.
 
     Datasets of different widths stack into one table; the extra columns of a
     narrower dataset stay empty and the evaluator drops them per dataset, so each
     dataset keeps its own embedding size.
     """
     blocks = []
-    for entry in datasets(root / MANIFEST_FILENAME, modality):
-        dataset_id = str(entry["id"])
-        adata = ad.read_h5ad(root / entry["path"])
-        sample_ids = prediction_ids(adata)
+    for dataset_id in dataset_ids(root):
+        adata = ad.read_h5ad(root / dataset_id / "expression.h5ad")
         vectors = embed(adata)
-        print(
-            f"{dataset_id}: {adata.n_obs} observations, {len(sample_ids)} samples "
-            f"-> {vectors.shape[1]} dims"
-        )
+        print(f"{dataset_id}: {adata.n_obs} samples -> {vectors.shape[1]} dims")
         block = pd.DataFrame(
             vectors, columns=[f"e{i}" for i in range(vectors.shape[1])]
         )
-        block.insert(0, SAMPLE_ID, sample_ids)
+        block.insert(0, SAMPLE_ID, adata.obs_names.to_numpy())
         block.insert(0, DATASET_ID, dataset_id)
         blocks.append(block)
     return pd.concat(blocks, ignore_index=True)
@@ -127,13 +84,11 @@ def build(root: Path, modality: str) -> pd.DataFrame:
 
 def main() -> None:
     parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
-    parser.add_argument("--modality", required=True, choices=MODALITIES)
     parser.add_argument("--out", type=Path, default=Path("submission.parquet"))
     parser.add_argument("--token", default=None, help="HF token, if you need one.")
     args = parser.parse_args()
 
-    modality = MODALITIES[args.modality]
-    submission = build(download(modality, args.token), modality)
+    submission = build(download(args.token))
     if args.out.suffix == ".csv":
         submission.to_csv(args.out, index=False)
     else:

render.py

@@ -67,42 +67,10 @@ METRIC_GUIDES = {
     "Centered Spearman score": "Range: -1 to 1 · Random predictor: 0 (expected)",
 }
 
-
-def evaluation_confirmation() -> str:
-    """Render the confirmation shown before a model evaluation starts."""
-    return (
-        '<dialog id="pm-evaluation-confirmation" class="pm-dialog" '
-        'aria-labelledby="pm-evaluation-confirmation-title">'
-        '<form method="dialog" class="pm-dialog-surface">'
-        '<p class="pm-over">Confirm submission</p>'
-        '<h2 id="pm-evaluation-confirmation-title">'
-        "Evaluate and submit this model?</h2>"
-        "<p>Evaluation can take several minutes. If it succeeds, this submission "
-        "will update the public leaderboards.</p>"
-        '<div class="pm-dialog-actions">'
-        '<button class="pm-btn pm-btn--outline" value="cancel">Cancel</button>'
-        '<button class="pm-btn pm-btn--primary" value="confirm">'
-        "Evaluate model</button>"
-        "</div></form></dialog>"
-    )
-
-
-def evaluation_status() -> str:
-    """Render the progress message shown while an evaluation runs."""
-    return (
-        '<div class="pm-evaluation-status" role="status" aria-live="polite">'
-        '<svg class="pm-spinner" aria-hidden="true" viewBox="0 0 24 24">'
-        '<circle cx="12" cy="12" r="9"></circle>'
-        '<path d="M12 3a9 9 0 0 1 9 9"></path></svg>'
-        "<span>Your embeddings are being evaluated... This can take a few "
-        "minutes.</span></div>"
-    )
-
-
 CHEVRON = (
-    '<svg class="pm-chevron" viewBox="0 0 16 16" width="16" height="16" '
-    'aria-hidden="true"><path d="M6 4l4 4-4 4" fill="none" stroke="#fff" '
-    'stroke-width="2" stroke-linecap="round" stroke-linejoin="round"/></svg>'
+    '<svg class="pm-chevron" viewBox="0 0 16 16" width="12" height="12" '
+    'aria-hidden="true"><path d="M6 4l4 4-4 4" fill="none" stroke="currentColor" '
+    'stroke-width="1.6" stroke-linecap="round" stroke-linejoin="round"/></svg>'
 )
 
 
@@ -343,19 +311,12 @@ def _task_tooltip(task: dict) -> str:
     diseases = (
         ", ".join(str(d) for d in task.get("diseases") or []) or "the listed cohort"
     )
-    patients = task.get("n_subjects") or task.get("n_samples")
+    patients = task.get("n_samples")
     patient_text = (
         f"{patients:,} patients"
         if isinstance(patients, int)
         else "an unspecified number of patients"
     )
-    n_samples = task.get("n_samples")
-    if (
-        isinstance(n_samples, int)
-        and isinstance(patients, int)
-        and n_samples != patients
-    ):
-        patient_text += f" ({n_samples:,} collection samples)"
     modality = str(task.get("modality") or "omics")
     tissue = str(task.get("tissue") or "unspecified tissue")
     target = str(task.get("target") or task.get("title") or "the task target")