feat(primo): add task tooltips and custom social preview

README.md

@@ -3,6 +3,7 @@ title: PRIMO Benchmark
 emoji: 🧬
 colorFrom: indigo
 colorTo: blue
+thumbnail: https://huggingface.co/spaces/ScientaLab/primo-eval/resolve/main/assets/primo-social-preview.png
 sdk: gradio
 sdk_version: 5.50.0
 python_version: "3.10"
@@ -17,7 +18,7 @@ hf_oauth: true
 
 PRIMO grades how well a model turns a patient's omics data into a useful
 **patient embedding**. You embed every dataset and upload **one** file; a fixed
-fixed probe scores each hidden task, and the results roll up into a blind,
+linear probe scores each hidden task, and the results roll up into a blind,
 per-category leaderboard. The datasets are opaque (`d001`, `d002`…) and you never
 see the disease, tissue, or target, which leaves you grading the *embedding*
 itself with no room for per-task tuning.
@@ -41,16 +42,13 @@ that does not exist.
 
 ## What's in the data
 
-PRIMO benchmarks any omics modality. Today's datasets are all **bulk RNA**,
+PRIMO benchmarks any omics modality. Today's datasets are all **bulk RNA-seq**,
 covering **immune-mediated inflammatory diseases (IMIDs)** with real clinical
-labels or treatment-induced expression responses from published cohorts:
+labels from published cohorts:
 
 - **Gastroenterology**: Crohn's disease, ulcerative colitis (anti-TNF response, severity scores)
 - **Dermatology**: atopic dermatitis, psoriasis (severity scores)
 - **Rheumatology**: rheumatoid arthritis (joint counts, molecular endotype)
-- **Perturbation response**: adalimumab transfer across inflammatory skin
-  diseases, rituximab response in Sjögren salivary gland, and mouse intestinal
-  anti-TNF response
 
 ## Submission format
 
@@ -78,40 +76,17 @@ partitions: **AUROC** (classification) or **Pearson r** (regression). A few
 tasks instead use a fixed train/test split: the probe is fit once on the training
 portion and scored on the held-out portion.
 
-Tasks are grouped into **four families**, each reported in its own native
+Tasks are grouped into **three families**, each reported in its own native
 metric. Two families are never merged into one column:
 
 - **Treatment outcome**: response to anti-TNF therapy (**AUROC**)
 - **Clinical scores**: disease-severity regression (**Pearson r**)
 - **Endotype**: molecular-subtype classification (**AUROC**)
-- **Perturbation response**: hidden post-minus-pre DEG decoding (**centered
-  Spearman**), where 1 is perfect, 0 is uninformative, and -1 is an inverse gene
-  ranking.
-
-Perturbation inputs contain baseline expression only. Participants still submit
-one embedding per public sample. The evaluator independently applies
-`log2(CPM + 1)` to baseline and post-treatment profiles, selects the top 600
-DEGs inside each outer training fold, standardizes embeddings on the training
-side, and fits a multi-output `RidgeCV` decoder to the hidden response matrix.
-Test coordinates are clipped to ±20 training standard deviations to limit
-cross-cohort extrapolation. Directional transfer tasks learn standardization,
-DEGs, and the decoder solely from their source disease. The mouse IBD task is a
-regular weighted task: three public baseline animals are reused across nine
-private control×perturbed responses, evaluated over all six specimen-isolated
-permutations.
-
-For ordinary paired and same-species transfer tasks, the evaluator subtracts the
-training-fold mean response from the observed and decoded responses. For
-mouse-to-human transfer, it instead subtracts each gene's mean across the human
-target cohort, separately for observations and predictions. It then computes
-Spearman across the 600 genes for every held-out patient. The task score is the
-mean Spearman itself: inverse rankings score -1, an uninformative shared response
-scores 0, and perfect rankings score 1.
 
 A leaderboard shows one column per family. A board holding more than one family
 also shows **Mean**, the average of those columns. It is what orders the rows,
-but it does mix AUROC, Pearson and centered Spearman, so treat it as a tie-break and
-compare models on the family columns.
+but it does mix AUROC with Pearson, so treat it as a tie-break and compare models
+on the family columns.
 
 ## Boards
 
@@ -145,35 +120,6 @@ export HF_TOKEN=...   # read access to the PRIMO datasets
 python evaluator.py --submission my_embeddings.parquet
 ```
 
-Run the complete perturbation artifact and baseline validation harness from the
-repository root. Synthetic mode needs no private data or network access:
-
-```bash
-uv run python -m benchmark.public_benchmark.validate_perturbation
-```
-
-To validate against datasets already present in the local Scienta Hub cache:
-
-```bash
-uv run python -m benchmark.public_benchmark.validate_perturbation \
-  --source cache --output .context/perturbation-real
-```
-
-The harness writes public/private artifacts, raw-expression, HVG-1200, random,
-and constant submissions, plus `report.json`, beneath `.context/`. It checks the
-privacy boundary and zero anchor, and reports fold-local DEG overlap, score
-distributions, ridge conditioning, and all mouse permutations. It never
-downloads, uploads, publishes, or deploys anything. Pass `--overwrite` to
-replace an existing output directory.
-
-## Perturbation rollout
-
-Keep the live benchmark unchanged until the local artifact and baseline checks
-pass. Then deploy in this order: the backward-compatible scorer, public baseline
-inputs, private targets and registry, and finally refreshed baselines. Existing
-task-result rows stay in place; their submissions simply lack coverage for the
-new datasets until their owners submit embeddings for `d011`–`d016`.
-
 ## Baselines
 
 `task_results.csv` carries an `is_baseline` flag. Reference submissions we
@@ -192,10 +138,10 @@ dataset, so read both as a floor set by the raw features.
 - Set an **`HF_TOKEN`** Space secret (fine-grained) with: **read** on
   `ScientaLab/primo` (the public `datasets.yaml` manifest) and
   `ScientaLab/primo-labels` (the private `tasks.yaml` registry +
-  `<task_id>/labels.csv` and perturbation `<task_id>/targets.npz`), and **read + write** on
+  `<task_id>/labels.csv`), and **read + write** on
   `ScientaLab/primo-results` (the persisted leaderboard).
 - Results persist as one normalized `task_results.csv` (`model_name, task_id,
-  score, repeat_scores, diagnostics, submitted_at, is_baseline, hf_username`) in the results dataset; the
+  score, repeat_scores, submitted_at, is_baseline, hf_username`) in the results dataset; the
   leaderboard is recomputed from it by joining the registry, so it survives
   Space restarts.
 - A board keeps each name's **latest** submission, so a model name is owned by

app.py

@@ -8,18 +8,17 @@ board and ``?tab=contribute`` opens a tab -- which is what the rail links and th
 open cards use. The tab strip is hidden in CSS; the rail is the navigation.
 
 Upload one embedding file spanning every dataset (rows keyed by ``dataset_id``
-+ ``sample_id``); a fixed task probe scores each task (a dataset may carry
++ ``sample_id``); a fixed linear probe scores each task (a dataset may carry
 several hidden targets). Results roll up into boards -- the whole modality, one
 therapeutic area, one task family -- and each board ranks the models that
-covered every task our baseline scored, one column per category in its native
-metric (AUROC, Pearson, or centered Spearman), plus a ``Mean`` of those columns
-that orders the rows and is labelled as the cross-metric average it is.
+covered all of its tasks, one column per category in its native metric (AUROC or
+Pearson), plus a ``Mean`` of those columns that orders the rows and is labelled
+as the cross-metric average it is.
 
 Disclosure policy -- what the public pages may show:
   per task   disease, tissue, area, what is predicted, class names, n, metric
   aggregated the public archives the cohorts sit in, and their licences
-  never      study accessions, dataset_id -> cohort, hub keys, per-sample labels,
-             pairing identifiers or post-treatment responses
+  never      study accessions, dataset_id -> cohort, hub keys, per-sample labels
 Naming the accession behind ``d002`` would put every label one GEO download
 away, so ``sources`` is collapsed to its archive (NCBI GEO / EMBL-EBI
 ArrayExpress) and ``citation`` is never rendered at all. Submitters can publish
@@ -209,9 +208,9 @@ def _summary(result: dict, model_name: str) -> str:
         lines.append("\n✅ **full coverage.** You are ranked on every board.")
     else:
         lines.append(
-            "\n⚠️ **partial coverage.** You are ranked on the boards whose scored "
-            "tasks you covered in full, and your scores always appear in each "
-            "board's **per-task** table."
+            "\n⚠️ **partial coverage.** You are ranked on the boards whose tasks you "
+            "covered in full, and your scores always appear in each board's "
+            "**per-task** table."
         )
         if result["missing"]:
             lines.append(f"- missing from file: {result['missing']}")
@@ -305,9 +304,6 @@ def evaluate(
             "repeat_scores": json.dumps(
                 [round(value, 4) for value in task.repeat_scores]
             ),
-            "diagnostics": json.dumps(
-                {key: round(value, 4) for key, value in task.diagnostics.items()}
-            ),
             "submitted_at": submitted_at,
             IS_BASELINE: False,
             OWNER: profile.username,

boards.py

@@ -31,12 +31,7 @@ from dataclasses import dataclass
 
 from evaluator import _norm_id
 
-METRIC_LABEL = {
-    "auroc": "AUROC",
-    "pearson": "Pearson",
-    "residual_spearman": "Residual Spearman score",
-    "centered_spearman": "Centered Spearman score",
-}
+METRIC_LABEL = {"auroc": "AUROC", "pearson": "Pearson"}
 MODALITY_LABEL = {
     "bulk RNA": "bulk RNAseq",
     "single-cell RNA": "single-cell RNAseq",
@@ -54,13 +49,11 @@ GROUP_NOTE = {
 
 CODE_LENGTH = 3
 FALLBACK_CODE = "n/a"
-DISPLAY_LABEL = {"perturbation_prediction": "Perturbation response"}
 
 CATEGORY_BLURB = {
     "treatment_outcome": "Will this patient respond to the drug?",
     "clinical_scores": "How severe is this patient's disease?",
     "endotype": "Which molecular subtype is this patient?",
-    "perturbation_prediction": "How will treatment change this patient's expression?",
 }
 
 N_FEATURED = 3
@@ -69,7 +62,7 @@ MAX_LISTED = 3
 
 def label(value: str) -> str:
     """``treatment_outcome`` -> ``Treatment outcome``; leaves free text alone."""
-    return DISPLAY_LABEL.get(value, value.replace("_", " ").capitalize())
+    return value.replace("_", " ").capitalize()
 
 
 def metric_label(metric: str) -> str:

evaluator.py

@@ -3,8 +3,8 @@
 Loads one embedding submission that spans every dataset (rows keyed by
 ``dataset_id`` + ``sample_id``), then scores each TASK = (dataset, target): a
 dataset is embedded once and reused across all its tasks. Per task it fits a
-fixed probe—linear for scalar targets and multi-output ridge for response
-matrices—over repeated frozen CV folds or a fixed transfer split. Turning those
+fixed linear probe, averaged over repeated frozen CV folds or fit once on a
+fixed train/test split (transfer), and produces predictions. Turning those
 predictions into scores + leaderboard aggregates is the job of ``scoring.py``;
 this module only reads data and runs the probe.
 
@@ -14,9 +14,8 @@ failed Hugging Face fetch or an unexpected bug, raises ``EvaluatorError`` so it
 is never silently charged against the submitter.
 
 Reads the public ``datasets.yaml`` manifest (what the submitter embeds) plus the
-private ``tasks.yaml`` registry, each task's private ``labels.csv``, and hidden
-perturbation ``targets.npz`` matrices where applicable. No monorepo imports, so
-it runs unchanged inside a public Hugging Face Space.
+private ``tasks.yaml`` registry and each task's private ``labels.csv``. No
+monorepo imports, so it runs unchanged inside a public Hugging Face Space.
 
 Deps: numpy, pandas, scikit-learn, pyyaml. ``huggingface_hub`` is used only by
 the fetch helpers / CLI, imported lazily.
@@ -33,13 +32,7 @@ from pathlib import Path
 import numpy as np
 import pandas as pd
 import yaml
-from scoring import (
-    METRICS,
-    TaskScore,
-    category_means,
-    compute_residual_sample_spearman,
-    compute_target_centered_sample_spearman,
-)
+from scoring import METRICS, TaskScore, category_means
 from sklearn.linear_model import LogisticRegressionCV, RidgeCV
 from sklearn.model_selection import StratifiedKFold
 from sklearn.preprocessing import StandardScaler
@@ -53,15 +46,7 @@ FOLD_PREFIX = f"{FOLD}_"
 SPLIT = "split"
 SPLIT_TRAIN = "train"
 SPLIT_TEST = "test"
-PAIR_ID = "pair_id"
-CONTROL_ID = "control_id"
-PERTURBED_ID = "perturbed_id"
-PAIRING = "pairing"
-PAIRED = "paired"
-CARTESIAN = "cartesian"
-PERTURBATION = "perturbation"
-RESIDUAL_SPEARMAN = "residual_spearman"
-CENTERED_SPEARMAN = "centered_spearman"
+
 ORG = "ScientaLab"
 PUBLIC_REPO = f"{ORG}/primo"
 LABELS_REPO = f"{ORG}/primo-labels"
@@ -70,10 +55,8 @@ RESULTS_REPO = "ScientaLab/primo-results"
 MANIFEST_FILENAME = "datasets.yaml"
 TASKS_FILENAME = "tasks.yaml"
 LABELS_FILENAME = "labels.csv"
-TARGETS_FILENAME = "targets.npz"
 
 RIDGE_ALPHAS = np.logspace(-3.0, 6.0, 19)
-PERTURBATION_TEST_Z_CLIP = 20.0
 LOGREG_CS = 10
 INNER_CV = 5
 MAX_ITER = 5000
@@ -111,14 +94,6 @@ class TaskOutcome:
     reason: str | None = None
 
 
-@dataclass(frozen=True)
-class PerturbationTargets:
-    """Hidden response vectors and their aligned gene identifiers."""
-
-    delta: np.ndarray
-    gene_ids: np.ndarray
-
-
 def _norm_id(value: object) -> str:
     """Normalise an id so int/str/float spellings of the same id join."""
     if isinstance(value, float) and value.is_integer():
@@ -145,9 +120,8 @@ def _fold_columns(labels: pd.DataFrame) -> list[str]:
     ]
     if invalid:
         raise EvaluatorError(f"labels.csv has invalid repeat-fold columns: {invalid}")
-    first = [FOLD] if FOLD in labels.columns else []
     return [
-        *first,
+        FOLD,
         *sorted(repeated, key=lambda column: int(column.removeprefix(FOLD_PREFIX))),
     ]
 
@@ -155,48 +129,19 @@ def _fold_columns(labels: pd.DataFrame) -> list[str]:
 def load_labels(path: str | Path) -> pd.DataFrame:
     """Read private labels with repeated CV folds or one transfer split."""
     df = pd.read_csv(path)
-    is_perturbation = PAIR_ID in df.columns
-    required = {SAMPLE_ID, PAIR_ID} if is_perturbation else {SAMPLE_ID, LABEL}
-    missing = required - set(df.columns)
+    missing = {SAMPLE_ID, LABEL} - set(df.columns)
     if missing:
         raise EvaluatorError(f"labels.csv missing columns: {sorted(missing)}")
-    has_folds = bool(_fold_columns(df))
-    has_split = SPLIT in df.columns
-    pairing_values = (
-        set(df[PAIRING].astype(str)) if is_perturbation and PAIRING in df else set()
-    )
-    is_cartesian = pairing_values == {CARTESIAN}
-    if sum((has_folds, has_split, is_cartesian)) != 1:
+    has_folds = FOLD in df.columns
+    if has_folds == (SPLIT in df.columns):
         raise EvaluatorError(
-            "labels.csv needs exactly one evaluation assignment: folds, transfer "
-            "split, or Cartesian pairing."
+            f"labels.csv needs exactly one of '{FOLD}' (CV) or '{SPLIT}' (transfer)."
         )
-    if is_cartesian:
-        missing = {CONTROL_ID, PERTURBED_ID, PAIRING} - set(df.columns)
-        if missing:
-            raise EvaluatorError(
-                f"Cartesian labels.csv missing columns: {sorted(missing)}"
-            )
+    if has_folds:
+        _fold_columns(df)
     return df
 
 
-def load_targets(path: str | Path) -> PerturbationTargets:
-    """Read and validate one task's hidden perturbation responses."""
-    with np.load(path, allow_pickle=False) as data:
-        missing = {"delta", "gene_ids"} - set(data.files)
-        if missing:
-            raise EvaluatorError(f"targets.npz missing arrays: {sorted(missing)}")
-        delta = np.asarray(data["delta"], dtype=float)
-        gene_ids = np.asarray(data["gene_ids"]).astype(str)
-    if delta.ndim != 2:
-        raise EvaluatorError(f"targets delta must be 2D, got shape {delta.shape}")
-    if gene_ids.ndim != 1 or len(gene_ids) != delta.shape[1]:
-        raise EvaluatorError("targets gene_ids must align with delta columns")
-    if not np.isfinite(delta).all():
-        raise EvaluatorError("targets delta contains NaN or inf values")
-    return PerturbationTargets(delta=delta, gene_ids=gene_ids)
-
-
 def load_submission(path: str | Path) -> dict[str, pd.DataFrame]:
     """Load a multi-dataset submission into ``{dataset_id: raw_block_frame}``."""
     ext = Path(path).suffix.lower()
@@ -407,198 +352,13 @@ def _transfer_predict(
     return y[test], preds
 
 
-def _select_degs(delta_train: np.ndarray, n_top: int) -> np.ndarray:
-    """Select the strongest mean absolute responses using training rows only."""
-    n_keep = min(int(n_top), delta_train.shape[1])
-    if n_keep <= 0:
-        raise EvaluatorError("perturbation task has no response genes to select")
-    effect = np.abs(np.mean(delta_train, axis=0))
-    return np.argsort(effect, kind="stable")[-n_keep:]
-
-
-def _mean_scores(scores: list[float]) -> float:
-    """Average finite fold scores."""
-    finite = np.asarray(scores)[np.isfinite(scores)]
-    return float(np.mean(finite)) if len(finite) else float("nan")
-
-
-def _perturbation_fold(
-    matrix: np.ndarray,
-    delta: np.ndarray,
-    train: np.ndarray,
-    test: np.ndarray,
-    n_top: int,
-    target_centered: bool,
-) -> float:
-    """Fit and score one leakage-safe perturbation fold."""
-    if not train.any() or not test.any():
-        raise EvaluatorError("perturbation fold has an empty train or test side")
-    genes = _select_degs(delta[train], n_top)
-    y_train = delta[train][:, genes]
-    y_test = delta[test][:, genes]
-    scaler = StandardScaler().fit(matrix[train])
-    x_train = scaler.transform(matrix[train])
-    x_test = np.clip(
-        scaler.transform(matrix[test]),
-        -PERTURBATION_TEST_Z_CLIP,
-        PERTURBATION_TEST_Z_CLIP,
-    )
-    predictions = RidgeCV(alphas=RIDGE_ALPHAS).fit(x_train, y_train).predict(x_test)
-    if target_centered:
-        correlation = compute_target_centered_sample_spearman(y_test, predictions)
-    else:
-        training_mean = np.mean(y_train, axis=0)
-        correlation = compute_residual_sample_spearman(
-            y_test, predictions, training_mean
-        )
-    if not np.isfinite(correlation):
-        raise SubmissionError("a perturbation fold has a degenerate Spearman score")
-    return float(np.clip(correlation, -1.0, 1.0))
-
-
-def _paired_perturbation_scores(
-    matrix: np.ndarray,
-    delta: np.ndarray,
-    labels: pd.DataFrame,
-    n_top: int,
-) -> tuple[float, ...]:
-    """Score three frozen subject-grouped CV partitions."""
-    repeat_scores = []
-    for column in _fold_columns(labels):
-        fold_metrics = []
-        folds = labels[column].to_numpy()
-        for fold in sorted(np.unique(folds)):
-            test = folds == fold
-            fold_metrics.append(
-                _perturbation_fold(matrix, delta, ~test, test, n_top, False)
-            )
-        repeat_scores.append(_mean_scores(fold_metrics))
-    return tuple(repeat_scores)
-
-
-def _transfer_perturbation_scores(
-    matrix: np.ndarray,
-    delta: np.ndarray,
-    labels: pd.DataFrame,
-    n_top: int,
-    target_centered: bool,
-) -> tuple[tuple[float, ...], int]:
-    """Train every response-decoding component on the source disease only."""
-    split = labels[SPLIT].astype(str).to_numpy()
-    train, test = split == SPLIT_TRAIN, split == SPLIT_TEST
-    metrics = _perturbation_fold(matrix, delta, train, test, n_top, target_centered)
-    return (metrics,), int(test.sum())
-
-
-def _cartesian_partitions(
-    control_ids: np.ndarray, perturbed_ids: np.ndarray
-) -> list[list[tuple[np.ndarray, np.ndarray]]]:
-    """Build six 3-fold specimen-isolated partitions for a 3x3 response grid."""
-    from itertools import permutations
-
-    controls = np.unique(control_ids)
-    perturbed = np.unique(perturbed_ids)
-    if len(controls) != 3 or len(perturbed) != 3:
-        raise EvaluatorError(
-            "Cartesian perturbation evaluation requires exactly three controls "
-            "and three perturbed specimens."
-        )
-    repeats = []
-    for assignment in permutations(perturbed):
-        folds = []
-        for control_id, perturbed_id in zip(controls, assignment):
-            test = (control_ids == control_id) & (perturbed_ids == perturbed_id)
-            train = (control_ids != control_id) & (perturbed_ids != perturbed_id)
-            if test.sum() != 1 or train.sum() != 4:
-                raise EvaluatorError("Cartesian labels do not form a complete 3x3 grid")
-            folds.append((train, test))
-        repeats.append(folds)
-    return repeats
-
-
-def _cartesian_perturbation_scores(
-    matrix: np.ndarray,
-    delta: np.ndarray,
-    labels: pd.DataFrame,
-    n_top: int,
-) -> tuple[float, ...]:
-    """Evaluate all control-to-perturbed permutations without specimen leakage."""
-    repeats = _cartesian_partitions(
-        labels[CONTROL_ID].astype(str).to_numpy(),
-        labels[PERTURBED_ID].astype(str).to_numpy(),
-    )
-    repeat_scores = []
-    for folds in repeats:
-        fold_metrics = []
-        for train, test in folds:
-            fold_metrics.append(
-                _perturbation_fold(matrix, delta, train, test, n_top, False)
-            )
-        repeat_scores.append(_mean_scores(fold_metrics))
-    return tuple(repeat_scores)
-
-
-def _score_perturbation_task(
-    task: dict,
-    labels: pd.DataFrame,
-    emb: pd.DataFrame,
-    targets: PerturbationTargets,
-) -> TaskScore:
-    """Decode one hidden response matrix from submitted baseline embeddings."""
-    if len(labels) != len(targets.delta):
-        raise EvaluatorError("labels.csv and targets.npz have different row counts")
-    matrix = _align(labels, emb)
-    n_top = int(task.get("n_top_de_genes", 0))
-    pairing = str(task.get(PAIRING, labels.get(PAIRING, pd.Series([PAIRED])).iloc[0]))
-    if SPLIT in labels:
-        repeat_scores, n_samples = _transfer_perturbation_scores(
-            matrix,
-            targets.delta,
-            labels,
-            n_top,
-            bool(task.get("target_centered", False)),
-        )
-    elif pairing == CARTESIAN:
-        repeat_scores = _cartesian_perturbation_scores(
-            matrix, targets.delta, labels, n_top
-        )
-        n_samples = len(labels)
-    else:
-        repeat_scores = _paired_perturbation_scores(
-            matrix, targets.delta, labels, n_top
-        )
-        n_samples = len(labels)
-    return TaskScore(
-        task_id=_norm_id(task[TASK_ID]),
-        dataset_id=_norm_id(task[DATASET_ID]),
-        category=str(task["category"]),
-        metric=str(task.get("metric", RESIDUAL_SPEARMAN)),
-        score=float(np.mean(repeat_scores)),
-        n_samples=n_samples,
-        repeat_scores=repeat_scores,
-    )
-
-
-def score_task(
-    task: dict,
-    labels: pd.DataFrame,
-    emb: pd.DataFrame,
-    targets: PerturbationTargets | None = None,
-) -> TaskScore:
+def score_task(task: dict, labels: pd.DataFrame, emb: pd.DataFrame) -> TaskScore:
     """Score one task: repeated fold-wise CV, or a fixed transfer split.
 
-    Private biology (disease, tissue, and target names) never leaks in.
+    Reads only ``task_id``, ``dataset_id``, ``metric``, ``task_type`` and
+    ``category``. Private biology (disease/tissue/target) never leaks in.
     """
     metric = task["metric"]
-    if task.get("task_type") == PERTURBATION:
-        if metric not in {CENTERED_SPEARMAN, RESIDUAL_SPEARMAN}:
-            raise EvaluatorError(
-                f"perturbation task {task.get(TASK_ID)} needs metric "
-                f"'{CENTERED_SPEARMAN}'"
-            )
-        if targets is None:
-            raise EvaluatorError("perturbation task is missing targets.npz")
-        return _score_perturbation_task(task, labels, emb, targets)
     if metric not in METRICS:
         raise EvaluatorError(f"unknown metric '{metric}' for task {task.get(TASK_ID)}")
     task_type = task["task_type"]
@@ -695,16 +455,6 @@ def fetch_task_labels(task_id: str, token: str | None = None) -> pd.DataFrame:
     return load_labels(path)
 
 
-def fetch_task_targets(task_id: str, token: str | None = None) -> PerturbationTargets:
-    """Download a task's private compressed response matrix."""
-    from huggingface_hub import hf_hub_download
-
-    path = hf_hub_download(
-        LABELS_REPO, f"{task_id}/{TARGETS_FILENAME}", repo_type="dataset", token=token
-    )
-    return load_targets(path)
-
-
 def _retry(fn):
     """Call ``fn``, retrying transient failures with linear backoff."""
     last: Exception | None = None
@@ -718,14 +468,7 @@ def _retry(fn):
     raise last
 
 
-def _task_outcome(
-    task: dict,
-    emb,
-    block_error,
-    fetch_labels,
-    fetch_targets,
-    token,
-) -> TaskOutcome:
+def _task_outcome(task: dict, emb, block_error, fetch_labels, token) -> TaskOutcome:
     """Score one task, mapping each failure to the right category."""
     task_id = _norm_id(task[TASK_ID])
     dataset_id = _norm_id(task[DATASET_ID])
@@ -739,16 +482,8 @@ def _task_outcome(
         raise EvaluatorError(
             f"could not load evaluation data for a task: {error}"
         ) from error
-    targets = None
-    if task.get("task_type") == PERTURBATION:
-        try:
-            targets = _retry(lambda: fetch_targets(task_id, token))
-        except Exception as error:  # noqa: BLE001
-            raise EvaluatorError(
-                f"could not load perturbation targets for a task: {error}"
-            ) from error
     try:
-        score = score_task(task, labels, emb, targets)
+        score = score_task(task, labels, emb)
     except SubmissionError as error:
         return TaskOutcome(task_id, dataset_id, "invalid", reason=str(error))
     if not np.isfinite(score.score):
@@ -768,7 +503,6 @@ def score_all(
     datasets: list[dict] | None = None,
     tasks: list[dict] | None = None,
     fetch_labels=None,
-    fetch_targets=None,
 ) -> dict:
     """Score a submission per task and roll it up into a blind result.
 
@@ -798,7 +532,6 @@ def score_all(
                 f"could not load the task registry: {error}"
             ) from error
     fetch_labels = fetch_labels or fetch_task_labels
-    fetch_targets = fetch_targets or fetch_task_targets
 
     emb_by_ds, block_error = {}, {}
     for dataset_id, block in blocks.items():
@@ -813,7 +546,6 @@ def score_all(
             emb_by_ds.get(_norm_id(task[DATASET_ID])),
             block_error.get(_norm_id(task[DATASET_ID])),
             fetch_labels,
-            fetch_targets,
             token,
         )
         for task in scoreable_tasks(tasks, valid)

leaderboard.py

@@ -6,9 +6,9 @@ apart from Gradio and from any I/O so it unit-tests without a network
 
 Two tables per board, deliberately different:
 
-``ranked_table``   the leaderboard. Only models that covered EVERY task the Elo
-                   anchor scored are listed -- a partial run cannot win by
-                   skipping the hard cohorts.
+``ranked_table``   the leaderboard. Only models that covered EVERY task of the
+                   board are listed -- a partial run cannot win by skipping the
+                   hard cohorts.
 ``per_task_table`` one row per task, one column per model. Keeps partial
                    submissions, so a newcomer who covered three cohorts sees
                    their numbers instead of vanishing. Never a ranking.
@@ -53,7 +53,6 @@ SCORE_DECIMALS = 3
 ELO_START = 1000.0
 ELO_SCALE = 400 / math.log(10)
 ELO_REGULARIZATION = 0.1
-ELO_ANCHOR_MODEL = "HVG-1200-genes"
 
 
 @dataclass(frozen=True)
@@ -163,58 +162,28 @@ def _repeat_scores(row: pd.Series) -> tuple[float, ...]:
     return (float(row["score"]),)
 
 
-def _is_anchor(model: str, is_baseline: bool) -> bool:
-    """Is this entry the Elo anchor?
-
-    Matched on ``(name, is_baseline)``, never the name alone. A published
-    baseline carries no ``hf_username``, which deliberately leaves its name free
-    for a submitter to reuse -- ``latest_only`` then keeps the two as separate
-    entries. Matching on the name alone would let that submission take over the
-    anchor and redefine the scale.
-    """
-    return is_baseline and model == ELO_ANCHOR_MODEL
-
-
 def _entries(df: pd.DataFrame, by_id: dict[str, dict], board: Board) -> list[dict]:
     """Per-category means for every model covering the board's scored tasks.
 
-    The ANCHOR decides which tasks are ranked: a task it did not score is not a
-    leaderboard task, however many submissions covered it. That is what keeps the
-    anchor at full coverage on every board, hence always present to hold the Elo
-    scale down. The task stays visible in the per-task table either way.
-
-    Without the anchor the task set falls back to every task with a finite score
-    from any latest submission, and the Elo scale to a mean-centred one (see
-    ``_comparison_stats``). Ranking a board is worth more than ranking it on a
-    fixed scale. That fallback is decided PER BOARD, not globally: a board the
-    anchor has not been scored on yet -- a task family added since the baseline
-    last ran -- still ranks the models that covered it.
+    A task with no finite score from any latest submission is not a leaderboard
+    task yet. Keeping it out of the coverage check lets the rest of the board be
+    ranked while still leaving the task visible in the per-task table.
     """
     scoped = _board_registry(by_id, board)
     latest = latest_only(df)
-    if latest.empty:
-        return []
-    on_board = latest[latest["task_id"].map(_norm_id).isin(list(board.task_ids))]
-    anchor_rows = on_board[
-        [
-            _is_anchor(str(model), bool(flag))
-            for model, flag in zip(on_board[MODEL_NAME], on_board[IS_BASELINE])
-        ]
-    ]
-    ranked_rows = on_board if anchor_rows.empty else anchor_rows
     scored_tasks = {
         _norm_id(row["task_id"])
-        for _, row in ranked_rows.iterrows()
-        if math.isfinite(float(row["score"]))
+        for _, row in latest.iterrows()
+        if _norm_id(row["task_id"]) in board.task_ids
+        and math.isfinite(float(row["score"]))
     }
     if not scored_tasks:
         return []
 
     entries = []
-    for (model, is_baseline, submission_id), rows in latest.groupby(
-        [MODEL_NAME, IS_BASELINE, SUBMISSION_ID]
+    for (model, is_baseline, submitted, submission_id), rows in latest.groupby(
+        [MODEL_NAME, IS_BASELINE, "submitted_at", SUBMISSION_ID]
     ):
-        submitted = rows["submitted_at"].max()
         task_rows = rows[
             rows["task_id"].map(_norm_id).isin(list(scored_tasks))
             & rows["score"].map(lambda score: math.isfinite(float(score)))
@@ -260,26 +229,12 @@ def _comparison_stats(entries: list[dict], task_ids: set[str]) -> None:
     updating a rating sequentially. Thus the result is invariant to the order
     in which tasks or models are visited. Repeats of one task share total weight
     one, matching TabArena's equal-per-dataset treatment.
-
-    The scale is ANCHORED: ``ELO_ANCHOR_MODEL`` is held at ``ELO_START`` the way
-    TabArena calibrates 1000 Elo to its default random forest. That is what gives
-    a rating a meaning of its own -- "beats the 1,200 highest-variance genes this
-    often" -- and keeps it steady as submissions arrive; centring on the mean
-    instead moves every model each time a new one is published.
-
-    The anchor is pinned DURING the fit rather than subtracted after it, because
-    the ridge penalty is not gauge-invariant: it shrinks toward whichever model
-    the parameterization pins, and a later shift cannot undo that. Pinning the
-    anchor aims the shrinkage at it, so a model with little evidence reads as the
-    baseline rather than as an alphabetical accident.
-
-    Entries are keyed by POSITION, never by name -- see ``_is_anchor`` for why two
-    entries can share one.
     """
     if not entries:
         return
-    indices = list(range(len(entries)))
-    ranks: dict[int, list[float]] = {index: [] for index in indices}
+    models = [entry["model_name"] for entry in entries]
+    model_index = {model: index for index, model in enumerate(models)}
+    ranks = {model: [] for model in models}
     valid_games = [
         task_id
         for task_id in task_ids
@@ -292,44 +247,32 @@ def _comparison_stats(entries: list[dict], task_ids: set[str]) -> None:
     battles: list[tuple[int, int, float, float]] = []
     for task_id in sorted(valid_games):
         ordered = sorted(
-            indices,
-            key=lambda index: (
-                -entries[index]["game_scores"][task_id],
-                entries[index]["model_name"],
-            ),
+            entries,
+            key=lambda entry: (-entry["game_scores"][task_id], entry["model_name"]),
         )
-        for position, index in enumerate(ordered, start=1):
-            score = entries[index]["game_scores"][task_id]
+        for position, entry in enumerate(ordered, start=1):
+            model, score = entry["model_name"], entry["game_scores"][task_id]
             tied_positions = [
-                rank
-                for rank, other in enumerate(ordered, start=1)
-                if entries[other]["game_scores"][task_id] == score
+                index
+                for index, other in enumerate(ordered, start=1)
+                if other["game_scores"][task_id] == score
             ]
-            ranks[index].append(sum(tied_positions) / len(tied_positions))
+            ranks[model].append(sum(tied_positions) / len(tied_positions))
             for opponent in ordered[position:]:
-                other_score = entries[opponent]["game_scores"][task_id]
+                other = opponent["model_name"]
+                outcome = 1.0 if score > opponent["game_scores"][task_id] else 0.5
                 task, _ = task_id.rsplit(":", maxsplit=1)
                 battles.append(
                     (
-                        index,
-                        opponent,
-                        1.0 if score > other_score else 0.5,
+                        model_index[model],
+                        model_index[other],
+                        outcome,
                         1 / repeats_per_task[task],
                     )
                 )
 
-    anchor = next(
-        (
-            index
-            for index, entry in enumerate(entries)
-            if _is_anchor(entry["model_name"], entry["is_baseline"])
-        ),
-        None,
-    )
-    pinned = len(entries) - 1 if anchor is None else anchor
-
     def objective(theta_free: np.ndarray) -> tuple[float, np.ndarray]:
-        theta = np.insert(theta_free, pinned, 0.0)
+        theta = np.append(theta_free, 0.0)
         loss = 0.0
         gradient = np.zeros_like(theta)
         for left, right, outcome, weight in battles:
@@ -340,25 +283,25 @@ def _comparison_stats(entries: list[dict], task_ids: set[str]) -> None:
             gradient[right] -= error
         loss += ELO_REGULARIZATION * float(np.dot(theta, theta)) / 2
         gradient += ELO_REGULARIZATION * theta
-        return float(loss), np.delete(gradient, pinned)
+        return float(loss), gradient[:-1]
 
-    if battles and len(entries) > 1:
+    if battles and len(models) > 1:
         fit = minimize(
             objective,
-            np.zeros(len(entries) - 1),
+            np.zeros(len(models) - 1),
             jac=True,
             method="BFGS",
         )
-        theta = np.insert(fit.x, pinned, 0.0) if fit.success else np.zeros(len(entries))
-        if anchor is None:
-            theta -= theta.mean()
-        ratings = ELO_START + ELO_SCALE * theta
+        theta = np.append(fit.x, 0.0) if fit.success else np.zeros(len(models))
+        theta -= theta.mean()
+        ratings = dict(zip(models, ELO_START + ELO_SCALE * theta))
     else:
-        ratings = np.full(len(entries), ELO_START)
-    for index, entry in enumerate(entries):
-        entry["elo"] = float(ratings[index])
+        ratings = {model: ELO_START for model in models}
+    for entry in entries:
+        model = entry["model_name"]
+        entry["elo"] = ratings[model]
         entry["mean_rank"] = (
-            float(np.mean(ranks[index])) if ranks[index] else float("nan")
+            float(np.mean(ranks[model])) if ranks[model] else float("nan")
         )
         entry["mean_score"] = float(np.mean(list(entry["task_scores"].values())))
 
@@ -366,13 +309,13 @@ def _comparison_stats(entries: list[dict], task_ids: set[str]) -> None:
 def ranked_table(
     df: pd.DataFrame, by_id: dict[str, dict], board: Board
 ) -> pd.DataFrame:
-    """Rank the models that covered every task of ``board`` the anchor scored.
+    """Rank the models that covered every task of ``board``.
 
-    One column per task category, each in its native metric (AUROC, Pearson, or
-    centered Spearman); two metrics never share a column. ``Mean`` averages those
-    columns -- it does cross metrics, which is why the pages call it a tie-break
-    rather than a score -- and is dropped when the board holds a single category,
-    where it would just repeat it.
+    One column per task category, each in its native metric (AUROC or Pearson);
+    two metrics never share a column. ``Mean`` averages those columns -- it does
+    cross metrics, which is why the pages call it a tie-break rather than a
+    score -- and is dropped when the board holds a single category, where it
+    would just repeat it.
     """
     entries = _entries(df, by_id, board)
     if not entries:

pages/about.md

@@ -3,7 +3,7 @@
 **Can a foundation model turn a patient's omics data into an embedding that
 predicts meaningful outcomes?**
 
-You embed each patient. We fit a fixed probe on top and check whether that
+You embed each patient. We fit a linear probe on top and check whether that
 embedding predicts a real clinical outcome. You never see the labels, so you
 cannot tune to them, and when a score comes out high the credit belongs to the
 *embedding*.
@@ -13,32 +13,26 @@ representation is any use in the clinic, and that is the gap PRIMO tries to fill
 
 ## How a submission is scored
 
-1. **You send embeddings, never predictions**: one vector per baseline sample.
-2. **We fit the probe**: ridge for scalar regression, logistic for
-   classification, or multi-output ridge regression for perturbation
-   responses. Hyperparameters are picked by inner cross-validation.
+1. **You send embeddings, never predictions**: one vector per sample.
+2. **We fit the probe**: ridge for regression, logistic for classification,
+   regularisation picked by inner cross-validation.
 3. **Scores use repeated subject-grouped cross-validation.** We average the
    fold-level metric over three frozen five-fold partitions. Transfer tasks use
-   a fixed split instead: train on one source cohort, test on the target cohort.
-   Perturbation tasks decode hidden post-minus-pre expression responses; the
-   mouse IBD task reuses three baseline embeddings across its hidden 3×3 response
-   grid and isolates both specimens in every held-out fold.
-4. **Each task family keeps its own metric.** AUROC, Pearson r and centered
-   Spearman never share a category column.
+   a fixed split instead: train on one drug's patients, test on the other's.
+4. **Each task family keeps its own metric.** We never average an AUROC with a
+   Pearson r inside one task score, and the two never share a column.
 
 The probe is identical for everyone, which is what makes the numbers comparable:
 what changes between two rows is the representation behind them. The rows marked
 `(baseline)` are our own submissions, scored by that same probe on the log-CPM
 expression itself, whole or cut to its most variable genes. That cut never reads
 a label, but it is made over every sample, so read those rows as a floor set by
-the raw features rather than as a competing model. Perturbation centered Spearman
-is reported directly on `[-1, 1]`, with 0 as the uninformative shared-response
-anchor.
+the raw features rather than as a competing model.
 
 A board covering more than one family also shows a **Mean** of the family
-columns. It is there to give the table an order, but it does average AUROC,
-Pearson and centered Spearman, so use it as a tie-break and do the real
-comparison on the family columns.
+columns. It is there to give the table an order, but it does average AUROC with
+Pearson, so use it as a tie-break and do the real comparison on the family
+columns.
 
 ## How the results are organised
 
@@ -62,7 +56,7 @@ embeddings.
 |---|---|
 | **Per task** | disease, tissue, therapeutic area, what is predicted, class names, patient count, metric |
 | **In aggregate** | the public archives the cohorts sit in |
-| **Never** | study accessions, which cohort is behind which `dataset_id`, labels, pairing identifiers, and post-treatment responses |
+| **Never** | study accessions, which cohort is behind which `dataset_id`, and the labels |
 
 Every cohort comes from a published study deposited in {repositories}. We name
 the archive but never the accession, as publishing the study id behind a

pages/leaderboard.md

@@ -1,13 +1,6 @@
-**🏆 Ranked**: only models that covered every task of this board the
-`HVG-1200-genes` baseline scored.
+**🏆 Ranked**: only models that covered every task of this board.
 **🔍 Per task**: everyone, partial submissions included.
 
 One column per task family, each in its own metric. **Mean** is the average of
-those columns: it orders the table, but it averages AUROC, Pearson and residual
-Spearman, so read it as a tie-break and compare models on the family columns.
-
-**Elo** is pinned to our `HVG-1200-genes` baseline, which sits at exactly 1000:
-above that line a model beats the 1,200 highest-variance genes, below it it does
-not. +400 Elo is a 10-to-1 expected win rate. The scale does not move when a new
-model is submitted, so ratings stay comparable over time — but only within one
-board, since each board is fitted on its own tasks.
+those columns: it orders the table, but it averages AUROC with Pearson, so read
+it as a tie-break and compare models on the family columns.

pages/submit.md

@@ -17,7 +17,7 @@ Or do it by hand, in **three steps**:
 
 1. **Get the data** → download the datasets from [ScientaLab/primo](https://huggingface.co/datasets/ScientaLab/primo) (start with its `datasets.yaml`).
 2. **Embed every dataset** → build **one** file: `dataset_id`, `sample_id`, then one column per embedding dim (`e0`, `e1`, …). CSV / TSV / Parquet, or NPZ.
-3. **Sign in, fill the form, and hit Evaluate.** Add an institution for group submissions, check **Submitted by the model's authors** when applicable, and provide a paper link to make the model name clickable. A fixed task probe scores each hidden task (AUROC, Pearson or centered Spearman), reported per task category in its native metric.
+3. **Sign in, fill the form, and hit Evaluate.** Add an institution for group submissions, check **Submitted by the model's authors** when applicable, and provide a paper link to make the model name clickable. A fixed linear probe scores each hidden task (AUROC or Pearson), reported per task category in its native metric.
 
 **Example file**
 
@@ -28,9 +28,9 @@ d002,S1,0.31,0.02,-0.15
 ```
 
 **Partial submissions are welcome.** Cover fewer datasets and you are still
-scored: you get ranked on every **board** whose scored tasks you covered in
-full, and your numbers still show up in each board's **per-task** table, so
-nothing you send is thrown away.
+scored: you get ranked on every **board** whose tasks you covered in full, and
+your numbers still show up in each board's **per-task** table, so nothing you
+send is thrown away.
 
 **Your first target is the baselines.** We run our own reference submissions on
 the log-CPM expression itself, whole or cut down to its most variable genes, and

pages/tasks.md

@@ -1,3 +1,2 @@
-Every scoreable task. Each row is a hidden clinical target or perturbation
-response, scored by the same fixed task probe. The cohorts stay anonymous, but the
-biology is on the table.
+Every scoreable task. Each row is a hidden clinical target, scored by the same
+linear probe. The cohorts stay anonymous, but the biology is on the table.

primo.css

@@ -194,6 +194,13 @@ a:hover { color: var(--marine-700); }
   background: var(--marine-700); color: #fff; box-shadow: var(--shadow-sm); font: 400 12px/1.4 var(--font-sans);
   text-align: left; white-space: normal; opacity: 0; pointer-events: none; transform: translateY(3px); transition: opacity .12s ease, transform .12s ease; }
 .pm-score-guide:hover::after, .pm-score-guide:focus::after { opacity: 1; transform: translateY(0); }
+.pm-task-guide { position: relative; display: inline-block; cursor: help; border-bottom: 1px dotted var(--marine-200); }
+.pm-task-guide:focus-visible { outline: 2px solid var(--marine-400); outline-offset: 2px; }
+.pm-task-guide::after { content: attr(data-tooltip); position: absolute; z-index: 10; left: 0; bottom: calc(100% + 7px); width: min(360px, 72vw);
+  padding: 9px 11px; border: 1px solid var(--marine-200); border-radius: var(--radius-sm); background: var(--marine-700); color: #fff;
+  box-shadow: var(--shadow-sm); font: 400 12px/1.45 var(--font-sans); text-align: left; white-space: normal;
+  opacity: 0; pointer-events: none; transform: translateY(3px); transition: opacity .12s ease, transform .12s ease; }
+.pm-task-guide:hover::after, .pm-task-guide:focus::after { opacity: 1; transform: translateY(0); }
 .pm-table .pm-score { text-align: right; font-family: var(--font-mono); font-size: .9375rem; color: var(--marine-700); }
 .pm-table .pm-metric { font-family: var(--font-mono); font-size: 12px; color: var(--marine-700); }
 .pm-table .pm-groupline td { padding-top: 22px; border-bottom: 0;

render.py

@@ -268,6 +268,28 @@ def _score_guide(col: str, row: pd.Series) -> str | None:
     return METRIC_GUIDES.get(metric) if metric is not None else None
 
 
+def _task_tooltip(task: dict) -> str:
+    """Summarize a task's cohort, input profiles, and prediction target."""
+    patients = int(task.get("n_samples") or 0)
+    patient_label = "patient" if patients == 1 else "patients"
+    diseases = ", ".join(str(d) for d in task.get("diseases") or [])
+    tissue = str(task.get("tissue") or "unknown tissue")
+    modality = "bulk RNAseq" if task.get("modality") == "bulk RNA" else str(
+        task.get("modality") or "omics"
+    )
+    target = str(task.get("title") or "this clinical target").lower()
+    description = str(task.get("description") or "").strip().rstrip(".")
+    cohort = (
+        f"{patients:,} {patient_label} with {diseases}"
+        if diseases
+        else f"{patients:,} {patient_label}"
+    )
+    return (
+        f"{description}. {cohort}; uses {modality} profiles from {tissue.lower()} tissue; "
+        f"predicts the target: {target}."
+    )
+
+
 def _bold_cells(df: pd.DataFrame, score_cols: list[str], axis: int) -> set:
     """Which ``(row, col)`` cells hold the best value.
 
@@ -305,7 +327,12 @@ def _cell_class(col: str, is_score: bool, blank: bool) -> str:
     return ""
 
 
-def _df_to_table(df: pd.DataFrame, bold_axis: int | None, empty: str) -> str:
+def _df_to_table(
+    df: pd.DataFrame,
+    bold_axis: int | None,
+    empty: str,
+    task_tooltips: dict[str, str] | None = None,
+) -> str:
     """Render a DataFrame as a ``pm-table``, escaping every header and cell."""
     if df.empty:
         return f'<p class="pm-caption">{escape(empty)}</p>'
@@ -350,6 +377,18 @@ def _df_to_table(df: pd.DataFrame, bold_axis: int | None, empty: str) -> str:
                     f'aria-label="{score_label}" '
                     f'data-tooltip="{escaped_guide}">{shown}</span>'
                 )
+            if col == "Task" and text is not None and task_tooltips:
+                tooltip = task_tooltips.get(str(text))
+                if tooltip:
+                    escaped_tooltip = escape(tooltip, quote=True)
+                    task_label = escape(
+                        f"{text}. {tooltip}", quote=True
+                    )
+                    shown = (
+                        '<span class="pm-task-guide" tabindex="0" '
+                        f'aria-label="{task_label}" '
+                        f'data-tooltip="{escaped_tooltip}">{shown}</span>'
+                    )
             cells.append(f'<td class="{cls}"{weight}>{shown}</td>')
         body.append(f"<tr>{''.join(cells)}</tr>")
     return (
@@ -384,6 +423,11 @@ def render_board(board: Board | None, df: pd.DataFrame, by_id: dict[str, dict])
         per_task_table(df, by_id, board),
         bold_axis=1,
         empty="No submission has scored on this board yet.",
+        task_tooltips={
+            str(task.get("title") or task_id): _task_tooltip(task)
+            for task_id, task in by_id.items()
+            if task_id in board.task_ids
+        },
     )
     return (
         '<div class="pm-head"><div><p class="pm-over">Board</p>'
@@ -391,11 +435,9 @@ def render_board(board: Board | None, df: pd.DataFrame, by_id: dict[str, dict])
         '<div class="pm-body">'
         '<div class="pm-group"><div class="pm-group-head">'
         '<p class="pm-over pm-over--marine">Ranked</p>'
-        '<p class="pm-note">Only models that covered every task scored by the '
-        "<code>HVG-1200-genes</code> baseline are ranked. Elo compares models "
-        "pairwise within each task and never compares AUROC, Pearson and "
-        "centered Spearman directly; the baseline holds 1000, so a rating reads "
-        "as how far a model sits above or below the raw expression.</p>"
+        f'<p class="pm-note">Only models that covered all {board.n_tasks} tasks are '
+        "ranked by Elo. Elo compares models pairwise within each task and never "
+        "compares AUROC with Pearson directly.</p>"
         f"</div>{ranked}</div>"
         '<div class="pm-group"><div class="pm-group-head">'
         '<p class="pm-over pm-over--marine">Per task</p>'
@@ -414,7 +456,7 @@ def render_tasks(by_id: dict[str, dict]) -> str:
     )
     return (
         '<div class="pm-head"><div><h1>Tasks</h1>'
-        f"<p>{len(df)} hidden targets. One fixed task probe reads each "
+        f"<p>{len(df)} hidden clinical targets. One fixed linear probe reads each "
         "one out of your embedding; the cohorts stay anonymous, the biology does "
         "not.</p></div></div>"
         f'<div class="pm-body">{table}</div>'

results.py

@@ -44,7 +44,6 @@ RESULT_COLUMNS = [
     "task_id",
     "score",
     "repeat_scores",
-    "diagnostics",
     "submitted_at",
     IS_BASELINE,
     OWNER,

scoring.py

@@ -5,10 +5,10 @@ numbers lives here, kept apart from the probe and from any I/O so it stays easy
 to change as the benchmark grows.
 
 The scoring unit is a task = (dataset, target). Tasks are grouped by their
-``category`` (including treatment outcome, clinical scores, endotype, and
-perturbation response). A category uses a SINGLE metric (enforced in the registry),
-so its leaderboard number is a plain mean of that metric -- AUROC, Pearson and
-centered Spearman are never averaged together inside a category column.
+``category`` (``treatment_outcome`` / ``clinical_scores`` / ``endotype``). A
+category uses a SINGLE metric (enforced in the registry), so its leaderboard
+number is a plain mean of that metric -- AUROC and Pearson are never averaged
+together inside a category column.
 
 ``sort_key`` is the one place they are averaged, to give the board a single
 order. It is shown as the ``Mean`` column, labelled as a cross-metric average so
@@ -21,7 +21,7 @@ without a network and is safe to rework mid-project.
 
 from collections import defaultdict
 from collections.abc import Callable
-from dataclasses import dataclass, field
+from dataclasses import dataclass
 
 import numpy as np
 from sklearn.metrics import roc_auc_score
@@ -66,52 +66,6 @@ def compute_pearson(
     return float(np.corrcoef(y_pred, y_true)[0, 1])
 
 
-def _average_ranks(values: np.ndarray) -> np.ndarray:
-    """Return stable, one-based average ranks with deterministic tie handling."""
-    values = np.asarray(values)
-    order = np.argsort(values, kind="stable")
-    sorted_values = values[order]
-    sorted_ranks = np.empty(len(values), dtype=float)
-    start = 0
-    while start < len(values):
-        stop = start + 1
-        while stop < len(values) and sorted_values[stop] == sorted_values[start]:
-            stop += 1
-        sorted_ranks[start:stop] = 0.5 * (start + stop - 1) + 1.0
-        start = stop
-    ranks = np.empty(len(values), dtype=float)
-    ranks[order] = sorted_ranks
-    return ranks
-
-
-def _mean_sample_spearman(y_true: np.ndarray, y_pred: np.ndarray) -> float:
-    """Return mean row-wise rank correlation, scoring degenerate rows at zero."""
-    correlations = []
-    for truth, prediction in zip(y_true, y_pred):
-        correlation = compute_pearson(_average_ranks(truth), _average_ranks(prediction))
-        correlations.append(correlation if np.isfinite(correlation) else 0.0)
-    return float(np.mean(correlations)) if correlations else float("nan")
-
-
-def compute_residual_sample_spearman(
-    y_true: np.ndarray,
-    y_pred: np.ndarray,
-    training_mean: np.ndarray,
-) -> float:
-    """Mean sample-wise Spearman beyond the training-fold mean response."""
-    return _mean_sample_spearman(y_true - training_mean, y_pred - training_mean)
-
-
-def compute_target_centered_sample_spearman(
-    y_true: np.ndarray,
-    y_pred: np.ndarray,
-) -> float:
-    """Mean sample-wise Spearman after removing each target-cohort gene mean."""
-    truth_centered = y_true - np.mean(y_true, axis=0)
-    prediction_centered = y_pred - np.mean(y_pred, axis=0)
-    return _mean_sample_spearman(truth_centered, prediction_centered)
-
-
 METRICS: dict[str, Callable[[np.ndarray, np.ndarray, np.ndarray | None], float]] = {
     "auroc": compute_auroc,
     "pearson": compute_pearson,
@@ -129,14 +83,13 @@ class TaskScore:
     score: float
     n_samples: int
     repeat_scores: tuple[float, ...] = ()
-    diagnostics: dict[str, float] = field(default_factory=dict)
 
 
 def category_means(scores: list[TaskScore]) -> dict[str, dict]:
     """Mean of the native metric per task category.
 
     A category uses one metric, so this is a plain mean of that metric -- never a
-    mix of native metrics. Degenerate (non-finite) task scores are dropped from
+    mix of AUROC and Pearson. Degenerate (non-finite) task scores are dropped from
     the mean. Returns ``{category: {metric, mean, n_tasks}}`` for the categories
     present in ``scores``.
     """
@@ -158,7 +111,7 @@ def sort_key(categories: dict[str, dict]) -> float:
     """Leaderboard ranking key: mean of the per-category means.
 
     Orders the rows, and is shown as the ``Mean`` column on boards holding more
-    than one category. It does average across native metrics, a
+    than one category. It does average across metrics (AUROC + Pearson), a
     deliberate compromise for a single order; swap for a per-category-normalized
     mean if the ranking needs to be metric-fair.