Spaces:
Running
Running
PRIMO Space redesign: baselines, open boards, Contribute tab, quickstart, PRIMO identity
Browse files- README.md +44 -5
- app.py +105 -114
- boards.py +71 -15
- evaluator.py +4 -3
- example_submission.csv +5 -0
- home.py +81 -18
- leaderboard.py +41 -12
- pages/about.md +4 -5
- pages/contribute.md +44 -0
- pages/submit.md +20 -1
- quickstart.py +100 -0
- results.py +111 -0
- style.css +166 -24
README.md
CHANGED
|
@@ -22,10 +22,14 @@ per-category leaderboard. The datasets are opaque (`d001`, `d002`…) — you ne
|
|
| 22 |
see the disease, tissue, or target — so you grade the *embedding*, not per-task
|
| 23 |
tuning.
|
| 24 |
|
| 25 |
-
The Space has
|
| 26 |
-
at a time), **Tasks**, a **Submit** form (sign in with Hugging Face),
|
| 27 |
-
**About**. Every board has its own URL —
|
| 28 |
-
board
|
|
|
|
|
|
|
|
|
|
|
|
|
| 29 |
|
| 30 |
🌐 Website: http://primomics.org/ ·
|
| 31 |
📄 Paper: https://openreview.net/forum?id=v2SA8gHwqo ·
|
|
@@ -35,7 +39,7 @@ board can be linked to directly.
|
|
| 35 |
|
| 36 |
PRIMO benchmarks any omics modality. Today's datasets are all **bulk RNA-seq**,
|
| 37 |
covering **immune-mediated inflammatory diseases (IMIDs)** with real clinical
|
| 38 |
-
labels from published cohorts
|
| 39 |
|
| 40 |
- **Gastroenterology** — Crohn's disease, ulcerative colitis (anti-TNF response, severity scores)
|
| 41 |
- **Dermatology** — atopic dermatitis, psoriasis (severity scores)
|
|
@@ -87,6 +91,18 @@ and an AUROC on single-cell are not the same number.
|
|
| 87 |
Partial and failed submissions still get feedback, and their scores always appear
|
| 88 |
in each board's **per-task** table even when they are not ranked.
|
| 89 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 90 |
## Run the scorer locally
|
| 91 |
|
| 92 |
```bash
|
|
@@ -95,6 +111,15 @@ export HF_TOKEN=... # read access to the PRIMO datasets
|
|
| 95 |
python evaluator.py --submission my_embeddings.parquet
|
| 96 |
```
|
| 97 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 98 |
## Space configuration
|
| 99 |
|
| 100 |
- **`hf_oauth: true`** (set above) turns on the Submit tab's *Sign in with
|
|
@@ -110,3 +135,17 @@ python evaluator.py --submission my_embeddings.parquet
|
|
| 110 |
- Submitter contact metadata (HF username, email, paper / model links, notes)
|
| 111 |
persists to a separate `submissions.csv` in the same **private** results
|
| 112 |
dataset — it never reaches the public leaderboard.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 22 |
see the disease, tissue, or target — so you grade the *embedding*, not per-task
|
| 23 |
tuning.
|
| 24 |
|
| 25 |
+
The Space has six tabs: **Home** (a grid of boards), **Leaderboard** (one board
|
| 26 |
+
at a time), **Tasks**, a **Submit** form (sign in with Hugging Face),
|
| 27 |
+
**Contribute**, and **About**. Every board has its own URL —
|
| 28 |
+
`?board=rheumatology-bulk-rna` — and every tab too, as `?tab=contribute`.
|
| 29 |
+
|
| 30 |
+
Home also shows **open boards**: greyed-out cards for the omics layers PRIMO does
|
| 31 |
+
not cover yet. They are declared in `boards.py` (`OPEN_BOARDS`) and drop out on
|
| 32 |
+
their own once the registry covers that slice.
|
| 33 |
|
| 34 |
🌐 Website: http://primomics.org/ ·
|
| 35 |
📄 Paper: https://openreview.net/forum?id=v2SA8gHwqo ·
|
|
|
|
| 39 |
|
| 40 |
PRIMO benchmarks any omics modality. Today's datasets are all **bulk RNA-seq**,
|
| 41 |
covering **immune-mediated inflammatory diseases (IMIDs)** with real clinical
|
| 42 |
+
labels from published cohorts:
|
| 43 |
|
| 44 |
- **Gastroenterology** — Crohn's disease, ulcerative colitis (anti-TNF response, severity scores)
|
| 45 |
- **Dermatology** — atopic dermatitis, psoriasis (severity scores)
|
|
|
|
| 91 |
Partial and failed submissions still get feedback, and their scores always appear
|
| 92 |
in each board's **per-task** table even when they are not ranked.
|
| 93 |
|
| 94 |
+
## Make a submission
|
| 95 |
+
|
| 96 |
+
`quickstart.py` is the shortest path: it downloads every dataset, embeds each one
|
| 97 |
+
(log2(CPM+1) → PCA) and writes the file the Submit tab wants. Swap its `embed`
|
| 98 |
+
function for your encoder and nothing else changes. `example_submission.csv`
|
| 99 |
+
shows the expected shape in four lines.
|
| 100 |
+
|
| 101 |
+
```bash
|
| 102 |
+
pip install anndata scikit-learn pandas pyyaml huggingface_hub
|
| 103 |
+
python quickstart.py --out submission.parquet
|
| 104 |
+
```
|
| 105 |
+
|
| 106 |
## Run the scorer locally
|
| 107 |
|
| 108 |
```bash
|
|
|
|
| 111 |
python evaluator.py --submission my_embeddings.parquet
|
| 112 |
```
|
| 113 |
|
| 114 |
+
## Baselines
|
| 115 |
+
|
| 116 |
+
`task_results.csv` carries an `is_baseline` flag. Reference submissions we
|
| 117 |
+
produce ourselves (a random embedding, PCA / HVG recipes over log-CPM) are
|
| 118 |
+
published with it set, rendered as `name (baseline)`, and **ranked in place** —
|
| 119 |
+
a foundation model losing to a PCA is the result worth publishing, so it is
|
| 120 |
+
never hidden at the bottom of the table. They are generated and pushed by
|
| 121 |
+
`benchmark/public_benchmark/baselines.py --score --publish`.
|
| 122 |
+
|
| 123 |
## Space configuration
|
| 124 |
|
| 125 |
- **`hf_oauth: true`** (set above) turns on the Submit tab's *Sign in with
|
|
|
|
| 135 |
- Submitter contact metadata (HF username, email, paper / model links, notes)
|
| 136 |
persists to a separate `submissions.csv` in the same **private** results
|
| 137 |
dataset — it never reaches the public leaderboard.
|
| 138 |
+
|
| 139 |
+
## Moving to another Hugging Face org
|
| 140 |
+
|
| 141 |
+
The three dataset repos are derived from one constant, `ORG` in `evaluator.py`.
|
| 142 |
+
The rest of the org name is spelled out and has to be changed by hand:
|
| 143 |
+
|
| 144 |
+
- `SPACE_REPO` in `benchmark/public_benchmark/deploy_space.py`
|
| 145 |
+
- `PUBLIC_REPO` in `quickstart.py`
|
| 146 |
+
- the links in this file and in `pages/*.md`
|
| 147 |
+
|
| 148 |
+
The theme follows the PRIMO charter: Funnel Display for headings, Funnel Sans for
|
| 149 |
+
everything else, Scienta Navy `#080F5F` / PRIMO Cyan `#16B3C0` on Paper
|
| 150 |
+
`#F3F8F8`. `colorFrom`/`colorTo` above stay `indigo`/`blue` because Hugging Face
|
| 151 |
+
only accepts eight named colours and none of them is cyan.
|
app.py
CHANGED
|
@@ -1,8 +1,9 @@
|
|
| 1 |
"""Gradio front-end for the PRIMO public benchmark.
|
| 2 |
|
| 3 |
-
|
| 4 |
-
("what is actually being tested?"), Submit ("how do I enter?"),
|
| 5 |
-
trust this?").
|
|
|
|
| 6 |
|
| 7 |
Upload one embedding file spanning every dataset (rows keyed by ``dataset_id``
|
| 8 |
+ ``sample_id``); a fixed linear probe scores each task (a dataset may carry
|
|
@@ -24,17 +25,16 @@ what to fix; an evaluator-side failure says "our side, please retry" and logs
|
|
| 24 |
the traceback rather than blaming the submission.
|
| 25 |
"""
|
| 26 |
|
| 27 |
-
import io
|
| 28 |
import os
|
| 29 |
import traceback
|
| 30 |
from datetime import datetime, timezone
|
|
|
|
| 31 |
from pathlib import Path
|
| 32 |
|
| 33 |
import gradio as gr
|
| 34 |
import pandas as pd
|
| 35 |
from boards import Board, build_boards, by_slug
|
| 36 |
from evaluator import (
|
| 37 |
-
RESULTS_REPO,
|
| 38 |
EvaluatorError,
|
| 39 |
SubmissionError,
|
| 40 |
_norm_id,
|
|
@@ -44,7 +44,7 @@ from evaluator import (
|
|
| 44 |
score_all,
|
| 45 |
scoreable_tasks,
|
| 46 |
)
|
| 47 |
-
from home import home_html
|
| 48 |
from leaderboard import (
|
| 49 |
TASK_COLUMNS,
|
| 50 |
per_task_table,
|
|
@@ -52,62 +52,71 @@ from leaderboard import (
|
|
| 52 |
source_repositories,
|
| 53 |
tasks_table,
|
| 54 |
)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 55 |
|
| 56 |
PAGES_DIR = Path(__file__).parent / "pages"
|
| 57 |
STYLE = (Path(__file__).parent / "style.css").read_text()
|
| 58 |
|
| 59 |
TOKEN = os.environ.get("HF_TOKEN")
|
| 60 |
-
|
| 61 |
-
|
| 62 |
-
|
| 63 |
-
|
| 64 |
-
|
| 65 |
-
|
| 66 |
-
|
| 67 |
-
|
| 68 |
-
|
| 69 |
-
|
| 70 |
-
"notes",
|
| 71 |
-
]
|
| 72 |
-
|
| 73 |
-
INK = "#0b0f19"
|
| 74 |
-
PANEL = "#10151f"
|
| 75 |
-
LINE = "#1f2733"
|
| 76 |
-
TEXT = "#e8ebf2"
|
| 77 |
-
MUTED = "#8b93a7"
|
| 78 |
|
| 79 |
THEME = gr.themes.Base(
|
| 80 |
-
primary_hue=gr.themes.colors.
|
| 81 |
secondary_hue=gr.themes.colors.blue,
|
| 82 |
neutral_hue=gr.themes.colors.slate,
|
| 83 |
-
font=(
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 84 |
).set(
|
| 85 |
-
|
| 86 |
-
|
| 87 |
-
|
| 88 |
-
|
| 89 |
-
|
| 90 |
-
|
| 91 |
-
|
| 92 |
-
|
| 93 |
-
|
| 94 |
-
|
| 95 |
-
|
| 96 |
-
|
| 97 |
-
|
| 98 |
-
|
| 99 |
-
|
| 100 |
-
|
| 101 |
-
|
| 102 |
-
|
| 103 |
-
|
| 104 |
-
|
| 105 |
-
|
| 106 |
-
|
| 107 |
-
|
| 108 |
-
|
| 109 |
-
|
| 110 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 111 |
)
|
| 112 |
|
| 113 |
|
|
@@ -120,53 +129,6 @@ def _page_text(name: str) -> str:
|
|
| 120 |
return (PAGES_DIR / f"{name}.md").read_text()
|
| 121 |
|
| 122 |
|
| 123 |
-
def _read_csv(filename: str, columns: list[str]) -> pd.DataFrame:
|
| 124 |
-
from huggingface_hub import hf_hub_download
|
| 125 |
-
from huggingface_hub.utils import EntryNotFoundError, RepositoryNotFoundError
|
| 126 |
-
|
| 127 |
-
try:
|
| 128 |
-
path = hf_hub_download(RESULTS_REPO, filename, repo_type="dataset", token=TOKEN)
|
| 129 |
-
except (RepositoryNotFoundError, EntryNotFoundError):
|
| 130 |
-
return pd.DataFrame(columns=columns)
|
| 131 |
-
return pd.read_csv(path)
|
| 132 |
-
|
| 133 |
-
|
| 134 |
-
def _upload_csv(filename: str, df: pd.DataFrame) -> None:
|
| 135 |
-
from huggingface_hub import HfApi
|
| 136 |
-
|
| 137 |
-
api = HfApi(token=TOKEN)
|
| 138 |
-
api.create_repo(RESULTS_REPO, repo_type="dataset", private=True, exist_ok=True)
|
| 139 |
-
buffer = io.BytesIO()
|
| 140 |
-
df.to_csv(buffer, index=False)
|
| 141 |
-
buffer.seek(0)
|
| 142 |
-
api.upload_file(
|
| 143 |
-
path_or_fileobj=buffer,
|
| 144 |
-
path_in_repo=filename,
|
| 145 |
-
repo_id=RESULTS_REPO,
|
| 146 |
-
repo_type="dataset",
|
| 147 |
-
)
|
| 148 |
-
|
| 149 |
-
|
| 150 |
-
def _read_results() -> pd.DataFrame:
|
| 151 |
-
return _read_csv(RESULTS_FILE, RESULT_COLUMNS)
|
| 152 |
-
|
| 153 |
-
|
| 154 |
-
def _append_results(rows: list[dict]) -> None:
|
| 155 |
-
if not rows:
|
| 156 |
-
return
|
| 157 |
-
df = pd.concat([_read_results(), pd.DataFrame(rows)], ignore_index=True)
|
| 158 |
-
_upload_csv(RESULTS_FILE, df)
|
| 159 |
-
|
| 160 |
-
|
| 161 |
-
def _append_submission(meta: dict) -> None:
|
| 162 |
-
"""Persist a submitter's contact metadata to the private results repo."""
|
| 163 |
-
df = pd.concat(
|
| 164 |
-
[_read_csv(SUBMISSIONS_FILE, SUBMISSION_COLUMNS), pd.DataFrame([meta])],
|
| 165 |
-
ignore_index=True,
|
| 166 |
-
)
|
| 167 |
-
_upload_csv(SUBMISSIONS_FILE, df)
|
| 168 |
-
|
| 169 |
-
|
| 170 |
def _registry_by_id() -> dict[str, dict]:
|
| 171 |
"""Scoreable tasks keyed by task_id (dataset present in the public manifest)."""
|
| 172 |
datasets = manifest_ids(fetch_manifest(TOKEN))
|
|
@@ -182,7 +144,7 @@ def _page_state() -> tuple[dict[str, dict], list[Board], pd.DataFrame]:
|
|
| 182 |
"""
|
| 183 |
try:
|
| 184 |
by_id = _registry_by_id()
|
| 185 |
-
return by_id, build_boards(by_id),
|
| 186 |
except Exception: # noqa: BLE001
|
| 187 |
traceback.print_exc()
|
| 188 |
return {}, [], pd.DataFrame(columns=RESULT_COLUMNS)
|
|
@@ -206,7 +168,7 @@ def _score_columns(df: pd.DataFrame) -> list[str]:
|
|
| 206 |
|
| 207 |
|
| 208 |
def _styled(df: pd.DataFrame, label: str, axis: int, pinned: int) -> gr.DataFrame:
|
| 209 |
-
"""
|
| 210 |
|
| 211 |
``axis=0`` bolds the best model per column (the ranked table); ``axis=1``
|
| 212 |
bolds the best model per row (the per-task table, read across). The label
|
|
@@ -235,15 +197,19 @@ def _styled(df: pd.DataFrame, label: str, axis: int, pinned: int) -> gr.DataFram
|
|
| 235 |
|
| 236 |
|
| 237 |
def _board_header(board: Board | None) -> str:
|
|
|
|
| 238 |
if board is None:
|
| 239 |
return (
|
| 240 |
'<div class="primo-board-head"><h2>No board available</h2>'
|
| 241 |
"<p>The task registry could not be loaded — please retry shortly.</p></div>"
|
| 242 |
)
|
| 243 |
return (
|
| 244 |
-
|
| 245 |
-
f
|
| 246 |
-
f"{board.
|
|
|
|
|
|
|
|
|
|
| 247 |
)
|
| 248 |
|
| 249 |
|
|
@@ -270,7 +236,7 @@ def _board_page(slug: str | None):
|
|
| 270 |
|
| 271 |
|
| 272 |
def _board_choices(boards: list[Board]) -> list[tuple[str, str]]:
|
| 273 |
-
return [(f"{b.
|
| 274 |
|
| 275 |
|
| 276 |
def _tasks_page() -> pd.DataFrame:
|
|
@@ -345,6 +311,11 @@ def evaluate(
|
|
| 345 |
return _refuse("Please enter a model name.")
|
| 346 |
if not email or not email.strip():
|
| 347 |
return _refuse("Please enter a contact email.")
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 348 |
try:
|
| 349 |
result = score_all(submission_path, TOKEN)
|
| 350 |
except SubmissionError as error:
|
|
@@ -368,6 +339,7 @@ def evaluate(
|
|
| 368 |
"task_id": task.task_id,
|
| 369 |
"score": round(float(task.score), 4),
|
| 370 |
"submitted_at": submitted_at,
|
|
|
|
| 371 |
}
|
| 372 |
for task in result["per_task"]
|
| 373 |
]
|
|
@@ -381,25 +353,38 @@ def evaluate(
|
|
| 381 |
"notes": (notes or "").strip(),
|
| 382 |
}
|
| 383 |
try:
|
| 384 |
-
|
| 385 |
-
|
| 386 |
except Exception as error: # noqa: BLE001
|
| 387 |
traceback.print_exc()
|
| 388 |
summary += f"\n\n⚠️ scored, but the leaderboard was not saved: {error}"
|
| 389 |
return (summary, *_board_page(slug))
|
| 390 |
|
| 391 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 392 |
def _init(request: gr.Request):
|
| 393 |
-
"""Render home + the board named by ``?board=``, landing
|
| 394 |
by_id, boards, df = _page_state()
|
| 395 |
-
|
| 396 |
-
board = by_slug(boards,
|
| 397 |
-
selected = "
|
| 398 |
return (
|
| 399 |
gr.Tabs(selected=selected),
|
| 400 |
gr.Dropdown(
|
| 401 |
choices=_board_choices(boards), value=board.slug if board else None
|
| 402 |
),
|
|
|
|
| 403 |
home_html(boards, df, by_id),
|
| 404 |
*_board_page(board.slug if board else None),
|
| 405 |
)
|
|
@@ -411,9 +396,13 @@ def build_demo() -> gr.Blocks:
|
|
| 411 |
) as demo:
|
| 412 |
gr.Markdown(
|
| 413 |
"# 🧬 PRIMO — Patient Representations in Multi-Omics\n\n"
|
| 414 |
-
"
|
| 415 |
-
"
|
|
|
|
|
|
|
|
|
|
| 416 |
)
|
|
|
|
| 417 |
with gr.Tabs() as tabs:
|
| 418 |
with gr.Tab("Home", id="home"):
|
| 419 |
home = gr.HTML()
|
|
@@ -465,6 +454,8 @@ def build_demo() -> gr.Blocks:
|
|
| 465 |
)
|
| 466 |
run_btn = gr.Button("Evaluate", variant="primary")
|
| 467 |
result_md = gr.Markdown()
|
|
|
|
|
|
|
| 468 |
with gr.Tab("About", id="about"):
|
| 469 |
about_md = gr.Markdown()
|
| 470 |
|
|
@@ -475,7 +466,7 @@ def build_demo() -> gr.Blocks:
|
|
| 475 |
[result_md, *board_view],
|
| 476 |
)
|
| 477 |
board_sel.change(_board_page, board_sel, board_view)
|
| 478 |
-
demo.load(_init, None, [tabs, board_sel, home, *board_view])
|
| 479 |
demo.load(_tasks_page, None, tasks_df)
|
| 480 |
demo.load(_about_page, None, about_md)
|
| 481 |
return demo
|
|
|
|
| 1 |
"""Gradio front-end for the PRIMO public benchmark.
|
| 2 |
|
| 3 |
+
Six pages: Home (a grid of boards), Leaderboard (one board at a time), Tasks
|
| 4 |
+
("what is actually being tested?"), Submit ("how do I enter?"), Contribute
|
| 5 |
+
("what is missing, and how do I add it?"), About ("can I trust this?"). A tab is
|
| 6 |
+
addressable as ``?tab=contribute``, which is what the open cards on Home link to.
|
| 7 |
|
| 8 |
Upload one embedding file spanning every dataset (rows keyed by ``dataset_id``
|
| 9 |
+ ``sample_id``); a fixed linear probe scores each task (a dataset may carry
|
|
|
|
| 25 |
the traceback rather than blaming the submission.
|
| 26 |
"""
|
| 27 |
|
|
|
|
| 28 |
import os
|
| 29 |
import traceback
|
| 30 |
from datetime import datetime, timezone
|
| 31 |
+
from html import escape
|
| 32 |
from pathlib import Path
|
| 33 |
|
| 34 |
import gradio as gr
|
| 35 |
import pandas as pd
|
| 36 |
from boards import Board, build_boards, by_slug
|
| 37 |
from evaluator import (
|
|
|
|
| 38 |
EvaluatorError,
|
| 39 |
SubmissionError,
|
| 40 |
_norm_id,
|
|
|
|
| 44 |
score_all,
|
| 45 |
scoreable_tasks,
|
| 46 |
)
|
| 47 |
+
from home import banner_html, home_html
|
| 48 |
from leaderboard import (
|
| 49 |
TASK_COLUMNS,
|
| 50 |
per_task_table,
|
|
|
|
| 52 |
source_repositories,
|
| 53 |
tasks_table,
|
| 54 |
)
|
| 55 |
+
from results import (
|
| 56 |
+
BASELINE_TAG,
|
| 57 |
+
IS_BASELINE,
|
| 58 |
+
RESULT_COLUMNS,
|
| 59 |
+
append_results,
|
| 60 |
+
append_submission,
|
| 61 |
+
read_results,
|
| 62 |
+
)
|
| 63 |
|
| 64 |
PAGES_DIR = Path(__file__).parent / "pages"
|
| 65 |
STYLE = (Path(__file__).parent / "style.css").read_text()
|
| 66 |
|
| 67 |
TOKEN = os.environ.get("HF_TOKEN")
|
| 68 |
+
TAB_IDS = ("home", "leaderboard", "tasks", "submit", "contribute", "about")
|
| 69 |
+
|
| 70 |
+
NAVY = "#080F5F"
|
| 71 |
+
DEEP_NAVY = "#050A3C"
|
| 72 |
+
CYAN = "#16B3C0"
|
| 73 |
+
PAPER = "#F3F8F8"
|
| 74 |
+
SLATE = "#54686B"
|
| 75 |
+
HAIRLINE = "#D6E2E3"
|
| 76 |
+
SURFACE = "#FFFFFF"
|
| 77 |
+
CYAN_INK = "#0F7F89"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 78 |
|
| 79 |
THEME = gr.themes.Base(
|
| 80 |
+
primary_hue=gr.themes.colors.cyan,
|
| 81 |
secondary_hue=gr.themes.colors.blue,
|
| 82 |
neutral_hue=gr.themes.colors.slate,
|
| 83 |
+
font=(
|
| 84 |
+
gr.themes.GoogleFont("Funnel Sans"),
|
| 85 |
+
"ui-sans-serif",
|
| 86 |
+
"system-ui",
|
| 87 |
+
"sans-serif",
|
| 88 |
+
),
|
| 89 |
).set(
|
| 90 |
+
color_accent=CYAN,
|
| 91 |
+
border_color_accent=CYAN,
|
| 92 |
+
body_background_fill=PAPER,
|
| 93 |
+
body_background_fill_dark=PAPER,
|
| 94 |
+
background_fill_primary=SURFACE,
|
| 95 |
+
background_fill_primary_dark=SURFACE,
|
| 96 |
+
background_fill_secondary=PAPER,
|
| 97 |
+
background_fill_secondary_dark=PAPER,
|
| 98 |
+
block_background_fill=SURFACE,
|
| 99 |
+
block_background_fill_dark=SURFACE,
|
| 100 |
+
panel_background_fill=PAPER,
|
| 101 |
+
block_label_background_fill=PAPER,
|
| 102 |
+
block_label_text_color=SLATE,
|
| 103 |
+
body_text_color=DEEP_NAVY,
|
| 104 |
+
body_text_color_dark=DEEP_NAVY,
|
| 105 |
+
body_text_color_subdued=SLATE,
|
| 106 |
+
body_text_color_subdued_dark=SLATE,
|
| 107 |
+
border_color_primary=HAIRLINE,
|
| 108 |
+
border_color_primary_dark=HAIRLINE,
|
| 109 |
+
block_border_color=HAIRLINE,
|
| 110 |
+
input_background_fill=SURFACE,
|
| 111 |
+
input_background_fill_dark=SURFACE,
|
| 112 |
+
button_primary_background_fill=NAVY,
|
| 113 |
+
button_primary_text_color=PAPER,
|
| 114 |
+
button_secondary_background_fill=SURFACE,
|
| 115 |
+
table_border_color=HAIRLINE,
|
| 116 |
+
table_text_color=DEEP_NAVY,
|
| 117 |
+
table_even_background_fill=SURFACE,
|
| 118 |
+
table_odd_background_fill=PAPER,
|
| 119 |
+
link_text_color=CYAN_INK,
|
| 120 |
)
|
| 121 |
|
| 122 |
|
|
|
|
| 129 |
return (PAGES_DIR / f"{name}.md").read_text()
|
| 130 |
|
| 131 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 132 |
def _registry_by_id() -> dict[str, dict]:
|
| 133 |
"""Scoreable tasks keyed by task_id (dataset present in the public manifest)."""
|
| 134 |
datasets = manifest_ids(fetch_manifest(TOKEN))
|
|
|
|
| 144 |
"""
|
| 145 |
try:
|
| 146 |
by_id = _registry_by_id()
|
| 147 |
+
return by_id, build_boards(by_id), read_results(TOKEN)
|
| 148 |
except Exception: # noqa: BLE001
|
| 149 |
traceback.print_exc()
|
| 150 |
return {}, [], pd.DataFrame(columns=RESULT_COLUMNS)
|
|
|
|
| 168 |
|
| 169 |
|
| 170 |
def _styled(df: pd.DataFrame, label: str, axis: int, pinned: int) -> gr.DataFrame:
|
| 171 |
+
"""An MTEB-style table: best value in bold, ids pinned, searchable.
|
| 172 |
|
| 173 |
``axis=0`` bolds the best model per column (the ranked table); ``axis=1``
|
| 174 |
bolds the best model per row (the per-task table, read across). The label
|
|
|
|
| 197 |
|
| 198 |
|
| 199 |
def _board_header(board: Board | None) -> str:
|
| 200 |
+
"""The board title strip. Registry strings are escaped: they are data, not code."""
|
| 201 |
if board is None:
|
| 202 |
return (
|
| 203 |
'<div class="primo-board-head"><h2>No board available</h2>'
|
| 204 |
"<p>The task registry could not be loaded — please retry shortly.</p></div>"
|
| 205 |
)
|
| 206 |
return (
|
| 207 |
+
'<div class="primo-board-head primo-accent"><h2>'
|
| 208 |
+
f'<span class="primo-code primo-code-modality">{escape(board.code)}</span>'
|
| 209 |
+
f"{escape(board.name)}</h2>"
|
| 210 |
+
f"<p>{escape(board.blurb)} — {board.n_tasks} tasks · "
|
| 211 |
+
f"{board.n_cohorts} cohorts · {board.n_patients:,} patients · "
|
| 212 |
+
f"{escape(board.modality)}</p></div>"
|
| 213 |
)
|
| 214 |
|
| 215 |
|
|
|
|
| 236 |
|
| 237 |
|
| 238 |
def _board_choices(boards: list[Board]) -> list[tuple[str, str]]:
|
| 239 |
+
return [(f"{b.code} · {b.name} · {b.group}", b.slug) for b in boards]
|
| 240 |
|
| 241 |
|
| 242 |
def _tasks_page() -> pd.DataFrame:
|
|
|
|
| 311 |
return _refuse("Please enter a model name.")
|
| 312 |
if not email or not email.strip():
|
| 313 |
return _refuse("Please enter a contact email.")
|
| 314 |
+
if BASELINE_TAG in model_name.lower():
|
| 315 |
+
return _refuse(
|
| 316 |
+
f"`{BASELINE_TAG}` is reserved for our reference submissions — please "
|
| 317 |
+
"pick another model name."
|
| 318 |
+
)
|
| 319 |
try:
|
| 320 |
result = score_all(submission_path, TOKEN)
|
| 321 |
except SubmissionError as error:
|
|
|
|
| 339 |
"task_id": task.task_id,
|
| 340 |
"score": round(float(task.score), 4),
|
| 341 |
"submitted_at": submitted_at,
|
| 342 |
+
IS_BASELINE: False,
|
| 343 |
}
|
| 344 |
for task in result["per_task"]
|
| 345 |
]
|
|
|
|
| 353 |
"notes": (notes or "").strip(),
|
| 354 |
}
|
| 355 |
try:
|
| 356 |
+
append_results(rows, TOKEN)
|
| 357 |
+
append_submission(meta, TOKEN)
|
| 358 |
except Exception as error: # noqa: BLE001
|
| 359 |
traceback.print_exc()
|
| 360 |
summary += f"\n\n⚠️ scored, but the leaderboard was not saved: {error}"
|
| 361 |
return (summary, *_board_page(slug))
|
| 362 |
|
| 363 |
|
| 364 |
+
def _landing_tab(params: dict, has_board: bool) -> str:
|
| 365 |
+
"""Which tab a visitor lands on: ``?tab=`` wins, then ``?board=``, else Home.
|
| 366 |
+
|
| 367 |
+
An unknown ``?tab=`` falls through to Home rather than selecting nothing,
|
| 368 |
+
which would render the Space with every panel collapsed.
|
| 369 |
+
"""
|
| 370 |
+
tab = params.get("tab")
|
| 371 |
+
if tab in TAB_IDS:
|
| 372 |
+
return tab
|
| 373 |
+
return "leaderboard" if has_board else "home"
|
| 374 |
+
|
| 375 |
+
|
| 376 |
def _init(request: gr.Request):
|
| 377 |
+
"""Render home + the board named by ``?board=``, landing where asked."""
|
| 378 |
by_id, boards, df = _page_state()
|
| 379 |
+
params = dict(request.query_params) if request else {}
|
| 380 |
+
board = by_slug(boards, params.get("board"))
|
| 381 |
+
selected = _landing_tab(params, bool(params.get("board") and board))
|
| 382 |
return (
|
| 383 |
gr.Tabs(selected=selected),
|
| 384 |
gr.Dropdown(
|
| 385 |
choices=_board_choices(boards), value=board.slug if board else None
|
| 386 |
),
|
| 387 |
+
banner_html(boards),
|
| 388 |
home_html(boards, df, by_id),
|
| 389 |
*_board_page(board.slug if board else None),
|
| 390 |
)
|
|
|
|
| 396 |
) as demo:
|
| 397 |
gr.Markdown(
|
| 398 |
"# 🧬 PRIMO — Patient Representations in Multi-Omics\n\n"
|
| 399 |
+
"**Omics foundation models are benchmarked on cells and genes. "
|
| 400 |
+
"Medicine acts on patients.** PRIMO scores one thing: does your "
|
| 401 |
+
"model's patient embedding predict a real clinical outcome — drug "
|
| 402 |
+
"response, disease severity, molecular subtype — on cohorts whose "
|
| 403 |
+
"labels you never see?"
|
| 404 |
)
|
| 405 |
+
banner = gr.HTML()
|
| 406 |
with gr.Tabs() as tabs:
|
| 407 |
with gr.Tab("Home", id="home"):
|
| 408 |
home = gr.HTML()
|
|
|
|
| 454 |
)
|
| 455 |
run_btn = gr.Button("Evaluate", variant="primary")
|
| 456 |
result_md = gr.Markdown()
|
| 457 |
+
with gr.Tab("Contribute", id="contribute"):
|
| 458 |
+
gr.Markdown(_page_text("contribute"))
|
| 459 |
with gr.Tab("About", id="about"):
|
| 460 |
about_md = gr.Markdown()
|
| 461 |
|
|
|
|
| 466 |
[result_md, *board_view],
|
| 467 |
)
|
| 468 |
board_sel.change(_board_page, board_sel, board_view)
|
| 469 |
+
demo.load(_init, None, [tabs, board_sel, banner, home, *board_view])
|
| 470 |
demo.load(_tasks_page, None, tasks_df)
|
| 471 |
demo.load(_about_page, None, about_md)
|
| 472 |
return demo
|
boards.py
CHANGED
|
@@ -12,6 +12,12 @@ because an AUROC on bulk RNA and an AUROC on single-cell are not the same
|
|
| 12 |
number. With one modality in the registry that is invisible; a second one
|
| 13 |
doubles the cards instead of silently mixing them.
|
| 14 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 15 |
Pure functions over registry dicts -- no Gradio, no network, no HTML.
|
| 16 |
"""
|
| 17 |
|
|
@@ -32,18 +38,8 @@ GROUP_NOTE = {
|
|
| 32 |
CATEGORY_GROUP: "Per-question leaderboards",
|
| 33 |
}
|
| 34 |
|
| 35 |
-
|
| 36 |
-
|
| 37 |
-
"single-cell RNA": "🔬",
|
| 38 |
-
"proteomics": "⚗️",
|
| 39 |
-
"Gastroenterology": "🩺",
|
| 40 |
-
"Dermatology": "🧴",
|
| 41 |
-
"Rheumatology": "🦴",
|
| 42 |
-
"treatment_outcome": "💊",
|
| 43 |
-
"clinical_scores": "📈",
|
| 44 |
-
"endotype": "🧩",
|
| 45 |
-
}
|
| 46 |
-
FALLBACK_ICON = "🔹"
|
| 47 |
|
| 48 |
CATEGORY_BLURB = {
|
| 49 |
"treatment_outcome": "Will this patient respond to the drug?",
|
|
@@ -64,6 +60,18 @@ def metric_label(metric: str) -> str:
|
|
| 64 |
return METRIC_LABEL.get(metric, metric)
|
| 65 |
|
| 66 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 67 |
def slugify(*parts: str) -> str:
|
| 68 |
"""URL-safe key for a board, stable enough to paste into a link."""
|
| 69 |
joined = "-".join(str(p) for p in parts)
|
|
@@ -90,7 +98,7 @@ class Board:
|
|
| 90 |
slug: str
|
| 91 |
group: str
|
| 92 |
name: str
|
| 93 |
-
|
| 94 |
blurb: str
|
| 95 |
modality: str
|
| 96 |
task_ids: frozenset[str]
|
|
@@ -152,11 +160,12 @@ def _blurb(group: str, name: str, tasks: list[dict]) -> str:
|
|
| 152 |
def _board(group: str, name: str, modality: str, tasks: list[dict]) -> Board:
|
| 153 |
"""One card. The modality board owns the bare slug; the rest are suffixed by it."""
|
| 154 |
n_cohorts, n_patients, n_diseases = _cohort_stats(tasks)
|
|
|
|
| 155 |
return Board(
|
| 156 |
slug=slugify(modality) if group == MODALITY_GROUP else slugify(name, modality),
|
| 157 |
group=group,
|
| 158 |
-
name=
|
| 159 |
-
|
| 160 |
blurb=_blurb(group, name, tasks),
|
| 161 |
modality=modality,
|
| 162 |
task_ids=frozenset(_norm_id(t["task_id"]) for t in tasks),
|
|
@@ -188,6 +197,39 @@ def build_boards(by_id: dict[str, dict]) -> list[Board]:
|
|
| 188 |
return boards
|
| 189 |
|
| 190 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 191 |
def in_group(boards: list[Board], group: str) -> list[Board]:
|
| 192 |
"""Cards of one section, biggest first -- the fullest board reads as the headline."""
|
| 193 |
return sorted(
|
|
@@ -195,6 +237,20 @@ def in_group(boards: list[Board], group: str) -> list[Board]:
|
|
| 195 |
)
|
| 196 |
|
| 197 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 198 |
def featured(boards: list[Board]) -> list[Board]:
|
| 199 |
"""The hero row: every modality board, topped up with the largest others."""
|
| 200 |
heroes = in_group(boards, MODALITY_GROUP)
|
|
|
|
| 12 |
number. With one modality in the registry that is invisible; a second one
|
| 13 |
doubles the cards instead of silently mixing them.
|
| 14 |
|
| 15 |
+
``OPEN_BOARDS`` names the slices PRIMO does NOT cover, so the home page states
|
| 16 |
+
its own gaps instead of implying the registry is the whole territory. They are a
|
| 17 |
+
hand-written constant, not a roadmap: a slice belongs here when a contributor
|
| 18 |
+
could plausibly bring it, and it disappears on its own the day the registry
|
| 19 |
+
covers it.
|
| 20 |
+
|
| 21 |
Pure functions over registry dicts -- no Gradio, no network, no HTML.
|
| 22 |
"""
|
| 23 |
|
|
|
|
| 38 |
CATEGORY_GROUP: "Per-question leaderboards",
|
| 39 |
}
|
| 40 |
|
| 41 |
+
CODE_LENGTH = 3
|
| 42 |
+
FALLBACK_CODE = "—"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 43 |
|
| 44 |
CATEGORY_BLURB = {
|
| 45 |
"treatment_outcome": "Will this patient respond to the drug?",
|
|
|
|
| 60 |
return METRIC_LABEL.get(metric, metric)
|
| 61 |
|
| 62 |
|
| 63 |
+
def short_code(name: str) -> str:
|
| 64 |
+
"""The board's letter tag, standing in for what used to be a per-board emoji.
|
| 65 |
+
|
| 66 |
+
Derived from the name rather than looked up, because a lookup table is what
|
| 67 |
+
breaks: spatial transcriptomics or metabolomics would land on a shrug the day
|
| 68 |
+
somebody adds them. Colour carries the group; these letters only carry the
|
| 69 |
+
board.
|
| 70 |
+
"""
|
| 71 |
+
letters = re.sub(r"[^a-z]", "", name.lower())
|
| 72 |
+
return letters[:CODE_LENGTH].upper() or FALLBACK_CODE
|
| 73 |
+
|
| 74 |
+
|
| 75 |
def slugify(*parts: str) -> str:
|
| 76 |
"""URL-safe key for a board, stable enough to paste into a link."""
|
| 77 |
joined = "-".join(str(p) for p in parts)
|
|
|
|
| 98 |
slug: str
|
| 99 |
group: str
|
| 100 |
name: str
|
| 101 |
+
code: str
|
| 102 |
blurb: str
|
| 103 |
modality: str
|
| 104 |
task_ids: frozenset[str]
|
|
|
|
| 160 |
def _board(group: str, name: str, modality: str, tasks: list[dict]) -> Board:
|
| 161 |
"""One card. The modality board owns the bare slug; the rest are suffixed by it."""
|
| 162 |
n_cohorts, n_patients, n_diseases = _cohort_stats(tasks)
|
| 163 |
+
display = name if group == MODALITY_GROUP else label(name)
|
| 164 |
return Board(
|
| 165 |
slug=slugify(modality) if group == MODALITY_GROUP else slugify(name, modality),
|
| 166 |
group=group,
|
| 167 |
+
name=display,
|
| 168 |
+
code=short_code(display),
|
| 169 |
blurb=_blurb(group, name, tasks),
|
| 170 |
modality=modality,
|
| 171 |
task_ids=frozenset(_norm_id(t["task_id"]) for t in tasks),
|
|
|
|
| 197 |
return boards
|
| 198 |
|
| 199 |
|
| 200 |
+
@dataclass(frozen=True)
|
| 201 |
+
class OpenBoard:
|
| 202 |
+
"""A slice nobody can be ranked on yet: a stated gap, not a leaderboard.
|
| 203 |
+
|
| 204 |
+
Deliberately not a ``Board``: it has no tasks, no cohorts and no patients, and
|
| 205 |
+
zeroing those fields would print "0 patients" on a card whose whole job is to
|
| 206 |
+
read as an invitation.
|
| 207 |
+
"""
|
| 208 |
+
|
| 209 |
+
group: str
|
| 210 |
+
name: str
|
| 211 |
+
blurb: str
|
| 212 |
+
|
| 213 |
+
|
| 214 |
+
OPEN_BOARDS: tuple[OpenBoard, ...] = (
|
| 215 |
+
OpenBoard(
|
| 216 |
+
MODALITY_GROUP,
|
| 217 |
+
"single-cell RNA",
|
| 218 |
+
"Dissociated tissue, labelled at the patient level. No cohort yet.",
|
| 219 |
+
),
|
| 220 |
+
OpenBoard(
|
| 221 |
+
MODALITY_GROUP,
|
| 222 |
+
"proteomics",
|
| 223 |
+
"Plasma or tissue proteins paired with clinical follow-up. No cohort yet.",
|
| 224 |
+
),
|
| 225 |
+
OpenBoard(
|
| 226 |
+
MODALITY_GROUP,
|
| 227 |
+
"spatial transcriptomics",
|
| 228 |
+
"Expression kept in place in the tissue, with patient outcomes. No cohort yet.",
|
| 229 |
+
),
|
| 230 |
+
)
|
| 231 |
+
|
| 232 |
+
|
| 233 |
def in_group(boards: list[Board], group: str) -> list[Board]:
|
| 234 |
"""Cards of one section, biggest first -- the fullest board reads as the headline."""
|
| 235 |
return sorted(
|
|
|
|
| 237 |
)
|
| 238 |
|
| 239 |
|
| 240 |
+
def open_in_group(boards: list[Board], group: str) -> list[OpenBoard]:
|
| 241 |
+
"""Open cards of one section, minus any slice the registry has since covered.
|
| 242 |
+
|
| 243 |
+
The day a single-cell cohort lands, its board is built from the registry and
|
| 244 |
+
the matching open card drops out with no edit here.
|
| 245 |
+
"""
|
| 246 |
+
covered = {board.name.lower() for board in in_group(boards, group)}
|
| 247 |
+
return [
|
| 248 |
+
board
|
| 249 |
+
for board in OPEN_BOARDS
|
| 250 |
+
if board.group == group and board.name.lower() not in covered
|
| 251 |
+
]
|
| 252 |
+
|
| 253 |
+
|
| 254 |
def featured(boards: list[Board]) -> list[Board]:
|
| 255 |
"""The hero row: every modality board, topped up with the largest others."""
|
| 256 |
heroes = in_group(boards, MODALITY_GROUP)
|
evaluator.py
CHANGED
|
@@ -46,9 +46,10 @@ SPLIT = "split"
|
|
| 46 |
SPLIT_TRAIN = "train"
|
| 47 |
SPLIT_TEST = "test"
|
| 48 |
|
| 49 |
-
|
| 50 |
-
|
| 51 |
-
|
|
|
|
| 52 |
MANIFEST_FILENAME = "datasets.yaml"
|
| 53 |
TASKS_FILENAME = "tasks.yaml"
|
| 54 |
LABELS_FILENAME = "labels.csv"
|
|
|
|
| 46 |
SPLIT_TRAIN = "train"
|
| 47 |
SPLIT_TEST = "test"
|
| 48 |
|
| 49 |
+
ORG = "ScientaLab"
|
| 50 |
+
PUBLIC_REPO = f"{ORG}/primo"
|
| 51 |
+
LABELS_REPO = f"{ORG}/primo-labels"
|
| 52 |
+
RESULTS_REPO = f"{ORG}/primo-results"
|
| 53 |
MANIFEST_FILENAME = "datasets.yaml"
|
| 54 |
TASKS_FILENAME = "tasks.yaml"
|
| 55 |
LABELS_FILENAME = "labels.csv"
|
example_submission.csv
ADDED
|
@@ -0,0 +1,5 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
dataset_id,sample_id,e0,e1,e2
|
| 2 |
+
d001,SAMPLE_A,0.121,-0.443,0.982
|
| 3 |
+
d001,SAMPLE_B,-0.075,0.310,0.044
|
| 4 |
+
d002,SAMPLE_C,0.512,0.028,-0.157
|
| 5 |
+
d002,SAMPLE_D,-0.301,-0.119,0.663
|
home.py
CHANGED
|
@@ -5,6 +5,11 @@ to ``gr.HTML``. Each card is a plain ``<a href="?board=...">``: clicking it
|
|
| 5 |
reloads the Space on that board, which costs a page load but buys shareable
|
| 6 |
per-board URLs and needs no JavaScript.
|
| 7 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 8 |
Every string that comes from the registry goes through ``html.escape`` -- the
|
| 9 |
titles, blurbs and disease names are authored data, not constants.
|
| 10 |
"""
|
|
@@ -18,11 +23,13 @@ from boards import (
|
|
| 18 |
GROUP_NOTE,
|
| 19 |
MODALITY_GROUP,
|
| 20 |
Board,
|
|
|
|
| 21 |
featured,
|
| 22 |
in_group,
|
| 23 |
metric_label,
|
|
|
|
| 24 |
)
|
| 25 |
-
from leaderboard import top_models
|
| 26 |
|
| 27 |
SECTIONS = (MODALITY_GROUP, AREA_GROUP, CATEGORY_GROUP)
|
| 28 |
N_TOP_MODELS = 3
|
|
@@ -31,6 +38,32 @@ GROUP_ACCENT = {
|
|
| 31 |
AREA_GROUP: "area",
|
| 32 |
CATEGORY_GROUP: "category",
|
| 33 |
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 34 |
|
| 35 |
|
| 36 |
def _stat(name: str, value: object) -> str:
|
|
@@ -51,29 +84,43 @@ def _pills(board: Board) -> str:
|
|
| 51 |
return f'<div class="primo-pills">{"".join(pills)}</div>'
|
| 52 |
|
| 53 |
|
| 54 |
-
def
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 55 |
if not rows:
|
| 56 |
return (
|
| 57 |
'<div class="primo-top primo-top-empty">No ranked model yet — '
|
| 58 |
"be the first.</div>"
|
| 59 |
)
|
| 60 |
-
|
| 61 |
-
|
| 62 |
-
|
| 63 |
-
|
| 64 |
)
|
| 65 |
return (
|
| 66 |
'<div class="primo-top"><div class="primo-top-head">'
|
| 67 |
"<span>TOP MODELS</span><span>MEAN</span></div>"
|
| 68 |
-
f"{
|
| 69 |
)
|
| 70 |
|
| 71 |
|
| 72 |
-
def _card(board: Board, top: list[
|
| 73 |
accent = GROUP_ACCENT.get(board.group, "modality")
|
| 74 |
inner = [
|
| 75 |
'<div class="primo-card-head">'
|
| 76 |
-
f'<span class="primo-
|
| 77 |
f'<span class="primo-card-title">{escape(board.name)}</span></div>'
|
| 78 |
]
|
| 79 |
if top is None:
|
|
@@ -99,30 +146,46 @@ def _card(board: Board, top: list[tuple[str, float]] | None) -> str:
|
|
| 99 |
)
|
| 100 |
|
| 101 |
|
| 102 |
-
def
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 103 |
cards = "".join(_card(b, None) for b in boards)
|
|
|
|
| 104 |
return (
|
| 105 |
'<section class="primo-section">'
|
| 106 |
'<div class="primo-section-head"><h3>'
|
| 107 |
-
f'{escape(group)} <span class="primo-count">{len(boards)}</span>
|
|
|
|
| 108 |
f'<span class="primo-section-note">{escape(GROUP_NOTE.get(group, ""))}</span>'
|
| 109 |
f'</div><div class="primo-grid">{cards}</div></section>'
|
| 110 |
)
|
| 111 |
|
| 112 |
|
| 113 |
def _sections(boards: list[Board]) -> str:
|
| 114 |
-
"""One block per facet
|
| 115 |
|
| 116 |
-
|
| 117 |
-
|
| 118 |
-
|
| 119 |
"""
|
| 120 |
out = []
|
| 121 |
for group in SECTIONS:
|
| 122 |
-
cards = in_group(boards, group)
|
| 123 |
-
if not cards
|
| 124 |
continue
|
| 125 |
-
out.append(_section(group, cards))
|
| 126 |
return "".join(out)
|
| 127 |
|
| 128 |
|
|
|
|
| 5 |
reloads the Space on that board, which costs a page load but buys shareable
|
| 6 |
per-board URLs and needs no JavaScript.
|
| 7 |
|
| 8 |
+
Two things here are not leaderboards. The version strip states what v0 actually
|
| 9 |
+
covers, counted off the registry so it cannot go stale. The greyed-out OPEN cards
|
| 10 |
+
state what it does not cover, and link to Contribute instead of to a board: the
|
| 11 |
+
page is supposed to show its own gaps, not imply the registry is the territory.
|
| 12 |
+
|
| 13 |
Every string that comes from the registry goes through ``html.escape`` -- the
|
| 14 |
titles, blurbs and disease names are authored data, not constants.
|
| 15 |
"""
|
|
|
|
| 23 |
GROUP_NOTE,
|
| 24 |
MODALITY_GROUP,
|
| 25 |
Board,
|
| 26 |
+
OpenBoard,
|
| 27 |
featured,
|
| 28 |
in_group,
|
| 29 |
metric_label,
|
| 30 |
+
open_in_group,
|
| 31 |
)
|
| 32 |
+
from leaderboard import TopModel, top_models
|
| 33 |
|
| 34 |
SECTIONS = (MODALITY_GROUP, AREA_GROUP, CATEGORY_GROUP)
|
| 35 |
N_TOP_MODELS = 3
|
|
|
|
| 38 |
AREA_GROUP: "area",
|
| 39 |
CATEGORY_GROUP: "category",
|
| 40 |
}
|
| 41 |
+
VERSION = "v0"
|
| 42 |
+
|
| 43 |
+
|
| 44 |
+
def _plural(count: int, noun: str) -> str:
|
| 45 |
+
return f"{count} {noun}" if count == 1 else f"{count} {noun}s"
|
| 46 |
+
|
| 47 |
+
|
| 48 |
+
def banner_html(boards: list[Board]) -> str:
|
| 49 |
+
"""The version strip shown above every tab: what the benchmark covers today.
|
| 50 |
+
|
| 51 |
+
Counted off the registry rather than written by hand, so it can never claim a
|
| 52 |
+
scope that has since changed. An unavailable registry yields no strip at all
|
| 53 |
+
-- a version claim nobody can check is worse than none.
|
| 54 |
+
"""
|
| 55 |
+
modalities = in_group(boards, MODALITY_GROUP)
|
| 56 |
+
if not modalities:
|
| 57 |
+
return ""
|
| 58 |
+
areas = len({board.name for board in in_group(boards, AREA_GROUP)})
|
| 59 |
+
return (
|
| 60 |
+
f'<div class="primo-banner"><span class="primo-banner-tag">{VERSION}</span>'
|
| 61 |
+
f'<span>{escape(", ".join(b.name for b in modalities))} · '
|
| 62 |
+
f'{_plural(areas, "therapeutic area")} · '
|
| 63 |
+
f'{_plural(sum(b.n_tasks for b in modalities), "task")}. '
|
| 64 |
+
"The scope is incomplete by design — the open boards on Home say where."
|
| 65 |
+
"</span></div>"
|
| 66 |
+
)
|
| 67 |
|
| 68 |
|
| 69 |
def _stat(name: str, value: object) -> str:
|
|
|
|
| 84 |
return f'<div class="primo-pills">{"".join(pills)}</div>'
|
| 85 |
|
| 86 |
|
| 87 |
+
def _top_row(row: TopModel) -> str:
|
| 88 |
+
tag = '<span class="primo-tag">baseline</span>' if row.is_baseline else ""
|
| 89 |
+
css = "primo-top-row primo-top-baseline" if row.is_baseline else "primo-top-row"
|
| 90 |
+
return (
|
| 91 |
+
f'<div class="{css}"><span>{escape(row.name)}{tag}</span>'
|
| 92 |
+
f"<span>{row.score:.3f}</span></div>"
|
| 93 |
+
)
|
| 94 |
+
|
| 95 |
+
|
| 96 |
+
def _top_models(rows: list[TopModel]) -> str:
|
| 97 |
+
"""The card's mini-ranking.
|
| 98 |
+
|
| 99 |
+
A board whose only entries are our own baselines says so: "be the first" is
|
| 100 |
+
misleading once a PCA already holds a score somebody has to beat.
|
| 101 |
+
"""
|
| 102 |
if not rows:
|
| 103 |
return (
|
| 104 |
'<div class="primo-top primo-top-empty">No ranked model yet — '
|
| 105 |
"be the first.</div>"
|
| 106 |
)
|
| 107 |
+
foot = (
|
| 108 |
+
'<div class="primo-top-foot">No submitted model yet — beat the baseline.</div>'
|
| 109 |
+
if all(row.is_baseline for row in rows)
|
| 110 |
+
else ""
|
| 111 |
)
|
| 112 |
return (
|
| 113 |
'<div class="primo-top"><div class="primo-top-head">'
|
| 114 |
"<span>TOP MODELS</span><span>MEAN</span></div>"
|
| 115 |
+
f'{"".join(_top_row(row) for row in rows)}{foot}</div>'
|
| 116 |
)
|
| 117 |
|
| 118 |
|
| 119 |
+
def _card(board: Board, top: list[TopModel] | None) -> str:
|
| 120 |
accent = GROUP_ACCENT.get(board.group, "modality")
|
| 121 |
inner = [
|
| 122 |
'<div class="primo-card-head">'
|
| 123 |
+
f'<span class="primo-code primo-code-{accent}">{escape(board.code)}</span>'
|
| 124 |
f'<span class="primo-card-title">{escape(board.name)}</span></div>'
|
| 125 |
]
|
| 126 |
if top is None:
|
|
|
|
| 146 |
)
|
| 147 |
|
| 148 |
|
| 149 |
+
def _open_card(board: OpenBoard) -> str:
|
| 150 |
+
"""A greyed-out card for a slice nobody covers: no stats, one call to action."""
|
| 151 |
+
return (
|
| 152 |
+
'<a class="primo-card primo-card-open" target="_self" href="?tab=contribute">'
|
| 153 |
+
f'<div class="primo-card-head"><span class="primo-card-title">'
|
| 154 |
+
f'{escape(board.name)}</span><span class="primo-badge-open">OPEN</span></div>'
|
| 155 |
+
f'<p class="primo-blurb">{escape(board.blurb)}</p>'
|
| 156 |
+
'<div class="primo-open-cta">Propose a cohort →</div></a>'
|
| 157 |
+
)
|
| 158 |
+
|
| 159 |
+
|
| 160 |
+
def _section(group: str, boards: list[Board], opens: list[OpenBoard]) -> str:
|
| 161 |
+
open_count = (
|
| 162 |
+
f'<span class="primo-count-open">+{len(opens)} open</span>' if opens else ""
|
| 163 |
+
)
|
| 164 |
cards = "".join(_card(b, None) for b in boards)
|
| 165 |
+
cards += "".join(_open_card(b) for b in opens)
|
| 166 |
return (
|
| 167 |
'<section class="primo-section">'
|
| 168 |
'<div class="primo-section-head"><h3>'
|
| 169 |
+
f'{escape(group)} <span class="primo-count">{len(boards)}</span>'
|
| 170 |
+
f"{open_count}</h3>"
|
| 171 |
f'<span class="primo-section-note">{escape(GROUP_NOTE.get(group, ""))}</span>'
|
| 172 |
f'</div><div class="primo-grid">{cards}</div></section>'
|
| 173 |
)
|
| 174 |
|
| 175 |
|
| 176 |
def _sections(boards: list[Board]) -> str:
|
| 177 |
+
"""One block per facet: what is covered, then what is openly missing.
|
| 178 |
|
| 179 |
+
The Modality section is shown even when a single modality exists. A lone card
|
| 180 |
+
would read as a bug; a lone card beside the omics layers nobody has brought
|
| 181 |
+
yet reads as the point of the page.
|
| 182 |
"""
|
| 183 |
out = []
|
| 184 |
for group in SECTIONS:
|
| 185 |
+
cards, opens = in_group(boards, group), open_in_group(boards, group)
|
| 186 |
+
if not cards and not opens:
|
| 187 |
continue
|
| 188 |
+
out.append(_section(group, cards, opens))
|
| 189 |
return "".join(out)
|
| 190 |
|
| 191 |
|
leaderboard.py
CHANGED
|
@@ -13,13 +13,20 @@ Two tables per board, deliberately different:
|
|
| 13 |
submissions, so a newcomer who covered three cohorts sees
|
| 14 |
their numbers instead of vanishing. Never a ranking.
|
| 15 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 16 |
Task metadata is read defensively: a registry written before diseases were
|
| 17 |
recorded yields blank cells rather than breaking the page.
|
| 18 |
"""
|
| 19 |
|
|
|
|
|
|
|
| 20 |
import pandas as pd
|
| 21 |
from boards import Board, label, metric_label
|
| 22 |
from evaluator import _norm_id
|
|
|
|
| 23 |
from scoring import TaskScore, category_means, sort_key
|
| 24 |
|
| 25 |
EMPTY_RANKED_COLUMNS = ["Model"]
|
|
@@ -27,16 +34,31 @@ EMPTY_PER_TASK_COLUMNS = ["Task"]
|
|
| 27 |
SCORE_DECIMALS = 3
|
| 28 |
|
| 29 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 30 |
def _round(value: float | None) -> float | None:
|
| 31 |
return round(value, SCORE_DECIMALS) if value is not None else None
|
| 32 |
|
| 33 |
|
| 34 |
def latest_only(df: pd.DataFrame) -> pd.DataFrame:
|
| 35 |
-
"""Keep each
|
| 36 |
-
|
| 37 |
-
|
| 38 |
-
|
| 39 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 40 |
|
| 41 |
|
| 42 |
def scores_from_rows(rows: pd.DataFrame, by_id: dict[str, dict]) -> list[TaskScore]:
|
|
@@ -72,8 +94,8 @@ def _entries(df: pd.DataFrame, by_id: dict[str, dict], board: Board) -> list[dic
|
|
| 72 |
"""Per-category means for every model that covered the whole board."""
|
| 73 |
scoped = _board_registry(by_id, board)
|
| 74 |
entries = []
|
| 75 |
-
for (model, submitted), rows in latest_only(df).groupby(
|
| 76 |
-
[
|
| 77 |
):
|
| 78 |
if not board.task_ids.issubset({_norm_id(t) for t in rows["task_id"]}):
|
| 79 |
continue
|
|
@@ -84,6 +106,7 @@ def _entries(df: pd.DataFrame, by_id: dict[str, dict], board: Board) -> list[dic
|
|
| 84 |
"submitted_at": submitted,
|
| 85 |
"categories": categories,
|
| 86 |
"rank": sort_key(categories),
|
|
|
|
| 87 |
}
|
| 88 |
)
|
| 89 |
return sorted(entries, key=lambda e: -e["rank"])
|
|
@@ -111,7 +134,8 @@ def ranked_table(
|
|
| 111 |
rows = []
|
| 112 |
for position, entry in enumerate(entries, start=1):
|
| 113 |
categories = entry["categories"]
|
| 114 |
-
|
|
|
|
| 115 |
if len(columns) > 1:
|
| 116 |
row["Mean"] = _round(entry["rank"])
|
| 117 |
for cat in columns:
|
|
@@ -138,7 +162,8 @@ def per_task_table(
|
|
| 138 |
for _, r in latest_only(df).iterrows():
|
| 139 |
task_id = _norm_id(r["task_id"])
|
| 140 |
if task_id in scoped:
|
| 141 |
-
|
|
|
|
| 142 |
if not scores:
|
| 143 |
return pd.DataFrame(columns=EMPTY_PER_TASK_COLUMNS)
|
| 144 |
|
|
@@ -165,10 +190,14 @@ def per_task_table(
|
|
| 165 |
|
| 166 |
def top_models(
|
| 167 |
df: pd.DataFrame, by_id: dict[str, dict], board: Board, limit: int
|
| 168 |
-
) -> list[
|
| 169 |
-
"""
|
| 170 |
return [
|
| 171 |
-
(
|
|
|
|
|
|
|
|
|
|
|
|
|
| 172 |
for e in _entries(df, by_id, board)[:limit]
|
| 173 |
]
|
| 174 |
|
|
|
|
| 13 |
submissions, so a newcomer who covered three cohorts sees
|
| 14 |
their numbers instead of vanishing. Never a ranking.
|
| 15 |
|
| 16 |
+
Our own baselines are entries like any other: they are labelled, and they rank
|
| 17 |
+
where their score puts them. A baseline pinned to the bottom would hide the one
|
| 18 |
+
result worth publishing -- a foundation model losing to a PCA.
|
| 19 |
+
|
| 20 |
Task metadata is read defensively: a registry written before diseases were
|
| 21 |
recorded yields blank cells rather than breaking the page.
|
| 22 |
"""
|
| 23 |
|
| 24 |
+
from dataclasses import dataclass
|
| 25 |
+
|
| 26 |
import pandas as pd
|
| 27 |
from boards import Board, label, metric_label
|
| 28 |
from evaluator import _norm_id
|
| 29 |
+
from results import IS_BASELINE, MODEL_NAME, display_name, with_baseline_flag
|
| 30 |
from scoring import TaskScore, category_means, sort_key
|
| 31 |
|
| 32 |
EMPTY_RANKED_COLUMNS = ["Model"]
|
|
|
|
| 34 |
SCORE_DECIMALS = 3
|
| 35 |
|
| 36 |
|
| 37 |
+
@dataclass(frozen=True)
|
| 38 |
+
class TopModel:
|
| 39 |
+
"""One line of a home card's mini-ranking."""
|
| 40 |
+
|
| 41 |
+
name: str
|
| 42 |
+
score: float
|
| 43 |
+
is_baseline: bool
|
| 44 |
+
|
| 45 |
+
|
| 46 |
def _round(value: float | None) -> float | None:
|
| 47 |
return round(value, SCORE_DECIMALS) if value is not None else None
|
| 48 |
|
| 49 |
|
| 50 |
def latest_only(df: pd.DataFrame) -> pd.DataFrame:
|
| 51 |
+
"""Keep each entry's most recent submission, so nobody can shop for a lucky run.
|
| 52 |
+
|
| 53 |
+
An entry is ``(name, is_baseline)``, not just the name. Sharing one namespace
|
| 54 |
+
would let somebody who submits a model called ``pca-50`` bury the published
|
| 55 |
+
``pca-50`` baseline simply by submitting after it.
|
| 56 |
+
"""
|
| 57 |
+
flagged = with_baseline_flag(df)
|
| 58 |
+
if flagged.empty:
|
| 59 |
+
return flagged
|
| 60 |
+
latest = flagged.groupby([MODEL_NAME, IS_BASELINE])["submitted_at"].transform("max")
|
| 61 |
+
return flagged[flagged["submitted_at"] == latest]
|
| 62 |
|
| 63 |
|
| 64 |
def scores_from_rows(rows: pd.DataFrame, by_id: dict[str, dict]) -> list[TaskScore]:
|
|
|
|
| 94 |
"""Per-category means for every model that covered the whole board."""
|
| 95 |
scoped = _board_registry(by_id, board)
|
| 96 |
entries = []
|
| 97 |
+
for (model, is_baseline, submitted), rows in latest_only(df).groupby(
|
| 98 |
+
[MODEL_NAME, IS_BASELINE, "submitted_at"]
|
| 99 |
):
|
| 100 |
if not board.task_ids.issubset({_norm_id(t) for t in rows["task_id"]}):
|
| 101 |
continue
|
|
|
|
| 106 |
"submitted_at": submitted,
|
| 107 |
"categories": categories,
|
| 108 |
"rank": sort_key(categories),
|
| 109 |
+
"is_baseline": bool(is_baseline),
|
| 110 |
}
|
| 111 |
)
|
| 112 |
return sorted(entries, key=lambda e: -e["rank"])
|
|
|
|
| 134 |
rows = []
|
| 135 |
for position, entry in enumerate(entries, start=1):
|
| 136 |
categories = entry["categories"]
|
| 137 |
+
model = display_name(entry["model_name"], entry["is_baseline"])
|
| 138 |
+
row = {"Rank": position, "Model": model}
|
| 139 |
if len(columns) > 1:
|
| 140 |
row["Mean"] = _round(entry["rank"])
|
| 141 |
for cat in columns:
|
|
|
|
| 162 |
for _, r in latest_only(df).iterrows():
|
| 163 |
task_id = _norm_id(r["task_id"])
|
| 164 |
if task_id in scoped:
|
| 165 |
+
model = display_name(str(r[MODEL_NAME]), bool(r[IS_BASELINE]))
|
| 166 |
+
scores[(task_id, model)] = float(r["score"])
|
| 167 |
if not scores:
|
| 168 |
return pd.DataFrame(columns=EMPTY_PER_TASK_COLUMNS)
|
| 169 |
|
|
|
|
| 190 |
|
| 191 |
def top_models(
|
| 192 |
df: pd.DataFrame, by_id: dict[str, dict], board: Board, limit: int
|
| 193 |
+
) -> list[TopModel]:
|
| 194 |
+
"""The board's best full-coverage entries -- the card teaser, baselines included."""
|
| 195 |
return [
|
| 196 |
+
TopModel(
|
| 197 |
+
name=str(e["model_name"]),
|
| 198 |
+
score=round(e["rank"], SCORE_DECIMALS),
|
| 199 |
+
is_baseline=e["is_baseline"],
|
| 200 |
+
)
|
| 201 |
for e in _entries(df, by_id, board)[:limit]
|
| 202 |
]
|
| 203 |
|
pages/about.md
CHANGED
|
@@ -11,8 +11,7 @@ Reconstruction is easy to score. Clinical usefulness is not. That is the gap
|
|
| 11 |
PRIMO measures.
|
| 12 |
|
| 13 |
Today every cohort is **bulk RNA-seq**, in **immune-mediated inflammatory
|
| 14 |
-
diseases**.
|
| 15 |
-
patient:
|
| 16 |
|
| 17 |
| Task family | The question | Metric |
|
| 18 |
|---|---|---|
|
|
@@ -64,12 +63,12 @@ Licences in force:
|
|
| 64 |
|
| 65 |
{licenses}
|
| 66 |
|
| 67 |
-
## Cite PRIMO
|
| 68 |
|
| 69 |
Cite the benchmark paper: <https://openreview.net/forum?id=v2SA8gHwqo>
|
| 70 |
|
| 71 |
-
|
| 72 |
-
|
| 73 |
|
| 74 |
## Links
|
| 75 |
|
|
|
|
| 11 |
PRIMO measures.
|
| 12 |
|
| 13 |
Today every cohort is **bulk RNA-seq**, in **immune-mediated inflammatory
|
| 14 |
+
diseases**. Three questions are asked of each patient:
|
|
|
|
| 15 |
|
| 16 |
| Task family | The question | Metric |
|
| 17 |
|---|---|---|
|
|
|
|
| 63 |
|
| 64 |
{licenses}
|
| 65 |
|
| 66 |
+
## Cite PRIMO
|
| 67 |
|
| 68 |
Cite the benchmark paper: <https://openreview.net/forum?id=v2SA8gHwqo>
|
| 69 |
|
| 70 |
+
To add a cohort, open a modality, or partner on the methodology, see the
|
| 71 |
+
**Contribute** tab.
|
| 72 |
|
| 73 |
## Links
|
| 74 |
|
pages/contribute.md
ADDED
|
@@ -0,0 +1,44 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
PRIMO is an open benchmark, and it is **deliberately incomplete** — the strip at
|
| 2 |
+
the top of this page says exactly how far it reaches today. Everything past that
|
| 3 |
+
line is something somebody else has to bring.
|
| 4 |
+
|
| 5 |
+
## Add a cohort or a task
|
| 6 |
+
|
| 7 |
+
What we need is rare and it is not a model: **an omics cohort with patient-level
|
| 8 |
+
clinical labels** — drug response, a severity score, a molecular subtype. If you
|
| 9 |
+
hold one, or know a group that does, tell us:
|
| 10 |
+
|
| 11 |
+
- **modality** and roughly how many patients
|
| 12 |
+
- **what is labelled**, and how many patients carry that label
|
| 13 |
+
- **disease and tissue**
|
| 14 |
+
- **licence**, and whether the data can be redistributed
|
| 15 |
+
|
| 16 |
+
[Propose a cohort](mailto:karim.elkanbi@scientalab.com?subject=PRIMO%20—%20cohort%20proposal)
|
| 17 |
+
|
| 18 |
+
We do not have a formal acceptance process yet. It is being written with the
|
| 19 |
+
first partners, and we would rather say that than invent rules we have not
|
| 20 |
+
agreed on. What we can promise today is a real answer and a technical
|
| 21 |
+
conversation.
|
| 22 |
+
|
| 23 |
+
## Add a modality
|
| 24 |
+
|
| 25 |
+
The **OPEN** cards on Home are the omics layers PRIMO does not reach yet. They
|
| 26 |
+
are not a roadmap — they are gaps, and each one is claimable. Opening a modality
|
| 27 |
+
takes one cohort with clinical labels, not a whole atlas.
|
| 28 |
+
|
| 29 |
+
[Claim an open board](mailto:karim.elkanbi@scientalab.com?subject=PRIMO%20—%20open%20board)
|
| 30 |
+
|
| 31 |
+
## Become a founding partner
|
| 32 |
+
|
| 33 |
+
Founding partners shape PRIMO rather than just appear on it: a voice on the
|
| 34 |
+
methodology, co-signature on what we publish, and early access to the results.
|
| 35 |
+
|
| 36 |
+
[Talk to us about partnership](mailto:karim.elkanbi@scientalab.com?subject=PRIMO%20founding%20partner)
|
| 37 |
+
· [primomics.org](http://primomics.org/)
|
| 38 |
+
|
| 39 |
+
## Smaller ways in
|
| 40 |
+
|
| 41 |
+
- **Question the method.** Open a thread in the Space's **Community** tab.
|
| 42 |
+
- **Submit a model.** Baselines are on the leaderboard already; beat them.
|
| 43 |
+
- **Read the paper** — [OpenReview](https://openreview.net/forum?id=v2SA8gHwqo)
|
| 44 |
+
- **Look at the data** — [ScientaLab/primo](https://huggingface.co/datasets/ScientaLab/primo)
|
pages/submit.md
CHANGED
|
@@ -1,4 +1,19 @@
|
|
| 1 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2 |
|
| 3 |
1. **Get the data** → download the datasets from [ScientaLab/primo](https://huggingface.co/datasets/ScientaLab/primo) (start with its `datasets.yaml`).
|
| 4 |
2. **Embed every dataset** → build **one** file: `dataset_id`, `sample_id`, then one column per embedding dim (`e0`, `e1`, …). CSV / TSV / Parquet, or NPZ.
|
|
@@ -16,3 +31,7 @@ d002,S1,0.31,0.02,-0.15
|
|
| 16 |
scored: you get ranked on every **board** whose tasks you covered in full, and
|
| 17 |
your numbers always appear in each board's **per-task** table. Nothing is
|
| 18 |
thrown away.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
**Start here, not with the docs.** Two files, both in this Space's repo:
|
| 2 |
+
|
| 3 |
+
- 📥 [`quickstart.py`](https://huggingface.co/spaces/ScientaLab/primo-eval/blob/main/quickstart.py)
|
| 4 |
+
— downloads every dataset, embeds them, writes a valid submission. Swap its
|
| 5 |
+
`embed` function for your model and you are done.
|
| 6 |
+
- 📄 [`example_submission.csv`](https://huggingface.co/spaces/ScientaLab/primo-eval/blob/main/example_submission.csv)
|
| 7 |
+
— four lines, fake numbers, the exact shape we expect.
|
| 8 |
+
|
| 9 |
+
```bash
|
| 10 |
+
pip install anndata scikit-learn pandas pyyaml huggingface_hub
|
| 11 |
+
python quickstart.py --out submission.parquet
|
| 12 |
+
```
|
| 13 |
+
|
| 14 |
+
---
|
| 15 |
+
|
| 16 |
+
Or do it by hand, in **three steps**:
|
| 17 |
|
| 18 |
1. **Get the data** → download the datasets from [ScientaLab/primo](https://huggingface.co/datasets/ScientaLab/primo) (start with its `datasets.yaml`).
|
| 19 |
2. **Embed every dataset** → build **one** file: `dataset_id`, `sample_id`, then one column per embedding dim (`e0`, `e1`, …). CSV / TSV / Parquet, or NPZ.
|
|
|
|
| 31 |
scored: you get ranked on every **board** whose tasks you covered in full, and
|
| 32 |
your numbers always appear in each board's **per-task** table. Nothing is
|
| 33 |
thrown away.
|
| 34 |
+
|
| 35 |
+
**Your first target is the baselines.** Reference submissions we run ourselves —
|
| 36 |
+
a random embedding, a PCA of log-CPM — appear on the boards labelled
|
| 37 |
+
`(baseline)`. Beating them is the bar.
|
quickstart.py
ADDED
|
@@ -0,0 +1,100 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Produce a valid PRIMO submission in one command, then swap in your own model.
|
| 2 |
+
|
| 3 |
+
Downloads every public dataset, embeds each one, and writes the single file the
|
| 4 |
+
Submit tab expects. The embedding here is deliberately dumb -- log2(CPM+1) then
|
| 5 |
+
PCA -- because the point is the plumbing, not the score: replace ``embed`` with
|
| 6 |
+
your encoder and nothing else changes.
|
| 7 |
+
|
| 8 |
+
pip install anndata scikit-learn pandas pyyaml huggingface_hub
|
| 9 |
+
python quickstart.py --out submission.parquet
|
| 10 |
+
|
| 11 |
+
Standalone on purpose: no import from this Space and none from our monorepo, so
|
| 12 |
+
it keeps working if you copy the file into your own project.
|
| 13 |
+
"""
|
| 14 |
+
|
| 15 |
+
import argparse
|
| 16 |
+
from pathlib import Path
|
| 17 |
+
|
| 18 |
+
import anndata as ad
|
| 19 |
+
import numpy as np
|
| 20 |
+
import pandas as pd
|
| 21 |
+
import yaml
|
| 22 |
+
from huggingface_hub import snapshot_download
|
| 23 |
+
from sklearn.decomposition import PCA
|
| 24 |
+
|
| 25 |
+
PUBLIC_REPO = "ScientaLab/primo"
|
| 26 |
+
MANIFEST_FILENAME = "datasets.yaml"
|
| 27 |
+
|
| 28 |
+
DATASET_ID = "dataset_id"
|
| 29 |
+
SAMPLE_ID = "sample_id"
|
| 30 |
+
|
| 31 |
+
TARGET_SUM = 1_000_000
|
| 32 |
+
N_COMPONENTS = 50
|
| 33 |
+
RANDOM_STATE = 0
|
| 34 |
+
|
| 35 |
+
|
| 36 |
+
def embed(adata: ad.AnnData) -> np.ndarray:
|
| 37 |
+
"""One dataset's raw counts -> one vector per patient. Replace me.
|
| 38 |
+
|
| 39 |
+
Whatever you return, the contract is the same: one row per sample, in
|
| 40 |
+
``adata.obs_names`` order, all finite. The embedding width is yours to pick
|
| 41 |
+
and may differ from one dataset to the next.
|
| 42 |
+
"""
|
| 43 |
+
x = adata.X
|
| 44 |
+
x = x.toarray() if hasattr(x, "toarray") else np.asarray(x)
|
| 45 |
+
x = x.astype(float)
|
| 46 |
+
counts = x.sum(axis=1, keepdims=True)
|
| 47 |
+
x = np.log2(x / np.where(counts == 0, 1, counts) * TARGET_SUM + 1)
|
| 48 |
+
k = min(N_COMPONENTS, x.shape[0] - 1, x.shape[1])
|
| 49 |
+
return PCA(n_components=k, random_state=RANDOM_STATE).fit_transform(x)
|
| 50 |
+
|
| 51 |
+
|
| 52 |
+
def download(token: str | None) -> Path:
|
| 53 |
+
"""Pull the public benchmark (manifest + every ``expression.h5ad``)."""
|
| 54 |
+
return Path(snapshot_download(PUBLIC_REPO, repo_type="dataset", token=token))
|
| 55 |
+
|
| 56 |
+
|
| 57 |
+
def dataset_ids(root: Path) -> list[str]:
|
| 58 |
+
"""The opaque ids to embed, read off the public manifest."""
|
| 59 |
+
manifest = yaml.safe_load((root / MANIFEST_FILENAME).read_text())
|
| 60 |
+
entries = manifest.get("datasets", []) if isinstance(manifest, dict) else manifest
|
| 61 |
+
return [str(entry["id"]) for entry in entries]
|
| 62 |
+
|
| 63 |
+
|
| 64 |
+
def build(root: Path) -> pd.DataFrame:
|
| 65 |
+
"""Embed every dataset into the one frame the Submit tab expects.
|
| 66 |
+
|
| 67 |
+
Datasets of different widths stack into one table; the extra columns of a
|
| 68 |
+
narrower dataset stay empty and the evaluator drops them per dataset, so each
|
| 69 |
+
dataset keeps its own embedding size.
|
| 70 |
+
"""
|
| 71 |
+
blocks = []
|
| 72 |
+
for dataset_id in dataset_ids(root):
|
| 73 |
+
adata = ad.read_h5ad(root / dataset_id / "expression.h5ad")
|
| 74 |
+
vectors = embed(adata)
|
| 75 |
+
print(f"{dataset_id}: {adata.n_obs} samples -> {vectors.shape[1]} dims")
|
| 76 |
+
block = pd.DataFrame(
|
| 77 |
+
vectors, columns=[f"e{i}" for i in range(vectors.shape[1])]
|
| 78 |
+
)
|
| 79 |
+
block.insert(0, SAMPLE_ID, adata.obs_names.to_numpy())
|
| 80 |
+
block.insert(0, DATASET_ID, dataset_id)
|
| 81 |
+
blocks.append(block)
|
| 82 |
+
return pd.concat(blocks, ignore_index=True)
|
| 83 |
+
|
| 84 |
+
|
| 85 |
+
def main() -> None:
|
| 86 |
+
parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
| 87 |
+
parser.add_argument("--out", type=Path, default=Path("submission.parquet"))
|
| 88 |
+
parser.add_argument("--token", default=None, help="HF token, if you need one.")
|
| 89 |
+
args = parser.parse_args()
|
| 90 |
+
|
| 91 |
+
submission = build(download(args.token))
|
| 92 |
+
if args.out.suffix == ".csv":
|
| 93 |
+
submission.to_csv(args.out, index=False)
|
| 94 |
+
else:
|
| 95 |
+
submission.to_parquet(args.out, index=False)
|
| 96 |
+
print(f"\nWrote {args.out} — {len(submission)} rows. Upload it on the Submit tab.")
|
| 97 |
+
|
| 98 |
+
|
| 99 |
+
if __name__ == "__main__":
|
| 100 |
+
main()
|
results.py
ADDED
|
@@ -0,0 +1,111 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Persisted leaderboard rows: the results-CSV schema and its Hugging Face IO.
|
| 2 |
+
|
| 3 |
+
Kept out of ``app.py`` so the schema has one owner and so the baseline publisher
|
| 4 |
+
(``benchmark/public_benchmark/baselines.py``) can append rows without importing
|
| 5 |
+
Gradio. ``huggingface_hub`` is imported lazily, so the unit tests touch no
|
| 6 |
+
network.
|
| 7 |
+
|
| 8 |
+
``is_baseline`` marks a reference submission we produced ourselves (a random
|
| 9 |
+
embedding, a PCA of log-CPM) rather than a model somebody sent us. Baselines are
|
| 10 |
+
ranked in place, never pinned: the point of showing them is that a foundation
|
| 11 |
+
model can lose to a PCA, and a row pushed to the bottom of the table would hide
|
| 12 |
+
exactly that.
|
| 13 |
+
|
| 14 |
+
Rows written before the flag existed carry no column at all, so every reader
|
| 15 |
+
goes through ``with_baseline_flag``: a missing or unparsable flag means "a
|
| 16 |
+
submitted model", the conservative reading.
|
| 17 |
+
"""
|
| 18 |
+
|
| 19 |
+
import io
|
| 20 |
+
|
| 21 |
+
import pandas as pd
|
| 22 |
+
from evaluator import RESULTS_REPO
|
| 23 |
+
|
| 24 |
+
RESULTS_FILE = "task_results.csv"
|
| 25 |
+
SUBMISSIONS_FILE = "submissions.csv"
|
| 26 |
+
|
| 27 |
+
MODEL_NAME = "model_name"
|
| 28 |
+
IS_BASELINE = "is_baseline"
|
| 29 |
+
|
| 30 |
+
RESULT_COLUMNS = [MODEL_NAME, "task_id", "score", "submitted_at", IS_BASELINE]
|
| 31 |
+
SUBMISSION_COLUMNS = [
|
| 32 |
+
MODEL_NAME,
|
| 33 |
+
"submitted_at",
|
| 34 |
+
"hf_username",
|
| 35 |
+
"email",
|
| 36 |
+
"paper_link",
|
| 37 |
+
"hf_model_link",
|
| 38 |
+
"notes",
|
| 39 |
+
]
|
| 40 |
+
|
| 41 |
+
TRUTHY = ("true", "1")
|
| 42 |
+
BASELINE_TAG = "(baseline)"
|
| 43 |
+
|
| 44 |
+
|
| 45 |
+
def with_baseline_flag(df: pd.DataFrame) -> pd.DataFrame:
|
| 46 |
+
"""Guarantee a boolean ``is_baseline`` column, whatever the CSV held.
|
| 47 |
+
|
| 48 |
+
Read back from CSV the column can be bool, the strings ``True``/``False``, or
|
| 49 |
+
absent on rows written before baselines existed; all of those must collapse
|
| 50 |
+
to a real boolean before anything ranks on it.
|
| 51 |
+
"""
|
| 52 |
+
if IS_BASELINE not in df.columns:
|
| 53 |
+
return df.assign(**{IS_BASELINE: False})
|
| 54 |
+
flags = df[IS_BASELINE].astype(str).str.strip().str.lower().isin(TRUTHY)
|
| 55 |
+
return df.assign(**{IS_BASELINE: flags})
|
| 56 |
+
|
| 57 |
+
|
| 58 |
+
def display_name(model: str, is_baseline: bool) -> str:
|
| 59 |
+
"""Leaderboard label: a baseline says so, in the one column everybody reads."""
|
| 60 |
+
return f"{model} {BASELINE_TAG}" if is_baseline else model
|
| 61 |
+
|
| 62 |
+
|
| 63 |
+
def read_csv(filename: str, columns: list[str], token: str | None) -> pd.DataFrame:
|
| 64 |
+
"""One CSV from the private results dataset; an absent file is an empty table."""
|
| 65 |
+
from huggingface_hub import hf_hub_download
|
| 66 |
+
from huggingface_hub.utils import EntryNotFoundError, RepositoryNotFoundError
|
| 67 |
+
|
| 68 |
+
try:
|
| 69 |
+
path = hf_hub_download(RESULTS_REPO, filename, repo_type="dataset", token=token)
|
| 70 |
+
except (RepositoryNotFoundError, EntryNotFoundError):
|
| 71 |
+
return pd.DataFrame(columns=columns)
|
| 72 |
+
return pd.read_csv(path)
|
| 73 |
+
|
| 74 |
+
|
| 75 |
+
def upload_csv(filename: str, df: pd.DataFrame, token: str | None) -> None:
|
| 76 |
+
"""Overwrite one CSV in the private results dataset."""
|
| 77 |
+
from huggingface_hub import HfApi
|
| 78 |
+
|
| 79 |
+
api = HfApi(token=token)
|
| 80 |
+
api.create_repo(RESULTS_REPO, repo_type="dataset", private=True, exist_ok=True)
|
| 81 |
+
buffer = io.BytesIO()
|
| 82 |
+
df.to_csv(buffer, index=False)
|
| 83 |
+
buffer.seek(0)
|
| 84 |
+
api.upload_file(
|
| 85 |
+
path_or_fileobj=buffer,
|
| 86 |
+
path_in_repo=filename,
|
| 87 |
+
repo_id=RESULTS_REPO,
|
| 88 |
+
repo_type="dataset",
|
| 89 |
+
)
|
| 90 |
+
|
| 91 |
+
|
| 92 |
+
def read_results(token: str | None) -> pd.DataFrame:
|
| 93 |
+
"""Every persisted task score, with the baseline flag normalised."""
|
| 94 |
+
return with_baseline_flag(read_csv(RESULTS_FILE, RESULT_COLUMNS, token))
|
| 95 |
+
|
| 96 |
+
|
| 97 |
+
def append_results(rows: list[dict], token: str | None) -> None:
|
| 98 |
+
"""Append scored rows to the leaderboard, keeping the history intact."""
|
| 99 |
+
if not rows:
|
| 100 |
+
return
|
| 101 |
+
df = pd.concat([read_results(token), pd.DataFrame(rows)], ignore_index=True)
|
| 102 |
+
upload_csv(RESULTS_FILE, with_baseline_flag(df), token)
|
| 103 |
+
|
| 104 |
+
|
| 105 |
+
def append_submission(meta: dict, token: str | None) -> None:
|
| 106 |
+
"""Persist a submitter's contact metadata, never shown on a public page."""
|
| 107 |
+
df = pd.concat(
|
| 108 |
+
[read_csv(SUBMISSIONS_FILE, SUBMISSION_COLUMNS, token), pd.DataFrame([meta])],
|
| 109 |
+
ignore_index=True,
|
| 110 |
+
)
|
| 111 |
+
upload_csv(SUBMISSIONS_FILE, df, token)
|
style.css
CHANGED
|
@@ -1,15 +1,64 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
:root {
|
| 2 |
-
--primo-
|
| 3 |
-
--primo-
|
| 4 |
-
--primo-
|
| 5 |
-
--primo-
|
| 6 |
-
--primo-
|
| 7 |
-
--primo-
|
| 8 |
-
--primo-
|
| 9 |
-
|
| 10 |
-
--primo-
|
| 11 |
-
--primo-
|
| 12 |
-
--primo-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 13 |
}
|
| 14 |
|
| 15 |
.primo-home {
|
|
@@ -47,8 +96,8 @@
|
|
| 47 |
font-weight: 600;
|
| 48 |
letter-spacing: 0.08em;
|
| 49 |
color: var(--primo-featured);
|
| 50 |
-
background:
|
| 51 |
-
border: 1px solid rgba(
|
| 52 |
border-radius: 999px;
|
| 53 |
padding: 4px 12px;
|
| 54 |
margin-bottom: 12px;
|
|
@@ -75,12 +124,13 @@
|
|
| 75 |
padding: 16px 18px;
|
| 76 |
text-decoration: none !important;
|
| 77 |
color: inherit;
|
| 78 |
-
transition:
|
| 79 |
}
|
| 80 |
|
| 81 |
.primo-card:hover {
|
| 82 |
background: var(--primo-card-hover);
|
| 83 |
transform: translateY(-2px);
|
|
|
|
| 84 |
}
|
| 85 |
|
| 86 |
.primo-accent-modality {
|
|
@@ -102,6 +152,50 @@
|
|
| 102 |
border-color: var(--primo-category);
|
| 103 |
}
|
| 104 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 105 |
.primo-card-head {
|
| 106 |
display: flex;
|
| 107 |
align-items: center;
|
|
@@ -109,9 +203,32 @@
|
|
| 109 |
margin-bottom: 8px;
|
| 110 |
}
|
| 111 |
|
| 112 |
-
|
| 113 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 114 |
line-height: 1;
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 115 |
}
|
| 116 |
|
| 117 |
.primo-card-title {
|
|
@@ -198,6 +315,29 @@
|
|
| 198 |
font-style: italic;
|
| 199 |
}
|
| 200 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 201 |
.primo-pills {
|
| 202 |
display: flex;
|
| 203 |
flex-wrap: wrap;
|
|
@@ -207,7 +347,7 @@
|
|
| 207 |
.primo-pill {
|
| 208 |
font-size: 0.72rem;
|
| 209 |
color: var(--primo-muted);
|
| 210 |
-
background:
|
| 211 |
border: 1px solid var(--primo-line);
|
| 212 |
border-radius: 999px;
|
| 213 |
padding: 2px 10px;
|
|
@@ -215,16 +355,18 @@
|
|
| 215 |
|
| 216 |
.primo-pill-modality {
|
| 217 |
color: var(--primo-modality);
|
| 218 |
-
border-color: rgba(
|
| 219 |
-
background:
|
| 220 |
}
|
| 221 |
|
|
|
|
|
|
|
| 222 |
.primo-section {
|
| 223 |
border: 1px solid var(--primo-line);
|
| 224 |
border-radius: 14px;
|
| 225 |
padding: 0 0 18px;
|
| 226 |
margin-bottom: 20px;
|
| 227 |
-
background:
|
| 228 |
}
|
| 229 |
|
| 230 |
.primo-section-head {
|
|
@@ -249,7 +391,7 @@
|
|
| 249 |
font-size: 0.75rem;
|
| 250 |
font-weight: 600;
|
| 251 |
color: var(--primo-muted);
|
| 252 |
-
background:
|
| 253 |
border-radius: 999px;
|
| 254 |
padding: 2px 9px;
|
| 255 |
margin-left: 6px;
|
|
@@ -291,9 +433,9 @@
|
|
| 291 |
theme. Scoped to :not(pre) so fenced blocks keep their own styling. */
|
| 292 |
.md :not(pre) > code,
|
| 293 |
.prose :not(pre) > code {
|
| 294 |
-
background: rgba(
|
| 295 |
-
color:
|
| 296 |
-
border: 1px solid rgba(
|
| 297 |
border-radius: 5px;
|
| 298 |
padding: 1px 6px;
|
| 299 |
font-size: 0.9em;
|
|
|
|
| 1 |
+
@import url("https://fonts.googleapis.com/css2?family=Funnel+Display:wght@400..800&display=swap");
|
| 2 |
+
|
| 3 |
+
/* PRIMO charter, light. Cyan is a mark colour: at 2.2:1 on Paper it is
|
| 4 |
+
unreadable as text, so anything you have to read uses --primo-cyan-ink
|
| 5 |
+
(4.5:1), and raw cyan is kept for rules, dots and borders. */
|
| 6 |
:root {
|
| 7 |
+
--primo-navy: #080f5f;
|
| 8 |
+
--primo-deep-navy: #050a3c;
|
| 9 |
+
--primo-cyan: #16b3c0;
|
| 10 |
+
--primo-cyan-ink: #0f7f89;
|
| 11 |
+
--primo-paper: #f3f8f8;
|
| 12 |
+
--primo-slate: #54686b;
|
| 13 |
+
--primo-hairline: #d6e2e3;
|
| 14 |
+
|
| 15 |
+
--primo-bg: var(--primo-paper);
|
| 16 |
+
--primo-card: #ffffff;
|
| 17 |
+
--primo-card-hover: #ffffff;
|
| 18 |
+
--primo-line: var(--primo-hairline);
|
| 19 |
+
--primo-text: var(--primo-deep-navy);
|
| 20 |
+
--primo-muted: var(--primo-slate);
|
| 21 |
+
--primo-dim: #8ca0a3;
|
| 22 |
+
--primo-modality: var(--primo-navy);
|
| 23 |
+
--primo-area: var(--primo-cyan-ink);
|
| 24 |
+
--primo-category: var(--primo-slate);
|
| 25 |
+
--primo-featured: var(--primo-navy);
|
| 26 |
+
--primo-wash: rgba(8, 15, 95, 0.04);
|
| 27 |
+
}
|
| 28 |
+
|
| 29 |
+
.primo-home h2,
|
| 30 |
+
.primo-home h3,
|
| 31 |
+
.primo-board-head h2,
|
| 32 |
+
.prose h1,
|
| 33 |
+
.prose h2,
|
| 34 |
+
.prose h3 {
|
| 35 |
+
font-family: "Funnel Display", "Funnel Sans", ui-sans-serif, system-ui, sans-serif;
|
| 36 |
+
letter-spacing: -0.015em;
|
| 37 |
+
}
|
| 38 |
+
|
| 39 |
+
.primo-banner {
|
| 40 |
+
display: flex;
|
| 41 |
+
align-items: center;
|
| 42 |
+
gap: 10px;
|
| 43 |
+
flex-wrap: wrap;
|
| 44 |
+
font-size: 0.82rem;
|
| 45 |
+
color: var(--primo-muted);
|
| 46 |
+
border: 1px solid var(--primo-line);
|
| 47 |
+
border-radius: 10px;
|
| 48 |
+
padding: 9px 14px;
|
| 49 |
+
margin-bottom: 4px;
|
| 50 |
+
}
|
| 51 |
+
|
| 52 |
+
.primo-banner-tag {
|
| 53 |
+
font-size: 0.7rem;
|
| 54 |
+
font-weight: 700;
|
| 55 |
+
letter-spacing: 0.06em;
|
| 56 |
+
text-transform: uppercase;
|
| 57 |
+
color: var(--primo-featured);
|
| 58 |
+
background: var(--primo-wash);
|
| 59 |
+
border: 1px solid rgba(8, 15, 95, 0.18);
|
| 60 |
+
border-radius: 999px;
|
| 61 |
+
padding: 2px 9px;
|
| 62 |
}
|
| 63 |
|
| 64 |
.primo-home {
|
|
|
|
| 96 |
font-weight: 600;
|
| 97 |
letter-spacing: 0.08em;
|
| 98 |
color: var(--primo-featured);
|
| 99 |
+
background: var(--primo-wash);
|
| 100 |
+
border: 1px solid rgba(8, 15, 95, 0.18);
|
| 101 |
border-radius: 999px;
|
| 102 |
padding: 4px 12px;
|
| 103 |
margin-bottom: 12px;
|
|
|
|
| 124 |
padding: 16px 18px;
|
| 125 |
text-decoration: none !important;
|
| 126 |
color: inherit;
|
| 127 |
+
transition: box-shadow 0.15s ease, border-color 0.15s ease, transform 0.15s ease;
|
| 128 |
}
|
| 129 |
|
| 130 |
.primo-card:hover {
|
| 131 |
background: var(--primo-card-hover);
|
| 132 |
transform: translateY(-2px);
|
| 133 |
+
box-shadow: 0 6px 18px rgba(8, 15, 95, 0.1);
|
| 134 |
}
|
| 135 |
|
| 136 |
.primo-accent-modality {
|
|
|
|
| 152 |
border-color: var(--primo-category);
|
| 153 |
}
|
| 154 |
|
| 155 |
+
/* ``align-self`` so an open card sizes to its own text: stretched to match a
|
| 156 |
+
full board it becomes a large empty rectangle, which reads as broken. */
|
| 157 |
+
.primo-card-open {
|
| 158 |
+
align-self: start;
|
| 159 |
+
background: transparent;
|
| 160 |
+
border-style: dashed;
|
| 161 |
+
border-left-color: var(--primo-line);
|
| 162 |
+
opacity: 0.72;
|
| 163 |
+
}
|
| 164 |
+
|
| 165 |
+
.primo-card-open:hover {
|
| 166 |
+
opacity: 1;
|
| 167 |
+
background: var(--primo-card);
|
| 168 |
+
border-color: var(--primo-dim);
|
| 169 |
+
}
|
| 170 |
+
|
| 171 |
+
.primo-card-open .primo-card-title {
|
| 172 |
+
color: var(--primo-muted);
|
| 173 |
+
}
|
| 174 |
+
|
| 175 |
+
.primo-badge-open {
|
| 176 |
+
margin-left: auto;
|
| 177 |
+
font-size: 0.6rem;
|
| 178 |
+
font-weight: 700;
|
| 179 |
+
letter-spacing: 0.08em;
|
| 180 |
+
color: var(--primo-dim);
|
| 181 |
+
border: 1px solid var(--primo-line);
|
| 182 |
+
border-radius: 999px;
|
| 183 |
+
padding: 2px 8px;
|
| 184 |
+
}
|
| 185 |
+
|
| 186 |
+
.primo-open-cta {
|
| 187 |
+
font-size: 0.8rem;
|
| 188 |
+
font-weight: 600;
|
| 189 |
+
color: var(--primo-area);
|
| 190 |
+
}
|
| 191 |
+
|
| 192 |
+
.primo-count-open {
|
| 193 |
+
font-size: 0.7rem;
|
| 194 |
+
font-weight: 600;
|
| 195 |
+
color: var(--primo-dim);
|
| 196 |
+
margin-left: 6px;
|
| 197 |
+
}
|
| 198 |
+
|
| 199 |
.primo-card-head {
|
| 200 |
display: flex;
|
| 201 |
align-items: center;
|
|
|
|
| 203 |
margin-bottom: 8px;
|
| 204 |
}
|
| 205 |
|
| 206 |
+
/* Replaces the per-board emoji: letters scale to any modality anyone adds, and
|
| 207 |
+
the colour is the group's, so a new board needs no design decision. */
|
| 208 |
+
.primo-code {
|
| 209 |
+
flex: none;
|
| 210 |
+
font-size: 0.66rem;
|
| 211 |
+
font-weight: 700;
|
| 212 |
+
letter-spacing: 0.07em;
|
| 213 |
line-height: 1;
|
| 214 |
+
color: var(--primo-card);
|
| 215 |
+
background: var(--primo-modality);
|
| 216 |
+
border-radius: 5px;
|
| 217 |
+
padding: 5px 6px;
|
| 218 |
+
}
|
| 219 |
+
|
| 220 |
+
.primo-code-area {
|
| 221 |
+
background: var(--primo-area);
|
| 222 |
+
}
|
| 223 |
+
|
| 224 |
+
.primo-code-category {
|
| 225 |
+
background: var(--primo-category);
|
| 226 |
+
}
|
| 227 |
+
|
| 228 |
+
.primo-board-head .primo-code {
|
| 229 |
+
vertical-align: middle;
|
| 230 |
+
display: inline-block;
|
| 231 |
+
margin-right: 9px;
|
| 232 |
}
|
| 233 |
|
| 234 |
.primo-card-title {
|
|
|
|
| 315 |
font-style: italic;
|
| 316 |
}
|
| 317 |
|
| 318 |
+
.primo-top-baseline span:first-child {
|
| 319 |
+
color: var(--primo-muted);
|
| 320 |
+
}
|
| 321 |
+
|
| 322 |
+
.primo-tag {
|
| 323 |
+
font-size: 0.6rem;
|
| 324 |
+
letter-spacing: 0.06em;
|
| 325 |
+
text-transform: uppercase;
|
| 326 |
+
color: var(--primo-dim);
|
| 327 |
+
border: 1px solid var(--primo-line);
|
| 328 |
+
border-radius: 999px;
|
| 329 |
+
padding: 1px 6px;
|
| 330 |
+
margin-left: 7px;
|
| 331 |
+
}
|
| 332 |
+
|
| 333 |
+
.primo-top-foot {
|
| 334 |
+
font-size: 0.75rem;
|
| 335 |
+
color: var(--primo-dim);
|
| 336 |
+
font-style: italic;
|
| 337 |
+
border-top: 1px solid var(--primo-line);
|
| 338 |
+
padding-top: 7px;
|
| 339 |
+
}
|
| 340 |
+
|
| 341 |
.primo-pills {
|
| 342 |
display: flex;
|
| 343 |
flex-wrap: wrap;
|
|
|
|
| 347 |
.primo-pill {
|
| 348 |
font-size: 0.72rem;
|
| 349 |
color: var(--primo-muted);
|
| 350 |
+
background: var(--primo-wash);
|
| 351 |
border: 1px solid var(--primo-line);
|
| 352 |
border-radius: 999px;
|
| 353 |
padding: 2px 10px;
|
|
|
|
| 355 |
|
| 356 |
.primo-pill-modality {
|
| 357 |
color: var(--primo-modality);
|
| 358 |
+
border-color: rgba(8, 15, 95, 0.22);
|
| 359 |
+
background: var(--primo-wash);
|
| 360 |
}
|
| 361 |
|
| 362 |
+
/* Transparent, not white: the cards inside are white, and a white box behind
|
| 363 |
+
them would flatten the two into one surface. */
|
| 364 |
.primo-section {
|
| 365 |
border: 1px solid var(--primo-line);
|
| 366 |
border-radius: 14px;
|
| 367 |
padding: 0 0 18px;
|
| 368 |
margin-bottom: 20px;
|
| 369 |
+
background: transparent;
|
| 370 |
}
|
| 371 |
|
| 372 |
.primo-section-head {
|
|
|
|
| 391 |
font-size: 0.75rem;
|
| 392 |
font-weight: 600;
|
| 393 |
color: var(--primo-muted);
|
| 394 |
+
background: var(--primo-wash);
|
| 395 |
border-radius: 999px;
|
| 396 |
padding: 2px 9px;
|
| 397 |
margin-left: 6px;
|
|
|
|
| 433 |
theme. Scoped to :not(pre) so fenced blocks keep their own styling. */
|
| 434 |
.md :not(pre) > code,
|
| 435 |
.prose :not(pre) > code {
|
| 436 |
+
background: rgba(22, 179, 192, 0.1) !important;
|
| 437 |
+
color: var(--primo-cyan-ink) !important;
|
| 438 |
+
border: 1px solid rgba(22, 179, 192, 0.3);
|
| 439 |
border-radius: 5px;
|
| 440 |
padding: 1px 6px;
|
| 441 |
font-size: 0.9em;
|