karimox commited on
Commit
d2058b2
·
verified ·
1 Parent(s): 96c2406

PRIMO Space redesign: baselines, open boards, Contribute tab, quickstart, PRIMO identity

Browse files
Files changed (13) hide show
  1. README.md +44 -5
  2. app.py +105 -114
  3. boards.py +71 -15
  4. evaluator.py +4 -3
  5. example_submission.csv +5 -0
  6. home.py +81 -18
  7. leaderboard.py +41 -12
  8. pages/about.md +4 -5
  9. pages/contribute.md +44 -0
  10. pages/submit.md +20 -1
  11. quickstart.py +100 -0
  12. results.py +111 -0
  13. style.css +166 -24
README.md CHANGED
@@ -22,10 +22,14 @@ per-category leaderboard. The datasets are opaque (`d001`, `d002`…) — you ne
22
  see the disease, tissue, or target — so you grade the *embedding*, not per-task
23
  tuning.
24
 
25
- The Space has five tabs: **Home** (a grid of boards), **Leaderboard** (one board
26
- at a time), **Tasks**, a **Submit** form (sign in with Hugging Face), and
27
- **About**. Every board has its own URL — `?board=rheumatology-bulk-rna` — so a
28
- board can be linked to directly.
 
 
 
 
29
 
30
  🌐 Website: http://primomics.org/ ·
31
  📄 Paper: https://openreview.net/forum?id=v2SA8gHwqo ·
@@ -35,7 +39,7 @@ board can be linked to directly.
35
 
36
  PRIMO benchmarks any omics modality. Today's datasets are all **bulk RNA-seq**,
37
  covering **immune-mediated inflammatory diseases (IMIDs)** with real clinical
38
- labels from published cohorts (more modalities are coming):
39
 
40
  - **Gastroenterology** — Crohn's disease, ulcerative colitis (anti-TNF response, severity scores)
41
  - **Dermatology** — atopic dermatitis, psoriasis (severity scores)
@@ -87,6 +91,18 @@ and an AUROC on single-cell are not the same number.
87
  Partial and failed submissions still get feedback, and their scores always appear
88
  in each board's **per-task** table even when they are not ranked.
89
 
 
 
 
 
 
 
 
 
 
 
 
 
90
  ## Run the scorer locally
91
 
92
  ```bash
@@ -95,6 +111,15 @@ export HF_TOKEN=... # read access to the PRIMO datasets
95
  python evaluator.py --submission my_embeddings.parquet
96
  ```
97
 
 
 
 
 
 
 
 
 
 
98
  ## Space configuration
99
 
100
  - **`hf_oauth: true`** (set above) turns on the Submit tab's *Sign in with
@@ -110,3 +135,17 @@ python evaluator.py --submission my_embeddings.parquet
110
  - Submitter contact metadata (HF username, email, paper / model links, notes)
111
  persists to a separate `submissions.csv` in the same **private** results
112
  dataset — it never reaches the public leaderboard.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
22
  see the disease, tissue, or target — so you grade the *embedding*, not per-task
23
  tuning.
24
 
25
+ The Space has six tabs: **Home** (a grid of boards), **Leaderboard** (one board
26
+ at a time), **Tasks**, a **Submit** form (sign in with Hugging Face),
27
+ **Contribute**, and **About**. Every board has its own URL —
28
+ `?board=rheumatology-bulk-rna` and every tab too, as `?tab=contribute`.
29
+
30
+ Home also shows **open boards**: greyed-out cards for the omics layers PRIMO does
31
+ not cover yet. They are declared in `boards.py` (`OPEN_BOARDS`) and drop out on
32
+ their own once the registry covers that slice.
33
 
34
  🌐 Website: http://primomics.org/ ·
35
  📄 Paper: https://openreview.net/forum?id=v2SA8gHwqo ·
 
39
 
40
  PRIMO benchmarks any omics modality. Today's datasets are all **bulk RNA-seq**,
41
  covering **immune-mediated inflammatory diseases (IMIDs)** with real clinical
42
+ labels from published cohorts:
43
 
44
  - **Gastroenterology** — Crohn's disease, ulcerative colitis (anti-TNF response, severity scores)
45
  - **Dermatology** — atopic dermatitis, psoriasis (severity scores)
 
91
  Partial and failed submissions still get feedback, and their scores always appear
92
  in each board's **per-task** table even when they are not ranked.
93
 
94
+ ## Make a submission
95
+
96
+ `quickstart.py` is the shortest path: it downloads every dataset, embeds each one
97
+ (log2(CPM+1) → PCA) and writes the file the Submit tab wants. Swap its `embed`
98
+ function for your encoder and nothing else changes. `example_submission.csv`
99
+ shows the expected shape in four lines.
100
+
101
+ ```bash
102
+ pip install anndata scikit-learn pandas pyyaml huggingface_hub
103
+ python quickstart.py --out submission.parquet
104
+ ```
105
+
106
  ## Run the scorer locally
107
 
108
  ```bash
 
111
  python evaluator.py --submission my_embeddings.parquet
112
  ```
113
 
114
+ ## Baselines
115
+
116
+ `task_results.csv` carries an `is_baseline` flag. Reference submissions we
117
+ produce ourselves (a random embedding, PCA / HVG recipes over log-CPM) are
118
+ published with it set, rendered as `name (baseline)`, and **ranked in place** —
119
+ a foundation model losing to a PCA is the result worth publishing, so it is
120
+ never hidden at the bottom of the table. They are generated and pushed by
121
+ `benchmark/public_benchmark/baselines.py --score --publish`.
122
+
123
  ## Space configuration
124
 
125
  - **`hf_oauth: true`** (set above) turns on the Submit tab's *Sign in with
 
135
  - Submitter contact metadata (HF username, email, paper / model links, notes)
136
  persists to a separate `submissions.csv` in the same **private** results
137
  dataset — it never reaches the public leaderboard.
138
+
139
+ ## Moving to another Hugging Face org
140
+
141
+ The three dataset repos are derived from one constant, `ORG` in `evaluator.py`.
142
+ The rest of the org name is spelled out and has to be changed by hand:
143
+
144
+ - `SPACE_REPO` in `benchmark/public_benchmark/deploy_space.py`
145
+ - `PUBLIC_REPO` in `quickstart.py`
146
+ - the links in this file and in `pages/*.md`
147
+
148
+ The theme follows the PRIMO charter: Funnel Display for headings, Funnel Sans for
149
+ everything else, Scienta Navy `#080F5F` / PRIMO Cyan `#16B3C0` on Paper
150
+ `#F3F8F8`. `colorFrom`/`colorTo` above stay `indigo`/`blue` because Hugging Face
151
+ only accepts eight named colours and none of them is cyan.
app.py CHANGED
@@ -1,8 +1,9 @@
1
  """Gradio front-end for the PRIMO public benchmark.
2
 
3
- Five pages: Home (a grid of boards), Leaderboard (one board at a time), Tasks
4
- ("what is actually being tested?"), Submit ("how do I enter?"), About ("can I
5
- trust this?").
 
6
 
7
  Upload one embedding file spanning every dataset (rows keyed by ``dataset_id``
8
  + ``sample_id``); a fixed linear probe scores each task (a dataset may carry
@@ -24,17 +25,16 @@ what to fix; an evaluator-side failure says "our side, please retry" and logs
24
  the traceback rather than blaming the submission.
25
  """
26
 
27
- import io
28
  import os
29
  import traceback
30
  from datetime import datetime, timezone
 
31
  from pathlib import Path
32
 
33
  import gradio as gr
34
  import pandas as pd
35
  from boards import Board, build_boards, by_slug
36
  from evaluator import (
37
- RESULTS_REPO,
38
  EvaluatorError,
39
  SubmissionError,
40
  _norm_id,
@@ -44,7 +44,7 @@ from evaluator import (
44
  score_all,
45
  scoreable_tasks,
46
  )
47
- from home import home_html
48
  from leaderboard import (
49
  TASK_COLUMNS,
50
  per_task_table,
@@ -52,62 +52,71 @@ from leaderboard import (
52
  source_repositories,
53
  tasks_table,
54
  )
 
 
 
 
 
 
 
 
55
 
56
  PAGES_DIR = Path(__file__).parent / "pages"
57
  STYLE = (Path(__file__).parent / "style.css").read_text()
58
 
59
  TOKEN = os.environ.get("HF_TOKEN")
60
- RESULTS_FILE = "task_results.csv"
61
- SUBMISSIONS_FILE = "submissions.csv"
62
- RESULT_COLUMNS = ["model_name", "task_id", "score", "submitted_at"]
63
- SUBMISSION_COLUMNS = [
64
- "model_name",
65
- "submitted_at",
66
- "hf_username",
67
- "email",
68
- "paper_link",
69
- "hf_model_link",
70
- "notes",
71
- ]
72
-
73
- INK = "#0b0f19"
74
- PANEL = "#10151f"
75
- LINE = "#1f2733"
76
- TEXT = "#e8ebf2"
77
- MUTED = "#8b93a7"
78
 
79
  THEME = gr.themes.Base(
80
- primary_hue=gr.themes.colors.blue,
81
  secondary_hue=gr.themes.colors.blue,
82
  neutral_hue=gr.themes.colors.slate,
83
- font=(gr.themes.GoogleFont("Inter"), "ui-sans-serif", "system-ui", "sans-serif"),
 
 
 
 
 
84
  ).set(
85
- body_background_fill=INK,
86
- body_background_fill_dark=INK,
87
- background_fill_primary=INK,
88
- background_fill_primary_dark=INK,
89
- background_fill_secondary=PANEL,
90
- background_fill_secondary_dark=PANEL,
91
- block_background_fill=INK,
92
- block_background_fill_dark=INK,
93
- panel_background_fill=PANEL,
94
- block_label_background_fill=PANEL,
95
- block_label_text_color=MUTED,
96
- body_text_color=TEXT,
97
- body_text_color_dark=TEXT,
98
- body_text_color_subdued=MUTED,
99
- body_text_color_subdued_dark=MUTED,
100
- border_color_primary=LINE,
101
- border_color_primary_dark=LINE,
102
- block_border_color=LINE,
103
- input_background_fill=PANEL,
104
- input_background_fill_dark=PANEL,
105
- button_secondary_background_fill=PANEL,
106
- table_border_color=LINE,
107
- table_text_color=TEXT,
108
- table_even_background_fill=INK,
109
- table_odd_background_fill=PANEL,
110
- link_text_color="#60a5fa",
 
 
 
 
111
  )
112
 
113
 
@@ -120,53 +129,6 @@ def _page_text(name: str) -> str:
120
  return (PAGES_DIR / f"{name}.md").read_text()
121
 
122
 
123
- def _read_csv(filename: str, columns: list[str]) -> pd.DataFrame:
124
- from huggingface_hub import hf_hub_download
125
- from huggingface_hub.utils import EntryNotFoundError, RepositoryNotFoundError
126
-
127
- try:
128
- path = hf_hub_download(RESULTS_REPO, filename, repo_type="dataset", token=TOKEN)
129
- except (RepositoryNotFoundError, EntryNotFoundError):
130
- return pd.DataFrame(columns=columns)
131
- return pd.read_csv(path)
132
-
133
-
134
- def _upload_csv(filename: str, df: pd.DataFrame) -> None:
135
- from huggingface_hub import HfApi
136
-
137
- api = HfApi(token=TOKEN)
138
- api.create_repo(RESULTS_REPO, repo_type="dataset", private=True, exist_ok=True)
139
- buffer = io.BytesIO()
140
- df.to_csv(buffer, index=False)
141
- buffer.seek(0)
142
- api.upload_file(
143
- path_or_fileobj=buffer,
144
- path_in_repo=filename,
145
- repo_id=RESULTS_REPO,
146
- repo_type="dataset",
147
- )
148
-
149
-
150
- def _read_results() -> pd.DataFrame:
151
- return _read_csv(RESULTS_FILE, RESULT_COLUMNS)
152
-
153
-
154
- def _append_results(rows: list[dict]) -> None:
155
- if not rows:
156
- return
157
- df = pd.concat([_read_results(), pd.DataFrame(rows)], ignore_index=True)
158
- _upload_csv(RESULTS_FILE, df)
159
-
160
-
161
- def _append_submission(meta: dict) -> None:
162
- """Persist a submitter's contact metadata to the private results repo."""
163
- df = pd.concat(
164
- [_read_csv(SUBMISSIONS_FILE, SUBMISSION_COLUMNS), pd.DataFrame([meta])],
165
- ignore_index=True,
166
- )
167
- _upload_csv(SUBMISSIONS_FILE, df)
168
-
169
-
170
  def _registry_by_id() -> dict[str, dict]:
171
  """Scoreable tasks keyed by task_id (dataset present in the public manifest)."""
172
  datasets = manifest_ids(fetch_manifest(TOKEN))
@@ -182,7 +144,7 @@ def _page_state() -> tuple[dict[str, dict], list[Board], pd.DataFrame]:
182
  """
183
  try:
184
  by_id = _registry_by_id()
185
- return by_id, build_boards(by_id), _read_results()
186
  except Exception: # noqa: BLE001
187
  traceback.print_exc()
188
  return {}, [], pd.DataFrame(columns=RESULT_COLUMNS)
@@ -206,7 +168,7 @@ def _score_columns(df: pd.DataFrame) -> list[str]:
206
 
207
 
208
  def _styled(df: pd.DataFrame, label: str, axis: int, pinned: int) -> gr.DataFrame:
209
- """A dark, MTEB-style table: best value in bold, ids pinned, searchable.
210
 
211
  ``axis=0`` bolds the best model per column (the ranked table); ``axis=1``
212
  bolds the best model per row (the per-task table, read across). The label
@@ -235,15 +197,19 @@ def _styled(df: pd.DataFrame, label: str, axis: int, pinned: int) -> gr.DataFram
235
 
236
 
237
  def _board_header(board: Board | None) -> str:
 
238
  if board is None:
239
  return (
240
  '<div class="primo-board-head"><h2>No board available</h2>'
241
  "<p>The task registry could not be loaded — please retry shortly.</p></div>"
242
  )
243
  return (
244
- f'<div class="primo-board-head primo-accent"><h2>{board.icon} {board.name}</h2>'
245
- f"<p>{board.blurb} {board.n_tasks} tasks · {board.n_cohorts} cohorts · "
246
- f"{board.n_patients:,} patients · {board.modality}</p></div>"
 
 
 
247
  )
248
 
249
 
@@ -270,7 +236,7 @@ def _board_page(slug: str | None):
270
 
271
 
272
  def _board_choices(boards: list[Board]) -> list[tuple[str, str]]:
273
- return [(f"{b.icon} {b.name} · {b.group}", b.slug) for b in boards]
274
 
275
 
276
  def _tasks_page() -> pd.DataFrame:
@@ -345,6 +311,11 @@ def evaluate(
345
  return _refuse("Please enter a model name.")
346
  if not email or not email.strip():
347
  return _refuse("Please enter a contact email.")
 
 
 
 
 
348
  try:
349
  result = score_all(submission_path, TOKEN)
350
  except SubmissionError as error:
@@ -368,6 +339,7 @@ def evaluate(
368
  "task_id": task.task_id,
369
  "score": round(float(task.score), 4),
370
  "submitted_at": submitted_at,
 
371
  }
372
  for task in result["per_task"]
373
  ]
@@ -381,25 +353,38 @@ def evaluate(
381
  "notes": (notes or "").strip(),
382
  }
383
  try:
384
- _append_results(rows)
385
- _append_submission(meta)
386
  except Exception as error: # noqa: BLE001
387
  traceback.print_exc()
388
  summary += f"\n\n⚠️ scored, but the leaderboard was not saved: {error}"
389
  return (summary, *_board_page(slug))
390
 
391
 
 
 
 
 
 
 
 
 
 
 
 
 
392
  def _init(request: gr.Request):
393
- """Render home + the board named by ``?board=``, landing on it when asked."""
394
  by_id, boards, df = _page_state()
395
- slug = request.query_params.get("board") if request else None
396
- board = by_slug(boards, slug)
397
- selected = "leaderboard" if slug and board else "home"
398
  return (
399
  gr.Tabs(selected=selected),
400
  gr.Dropdown(
401
  choices=_board_choices(boards), value=board.slug if board else None
402
  ),
 
403
  home_html(boards, df, by_id),
404
  *_board_page(board.slug if board else None),
405
  )
@@ -411,9 +396,13 @@ def build_demo() -> gr.Blocks:
411
  ) as demo:
412
  gr.Markdown(
413
  "# 🧬 PRIMO — Patient Representations in Multi-Omics\n\n"
414
- "A **benchmark for omics foundation models**: does your model's patient "
415
- "embedding capture real clinical signal?"
 
 
 
416
  )
 
417
  with gr.Tabs() as tabs:
418
  with gr.Tab("Home", id="home"):
419
  home = gr.HTML()
@@ -465,6 +454,8 @@ def build_demo() -> gr.Blocks:
465
  )
466
  run_btn = gr.Button("Evaluate", variant="primary")
467
  result_md = gr.Markdown()
 
 
468
  with gr.Tab("About", id="about"):
469
  about_md = gr.Markdown()
470
 
@@ -475,7 +466,7 @@ def build_demo() -> gr.Blocks:
475
  [result_md, *board_view],
476
  )
477
  board_sel.change(_board_page, board_sel, board_view)
478
- demo.load(_init, None, [tabs, board_sel, home, *board_view])
479
  demo.load(_tasks_page, None, tasks_df)
480
  demo.load(_about_page, None, about_md)
481
  return demo
 
1
  """Gradio front-end for the PRIMO public benchmark.
2
 
3
+ Six pages: Home (a grid of boards), Leaderboard (one board at a time), Tasks
4
+ ("what is actually being tested?"), Submit ("how do I enter?"), Contribute
5
+ ("what is missing, and how do I add it?"), About ("can I trust this?"). A tab is
6
+ addressable as ``?tab=contribute``, which is what the open cards on Home link to.
7
 
8
  Upload one embedding file spanning every dataset (rows keyed by ``dataset_id``
9
  + ``sample_id``); a fixed linear probe scores each task (a dataset may carry
 
25
  the traceback rather than blaming the submission.
26
  """
27
 
 
28
  import os
29
  import traceback
30
  from datetime import datetime, timezone
31
+ from html import escape
32
  from pathlib import Path
33
 
34
  import gradio as gr
35
  import pandas as pd
36
  from boards import Board, build_boards, by_slug
37
  from evaluator import (
 
38
  EvaluatorError,
39
  SubmissionError,
40
  _norm_id,
 
44
  score_all,
45
  scoreable_tasks,
46
  )
47
+ from home import banner_html, home_html
48
  from leaderboard import (
49
  TASK_COLUMNS,
50
  per_task_table,
 
52
  source_repositories,
53
  tasks_table,
54
  )
55
+ from results import (
56
+ BASELINE_TAG,
57
+ IS_BASELINE,
58
+ RESULT_COLUMNS,
59
+ append_results,
60
+ append_submission,
61
+ read_results,
62
+ )
63
 
64
  PAGES_DIR = Path(__file__).parent / "pages"
65
  STYLE = (Path(__file__).parent / "style.css").read_text()
66
 
67
  TOKEN = os.environ.get("HF_TOKEN")
68
+ TAB_IDS = ("home", "leaderboard", "tasks", "submit", "contribute", "about")
69
+
70
+ NAVY = "#080F5F"
71
+ DEEP_NAVY = "#050A3C"
72
+ CYAN = "#16B3C0"
73
+ PAPER = "#F3F8F8"
74
+ SLATE = "#54686B"
75
+ HAIRLINE = "#D6E2E3"
76
+ SURFACE = "#FFFFFF"
77
+ CYAN_INK = "#0F7F89"
 
 
 
 
 
 
 
 
78
 
79
  THEME = gr.themes.Base(
80
+ primary_hue=gr.themes.colors.cyan,
81
  secondary_hue=gr.themes.colors.blue,
82
  neutral_hue=gr.themes.colors.slate,
83
+ font=(
84
+ gr.themes.GoogleFont("Funnel Sans"),
85
+ "ui-sans-serif",
86
+ "system-ui",
87
+ "sans-serif",
88
+ ),
89
  ).set(
90
+ color_accent=CYAN,
91
+ border_color_accent=CYAN,
92
+ body_background_fill=PAPER,
93
+ body_background_fill_dark=PAPER,
94
+ background_fill_primary=SURFACE,
95
+ background_fill_primary_dark=SURFACE,
96
+ background_fill_secondary=PAPER,
97
+ background_fill_secondary_dark=PAPER,
98
+ block_background_fill=SURFACE,
99
+ block_background_fill_dark=SURFACE,
100
+ panel_background_fill=PAPER,
101
+ block_label_background_fill=PAPER,
102
+ block_label_text_color=SLATE,
103
+ body_text_color=DEEP_NAVY,
104
+ body_text_color_dark=DEEP_NAVY,
105
+ body_text_color_subdued=SLATE,
106
+ body_text_color_subdued_dark=SLATE,
107
+ border_color_primary=HAIRLINE,
108
+ border_color_primary_dark=HAIRLINE,
109
+ block_border_color=HAIRLINE,
110
+ input_background_fill=SURFACE,
111
+ input_background_fill_dark=SURFACE,
112
+ button_primary_background_fill=NAVY,
113
+ button_primary_text_color=PAPER,
114
+ button_secondary_background_fill=SURFACE,
115
+ table_border_color=HAIRLINE,
116
+ table_text_color=DEEP_NAVY,
117
+ table_even_background_fill=SURFACE,
118
+ table_odd_background_fill=PAPER,
119
+ link_text_color=CYAN_INK,
120
  )
121
 
122
 
 
129
  return (PAGES_DIR / f"{name}.md").read_text()
130
 
131
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
132
  def _registry_by_id() -> dict[str, dict]:
133
  """Scoreable tasks keyed by task_id (dataset present in the public manifest)."""
134
  datasets = manifest_ids(fetch_manifest(TOKEN))
 
144
  """
145
  try:
146
  by_id = _registry_by_id()
147
+ return by_id, build_boards(by_id), read_results(TOKEN)
148
  except Exception: # noqa: BLE001
149
  traceback.print_exc()
150
  return {}, [], pd.DataFrame(columns=RESULT_COLUMNS)
 
168
 
169
 
170
  def _styled(df: pd.DataFrame, label: str, axis: int, pinned: int) -> gr.DataFrame:
171
+ """An MTEB-style table: best value in bold, ids pinned, searchable.
172
 
173
  ``axis=0`` bolds the best model per column (the ranked table); ``axis=1``
174
  bolds the best model per row (the per-task table, read across). The label
 
197
 
198
 
199
  def _board_header(board: Board | None) -> str:
200
+ """The board title strip. Registry strings are escaped: they are data, not code."""
201
  if board is None:
202
  return (
203
  '<div class="primo-board-head"><h2>No board available</h2>'
204
  "<p>The task registry could not be loaded — please retry shortly.</p></div>"
205
  )
206
  return (
207
+ '<div class="primo-board-head primo-accent"><h2>'
208
+ f'<span class="primo-code primo-code-modality">{escape(board.code)}</span>'
209
+ f"{escape(board.name)}</h2>"
210
+ f"<p>{escape(board.blurb)} — {board.n_tasks} tasks · "
211
+ f"{board.n_cohorts} cohorts · {board.n_patients:,} patients · "
212
+ f"{escape(board.modality)}</p></div>"
213
  )
214
 
215
 
 
236
 
237
 
238
  def _board_choices(boards: list[Board]) -> list[tuple[str, str]]:
239
+ return [(f"{b.code} · {b.name} · {b.group}", b.slug) for b in boards]
240
 
241
 
242
  def _tasks_page() -> pd.DataFrame:
 
311
  return _refuse("Please enter a model name.")
312
  if not email or not email.strip():
313
  return _refuse("Please enter a contact email.")
314
+ if BASELINE_TAG in model_name.lower():
315
+ return _refuse(
316
+ f"`{BASELINE_TAG}` is reserved for our reference submissions — please "
317
+ "pick another model name."
318
+ )
319
  try:
320
  result = score_all(submission_path, TOKEN)
321
  except SubmissionError as error:
 
339
  "task_id": task.task_id,
340
  "score": round(float(task.score), 4),
341
  "submitted_at": submitted_at,
342
+ IS_BASELINE: False,
343
  }
344
  for task in result["per_task"]
345
  ]
 
353
  "notes": (notes or "").strip(),
354
  }
355
  try:
356
+ append_results(rows, TOKEN)
357
+ append_submission(meta, TOKEN)
358
  except Exception as error: # noqa: BLE001
359
  traceback.print_exc()
360
  summary += f"\n\n⚠️ scored, but the leaderboard was not saved: {error}"
361
  return (summary, *_board_page(slug))
362
 
363
 
364
+ def _landing_tab(params: dict, has_board: bool) -> str:
365
+ """Which tab a visitor lands on: ``?tab=`` wins, then ``?board=``, else Home.
366
+
367
+ An unknown ``?tab=`` falls through to Home rather than selecting nothing,
368
+ which would render the Space with every panel collapsed.
369
+ """
370
+ tab = params.get("tab")
371
+ if tab in TAB_IDS:
372
+ return tab
373
+ return "leaderboard" if has_board else "home"
374
+
375
+
376
  def _init(request: gr.Request):
377
+ """Render home + the board named by ``?board=``, landing where asked."""
378
  by_id, boards, df = _page_state()
379
+ params = dict(request.query_params) if request else {}
380
+ board = by_slug(boards, params.get("board"))
381
+ selected = _landing_tab(params, bool(params.get("board") and board))
382
  return (
383
  gr.Tabs(selected=selected),
384
  gr.Dropdown(
385
  choices=_board_choices(boards), value=board.slug if board else None
386
  ),
387
+ banner_html(boards),
388
  home_html(boards, df, by_id),
389
  *_board_page(board.slug if board else None),
390
  )
 
396
  ) as demo:
397
  gr.Markdown(
398
  "# 🧬 PRIMO — Patient Representations in Multi-Omics\n\n"
399
+ "**Omics foundation models are benchmarked on cells and genes. "
400
+ "Medicine acts on patients.** PRIMO scores one thing: does your "
401
+ "model's patient embedding predict a real clinical outcome — drug "
402
+ "response, disease severity, molecular subtype — on cohorts whose "
403
+ "labels you never see?"
404
  )
405
+ banner = gr.HTML()
406
  with gr.Tabs() as tabs:
407
  with gr.Tab("Home", id="home"):
408
  home = gr.HTML()
 
454
  )
455
  run_btn = gr.Button("Evaluate", variant="primary")
456
  result_md = gr.Markdown()
457
+ with gr.Tab("Contribute", id="contribute"):
458
+ gr.Markdown(_page_text("contribute"))
459
  with gr.Tab("About", id="about"):
460
  about_md = gr.Markdown()
461
 
 
466
  [result_md, *board_view],
467
  )
468
  board_sel.change(_board_page, board_sel, board_view)
469
+ demo.load(_init, None, [tabs, board_sel, banner, home, *board_view])
470
  demo.load(_tasks_page, None, tasks_df)
471
  demo.load(_about_page, None, about_md)
472
  return demo
boards.py CHANGED
@@ -12,6 +12,12 @@ because an AUROC on bulk RNA and an AUROC on single-cell are not the same
12
  number. With one modality in the registry that is invisible; a second one
13
  doubles the cards instead of silently mixing them.
14
 
 
 
 
 
 
 
15
  Pure functions over registry dicts -- no Gradio, no network, no HTML.
16
  """
17
 
@@ -32,18 +38,8 @@ GROUP_NOTE = {
32
  CATEGORY_GROUP: "Per-question leaderboards",
33
  }
34
 
35
- ICONS = {
36
- "bulk RNA": "🧬",
37
- "single-cell RNA": "🔬",
38
- "proteomics": "⚗️",
39
- "Gastroenterology": "🩺",
40
- "Dermatology": "🧴",
41
- "Rheumatology": "🦴",
42
- "treatment_outcome": "💊",
43
- "clinical_scores": "📈",
44
- "endotype": "🧩",
45
- }
46
- FALLBACK_ICON = "🔹"
47
 
48
  CATEGORY_BLURB = {
49
  "treatment_outcome": "Will this patient respond to the drug?",
@@ -64,6 +60,18 @@ def metric_label(metric: str) -> str:
64
  return METRIC_LABEL.get(metric, metric)
65
 
66
 
 
 
 
 
 
 
 
 
 
 
 
 
67
  def slugify(*parts: str) -> str:
68
  """URL-safe key for a board, stable enough to paste into a link."""
69
  joined = "-".join(str(p) for p in parts)
@@ -90,7 +98,7 @@ class Board:
90
  slug: str
91
  group: str
92
  name: str
93
- icon: str
94
  blurb: str
95
  modality: str
96
  task_ids: frozenset[str]
@@ -152,11 +160,12 @@ def _blurb(group: str, name: str, tasks: list[dict]) -> str:
152
  def _board(group: str, name: str, modality: str, tasks: list[dict]) -> Board:
153
  """One card. The modality board owns the bare slug; the rest are suffixed by it."""
154
  n_cohorts, n_patients, n_diseases = _cohort_stats(tasks)
 
155
  return Board(
156
  slug=slugify(modality) if group == MODALITY_GROUP else slugify(name, modality),
157
  group=group,
158
- name=name if group == MODALITY_GROUP else label(name),
159
- icon=ICONS.get(name, FALLBACK_ICON),
160
  blurb=_blurb(group, name, tasks),
161
  modality=modality,
162
  task_ids=frozenset(_norm_id(t["task_id"]) for t in tasks),
@@ -188,6 +197,39 @@ def build_boards(by_id: dict[str, dict]) -> list[Board]:
188
  return boards
189
 
190
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
191
  def in_group(boards: list[Board], group: str) -> list[Board]:
192
  """Cards of one section, biggest first -- the fullest board reads as the headline."""
193
  return sorted(
@@ -195,6 +237,20 @@ def in_group(boards: list[Board], group: str) -> list[Board]:
195
  )
196
 
197
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
198
  def featured(boards: list[Board]) -> list[Board]:
199
  """The hero row: every modality board, topped up with the largest others."""
200
  heroes = in_group(boards, MODALITY_GROUP)
 
12
  number. With one modality in the registry that is invisible; a second one
13
  doubles the cards instead of silently mixing them.
14
 
15
+ ``OPEN_BOARDS`` names the slices PRIMO does NOT cover, so the home page states
16
+ its own gaps instead of implying the registry is the whole territory. They are a
17
+ hand-written constant, not a roadmap: a slice belongs here when a contributor
18
+ could plausibly bring it, and it disappears on its own the day the registry
19
+ covers it.
20
+
21
  Pure functions over registry dicts -- no Gradio, no network, no HTML.
22
  """
23
 
 
38
  CATEGORY_GROUP: "Per-question leaderboards",
39
  }
40
 
41
+ CODE_LENGTH = 3
42
+ FALLBACK_CODE = ""
 
 
 
 
 
 
 
 
 
 
43
 
44
  CATEGORY_BLURB = {
45
  "treatment_outcome": "Will this patient respond to the drug?",
 
60
  return METRIC_LABEL.get(metric, metric)
61
 
62
 
63
+ def short_code(name: str) -> str:
64
+ """The board's letter tag, standing in for what used to be a per-board emoji.
65
+
66
+ Derived from the name rather than looked up, because a lookup table is what
67
+ breaks: spatial transcriptomics or metabolomics would land on a shrug the day
68
+ somebody adds them. Colour carries the group; these letters only carry the
69
+ board.
70
+ """
71
+ letters = re.sub(r"[^a-z]", "", name.lower())
72
+ return letters[:CODE_LENGTH].upper() or FALLBACK_CODE
73
+
74
+
75
  def slugify(*parts: str) -> str:
76
  """URL-safe key for a board, stable enough to paste into a link."""
77
  joined = "-".join(str(p) for p in parts)
 
98
  slug: str
99
  group: str
100
  name: str
101
+ code: str
102
  blurb: str
103
  modality: str
104
  task_ids: frozenset[str]
 
160
  def _board(group: str, name: str, modality: str, tasks: list[dict]) -> Board:
161
  """One card. The modality board owns the bare slug; the rest are suffixed by it."""
162
  n_cohorts, n_patients, n_diseases = _cohort_stats(tasks)
163
+ display = name if group == MODALITY_GROUP else label(name)
164
  return Board(
165
  slug=slugify(modality) if group == MODALITY_GROUP else slugify(name, modality),
166
  group=group,
167
+ name=display,
168
+ code=short_code(display),
169
  blurb=_blurb(group, name, tasks),
170
  modality=modality,
171
  task_ids=frozenset(_norm_id(t["task_id"]) for t in tasks),
 
197
  return boards
198
 
199
 
200
+ @dataclass(frozen=True)
201
+ class OpenBoard:
202
+ """A slice nobody can be ranked on yet: a stated gap, not a leaderboard.
203
+
204
+ Deliberately not a ``Board``: it has no tasks, no cohorts and no patients, and
205
+ zeroing those fields would print "0 patients" on a card whose whole job is to
206
+ read as an invitation.
207
+ """
208
+
209
+ group: str
210
+ name: str
211
+ blurb: str
212
+
213
+
214
+ OPEN_BOARDS: tuple[OpenBoard, ...] = (
215
+ OpenBoard(
216
+ MODALITY_GROUP,
217
+ "single-cell RNA",
218
+ "Dissociated tissue, labelled at the patient level. No cohort yet.",
219
+ ),
220
+ OpenBoard(
221
+ MODALITY_GROUP,
222
+ "proteomics",
223
+ "Plasma or tissue proteins paired with clinical follow-up. No cohort yet.",
224
+ ),
225
+ OpenBoard(
226
+ MODALITY_GROUP,
227
+ "spatial transcriptomics",
228
+ "Expression kept in place in the tissue, with patient outcomes. No cohort yet.",
229
+ ),
230
+ )
231
+
232
+
233
  def in_group(boards: list[Board], group: str) -> list[Board]:
234
  """Cards of one section, biggest first -- the fullest board reads as the headline."""
235
  return sorted(
 
237
  )
238
 
239
 
240
+ def open_in_group(boards: list[Board], group: str) -> list[OpenBoard]:
241
+ """Open cards of one section, minus any slice the registry has since covered.
242
+
243
+ The day a single-cell cohort lands, its board is built from the registry and
244
+ the matching open card drops out with no edit here.
245
+ """
246
+ covered = {board.name.lower() for board in in_group(boards, group)}
247
+ return [
248
+ board
249
+ for board in OPEN_BOARDS
250
+ if board.group == group and board.name.lower() not in covered
251
+ ]
252
+
253
+
254
  def featured(boards: list[Board]) -> list[Board]:
255
  """The hero row: every modality board, topped up with the largest others."""
256
  heroes = in_group(boards, MODALITY_GROUP)
evaluator.py CHANGED
@@ -46,9 +46,10 @@ SPLIT = "split"
46
  SPLIT_TRAIN = "train"
47
  SPLIT_TEST = "test"
48
 
49
- PUBLIC_REPO = "ScientaLab/primo"
50
- LABELS_REPO = "ScientaLab/primo-labels"
51
- RESULTS_REPO = "ScientaLab/primo-results"
 
52
  MANIFEST_FILENAME = "datasets.yaml"
53
  TASKS_FILENAME = "tasks.yaml"
54
  LABELS_FILENAME = "labels.csv"
 
46
  SPLIT_TRAIN = "train"
47
  SPLIT_TEST = "test"
48
 
49
+ ORG = "ScientaLab"
50
+ PUBLIC_REPO = f"{ORG}/primo"
51
+ LABELS_REPO = f"{ORG}/primo-labels"
52
+ RESULTS_REPO = f"{ORG}/primo-results"
53
  MANIFEST_FILENAME = "datasets.yaml"
54
  TASKS_FILENAME = "tasks.yaml"
55
  LABELS_FILENAME = "labels.csv"
example_submission.csv ADDED
@@ -0,0 +1,5 @@
 
 
 
 
 
 
1
+ dataset_id,sample_id,e0,e1,e2
2
+ d001,SAMPLE_A,0.121,-0.443,0.982
3
+ d001,SAMPLE_B,-0.075,0.310,0.044
4
+ d002,SAMPLE_C,0.512,0.028,-0.157
5
+ d002,SAMPLE_D,-0.301,-0.119,0.663
home.py CHANGED
@@ -5,6 +5,11 @@ to ``gr.HTML``. Each card is a plain ``<a href="?board=...">``: clicking it
5
  reloads the Space on that board, which costs a page load but buys shareable
6
  per-board URLs and needs no JavaScript.
7
 
 
 
 
 
 
8
  Every string that comes from the registry goes through ``html.escape`` -- the
9
  titles, blurbs and disease names are authored data, not constants.
10
  """
@@ -18,11 +23,13 @@ from boards import (
18
  GROUP_NOTE,
19
  MODALITY_GROUP,
20
  Board,
 
21
  featured,
22
  in_group,
23
  metric_label,
 
24
  )
25
- from leaderboard import top_models
26
 
27
  SECTIONS = (MODALITY_GROUP, AREA_GROUP, CATEGORY_GROUP)
28
  N_TOP_MODELS = 3
@@ -31,6 +38,32 @@ GROUP_ACCENT = {
31
  AREA_GROUP: "area",
32
  CATEGORY_GROUP: "category",
33
  }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
34
 
35
 
36
  def _stat(name: str, value: object) -> str:
@@ -51,29 +84,43 @@ def _pills(board: Board) -> str:
51
  return f'<div class="primo-pills">{"".join(pills)}</div>'
52
 
53
 
54
- def _top_models(rows: list[tuple[str, float]]) -> str:
 
 
 
 
 
 
 
 
 
 
 
 
 
 
55
  if not rows:
56
  return (
57
  '<div class="primo-top primo-top-empty">No ranked model yet — '
58
  "be the first.</div>"
59
  )
60
- body = "".join(
61
- f'<div class="primo-top-row"><span>{escape(model)}</span>'
62
- f"<span>{score:.3f}</span></div>"
63
- for model, score in rows
64
  )
65
  return (
66
  '<div class="primo-top"><div class="primo-top-head">'
67
  "<span>TOP MODELS</span><span>MEAN</span></div>"
68
- f"{body}</div>"
69
  )
70
 
71
 
72
- def _card(board: Board, top: list[tuple[str, float]] | None) -> str:
73
  accent = GROUP_ACCENT.get(board.group, "modality")
74
  inner = [
75
  '<div class="primo-card-head">'
76
- f'<span class="primo-icon">{escape(board.icon)}</span>'
77
  f'<span class="primo-card-title">{escape(board.name)}</span></div>'
78
  ]
79
  if top is None:
@@ -99,30 +146,46 @@ def _card(board: Board, top: list[tuple[str, float]] | None) -> str:
99
  )
100
 
101
 
102
- def _section(group: str, boards: list[Board]) -> str:
 
 
 
 
 
 
 
 
 
 
 
 
 
 
103
  cards = "".join(_card(b, None) for b in boards)
 
104
  return (
105
  '<section class="primo-section">'
106
  '<div class="primo-section-head"><h3>'
107
- f'{escape(group)} <span class="primo-count">{len(boards)}</span></h3>'
 
108
  f'<span class="primo-section-note">{escape(GROUP_NOTE.get(group, ""))}</span>'
109
  f'</div><div class="primo-grid">{cards}</div></section>'
110
  )
111
 
112
 
113
  def _sections(boards: list[Board]) -> str:
114
- """One block per facet, skipping the ones that would say nothing.
115
 
116
- A registry with a single modality gets no Modality section: that board is
117
- already the first hero card, and a lone card in a full-width box reads as a
118
- bug. The section reappears by itself the day a second modality lands.
119
  """
120
  out = []
121
  for group in SECTIONS:
122
- cards = in_group(boards, group)
123
- if not cards or (group == MODALITY_GROUP and len(cards) == 1):
124
  continue
125
- out.append(_section(group, cards))
126
  return "".join(out)
127
 
128
 
 
5
  reloads the Space on that board, which costs a page load but buys shareable
6
  per-board URLs and needs no JavaScript.
7
 
8
+ Two things here are not leaderboards. The version strip states what v0 actually
9
+ covers, counted off the registry so it cannot go stale. The greyed-out OPEN cards
10
+ state what it does not cover, and link to Contribute instead of to a board: the
11
+ page is supposed to show its own gaps, not imply the registry is the territory.
12
+
13
  Every string that comes from the registry goes through ``html.escape`` -- the
14
  titles, blurbs and disease names are authored data, not constants.
15
  """
 
23
  GROUP_NOTE,
24
  MODALITY_GROUP,
25
  Board,
26
+ OpenBoard,
27
  featured,
28
  in_group,
29
  metric_label,
30
+ open_in_group,
31
  )
32
+ from leaderboard import TopModel, top_models
33
 
34
  SECTIONS = (MODALITY_GROUP, AREA_GROUP, CATEGORY_GROUP)
35
  N_TOP_MODELS = 3
 
38
  AREA_GROUP: "area",
39
  CATEGORY_GROUP: "category",
40
  }
41
+ VERSION = "v0"
42
+
43
+
44
+ def _plural(count: int, noun: str) -> str:
45
+ return f"{count} {noun}" if count == 1 else f"{count} {noun}s"
46
+
47
+
48
+ def banner_html(boards: list[Board]) -> str:
49
+ """The version strip shown above every tab: what the benchmark covers today.
50
+
51
+ Counted off the registry rather than written by hand, so it can never claim a
52
+ scope that has since changed. An unavailable registry yields no strip at all
53
+ -- a version claim nobody can check is worse than none.
54
+ """
55
+ modalities = in_group(boards, MODALITY_GROUP)
56
+ if not modalities:
57
+ return ""
58
+ areas = len({board.name for board in in_group(boards, AREA_GROUP)})
59
+ return (
60
+ f'<div class="primo-banner"><span class="primo-banner-tag">{VERSION}</span>'
61
+ f'<span>{escape(", ".join(b.name for b in modalities))} · '
62
+ f'{_plural(areas, "therapeutic area")} · '
63
+ f'{_plural(sum(b.n_tasks for b in modalities), "task")}. '
64
+ "The scope is incomplete by design — the open boards on Home say where."
65
+ "</span></div>"
66
+ )
67
 
68
 
69
  def _stat(name: str, value: object) -> str:
 
84
  return f'<div class="primo-pills">{"".join(pills)}</div>'
85
 
86
 
87
+ def _top_row(row: TopModel) -> str:
88
+ tag = '<span class="primo-tag">baseline</span>' if row.is_baseline else ""
89
+ css = "primo-top-row primo-top-baseline" if row.is_baseline else "primo-top-row"
90
+ return (
91
+ f'<div class="{css}"><span>{escape(row.name)}{tag}</span>'
92
+ f"<span>{row.score:.3f}</span></div>"
93
+ )
94
+
95
+
96
+ def _top_models(rows: list[TopModel]) -> str:
97
+ """The card's mini-ranking.
98
+
99
+ A board whose only entries are our own baselines says so: "be the first" is
100
+ misleading once a PCA already holds a score somebody has to beat.
101
+ """
102
  if not rows:
103
  return (
104
  '<div class="primo-top primo-top-empty">No ranked model yet — '
105
  "be the first.</div>"
106
  )
107
+ foot = (
108
+ '<div class="primo-top-foot">No submitted model yet — beat the baseline.</div>'
109
+ if all(row.is_baseline for row in rows)
110
+ else ""
111
  )
112
  return (
113
  '<div class="primo-top"><div class="primo-top-head">'
114
  "<span>TOP MODELS</span><span>MEAN</span></div>"
115
+ f'{"".join(_top_row(row) for row in rows)}{foot}</div>'
116
  )
117
 
118
 
119
+ def _card(board: Board, top: list[TopModel] | None) -> str:
120
  accent = GROUP_ACCENT.get(board.group, "modality")
121
  inner = [
122
  '<div class="primo-card-head">'
123
+ f'<span class="primo-code primo-code-{accent}">{escape(board.code)}</span>'
124
  f'<span class="primo-card-title">{escape(board.name)}</span></div>'
125
  ]
126
  if top is None:
 
146
  )
147
 
148
 
149
+ def _open_card(board: OpenBoard) -> str:
150
+ """A greyed-out card for a slice nobody covers: no stats, one call to action."""
151
+ return (
152
+ '<a class="primo-card primo-card-open" target="_self" href="?tab=contribute">'
153
+ f'<div class="primo-card-head"><span class="primo-card-title">'
154
+ f'{escape(board.name)}</span><span class="primo-badge-open">OPEN</span></div>'
155
+ f'<p class="primo-blurb">{escape(board.blurb)}</p>'
156
+ '<div class="primo-open-cta">Propose a cohort →</div></a>'
157
+ )
158
+
159
+
160
+ def _section(group: str, boards: list[Board], opens: list[OpenBoard]) -> str:
161
+ open_count = (
162
+ f'<span class="primo-count-open">+{len(opens)} open</span>' if opens else ""
163
+ )
164
  cards = "".join(_card(b, None) for b in boards)
165
+ cards += "".join(_open_card(b) for b in opens)
166
  return (
167
  '<section class="primo-section">'
168
  '<div class="primo-section-head"><h3>'
169
+ f'{escape(group)} <span class="primo-count">{len(boards)}</span>'
170
+ f"{open_count}</h3>"
171
  f'<span class="primo-section-note">{escape(GROUP_NOTE.get(group, ""))}</span>'
172
  f'</div><div class="primo-grid">{cards}</div></section>'
173
  )
174
 
175
 
176
  def _sections(boards: list[Board]) -> str:
177
+ """One block per facet: what is covered, then what is openly missing.
178
 
179
+ The Modality section is shown even when a single modality exists. A lone card
180
+ would read as a bug; a lone card beside the omics layers nobody has brought
181
+ yet reads as the point of the page.
182
  """
183
  out = []
184
  for group in SECTIONS:
185
+ cards, opens = in_group(boards, group), open_in_group(boards, group)
186
+ if not cards and not opens:
187
  continue
188
+ out.append(_section(group, cards, opens))
189
  return "".join(out)
190
 
191
 
leaderboard.py CHANGED
@@ -13,13 +13,20 @@ Two tables per board, deliberately different:
13
  submissions, so a newcomer who covered three cohorts sees
14
  their numbers instead of vanishing. Never a ranking.
15
 
 
 
 
 
16
  Task metadata is read defensively: a registry written before diseases were
17
  recorded yields blank cells rather than breaking the page.
18
  """
19
 
 
 
20
  import pandas as pd
21
  from boards import Board, label, metric_label
22
  from evaluator import _norm_id
 
23
  from scoring import TaskScore, category_means, sort_key
24
 
25
  EMPTY_RANKED_COLUMNS = ["Model"]
@@ -27,16 +34,31 @@ EMPTY_PER_TASK_COLUMNS = ["Task"]
27
  SCORE_DECIMALS = 3
28
 
29
 
 
 
 
 
 
 
 
 
 
30
  def _round(value: float | None) -> float | None:
31
  return round(value, SCORE_DECIMALS) if value is not None else None
32
 
33
 
34
  def latest_only(df: pd.DataFrame) -> pd.DataFrame:
35
- """Keep each model's most recent submission, so nobody can shop for a lucky run."""
36
- if df.empty:
37
- return df
38
- latest = df.groupby("model_name")["submitted_at"].transform("max")
39
- return df[df["submitted_at"] == latest]
 
 
 
 
 
 
40
 
41
 
42
  def scores_from_rows(rows: pd.DataFrame, by_id: dict[str, dict]) -> list[TaskScore]:
@@ -72,8 +94,8 @@ def _entries(df: pd.DataFrame, by_id: dict[str, dict], board: Board) -> list[dic
72
  """Per-category means for every model that covered the whole board."""
73
  scoped = _board_registry(by_id, board)
74
  entries = []
75
- for (model, submitted), rows in latest_only(df).groupby(
76
- ["model_name", "submitted_at"]
77
  ):
78
  if not board.task_ids.issubset({_norm_id(t) for t in rows["task_id"]}):
79
  continue
@@ -84,6 +106,7 @@ def _entries(df: pd.DataFrame, by_id: dict[str, dict], board: Board) -> list[dic
84
  "submitted_at": submitted,
85
  "categories": categories,
86
  "rank": sort_key(categories),
 
87
  }
88
  )
89
  return sorted(entries, key=lambda e: -e["rank"])
@@ -111,7 +134,8 @@ def ranked_table(
111
  rows = []
112
  for position, entry in enumerate(entries, start=1):
113
  categories = entry["categories"]
114
- row = {"Rank": position, "Model": entry["model_name"]}
 
115
  if len(columns) > 1:
116
  row["Mean"] = _round(entry["rank"])
117
  for cat in columns:
@@ -138,7 +162,8 @@ def per_task_table(
138
  for _, r in latest_only(df).iterrows():
139
  task_id = _norm_id(r["task_id"])
140
  if task_id in scoped:
141
- scores[(task_id, str(r["model_name"]))] = float(r["score"])
 
142
  if not scores:
143
  return pd.DataFrame(columns=EMPTY_PER_TASK_COLUMNS)
144
 
@@ -165,10 +190,14 @@ def per_task_table(
165
 
166
  def top_models(
167
  df: pd.DataFrame, by_id: dict[str, dict], board: Board, limit: int
168
- ) -> list[tuple[str, float]]:
169
- """``(model, mean)`` for the board's best full-coverage models -- the card teaser."""
170
  return [
171
- (str(e["model_name"]), round(e["rank"], SCORE_DECIMALS))
 
 
 
 
172
  for e in _entries(df, by_id, board)[:limit]
173
  ]
174
 
 
13
  submissions, so a newcomer who covered three cohorts sees
14
  their numbers instead of vanishing. Never a ranking.
15
 
16
+ Our own baselines are entries like any other: they are labelled, and they rank
17
+ where their score puts them. A baseline pinned to the bottom would hide the one
18
+ result worth publishing -- a foundation model losing to a PCA.
19
+
20
  Task metadata is read defensively: a registry written before diseases were
21
  recorded yields blank cells rather than breaking the page.
22
  """
23
 
24
+ from dataclasses import dataclass
25
+
26
  import pandas as pd
27
  from boards import Board, label, metric_label
28
  from evaluator import _norm_id
29
+ from results import IS_BASELINE, MODEL_NAME, display_name, with_baseline_flag
30
  from scoring import TaskScore, category_means, sort_key
31
 
32
  EMPTY_RANKED_COLUMNS = ["Model"]
 
34
  SCORE_DECIMALS = 3
35
 
36
 
37
+ @dataclass(frozen=True)
38
+ class TopModel:
39
+ """One line of a home card's mini-ranking."""
40
+
41
+ name: str
42
+ score: float
43
+ is_baseline: bool
44
+
45
+
46
  def _round(value: float | None) -> float | None:
47
  return round(value, SCORE_DECIMALS) if value is not None else None
48
 
49
 
50
  def latest_only(df: pd.DataFrame) -> pd.DataFrame:
51
+ """Keep each entry's most recent submission, so nobody can shop for a lucky run.
52
+
53
+ An entry is ``(name, is_baseline)``, not just the name. Sharing one namespace
54
+ would let somebody who submits a model called ``pca-50`` bury the published
55
+ ``pca-50`` baseline simply by submitting after it.
56
+ """
57
+ flagged = with_baseline_flag(df)
58
+ if flagged.empty:
59
+ return flagged
60
+ latest = flagged.groupby([MODEL_NAME, IS_BASELINE])["submitted_at"].transform("max")
61
+ return flagged[flagged["submitted_at"] == latest]
62
 
63
 
64
  def scores_from_rows(rows: pd.DataFrame, by_id: dict[str, dict]) -> list[TaskScore]:
 
94
  """Per-category means for every model that covered the whole board."""
95
  scoped = _board_registry(by_id, board)
96
  entries = []
97
+ for (model, is_baseline, submitted), rows in latest_only(df).groupby(
98
+ [MODEL_NAME, IS_BASELINE, "submitted_at"]
99
  ):
100
  if not board.task_ids.issubset({_norm_id(t) for t in rows["task_id"]}):
101
  continue
 
106
  "submitted_at": submitted,
107
  "categories": categories,
108
  "rank": sort_key(categories),
109
+ "is_baseline": bool(is_baseline),
110
  }
111
  )
112
  return sorted(entries, key=lambda e: -e["rank"])
 
134
  rows = []
135
  for position, entry in enumerate(entries, start=1):
136
  categories = entry["categories"]
137
+ model = display_name(entry["model_name"], entry["is_baseline"])
138
+ row = {"Rank": position, "Model": model}
139
  if len(columns) > 1:
140
  row["Mean"] = _round(entry["rank"])
141
  for cat in columns:
 
162
  for _, r in latest_only(df).iterrows():
163
  task_id = _norm_id(r["task_id"])
164
  if task_id in scoped:
165
+ model = display_name(str(r[MODEL_NAME]), bool(r[IS_BASELINE]))
166
+ scores[(task_id, model)] = float(r["score"])
167
  if not scores:
168
  return pd.DataFrame(columns=EMPTY_PER_TASK_COLUMNS)
169
 
 
190
 
191
  def top_models(
192
  df: pd.DataFrame, by_id: dict[str, dict], board: Board, limit: int
193
+ ) -> list[TopModel]:
194
+ """The board's best full-coverage entries -- the card teaser, baselines included."""
195
  return [
196
+ TopModel(
197
+ name=str(e["model_name"]),
198
+ score=round(e["rank"], SCORE_DECIMALS),
199
+ is_baseline=e["is_baseline"],
200
+ )
201
  for e in _entries(df, by_id, board)[:limit]
202
  ]
203
 
pages/about.md CHANGED
@@ -11,8 +11,7 @@ Reconstruction is easy to score. Clinical usefulness is not. That is the gap
11
  PRIMO measures.
12
 
13
  Today every cohort is **bulk RNA-seq**, in **immune-mediated inflammatory
14
- diseases**. More modalities are coming. Three questions are asked of each
15
- patient:
16
 
17
  | Task family | The question | Metric |
18
  |---|---|---|
@@ -64,12 +63,12 @@ Licences in force:
64
 
65
  {licenses}
66
 
67
- ## Cite PRIMO, or add a cohort to it
68
 
69
  Cite the benchmark paper: <https://openreview.net/forum?id=v2SA8gHwqo>
70
 
71
- PRIMO grows by adding cohorts, not models. If you hold a clinically annotated
72
- omics cohort, [get in touch](mailto:karim.elkanbi@scientalab.com).
73
 
74
  ## Links
75
 
 
11
  PRIMO measures.
12
 
13
  Today every cohort is **bulk RNA-seq**, in **immune-mediated inflammatory
14
+ diseases**. Three questions are asked of each patient:
 
15
 
16
  | Task family | The question | Metric |
17
  |---|---|---|
 
63
 
64
  {licenses}
65
 
66
+ ## Cite PRIMO
67
 
68
  Cite the benchmark paper: <https://openreview.net/forum?id=v2SA8gHwqo>
69
 
70
+ To add a cohort, open a modality, or partner on the methodology, see the
71
+ **Contribute** tab.
72
 
73
  ## Links
74
 
pages/contribute.md ADDED
@@ -0,0 +1,44 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ PRIMO is an open benchmark, and it is **deliberately incomplete** — the strip at
2
+ the top of this page says exactly how far it reaches today. Everything past that
3
+ line is something somebody else has to bring.
4
+
5
+ ## Add a cohort or a task
6
+
7
+ What we need is rare and it is not a model: **an omics cohort with patient-level
8
+ clinical labels** — drug response, a severity score, a molecular subtype. If you
9
+ hold one, or know a group that does, tell us:
10
+
11
+ - **modality** and roughly how many patients
12
+ - **what is labelled**, and how many patients carry that label
13
+ - **disease and tissue**
14
+ - **licence**, and whether the data can be redistributed
15
+
16
+ [Propose a cohort](mailto:karim.elkanbi@scientalab.com?subject=PRIMO%20—%20cohort%20proposal)
17
+
18
+ We do not have a formal acceptance process yet. It is being written with the
19
+ first partners, and we would rather say that than invent rules we have not
20
+ agreed on. What we can promise today is a real answer and a technical
21
+ conversation.
22
+
23
+ ## Add a modality
24
+
25
+ The **OPEN** cards on Home are the omics layers PRIMO does not reach yet. They
26
+ are not a roadmap — they are gaps, and each one is claimable. Opening a modality
27
+ takes one cohort with clinical labels, not a whole atlas.
28
+
29
+ [Claim an open board](mailto:karim.elkanbi@scientalab.com?subject=PRIMO%20—%20open%20board)
30
+
31
+ ## Become a founding partner
32
+
33
+ Founding partners shape PRIMO rather than just appear on it: a voice on the
34
+ methodology, co-signature on what we publish, and early access to the results.
35
+
36
+ [Talk to us about partnership](mailto:karim.elkanbi@scientalab.com?subject=PRIMO%20founding%20partner)
37
+ · [primomics.org](http://primomics.org/)
38
+
39
+ ## Smaller ways in
40
+
41
+ - **Question the method.** Open a thread in the Space's **Community** tab.
42
+ - **Submit a model.** Baselines are on the leaderboard already; beat them.
43
+ - **Read the paper** — [OpenReview](https://openreview.net/forum?id=v2SA8gHwqo)
44
+ - **Look at the data** — [ScientaLab/primo](https://huggingface.co/datasets/ScientaLab/primo)
pages/submit.md CHANGED
@@ -1,4 +1,19 @@
1
- Submit your model in **three steps**:
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
2
 
3
  1. **Get the data** → download the datasets from [ScientaLab/primo](https://huggingface.co/datasets/ScientaLab/primo) (start with its `datasets.yaml`).
4
  2. **Embed every dataset** → build **one** file: `dataset_id`, `sample_id`, then one column per embedding dim (`e0`, `e1`, …). CSV / TSV / Parquet, or NPZ.
@@ -16,3 +31,7 @@ d002,S1,0.31,0.02,-0.15
16
  scored: you get ranked on every **board** whose tasks you covered in full, and
17
  your numbers always appear in each board's **per-task** table. Nothing is
18
  thrown away.
 
 
 
 
 
1
+ **Start here, not with the docs.** Two files, both in this Space's repo:
2
+
3
+ - 📥 [`quickstart.py`](https://huggingface.co/spaces/ScientaLab/primo-eval/blob/main/quickstart.py)
4
+ — downloads every dataset, embeds them, writes a valid submission. Swap its
5
+ `embed` function for your model and you are done.
6
+ - 📄 [`example_submission.csv`](https://huggingface.co/spaces/ScientaLab/primo-eval/blob/main/example_submission.csv)
7
+ — four lines, fake numbers, the exact shape we expect.
8
+
9
+ ```bash
10
+ pip install anndata scikit-learn pandas pyyaml huggingface_hub
11
+ python quickstart.py --out submission.parquet
12
+ ```
13
+
14
+ ---
15
+
16
+ Or do it by hand, in **three steps**:
17
 
18
  1. **Get the data** → download the datasets from [ScientaLab/primo](https://huggingface.co/datasets/ScientaLab/primo) (start with its `datasets.yaml`).
19
  2. **Embed every dataset** → build **one** file: `dataset_id`, `sample_id`, then one column per embedding dim (`e0`, `e1`, …). CSV / TSV / Parquet, or NPZ.
 
31
  scored: you get ranked on every **board** whose tasks you covered in full, and
32
  your numbers always appear in each board's **per-task** table. Nothing is
33
  thrown away.
34
+
35
+ **Your first target is the baselines.** Reference submissions we run ourselves —
36
+ a random embedding, a PCA of log-CPM — appear on the boards labelled
37
+ `(baseline)`. Beating them is the bar.
quickstart.py ADDED
@@ -0,0 +1,100 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Produce a valid PRIMO submission in one command, then swap in your own model.
2
+
3
+ Downloads every public dataset, embeds each one, and writes the single file the
4
+ Submit tab expects. The embedding here is deliberately dumb -- log2(CPM+1) then
5
+ PCA -- because the point is the plumbing, not the score: replace ``embed`` with
6
+ your encoder and nothing else changes.
7
+
8
+ pip install anndata scikit-learn pandas pyyaml huggingface_hub
9
+ python quickstart.py --out submission.parquet
10
+
11
+ Standalone on purpose: no import from this Space and none from our monorepo, so
12
+ it keeps working if you copy the file into your own project.
13
+ """
14
+
15
+ import argparse
16
+ from pathlib import Path
17
+
18
+ import anndata as ad
19
+ import numpy as np
20
+ import pandas as pd
21
+ import yaml
22
+ from huggingface_hub import snapshot_download
23
+ from sklearn.decomposition import PCA
24
+
25
+ PUBLIC_REPO = "ScientaLab/primo"
26
+ MANIFEST_FILENAME = "datasets.yaml"
27
+
28
+ DATASET_ID = "dataset_id"
29
+ SAMPLE_ID = "sample_id"
30
+
31
+ TARGET_SUM = 1_000_000
32
+ N_COMPONENTS = 50
33
+ RANDOM_STATE = 0
34
+
35
+
36
+ def embed(adata: ad.AnnData) -> np.ndarray:
37
+ """One dataset's raw counts -> one vector per patient. Replace me.
38
+
39
+ Whatever you return, the contract is the same: one row per sample, in
40
+ ``adata.obs_names`` order, all finite. The embedding width is yours to pick
41
+ and may differ from one dataset to the next.
42
+ """
43
+ x = adata.X
44
+ x = x.toarray() if hasattr(x, "toarray") else np.asarray(x)
45
+ x = x.astype(float)
46
+ counts = x.sum(axis=1, keepdims=True)
47
+ x = np.log2(x / np.where(counts == 0, 1, counts) * TARGET_SUM + 1)
48
+ k = min(N_COMPONENTS, x.shape[0] - 1, x.shape[1])
49
+ return PCA(n_components=k, random_state=RANDOM_STATE).fit_transform(x)
50
+
51
+
52
+ def download(token: str | None) -> Path:
53
+ """Pull the public benchmark (manifest + every ``expression.h5ad``)."""
54
+ return Path(snapshot_download(PUBLIC_REPO, repo_type="dataset", token=token))
55
+
56
+
57
+ def dataset_ids(root: Path) -> list[str]:
58
+ """The opaque ids to embed, read off the public manifest."""
59
+ manifest = yaml.safe_load((root / MANIFEST_FILENAME).read_text())
60
+ entries = manifest.get("datasets", []) if isinstance(manifest, dict) else manifest
61
+ return [str(entry["id"]) for entry in entries]
62
+
63
+
64
+ def build(root: Path) -> pd.DataFrame:
65
+ """Embed every dataset into the one frame the Submit tab expects.
66
+
67
+ Datasets of different widths stack into one table; the extra columns of a
68
+ narrower dataset stay empty and the evaluator drops them per dataset, so each
69
+ dataset keeps its own embedding size.
70
+ """
71
+ blocks = []
72
+ for dataset_id in dataset_ids(root):
73
+ adata = ad.read_h5ad(root / dataset_id / "expression.h5ad")
74
+ vectors = embed(adata)
75
+ print(f"{dataset_id}: {adata.n_obs} samples -> {vectors.shape[1]} dims")
76
+ block = pd.DataFrame(
77
+ vectors, columns=[f"e{i}" for i in range(vectors.shape[1])]
78
+ )
79
+ block.insert(0, SAMPLE_ID, adata.obs_names.to_numpy())
80
+ block.insert(0, DATASET_ID, dataset_id)
81
+ blocks.append(block)
82
+ return pd.concat(blocks, ignore_index=True)
83
+
84
+
85
+ def main() -> None:
86
+ parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
87
+ parser.add_argument("--out", type=Path, default=Path("submission.parquet"))
88
+ parser.add_argument("--token", default=None, help="HF token, if you need one.")
89
+ args = parser.parse_args()
90
+
91
+ submission = build(download(args.token))
92
+ if args.out.suffix == ".csv":
93
+ submission.to_csv(args.out, index=False)
94
+ else:
95
+ submission.to_parquet(args.out, index=False)
96
+ print(f"\nWrote {args.out} — {len(submission)} rows. Upload it on the Submit tab.")
97
+
98
+
99
+ if __name__ == "__main__":
100
+ main()
results.py ADDED
@@ -0,0 +1,111 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Persisted leaderboard rows: the results-CSV schema and its Hugging Face IO.
2
+
3
+ Kept out of ``app.py`` so the schema has one owner and so the baseline publisher
4
+ (``benchmark/public_benchmark/baselines.py``) can append rows without importing
5
+ Gradio. ``huggingface_hub`` is imported lazily, so the unit tests touch no
6
+ network.
7
+
8
+ ``is_baseline`` marks a reference submission we produced ourselves (a random
9
+ embedding, a PCA of log-CPM) rather than a model somebody sent us. Baselines are
10
+ ranked in place, never pinned: the point of showing them is that a foundation
11
+ model can lose to a PCA, and a row pushed to the bottom of the table would hide
12
+ exactly that.
13
+
14
+ Rows written before the flag existed carry no column at all, so every reader
15
+ goes through ``with_baseline_flag``: a missing or unparsable flag means "a
16
+ submitted model", the conservative reading.
17
+ """
18
+
19
+ import io
20
+
21
+ import pandas as pd
22
+ from evaluator import RESULTS_REPO
23
+
24
+ RESULTS_FILE = "task_results.csv"
25
+ SUBMISSIONS_FILE = "submissions.csv"
26
+
27
+ MODEL_NAME = "model_name"
28
+ IS_BASELINE = "is_baseline"
29
+
30
+ RESULT_COLUMNS = [MODEL_NAME, "task_id", "score", "submitted_at", IS_BASELINE]
31
+ SUBMISSION_COLUMNS = [
32
+ MODEL_NAME,
33
+ "submitted_at",
34
+ "hf_username",
35
+ "email",
36
+ "paper_link",
37
+ "hf_model_link",
38
+ "notes",
39
+ ]
40
+
41
+ TRUTHY = ("true", "1")
42
+ BASELINE_TAG = "(baseline)"
43
+
44
+
45
+ def with_baseline_flag(df: pd.DataFrame) -> pd.DataFrame:
46
+ """Guarantee a boolean ``is_baseline`` column, whatever the CSV held.
47
+
48
+ Read back from CSV the column can be bool, the strings ``True``/``False``, or
49
+ absent on rows written before baselines existed; all of those must collapse
50
+ to a real boolean before anything ranks on it.
51
+ """
52
+ if IS_BASELINE not in df.columns:
53
+ return df.assign(**{IS_BASELINE: False})
54
+ flags = df[IS_BASELINE].astype(str).str.strip().str.lower().isin(TRUTHY)
55
+ return df.assign(**{IS_BASELINE: flags})
56
+
57
+
58
+ def display_name(model: str, is_baseline: bool) -> str:
59
+ """Leaderboard label: a baseline says so, in the one column everybody reads."""
60
+ return f"{model} {BASELINE_TAG}" if is_baseline else model
61
+
62
+
63
+ def read_csv(filename: str, columns: list[str], token: str | None) -> pd.DataFrame:
64
+ """One CSV from the private results dataset; an absent file is an empty table."""
65
+ from huggingface_hub import hf_hub_download
66
+ from huggingface_hub.utils import EntryNotFoundError, RepositoryNotFoundError
67
+
68
+ try:
69
+ path = hf_hub_download(RESULTS_REPO, filename, repo_type="dataset", token=token)
70
+ except (RepositoryNotFoundError, EntryNotFoundError):
71
+ return pd.DataFrame(columns=columns)
72
+ return pd.read_csv(path)
73
+
74
+
75
+ def upload_csv(filename: str, df: pd.DataFrame, token: str | None) -> None:
76
+ """Overwrite one CSV in the private results dataset."""
77
+ from huggingface_hub import HfApi
78
+
79
+ api = HfApi(token=token)
80
+ api.create_repo(RESULTS_REPO, repo_type="dataset", private=True, exist_ok=True)
81
+ buffer = io.BytesIO()
82
+ df.to_csv(buffer, index=False)
83
+ buffer.seek(0)
84
+ api.upload_file(
85
+ path_or_fileobj=buffer,
86
+ path_in_repo=filename,
87
+ repo_id=RESULTS_REPO,
88
+ repo_type="dataset",
89
+ )
90
+
91
+
92
+ def read_results(token: str | None) -> pd.DataFrame:
93
+ """Every persisted task score, with the baseline flag normalised."""
94
+ return with_baseline_flag(read_csv(RESULTS_FILE, RESULT_COLUMNS, token))
95
+
96
+
97
+ def append_results(rows: list[dict], token: str | None) -> None:
98
+ """Append scored rows to the leaderboard, keeping the history intact."""
99
+ if not rows:
100
+ return
101
+ df = pd.concat([read_results(token), pd.DataFrame(rows)], ignore_index=True)
102
+ upload_csv(RESULTS_FILE, with_baseline_flag(df), token)
103
+
104
+
105
+ def append_submission(meta: dict, token: str | None) -> None:
106
+ """Persist a submitter's contact metadata, never shown on a public page."""
107
+ df = pd.concat(
108
+ [read_csv(SUBMISSIONS_FILE, SUBMISSION_COLUMNS, token), pd.DataFrame([meta])],
109
+ ignore_index=True,
110
+ )
111
+ upload_csv(SUBMISSIONS_FILE, df, token)
style.css CHANGED
@@ -1,15 +1,64 @@
 
 
 
 
 
1
  :root {
2
- --primo-bg: #0b0f19;
3
- --primo-card: #10151f;
4
- --primo-card-hover: #151b28;
5
- --primo-line: #1f2733;
6
- --primo-text: #e8ebf2;
7
- --primo-muted: #8b93a7;
8
- --primo-dim: #626b7e;
9
- --primo-modality: #60a5fa;
10
- --primo-area: #2dd4bf;
11
- --primo-category: #fbbf24;
12
- --primo-featured: #818cf8;
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
13
  }
14
 
15
  .primo-home {
@@ -47,8 +96,8 @@
47
  font-weight: 600;
48
  letter-spacing: 0.08em;
49
  color: var(--primo-featured);
50
- background: rgba(129, 140, 248, 0.12);
51
- border: 1px solid rgba(129, 140, 248, 0.25);
52
  border-radius: 999px;
53
  padding: 4px 12px;
54
  margin-bottom: 12px;
@@ -75,12 +124,13 @@
75
  padding: 16px 18px;
76
  text-decoration: none !important;
77
  color: inherit;
78
- transition: background 0.15s ease, border-color 0.15s ease, transform 0.15s ease;
79
  }
80
 
81
  .primo-card:hover {
82
  background: var(--primo-card-hover);
83
  transform: translateY(-2px);
 
84
  }
85
 
86
  .primo-accent-modality {
@@ -102,6 +152,50 @@
102
  border-color: var(--primo-category);
103
  }
104
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
105
  .primo-card-head {
106
  display: flex;
107
  align-items: center;
@@ -109,9 +203,32 @@
109
  margin-bottom: 8px;
110
  }
111
 
112
- .primo-icon {
113
- font-size: 1.25rem;
 
 
 
 
 
114
  line-height: 1;
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
115
  }
116
 
117
  .primo-card-title {
@@ -198,6 +315,29 @@
198
  font-style: italic;
199
  }
200
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
201
  .primo-pills {
202
  display: flex;
203
  flex-wrap: wrap;
@@ -207,7 +347,7 @@
207
  .primo-pill {
208
  font-size: 0.72rem;
209
  color: var(--primo-muted);
210
- background: rgba(255, 255, 255, 0.04);
211
  border: 1px solid var(--primo-line);
212
  border-radius: 999px;
213
  padding: 2px 10px;
@@ -215,16 +355,18 @@
215
 
216
  .primo-pill-modality {
217
  color: var(--primo-modality);
218
- border-color: rgba(96, 165, 250, 0.3);
219
- background: rgba(96, 165, 250, 0.08);
220
  }
221
 
 
 
222
  .primo-section {
223
  border: 1px solid var(--primo-line);
224
  border-radius: 14px;
225
  padding: 0 0 18px;
226
  margin-bottom: 20px;
227
- background: rgba(255, 255, 255, 0.012);
228
  }
229
 
230
  .primo-section-head {
@@ -249,7 +391,7 @@
249
  font-size: 0.75rem;
250
  font-weight: 600;
251
  color: var(--primo-muted);
252
- background: rgba(255, 255, 255, 0.06);
253
  border-radius: 999px;
254
  padding: 2px 9px;
255
  margin-left: 6px;
@@ -291,9 +433,9 @@
291
  theme. Scoped to :not(pre) so fenced blocks keep their own styling. */
292
  .md :not(pre) > code,
293
  .prose :not(pre) > code {
294
- background: rgba(129, 140, 248, 0.12) !important;
295
- color: #a5b4fc !important;
296
- border: 1px solid rgba(129, 140, 248, 0.22);
297
  border-radius: 5px;
298
  padding: 1px 6px;
299
  font-size: 0.9em;
 
1
+ @import url("https://fonts.googleapis.com/css2?family=Funnel+Display:wght@400..800&display=swap");
2
+
3
+ /* PRIMO charter, light. Cyan is a mark colour: at 2.2:1 on Paper it is
4
+ unreadable as text, so anything you have to read uses --primo-cyan-ink
5
+ (4.5:1), and raw cyan is kept for rules, dots and borders. */
6
  :root {
7
+ --primo-navy: #080f5f;
8
+ --primo-deep-navy: #050a3c;
9
+ --primo-cyan: #16b3c0;
10
+ --primo-cyan-ink: #0f7f89;
11
+ --primo-paper: #f3f8f8;
12
+ --primo-slate: #54686b;
13
+ --primo-hairline: #d6e2e3;
14
+
15
+ --primo-bg: var(--primo-paper);
16
+ --primo-card: #ffffff;
17
+ --primo-card-hover: #ffffff;
18
+ --primo-line: var(--primo-hairline);
19
+ --primo-text: var(--primo-deep-navy);
20
+ --primo-muted: var(--primo-slate);
21
+ --primo-dim: #8ca0a3;
22
+ --primo-modality: var(--primo-navy);
23
+ --primo-area: var(--primo-cyan-ink);
24
+ --primo-category: var(--primo-slate);
25
+ --primo-featured: var(--primo-navy);
26
+ --primo-wash: rgba(8, 15, 95, 0.04);
27
+ }
28
+
29
+ .primo-home h2,
30
+ .primo-home h3,
31
+ .primo-board-head h2,
32
+ .prose h1,
33
+ .prose h2,
34
+ .prose h3 {
35
+ font-family: "Funnel Display", "Funnel Sans", ui-sans-serif, system-ui, sans-serif;
36
+ letter-spacing: -0.015em;
37
+ }
38
+
39
+ .primo-banner {
40
+ display: flex;
41
+ align-items: center;
42
+ gap: 10px;
43
+ flex-wrap: wrap;
44
+ font-size: 0.82rem;
45
+ color: var(--primo-muted);
46
+ border: 1px solid var(--primo-line);
47
+ border-radius: 10px;
48
+ padding: 9px 14px;
49
+ margin-bottom: 4px;
50
+ }
51
+
52
+ .primo-banner-tag {
53
+ font-size: 0.7rem;
54
+ font-weight: 700;
55
+ letter-spacing: 0.06em;
56
+ text-transform: uppercase;
57
+ color: var(--primo-featured);
58
+ background: var(--primo-wash);
59
+ border: 1px solid rgba(8, 15, 95, 0.18);
60
+ border-radius: 999px;
61
+ padding: 2px 9px;
62
  }
63
 
64
  .primo-home {
 
96
  font-weight: 600;
97
  letter-spacing: 0.08em;
98
  color: var(--primo-featured);
99
+ background: var(--primo-wash);
100
+ border: 1px solid rgba(8, 15, 95, 0.18);
101
  border-radius: 999px;
102
  padding: 4px 12px;
103
  margin-bottom: 12px;
 
124
  padding: 16px 18px;
125
  text-decoration: none !important;
126
  color: inherit;
127
+ transition: box-shadow 0.15s ease, border-color 0.15s ease, transform 0.15s ease;
128
  }
129
 
130
  .primo-card:hover {
131
  background: var(--primo-card-hover);
132
  transform: translateY(-2px);
133
+ box-shadow: 0 6px 18px rgba(8, 15, 95, 0.1);
134
  }
135
 
136
  .primo-accent-modality {
 
152
  border-color: var(--primo-category);
153
  }
154
 
155
+ /* ``align-self`` so an open card sizes to its own text: stretched to match a
156
+ full board it becomes a large empty rectangle, which reads as broken. */
157
+ .primo-card-open {
158
+ align-self: start;
159
+ background: transparent;
160
+ border-style: dashed;
161
+ border-left-color: var(--primo-line);
162
+ opacity: 0.72;
163
+ }
164
+
165
+ .primo-card-open:hover {
166
+ opacity: 1;
167
+ background: var(--primo-card);
168
+ border-color: var(--primo-dim);
169
+ }
170
+
171
+ .primo-card-open .primo-card-title {
172
+ color: var(--primo-muted);
173
+ }
174
+
175
+ .primo-badge-open {
176
+ margin-left: auto;
177
+ font-size: 0.6rem;
178
+ font-weight: 700;
179
+ letter-spacing: 0.08em;
180
+ color: var(--primo-dim);
181
+ border: 1px solid var(--primo-line);
182
+ border-radius: 999px;
183
+ padding: 2px 8px;
184
+ }
185
+
186
+ .primo-open-cta {
187
+ font-size: 0.8rem;
188
+ font-weight: 600;
189
+ color: var(--primo-area);
190
+ }
191
+
192
+ .primo-count-open {
193
+ font-size: 0.7rem;
194
+ font-weight: 600;
195
+ color: var(--primo-dim);
196
+ margin-left: 6px;
197
+ }
198
+
199
  .primo-card-head {
200
  display: flex;
201
  align-items: center;
 
203
  margin-bottom: 8px;
204
  }
205
 
206
+ /* Replaces the per-board emoji: letters scale to any modality anyone adds, and
207
+ the colour is the group's, so a new board needs no design decision. */
208
+ .primo-code {
209
+ flex: none;
210
+ font-size: 0.66rem;
211
+ font-weight: 700;
212
+ letter-spacing: 0.07em;
213
  line-height: 1;
214
+ color: var(--primo-card);
215
+ background: var(--primo-modality);
216
+ border-radius: 5px;
217
+ padding: 5px 6px;
218
+ }
219
+
220
+ .primo-code-area {
221
+ background: var(--primo-area);
222
+ }
223
+
224
+ .primo-code-category {
225
+ background: var(--primo-category);
226
+ }
227
+
228
+ .primo-board-head .primo-code {
229
+ vertical-align: middle;
230
+ display: inline-block;
231
+ margin-right: 9px;
232
  }
233
 
234
  .primo-card-title {
 
315
  font-style: italic;
316
  }
317
 
318
+ .primo-top-baseline span:first-child {
319
+ color: var(--primo-muted);
320
+ }
321
+
322
+ .primo-tag {
323
+ font-size: 0.6rem;
324
+ letter-spacing: 0.06em;
325
+ text-transform: uppercase;
326
+ color: var(--primo-dim);
327
+ border: 1px solid var(--primo-line);
328
+ border-radius: 999px;
329
+ padding: 1px 6px;
330
+ margin-left: 7px;
331
+ }
332
+
333
+ .primo-top-foot {
334
+ font-size: 0.75rem;
335
+ color: var(--primo-dim);
336
+ font-style: italic;
337
+ border-top: 1px solid var(--primo-line);
338
+ padding-top: 7px;
339
+ }
340
+
341
  .primo-pills {
342
  display: flex;
343
  flex-wrap: wrap;
 
347
  .primo-pill {
348
  font-size: 0.72rem;
349
  color: var(--primo-muted);
350
+ background: var(--primo-wash);
351
  border: 1px solid var(--primo-line);
352
  border-radius: 999px;
353
  padding: 2px 10px;
 
355
 
356
  .primo-pill-modality {
357
  color: var(--primo-modality);
358
+ border-color: rgba(8, 15, 95, 0.22);
359
+ background: var(--primo-wash);
360
  }
361
 
362
+ /* Transparent, not white: the cards inside are white, and a white box behind
363
+ them would flatten the two into one surface. */
364
  .primo-section {
365
  border: 1px solid var(--primo-line);
366
  border-radius: 14px;
367
  padding: 0 0 18px;
368
  margin-bottom: 20px;
369
+ background: transparent;
370
  }
371
 
372
  .primo-section-head {
 
391
  font-size: 0.75rem;
392
  font-weight: 600;
393
  color: var(--primo-muted);
394
+ background: var(--primo-wash);
395
  border-radius: 999px;
396
  padding: 2px 9px;
397
  margin-left: 6px;
 
433
  theme. Scoped to :not(pre) so fenced blocks keep their own styling. */
434
  .md :not(pre) > code,
435
  .prose :not(pre) > code {
436
+ background: rgba(22, 179, 192, 0.1) !important;
437
+ color: var(--primo-cyan-ink) !important;
438
+ border: 1px solid rgba(22, 179, 192, 0.3);
439
  border-radius: 5px;
440
  padding: 1px 6px;
441
  font-size: 0.9em;