.claude/skills/leaderboard-manager/references/manage-display.md CHANGED
@@ -6,34 +6,76 @@ renderer** (`render_leaderboard()` in `src/common/leaderboard.py`). Any change t
6
  function affects both boards simultaneously. Method-specific charts live in
7
  `src/pages/MethodDetails.py`.
8
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
9
  ---
10
 
11
  ## Leaderboard table
12
 
13
  ### Column visibility and order
14
 
15
- The columns shown depend on whether "All" or a specific benchmark is selected. Both lists
16
- are defined in `src/common/leaderboard.py`:
17
-
18
- - **All-benchmarks view** (`table_column_order`, line 296–305):
19
- ```python
20
- ["Placement", "abbreviation", "Method", "family", "Score",
21
- "Targets Used", "Ensemble Sizes Used", "Benchmarks Used"]
22
- ```
23
- - **Single-benchmark view** (`table_column_order`, line 307–319):
24
- ```python
25
- ["Placement", "abbreviation", "Method", "family", "Score",
26
- "Mean Forward Model Runs", "Minimum Forward Model Runs",
27
- "Optimal Ensemble Size", "Targets Used", "Ensemble Sizes Used", "Benchmarks Used"]
28
- ```
29
-
30
- To **show or hide** a column, add/remove it from the relevant list.
31
  To **reorder** columns, rearrange entries within the list.
32
 
33
  ### Column labels and formatting
34
 
35
- Column display names and number formats are set in `st.dataframe(... column_config=...)`
36
- at lines 326–339. Each column has an entry like:
37
 
38
  ```python
39
  "Mean Forward Model Runs": st.column_config.NumberColumn(
@@ -48,8 +90,8 @@ To **rename** a displayed column header, change the string argument (second posi
48
  in the `column_config` entry. The dict key must still match the DataFrame column name.
49
 
50
  To **add a new column to the table**, you also need to produce it in `build_scored_table()`
51
- (line 111) — add it to the `.agg()` call or compute it after the groupby, then add a
52
- `column_config` entry and include it in `table_column_order`.
53
 
54
  ### Adding `failure_rate` to the leaderboard table
55
 
@@ -64,7 +106,8 @@ To add it:
64
 
65
  ## Scoring modes
66
 
67
- The four scoring modes are defined in `scoring_options` (line 82):
 
68
  ```python
69
  scoring_options = [
70
  "Mean Forward Model Runs",
@@ -74,24 +117,27 @@ scoring_options = [
74
  ]
75
  ```
76
 
77
- Each mode maps to a sort key and a score column assignment in `build_scored_table()` at
78
- lines 185–203. To **add a new scoring mode**, add its name to `scoring_options` and add
79
- an `elif` branch in the scoring logic block that sets `scored_df["Score"]`,
80
- `sort_columns`, and `ascending`.
81
 
82
- The "Custom Blend" slider weight (lines 357–368) is keyed to the session-state key
83
- `k_weight` no change needed when adding a non-blend mode.
 
 
84
 
85
  ---
86
 
87
  ## Leaderboard line chart
88
 
89
- The chart that appears in single-benchmark view ("Mean Forward Model Runs vs Ensemble
90
- Size") is at lines 387–396 (inside the `if selected_benchmark != "All":` guard at line
91
- 376):
92
 
93
  ```python
94
- ens_ticks = sorted(chart_df["ensemble_size"].unique().tolist())
 
95
  chart = (
96
  alt.Chart(chart_df)
97
  .mark_line(point=True)
@@ -102,7 +148,7 @@ chart = (
102
  axis=alt.Axis(values=ens_ticks, format="d"), # integer ticks only
103
  ),
104
  y=alt.Y("mean_forward_runs:Q", title="Mean Forward Model Runs"),
105
- color=alt.Color("abbreviation:N", title="Method"),
106
  tooltip=["abbreviation", "ensemble_size",
107
  alt.Tooltip("mean_forward_runs:Q", format=".4f")],
108
  )
@@ -119,11 +165,14 @@ Common changes:
119
  - **Add failure rate as a second Y-axis or tooltip**: add `failure_rate` to `chart_df`
120
  groupby and include it in `tooltip`.
121
  - **Change mark type** (e.g. bars): replace `mark_line(point=True)` with `mark_bar()`.
122
- - **Color palette**: add `.configure_range(category={"scheme": "tableau10"})` after the
123
- chart construction.
 
 
124
 
125
- The chart only renders when a single benchmark is selected (`if selected_benchmark != "All":`
126
- guard at line 376).
 
127
 
128
  ---
129
 
@@ -133,22 +182,17 @@ The per-method page (`src/pages/MethodDetails.py`) has its own controls.
133
 
134
  ### Target level radio
135
 
136
- Line 111:
137
- ```python
138
- target_options = ["1.0", "1.1", "1.2"]
139
- selected_target = st.radio("RMSE Target Level", options=target_options, horizontal=True)
140
- ```
141
-
142
- These are **hardcoded strings**. If the benchmark gains new target levels, update this
143
- list. A more robust approach is to derive them from the data:
144
  ```python
145
  target_options = sorted(metric_store["rmse_target"].astype(str).unique().tolist())
146
  ```
147
 
148
  ### Per-method special-case charts
149
 
150
- The HM (History Matching) failure-analysis chart (lines 122–138) is the canonical
151
- example of a per-method detail chart:
152
 
153
  ```python
154
  if sel == "HM":
@@ -178,19 +222,25 @@ encoding, change the `.encode()` arguments analogously to the leaderboard chart
178
 
179
  The UQ leaderboard passes a `budget_store` DataFrame to `render_leaderboard()` via the
180
  `budget_store` keyword argument. This enables two additional chart sections rendered in
181
- `src/common/leaderboard.py` after the main performance chart:
 
182
 
183
  1. **"Mean Iterations for Coverage vs Ensemble Size"** — line chart of `mean_iters` vs
184
- ensemble size. Uses the same integer-tick pattern as the main chart.
185
-
186
- 2. **"Failure Rate of Hitting Target ..."**grouped bar chart with `ensemble_size:O`
187
- on the x-axis (ordinal, not quantitative) and `mean_failure_rate:Q` on y. The title
188
- reflects the currently selected target level.
189
-
190
- To **add or modify a UQ chart**, edit the `if budget_store is not None` block in
191
- `src/common/leaderboard.py`. The `budget_store` DataFrame has columns: `benchmark`,
192
- `algorithm_type`, `abbreviation`, `family`, `uq_target`, `ensemble_size`, `mean_budget`,
193
- `mean_iters`, `failure_count`, `failure_rate`, `n_seeds`.
 
 
 
 
 
194
 
195
  **Bar chart integer x-axis:** Use `ensemble_size:O` (ordinal) with an explicit sort list
196
  and `axis=alt.Axis(labelAngle=0)` to suppress diagonal labels:
@@ -224,23 +274,63 @@ When **adding a new method**, its color is assigned automatically — no manual
224
  needed, as long as it is appended to the end of `KNOWN_METHODS`. The palette cycles every
225
  10 methods.
226
 
227
- ### How to import shared colors in a new chart
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
228
 
229
  ```python
230
  try:
231
  from common.method_registry import METHOD_COLORS
232
  except ModuleNotFoundError:
233
  from src.common.method_registry import METHOD_COLORS
234
-
235
- method_color = alt.Color(
236
- "abbreviation:N",
237
- title="Method",
238
- scale=alt.Scale(domain=list(METHOD_COLORS.keys()), range=list(METHOD_COLORS.values())),
239
- )
240
  ```
241
 
242
- Always use the full registry domain/range (not a filtered per-benchmark subset) so colors
243
- are stable when the user switches benchmarks or navigates between pages.
 
 
 
244
 
245
  ### General plotting conventions
246
 
@@ -249,7 +339,7 @@ new Altair chart in the app:
249
 
250
  | Concern | Convention |
251
  |---|---|
252
- | Method color | `METHOD_COLORS` via the import above — never a local palette |
253
  | Benchmark color | `alt.Color("benchmark:N")` — Altair assigns automatically; no custom palette needed |
254
  | Full-width charts | `st.altair_chart(chart, use_container_width=True)` — never `width="stretch"` |
255
  | Ensemble-size x-axis (quantitative) | `alt.Axis(values=ens_ticks, format="d")` — derive ticks from data, integer format |
@@ -266,7 +356,12 @@ new Altair chart in the app:
266
  per-method surfaces).
267
  2. If adding a column: produce it in the aggregation AND add a `column_config` entry AND
268
  add it to `table_column_order`.
269
- 3. Restart or clear Streamlit cache to pick up Python changes (code changes take effect on
 
 
 
 
 
270
  restart; cache only matters for data changes).
271
- 4. Smoke-test on both the "All" view and a single-benchmark view to catch the different
272
- `table_column_order` branches.
 
6
  function affects both boards simultaneously. Method-specific charts live in
7
  `src/pages/MethodDetails.py`.
8
 
9
+ **A note on line numbers:** this file used to cite specific line numbers (e.g. "line 296").
10
+ They drift out of date after almost any edit to `leaderboard.py` — a single reordering
11
+ task can shift everything below it by 50+ lines. Below, locations are given as
12
+ **function/variable/string anchors** you can `grep` for instead — these survive
13
+ refactors. If you add a new durable line-number reference anyway, expect to fix it again
14
+ next time someone edits the file.
15
+
16
+ ---
17
+
18
+ ## Page layout (top to bottom)
19
+
20
+ `render_leaderboard()` renders, in order:
21
+
22
+ 1. Suitability table (`_render_suitability_table()`)
23
+ 2. Benchmark selector (`st.selectbox("Benchmark", ...)`) — always a single benchmark;
24
+ there is **no "All benchmarks" option**. Every chart and the table below operate on
25
+ whatever `selected_benchmark` is currently chosen.
26
+ 3. **Controls — always visible**, not collapsed. Look for `st.subheader("Scoring &
27
+ Target Controls")`: target-level radio, scoring-mode radio (+ blend-weight slider
28
+ when "Custom Blend" is selected), and the "Methods to display in charts" multiselect.
29
+ These used to be wrapped in `st.expander(..., expanded=False)`; that was removed so
30
+ users don't have to open a dropdown to see or change the target level.
31
+ 4. **Charts** — main "Mean Forward Model Runs vs Ensemble Size" chart, then (UQ only)
32
+ the "Mean Iterations for Coverage" chart, then (if `show_failure_panel=True`) the
33
+ failure-rate bar chart.
34
+ 5. **Leaderboard table** — `st.dataframe(leaderboard_df, ...)`.
35
+
36
+ Charts render above the table and below the controls. If you need to reorder these
37
+ sections again, see the gotcha below before moving the main chart.
38
+
39
+ ### Gotcha: the main chart is gated by `leaderboard_df`, not by its own emptiness check
40
+
41
+ The main forward-model-runs chart is wrapped in `if not leaderboard_df.empty:` — the
42
+ same DataFrame the table renders from — rather than an independent check on its own
43
+ `chart_df`/`fail_df`. This is deliberate: `leaderboard_df` (from `build_scored_table()`)
44
+ is empty in two distinct cases:
45
+
46
+ - No rows at all for the current benchmark/target (nothing was ever run).
47
+ - Every seed failed (`metric` is NaN for all rows) — a real "DNF" case.
48
+
49
+ In the DNF case, the chart is intentionally suppressed too — the user sees a warning
50
+ ("All runs failed...") instead of a near-empty line chart with only fail-crosses at
51
+ y=0. If you're adding a new chart section anywhere near the main chart, decide
52
+ deliberately whether it should share this gate or have its own (the UQ iters chart and
53
+ the failure-rate chart do **not** share it — they have their own internal emptiness
54
+ checks, e.g. `if not all_ens_combos_iters.empty:`, so they can render even when the
55
+ scored table is empty).
56
+
57
  ---
58
 
59
  ## Leaderboard table
60
 
61
  ### Column visibility and order
62
 
63
+ Search for `table_column_order = [` in `src/common/leaderboard.py` there is a single
64
+ list (not one-per-view):
65
+ ```python
66
+ table_column_order = [
67
+ "Placement", "abbreviation", "Method", "family", "Score",
68
+ "Mean Forward Model Runs", "Minimum Forward Model Runs",
69
+ "Mean Failure Rate (%)", "Optimal Ensemble Size", "Ensemble Sizes Used",
70
+ ]
71
+ ```
72
+ To **show or hide** a column, add/remove it from this list.
 
 
 
 
 
 
73
  To **reorder** columns, rearrange entries within the list.
74
 
75
  ### Column labels and formatting
76
 
77
+ Column display names and number formats are set in the `st.dataframe(... column_config=...)`
78
+ call right after `table_column_order` is used. Each column has an entry like:
79
 
80
  ```python
81
  "Mean Forward Model Runs": st.column_config.NumberColumn(
 
90
  in the `column_config` entry. The dict key must still match the DataFrame column name.
91
 
92
  To **add a new column to the table**, you also need to produce it in `build_scored_table()`
93
+ — add it to a `.agg()` call or compute it after the groupby, then add a `column_config`
94
+ entry and include it in `table_column_order`.
95
 
96
  ### Adding `failure_rate` to the leaderboard table
97
 
 
106
 
107
  ## Scoring modes
108
 
109
+ The four scoring modes are defined in `scoring_options` (a list literal near the top of
110
+ `render_leaderboard()`, right after `target_options` is computed):
111
  ```python
112
  scoring_options = [
113
  "Mean Forward Model Runs",
 
117
  ]
118
  ```
119
 
120
+ Each mode maps to a sort key and a score column assignment inside `build_scored_table()`
121
+ search for the `if scoring_mode == "Mean Forward Model Runs": ... elif ...` chain. To
122
+ **add a new scoring mode**, add its name to `scoring_options` and add an `elif` branch in
123
+ that chain that sets `scored_df["Score"]`, `sort_columns`, and `ascending`.
124
 
125
+ The "Custom Blend" slider (`st.slider("Blend Weight: Forward Runs vs Ensemble Size", ...)`)
126
+ is keyed to the session-state key `k_weight` and only rendered when
127
+ `st.session_state[k_scoring] == "Custom Blend"` — no change needed when adding a
128
+ non-blend mode. It lives in the always-visible controls section, not behind an expander.
129
 
130
  ---
131
 
132
  ## Leaderboard line chart
133
 
134
+ The chart that appears above the table ("Mean Forward Model Runs vs Ensemble Size") is
135
+ built right after `st.subheader("Mean Forward Model Runs vs Ensemble Size")`, inside the
136
+ `if not leaderboard_df.empty:` guard described in the gotcha above:
137
 
138
  ```python
139
+ ens_ticks = sorted(all_ens_combos["ensemble_size"].unique().tolist()) if not all_ens_combos.empty else []
140
+ main_color = _method_color(all_ens_combos["abbreviation"].unique().tolist())
141
  chart = (
142
  alt.Chart(chart_df)
143
  .mark_line(point=True)
 
148
  axis=alt.Axis(values=ens_ticks, format="d"), # integer ticks only
149
  ),
150
  y=alt.Y("mean_forward_runs:Q", title="Mean Forward Model Runs"),
151
+ color=main_color,
152
  tooltip=["abbreviation", "ensemble_size",
153
  alt.Tooltip("mean_forward_runs:Q", format=".4f")],
154
  )
 
165
  - **Add failure rate as a second Y-axis or tooltip**: add `failure_rate` to `chart_df`
166
  groupby and include it in `tooltip`.
167
  - **Change mark type** (e.g. bars): replace `mark_line(point=True)` with `mark_bar()`.
168
+ - **Restrict the legend to the methods actually plotted**: use `_method_color(...)` see
169
+ "Method color legends" below. Do **not** reach for `.configure_range(category={"scheme":
170
+ ...})`; that overrides the whole palette rather than filtering the legend, and breaks
171
+ the per-method color stability the app relies on.
172
 
173
+ There is no `if selected_benchmark != "All":` guard — a benchmark is always selected, so
174
+ this chart renders whenever there's at least one non-DNF row for it (see the gotcha above
175
+ for the exact condition).
176
 
177
  ---
178
 
 
182
 
183
  ### Target level radio
184
 
185
+ Search for `target_options = ["1.0", "1.1", "1.2"]` (a hardcoded list). If the benchmark
186
+ gains new target levels, update this list. A more robust approach is to derive them from
187
+ the data:
 
 
 
 
 
188
  ```python
189
  target_options = sorted(metric_store["rmse_target"].astype(str).unique().tolist())
190
  ```
191
 
192
  ### Per-method special-case charts
193
 
194
+ The HM (History Matching) failure-analysis chart (search for `if sel == "HM":`) is the
195
+ canonical example of a per-method detail chart:
196
 
197
  ```python
198
  if sel == "HM":
 
222
 
223
  The UQ leaderboard passes a `budget_store` DataFrame to `render_leaderboard()` via the
224
  `budget_store` keyword argument. This enables two additional chart sections rendered in
225
+ `src/common/leaderboard.py`, both **after** the main chart and **before** the leaderboard
226
+ table:
227
 
228
  1. **"Mean Iterations for Coverage vs Ensemble Size"** — line chart of `mean_iters` vs
229
+ ensemble size (`if budget_store is not None and not budget_store.empty:`). Uses the
230
+ same integer-tick pattern as the main chart. This section has its own emptiness check
231
+ (`if not all_ens_combos_iters.empty:`)it is independent of `leaderboard_df`, so it
232
+ can render even when the scored table/main chart are suppressed by an all-failed
233
+ selection.
234
+
235
+ 2. **"Failure Rate of Hitting Target ..."** grouped bar chart (`if show_failure_panel:`)
236
+ with `ensemble_size:O` on the x-axis (ordinal, not quantitative) and
237
+ `mean_failure_rate:Q` on y. The title reflects the currently selected target level.
238
+ Also independent of `leaderboard_df`.
239
+
240
+ To **add or modify a UQ chart**, edit the `if budget_store is not None` block. The
241
+ `budget_store` DataFrame has columns: `benchmark`, `algorithm_type`, `abbreviation`,
242
+ `family`, `uq_target`, `ensemble_size`, `mean_budget`, `mean_iters`, `failure_count`,
243
+ `failure_rate`, `n_seeds`.
244
 
245
  **Bar chart integer x-axis:** Use `ensemble_size:O` (ordinal) with an explicit sort list
246
  and `axis=alt.Axis(labelAngle=0)` to suppress diagonal labels:
 
274
  needed, as long as it is appended to the end of `KNOWN_METHODS`. The palette cycles every
275
  10 methods.
276
 
277
+ ### Method color legends: scale stability vs. legend filtering
278
+
279
+ `src/common/leaderboard.py` defines a `_method_color(present_abbrevs: list[str])` helper
280
+ — **use it (or its pattern) for every method-colored chart**, rather than building
281
+ `alt.Color(...)` by hand:
282
+
283
+ ```python
284
+ def _method_color(present_abbrevs: list[str]) -> alt.Color:
285
+ return alt.Color(
286
+ "abbreviation:N",
287
+ title="Method",
288
+ scale=alt.Scale(domain=list(METHOD_COLORS.keys()), range=list(METHOD_COLORS.values())),
289
+ legend=alt.Legend(values=sorted(present_abbrevs)),
290
+ )
291
+ ```
292
+
293
+ This encodes two *separate* requirements that are easy to conflate:
294
+
295
+ - **The `scale` domain/range must always be the full registry** (`METHOD_COLORS.keys()`
296
+ / `.values()`), never a filtered subset. This is what keeps a given method's color
297
+ stable across every chart, benchmark, and page — if you filtered the scale domain to
298
+ "just the methods in this chart," a method could get a different color depending on
299
+ which benchmark happens to be selected.
300
+ - **The `legend` should be filtered to just the methods present in *this* chart's data**
301
+ (`legend=alt.Legend(values=...)`). Without this, Vega-Lite lists every method in the
302
+ domain in the legend regardless of whether it appears in the plotted data — on a
303
+ benchmark with 3 methods, the legend would still show all 9+ registered methods.
304
+
305
+ When calling `_method_color()`, compute `present_abbrevs` from the actual DataFrame(s)
306
+ plotted in that chart section (e.g. `all_ens_combos["abbreviation"].unique()`), **not**
307
+ from the global "Methods to display in charts" multiselect. A method can be selected in
308
+ the multiselect but have no data for the current benchmark/target combination — including
309
+ it in the legend anyway would be misleading. Each chart section (main chart, UQ iters
310
+ chart, failure-rate chart) computes and passes its own `present_abbrevs`, since the set
311
+ of methods with data can differ section to section (e.g. a method might have forward-run
312
+ data but no coverage/iters data).
313
+
314
+ If a layered chart (`alt.layer(...)`) has multiple encode calls sharing one color field
315
+ (e.g. the main chart's success-line layer, fail-cross layer, and single-ensemble rule
316
+ layer), pass the **same** `_method_color(...)` result (same `present_abbrevs`) to all of
317
+ them — Vega-Lite needs matching legend/scale specs across layers to merge them into one
318
+ legend instead of drawing duplicates.
319
+
320
+ ### How to import shared colors in a new page/file
321
 
322
  ```python
323
  try:
324
  from common.method_registry import METHOD_COLORS
325
  except ModuleNotFoundError:
326
  from src.common.method_registry import METHOD_COLORS
 
 
 
 
 
 
327
  ```
328
 
329
+ `_method_color()` itself currently lives in `leaderboard.py` (not `method_registry.py`)
330
+ and is private (`_`-prefixed). If a future page outside `leaderboard.py` needs
331
+ method-colored charts with the same scale-stable/legend-filtered behavior, promote
332
+ `_method_color()` to `method_registry.py` (dropping the underscore) rather than
333
+ duplicating the function — don't hand-roll a second copy.
334
 
335
  ### General plotting conventions
336
 
 
339
 
340
  | Concern | Convention |
341
  |---|---|
342
+ | Method color | `_method_color(present_abbrevs)` (see above) — never a local palette or an unfiltered legend |
343
  | Benchmark color | `alt.Color("benchmark:N")` — Altair assigns automatically; no custom palette needed |
344
  | Full-width charts | `st.altair_chart(chart, use_container_width=True)` — never `width="stretch"` |
345
  | Ensemble-size x-axis (quantitative) | `alt.Axis(values=ens_ticks, format="d")` — derive ticks from data, integer format |
 
356
  per-method surfaces).
357
  2. If adding a column: produce it in the aggregation AND add a `column_config` entry AND
358
  add it to `table_column_order`.
359
+ 3. If adding or reordering a chart section: check whether it should share the
360
+ `leaderboard_df`-emptiness gate (main chart) or have its own independent emptiness
361
+ check (UQ iters chart, failure panel) — see the gotcha under "Page layout" above.
362
+ 4. If the chart colors by `abbreviation`: use `_method_color(present_abbrevs)` computed
363
+ from that section's actual plotted data, not the global multiselect.
364
+ 5. Restart or clear Streamlit cache to pick up Python changes (code changes take effect on
365
  restart; cache only matters for data changes).
366
+ 6. Smoke-test with a benchmark/target combination where every run fails (all-DNF) to
367
+ confirm the warning/chart-suppression behavior still matches what you intended.
src/common/leaderboard.py CHANGED
@@ -33,6 +33,22 @@ _SUITABLE = "#009E73" # Okabe-Ito teal-green (colorblind-safe)
33
  _UNSUITABLE = "#C0392B" # dark red
34
  _UNTESTED = "#BDBDBD" # gray
35
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
36
  # Family display order in the leaderboard table (lower = earlier).
37
  # Unknown families fall back to 99 and appear at the end.
38
  _FAMILY_ORDER: dict[str, int] = {
@@ -186,7 +202,7 @@ def render_leaderboard(
186
  If ``True``, render a grouped-bar failure-rate chart below the main
187
  performance chart.
188
  show_scoring_modes:
189
- If ``True``, show scoring-mode radio controls in the expander.
190
  canonical_target_levels:
191
  If provided, the target-level selector always offers exactly these
192
  values (as strings) regardless of what is present in the data. Use
@@ -269,14 +285,6 @@ def render_leaderboard(
269
  valid_saved = [m for m in saved_methods if m in available_methods]
270
  st.session_state[k_methods] = valid_saved if valid_saved else available_methods
271
 
272
- # Color scale fixed to the full registry so every chart in the app uses the same
273
- # color per method regardless of which benchmark or page is shown.
274
- method_color = alt.Color(
275
- "abbreviation:N",
276
- title="Method",
277
- scale=alt.Scale(domain=list(METHOD_COLORS.keys()), range=list(METHOD_COLORS.values())),
278
- )
279
-
280
  def build_scored_table(input_df: pd.DataFrame, add_rank: bool = True) -> pd.DataFrame:
281
  ranking_source = input_df[input_df[target_str_col] == selected_target]
282
  if ranking_source.empty:
@@ -437,93 +445,52 @@ def render_leaderboard(
437
  f"and normalized ensemble-size score ({ensemble_weight:.0%})"
438
  )
439
 
440
- # Controls expander — always shown so users can change target even when the
441
- # current selection yields all failures.
442
- with st.expander("Scoring & Target Controls", expanded=False):
 
 
 
 
 
 
 
 
 
443
  st.radio(
444
- target_label,
445
- options=target_options,
446
  horizontal=True,
447
- key=k_target,
448
  )
449
 
450
- if show_scoring_modes:
451
- st.radio(
452
- "Scoring Method",
453
- options=scoring_options,
454
- horizontal=True,
455
- key=k_scoring,
 
 
 
 
 
456
  )
457
 
458
- if st.session_state.get(k_scoring, "Mean Forward Model Runs") == "Custom Blend":
459
- st.slider(
460
- "Blend Weight: Forward Runs vs Ensemble Size",
461
- min_value=0,
462
- max_value=100,
463
- step=5,
464
- key=k_weight,
465
- help=(
466
- "Higher forward-runs weight prioritizes fewer model evaluations; "
467
- "higher ensemble-size weight prioritizes smaller ensembles."
468
- ),
469
- )
470
-
471
- st.multiselect(
472
- "Methods to display in charts",
473
- options=available_methods,
474
- key=k_methods,
475
- )
476
 
477
  selected_methods = st.session_state.get(k_methods, available_methods)
478
  if not selected_methods:
479
  selected_methods = available_methods
480
 
481
- table_column_order = [
482
- "Placement",
483
- "abbreviation",
484
- "Method",
485
- "family",
486
- "Score",
487
- "Mean Forward Model Runs",
488
- "Minimum Forward Model Runs",
489
- "Mean Failure Rate (%)",
490
- "Optimal Ensemble Size",
491
- "Ensemble Sizes Used",
492
- ]
493
-
494
- if leaderboard_df.empty:
495
- st.warning(
496
- "All runs failed to reach the target at this selection. "
497
- "See the failure rate chart below."
498
- if show_failure_panel
499
- else "No rows available for the current benchmark/target selection."
500
- )
501
- else:
502
- st.subheader(f"Ranked Leaderboard — {selected_benchmark}")
503
- st.dataframe(
504
- leaderboard_df,
505
- hide_index=True,
506
- use_container_width=True,
507
- column_config={
508
- "Placement": st.column_config.TextColumn("Placement"),
509
- "family": st.column_config.TextColumn("Family"),
510
- "Method": st.column_config.TextColumn("Method"),
511
- "abbreviation": st.column_config.TextColumn("Abbrev."),
512
- "Mean Forward Model Runs": st.column_config.NumberColumn("Mean Forward Model Runs", format="%.4f"),
513
- "Minimum Forward Model Runs": st.column_config.NumberColumn("Minimum Forward Model Runs", format="%.4f"),
514
- "Score": st.column_config.ProgressColumn("Score (0-100)", min_value=0.0, max_value=100.0, format="%.1f"),
515
- "Optimal Ensemble Size": st.column_config.NumberColumn("Mean Optimal Ensemble Size", format="%.2f"),
516
- "Mean Failure Rate (%)": st.column_config.NumberColumn("Mean Failure Rate (%)", format="%.1f"),
517
- "Ensemble Sizes Used": st.column_config.NumberColumn("Ensemble Sizes Used", format="%d"),
518
- },
519
- column_order=table_column_order,
520
- )
521
-
522
- st.info(
523
- f"Score is a normalized 0–100 ranking based on **{score_basis}**. "
524
- "Values are computed from all ensemble sizes after averaging over random seeds."
525
- )
526
 
 
 
527
  st.subheader("Mean Forward Model Runs vs Ensemble Size")
528
  chart_source = filtered[filtered[target_str_col] == selected_target]
529
  chart_source = chart_source[chart_source["abbreviation"].isin(selected_methods)]
@@ -533,6 +500,7 @@ def render_leaderboard(
533
 
534
  all_ens_combos = chart_source[["abbreviation", "ensemble_size"]].drop_duplicates()
535
  ens_ticks = sorted(all_ens_combos["ensemble_size"].unique().tolist()) if not all_ens_combos.empty else []
 
536
 
537
  if not chart_df.empty:
538
  _ok = chart_df[["abbreviation", "ensemble_size"]].assign(_ok=True)
@@ -554,7 +522,7 @@ def render_leaderboard(
554
  axis=alt.Axis(values=ens_ticks, format="d"),
555
  ),
556
  y=alt.Y("mean_forward_runs:Q", title="Mean Forward Model Runs"),
557
- color=method_color,
558
  tooltip=["abbreviation", "ensemble_size", alt.Tooltip("mean_forward_runs:Q", format=".4f")],
559
  )
560
  )
@@ -574,7 +542,7 @@ def render_leaderboard(
574
  axis=alt.Axis(values=ens_ticks, format="d"),
575
  ),
576
  y=y_fwd,
577
- color=method_color,
578
  tooltip=[
579
  alt.Tooltip("abbreviation:N", title="Method"),
580
  alt.Tooltip("ensemble_size:Q", title="Ensemble Size"),
@@ -603,7 +571,7 @@ def render_leaderboard(
603
  "ensemble_size:Q",
604
  axis=alt.Axis(values=ens_ticks, format="d"),
605
  ),
606
- color=method_color,
607
  tooltip=[
608
  alt.Tooltip("abbreviation:N", title="Method"),
609
  alt.Tooltip("ensemble_size:Q", title="Ensemble Size"),
@@ -630,6 +598,7 @@ def render_leaderboard(
630
  all_ens_combos_iters = bf[["abbreviation", "ensemble_size"]].drop_duplicates()
631
  if not all_ens_combos_iters.empty:
632
  iters_ticks = sorted(all_ens_combos_iters["ensemble_size"].unique().tolist())
 
633
 
634
  if not iters_df.empty:
635
  _ok_iters = iters_df[["abbreviation", "ensemble_size"]].assign(_ok=True)
@@ -652,7 +621,7 @@ def render_leaderboard(
652
  axis=alt.Axis(values=iters_ticks, format="d"),
653
  ),
654
  y=alt.Y("mean_iters:Q", title="Mean Iterations"),
655
- color=method_color,
656
  tooltip=[
657
  alt.Tooltip("abbreviation:N", title="Method"),
658
  alt.Tooltip("ensemble_size:Q", title="Ensemble Size"),
@@ -676,7 +645,7 @@ def render_leaderboard(
676
  axis=alt.Axis(values=iters_ticks, format="d"),
677
  ),
678
  y=y_iters,
679
- color=method_color,
680
  tooltip=[
681
  alt.Tooltip("abbreviation:N", title="Method"),
682
  alt.Tooltip("ensemble_size:Q", title="Ensemble Size"),
@@ -715,7 +684,7 @@ def render_leaderboard(
715
  title="Failure Rate (%)",
716
  scale=alt.Scale(domain=[0, 100]),
717
  ),
718
- color=method_color,
719
  tooltip=[
720
  alt.Tooltip("abbreviation:N", title="Method"),
721
  alt.Tooltip("ensemble_size:O", title="Ensemble Size"),
@@ -730,6 +699,55 @@ def render_leaderboard(
730
  )
731
  st.altair_chart(failure_chart + ceiling_line, use_container_width=True)
732
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
733
  st.caption("Top 3 are shown as podium spots; remaining methods are directly comparable via normalized score.")
734
 
735
  if raw_page is not None:
 
33
  _UNSUITABLE = "#C0392B" # dark red
34
  _UNTESTED = "#BDBDBD" # gray
35
 
36
+
37
+ def _method_color(present_abbrevs: list[str]) -> alt.Color:
38
+ """Color encoding for the ``abbreviation`` field.
39
+
40
+ The scale domain/range is always the full method registry so a given
41
+ method keeps the same color across every chart and page. The legend,
42
+ however, is restricted to *present_abbrevs* so it only lists methods
43
+ actually plotted in this chart section rather than every known method.
44
+ """
45
+ return alt.Color(
46
+ "abbreviation:N",
47
+ title="Method",
48
+ scale=alt.Scale(domain=list(METHOD_COLORS.keys()), range=list(METHOD_COLORS.values())),
49
+ legend=alt.Legend(values=sorted(present_abbrevs)),
50
+ )
51
+
52
  # Family display order in the leaderboard table (lower = earlier).
53
  # Unknown families fall back to 99 and appear at the end.
54
  _FAMILY_ORDER: dict[str, int] = {
 
202
  If ``True``, render a grouped-bar failure-rate chart below the main
203
  performance chart.
204
  show_scoring_modes:
205
+ If ``True``, show scoring-mode radio controls.
206
  canonical_target_levels:
207
  If provided, the target-level selector always offers exactly these
208
  values (as strings) regardless of what is present in the data. Use
 
285
  valid_saved = [m for m in saved_methods if m in available_methods]
286
  st.session_state[k_methods] = valid_saved if valid_saved else available_methods
287
 
 
 
 
 
 
 
 
 
288
  def build_scored_table(input_df: pd.DataFrame, add_rank: bool = True) -> pd.DataFrame:
289
  ranking_source = input_df[input_df[target_str_col] == selected_target]
290
  if ranking_source.empty:
 
445
  f"and normalized ensemble-size score ({ensemble_weight:.0%})"
446
  )
447
 
448
+ # Controls — always visible (not collapsed behind a dropdown) so users can
449
+ # change target/scoring/methods even when the current selection yields all
450
+ # failures.
451
+ st.subheader("Scoring & Target Controls")
452
+ st.radio(
453
+ target_label,
454
+ options=target_options,
455
+ horizontal=True,
456
+ key=k_target,
457
+ )
458
+
459
+ if show_scoring_modes:
460
  st.radio(
461
+ "Scoring Method",
462
+ options=scoring_options,
463
  horizontal=True,
464
+ key=k_scoring,
465
  )
466
 
467
+ if st.session_state.get(k_scoring, "Mean Forward Model Runs") == "Custom Blend":
468
+ st.slider(
469
+ "Blend Weight: Forward Runs vs Ensemble Size",
470
+ min_value=0,
471
+ max_value=100,
472
+ step=5,
473
+ key=k_weight,
474
+ help=(
475
+ "Higher forward-runs weight prioritizes fewer model evaluations; "
476
+ "higher ensemble-size weight prioritizes smaller ensembles."
477
+ ),
478
  )
479
 
480
+ st.multiselect(
481
+ "Methods to display in charts",
482
+ options=available_methods,
483
+ key=k_methods,
484
+ )
 
 
 
 
 
 
 
 
 
 
 
 
 
485
 
486
  selected_methods = st.session_state.get(k_methods, available_methods)
487
  if not selected_methods:
488
  selected_methods = available_methods
489
 
490
+ st.divider()
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
491
 
492
+ # --- Charts (rendered above the leaderboard table, below the controls) ---
493
+ if not leaderboard_df.empty:
494
  st.subheader("Mean Forward Model Runs vs Ensemble Size")
495
  chart_source = filtered[filtered[target_str_col] == selected_target]
496
  chart_source = chart_source[chart_source["abbreviation"].isin(selected_methods)]
 
500
 
501
  all_ens_combos = chart_source[["abbreviation", "ensemble_size"]].drop_duplicates()
502
  ens_ticks = sorted(all_ens_combos["ensemble_size"].unique().tolist()) if not all_ens_combos.empty else []
503
+ main_color = _method_color(all_ens_combos["abbreviation"].unique().tolist())
504
 
505
  if not chart_df.empty:
506
  _ok = chart_df[["abbreviation", "ensemble_size"]].assign(_ok=True)
 
522
  axis=alt.Axis(values=ens_ticks, format="d"),
523
  ),
524
  y=alt.Y("mean_forward_runs:Q", title="Mean Forward Model Runs"),
525
+ color=main_color,
526
  tooltip=["abbreviation", "ensemble_size", alt.Tooltip("mean_forward_runs:Q", format=".4f")],
527
  )
528
  )
 
542
  axis=alt.Axis(values=ens_ticks, format="d"),
543
  ),
544
  y=y_fwd,
545
+ color=main_color,
546
  tooltip=[
547
  alt.Tooltip("abbreviation:N", title="Method"),
548
  alt.Tooltip("ensemble_size:Q", title="Ensemble Size"),
 
571
  "ensemble_size:Q",
572
  axis=alt.Axis(values=ens_ticks, format="d"),
573
  ),
574
+ color=main_color,
575
  tooltip=[
576
  alt.Tooltip("abbreviation:N", title="Method"),
577
  alt.Tooltip("ensemble_size:Q", title="Ensemble Size"),
 
598
  all_ens_combos_iters = bf[["abbreviation", "ensemble_size"]].drop_duplicates()
599
  if not all_ens_combos_iters.empty:
600
  iters_ticks = sorted(all_ens_combos_iters["ensemble_size"].unique().tolist())
601
+ iters_color = _method_color(all_ens_combos_iters["abbreviation"].unique().tolist())
602
 
603
  if not iters_df.empty:
604
  _ok_iters = iters_df[["abbreviation", "ensemble_size"]].assign(_ok=True)
 
621
  axis=alt.Axis(values=iters_ticks, format="d"),
622
  ),
623
  y=alt.Y("mean_iters:Q", title="Mean Iterations"),
624
+ color=iters_color,
625
  tooltip=[
626
  alt.Tooltip("abbreviation:N", title="Method"),
627
  alt.Tooltip("ensemble_size:Q", title="Ensemble Size"),
 
645
  axis=alt.Axis(values=iters_ticks, format="d"),
646
  ),
647
  y=y_iters,
648
+ color=iters_color,
649
  tooltip=[
650
  alt.Tooltip("abbreviation:N", title="Method"),
651
  alt.Tooltip("ensemble_size:Q", title="Ensemble Size"),
 
684
  title="Failure Rate (%)",
685
  scale=alt.Scale(domain=[0, 100]),
686
  ),
687
+ color=_method_color(failure_df["abbreviation"].unique().tolist()),
688
  tooltip=[
689
  alt.Tooltip("abbreviation:N", title="Method"),
690
  alt.Tooltip("ensemble_size:O", title="Ensemble Size"),
 
699
  )
700
  st.altair_chart(failure_chart + ceiling_line, use_container_width=True)
701
 
702
+ st.divider()
703
+
704
+ # --- Leaderboard table (rendered below the charts) ---
705
+ table_column_order = [
706
+ "Placement",
707
+ "abbreviation",
708
+ "Method",
709
+ "family",
710
+ "Score",
711
+ "Mean Forward Model Runs",
712
+ "Minimum Forward Model Runs",
713
+ "Mean Failure Rate (%)",
714
+ "Optimal Ensemble Size",
715
+ "Ensemble Sizes Used",
716
+ ]
717
+
718
+ if leaderboard_df.empty:
719
+ st.warning(
720
+ "All runs failed to reach the target at this selection. "
721
+ "See the failure rate chart above."
722
+ if show_failure_panel
723
+ else "No rows available for the current benchmark/target selection."
724
+ )
725
+ else:
726
+ st.subheader(f"Ranked Leaderboard — {selected_benchmark}")
727
+ st.dataframe(
728
+ leaderboard_df,
729
+ hide_index=True,
730
+ use_container_width=True,
731
+ column_config={
732
+ "Placement": st.column_config.TextColumn("Placement"),
733
+ "family": st.column_config.TextColumn("Family"),
734
+ "Method": st.column_config.TextColumn("Method"),
735
+ "abbreviation": st.column_config.TextColumn("Abbrev."),
736
+ "Mean Forward Model Runs": st.column_config.NumberColumn("Mean Forward Model Runs", format="%.4f"),
737
+ "Minimum Forward Model Runs": st.column_config.NumberColumn("Minimum Forward Model Runs", format="%.4f"),
738
+ "Score": st.column_config.ProgressColumn("Score (0-100)", min_value=0.0, max_value=100.0, format="%.1f"),
739
+ "Optimal Ensemble Size": st.column_config.NumberColumn("Mean Optimal Ensemble Size", format="%.2f"),
740
+ "Mean Failure Rate (%)": st.column_config.NumberColumn("Mean Failure Rate (%)", format="%.1f"),
741
+ "Ensemble Sizes Used": st.column_config.NumberColumn("Ensemble Sizes Used", format="%d"),
742
+ },
743
+ column_order=table_column_order,
744
+ )
745
+
746
+ st.info(
747
+ f"Score is a normalized 0–100 ranking based on **{score_basis}**. "
748
+ "Values are computed from all ensemble sizes after averaging over random seeds."
749
+ )
750
+
751
  st.caption("Top 3 are shown as podium spots; remaining methods are directly comparable via normalized score.")
752
 
753
  if raw_page is not None: