Improved layouts
#6
by odunbar - opened
.claude/skills/leaderboard-manager/references/manage-display.md
CHANGED
|
@@ -6,34 +6,76 @@ renderer** (`render_leaderboard()` in `src/common/leaderboard.py`). Any change t
|
|
| 6 |
function affects both boards simultaneously. Method-specific charts live in
|
| 7 |
`src/pages/MethodDetails.py`.
|
| 8 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 9 |
---
|
| 10 |
|
| 11 |
## Leaderboard table
|
| 12 |
|
| 13 |
### Column visibility and order
|
| 14 |
|
| 15 |
-
|
| 16 |
-
|
| 17 |
-
|
| 18 |
-
|
| 19 |
-
|
| 20 |
-
|
| 21 |
-
|
| 22 |
-
|
| 23 |
-
|
| 24 |
-
|
| 25 |
-
["Placement", "abbreviation", "Method", "family", "Score",
|
| 26 |
-
"Mean Forward Model Runs", "Minimum Forward Model Runs",
|
| 27 |
-
"Optimal Ensemble Size", "Targets Used", "Ensemble Sizes Used", "Benchmarks Used"]
|
| 28 |
-
```
|
| 29 |
-
|
| 30 |
-
To **show or hide** a column, add/remove it from the relevant list.
|
| 31 |
To **reorder** columns, rearrange entries within the list.
|
| 32 |
|
| 33 |
### Column labels and formatting
|
| 34 |
|
| 35 |
-
Column display names and number formats are set in `st.dataframe(... column_config=...)`
|
| 36 |
-
|
| 37 |
|
| 38 |
```python
|
| 39 |
"Mean Forward Model Runs": st.column_config.NumberColumn(
|
|
@@ -48,8 +90,8 @@ To **rename** a displayed column header, change the string argument (second posi
|
|
| 48 |
in the `column_config` entry. The dict key must still match the DataFrame column name.
|
| 49 |
|
| 50 |
To **add a new column to the table**, you also need to produce it in `build_scored_table()`
|
| 51 |
-
|
| 52 |
-
|
| 53 |
|
| 54 |
### Adding `failure_rate` to the leaderboard table
|
| 55 |
|
|
@@ -64,7 +106,8 @@ To add it:
|
|
| 64 |
|
| 65 |
## Scoring modes
|
| 66 |
|
| 67 |
-
The four scoring modes are defined in `scoring_options` (
|
|
|
|
| 68 |
```python
|
| 69 |
scoring_options = [
|
| 70 |
"Mean Forward Model Runs",
|
|
@@ -74,24 +117,27 @@ scoring_options = [
|
|
| 74 |
]
|
| 75 |
```
|
| 76 |
|
| 77 |
-
Each mode maps to a sort key and a score column assignment
|
| 78 |
-
|
| 79 |
-
|
| 80 |
-
`sort_columns`, and `ascending`.
|
| 81 |
|
| 82 |
-
The "Custom Blend" slider
|
| 83 |
-
|
|
|
|
|
|
|
| 84 |
|
| 85 |
---
|
| 86 |
|
| 87 |
## Leaderboard line chart
|
| 88 |
|
| 89 |
-
The chart that appears
|
| 90 |
-
|
| 91 |
-
|
| 92 |
|
| 93 |
```python
|
| 94 |
-
ens_ticks = sorted(
|
|
|
|
| 95 |
chart = (
|
| 96 |
alt.Chart(chart_df)
|
| 97 |
.mark_line(point=True)
|
|
@@ -102,7 +148,7 @@ chart = (
|
|
| 102 |
axis=alt.Axis(values=ens_ticks, format="d"), # integer ticks only
|
| 103 |
),
|
| 104 |
y=alt.Y("mean_forward_runs:Q", title="Mean Forward Model Runs"),
|
| 105 |
-
color=
|
| 106 |
tooltip=["abbreviation", "ensemble_size",
|
| 107 |
alt.Tooltip("mean_forward_runs:Q", format=".4f")],
|
| 108 |
)
|
|
@@ -119,11 +165,14 @@ Common changes:
|
|
| 119 |
- **Add failure rate as a second Y-axis or tooltip**: add `failure_rate` to `chart_df`
|
| 120 |
groupby and include it in `tooltip`.
|
| 121 |
- **Change mark type** (e.g. bars): replace `mark_line(point=True)` with `mark_bar()`.
|
| 122 |
-
- **
|
| 123 |
-
|
|
|
|
|
|
|
| 124 |
|
| 125 |
-
|
| 126 |
-
|
|
|
|
| 127 |
|
| 128 |
---
|
| 129 |
|
|
@@ -133,22 +182,17 @@ The per-method page (`src/pages/MethodDetails.py`) has its own controls.
|
|
| 133 |
|
| 134 |
### Target level radio
|
| 135 |
|
| 136 |
-
|
| 137 |
-
|
| 138 |
-
|
| 139 |
-
selected_target = st.radio("RMSE Target Level", options=target_options, horizontal=True)
|
| 140 |
-
```
|
| 141 |
-
|
| 142 |
-
These are **hardcoded strings**. If the benchmark gains new target levels, update this
|
| 143 |
-
list. A more robust approach is to derive them from the data:
|
| 144 |
```python
|
| 145 |
target_options = sorted(metric_store["rmse_target"].astype(str).unique().tolist())
|
| 146 |
```
|
| 147 |
|
| 148 |
### Per-method special-case charts
|
| 149 |
|
| 150 |
-
The HM (History Matching) failure-analysis chart (
|
| 151 |
-
example of a per-method detail chart:
|
| 152 |
|
| 153 |
```python
|
| 154 |
if sel == "HM":
|
|
@@ -178,19 +222,25 @@ encoding, change the `.encode()` arguments analogously to the leaderboard chart
|
|
| 178 |
|
| 179 |
The UQ leaderboard passes a `budget_store` DataFrame to `render_leaderboard()` via the
|
| 180 |
`budget_store` keyword argument. This enables two additional chart sections rendered in
|
| 181 |
-
`src/common/leaderboard.py` after the main
|
|
|
|
| 182 |
|
| 183 |
1. **"Mean Iterations for Coverage vs Ensemble Size"** — line chart of `mean_iters` vs
|
| 184 |
-
ensemble size
|
| 185 |
-
|
| 186 |
-
|
| 187 |
-
|
| 188 |
-
|
| 189 |
-
|
| 190 |
-
|
| 191 |
-
|
| 192 |
-
`
|
| 193 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 194 |
|
| 195 |
**Bar chart integer x-axis:** Use `ensemble_size:O` (ordinal) with an explicit sort list
|
| 196 |
and `axis=alt.Axis(labelAngle=0)` to suppress diagonal labels:
|
|
@@ -224,23 +274,63 @@ When **adding a new method**, its color is assigned automatically — no manual
|
|
| 224 |
needed, as long as it is appended to the end of `KNOWN_METHODS`. The palette cycles every
|
| 225 |
10 methods.
|
| 226 |
|
| 227 |
-
###
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 228 |
|
| 229 |
```python
|
| 230 |
try:
|
| 231 |
from common.method_registry import METHOD_COLORS
|
| 232 |
except ModuleNotFoundError:
|
| 233 |
from src.common.method_registry import METHOD_COLORS
|
| 234 |
-
|
| 235 |
-
method_color = alt.Color(
|
| 236 |
-
"abbreviation:N",
|
| 237 |
-
title="Method",
|
| 238 |
-
scale=alt.Scale(domain=list(METHOD_COLORS.keys()), range=list(METHOD_COLORS.values())),
|
| 239 |
-
)
|
| 240 |
```
|
| 241 |
|
| 242 |
-
|
| 243 |
-
|
|
|
|
|
|
|
|
|
|
| 244 |
|
| 245 |
### General plotting conventions
|
| 246 |
|
|
@@ -249,7 +339,7 @@ new Altair chart in the app:
|
|
| 249 |
|
| 250 |
| Concern | Convention |
|
| 251 |
|---|---|
|
| 252 |
-
| Method color | `
|
| 253 |
| Benchmark color | `alt.Color("benchmark:N")` — Altair assigns automatically; no custom palette needed |
|
| 254 |
| Full-width charts | `st.altair_chart(chart, use_container_width=True)` — never `width="stretch"` |
|
| 255 |
| Ensemble-size x-axis (quantitative) | `alt.Axis(values=ens_ticks, format="d")` — derive ticks from data, integer format |
|
|
@@ -266,7 +356,12 @@ new Altair chart in the app:
|
|
| 266 |
per-method surfaces).
|
| 267 |
2. If adding a column: produce it in the aggregation AND add a `column_config` entry AND
|
| 268 |
add it to `table_column_order`.
|
| 269 |
-
3.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 270 |
restart; cache only matters for data changes).
|
| 271 |
-
|
| 272 |
-
|
|
|
|
| 6 |
function affects both boards simultaneously. Method-specific charts live in
|
| 7 |
`src/pages/MethodDetails.py`.
|
| 8 |
|
| 9 |
+
**A note on line numbers:** this file used to cite specific line numbers (e.g. "line 296").
|
| 10 |
+
They drift out of date after almost any edit to `leaderboard.py` — a single reordering
|
| 11 |
+
task can shift everything below it by 50+ lines. Below, locations are given as
|
| 12 |
+
**function/variable/string anchors** you can `grep` for instead — these survive
|
| 13 |
+
refactors. If you add a new durable line-number reference anyway, expect to fix it again
|
| 14 |
+
next time someone edits the file.
|
| 15 |
+
|
| 16 |
+
---
|
| 17 |
+
|
| 18 |
+
## Page layout (top to bottom)
|
| 19 |
+
|
| 20 |
+
`render_leaderboard()` renders, in order:
|
| 21 |
+
|
| 22 |
+
1. Suitability table (`_render_suitability_table()`)
|
| 23 |
+
2. Benchmark selector (`st.selectbox("Benchmark", ...)`) — always a single benchmark;
|
| 24 |
+
there is **no "All benchmarks" option**. Every chart and the table below operate on
|
| 25 |
+
whatever `selected_benchmark` is currently chosen.
|
| 26 |
+
3. **Controls — always visible**, not collapsed. Look for `st.subheader("Scoring &
|
| 27 |
+
Target Controls")`: target-level radio, scoring-mode radio (+ blend-weight slider
|
| 28 |
+
when "Custom Blend" is selected), and the "Methods to display in charts" multiselect.
|
| 29 |
+
These used to be wrapped in `st.expander(..., expanded=False)`; that was removed so
|
| 30 |
+
users don't have to open a dropdown to see or change the target level.
|
| 31 |
+
4. **Charts** — main "Mean Forward Model Runs vs Ensemble Size" chart, then (UQ only)
|
| 32 |
+
the "Mean Iterations for Coverage" chart, then (if `show_failure_panel=True`) the
|
| 33 |
+
failure-rate bar chart.
|
| 34 |
+
5. **Leaderboard table** — `st.dataframe(leaderboard_df, ...)`.
|
| 35 |
+
|
| 36 |
+
Charts render above the table and below the controls. If you need to reorder these
|
| 37 |
+
sections again, see the gotcha below before moving the main chart.
|
| 38 |
+
|
| 39 |
+
### Gotcha: the main chart is gated by `leaderboard_df`, not by its own emptiness check
|
| 40 |
+
|
| 41 |
+
The main forward-model-runs chart is wrapped in `if not leaderboard_df.empty:` — the
|
| 42 |
+
same DataFrame the table renders from — rather than an independent check on its own
|
| 43 |
+
`chart_df`/`fail_df`. This is deliberate: `leaderboard_df` (from `build_scored_table()`)
|
| 44 |
+
is empty in two distinct cases:
|
| 45 |
+
|
| 46 |
+
- No rows at all for the current benchmark/target (nothing was ever run).
|
| 47 |
+
- Every seed failed (`metric` is NaN for all rows) — a real "DNF" case.
|
| 48 |
+
|
| 49 |
+
In the DNF case, the chart is intentionally suppressed too — the user sees a warning
|
| 50 |
+
("All runs failed...") instead of a near-empty line chart with only fail-crosses at
|
| 51 |
+
y=0. If you're adding a new chart section anywhere near the main chart, decide
|
| 52 |
+
deliberately whether it should share this gate or have its own (the UQ iters chart and
|
| 53 |
+
the failure-rate chart do **not** share it — they have their own internal emptiness
|
| 54 |
+
checks, e.g. `if not all_ens_combos_iters.empty:`, so they can render even when the
|
| 55 |
+
scored table is empty).
|
| 56 |
+
|
| 57 |
---
|
| 58 |
|
| 59 |
## Leaderboard table
|
| 60 |
|
| 61 |
### Column visibility and order
|
| 62 |
|
| 63 |
+
Search for `table_column_order = [` in `src/common/leaderboard.py` — there is a single
|
| 64 |
+
list (not one-per-view):
|
| 65 |
+
```python
|
| 66 |
+
table_column_order = [
|
| 67 |
+
"Placement", "abbreviation", "Method", "family", "Score",
|
| 68 |
+
"Mean Forward Model Runs", "Minimum Forward Model Runs",
|
| 69 |
+
"Mean Failure Rate (%)", "Optimal Ensemble Size", "Ensemble Sizes Used",
|
| 70 |
+
]
|
| 71 |
+
```
|
| 72 |
+
To **show or hide** a column, add/remove it from this list.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 73 |
To **reorder** columns, rearrange entries within the list.
|
| 74 |
|
| 75 |
### Column labels and formatting
|
| 76 |
|
| 77 |
+
Column display names and number formats are set in the `st.dataframe(... column_config=...)`
|
| 78 |
+
call right after `table_column_order` is used. Each column has an entry like:
|
| 79 |
|
| 80 |
```python
|
| 81 |
"Mean Forward Model Runs": st.column_config.NumberColumn(
|
|
|
|
| 90 |
in the `column_config` entry. The dict key must still match the DataFrame column name.
|
| 91 |
|
| 92 |
To **add a new column to the table**, you also need to produce it in `build_scored_table()`
|
| 93 |
+
— add it to a `.agg()` call or compute it after the groupby, then add a `column_config`
|
| 94 |
+
entry and include it in `table_column_order`.
|
| 95 |
|
| 96 |
### Adding `failure_rate` to the leaderboard table
|
| 97 |
|
|
|
|
| 106 |
|
| 107 |
## Scoring modes
|
| 108 |
|
| 109 |
+
The four scoring modes are defined in `scoring_options` (a list literal near the top of
|
| 110 |
+
`render_leaderboard()`, right after `target_options` is computed):
|
| 111 |
```python
|
| 112 |
scoring_options = [
|
| 113 |
"Mean Forward Model Runs",
|
|
|
|
| 117 |
]
|
| 118 |
```
|
| 119 |
|
| 120 |
+
Each mode maps to a sort key and a score column assignment inside `build_scored_table()`
|
| 121 |
+
— search for the `if scoring_mode == "Mean Forward Model Runs": ... elif ...` chain. To
|
| 122 |
+
**add a new scoring mode**, add its name to `scoring_options` and add an `elif` branch in
|
| 123 |
+
that chain that sets `scored_df["Score"]`, `sort_columns`, and `ascending`.
|
| 124 |
|
| 125 |
+
The "Custom Blend" slider (`st.slider("Blend Weight: Forward Runs vs Ensemble Size", ...)`)
|
| 126 |
+
is keyed to the session-state key `k_weight` and only rendered when
|
| 127 |
+
`st.session_state[k_scoring] == "Custom Blend"` — no change needed when adding a
|
| 128 |
+
non-blend mode. It lives in the always-visible controls section, not behind an expander.
|
| 129 |
|
| 130 |
---
|
| 131 |
|
| 132 |
## Leaderboard line chart
|
| 133 |
|
| 134 |
+
The chart that appears above the table ("Mean Forward Model Runs vs Ensemble Size") is
|
| 135 |
+
built right after `st.subheader("Mean Forward Model Runs vs Ensemble Size")`, inside the
|
| 136 |
+
`if not leaderboard_df.empty:` guard described in the gotcha above:
|
| 137 |
|
| 138 |
```python
|
| 139 |
+
ens_ticks = sorted(all_ens_combos["ensemble_size"].unique().tolist()) if not all_ens_combos.empty else []
|
| 140 |
+
main_color = _method_color(all_ens_combos["abbreviation"].unique().tolist())
|
| 141 |
chart = (
|
| 142 |
alt.Chart(chart_df)
|
| 143 |
.mark_line(point=True)
|
|
|
|
| 148 |
axis=alt.Axis(values=ens_ticks, format="d"), # integer ticks only
|
| 149 |
),
|
| 150 |
y=alt.Y("mean_forward_runs:Q", title="Mean Forward Model Runs"),
|
| 151 |
+
color=main_color,
|
| 152 |
tooltip=["abbreviation", "ensemble_size",
|
| 153 |
alt.Tooltip("mean_forward_runs:Q", format=".4f")],
|
| 154 |
)
|
|
|
|
| 165 |
- **Add failure rate as a second Y-axis or tooltip**: add `failure_rate` to `chart_df`
|
| 166 |
groupby and include it in `tooltip`.
|
| 167 |
- **Change mark type** (e.g. bars): replace `mark_line(point=True)` with `mark_bar()`.
|
| 168 |
+
- **Restrict the legend to the methods actually plotted**: use `_method_color(...)` — see
|
| 169 |
+
"Method color legends" below. Do **not** reach for `.configure_range(category={"scheme":
|
| 170 |
+
...})`; that overrides the whole palette rather than filtering the legend, and breaks
|
| 171 |
+
the per-method color stability the app relies on.
|
| 172 |
|
| 173 |
+
There is no `if selected_benchmark != "All":` guard — a benchmark is always selected, so
|
| 174 |
+
this chart renders whenever there's at least one non-DNF row for it (see the gotcha above
|
| 175 |
+
for the exact condition).
|
| 176 |
|
| 177 |
---
|
| 178 |
|
|
|
|
| 182 |
|
| 183 |
### Target level radio
|
| 184 |
|
| 185 |
+
Search for `target_options = ["1.0", "1.1", "1.2"]` (a hardcoded list). If the benchmark
|
| 186 |
+
gains new target levels, update this list. A more robust approach is to derive them from
|
| 187 |
+
the data:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 188 |
```python
|
| 189 |
target_options = sorted(metric_store["rmse_target"].astype(str).unique().tolist())
|
| 190 |
```
|
| 191 |
|
| 192 |
### Per-method special-case charts
|
| 193 |
|
| 194 |
+
The HM (History Matching) failure-analysis chart (search for `if sel == "HM":`) is the
|
| 195 |
+
canonical example of a per-method detail chart:
|
| 196 |
|
| 197 |
```python
|
| 198 |
if sel == "HM":
|
|
|
|
| 222 |
|
| 223 |
The UQ leaderboard passes a `budget_store` DataFrame to `render_leaderboard()` via the
|
| 224 |
`budget_store` keyword argument. This enables two additional chart sections rendered in
|
| 225 |
+
`src/common/leaderboard.py`, both **after** the main chart and **before** the leaderboard
|
| 226 |
+
table:
|
| 227 |
|
| 228 |
1. **"Mean Iterations for Coverage vs Ensemble Size"** — line chart of `mean_iters` vs
|
| 229 |
+
ensemble size (`if budget_store is not None and not budget_store.empty:`). Uses the
|
| 230 |
+
same integer-tick pattern as the main chart. This section has its own emptiness check
|
| 231 |
+
(`if not all_ens_combos_iters.empty:`) — it is independent of `leaderboard_df`, so it
|
| 232 |
+
can render even when the scored table/main chart are suppressed by an all-failed
|
| 233 |
+
selection.
|
| 234 |
+
|
| 235 |
+
2. **"Failure Rate of Hitting Target ..."** — grouped bar chart (`if show_failure_panel:`)
|
| 236 |
+
with `ensemble_size:O` on the x-axis (ordinal, not quantitative) and
|
| 237 |
+
`mean_failure_rate:Q` on y. The title reflects the currently selected target level.
|
| 238 |
+
Also independent of `leaderboard_df`.
|
| 239 |
+
|
| 240 |
+
To **add or modify a UQ chart**, edit the `if budget_store is not None` block. The
|
| 241 |
+
`budget_store` DataFrame has columns: `benchmark`, `algorithm_type`, `abbreviation`,
|
| 242 |
+
`family`, `uq_target`, `ensemble_size`, `mean_budget`, `mean_iters`, `failure_count`,
|
| 243 |
+
`failure_rate`, `n_seeds`.
|
| 244 |
|
| 245 |
**Bar chart integer x-axis:** Use `ensemble_size:O` (ordinal) with an explicit sort list
|
| 246 |
and `axis=alt.Axis(labelAngle=0)` to suppress diagonal labels:
|
|
|
|
| 274 |
needed, as long as it is appended to the end of `KNOWN_METHODS`. The palette cycles every
|
| 275 |
10 methods.
|
| 276 |
|
| 277 |
+
### Method color legends: scale stability vs. legend filtering
|
| 278 |
+
|
| 279 |
+
`src/common/leaderboard.py` defines a `_method_color(present_abbrevs: list[str])` helper
|
| 280 |
+
— **use it (or its pattern) for every method-colored chart**, rather than building
|
| 281 |
+
`alt.Color(...)` by hand:
|
| 282 |
+
|
| 283 |
+
```python
|
| 284 |
+
def _method_color(present_abbrevs: list[str]) -> alt.Color:
|
| 285 |
+
return alt.Color(
|
| 286 |
+
"abbreviation:N",
|
| 287 |
+
title="Method",
|
| 288 |
+
scale=alt.Scale(domain=list(METHOD_COLORS.keys()), range=list(METHOD_COLORS.values())),
|
| 289 |
+
legend=alt.Legend(values=sorted(present_abbrevs)),
|
| 290 |
+
)
|
| 291 |
+
```
|
| 292 |
+
|
| 293 |
+
This encodes two *separate* requirements that are easy to conflate:
|
| 294 |
+
|
| 295 |
+
- **The `scale` domain/range must always be the full registry** (`METHOD_COLORS.keys()`
|
| 296 |
+
/ `.values()`), never a filtered subset. This is what keeps a given method's color
|
| 297 |
+
stable across every chart, benchmark, and page — if you filtered the scale domain to
|
| 298 |
+
"just the methods in this chart," a method could get a different color depending on
|
| 299 |
+
which benchmark happens to be selected.
|
| 300 |
+
- **The `legend` should be filtered to just the methods present in *this* chart's data**
|
| 301 |
+
(`legend=alt.Legend(values=...)`). Without this, Vega-Lite lists every method in the
|
| 302 |
+
domain in the legend regardless of whether it appears in the plotted data — on a
|
| 303 |
+
benchmark with 3 methods, the legend would still show all 9+ registered methods.
|
| 304 |
+
|
| 305 |
+
When calling `_method_color()`, compute `present_abbrevs` from the actual DataFrame(s)
|
| 306 |
+
plotted in that chart section (e.g. `all_ens_combos["abbreviation"].unique()`), **not**
|
| 307 |
+
from the global "Methods to display in charts" multiselect. A method can be selected in
|
| 308 |
+
the multiselect but have no data for the current benchmark/target combination — including
|
| 309 |
+
it in the legend anyway would be misleading. Each chart section (main chart, UQ iters
|
| 310 |
+
chart, failure-rate chart) computes and passes its own `present_abbrevs`, since the set
|
| 311 |
+
of methods with data can differ section to section (e.g. a method might have forward-run
|
| 312 |
+
data but no coverage/iters data).
|
| 313 |
+
|
| 314 |
+
If a layered chart (`alt.layer(...)`) has multiple encode calls sharing one color field
|
| 315 |
+
(e.g. the main chart's success-line layer, fail-cross layer, and single-ensemble rule
|
| 316 |
+
layer), pass the **same** `_method_color(...)` result (same `present_abbrevs`) to all of
|
| 317 |
+
them — Vega-Lite needs matching legend/scale specs across layers to merge them into one
|
| 318 |
+
legend instead of drawing duplicates.
|
| 319 |
+
|
| 320 |
+
### How to import shared colors in a new page/file
|
| 321 |
|
| 322 |
```python
|
| 323 |
try:
|
| 324 |
from common.method_registry import METHOD_COLORS
|
| 325 |
except ModuleNotFoundError:
|
| 326 |
from src.common.method_registry import METHOD_COLORS
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 327 |
```
|
| 328 |
|
| 329 |
+
`_method_color()` itself currently lives in `leaderboard.py` (not `method_registry.py`)
|
| 330 |
+
and is private (`_`-prefixed). If a future page outside `leaderboard.py` needs
|
| 331 |
+
method-colored charts with the same scale-stable/legend-filtered behavior, promote
|
| 332 |
+
`_method_color()` to `method_registry.py` (dropping the underscore) rather than
|
| 333 |
+
duplicating the function — don't hand-roll a second copy.
|
| 334 |
|
| 335 |
### General plotting conventions
|
| 336 |
|
|
|
|
| 339 |
|
| 340 |
| Concern | Convention |
|
| 341 |
|---|---|
|
| 342 |
+
| Method color | `_method_color(present_abbrevs)` (see above) — never a local palette or an unfiltered legend |
|
| 343 |
| Benchmark color | `alt.Color("benchmark:N")` — Altair assigns automatically; no custom palette needed |
|
| 344 |
| Full-width charts | `st.altair_chart(chart, use_container_width=True)` — never `width="stretch"` |
|
| 345 |
| Ensemble-size x-axis (quantitative) | `alt.Axis(values=ens_ticks, format="d")` — derive ticks from data, integer format |
|
|
|
|
| 356 |
per-method surfaces).
|
| 357 |
2. If adding a column: produce it in the aggregation AND add a `column_config` entry AND
|
| 358 |
add it to `table_column_order`.
|
| 359 |
+
3. If adding or reordering a chart section: check whether it should share the
|
| 360 |
+
`leaderboard_df`-emptiness gate (main chart) or have its own independent emptiness
|
| 361 |
+
check (UQ iters chart, failure panel) — see the gotcha under "Page layout" above.
|
| 362 |
+
4. If the chart colors by `abbreviation`: use `_method_color(present_abbrevs)` computed
|
| 363 |
+
from that section's actual plotted data, not the global multiselect.
|
| 364 |
+
5. Restart or clear Streamlit cache to pick up Python changes (code changes take effect on
|
| 365 |
restart; cache only matters for data changes).
|
| 366 |
+
6. Smoke-test with a benchmark/target combination where every run fails (all-DNF) to
|
| 367 |
+
confirm the warning/chart-suppression behavior still matches what you intended.
|
src/common/leaderboard.py
CHANGED
|
@@ -33,6 +33,22 @@ _SUITABLE = "#009E73" # Okabe-Ito teal-green (colorblind-safe)
|
|
| 33 |
_UNSUITABLE = "#C0392B" # dark red
|
| 34 |
_UNTESTED = "#BDBDBD" # gray
|
| 35 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 36 |
# Family display order in the leaderboard table (lower = earlier).
|
| 37 |
# Unknown families fall back to 99 and appear at the end.
|
| 38 |
_FAMILY_ORDER: dict[str, int] = {
|
|
@@ -186,7 +202,7 @@ def render_leaderboard(
|
|
| 186 |
If ``True``, render a grouped-bar failure-rate chart below the main
|
| 187 |
performance chart.
|
| 188 |
show_scoring_modes:
|
| 189 |
-
If ``True``, show scoring-mode radio controls
|
| 190 |
canonical_target_levels:
|
| 191 |
If provided, the target-level selector always offers exactly these
|
| 192 |
values (as strings) regardless of what is present in the data. Use
|
|
@@ -269,14 +285,6 @@ def render_leaderboard(
|
|
| 269 |
valid_saved = [m for m in saved_methods if m in available_methods]
|
| 270 |
st.session_state[k_methods] = valid_saved if valid_saved else available_methods
|
| 271 |
|
| 272 |
-
# Color scale fixed to the full registry so every chart in the app uses the same
|
| 273 |
-
# color per method regardless of which benchmark or page is shown.
|
| 274 |
-
method_color = alt.Color(
|
| 275 |
-
"abbreviation:N",
|
| 276 |
-
title="Method",
|
| 277 |
-
scale=alt.Scale(domain=list(METHOD_COLORS.keys()), range=list(METHOD_COLORS.values())),
|
| 278 |
-
)
|
| 279 |
-
|
| 280 |
def build_scored_table(input_df: pd.DataFrame, add_rank: bool = True) -> pd.DataFrame:
|
| 281 |
ranking_source = input_df[input_df[target_str_col] == selected_target]
|
| 282 |
if ranking_source.empty:
|
|
@@ -437,93 +445,52 @@ def render_leaderboard(
|
|
| 437 |
f"and normalized ensemble-size score ({ensemble_weight:.0%})"
|
| 438 |
)
|
| 439 |
|
| 440 |
-
# Controls
|
| 441 |
-
# current selection yields all
|
| 442 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 443 |
st.radio(
|
| 444 |
-
|
| 445 |
-
options=
|
| 446 |
horizontal=True,
|
| 447 |
-
key=
|
| 448 |
)
|
| 449 |
|
| 450 |
-
if
|
| 451 |
-
st.
|
| 452 |
-
"
|
| 453 |
-
|
| 454 |
-
|
| 455 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 456 |
)
|
| 457 |
|
| 458 |
-
|
| 459 |
-
|
| 460 |
-
|
| 461 |
-
|
| 462 |
-
|
| 463 |
-
step=5,
|
| 464 |
-
key=k_weight,
|
| 465 |
-
help=(
|
| 466 |
-
"Higher forward-runs weight prioritizes fewer model evaluations; "
|
| 467 |
-
"higher ensemble-size weight prioritizes smaller ensembles."
|
| 468 |
-
),
|
| 469 |
-
)
|
| 470 |
-
|
| 471 |
-
st.multiselect(
|
| 472 |
-
"Methods to display in charts",
|
| 473 |
-
options=available_methods,
|
| 474 |
-
key=k_methods,
|
| 475 |
-
)
|
| 476 |
|
| 477 |
selected_methods = st.session_state.get(k_methods, available_methods)
|
| 478 |
if not selected_methods:
|
| 479 |
selected_methods = available_methods
|
| 480 |
|
| 481 |
-
|
| 482 |
-
"Placement",
|
| 483 |
-
"abbreviation",
|
| 484 |
-
"Method",
|
| 485 |
-
"family",
|
| 486 |
-
"Score",
|
| 487 |
-
"Mean Forward Model Runs",
|
| 488 |
-
"Minimum Forward Model Runs",
|
| 489 |
-
"Mean Failure Rate (%)",
|
| 490 |
-
"Optimal Ensemble Size",
|
| 491 |
-
"Ensemble Sizes Used",
|
| 492 |
-
]
|
| 493 |
-
|
| 494 |
-
if leaderboard_df.empty:
|
| 495 |
-
st.warning(
|
| 496 |
-
"All runs failed to reach the target at this selection. "
|
| 497 |
-
"See the failure rate chart below."
|
| 498 |
-
if show_failure_panel
|
| 499 |
-
else "No rows available for the current benchmark/target selection."
|
| 500 |
-
)
|
| 501 |
-
else:
|
| 502 |
-
st.subheader(f"Ranked Leaderboard — {selected_benchmark}")
|
| 503 |
-
st.dataframe(
|
| 504 |
-
leaderboard_df,
|
| 505 |
-
hide_index=True,
|
| 506 |
-
use_container_width=True,
|
| 507 |
-
column_config={
|
| 508 |
-
"Placement": st.column_config.TextColumn("Placement"),
|
| 509 |
-
"family": st.column_config.TextColumn("Family"),
|
| 510 |
-
"Method": st.column_config.TextColumn("Method"),
|
| 511 |
-
"abbreviation": st.column_config.TextColumn("Abbrev."),
|
| 512 |
-
"Mean Forward Model Runs": st.column_config.NumberColumn("Mean Forward Model Runs", format="%.4f"),
|
| 513 |
-
"Minimum Forward Model Runs": st.column_config.NumberColumn("Minimum Forward Model Runs", format="%.4f"),
|
| 514 |
-
"Score": st.column_config.ProgressColumn("Score (0-100)", min_value=0.0, max_value=100.0, format="%.1f"),
|
| 515 |
-
"Optimal Ensemble Size": st.column_config.NumberColumn("Mean Optimal Ensemble Size", format="%.2f"),
|
| 516 |
-
"Mean Failure Rate (%)": st.column_config.NumberColumn("Mean Failure Rate (%)", format="%.1f"),
|
| 517 |
-
"Ensemble Sizes Used": st.column_config.NumberColumn("Ensemble Sizes Used", format="%d"),
|
| 518 |
-
},
|
| 519 |
-
column_order=table_column_order,
|
| 520 |
-
)
|
| 521 |
-
|
| 522 |
-
st.info(
|
| 523 |
-
f"Score is a normalized 0–100 ranking based on **{score_basis}**. "
|
| 524 |
-
"Values are computed from all ensemble sizes after averaging over random seeds."
|
| 525 |
-
)
|
| 526 |
|
|
|
|
|
|
|
| 527 |
st.subheader("Mean Forward Model Runs vs Ensemble Size")
|
| 528 |
chart_source = filtered[filtered[target_str_col] == selected_target]
|
| 529 |
chart_source = chart_source[chart_source["abbreviation"].isin(selected_methods)]
|
|
@@ -533,6 +500,7 @@ def render_leaderboard(
|
|
| 533 |
|
| 534 |
all_ens_combos = chart_source[["abbreviation", "ensemble_size"]].drop_duplicates()
|
| 535 |
ens_ticks = sorted(all_ens_combos["ensemble_size"].unique().tolist()) if not all_ens_combos.empty else []
|
|
|
|
| 536 |
|
| 537 |
if not chart_df.empty:
|
| 538 |
_ok = chart_df[["abbreviation", "ensemble_size"]].assign(_ok=True)
|
|
@@ -554,7 +522,7 @@ def render_leaderboard(
|
|
| 554 |
axis=alt.Axis(values=ens_ticks, format="d"),
|
| 555 |
),
|
| 556 |
y=alt.Y("mean_forward_runs:Q", title="Mean Forward Model Runs"),
|
| 557 |
-
color=
|
| 558 |
tooltip=["abbreviation", "ensemble_size", alt.Tooltip("mean_forward_runs:Q", format=".4f")],
|
| 559 |
)
|
| 560 |
)
|
|
@@ -574,7 +542,7 @@ def render_leaderboard(
|
|
| 574 |
axis=alt.Axis(values=ens_ticks, format="d"),
|
| 575 |
),
|
| 576 |
y=y_fwd,
|
| 577 |
-
color=
|
| 578 |
tooltip=[
|
| 579 |
alt.Tooltip("abbreviation:N", title="Method"),
|
| 580 |
alt.Tooltip("ensemble_size:Q", title="Ensemble Size"),
|
|
@@ -603,7 +571,7 @@ def render_leaderboard(
|
|
| 603 |
"ensemble_size:Q",
|
| 604 |
axis=alt.Axis(values=ens_ticks, format="d"),
|
| 605 |
),
|
| 606 |
-
color=
|
| 607 |
tooltip=[
|
| 608 |
alt.Tooltip("abbreviation:N", title="Method"),
|
| 609 |
alt.Tooltip("ensemble_size:Q", title="Ensemble Size"),
|
|
@@ -630,6 +598,7 @@ def render_leaderboard(
|
|
| 630 |
all_ens_combos_iters = bf[["abbreviation", "ensemble_size"]].drop_duplicates()
|
| 631 |
if not all_ens_combos_iters.empty:
|
| 632 |
iters_ticks = sorted(all_ens_combos_iters["ensemble_size"].unique().tolist())
|
|
|
|
| 633 |
|
| 634 |
if not iters_df.empty:
|
| 635 |
_ok_iters = iters_df[["abbreviation", "ensemble_size"]].assign(_ok=True)
|
|
@@ -652,7 +621,7 @@ def render_leaderboard(
|
|
| 652 |
axis=alt.Axis(values=iters_ticks, format="d"),
|
| 653 |
),
|
| 654 |
y=alt.Y("mean_iters:Q", title="Mean Iterations"),
|
| 655 |
-
color=
|
| 656 |
tooltip=[
|
| 657 |
alt.Tooltip("abbreviation:N", title="Method"),
|
| 658 |
alt.Tooltip("ensemble_size:Q", title="Ensemble Size"),
|
|
@@ -676,7 +645,7 @@ def render_leaderboard(
|
|
| 676 |
axis=alt.Axis(values=iters_ticks, format="d"),
|
| 677 |
),
|
| 678 |
y=y_iters,
|
| 679 |
-
color=
|
| 680 |
tooltip=[
|
| 681 |
alt.Tooltip("abbreviation:N", title="Method"),
|
| 682 |
alt.Tooltip("ensemble_size:Q", title="Ensemble Size"),
|
|
@@ -715,7 +684,7 @@ def render_leaderboard(
|
|
| 715 |
title="Failure Rate (%)",
|
| 716 |
scale=alt.Scale(domain=[0, 100]),
|
| 717 |
),
|
| 718 |
-
color=
|
| 719 |
tooltip=[
|
| 720 |
alt.Tooltip("abbreviation:N", title="Method"),
|
| 721 |
alt.Tooltip("ensemble_size:O", title="Ensemble Size"),
|
|
@@ -730,6 +699,55 @@ def render_leaderboard(
|
|
| 730 |
)
|
| 731 |
st.altair_chart(failure_chart + ceiling_line, use_container_width=True)
|
| 732 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 733 |
st.caption("Top 3 are shown as podium spots; remaining methods are directly comparable via normalized score.")
|
| 734 |
|
| 735 |
if raw_page is not None:
|
|
|
|
| 33 |
_UNSUITABLE = "#C0392B" # dark red
|
| 34 |
_UNTESTED = "#BDBDBD" # gray
|
| 35 |
|
| 36 |
+
|
| 37 |
+
def _method_color(present_abbrevs: list[str]) -> alt.Color:
|
| 38 |
+
"""Color encoding for the ``abbreviation`` field.
|
| 39 |
+
|
| 40 |
+
The scale domain/range is always the full method registry so a given
|
| 41 |
+
method keeps the same color across every chart and page. The legend,
|
| 42 |
+
however, is restricted to *present_abbrevs* so it only lists methods
|
| 43 |
+
actually plotted in this chart section rather than every known method.
|
| 44 |
+
"""
|
| 45 |
+
return alt.Color(
|
| 46 |
+
"abbreviation:N",
|
| 47 |
+
title="Method",
|
| 48 |
+
scale=alt.Scale(domain=list(METHOD_COLORS.keys()), range=list(METHOD_COLORS.values())),
|
| 49 |
+
legend=alt.Legend(values=sorted(present_abbrevs)),
|
| 50 |
+
)
|
| 51 |
+
|
| 52 |
# Family display order in the leaderboard table (lower = earlier).
|
| 53 |
# Unknown families fall back to 99 and appear at the end.
|
| 54 |
_FAMILY_ORDER: dict[str, int] = {
|
|
|
|
| 202 |
If ``True``, render a grouped-bar failure-rate chart below the main
|
| 203 |
performance chart.
|
| 204 |
show_scoring_modes:
|
| 205 |
+
If ``True``, show scoring-mode radio controls.
|
| 206 |
canonical_target_levels:
|
| 207 |
If provided, the target-level selector always offers exactly these
|
| 208 |
values (as strings) regardless of what is present in the data. Use
|
|
|
|
| 285 |
valid_saved = [m for m in saved_methods if m in available_methods]
|
| 286 |
st.session_state[k_methods] = valid_saved if valid_saved else available_methods
|
| 287 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 288 |
def build_scored_table(input_df: pd.DataFrame, add_rank: bool = True) -> pd.DataFrame:
|
| 289 |
ranking_source = input_df[input_df[target_str_col] == selected_target]
|
| 290 |
if ranking_source.empty:
|
|
|
|
| 445 |
f"and normalized ensemble-size score ({ensemble_weight:.0%})"
|
| 446 |
)
|
| 447 |
|
| 448 |
+
# Controls — always visible (not collapsed behind a dropdown) so users can
|
| 449 |
+
# change target/scoring/methods even when the current selection yields all
|
| 450 |
+
# failures.
|
| 451 |
+
st.subheader("Scoring & Target Controls")
|
| 452 |
+
st.radio(
|
| 453 |
+
target_label,
|
| 454 |
+
options=target_options,
|
| 455 |
+
horizontal=True,
|
| 456 |
+
key=k_target,
|
| 457 |
+
)
|
| 458 |
+
|
| 459 |
+
if show_scoring_modes:
|
| 460 |
st.radio(
|
| 461 |
+
"Scoring Method",
|
| 462 |
+
options=scoring_options,
|
| 463 |
horizontal=True,
|
| 464 |
+
key=k_scoring,
|
| 465 |
)
|
| 466 |
|
| 467 |
+
if st.session_state.get(k_scoring, "Mean Forward Model Runs") == "Custom Blend":
|
| 468 |
+
st.slider(
|
| 469 |
+
"Blend Weight: Forward Runs vs Ensemble Size",
|
| 470 |
+
min_value=0,
|
| 471 |
+
max_value=100,
|
| 472 |
+
step=5,
|
| 473 |
+
key=k_weight,
|
| 474 |
+
help=(
|
| 475 |
+
"Higher forward-runs weight prioritizes fewer model evaluations; "
|
| 476 |
+
"higher ensemble-size weight prioritizes smaller ensembles."
|
| 477 |
+
),
|
| 478 |
)
|
| 479 |
|
| 480 |
+
st.multiselect(
|
| 481 |
+
"Methods to display in charts",
|
| 482 |
+
options=available_methods,
|
| 483 |
+
key=k_methods,
|
| 484 |
+
)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 485 |
|
| 486 |
selected_methods = st.session_state.get(k_methods, available_methods)
|
| 487 |
if not selected_methods:
|
| 488 |
selected_methods = available_methods
|
| 489 |
|
| 490 |
+
st.divider()
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 491 |
|
| 492 |
+
# --- Charts (rendered above the leaderboard table, below the controls) ---
|
| 493 |
+
if not leaderboard_df.empty:
|
| 494 |
st.subheader("Mean Forward Model Runs vs Ensemble Size")
|
| 495 |
chart_source = filtered[filtered[target_str_col] == selected_target]
|
| 496 |
chart_source = chart_source[chart_source["abbreviation"].isin(selected_methods)]
|
|
|
|
| 500 |
|
| 501 |
all_ens_combos = chart_source[["abbreviation", "ensemble_size"]].drop_duplicates()
|
| 502 |
ens_ticks = sorted(all_ens_combos["ensemble_size"].unique().tolist()) if not all_ens_combos.empty else []
|
| 503 |
+
main_color = _method_color(all_ens_combos["abbreviation"].unique().tolist())
|
| 504 |
|
| 505 |
if not chart_df.empty:
|
| 506 |
_ok = chart_df[["abbreviation", "ensemble_size"]].assign(_ok=True)
|
|
|
|
| 522 |
axis=alt.Axis(values=ens_ticks, format="d"),
|
| 523 |
),
|
| 524 |
y=alt.Y("mean_forward_runs:Q", title="Mean Forward Model Runs"),
|
| 525 |
+
color=main_color,
|
| 526 |
tooltip=["abbreviation", "ensemble_size", alt.Tooltip("mean_forward_runs:Q", format=".4f")],
|
| 527 |
)
|
| 528 |
)
|
|
|
|
| 542 |
axis=alt.Axis(values=ens_ticks, format="d"),
|
| 543 |
),
|
| 544 |
y=y_fwd,
|
| 545 |
+
color=main_color,
|
| 546 |
tooltip=[
|
| 547 |
alt.Tooltip("abbreviation:N", title="Method"),
|
| 548 |
alt.Tooltip("ensemble_size:Q", title="Ensemble Size"),
|
|
|
|
| 571 |
"ensemble_size:Q",
|
| 572 |
axis=alt.Axis(values=ens_ticks, format="d"),
|
| 573 |
),
|
| 574 |
+
color=main_color,
|
| 575 |
tooltip=[
|
| 576 |
alt.Tooltip("abbreviation:N", title="Method"),
|
| 577 |
alt.Tooltip("ensemble_size:Q", title="Ensemble Size"),
|
|
|
|
| 598 |
all_ens_combos_iters = bf[["abbreviation", "ensemble_size"]].drop_duplicates()
|
| 599 |
if not all_ens_combos_iters.empty:
|
| 600 |
iters_ticks = sorted(all_ens_combos_iters["ensemble_size"].unique().tolist())
|
| 601 |
+
iters_color = _method_color(all_ens_combos_iters["abbreviation"].unique().tolist())
|
| 602 |
|
| 603 |
if not iters_df.empty:
|
| 604 |
_ok_iters = iters_df[["abbreviation", "ensemble_size"]].assign(_ok=True)
|
|
|
|
| 621 |
axis=alt.Axis(values=iters_ticks, format="d"),
|
| 622 |
),
|
| 623 |
y=alt.Y("mean_iters:Q", title="Mean Iterations"),
|
| 624 |
+
color=iters_color,
|
| 625 |
tooltip=[
|
| 626 |
alt.Tooltip("abbreviation:N", title="Method"),
|
| 627 |
alt.Tooltip("ensemble_size:Q", title="Ensemble Size"),
|
|
|
|
| 645 |
axis=alt.Axis(values=iters_ticks, format="d"),
|
| 646 |
),
|
| 647 |
y=y_iters,
|
| 648 |
+
color=iters_color,
|
| 649 |
tooltip=[
|
| 650 |
alt.Tooltip("abbreviation:N", title="Method"),
|
| 651 |
alt.Tooltip("ensemble_size:Q", title="Ensemble Size"),
|
|
|
|
| 684 |
title="Failure Rate (%)",
|
| 685 |
scale=alt.Scale(domain=[0, 100]),
|
| 686 |
),
|
| 687 |
+
color=_method_color(failure_df["abbreviation"].unique().tolist()),
|
| 688 |
tooltip=[
|
| 689 |
alt.Tooltip("abbreviation:N", title="Method"),
|
| 690 |
alt.Tooltip("ensemble_size:O", title="Ensemble Size"),
|
|
|
|
| 699 |
)
|
| 700 |
st.altair_chart(failure_chart + ceiling_line, use_container_width=True)
|
| 701 |
|
| 702 |
+
st.divider()
|
| 703 |
+
|
| 704 |
+
# --- Leaderboard table (rendered below the charts) ---
|
| 705 |
+
table_column_order = [
|
| 706 |
+
"Placement",
|
| 707 |
+
"abbreviation",
|
| 708 |
+
"Method",
|
| 709 |
+
"family",
|
| 710 |
+
"Score",
|
| 711 |
+
"Mean Forward Model Runs",
|
| 712 |
+
"Minimum Forward Model Runs",
|
| 713 |
+
"Mean Failure Rate (%)",
|
| 714 |
+
"Optimal Ensemble Size",
|
| 715 |
+
"Ensemble Sizes Used",
|
| 716 |
+
]
|
| 717 |
+
|
| 718 |
+
if leaderboard_df.empty:
|
| 719 |
+
st.warning(
|
| 720 |
+
"All runs failed to reach the target at this selection. "
|
| 721 |
+
"See the failure rate chart above."
|
| 722 |
+
if show_failure_panel
|
| 723 |
+
else "No rows available for the current benchmark/target selection."
|
| 724 |
+
)
|
| 725 |
+
else:
|
| 726 |
+
st.subheader(f"Ranked Leaderboard — {selected_benchmark}")
|
| 727 |
+
st.dataframe(
|
| 728 |
+
leaderboard_df,
|
| 729 |
+
hide_index=True,
|
| 730 |
+
use_container_width=True,
|
| 731 |
+
column_config={
|
| 732 |
+
"Placement": st.column_config.TextColumn("Placement"),
|
| 733 |
+
"family": st.column_config.TextColumn("Family"),
|
| 734 |
+
"Method": st.column_config.TextColumn("Method"),
|
| 735 |
+
"abbreviation": st.column_config.TextColumn("Abbrev."),
|
| 736 |
+
"Mean Forward Model Runs": st.column_config.NumberColumn("Mean Forward Model Runs", format="%.4f"),
|
| 737 |
+
"Minimum Forward Model Runs": st.column_config.NumberColumn("Minimum Forward Model Runs", format="%.4f"),
|
| 738 |
+
"Score": st.column_config.ProgressColumn("Score (0-100)", min_value=0.0, max_value=100.0, format="%.1f"),
|
| 739 |
+
"Optimal Ensemble Size": st.column_config.NumberColumn("Mean Optimal Ensemble Size", format="%.2f"),
|
| 740 |
+
"Mean Failure Rate (%)": st.column_config.NumberColumn("Mean Failure Rate (%)", format="%.1f"),
|
| 741 |
+
"Ensemble Sizes Used": st.column_config.NumberColumn("Ensemble Sizes Used", format="%d"),
|
| 742 |
+
},
|
| 743 |
+
column_order=table_column_order,
|
| 744 |
+
)
|
| 745 |
+
|
| 746 |
+
st.info(
|
| 747 |
+
f"Score is a normalized 0–100 ranking based on **{score_basis}**. "
|
| 748 |
+
"Values are computed from all ensemble sizes after averaging over random seeds."
|
| 749 |
+
)
|
| 750 |
+
|
| 751 |
st.caption("Top 3 are shown as podium spots; remaining methods are directly comparable via normalized score.")
|
| 752 |
|
| 753 |
if raw_page is not None:
|