binarybeard commited on
Commit
5241248
Β·
unverified Β·
2 Parent(s): 0ee642ec952d77

Merge pull request #22 from abducodez/feat/hall-of-fame-leaderboard

Browse files

feat: Add Hall of Fame leaderboard backed by a dedicated results table

docs/adr/0035-hall-of-fame-leaderboard.md ADDED
@@ -0,0 +1,248 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # ADR-0035: Hall of Fame β€” Dedicated Scoreboard Table, Detached from the Event Ledger
2
+
3
+ **Status:** Accepted
4
+ **Date:** 2026-06-14
5
+
6
+ ## Context
7
+
8
+ Every competitive run now crowns a winner ([ADR-0029](0029-structured-verdicts.md),
9
+ [ADR-0030](0030-arena-scenarios-and-offline-winners.md)). What was missing was a
10
+ place to *remember*. Without a leaderboard the question "which model wins Debate Duel
11
+ the most?" has no answer β€” wins happen, are faithfully recorded on the ledger, and
12
+ then vanish from view the moment the next run starts.
13
+
14
+ The original design (the initial version of this ADR) proposed the leaderboard as a
15
+ *pure projection over the event stream* β€” a read-only fold with no new storage. That
16
+ approach was replaced, at the operator's explicit request, with the dedicated-table
17
+ design described here. The core trade-off: pure projections are elegant and
18
+ philosophically clean, but at page-load time they must fold the *entire* events log
19
+ β€” every utterance, thought, and verdict in every run β€” just to count wins. As the
20
+ ledger grows, that becomes an O(all-events) query on every Hall of Fame tab refresh.
21
+ A dedicated scoreboard table turns that into a cheap `SELECT` over a small,
22
+ already-decided set of rows.
23
+
24
+ Four constraints shaped the final design:
25
+
26
+ 1. **One row per decided run, written once, at finish.** The scoreboard is not a live
27
+ aggregate; it is a durable receipt. Writing at finalization time, under an explicit
28
+ eligibility gate, keeps the table small and its rows trustworthy.
29
+ 2. **Detached from the event ledger, linked by `run_id`.** The `events` table stays
30
+ the source of truth for *what happened* in a run (the full trace β€” every utterance,
31
+ thought, verdict, and poke). The `leaderboard_entries` table is a separate,
32
+ denormalised record of *who won*. They share a database instance (the same
33
+ `DATABASE_URL` / Postgres instance) but never share rows. The `run_id` foreign key
34
+ on every leaderboard entry links back to the full trace for replay β€” so the
35
+ separation is clean without being lossy.
36
+ 3. **An explicit write gate.** A row is recorded only when the run is FINISHED, has a
37
+ WINNER, carries at least one concrete WINNING MODEL endpoint, and is COMPETITIVE
38
+ (`competition.kind != "none"`). This gate is the single choke point that keeps
39
+ the scoreboard honest: abandoned runs, budget closures with no verdict, and
40
+ exhibition-mode (`kind: none`) sessions never produce a row.
41
+ 4. **Fairness is first-class.** Raw win counts mislead when seats are not symmetric.
42
+ The `LeaderboardEntry` row carries the full competition shape (`kind`, `teams`,
43
+ `symmetric_seats`), so the fairness rollup in `leaderboard.py` needs no registry
44
+ lookup and no event replay.
45
+
46
+ Workstream 6 of the [arena roadmap](../architecture/next-steps/arena-roadmap.md)
47
+ specified the "Hall of Fame" UI tab. This ADR records what shipped.
48
+
49
+ ## Decision
50
+
51
+ ### 1. `src/core/leaderboard_store.py` β€” the dedicated, durable table
52
+
53
+ `LeaderboardEntry` (Pydantic v2) is the denormalised scoreboard row β€” one per decided
54
+ competitive run. It carries:
55
+
56
+ - `run_id` β€” links back to `ledger.events_for_run(run_id)` for replay (ADR-0027).
57
+ - `scenario`, `seed`, `cast` (cast→model bindings from the `run.started` stamp,
58
+ ADR-0030) β€” everything needed to describe the run without touching the events log.
59
+ - `winner`, `winner_kind`, `winning_model`, `winning_models` β€” the structured verdict
60
+ fields from ADR-0029.
61
+ - `competition_kind`, `teams`, `symmetric_seats` β€” the competition shape from
62
+ ADR-0030, needed for per-seat fairness rollups.
63
+ - `reason`, `turns`, `tokens`, `started_at`, `finished_at`, `recorded_at`.
64
+
65
+ `LeaderboardStore` (SQLAlchemy, mirroring `SqlAlchemyLedger`'s lazy-import pattern)
66
+ owns the `leaderboard_entries` table. Its `record()` method is an idempotent upsert
67
+ keyed on `run_id` β€” delete-then-insert in one transaction, so a verdict that
68
+ supersedes a budget close replaces the row rather than duplicating it. `entries()` and
69
+ `entries_for_scenario()` are the two read paths.
70
+
71
+ `make_leaderboard_store()` is a memoised factory that resolves the same `DATABASE_URL`
72
+ as `ledger_factory.make_ledger`. No extra configuration is needed: the store is a
73
+ separate table in the same Postgres instance. The in-memory (`sqlite://`) stage demo
74
+ works end-to-end because the factory memoises by URL β€” the write at `finalize` and the
75
+ Hall of Fame read share the same in-memory engine.
76
+
77
+ `build_entry(summary, competition)` is the eligibility gate. It accepts a `RunSummary`
78
+ (folded from the run's events by `index_runs`) and the scenario's `CompetitionConfig`.
79
+ It returns `None` β€” producing no row β€” unless all four conditions are met: `finished_at`
80
+ is set, `winner` is non-empty, at least one winning model endpoint is non-empty, and
81
+ `competition.kind != "none"`.
82
+
83
+ ### 2. The write path β€” `FishbowlSession._record_leaderboard()`
84
+
85
+ The write happens exactly once per run, inside `FishbowlSession.finalize()`, called
86
+ when the run reaches `"verdict"` or `"budget"`:
87
+
88
+ ```python
89
+ def _record_leaderboard(self) -> None:
90
+ # Folds the run's events into a RunSummary, checks eligibility via build_entry,
91
+ # and writes one row to leaderboard_entries. Idempotent. Fully defensive β€”
92
+ # any failure is swallowed so a store hiccup never breaks the show.
93
+ run_events = self.conductor.ledger.events_for_run(self.conductor.run_id)
94
+ summary = next((s for s in index_runs(run_events) if s.run_id == ...), None)
95
+ entry = build_entry(summary, getattr(scenario, "competition", None))
96
+ if entry is not None:
97
+ store.record(entry)
98
+ ```
99
+
100
+ The method is fully defensive: any store failure is logged and swallowed. A
101
+ leaderboard write hiccup never breaks a live run.
102
+
103
+ ### 3. `src/core/leaderboard.py` β€” aggregations over `LeaderboardEntry` rows
104
+
105
+ The five public aggregation functions take `Sequence[LeaderboardEntry]` (not an event
106
+ stream). They never touch the `events` table.
107
+
108
+ | Function | Returns | Sort order |
109
+ |---|---|---|
110
+ | `scenario_sessions(entries, scenario_name)` | `list[LeaderboardEntry]` | newest-first by `finished_at`, `run_id` tiebreak |
111
+ | `model_table(entries)` | `list[ModelRow]` | `(-win_rate, -wins, -plays, model)` |
112
+ | `agent_table(entries, scenario_name)` | `list[AgentRow]` | `(-win_rate, -wins, -plays, agent)` |
113
+ | `fairness_table(entries, scenario_name)` | `list[SeatRow]` | `(-win_rate, -wins, -plays, seat_type)` |
114
+ | `headline(entries)` | `str \| None` | single summary sentence |
115
+
116
+ `headline` considers only symmetric-seat scenarios, requires β‰₯2 distinct models that
117
+ have each won β‰₯1 game, picks the scenario with the most decided games, and shortens
118
+ model endpoint slugs to the segment after the last `/`. It returns `None` when the
119
+ table holds no qualifying data β€” safe at app start.
120
+
121
+ Attribution rules are identical to the previous pure-projection design: a model earns
122
+ one play per run (endpoint set deduplicated); win credit goes to every endpoint in
123
+ `winning_models βˆͺ {winning_model}`. `fairness_table` counts only declared seats
124
+ (team members and `symmetric_seats` members); cast members with no seat type (judges,
125
+ narrators) are excluded so they never appear as zero-percent rows.
126
+
127
+ ### 4. "Hall of Fame" Gradio tab in `src/ui/fishbowl/hall_of_fame.py`
128
+
129
+ The tab calls `make_leaderboard_store().entries()` on render, passes the resulting
130
+ `list[LeaderboardEntry]` to the five aggregation functions, and renders HTML. It never
131
+ touches the event ledger or the live `Conductor`. The per-row **Replay** button calls
132
+ `load_replay(run_id)` from `src/ui/fishbowl/archive.py` (ADR-0027) β€” the full event
133
+ trace is still in the ledger, accessible via `run_id`, so replay requires no change to
134
+ the transport mechanism.
135
+
136
+ ## Consequences
137
+
138
+ ### Positive
139
+
140
+ - **Cheap reads.** Hall of Fame renders are a `SELECT * FROM leaderboard_entries`,
141
+ not an O(all-events) fold of the event log. The table stays small by design: only
142
+ finished, won, competitive runs produce a row.
143
+ - **Clean separation of trace and scoreboard.** The `events` table is the authoritative
144
+ record of what every agent said and thought. The `leaderboard_entries` table is the
145
+ permanent record of who won. Neither needs the other to do its job.
146
+ - **Rebuildable.** The scoreboard is derived from events: every `LeaderboardEntry` can
147
+ be reconstructed by replaying `events_for_run(run_id)` through `index_runs` and
148
+ `build_entry`. The table is not a competing source of truth; it is a materialised
149
+ view that happens to live in a dedicated table. If the table is lost or corrupted, a
150
+ rebuild script using ADR-0026's `index_runs` can regenerate it.
151
+ - **Idempotent write.** The upsert-on-`run_id` means a corrective re-finalize (a
152
+ budget close later superseded by a verdict β€” the ADR-0026 "last-wins" pattern)
153
+ produces exactly one, current row. No duplicates; no gaps.
154
+ - **Fairness is first-class.** Per-seat win rates are built into the row schema, not
155
+ bolted on later. The competition shape travels with the result, so no registry lookup
156
+ is needed at render time.
157
+ - **Replay falls out for free.** `run_id` links every scoreboard row back to its full
158
+ trace. ADR-0027's `load_replay` / `ReplaySession` already knows how to feed
159
+ historical events into The Show; the Hall of Fame is just a new call site.
160
+ - **Deterministic tiebreaks.** Every sort ends on a string column (`model`, `agent`,
161
+ `seat_type`, `run_id`) so table order is reproducible across re-renders.
162
+
163
+ ### Negative / trade-offs
164
+
165
+ - **A second table can drift from the ledger.** The events log and the scoreboard are
166
+ separate tables. If a run's events are manually edited or a schema migration changes
167
+ the `run.finished` payload, the scoreboard row for that run will not update
168
+ automatically. The `run_id` link and the idempotent upsert mitigate this: a
169
+ re-finalize call regenerates the row from the current event state, and a rebuild
170
+ script can do the same in bulk. But the drift risk is real and was not present in a
171
+ pure-projection design.
172
+ - **Write-time eligibility is a point-in-time decision.** `build_entry` evaluates
173
+ eligibility at `finalize` time, against the competition config that was loaded into
174
+ the registry at that moment. A scenario YAML change that retroactively alters
175
+ `competition.kind` will not affect already-written rows. This is generally the
176
+ correct behaviour (the row reflects the game that was actually played), but operators
177
+ should be aware of it.
178
+ - **SQLAlchemy is now required on the write path.** The store's lazy-import pattern
179
+ (`SQLAlchemy` imported inside `__init__`) keeps the offline/stub path import-clean,
180
+ but any production `finalize` call on a Postgres deployment requires the driver.
181
+ This was already true of the `SqlAlchemyLedger` (ADR-0014); this ADR extends that
182
+ dependency to the write side of the Hall of Fame.
183
+
184
+ ### Neutral
185
+
186
+ - `headline` returns `None` at startup (no qualifying data). The UI renders a
187
+ cheerful empty state; nothing is fabricated.
188
+ - Sort order is stable but opinionated: win-rate before raw wins, then plays, then
189
+ name. A future "sort by column" affordance can override this without changing the
190
+ aggregations.
191
+ - The store is co-located with the ledger (same `DATABASE_URL`) but is architecturally
192
+ independent: it could be moved to a separate database by changing the URL passed to
193
+ `make_leaderboard_store()`.
194
+
195
+ ## Alternatives considered
196
+
197
+ - **Pure ledger projection (the original design).** The first version of this ADR
198
+ described five pure functions over `Iterable[Event]` β€” no new table, no new storage,
199
+ a perfect fit for ADR-0001's "ledger as source of truth" axiom. This was the
200
+ correct starting point. It was set aside because the O(all-events) cost per render
201
+ becomes perceptible as the ledger grows, and because the operator's explicit
202
+ requirement was a dedicated, durable scoreboard row rather than a live computation.
203
+ The pure-Python functions in `leaderboard.py` survive as aggregations over the
204
+ dedicated table's rows, so the computation model is preserved β€” only the input
205
+ changed from raw events to pre-decided `LeaderboardEntry` rows.
206
+
207
+ - **Extend `RunSummary` to carry the competition block.** This would have let a
208
+ pure-projection leaderboard reuse `index_runs` output cleanly (the original design's
209
+ main awkwardness). It was deferred rather than rejected: the change is mechanical but
210
+ touches ADR-0026's public surface. With the dedicated-table design, the competition
211
+ shape travels on the `LeaderboardEntry` row itself, making this less urgent.
212
+
213
+ - **SQL aggregate query as the primary path.** Faster at scale than folding Python
214
+ objects, but adds a backend-specific code path and a `SqlAlchemyLedger`-only
215
+ dependency into the aggregation module. The dedicated table achieves the same
216
+ read-performance goal with a simpler interface (fold a small list of Pydantic
217
+ objects). Revisit if the leaderboard's aggregation logic becomes a bottleneck.
218
+
219
+ - **Elo instead of win rate.** More statistically meaningful once session counts
220
+ grow, but requires at least ~30 decided games per model pair to be trustworthy.
221
+ Raw win rate is honest at small N and does not mislead when shown alongside the
222
+ play count. Elo is noted in the arena roadmap's "Beyond" section.
223
+
224
+ ## References
225
+
226
+ - [ADR-0001](0001-event-ledger.md) β€” event ledger as the single source of truth for
227
+ *the trace*. The scoreboard is a derived, rebuildable materialisation keyed by
228
+ `run_id`, not a competing source of truth.
229
+ - [ADR-0014](0014-postgres-event-store.md) β€” durable Postgres store behind the ledger
230
+ interface; the `DATABASE_URL` the leaderboard store shares.
231
+ - [ADR-0026](0026-run-lifecycle-and-history.md) β€” run lifecycle; `run.started` /
232
+ `run.finished`; `RunSummary`; `index_runs` β€” the fold `build_entry` calls.
233
+ - [ADR-0027](0027-per-user-sessions-and-archive-replay.md) β€” `ReplaySession` and
234
+ `load_replay`; the replay transport the Hall of Fame Replay button reuses via `run_id`.
235
+ - [ADR-0029](0029-structured-verdicts.md) β€” structured `winner` / `winner_kind` /
236
+ `winning_models` on `run.finished`; the source of the scoreboard's attribution fields.
237
+ - [ADR-0030](0030-arena-scenarios-and-offline-winners.md) β€” `competition` block
238
+ stamped on `run.started`; `symmetric_seats`; the eight arena-grade scenarios. The
239
+ competition shape travels onto every `LeaderboardEntry` row.
240
+ - `src/core/leaderboard_store.py` β€” `LeaderboardEntry`, `LeaderboardStore`,
241
+ `build_entry`, `make_leaderboard_store`.
242
+ - `src/core/leaderboard.py` β€” the five aggregation functions over `LeaderboardEntry`
243
+ rows.
244
+ - `src/ui/fishbowl/session.py` β€” `FishbowlSession._record_leaderboard()`, the write
245
+ path.
246
+ - `src/ui/fishbowl/hall_of_fame.py` β€” the Gradio tab.
247
+ - [arena roadmap Β§ Workstream 6](../architecture/next-steps/arena-roadmap.md) β€” the
248
+ original specification these projections realise.
docs/architecture/fishbowl-ui.md CHANGED
@@ -38,11 +38,11 @@ app.py (repo root) thin shim β†’ from src.ui.fishbowl.app import demo
38
  The dependency arrow is one-way: `src/ui/fishbowl/` reads the engine's public surface;
39
  **the engine never imports the UI** (test-enforced by `tests/test_modularity.py`).
40
 
41
- ## The two tabs
42
 
43
- Fishbowl is a `gr.Blocks` with a top bar and a `gr.Tabs` holding **The Lab** and
44
- **The Show**. A per-session `gr.State` carries the live `FishbowlSession` and the
45
- play-head, so concurrent visitors never share a world.
46
 
47
  ### The Lab β€” compose a run
48
 
@@ -73,6 +73,37 @@ Split); a **"Read their minds" `gr.Checkbox`**; a transport (scrubber `gr.Slider
73
  disturbances. `build_show` returns component handles only β€” every callback is wired in
74
  `app.py`.
75
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
76
  ## The render loop β€” `gr.HTML` + `gr.Timer`
77
 
78
  The Show does not stream component diffs; it **re-renders one `gr.HTML` block** each
@@ -265,3 +296,6 @@ stays green by construction.
265
  - [observer-pattern.md](observer-pattern.md) β€” the decoupled-rendering contract this
266
  realizes.
267
  - [ADR-0002](../adr/0002-gradio-first.md) β€” chose Gradio and anticipated this migration.
 
 
 
 
38
  The dependency arrow is one-way: `src/ui/fishbowl/` reads the engine's public surface;
39
  **the engine never imports the UI** (test-enforced by `tests/test_modularity.py`).
40
 
41
+ ## The three tabs
42
 
43
+ Fishbowl is a `gr.Blocks` with a top bar and a `gr.Tabs` holding **The Lab**,
44
+ **The Show**, and **Hall of Fame**. A per-session `gr.State` carries the live
45
+ `FishbowlSession` and the play-head, so concurrent visitors never share a world.
46
 
47
  ### The Lab β€” compose a run
48
 
 
73
  disturbances. `build_show` returns component handles only β€” every callback is wired in
74
  `app.py`.
75
 
76
+ ### Hall of Fame β€” the permanent record
77
+
78
+ `hall_of_fame.py` (`build_hall_of_fame`) is a read-only tab backed by the dedicated
79
+ `leaderboard_entries` table ([ADR-0035](../adr/0035-hall-of-fame-leaderboard.md)).
80
+ On each render it calls `make_leaderboard_store().entries()` to fetch the rows, then
81
+ passes them to the five aggregation functions in `src/core/leaderboard.py`. It never
82
+ reads the event ledger and never touches the live `Conductor` or the session state.
83
+ The `events` log stays the trace; this table is the scoreboard β€” linked back via
84
+ `run_id` for replay, not duplicated.
85
+
86
+ - **Scenario picker** (`gr.Dropdown`) filters all tables to one scenario. The "All
87
+ scenarios" view is the model leaderboard's default β€” the cross-scenario headline.
88
+ - **Sessions table** (`gr.Dataframe` of `SessionRow`) lists every finished competitive
89
+ run: date, cast summary, winner, winning model endpoint, turns, tokens, and end
90
+ reason. Newest first.
91
+ - **Replay button** β€” each row carries a run id; clicking it calls `load_replay` from
92
+ `src/ui/fishbowl/archive.py` (ADR-0027) and hands the `ReplaySession` to The Show.
93
+ No new transport mechanism; replay is the same path the Archive drawer uses.
94
+ - **Model leaderboard** (`gr.Dataframe` of `ModelRow`) β€” plays, wins, win rate, and
95
+ scenarios per model endpoint. This is the headline demo artifact: a single ledger
96
+ fold turns "MiniCPM-8B has beaten Gemma-12B 7–3 at Debate Duel" into a table cell.
97
+ - **Agent / fairness tables** (`gr.Dataframe` of `AgentRow` and `SeatRow`) β€” per-seat
98
+ win rates for the selected scenario. Judges and other non-seat cast members are
99
+ excluded from `SeatRow` counts so the asymmetry is honest, not hidden.
100
+ - **Headline** β€” the `headline()` projection renders a single prose sentence above the
101
+ tables when the ledger holds β‰₯1 symmetric-seat scenario with β‰₯2 models having won.
102
+ Returns `None` at app start (no runs yet) and renders nothing; the UI handles both.
103
+
104
+ The Hall of Fame refreshes on tab focus, not on a timer. It holds no derived state β€”
105
+ every render is a fresh projection fold.
106
+
107
  ## The render loop β€” `gr.HTML` + `gr.Timer`
108
 
109
  The Show does not stream component diffs; it **re-renders one `gr.HTML` block** each
 
296
  - [observer-pattern.md](observer-pattern.md) β€” the decoupled-rendering contract this
297
  realizes.
298
  - [ADR-0002](../adr/0002-gradio-first.md) β€” chose Gradio and anticipated this migration.
299
+ - [ADR-0035](../adr/0035-hall-of-fame-leaderboard.md) β€” Hall of Fame: dedicated
300
+ `leaderboard_entries` table detached from the event ledger; the five aggregation
301
+ functions in `leaderboard.py` and the replay reuse via `run_id`.
docs/architecture/next-steps/arena-roadmap.md CHANGED
@@ -257,8 +257,16 @@ here.
257
 
258
  ## Workstream 6 β€” Hall of Fame (leaderboard)
259
 
260
- Pure projection over the ledger β€” no new storage, fits ADR-0001 exactly.
261
- Requires W1 (run lifecycle) + W2 (structured winner).
 
 
 
 
 
 
 
 
262
 
263
  ### 6.1 `src/core/leaderboard.py` β€” pure functions
264
 
 
257
 
258
  ## Workstream 6 β€” Hall of Fame (leaderboard)
259
 
260
+ > βœ… *Shipped β€” 2026-06-14.* The dedicated `leaderboard_entries` table, the five
261
+ > aggregation functions, and the Gradio tab are live.
262
+ > See [ADR-0035](../../adr/0035-hall-of-fame-leaderboard.md) for the as-built record.
263
+ > The plan below is preserved as the original spec.
264
+
265
+ Dedicated detached table β€” not a pure projection. A separate `leaderboard_entries`
266
+ table (same Postgres instance as the event ledger, never sharing rows with `events`)
267
+ stores one denormalised result row per decided competitive run. The Hall of Fame reads
268
+ this small table directly; the `events` log stays the trace. `run_id` on every row
269
+ links back to the full trace for replay. Requires W1 (run lifecycle) + W2 (structured winner).
270
 
271
  ### 6.1 `src/core/leaderboard.py` β€” pure functions
272
 
src/core/leaderboard.py ADDED
@@ -0,0 +1,372 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Leaderboard β€” competitive-results aggregations over the dedicated scoreboard table.
2
+
3
+ This module is the *read model* of the Hall of Fame. It is deliberately **detached from
4
+ the event ledger**: it folds a list of :class:`~src.core.leaderboard_store.LeaderboardEntry`
5
+ rows β€” the materialised scoreboard persisted in the ``leaderboard_entries`` table β€” into
6
+ the model / agent / fairness tables the UI renders. It never touches the ``events`` log.
7
+
8
+ The split, in one line: :mod:`src.core.leaderboard_store` owns *persistence* (one durable
9
+ row per decided run, written at finish), and this module owns *aggregation* (cheap folds
10
+ over those rows). Because a row is only ever written for a finished, won, competitive run
11
+ (see ``build_entry`` and ``FishbowlSession.finalize``), the functions here can trust every
12
+ entry they receive is already "ranked" β€” they re-check ``winner`` only defensively.
13
+
14
+ Each entry is self-describing: it carries the cast→model bindings *and* the competition
15
+ shape (``competition_kind`` / ``teams`` / ``symmetric_seats``), so the per-seat fairness
16
+ rollup needs no registry lookup and no event replay.
17
+
18
+ Schema is additive only; ``schema_version`` is unaffected (this reads a separate table).
19
+ """
20
+
21
+ from __future__ import annotations
22
+
23
+ from collections import defaultdict
24
+ from datetime import datetime
25
+ from typing import Iterable, Sequence
26
+
27
+ from pydantic import BaseModel, ConfigDict, Field
28
+
29
+ from src.core.leaderboard_store import LeaderboardEntry
30
+
31
+
32
+ # ── competition shape (built per entry for seat mapping) ─────────────────────────
33
+
34
+
35
+ class CompetitionBlock(BaseModel):
36
+ """The competition shape of one run (built from a :class:`LeaderboardEntry`).
37
+
38
+ ``kind`` is ``"versus"`` / ``"judged"`` / ``"none"``; ``teams`` (versus only) maps a
39
+ team label β†’ member agent names; ``symmetric_seats`` (versus only) lists identical
40
+ seats that differ only by model.
41
+ """
42
+
43
+ model_config = ConfigDict(extra="ignore")
44
+
45
+ kind: str = "none"
46
+ teams: dict[str, list[str]] | None = None
47
+ symmetric_seats: list[str] | None = None
48
+
49
+
50
+ def _block(entry: LeaderboardEntry) -> CompetitionBlock:
51
+ """The competition shape carried by *entry* (no registry / ledger lookup)."""
52
+ return CompetitionBlock(
53
+ kind=entry.competition_kind,
54
+ teams=entry.teams,
55
+ symmetric_seats=entry.symmetric_seats,
56
+ )
57
+
58
+
59
+ # ── row models ─────────────────────────────────────────────────────────────────
60
+
61
+
62
+ class ModelRow(BaseModel):
63
+ """A model endpoint's record across *all* competitive decided runs."""
64
+
65
+ model_config = ConfigDict(extra="forbid")
66
+
67
+ model: str
68
+ plays: int = 0
69
+ """Decided competitive runs whose cast contained this endpoint."""
70
+ wins: int = 0
71
+ """Of those, how many this endpoint was credited with winning."""
72
+ win_rate: float = 0.0
73
+ """``wins / plays`` (0.0 when ``plays == 0``)."""
74
+ scenarios: list[str] = Field(default_factory=list)
75
+ """Sorted distinct scenario names this endpoint appeared in."""
76
+
77
+
78
+ class AgentRow(BaseModel):
79
+ """A persona's (cast seat *name*) record within a single scenario."""
80
+
81
+ model_config = ConfigDict(extra="forbid")
82
+
83
+ agent: str
84
+ """The cast member name β€” the seat, not the model."""
85
+ seat_type: str = ""
86
+ """Team label, symmetric-seat name, or ``""`` when the seat maps to neither."""
87
+ plays: int = 0
88
+ wins: int = 0
89
+ win_rate: float = 0.0
90
+ model_endpoints: list[str] = Field(default_factory=list)
91
+ """Sorted distinct model endpoints that filled this seat."""
92
+
93
+
94
+ class SeatRow(BaseModel):
95
+ """Win rate per *seat type* within a scenario (the 6.3 fairness footnote).
96
+
97
+ Surfaces structural asymmetry: spy vs herd, debater-a vs debater-b, the judge that
98
+ never wins. ``seat_type`` is a team label (versus-teams) or a symmetric-seat name.
99
+ """
100
+
101
+ model_config = ConfigDict(extra="forbid")
102
+
103
+ seat_type: str
104
+ plays: int = 0
105
+ wins: int = 0
106
+ win_rate: float = 0.0
107
+
108
+
109
+ # ── internal helpers ───────────────────────────────────────────────────────────
110
+
111
+
112
+ def _win_rate(wins: int, plays: int) -> float:
113
+ """``wins / plays``, or ``0.0`` when *plays* is zero (never divide by zero)."""
114
+ return (wins / plays) if plays else 0.0
115
+
116
+
117
+ def _ranked(entries: Iterable[LeaderboardEntry]) -> list[LeaderboardEntry]:
118
+ """Defensive gate: keep only entries that actually name a winner.
119
+
120
+ The store only ever persists finished + won + competitive runs, so this is belt-and-
121
+ suspenders β€” it drops any malformed/empty row rather than crediting a phantom win.
122
+ """
123
+ return [e for e in entries if e and e.winner]
124
+
125
+
126
+ def _seat_type_for(agent: str, block: CompetitionBlock) -> str:
127
+ """Map a cast member *name* to its seat type within *block*.
128
+
129
+ For ``versus`` teams the seat type is the team label the agent belongs to. For
130
+ ``symmetric_seats`` each named seat is its own type (the agent name == the seat). A
131
+ cast member that belongs to neither (e.g. a judge, a narrator) maps to ``""``.
132
+ """
133
+ if block.teams:
134
+ for label, members in block.teams.items():
135
+ if agent in (members or []):
136
+ return label
137
+ if block.symmetric_seats and agent in block.symmetric_seats:
138
+ return agent
139
+ return ""
140
+
141
+
142
+ def _winning_seat_types(entry: LeaderboardEntry, block: CompetitionBlock) -> set[str]:
143
+ """The seat type(s) credited with the win for fairness accounting."""
144
+ winner = entry.winner or ""
145
+ if not winner:
146
+ return set()
147
+ if block.teams and winner in block.teams:
148
+ return {winner}
149
+ seat = _seat_type_for(winner, block)
150
+ return {seat} if seat else set()
151
+
152
+
153
+ def _winning_agents(entry: LeaderboardEntry, block: CompetitionBlock) -> set[str]:
154
+ """Cast member name(s) credited with the win, for per-seat accounting.
155
+
156
+ A ``team`` winner credits every member of the winning team; an ``agent`` winner
157
+ (judged pick / symmetric-seat winner) credits just that name. Falls back to treating
158
+ ``winner`` as a bare agent name when ``winner_kind`` is absent.
159
+ """
160
+ winner = entry.winner or ""
161
+ if not winner:
162
+ return set()
163
+ if entry.winner_kind == "team" or (block.teams and winner in block.teams):
164
+ return set((block.teams or {}).get(winner) or [])
165
+ return {winner}
166
+
167
+
168
+ def _credited_models(entry: LeaderboardEntry) -> set[str]:
169
+ """The model endpoint(s) credited with this run's win (``winning_models`` βˆͺ single)."""
170
+ credited = {m for m in entry.winning_models if m}
171
+ if entry.winning_model:
172
+ credited.add(entry.winning_model)
173
+ return credited
174
+
175
+
176
+ # ── public aggregations (fold the scoreboard rows) ───────────────────────────────
177
+
178
+
179
+ def scenario_sessions(entries: Sequence[LeaderboardEntry], scenario_name: str) -> list[LeaderboardEntry]:
180
+ """The decided sessions of *scenario_name*, newest first.
181
+
182
+ A thin filter + deterministic sort over the stored rows: by ``finished_at``
183
+ descending (newest first), runs missing a finish time sorted last, ``run_id`` breaking
184
+ ties. Returns the :class:`LeaderboardEntry` rows themselves β€” they already carry the
185
+ winner, cast→model bindings and cost the sessions table renders.
186
+ """
187
+ rows = [e for e in _ranked(entries) if e.scenario == scenario_name]
188
+ rows.sort(key=lambda e: (e.finished_at is None, _neg_ts(e.finished_at), e.run_id))
189
+ return rows
190
+
191
+
192
+ def model_table(entries: Sequence[LeaderboardEntry]) -> list[ModelRow]:
193
+ """One :class:`ModelRow` per model endpoint across *all* decided competitive runs.
194
+
195
+ A model *plays* a run when its endpoint appears in that run's cast; it *wins* when its
196
+ endpoint is among the run's credited winners (``winning_models`` / ``winning_model``).
197
+ ``scenarios`` lists the distinct scenario names the model appeared in (sorted). Sorted
198
+ by ``win_rate`` desc, then ``wins`` desc, then ``plays`` desc, then ``model`` asc.
199
+ """
200
+ plays: dict[str, int] = defaultdict(int)
201
+ wins: dict[str, int] = defaultdict(int)
202
+ scenarios: dict[str, set[str]] = defaultdict(set)
203
+ for entry in _ranked(entries):
204
+ credited = _credited_models(entry)
205
+ seen: set[str] = set()
206
+ for binding in entry.cast.values():
207
+ endpoint = binding.model_endpoint
208
+ if not endpoint or endpoint in seen:
209
+ continue # one play per endpoint per run, even if it fills two seats
210
+ seen.add(endpoint)
211
+ plays[endpoint] += 1
212
+ scenarios[endpoint].add(entry.scenario)
213
+ if endpoint in credited:
214
+ wins[endpoint] += 1
215
+ rows = [
216
+ ModelRow(
217
+ model=endpoint,
218
+ plays=plays[endpoint],
219
+ wins=wins[endpoint],
220
+ win_rate=_win_rate(wins[endpoint], plays[endpoint]),
221
+ scenarios=sorted(scenarios[endpoint]),
222
+ )
223
+ for endpoint in plays
224
+ ]
225
+ rows.sort(key=lambda r: (-r.win_rate, -r.wins, -r.plays, r.model))
226
+ return rows
227
+
228
+
229
+ def agent_table(entries: Sequence[LeaderboardEntry], scenario_name: str) -> list[AgentRow]:
230
+ """Per-persona (cast seat *name*) wins within *scenario_name*.
231
+
232
+ One :class:`AgentRow` per cast member name that appears in a decided run of the
233
+ scenario. A seat *plays* a run when its name is in the cast and *wins* when it is the
234
+ run's winning agent, or a member of the winning team. ``seat_type`` is the seat's team
235
+ label / symmetric-seat name (or ``""``); ``model_endpoints`` lists the distinct models
236
+ that filled the seat. Deterministic sort matching :func:`model_table`.
237
+ """
238
+ plays: dict[str, int] = defaultdict(int)
239
+ wins: dict[str, int] = defaultdict(int)
240
+ seat_types: dict[str, str] = {}
241
+ endpoints: dict[str, set[str]] = defaultdict(set)
242
+ for entry in _ranked(entries):
243
+ if entry.scenario != scenario_name:
244
+ continue
245
+ block = _block(entry)
246
+ winners = _winning_agents(entry, block)
247
+ for name, binding in entry.cast.items():
248
+ plays[name] += 1
249
+ seat_types.setdefault(name, _seat_type_for(name, block))
250
+ if binding.model_endpoint:
251
+ endpoints[name].add(binding.model_endpoint)
252
+ if name in winners:
253
+ wins[name] += 1
254
+ rows = [
255
+ AgentRow(
256
+ agent=name,
257
+ seat_type=seat_types.get(name, ""),
258
+ plays=plays[name],
259
+ wins=wins[name],
260
+ win_rate=_win_rate(wins[name], plays[name]),
261
+ model_endpoints=sorted(endpoints[name]),
262
+ )
263
+ for name in plays
264
+ ]
265
+ rows.sort(key=lambda r: (-r.win_rate, -r.wins, -r.plays, r.agent))
266
+ return rows
267
+
268
+
269
+ def fairness_table(entries: Sequence[LeaderboardEntry], scenario_name: str) -> list[SeatRow]:
270
+ """Win rate per *seat type* within *scenario_name* β€” the 6.3 fairness footnote.
271
+
272
+ Aggregates the per-persona view up to seat types so structural asymmetry is visible:
273
+ spy vs herd, debater-a vs debater-b, a judge that never wins. Seat membership comes
274
+ from each entry's stored competition shape (``teams`` β†’ label per member;
275
+ ``symmetric_seats`` β†’ each seat its own type). A run contributes one *play* to each
276
+ seat type present in its cast, and one *win* to whichever seat type the winner maps to.
277
+ Unmapped cast members (``seat_type == ""``) are not counted β€” only declared seats
278
+ appear. Sorted by ``win_rate`` desc, ``wins`` desc, ``plays`` desc, ``seat_type`` asc.
279
+ """
280
+ plays: dict[str, int] = defaultdict(int)
281
+ wins: dict[str, int] = defaultdict(int)
282
+ for entry in _ranked(entries):
283
+ if entry.scenario != scenario_name:
284
+ continue
285
+ block = _block(entry)
286
+ seats_present = {st for st in (_seat_type_for(n, block) for n in entry.cast) if st}
287
+ for seat in seats_present:
288
+ plays[seat] += 1
289
+ for seat in _winning_seat_types(entry, block):
290
+ if seat in seats_present:
291
+ wins[seat] += 1
292
+ rows = [
293
+ SeatRow(
294
+ seat_type=seat,
295
+ plays=plays[seat],
296
+ wins=wins[seat],
297
+ win_rate=_win_rate(wins[seat], plays[seat]),
298
+ )
299
+ for seat in plays
300
+ ]
301
+ rows.sort(key=lambda r: (-r.win_rate, -r.wins, -r.plays, r.seat_type))
302
+ return rows
303
+
304
+
305
+ def headline(entries: Sequence[LeaderboardEntry]) -> str | None:
306
+ """The killer demo line, or ``None`` when there isn't enough data.
307
+
308
+ Looks for the most-played *symmetric-seat* scenario (the "which model argues better"
309
+ comparison) and, within it, the two models with the most head-to-head wins. Renders
310
+ e.g. ``"MiniCPM-8B beats Gemma-12B Β· 7-3 at Debate Duel"``. Returns ``None`` when no
311
+ competitive symmetric scenario has at least two distinct models that have each won at
312
+ least once (so the line is never a hollow "0-0").
313
+ """
314
+ ranked = _ranked(entries)
315
+ if not ranked:
316
+ return None
317
+
318
+ by_scenario: dict[str, list[LeaderboardEntry]] = defaultdict(list)
319
+ for entry in ranked:
320
+ if entry.symmetric_seats: # the model-vs-model comparison only
321
+ by_scenario[entry.scenario].append(entry)
322
+ if not by_scenario:
323
+ return None
324
+
325
+ best_line: str | None = None
326
+ best_key: tuple[int, int] = (-1, -1)
327
+ for scenario in sorted(by_scenario): # ascending scan: ties resolve to the first (alphabetical) name
328
+ runs = by_scenario[scenario]
329
+ wins: dict[str, int] = defaultdict(int)
330
+ plays: dict[str, int] = defaultdict(int)
331
+ for entry in runs:
332
+ credited = _credited_models(entry)
333
+ for endpoint in {b.model_endpoint for b in entry.cast.values() if b.model_endpoint}:
334
+ plays[endpoint] += 1
335
+ if endpoint in credited:
336
+ wins[endpoint] += 1
337
+ winners = sorted((m for m in plays if wins[m] > 0), key=lambda m: (-wins[m], -plays[m], m))
338
+ if len(winners) < 2:
339
+ continue
340
+ top, runner = winners[0], winners[1]
341
+ decided = wins[top] + wins[runner]
342
+ candidate_key = (decided, wins[top])
343
+ if candidate_key > best_key: # strict: a full tie keeps the earlier (alphabetical) scenario
344
+ best_key = candidate_key
345
+ best_line = f"{_short(top)} beats {_short(runner)} Β· {wins[top]}-{wins[runner]} at {scenario}"
346
+ return best_line
347
+
348
+
349
+ # ── tiny formatting helpers ─────────────────────────────────────────────────────
350
+
351
+
352
+ def _neg_ts(value: datetime | None) -> float:
353
+ """Negated POSIX timestamp for descending sort; ``0.0`` for ``None`` (sorted last)."""
354
+ return -value.timestamp() if value is not None else 0.0
355
+
356
+
357
+ def _short(endpoint: str) -> str:
358
+ """Compact a model endpoint for the headline (``"openai/openbmb/X"`` β†’ ``"X"``)."""
359
+ return endpoint.rsplit("/", 1)[-1] if endpoint else endpoint
360
+
361
+
362
+ __all__ = [
363
+ "AgentRow",
364
+ "CompetitionBlock",
365
+ "ModelRow",
366
+ "SeatRow",
367
+ "agent_table",
368
+ "fairness_table",
369
+ "headline",
370
+ "model_table",
371
+ "scenario_sessions",
372
+ ]
src/core/leaderboard_store.py ADDED
@@ -0,0 +1,380 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Leaderboard store β€” a dedicated, durable table for competitive run results.
2
+
3
+ Unlike :mod:`src.core.leaderboard` (which *aggregates* results), this module owns the
4
+ **persistence** of one row per decided competitive run. It is deliberately **detached
5
+ from the event ledger**: the append-only ``events`` table stays the source of truth for
6
+ *what happened* in a run (every utterance, thought, verdict β€” the full logs), while this
7
+ ``leaderboard_entries`` table is a separate, denormalised record of *who won* β€” a
8
+ materialised scoreboard the Hall of Fame reads without ever folding the event log.
9
+
10
+ Why a separate table (not a pure projection)
11
+ ---------------------------------------------
12
+ A run's result is written **once**, at finish, only when it is genuinely decided β€” so the
13
+ Hall of Fame is a cheap ``SELECT`` over a small table rather than an O(all-events) fold of
14
+ the whole ledger on every page load. The two tables share one database (the same
15
+ ``DATABASE_URL`` / Postgres instance) but never share rows: ``events`` is the trace,
16
+ ``leaderboard_entries`` is the scoreboard. The ``run_id`` on each entry links a row back
17
+ to its full trace (``ledger.events_for_run(run_id)``) for replay.
18
+
19
+ The write gate (the operator's explicit requirement)
20
+ ----------------------------------------------------
21
+ A row is recorded **only if** the run is finished AND a winner with a concrete winning
22
+ model is selected. Unfinished runs, abandoned runs (a budget close with no verdict), and
23
+ runs whose winner has no bound model endpoint never produce a row β€” see
24
+ :func:`build_entry` and the call site in ``FishbowlSession.finalize``.
25
+
26
+ SQLAlchemy is imported lazily inside ``LeaderboardStore.__init__`` so importing this
27
+ module β€” and therefore ``src.core.*`` β€” never requires SQLAlchemy or a DB driver. The
28
+ :class:`LeaderboardEntry` Pydantic model is import-clean and is reused by both the store
29
+ (persistence) and :mod:`src.core.leaderboard` (aggregation).
30
+ """
31
+
32
+ from __future__ import annotations
33
+
34
+ import json
35
+ import os
36
+ from datetime import datetime, timezone
37
+ from pathlib import Path
38
+ from typing import TYPE_CHECKING, Any
39
+
40
+ from pydantic import BaseModel, ConfigDict, Field
41
+
42
+ from src.core.run_index import CastBinding, RunSummary
43
+
44
+ if TYPE_CHECKING: # types only β€” never imported at runtime on the offline path
45
+ from sqlalchemy import Engine, MetaData, Table
46
+
47
+ from src.scenarios.base import CompetitionConfig
48
+
49
+
50
+ # ── the row model (shared by the store and the aggregations) ─────────────────────
51
+
52
+
53
+ class LeaderboardEntry(BaseModel):
54
+ """One decided competitive run β€” the durable scoreboard row.
55
+
56
+ Carries everything the Hall of Fame needs without touching the event ledger: the
57
+ cast→model bindings, the winner and the model(s) behind it, the competition shape
58
+ (``kind`` / ``teams`` / ``symmetric_seats`` β€” needed for per-seat fairness), and the
59
+ run's cost/timing. ``run_id`` links the row back to its full trace for replay.
60
+ """
61
+
62
+ model_config = ConfigDict(extra="forbid")
63
+
64
+ run_id: str
65
+ session_id: str | None = None
66
+ scenario: str = ""
67
+ seed: str = ""
68
+
69
+ # Competition shape (mirrors CompetitionConfig β€” needed for seat/fairness rollups).
70
+ competition_kind: str = "none"
71
+ teams: dict[str, list[str]] | None = None
72
+ symmetric_seats: list[str] | None = None
73
+
74
+ cast: dict[str, CastBinding] = Field(default_factory=dict)
75
+
76
+ winner: str | None = None
77
+ winner_kind: str | None = None
78
+ winning_model: str | None = None
79
+ winning_models: list[str] = Field(default_factory=list)
80
+
81
+ reason: str | None = None
82
+ turns: int = 0
83
+ tokens: int = 0
84
+
85
+ started_at: datetime | None = None
86
+ finished_at: datetime | None = None
87
+ recorded_at: datetime | None = None
88
+ """When this scoreboard row was written (insert time)."""
89
+
90
+
91
+ def build_entry(summary: RunSummary, competition: "CompetitionConfig | None") -> LeaderboardEntry | None:
92
+ """Build a :class:`LeaderboardEntry` from a finished run, or ``None`` if not eligible.
93
+
94
+ The single eligibility gate (the operator's requirement): the run must be **finished**
95
+ (``finished_at`` set), have a **winner**, carry at least one **winning model**, and be
96
+ **competitive** (``competition.kind != "none"``). Returns ``None`` otherwise, so the
97
+ caller simply doesn't record a row.
98
+
99
+ ``summary`` is the :class:`~src.core.run_index.RunSummary` folded from the run's events
100
+ (it already resolves winner / winner_kind / winning_model(s) / cast / turns / tokens);
101
+ ``competition`` is the scenario's config (for the seat/team shape). This function adds
102
+ no new attribution β€” it only decides eligibility and copies the fields across.
103
+ """
104
+ kind = getattr(competition, "kind", "none") or "none"
105
+ winning_models = [m for m in (summary.winning_models or []) if m]
106
+ if summary.winning_model and summary.winning_model not in winning_models:
107
+ winning_models.append(summary.winning_model)
108
+
109
+ finished = summary.finished_at is not None
110
+ if not finished or not summary.winner or not winning_models or kind == "none":
111
+ return None
112
+
113
+ return LeaderboardEntry(
114
+ run_id=summary.run_id,
115
+ session_id=summary.session_id,
116
+ scenario=summary.scenario,
117
+ seed=summary.seed,
118
+ competition_kind=kind,
119
+ teams=getattr(competition, "teams", None),
120
+ symmetric_seats=getattr(competition, "symmetric_seats", None),
121
+ cast=dict(summary.cast),
122
+ winner=summary.winner,
123
+ winner_kind=summary.winner_kind,
124
+ winning_model=summary.winning_model,
125
+ winning_models=winning_models,
126
+ reason=summary.reason,
127
+ turns=summary.turns,
128
+ tokens=summary.tokens,
129
+ started_at=summary.started_at,
130
+ finished_at=summary.finished_at,
131
+ )
132
+
133
+
134
+ # ── the durable store (mirrors SqlAlchemyLedger's lazy-SQLAlchemy pattern) ────────
135
+
136
+
137
+ def _normalise_url(url: str | Path) -> str:
138
+ """Accept a SQLAlchemy URL or a bare filesystem path (mirrors the ledger)."""
139
+ text = str(url)
140
+ if "://" in text:
141
+ return text
142
+ if text == ":memory:":
143
+ return "sqlite://"
144
+ return f"sqlite:///{text}"
145
+
146
+
147
+ class LeaderboardStore:
148
+ """Durable store for :class:`LeaderboardEntry` rows in a dedicated table.
149
+
150
+ Backed by SQLAlchemy (Postgres or SQLite), exactly like ``SqlAlchemyLedger`` β€” but it
151
+ owns the ``leaderboard_entries`` table, never ``events``. ``record`` is an idempotent
152
+ upsert keyed on ``run_id`` (a re-finalised run β€” e.g. a budget close later superseded
153
+ by a verdict β€” replaces its row rather than duplicating it).
154
+ """
155
+
156
+ def __init__(self, url: str | Path = ":memory:") -> None:
157
+ # Lazy import: keeps src.core.* importable without SQLAlchemy installed.
158
+ from sqlalchemy import (
159
+ Column,
160
+ DateTime,
161
+ Integer,
162
+ MetaData,
163
+ String,
164
+ Table,
165
+ Text,
166
+ create_engine,
167
+ )
168
+
169
+ self._url = _normalise_url(url)
170
+ self._engine: Engine = create_engine(self._url, pool_pre_ping=True)
171
+ self._metadata: MetaData = MetaData()
172
+ self._table: Table = Table(
173
+ "leaderboard_entries",
174
+ self._metadata,
175
+ Column("offset", Integer, primary_key=True, autoincrement=True),
176
+ Column("run_id", String(64), unique=True, nullable=False, index=True),
177
+ Column("session_id", String, nullable=True, index=True),
178
+ Column("scenario", String, nullable=False, index=True),
179
+ Column("seed", String, nullable=False, server_default=""),
180
+ Column("competition_kind", String, nullable=False, server_default="none"),
181
+ Column("teams", Text, nullable=True),
182
+ Column("symmetric_seats", Text, nullable=True),
183
+ Column("cast", Text, nullable=False, server_default="{}"),
184
+ Column("winner", String, nullable=True),
185
+ Column("winner_kind", String, nullable=True),
186
+ Column("winning_model", String, nullable=True),
187
+ Column("winning_models", Text, nullable=False, server_default="[]"),
188
+ Column("reason", String, nullable=True),
189
+ Column("turns", Integer, nullable=False, server_default="0"),
190
+ Column("tokens", Integer, nullable=False, server_default="0"),
191
+ Column("started_at", DateTime(timezone=True), nullable=True),
192
+ Column("finished_at", DateTime(timezone=True), nullable=True),
193
+ Column("recorded_at", DateTime(timezone=True), nullable=False),
194
+ )
195
+ self._metadata.create_all(self._engine)
196
+
197
+ # ── write ──────────────────────────────────────────────────────────────────
198
+
199
+ def record(self, entry: LeaderboardEntry) -> LeaderboardEntry:
200
+ """Persist *entry*, upserting on ``run_id`` (idempotent β€” never duplicates a run).
201
+
202
+ Stamps ``recorded_at`` if unset. A pre-existing row for the same ``run_id`` is
203
+ replaced (delete-then-insert in one transaction), so a corrective re-finalise
204
+ keeps exactly one, current scoreboard row per run.
205
+ """
206
+ recorded = entry.model_copy(update={"recorded_at": entry.recorded_at or datetime.now(timezone.utc)})
207
+ values = self._to_row(recorded)
208
+ with self._engine.begin() as conn:
209
+ conn.execute(self._table.delete().where(self._table.c.run_id == recorded.run_id))
210
+ conn.execute(self._table.insert().values(**values))
211
+ return recorded
212
+
213
+ # ── read ───────────────────────────────────────────────────────────────────
214
+
215
+ def entries(self) -> list[LeaderboardEntry]:
216
+ """Every recorded entry, newest finish first (``run_id`` breaks ties)."""
217
+ from sqlalchemy import select
218
+
219
+ t = self._table
220
+ stmt = select(t).order_by(t.c.offset)
221
+ with self._engine.connect() as conn:
222
+ rows = conn.execute(stmt).mappings().all()
223
+ out = [self._row_to_entry(row) for row in rows]
224
+ out.sort(key=lambda e: (e.finished_at is None, _neg_ts(e.finished_at), e.run_id))
225
+ return out
226
+
227
+ def entries_for_scenario(self, scenario: str) -> list[LeaderboardEntry]:
228
+ """The recorded entries for one scenario (indexed query), newest finish first."""
229
+ from sqlalchemy import select
230
+
231
+ t = self._table
232
+ stmt = select(t).where(t.c.scenario == scenario).order_by(t.c.offset)
233
+ with self._engine.connect() as conn:
234
+ rows = conn.execute(stmt).mappings().all()
235
+ out = [self._row_to_entry(row) for row in rows]
236
+ out.sort(key=lambda e: (e.finished_at is None, _neg_ts(e.finished_at), e.run_id))
237
+ return out
238
+
239
+ def close(self) -> None:
240
+ self._engine.dispose()
241
+
242
+ # ── internal (de)serialisation ───────────────────────────────────────────────
243
+
244
+ @staticmethod
245
+ def _to_row(entry: LeaderboardEntry) -> dict[str, Any]:
246
+ return {
247
+ "run_id": entry.run_id,
248
+ "session_id": entry.session_id,
249
+ "scenario": entry.scenario,
250
+ "seed": entry.seed,
251
+ "competition_kind": entry.competition_kind,
252
+ "teams": json.dumps(entry.teams) if entry.teams is not None else None,
253
+ "symmetric_seats": (json.dumps(entry.symmetric_seats) if entry.symmetric_seats is not None else None),
254
+ "cast": json.dumps({name: binding.model_dump() for name, binding in entry.cast.items()}),
255
+ "winner": entry.winner,
256
+ "winner_kind": entry.winner_kind,
257
+ "winning_model": entry.winning_model,
258
+ "winning_models": json.dumps(list(entry.winning_models)),
259
+ "reason": entry.reason,
260
+ "turns": int(entry.turns),
261
+ "tokens": int(entry.tokens),
262
+ "started_at": _aware(entry.started_at),
263
+ "finished_at": _aware(entry.finished_at),
264
+ "recorded_at": _aware(entry.recorded_at) or datetime.now(timezone.utc),
265
+ }
266
+
267
+ @staticmethod
268
+ def _row_to_entry(row: Any) -> LeaderboardEntry:
269
+ cast_raw = _loads(row["cast"], {})
270
+ cast = {name: CastBinding(**(binding or {})) for name, binding in cast_raw.items()}
271
+ return LeaderboardEntry(
272
+ run_id=row["run_id"],
273
+ session_id=row.get("session_id"),
274
+ scenario=row["scenario"],
275
+ seed=row["seed"] or "",
276
+ competition_kind=row["competition_kind"] or "none",
277
+ teams=_loads(row.get("teams"), None),
278
+ symmetric_seats=_loads(row.get("symmetric_seats"), None),
279
+ cast=cast,
280
+ winner=row.get("winner"),
281
+ winner_kind=row.get("winner_kind"),
282
+ winning_model=row.get("winning_model"),
283
+ winning_models=_loads(row.get("winning_models"), []) or [],
284
+ reason=row.get("reason"),
285
+ turns=row["turns"] or 0,
286
+ tokens=row["tokens"] or 0,
287
+ started_at=_aware(_parse_dt(row.get("started_at"))),
288
+ finished_at=_aware(_parse_dt(row.get("finished_at"))),
289
+ recorded_at=_aware(_parse_dt(row.get("recorded_at"))),
290
+ )
291
+
292
+
293
+ # ── factory (mirrors src.core.ledger_factory β€” same DATABASE_URL, separate table) ─
294
+
295
+ # Memoise one store per resolved URL. This is what makes the no-key, in-memory
296
+ # (``sqlite://``) stage demo work end-to-end within a process: the write at finalize and
297
+ # the Hall-of-Fame read share the *same* engine (a fresh in-memory engine would be a
298
+ # different, empty database). For a file/Postgres URL it just avoids re-opening engines.
299
+ _STORES: dict[str, LeaderboardStore] = {}
300
+
301
+
302
+ def make_leaderboard_store(url: str | None = None) -> LeaderboardStore:
303
+ """Construct (or reuse) the leaderboard store for the configured database.
304
+
305
+ Resolves the same ``DATABASE_URL`` as the event ledger (a separate *table* in the
306
+ same database), so no extra configuration is needed. *url* overrides it (tests pass
307
+ ``"sqlite://"``). Raises :class:`RuntimeError` when neither is set β€” like the ledger,
308
+ the durable store is not optional.
309
+ """
310
+ resolved = url or os.getenv("DATABASE_URL")
311
+ if not resolved:
312
+ raise RuntimeError(
313
+ "DATABASE_URL is required for the leaderboard store β€” it lives in the same "
314
+ "database as the event ledger (a separate table), so set DATABASE_URL or pass "
315
+ "an explicit url to make_leaderboard_store()."
316
+ )
317
+ from src.core.ledger_factory import _normalize_db_url
318
+
319
+ key = _normalize_db_url(resolved)
320
+ store = _STORES.get(key)
321
+ if store is None:
322
+ store = LeaderboardStore(key)
323
+ _STORES[key] = store
324
+ return store
325
+
326
+
327
+ def _reset_store_cache() -> None:
328
+ """Drop memoised stores (test hook β€” production never needs this)."""
329
+ for store in _STORES.values():
330
+ try:
331
+ store.close()
332
+ except Exception: # pragma: no cover - best-effort cleanup
333
+ pass
334
+ _STORES.clear()
335
+
336
+
337
+ # ── tiny datetime helpers (mirror the ledger's tz coercion) ──────────────────────
338
+
339
+
340
+ def _aware(dt: datetime | None) -> datetime | None:
341
+ """Coerce a datetime to tz-aware UTC; pass ``None`` through (SQLite drops tzinfo)."""
342
+ if dt is None:
343
+ return None
344
+ if dt.tzinfo is None:
345
+ return dt.replace(tzinfo=timezone.utc)
346
+ return dt
347
+
348
+
349
+ def _parse_dt(value: Any) -> datetime | None:
350
+ if value is None or isinstance(value, datetime):
351
+ return value
352
+ try:
353
+ return datetime.fromisoformat(str(value))
354
+ except (ValueError, TypeError): # pragma: no cover - defensive
355
+ return None
356
+
357
+
358
+ def _neg_ts(value: datetime | None) -> float:
359
+ """Negated POSIX timestamp for descending sort; ``0.0`` for ``None`` (sorted last)."""
360
+ return -value.timestamp() if value is not None else 0.0
361
+
362
+
363
+ def _loads(value: Any, default: Any) -> Any:
364
+ """JSON-decode a stored text column, tolerating already-decoded values / nulls."""
365
+ if value is None:
366
+ return default
367
+ if isinstance(value, (dict, list)):
368
+ return value
369
+ try:
370
+ return json.loads(value)
371
+ except (ValueError, TypeError): # pragma: no cover - defensive
372
+ return default
373
+
374
+
375
+ __all__ = [
376
+ "LeaderboardEntry",
377
+ "LeaderboardStore",
378
+ "build_entry",
379
+ "make_leaderboard_store",
380
+ ]
src/ui/fishbowl/app.py CHANGED
@@ -49,6 +49,18 @@ except Exception: # pragma: no cover - degrade gracefully if the archive unit i
49
  return "β–Ά (run)"
50
 
51
 
 
 
 
 
 
 
 
 
 
 
 
 
52
  # ── loop-safety backstop ────────────────────────────────────────────────────────
53
  # Belt-and-suspenders against a runaway autoplay loop: even when the governor never
54
  # trips (e.g. a generous budget) the timer halts after this many consecutive auto-ticks
@@ -709,6 +721,8 @@ def build_app() -> gr.Blocks:
709
  show_handles = build_show()
710
  with gr.Tab("Telemetry", id="telemetry"):
711
  build_telemetry()
 
 
712
 
713
  # CRT foreground layers (scanlines + vignette, above content, click-through).
714
  gr.HTML(_CRT_FG_HTML)
@@ -717,6 +731,23 @@ def build_app() -> gr.Blocks:
717
  # exists, so it reads the live panes through this ref.
718
  archive_refs["show_handles"] = show_handles or {}
719
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
720
  _wire(
721
  tabs=tabs,
722
  lab_handles=lab_handles or {},
 
49
  return "β–Ά (run)"
50
 
51
 
52
+ try: # hall of fame: the competitive high-score board (Workstream 6.2 + 6.3)
53
+ from src.ui.fishbowl.hall_of_fame import build_hall_of_fame, wire_sessions_render
54
+ except Exception: # pragma: no cover - degrade gracefully if the unit is absent
55
+
56
+ def build_hall_of_fame() -> dict:
57
+ gr.HTML('<div class="fishbowl"><div class="fishbowl-placeholder">πŸ† Hall of Fame β€” coming soon.</div></div>')
58
+ return {}
59
+
60
+ def wire_sessions_render(*_args, **_kwargs) -> None:
61
+ return None
62
+
63
+
64
  # ── loop-safety backstop ────────────────────────────────────────────────────────
65
  # Belt-and-suspenders against a runaway autoplay loop: even when the governor never
66
  # trips (e.g. a generous budget) the timer halts after this many consecutive auto-ticks
 
721
  show_handles = build_show()
722
  with gr.Tab("Telemetry", id="telemetry"):
723
  build_telemetry()
724
+ with gr.Tab("πŸ† Hall of Fame", id="hall"):
725
+ hall_handles = build_hall_of_fame()
726
 
727
  # CRT foreground layers (scanlines + vignette, above content, click-through).
728
  gr.HTML(_CRT_FG_HTML)
 
731
  # exists, so it reads the live panes through this ref.
732
  archive_refs["show_handles"] = show_handles or {}
733
 
734
+ # The Hall of Fame's Replay buttons target the Show panes; wire them after all
735
+ # tabs build (mirrors the Archive's deferred show_handles trick at :644).
736
+ wire_sessions_render(
737
+ hall_handles or {},
738
+ refs=archive_refs,
739
+ tabs=tabs,
740
+ states={
741
+ "session": session_state,
742
+ "k": k_state,
743
+ "scenario": scenario_state,
744
+ "layout": layout_state,
745
+ "mind": mind_reader_state,
746
+ "stopped": stopped_state,
747
+ "ticks": tick_count_state,
748
+ },
749
+ )
750
+
751
  _wire(
752
  tabs=tabs,
753
  lab_handles=lab_handles or {},
src/ui/fishbowl/hall_of_fame.py ADDED
@@ -0,0 +1,798 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """HALL OF FAME β€” the phosphor-arcade high-score board (Workstream 6.2 + 6.3).
2
+
3
+ A cheerful, screenshot-worthy scoreboard for the competitive results the engine has
4
+ recorded. It is a *pure read surface* over the dedicated leaderboard table (ADR-0035):
5
+ every number on the board comes from the aggregations in :mod:`src.core.leaderboard`
6
+ (``headline`` / ``model_table`` / ``scenario_sessions`` / ``fairness_table``), which fold
7
+ the :class:`~src.core.leaderboard_store.LeaderboardEntry` rows persisted in
8
+ ``leaderboard_entries`` β€” a table *detached from the event ledger* (the ``events`` log
9
+ stays the trace; this is the materialised scoreboard).
10
+
11
+ Design β€” "Phosphor Arcade High-Score Board"
12
+ -------------------------------------------
13
+ The tab lives inside the existing ``.fishbowl`` CRT-phosphor scope and reuses the
14
+ theme's design tokens, fonts (Martian Mono display, IBM Plex Mono body) and existing
15
+ classes (``eyebrow`` / ``chip`` / ``panel`` / verdict + winner chrome). Champions glow
16
+ **gold/amber** against the cool teal CRT β€” that warm/cool contrast is the whole
17
+ identity. Only two scoped vars are introduced (``--gold`` / ``--gold-soft``) plus the
18
+ Hall-of-Fame layout classes, all under ``.fishbowl.fb-hall``.
19
+
20
+ Everything here is **offline-safe**: pure HTML strings, an inlined ``<style>`` block,
21
+ no network calls and no external assets. When the store is empty or unconfigured (the
22
+ deterministic stub path before any competitive run finishes) every pane degrades to a
23
+ cheerful empty state rather than crashing.
24
+
25
+ The per-row **Replay** button mirrors the Lab's Archive drawer exactly: it turns a run
26
+ into a read-only :class:`~src.ui.fishbowl.session.ReplaySession` via
27
+ :func:`src.ui.fishbowl.archive.load_replay` and pushes it into the Show transport
28
+ through the same ``gr.render`` Load-button contract (see ``build_hall_of_fame``).
29
+ """
30
+
31
+ from __future__ import annotations
32
+
33
+ import html as _html
34
+ from datetime import datetime
35
+
36
+ import gradio as gr
37
+
38
+ from src.core.leaderboard import (
39
+ ModelRow,
40
+ SeatRow,
41
+ fairness_table,
42
+ headline,
43
+ model_table,
44
+ scenario_sessions,
45
+ )
46
+ from src.core.leaderboard_store import LeaderboardEntry
47
+ from src.core.registry import default_registry
48
+
49
+
50
+ # ── data access (defensive: empty store / no backend β†’ empty projections) ────────
51
+
52
+
53
+ def _entries() -> list[LeaderboardEntry]:
54
+ """Every scoreboard row from the dedicated leaderboard table, or ``[]`` if unavailable.
55
+
56
+ Reads the ``leaderboard_entries`` table (ADR-0035) β€” *detached from the event ledger*
57
+ β€” via :func:`make_leaderboard_store`. Defensive on every axis: a missing/unconfigured
58
+ store or any read error degrades to an empty list so the board shows its cheerful empty
59
+ state instead of raising.
60
+ """
61
+ try:
62
+ from src.core.leaderboard_store import make_leaderboard_store
63
+
64
+ return list(make_leaderboard_store().entries())
65
+ except Exception: # pragma: no cover - no store configured / read error (defensive)
66
+ return []
67
+
68
+
69
+ def competitive_scenarios() -> list[tuple[str, str]]:
70
+ """``(title, internal_name)`` pairs for scenarios whose ``competition.kind != none``.
71
+
72
+ These are the only worlds that can ever crown a champion, so the scenario picker
73
+ is scoped to them. Returns ``[]`` when the registry can't be read.
74
+ """
75
+ try:
76
+ registry = default_registry()
77
+ except Exception: # pragma: no cover - registry unavailable (defensive)
78
+ return []
79
+ out: list[tuple[str, str]] = []
80
+ for name in sorted(registry.scenarios):
81
+ cfg = registry.scenarios[name]
82
+ comp = getattr(cfg, "competition", None)
83
+ if getattr(comp, "kind", "none") not in ("none", None):
84
+ out.append(((getattr(cfg, "title", "") or name), name))
85
+ return out
86
+
87
+
88
+ # ── tiny formatting helpers ──────────────────────────────────────────────────────
89
+
90
+
91
+ def _esc(text: object) -> str:
92
+ """HTML-escape any value (defensive β€” inputs may be ``None`` / non-str)."""
93
+ return _html.escape(str(text if text is not None else ""))
94
+
95
+
96
+ def _short_model(endpoint: str) -> str:
97
+ """Compact a model endpoint to its last path segment (``a/b/Model`` β†’ ``Model``)."""
98
+ if not endpoint:
99
+ return "β€”"
100
+ return endpoint.rsplit("/", 1)[-1] or endpoint
101
+
102
+
103
+ def _pct(rate: float) -> str:
104
+ """A win-rate float as a whole-percent label (``0.7`` β†’ ``"70%"``)."""
105
+ try:
106
+ return f"{round(rate * 100)}%"
107
+ except Exception: # pragma: no cover - non-numeric (defensive)
108
+ return "0%"
109
+
110
+
111
+ def _bar_width(rate: float) -> float:
112
+ """Clamp a win-rate to a 0–100 bar width."""
113
+ try:
114
+ return max(0.0, min(100.0, float(rate) * 100.0))
115
+ except Exception: # pragma: no cover - non-numeric (defensive)
116
+ return 0.0
117
+
118
+
119
+ def _short_id(run_id: str) -> str:
120
+ """A glanceable run handle β€” last 4 of its uuid (matches the Archive)."""
121
+ tail = run_id.replace("-", "")[-4:] if run_id else "????"
122
+ return f"#{tail}"
123
+
124
+
125
+ def _fmt_when(value: datetime | None) -> str:
126
+ """A compact ``Jun 12 Β· 14:03`` timestamp, ``β€”`` when missing."""
127
+ if not isinstance(value, datetime):
128
+ return "β€”"
129
+ try:
130
+ return value.strftime("%b %-d Β· %H:%M")
131
+ except ValueError: # pragma: no cover - platforms without %-d
132
+ return value.strftime("%b %d Β· %H:%M")
133
+
134
+
135
+ def _fmt_tokens(tokens: int) -> str:
136
+ try:
137
+ tokens = int(tokens)
138
+ except Exception: # pragma: no cover - non-numeric (defensive)
139
+ return "0"
140
+ if tokens >= 1000:
141
+ return f"{tokens / 1000:.1f}k"
142
+ return str(tokens)
143
+
144
+
145
+ _REASON_LABEL = {
146
+ "verdict": "verdict",
147
+ "budget": "budget",
148
+ "tick_cap": "tick-cap",
149
+ "user_stop": "stopped",
150
+ }
151
+
152
+
153
+ def _cast_models(row: LeaderboardEntry) -> str:
154
+ """A compact roster of the distinct models that played a session."""
155
+ seen: list[str] = []
156
+ for binding in (row.cast or {}).values():
157
+ endpoint = getattr(binding, "model_endpoint", None)
158
+ if endpoint:
159
+ short = _short_model(endpoint)
160
+ if short not in seen:
161
+ seen.append(short)
162
+ return ", ".join(seen) if seen else "β€”"
163
+
164
+
165
+ # ── scoped stylesheet (offline; reuses theme tokens, adds only --gold) ────────────
166
+
167
+ _HALL_STYLE = """
168
+ <style>
169
+ .fishbowl.fb-hall { display:block; }
170
+ .fishbowl.fb-hall {
171
+ --gold:#ffcf6b; /* reuse the theme's --amber gold token */
172
+ --gold-soft:rgba(255,207,107,0.18);
173
+ }
174
+ .fishbowl.fb-hall .hall-wrap { display:flex; flex-direction:column; gap:18px; }
175
+
176
+ /* Headline marquee β€” the killer demo line, big and glowing. */
177
+ .fishbowl.fb-hall .hall-headline {
178
+ position:relative; overflow:hidden;
179
+ padding:20px 26px; border-radius:16px;
180
+ border:1px solid rgba(255,207,107,0.42);
181
+ background:
182
+ radial-gradient(120% 120% at 50% -20%, var(--gold-soft), transparent 60%),
183
+ rgba(6,20,27,0.96);
184
+ box-shadow:0 0 44px rgba(255,207,107,0.22), inset 0 1px 0 rgba(255,207,107,0.12);
185
+ }
186
+ .fishbowl.fb-hall .hall-headline .eyebrow { color:var(--gold); margin-bottom:8px; }
187
+ .fishbowl.fb-hall .hall-line {
188
+ font-family:var(--font-display); font-weight:800;
189
+ font-size:clamp(20px, 3vw, 34px); line-height:1.18;
190
+ color:var(--ink);
191
+ text-shadow:0 0 18px rgba(255,207,107,0.45);
192
+ }
193
+ .fishbowl.fb-hall .hall-line .hl-gold { color:var(--gold); }
194
+ /* A shimmer sweep across the headline (CSS-only). */
195
+ .fishbowl.fb-hall .hall-headline::after {
196
+ content:""; position:absolute; inset:0; pointer-events:none;
197
+ background:linear-gradient(105deg, transparent 30%,
198
+ rgba(255,207,107,0.16) 48%, transparent 66%);
199
+ transform:translateX(-120%);
200
+ animation:hallShimmer 5.5s ease-in-out infinite;
201
+ }
202
+ @keyframes hallShimmer { 0%,55% { transform:translateX(-120%);} 78%,100% { transform:translateX(120%);} }
203
+
204
+ /* Empty states β€” cheerful, never a dead screen. */
205
+ .fishbowl.fb-hall .hall-empty {
206
+ padding:22px 26px; border-radius:14px;
207
+ border:1px dashed var(--line); background:var(--panel);
208
+ }
209
+ .fishbowl.fb-hall .hall-empty .eyebrow { color:var(--gold); margin-bottom:8px; }
210
+ .fishbowl.fb-hall .hall-empty .he-body { color:var(--ink-mid); line-height:1.55; }
211
+
212
+ /* Section panels reuse the theme's card chrome. */
213
+ .fishbowl.fb-hall .hall-panel {
214
+ padding:18px 20px 20px; border-radius:var(--r-lg);
215
+ border:1px solid var(--line); background:var(--panel);
216
+ box-shadow:inset 0 1px 0 rgba(120,222,214,0.07), 0 16px 38px -32px rgba(0,0,0,0.9);
217
+ }
218
+ .fishbowl.fb-hall .hall-panel > .eyebrow { margin-bottom:14px; }
219
+
220
+ /* ── Podium (top 3): gold / silver / bronze phosphor ── */
221
+ .fishbowl.fb-hall .podium {
222
+ display:grid; grid-template-columns:1fr 1.2fr 1fr; gap:14px; align-items:end;
223
+ margin-bottom:18px;
224
+ }
225
+ .fishbowl.fb-hall .podium-slot {
226
+ display:flex; flex-direction:column; align-items:center; gap:8px;
227
+ padding:16px 12px; border-radius:14px; text-align:center;
228
+ border:1px solid var(--line); background:rgba(6,20,27,0.7);
229
+ animation:hallRise 0.5s cubic-bezier(0.2,0.9,0.3,1) both;
230
+ }
231
+ .fishbowl.fb-hall .podium-slot .ps-medal { font-size:30px; line-height:1; }
232
+ .fishbowl.fb-hall .podium-slot .ps-name {
233
+ font-family:var(--font-display); font-weight:700; font-size:14px; color:var(--ink);
234
+ word-break:break-word;
235
+ }
236
+ .fishbowl.fb-hall .podium-slot .ps-rate {
237
+ font-family:var(--font-display); font-weight:800; font-variant-numeric:tabular-nums;
238
+ font-size:22px;
239
+ }
240
+ .fishbowl.fb-hall .podium-slot .ps-sub { font-size:11px; color:var(--ink-dim); }
241
+ /* #1 β€” gold, taller, with a soft pulse. */
242
+ .fishbowl.fb-hall .podium-slot.rank-1 {
243
+ border-color:rgba(255,207,107,0.5);
244
+ background:radial-gradient(120% 120% at 50% -10%, var(--gold-soft), transparent 60%), rgba(6,20,27,0.92);
245
+ box-shadow:0 0 40px rgba(255,207,107,0.3); padding-top:26px; animation-delay:0.05s;
246
+ }
247
+ .fishbowl.fb-hall .podium-slot.rank-1 .ps-rate { color:var(--gold); text-shadow:0 0 14px rgba(255,207,107,0.6); }
248
+ .fishbowl.fb-hall .podium-slot.rank-1 .ps-medal { animation:hallChampPulse 2.4s ease-in-out infinite; }
249
+ .fishbowl.fb-hall .podium-slot.rank-2 {
250
+ border-color:rgba(159,220,210,0.45); animation-delay:0.16s;
251
+ }
252
+ .fishbowl.fb-hall .podium-slot.rank-2 .ps-rate { color:var(--ink-mid); }
253
+ .fishbowl.fb-hall .podium-slot.rank-3 {
254
+ border-color:rgba(255,143,125,0.4); animation-delay:0.27s;
255
+ }
256
+ .fishbowl.fb-hall .podium-slot.rank-3 .ps-rate { color:var(--coral); }
257
+
258
+ /* ── Ranked table with glowing win-rate bars ── */
259
+ .fishbowl.fb-hall .hall-table { display:flex; flex-direction:column; gap:2px; }
260
+ .fishbowl.fb-hall .ht-head, .fishbowl.fb-hall .ht-row {
261
+ display:grid; grid-template-columns:46px 1.6fr 0.7fr 0.7fr 1.5fr; gap:12px;
262
+ align-items:center; padding:9px 12px; border-radius:var(--r);
263
+ }
264
+ .fishbowl.fb-hall .ht-head {
265
+ font-family:var(--font-display); font-size:8.5px; letter-spacing:0.2em;
266
+ text-transform:uppercase; color:var(--ink-faint);
267
+ border-bottom:1px solid var(--line-soft); border-radius:0;
268
+ }
269
+ .fishbowl.fb-hall .ht-row { animation:hallRise 0.45s ease both; }
270
+ .fishbowl.fb-hall .ht-row:hover { background:var(--glass); }
271
+ .fishbowl.fb-hall .ht-row .ht-rank {
272
+ font-family:var(--font-display); font-weight:700; color:var(--ink-dim);
273
+ font-variant-numeric:tabular-nums;
274
+ }
275
+ .fishbowl.fb-hall .ht-row.is-champ .ht-rank { color:var(--gold); }
276
+ .fishbowl.fb-hall .ht-row .ht-model {
277
+ font-family:var(--font-display); font-weight:600; color:var(--ink); word-break:break-word;
278
+ }
279
+ .fishbowl.fb-hall .ht-row .ht-num { font-variant-numeric:tabular-nums; color:var(--ink-mid); }
280
+ .fishbowl.fb-hall .ht-bar {
281
+ position:relative; height:14px; border-radius:999px;
282
+ background:rgba(120,222,214,0.08); overflow:hidden;
283
+ }
284
+ .fishbowl.fb-hall .ht-bar .ht-fill {
285
+ position:absolute; inset:0 auto 0 0; height:100%; border-radius:999px;
286
+ background:linear-gradient(90deg, rgba(255,207,107,0.55), var(--gold));
287
+ box-shadow:0 0 12px rgba(255,207,107,0.45);
288
+ }
289
+ .fishbowl.fb-hall .ht-bar .ht-bar-label {
290
+ position:absolute; right:8px; top:50%; transform:translateY(-50%);
291
+ font-family:var(--font-display); font-size:9px; font-weight:700;
292
+ color:var(--bg-0); text-shadow:0 0 3px rgba(255,207,107,0.8);
293
+ font-variant-numeric:tabular-nums;
294
+ }
295
+
296
+ /* ── Sessions table ── */
297
+ .fishbowl.fb-hall .sess-head, .fishbowl.fb-hall .sess-row {
298
+ display:grid; grid-template-columns:0.7fr 1.4fr 1.1fr 0.9fr 0.7fr 0.9fr;
299
+ gap:10px; align-items:center; padding:9px 12px; border-radius:var(--r);
300
+ }
301
+ .fishbowl.fb-hall .sess-head {
302
+ font-family:var(--font-display); font-size:8.5px; letter-spacing:0.2em;
303
+ text-transform:uppercase; color:var(--ink-faint);
304
+ border-bottom:1px solid var(--line-soft); border-radius:0;
305
+ }
306
+ .fishbowl.fb-hall .sess-row:hover { background:var(--glass); }
307
+ .fishbowl.fb-hall .sess-row .sr-id {
308
+ font-family:var(--font-display); color:var(--ink-dim); font-size:12px;
309
+ }
310
+ .fishbowl.fb-hall .sess-row .sr-cast { color:var(--ink-mid); font-size:12px; word-break:break-word; }
311
+ .fishbowl.fb-hall .sess-row .sr-num { font-variant-numeric:tabular-nums; color:var(--ink-mid); }
312
+ .fishbowl.fb-hall .sr-won {
313
+ display:inline-flex; align-items:center; gap:5px;
314
+ font-family:var(--font-display); font-size:11px; font-weight:700;
315
+ text-transform:uppercase; letter-spacing:0.08em;
316
+ color:var(--gold); text-shadow:0 0 10px rgba(255,207,107,0.5);
317
+ }
318
+ .fishbowl.fb-hall .sr-reason { color:var(--ink-faint); font-size:11px; }
319
+
320
+ /* ── Fairness footnote panel ── */
321
+ .fishbowl.fb-hall .fair-grid {
322
+ display:grid; grid-template-columns:1.4fr 0.7fr 0.7fr 1.5fr; gap:10px;
323
+ align-items:center; padding:8px 12px; border-radius:var(--r);
324
+ }
325
+ .fishbowl.fb-hall .fair-head {
326
+ font-family:var(--font-display); font-size:8.5px; letter-spacing:0.2em;
327
+ text-transform:uppercase; color:var(--ink-faint);
328
+ border-bottom:1px solid var(--line-soft); border-radius:0;
329
+ }
330
+ .fishbowl.fb-hall .fair-grid:hover { background:var(--glass); }
331
+ .fishbowl.fb-hall .fair-seat { font-family:var(--font-display); font-weight:600; color:var(--ink); }
332
+ .fishbowl.fb-hall .fair-num { font-variant-numeric:tabular-nums; color:var(--ink-mid); }
333
+ .fishbowl.fb-hall .fair-bar { position:relative; height:10px; border-radius:999px;
334
+ background:rgba(120,222,214,0.08); overflow:hidden; }
335
+ .fishbowl.fb-hall .fair-bar .fair-fill { position:absolute; inset:0 auto 0 0; height:100%;
336
+ border-radius:999px; background:linear-gradient(90deg, var(--cyan), var(--teal));
337
+ box-shadow:0 0 8px rgba(79,230,210,0.4); }
338
+ .fishbowl.fb-hall .fair-note {
339
+ margin-top:14px; padding:10px 12px; border-radius:var(--r);
340
+ border:1px solid var(--line-soft); background:rgba(255,207,107,0.06);
341
+ color:var(--ink-dim); font-size:11px; line-height:1.5;
342
+ }
343
+
344
+ @keyframes hallRise { from { opacity:0; transform:translateY(8px);} to { opacity:1; transform:none;} }
345
+ @keyframes hallChampPulse {
346
+ 0%,100% { transform:scale(1); filter:drop-shadow(0 0 6px rgba(255,207,107,0.5)); }
347
+ 50% { transform:scale(1.12); filter:drop-shadow(0 0 14px rgba(255,207,107,0.85)); }
348
+ }
349
+ @media (prefers-reduced-motion: reduce) {
350
+ .fishbowl.fb-hall .hall-headline::after,
351
+ .fishbowl.fb-hall .podium-slot,
352
+ .fishbowl.fb-hall .podium-slot.rank-1 .ps-medal,
353
+ .fishbowl.fb-hall .ht-row { animation:none !important; }
354
+ }
355
+ </style>
356
+ """
357
+
358
+
359
+ # ── render functions (pure HTML strings, defensive, .fishbowl-scoped) ─────────────
360
+
361
+
362
+ def _wrap(inner: str) -> str:
363
+ """Wrap a Hall-of-Fame pane in the ``.fishbowl.fb-hall`` scope root.
364
+
365
+ Mirrors :func:`src.ui.fishbowl.app._fishbowl` so the theater stylesheet (and the
366
+ inlined Hall-of-Fame ``<style>``) win over Gradio's cascade.
367
+ """
368
+ return f'<div class="fishbowl fb-hall">{_HALL_STYLE}{inner}</div>'
369
+
370
+
371
+ def render_headline(entries) -> str:
372
+ """The big glowing demo line from :func:`leaderboard.headline`, or an empty state.
373
+
374
+ Renders e.g. ``MiniCPM-8B beats Gemma-12B Β· 7-3 at Debate Duel`` with the two model
375
+ names gilded. When ``headline`` returns ``None`` (no symmetric scenario with two
376
+ models that have each won), shows a cheerful "no champions yet" call to action.
377
+ """
378
+ try:
379
+ line = headline(entries)
380
+ except Exception: # pragma: no cover - projection error (defensive)
381
+ line = None
382
+ if not line:
383
+ return _wrap(
384
+ '<div class="hall-empty">'
385
+ '<div class="eyebrow">&#127942; Hall of Fame</div>'
386
+ '<div class="he-body">No champions crowned yet β€” run a competitive scenario '
387
+ "(a versus duel or a judged contest) to fill the Hall.</div>"
388
+ "</div>"
389
+ )
390
+ # Gild the two model names: the line is "<A> beats <B> Β· X-Y at <Scenario>".
391
+ safe = _esc(line)
392
+ if " beats " in line:
393
+ left, _, rest = line.partition(" beats ")
394
+ runner, _, tail = rest.partition(" Β· ")
395
+ safe = (
396
+ f'<span class="hl-gold">{_esc(left)}</span> beats '
397
+ f'<span class="hl-gold">{_esc(runner)}</span> Β· {_esc(tail)}'
398
+ )
399
+ return _wrap(
400
+ '<div class="hall-headline">'
401
+ '<div class="eyebrow">&#127942; Today\'s Champion</div>'
402
+ f'<div class="hall-line">{safe}</div>'
403
+ "</div>"
404
+ )
405
+
406
+
407
+ def _podium_html(rows: list[ModelRow]) -> str:
408
+ """The gold/silver/bronze top-3 podium (visual order: 2nd Β· 1st Β· 3rd)."""
409
+ if not rows:
410
+ return ""
411
+ top = rows[:3]
412
+ medals = {0: "&#129351;", 1: "&#129352;", 2: "&#129353;"} # πŸ₯‡ πŸ₯ˆ πŸ₯‰
413
+
414
+ def slot(idx: int) -> str:
415
+ if idx >= len(top):
416
+ return '<div class="podium-slot" style="visibility:hidden"></div>'
417
+ row = top[idx]
418
+ return (
419
+ f'<div class="podium-slot rank-{idx + 1}">'
420
+ f'<div class="ps-medal">{medals.get(idx, "")}</div>'
421
+ f'<div class="ps-name">{_esc(_short_model(row.model))}</div>'
422
+ f'<div class="ps-rate">{_pct(row.win_rate)}</div>'
423
+ f'<div class="ps-sub">{row.wins}/{row.plays} won</div>'
424
+ "</div>"
425
+ )
426
+
427
+ # Visual order puts #1 in the centre, raised.
428
+ order = [1, 0, 2]
429
+ return '<div class="podium">' + "".join(slot(i) for i in order) + "</div>"
430
+
431
+
432
+ def render_model_board(entries) -> str:
433
+ """The model podium + ranked table with glowing gold win-rate bars.
434
+
435
+ Folds :func:`leaderboard.model_table` (already sorted by win-rate) into the podium
436
+ for the top 3 and a ranked table below. Each row carries rank, model, plays, wins
437
+ and a horizontal bar whose width is the win-rate. Empty store β†’ empty state.
438
+ """
439
+ try:
440
+ rows = model_table(entries)
441
+ except Exception: # pragma: no cover - projection error (defensive)
442
+ rows = []
443
+ if not rows:
444
+ return _wrap(
445
+ '<div class="hall-empty">'
446
+ '<div class="eyebrow">&#128202; Model Leaderboard</div>'
447
+ '<div class="he-body">No decided competitive runs yet. Once a versus or judged '
448
+ "scenario crowns a winner, the models climb the board here.</div>"
449
+ "</div>"
450
+ )
451
+ head = '<div class="ht-head"><div>#</div><div>Model</div><div>Plays</div><div>Wins</div><div>Win rate</div></div>'
452
+ body_rows: list[str] = []
453
+ for idx, row in enumerate(rows):
454
+ champ = " is-champ" if idx == 0 else ""
455
+ width = _bar_width(row.win_rate)
456
+ body_rows.append(
457
+ f'<div class="ht-row{champ}" style="animation-delay:{min(idx, 12) * 0.04:.2f}s">'
458
+ f'<div class="ht-rank">{idx + 1}</div>'
459
+ f'<div class="ht-model">{_esc(_short_model(row.model))}</div>'
460
+ f'<div class="ht-num">{row.plays}</div>'
461
+ f'<div class="ht-num">{row.wins}</div>'
462
+ '<div class="ht-bar">'
463
+ f'<div class="ht-fill" style="width:{width:.1f}%"></div>'
464
+ f'<div class="ht-bar-label">{_pct(row.win_rate)}</div>'
465
+ "</div>"
466
+ "</div>"
467
+ )
468
+ return _wrap(
469
+ '<div class="hall-panel">'
470
+ '<div class="eyebrow">&#128202; Model Leaderboard</div>'
471
+ + _podium_html(rows)
472
+ + '<div class="hall-table">'
473
+ + head
474
+ + "".join(body_rows)
475
+ + "</div></div>"
476
+ )
477
+
478
+
479
+ def render_sessions(entries, scenario_name: str) -> str:
480
+ """The sessions table for *scenario_name* (without the Replay buttons).
481
+
482
+ Lists each finished, won, competitive run newest-first: a short id, the cast's
483
+ models, the gold WON badge, why it ended, turns/tokens and the date. The Replay
484
+ buttons themselves are real Gradio components rendered alongside this HTML by
485
+ ``build_hall_of_fame`` (gr.HTML can't host working buttons); this pane is the
486
+ glanceable record. Empty / non-competitive scenario β†’ empty state.
487
+ """
488
+ if not scenario_name:
489
+ return _wrap(
490
+ '<div class="hall-empty">'
491
+ '<div class="eyebrow">&#9654; Sessions</div>'
492
+ '<div class="he-body">Pick a competitive world above to see its decided '
493
+ "matches and replay any of them.</div>"
494
+ "</div>"
495
+ )
496
+ try:
497
+ rows = scenario_sessions(entries, scenario_name)
498
+ except Exception: # pragma: no cover - projection error (defensive)
499
+ rows = []
500
+ title = _esc(_scenario_title(scenario_name))
501
+ if not rows:
502
+ return _wrap(
503
+ '<div class="hall-panel">'
504
+ f'<div class="eyebrow">&#9654; {title} Β· Sessions</div>'
505
+ '<div class="hall-empty" style="border:none;background:transparent;padding:8px 0">'
506
+ '<div class="he-body">No decided matches in this world yet. Run it as a '
507
+ "competition and the results land here, replayable.</div>"
508
+ "</div></div>"
509
+ )
510
+ head = (
511
+ '<div class="sess-head">'
512
+ "<div>Run</div><div>Cast models</div><div>Winner</div>"
513
+ "<div>Why</div><div>Turns</div><div>When</div>"
514
+ "</div>"
515
+ )
516
+ body_rows = "".join(_session_row_html(row) for row in rows)
517
+ return _wrap(
518
+ '<div class="hall-panel">'
519
+ f'<div class="eyebrow">&#9654; {title} Β· Sessions</div>'
520
+ f'<div class="hall-table">{head}{body_rows}</div>'
521
+ "</div>"
522
+ )
523
+
524
+
525
+ def _session_row_html(row: LeaderboardEntry) -> str:
526
+ """One sessions-table row (glanceable; the Replay button sits beside it)."""
527
+ reason = _REASON_LABEL.get(row.reason or "", row.reason or "β€”")
528
+ won = (
529
+ f'<span class="sr-won">&#127942; {_esc(row.winner)}</span>'
530
+ if row.winner
531
+ else '<span class="sr-reason">β€”</span>'
532
+ )
533
+ return (
534
+ '<div class="sess-row">'
535
+ f'<div class="sr-id">{_esc(_short_id(row.run_id))}</div>'
536
+ f'<div class="sr-cast">{_esc(_cast_models(row))}</div>'
537
+ f"<div>{won}</div>"
538
+ f'<div class="sr-reason">{_esc(reason)}</div>'
539
+ f'<div class="sr-num">{row.turns}t Β· {_esc(_fmt_tokens(row.tokens))}</div>'
540
+ f'<div class="sr-num">{_esc(_fmt_when(row.finished_at))}</div>'
541
+ "</div>"
542
+ )
543
+
544
+
545
+ def render_fairness(entries, scenario_name: str) -> str:
546
+ """The 6.3 fairness footnote: win rate per seat type, with the asymmetry note.
547
+
548
+ Folds :func:`leaderboard.fairness_table` (only *declared* seats β€” teams /
549
+ symmetric seats; judges and other unmapped seats are excluded by design). A
550
+ footnote reminds viewers that asymmetric seats (spy vs herd) skew raw win rates.
551
+ Empty β†’ a quiet empty state.
552
+ """
553
+ if not scenario_name:
554
+ return ""
555
+ try:
556
+ rows = fairness_table(entries, scenario_name)
557
+ except Exception: # pragma: no cover - projection error (defensive)
558
+ rows = []
559
+ if not rows:
560
+ return _wrap(
561
+ '<div class="hall-panel">'
562
+ '<div class="eyebrow">&#9878;&#65039; Seat Fairness</div>'
563
+ '<div class="he-body" style="color:var(--ink-mid)">No per-seat data yet β€” '
564
+ "this footnote fills in once this world has decided competitive runs.</div>"
565
+ "</div>"
566
+ )
567
+ head = (
568
+ '<div class="fair-grid fair-head"><div>Seat type</div><div>Plays</div><div>Wins</div><div>Win rate</div></div>'
569
+ )
570
+ body = "".join(_fairness_row_html(row) for row in rows)
571
+ note = (
572
+ '<div class="fair-note">&#9888;&#65039; Seats are not always symmetric β€” a spy '
573
+ "faces a whole herd, a lone debater a panel. Raw seat win rates show the structural "
574
+ "tilt of the game, not just who played well.</div>"
575
+ )
576
+ return _wrap(
577
+ '<div class="hall-panel">'
578
+ '<div class="eyebrow">&#9878;&#65039; Seat Fairness</div>'
579
+ f'<div class="hall-table">{head}{body}</div>'
580
+ f"{note}</div>"
581
+ )
582
+
583
+
584
+ def _fairness_row_html(row: SeatRow) -> str:
585
+ """One fairness row: seat type, plays, wins, and a cyan win-rate bar."""
586
+ width = _bar_width(row.win_rate)
587
+ return (
588
+ '<div class="fair-grid">'
589
+ f'<div class="fair-seat">{_esc(row.seat_type)}</div>'
590
+ f'<div class="fair-num">{row.plays}</div>'
591
+ f'<div class="fair-num">{row.wins}</div>'
592
+ '<div class="fair-bar">'
593
+ f'<div class="fair-fill" style="width:{width:.1f}%"></div>'
594
+ "</div>"
595
+ "</div>"
596
+ )
597
+
598
+
599
+ def _scenario_title(scenario_name: str) -> str:
600
+ """The display title for an internal scenario name, falling back to the name."""
601
+ try:
602
+ cfg = default_registry().scenarios.get(scenario_name)
603
+ return (getattr(cfg, "title", "") or scenario_name) if cfg else scenario_name
604
+ except Exception: # pragma: no cover - registry unavailable (defensive)
605
+ return scenario_name
606
+
607
+
608
+ # ── builder ──────────────────────────────────────────────────────────────────────
609
+
610
+
611
+ def build_hall_of_fame() -> dict:
612
+ """Lay out the Hall of Fame tab and return a handles dict for the app to wire.
613
+
614
+ Layout (top β†’ bottom): the glowing **headline marquee**, a **scenario picker**
615
+ (only competitive worlds), the **model podium + ranked table**, the per-scenario
616
+ **sessions area** (an HTML record + a ``gr.render`` of per-run Replay buttons),
617
+ the **fairness footnote**, and a **refresh** control.
618
+
619
+ Returns a handles dict the app wires into the Show transport:
620
+
621
+ ``{"scenario_dd", "refresh", "headline_html", "model_html", "sessions_html",
622
+ "fairness_html", "sessions_render_anchor", "default_scenario"}``
623
+
624
+ The Replay buttons are *not* built here β€” they need the Show's pane handles, which
625
+ only exist after the Show tab builds. The app calls :func:`wire_sessions_render`
626
+ (below) with the shared ``archive_refs`` + states after all tabs build, mirroring
627
+ how ``_build_archive_drawer`` defers its ``show_handles`` lookup.
628
+ """
629
+ scenarios = competitive_scenarios()
630
+ choices = [(title, name) for title, name in scenarios]
631
+ default_scenario = choices[0][1] if choices else None
632
+ entries = _entries()
633
+
634
+ handles: dict = {"default_scenario": default_scenario}
635
+
636
+ with gr.Column(elem_classes=["hall-wrap"]):
637
+ handles["headline_html"] = gr.HTML(render_headline(entries))
638
+
639
+ handles["scenario_dd"] = gr.Dropdown(
640
+ choices=choices,
641
+ value=default_scenario,
642
+ label="World",
643
+ elem_classes=["hall-scenario"],
644
+ interactive=True,
645
+ visible=bool(choices),
646
+ )
647
+ handles["refresh"] = gr.Button("⟳ refresh", size="sm", scale=0)
648
+
649
+ handles["model_html"] = gr.HTML(render_model_board(entries))
650
+
651
+ handles["sessions_html"] = gr.HTML(render_sessions(entries, default_scenario))
652
+ # The per-run Replay buttons live in this column; the app fills it with a
653
+ # gr.render (it needs the Show panes, populated after all tabs build).
654
+ handles["sessions_render_anchor"] = gr.Column(elem_classes=["hall-replays"])
655
+
656
+ handles["fairness_html"] = gr.HTML(render_fairness(entries, default_scenario))
657
+
658
+ return handles
659
+
660
+
661
+ def wire_sessions_render(handles: dict, *, refs: dict, tabs, states: dict) -> None:
662
+ """Wire the Hall's data refresh + the per-run Replay buttons into the Show.
663
+
664
+ Called by the app **after all tabs build** so ``refs["show_handles"]`` is live
665
+ (mirrors ``_build_archive_drawer``'s deferred-ref trick). Three behaviours:
666
+
667
+ * Changing the scenario picker (or refresh) re-reads the ledger and repaints the
668
+ model board, sessions table and fairness panel.
669
+ * A ``gr.render`` keyed on (scenario, refresh) lists per-run **Replay** buttons.
670
+ * Each Replay click loads the run via :func:`load_replay` and pushes it into the
671
+ Show β€” the *exact* output contract from ``_build_archive_drawer`` (session, k,
672
+ scenario, switch Tabs to "show", repaint panes, stopped=False, ticks=0).
673
+
674
+ Defensive throughout: missing handles or an unavailable store degrade to no-ops /
675
+ empty states rather than raising.
676
+ """
677
+ if not handles:
678
+ return
679
+ # Late imports of the app's render/replay helpers β€” they live in app.py and importing
680
+ # them at module top would create an import cycle (app imports this module).
681
+ try:
682
+ from src.ui.fishbowl.app import (
683
+ _pad_values,
684
+ _registry,
685
+ _render_at,
686
+ _show_outs,
687
+ _title_for,
688
+ _tools,
689
+ load_replay,
690
+ )
691
+ except Exception: # pragma: no cover - app helpers unavailable (defensive)
692
+ return
693
+
694
+ scenario_dd = handles.get("scenario_dd")
695
+ refresh = handles.get("refresh")
696
+ model_html = handles.get("model_html")
697
+ sessions_html = handles.get("sessions_html")
698
+ fairness_html = handles.get("fairness_html")
699
+ anchor = handles.get("sessions_render_anchor")
700
+ if scenario_dd is None:
701
+ return
702
+
703
+ # ── refresh the boards on scenario change / refresh click ──────────────────────
704
+ def _repaint(scenario_name):
705
+ entries = _entries()
706
+ return (
707
+ render_model_board(entries),
708
+ render_sessions(entries, scenario_name),
709
+ render_fairness(entries, scenario_name),
710
+ )
711
+
712
+ repaint_outputs = [c for c in (model_html, sessions_html, fairness_html) if c is not None]
713
+ if repaint_outputs:
714
+ # Always emit a full tuple, but only route to present panes (defensive padding).
715
+ def _repaint_present(scenario_name):
716
+ full = _repaint(scenario_name)
717
+ picked = []
718
+ for comp, value in zip((model_html, sessions_html, fairness_html), full):
719
+ if comp is not None:
720
+ picked.append(value)
721
+ return tuple(picked) if len(picked) != 1 else picked[0]
722
+
723
+ scenario_dd.change(_repaint_present, inputs=[scenario_dd], outputs=repaint_outputs)
724
+ if refresh is not None:
725
+ refresh.click(_repaint_present, inputs=[scenario_dd], outputs=repaint_outputs)
726
+
727
+ # ── per-run Replay buttons (gr.render β†’ Show transport) ────────────────────────
728
+ if anchor is None:
729
+ return
730
+
731
+ with anchor:
732
+
733
+ @gr.render(
734
+ inputs=[scenario_dd],
735
+ triggers=[scenario_dd.change] + ([refresh.click] if refresh is not None else []),
736
+ )
737
+ def _render_replays(scenario_name):
738
+ if not scenario_name:
739
+ return
740
+ entries = _entries()
741
+ try:
742
+ rows = scenario_sessions(entries, scenario_name)
743
+ except Exception: # pragma: no cover - projection error (defensive)
744
+ rows = []
745
+ if not rows:
746
+ return
747
+
748
+ show_outs = _show_outs(refs.get("show_handles") or {})
749
+ n_out = 6 + len(show_outs) # session, k, scenario, tabs, *panes, stopped, ticks
750
+
751
+ def _loader(run_id: str):
752
+ def _load(layout, mind_reader):
753
+ session = load_replay(run_id, registry=_registry, tools=_tools)
754
+ if session is None:
755
+ return tuple(gr.update() for _ in range(n_out))
756
+ k = session.head # land on the full discussion; β–Ά replays it
757
+ out = _render_at(session, k, layout=layout, mind_reader=mind_reader)
758
+ return (
759
+ session,
760
+ k,
761
+ _title_for(session.scenario_name),
762
+ gr.update(selected="show"),
763
+ *_pad_values(out, show_outs),
764
+ False,
765
+ 0,
766
+ )
767
+
768
+ return _load
769
+
770
+ for row in rows:
771
+ label = f"β–Ά Replay {_short_id(row.run_id)}"
772
+ if row.winner:
773
+ label += f" Β· WON {row.winner}"
774
+ btn = gr.Button(label, elem_classes=["archive-card"], size="sm")
775
+ btn.click(
776
+ _loader(row.run_id),
777
+ inputs=[states["layout"], states["mind"]],
778
+ outputs=[
779
+ states["session"],
780
+ states["k"],
781
+ states["scenario"],
782
+ tabs,
783
+ *show_outs,
784
+ states["stopped"],
785
+ states["ticks"],
786
+ ],
787
+ )
788
+
789
+
790
+ __all__ = [
791
+ "build_hall_of_fame",
792
+ "competitive_scenarios",
793
+ "render_fairness",
794
+ "render_headline",
795
+ "render_model_board",
796
+ "render_sessions",
797
+ "wire_sessions_render",
798
+ ]
src/ui/fishbowl/session.py CHANGED
@@ -15,8 +15,10 @@ from __future__ import annotations
15
  from src import observability as obs
16
  from src.core.conductor import Conductor
17
  from src.core.ledger_factory import make_ledger
 
18
  from src.core.manifest import AgentManifest
19
  from src.core.registry import Registry, default_registry
 
20
  from src.tools.builtins import default_tool_registry
21
  from src.ui.fishbowl.view_model import view_model_at
22
 
@@ -50,6 +52,14 @@ class FishbowlSession:
50
  governor=self._registry.governor_for(scenario_name),
51
  ledger=make_ledger(),
52
  )
 
 
 
 
 
 
 
 
53
  obs.log(
54
  "session.created",
55
  scenario=scenario_name,
@@ -182,6 +192,34 @@ class FishbowlSession:
182
  winning_model=winning_model,
183
  winning_models=winning_models,
184
  )
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
185
 
186
  @property
187
  def cast(self) -> list[AgentManifest]:
 
15
  from src import observability as obs
16
  from src.core.conductor import Conductor
17
  from src.core.ledger_factory import make_ledger
18
+ from src.core.leaderboard_store import build_entry, make_leaderboard_store
19
  from src.core.manifest import AgentManifest
20
  from src.core.registry import Registry, default_registry
21
+ from src.core.run_index import index_runs
22
  from src.tools.builtins import default_tool_registry
23
  from src.ui.fishbowl.view_model import view_model_at
24
 
 
52
  governor=self._registry.governor_for(scenario_name),
53
  ledger=make_ledger(),
54
  )
55
+ # The leaderboard's dedicated scoreboard table (ADR-0035) β€” a *separate* table in
56
+ # the same database as the event ledger, never the events log. Built defensively
57
+ # so a store/config hiccup can never break a live run; a None store just means no
58
+ # scoreboard row is recorded.
59
+ try:
60
+ self._leaderboard = make_leaderboard_store()
61
+ except Exception: # pragma: no cover - no store configured (defensive)
62
+ self._leaderboard = None
63
  obs.log(
64
  "session.created",
65
  scenario=scenario_name,
 
192
  winning_model=winning_model,
193
  winning_models=winning_models,
194
  )
195
+ self._record_leaderboard()
196
+
197
+ def _record_leaderboard(self) -> None:
198
+ """Write this run's scoreboard row to the dedicated leaderboard table (ADR-0035).
199
+
200
+ Detached from the event ledger: the run's events stay the source of truth for the
201
+ trace, while this persists one denormalised result row to ``leaderboard_entries``
202
+ β€” but only when :func:`build_entry` deems the run eligible (finished, a winner, a
203
+ concrete winning model, and a competitive scenario). Idempotent via the store's
204
+ upsert-on-``run_id``, so a verdict that supersedes a budget close replaces the row.
205
+
206
+ Fully defensive: any failure is logged and swallowed so a leaderboard hiccup never
207
+ breaks the show."""
208
+ store = getattr(self, "_leaderboard", None)
209
+ if store is None:
210
+ return
211
+ try:
212
+ run_events = self.conductor.ledger.events_for_run(self.conductor.run_id)
213
+ summary = next((s for s in index_runs(run_events) if s.run_id == self.conductor.run_id), None)
214
+ if summary is None:
215
+ return
216
+ scenario = self._registry.scenarios.get(self._scenario_name)
217
+ entry = build_entry(summary, getattr(scenario, "competition", None))
218
+ if entry is not None:
219
+ store.record(entry)
220
+ obs.log("leaderboard.recorded", run_id=entry.run_id, scenario=entry.scenario, winner=entry.winner)
221
+ except Exception: # pragma: no cover - never let a scoreboard write break a run
222
+ obs.log("leaderboard.record_failed", level="warning", run_id=self.conductor.run_id)
223
 
224
  @property
225
  def cast(self) -> list[AgentManifest]:
tests/test_leaderboard.py ADDED
@@ -0,0 +1,977 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Leaderboard aggregations over LeaderboardEntry rows β€” zero-mock, deterministic.
2
+
3
+ This suite verifies the leaderboard's read-model invariants: that aggregations over a
4
+ list of LeaderboardEntry objects produce correct tables, win_rate math, sorting, and that
5
+ projections are deterministic regardless of input order.
6
+
7
+ Test strategy: Build LeaderboardEntry objects directly (no events, no ledger), call the
8
+ public aggregation functions, and assert on the resulting rows. This guards against:
9
+
10
+ - Incorrect model endpoint deduplication (one play per endpoint per run, even when
11
+ filling multiple seats).
12
+ - Team wins crediting every member's model vs single-agent wins.
13
+ - Judges and other unmapped cast members affecting fairness and agent tables.
14
+ - Sorting regressions (determinism is a contract).
15
+ - Headline generation requiring symmetric seats + β‰₯2 models with β‰₯1 win each.
16
+ - Edge cases: empty input, zero plays, missing fields, out-of-order entries.
17
+ """
18
+
19
+ from __future__ import annotations
20
+
21
+ from datetime import datetime, timedelta, timezone
22
+
23
+ from src.core.leaderboard import (
24
+ agent_table,
25
+ fairness_table,
26
+ headline,
27
+ model_table,
28
+ scenario_sessions,
29
+ )
30
+ from src.core.leaderboard_store import LeaderboardEntry
31
+ from src.core.run_index import CastBinding
32
+
33
+
34
+ # ── LeaderboardEntry builders ──────────────────────────────────────────────────────
35
+
36
+
37
+ def _entry(
38
+ run_id: str = "r1",
39
+ scenario: str = "Debate Duel",
40
+ seed: str = "seed123",
41
+ session_id: str | None = None,
42
+ competition_kind: str = "versus",
43
+ teams: dict[str, list[str]] | None = None,
44
+ symmetric_seats: list[str] | None = None,
45
+ cast: dict[str, CastBinding] | None = None,
46
+ winner: str | None = "alice",
47
+ winner_kind: str | None = "agent",
48
+ winning_model: str | None = "openai/openbmb/MiniCPM-8B",
49
+ winning_models: list[str] | None = None,
50
+ reason: str | None = "verdict",
51
+ turns: int = 5,
52
+ tokens: int = 200,
53
+ started_at: datetime | None = None,
54
+ finished_at: datetime | None = None,
55
+ ) -> LeaderboardEntry:
56
+ """Build a minimal LeaderboardEntry for testing."""
57
+ if started_at is None:
58
+ started_at = datetime(2025, 6, 14, 10, 0, 0, tzinfo=timezone.utc)
59
+ if finished_at is None:
60
+ finished_at = started_at + timedelta(minutes=5)
61
+ if cast is None:
62
+ cast = {"alice": CastBinding(model_endpoint=winning_model)}
63
+ if winning_models is None:
64
+ winning_models = [winning_model] if winning_model else []
65
+ return LeaderboardEntry(
66
+ run_id=run_id,
67
+ session_id=session_id,
68
+ scenario=scenario,
69
+ seed=seed,
70
+ competition_kind=competition_kind,
71
+ teams=teams,
72
+ symmetric_seats=symmetric_seats,
73
+ cast=cast,
74
+ winner=winner,
75
+ winner_kind=winner_kind,
76
+ winning_model=winning_model,
77
+ winning_models=winning_models,
78
+ reason=reason,
79
+ turns=turns,
80
+ tokens=tokens,
81
+ started_at=started_at,
82
+ finished_at=finished_at,
83
+ )
84
+
85
+
86
+ # ── Tests: scenario_sessions (newest-first filtering) ─────────────────────────────
87
+
88
+
89
+ class TestScenarioSessions:
90
+ """Verify scenario_sessions filters, orders, and projects correctly."""
91
+
92
+ def test_empty_entries_returns_empty_list(self):
93
+ """Empty entry list returns no sessions."""
94
+ result = scenario_sessions([], "Debate Duel")
95
+ assert result == []
96
+
97
+ def test_no_winner_entry_excluded(self):
98
+ """An entry with no winner is dropped (defensive gate)."""
99
+ entry = _entry(winner=None)
100
+ result = scenario_sessions([entry], "Debate Duel")
101
+ assert result == []
102
+
103
+ def test_scenario_filter_excludes_other_scenarios(self):
104
+ """Sessions from other scenarios are not returned."""
105
+ entry = _entry(scenario="Debate Duel")
106
+ result = scenario_sessions([entry], "Trivia Night")
107
+ assert result == []
108
+
109
+ def test_one_entry_has_all_fields(self):
110
+ """A single entry is projected with all fields intact."""
111
+ cast = {
112
+ "alice": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B", model_profile="large"),
113
+ "bob": CastBinding(model_endpoint="google/gemma-12B", model_profile="medium"),
114
+ }
115
+ started_at = datetime(2025, 6, 14, 10, 0, 0, tzinfo=timezone.utc)
116
+ finished_at = started_at + timedelta(minutes=5)
117
+ entry = _entry(
118
+ run_id="r1",
119
+ scenario="Debate Duel",
120
+ seed="abc123",
121
+ cast=cast,
122
+ symmetric_seats=["debater_a", "debater_b"],
123
+ winner="alice",
124
+ winner_kind="agent",
125
+ winning_model="openai/openbmb/MiniCPM-8B",
126
+ winning_models=["openai/openbmb/MiniCPM-8B"],
127
+ turns=7,
128
+ tokens=320,
129
+ started_at=started_at,
130
+ finished_at=finished_at,
131
+ )
132
+ result = scenario_sessions([entry], "Debate Duel")
133
+ assert len(result) == 1
134
+ row = result[0]
135
+ assert row.run_id == "r1"
136
+ assert row.scenario == "Debate Duel"
137
+ assert row.seed == "abc123"
138
+ assert row.winner == "alice"
139
+ assert row.winner_kind == "agent"
140
+ assert row.turns == 7
141
+ assert row.tokens == 320
142
+ assert row.started_at == started_at
143
+ assert row.finished_at == finished_at
144
+ assert "alice" in row.cast
145
+ assert row.cast["alice"].model_endpoint == "openai/openbmb/MiniCPM-8B"
146
+
147
+ def test_newest_first_order(self):
148
+ """Sessions are sorted newest first by finished_at."""
149
+ old = datetime(2025, 6, 1, 10, 0, 0, tzinfo=timezone.utc)
150
+ new = datetime(2025, 6, 14, 10, 0, 0, tzinfo=timezone.utc)
151
+ entries = [
152
+ _entry(run_id="r1", finished_at=old + timedelta(minutes=1)),
153
+ _entry(run_id="r2", finished_at=new + timedelta(minutes=1)),
154
+ ]
155
+ result = scenario_sessions(entries, "Debate Duel")
156
+ assert len(result) == 2
157
+ assert result[0].run_id == "r2" # newer first
158
+ assert result[1].run_id == "r1"
159
+
160
+ def test_run_id_tiebreak_when_finished_at_same(self):
161
+ """When finished_at is equal, run_id is the tiebreaker (ascending alphabetical)."""
162
+ same_time = datetime(2025, 6, 14, 10, 0, 0, tzinfo=timezone.utc)
163
+ entries = [
164
+ _entry(run_id="r2", finished_at=same_time + timedelta(minutes=1)),
165
+ _entry(run_id="r1", finished_at=same_time + timedelta(minutes=1)),
166
+ ]
167
+ result = scenario_sessions(entries, "Debate Duel")
168
+ assert len(result) == 2
169
+ # Same finished_at; run_id breaks ties in ascending order (r1 < r2)
170
+ assert result[0].run_id == "r1"
171
+ assert result[1].run_id == "r2"
172
+
173
+
174
+ # ── Tests: model_table (endpoint-level stats across all scenarios) ──────────────────
175
+
176
+
177
+ class TestModelTable:
178
+ """Verify model aggregation: plays, wins, win_rate, scenarios, deterministic sort."""
179
+
180
+ def test_empty_entries_returns_empty_list(self):
181
+ """No entries β†’ no rows."""
182
+ result = model_table([])
183
+ assert result == []
184
+
185
+ def test_no_winner_entry_not_counted(self):
186
+ """An entry with no winner contributes no stats."""
187
+ entry = _entry(winner=None)
188
+ result = model_table([entry])
189
+ assert result == []
190
+
191
+ def test_one_model_one_play_one_win(self):
192
+ """A single-entry win: plays=1, wins=1, win_rate=1.0."""
193
+ entry = _entry(
194
+ scenario="Debate Duel",
195
+ cast={"alice": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B")},
196
+ winner="alice",
197
+ winning_model="openai/openbmb/MiniCPM-8B",
198
+ winning_models=["openai/openbmb/MiniCPM-8B"],
199
+ )
200
+ result = model_table([entry])
201
+ assert len(result) == 1
202
+ assert result[0].model == "openai/openbmb/MiniCPM-8B"
203
+ assert result[0].plays == 1
204
+ assert result[0].wins == 1
205
+ assert result[0].win_rate == 1.0
206
+ assert result[0].scenarios == ["Debate Duel"]
207
+
208
+ def test_one_model_mixed_wins_and_losses(self):
209
+ """A model with 2 wins / 3 plays has win_rate β‰ˆ 0.667."""
210
+ entries = [
211
+ _entry(
212
+ run_id="r1",
213
+ scenario="Debate Duel",
214
+ cast={"alice": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B")},
215
+ winner="alice",
216
+ winning_model="openai/openbmb/MiniCPM-8B",
217
+ ),
218
+ _entry(
219
+ run_id="r2",
220
+ scenario="Debate Duel",
221
+ cast={"alice": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B")},
222
+ winner="alice",
223
+ winning_model="openai/openbmb/MiniCPM-8B",
224
+ ),
225
+ _entry(
226
+ run_id="r3",
227
+ scenario="Debate Duel",
228
+ cast={
229
+ "alice": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B"),
230
+ "bob": CastBinding(model_endpoint="google/gemma-12B"),
231
+ },
232
+ winner="bob",
233
+ winning_model="google/gemma-12B",
234
+ ),
235
+ ]
236
+ result = model_table(entries)
237
+ m8b = next((r for r in result if r.model == "openai/openbmb/MiniCPM-8B"), None)
238
+ assert m8b is not None
239
+ assert m8b.plays == 3
240
+ assert m8b.wins == 2
241
+ assert abs(m8b.win_rate - (2 / 3)) < 0.001
242
+
243
+ def test_model_endpoint_deduplication_one_play_per_endpoint_per_run(self):
244
+ """A model filling two seats in one run counts as one play (dedup by endpoint)."""
245
+ entry = _entry(
246
+ run_id="r1",
247
+ scenario="Some Scenario",
248
+ cast={
249
+ "alice": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B"),
250
+ "bob": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B"), # same endpoint
251
+ },
252
+ winner="alice",
253
+ winning_model="openai/openbmb/MiniCPM-8B",
254
+ symmetric_seats=["seat_a", "seat_b"],
255
+ )
256
+ result = model_table([entry])
257
+ assert len(result) == 1
258
+ assert result[0].plays == 1 # not 2!
259
+ assert result[0].wins == 1
260
+
261
+ def test_scenarios_lists_all_distinct_scenarios_sorted(self):
262
+ """A model in multiple scenarios lists all of them (sorted)."""
263
+ entries = [
264
+ _entry(
265
+ run_id="r1",
266
+ scenario="Zebra Debate",
267
+ cast={"alice": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B")},
268
+ winner="alice",
269
+ winning_model="openai/openbmb/MiniCPM-8B",
270
+ ),
271
+ _entry(
272
+ run_id="r2",
273
+ scenario="Alpha Trivia",
274
+ cast={"alice": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B")},
275
+ winner="alice",
276
+ winning_model="openai/openbmb/MiniCPM-8B",
277
+ ),
278
+ ]
279
+ result = model_table(entries)
280
+ assert len(result) == 1
281
+ assert result[0].scenarios == ["Alpha Trivia", "Zebra Debate"]
282
+
283
+ def test_deterministic_sort_win_rate_then_wins_then_plays_then_model(self):
284
+ """model_table sorts by (-win_rate, -wins, -plays, model asc)."""
285
+ entries = [
286
+ # MiniCPM-8B: 2 wins / 2 plays = 1.0
287
+ _entry(
288
+ run_id="r1",
289
+ cast={"alice": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B")},
290
+ winner="alice",
291
+ winning_model="openai/openbmb/MiniCPM-8B",
292
+ ),
293
+ _entry(
294
+ run_id="r2",
295
+ cast={"alice": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B")},
296
+ winner="alice",
297
+ winning_model="openai/openbmb/MiniCPM-8B",
298
+ ),
299
+ # Gemma: 1 win / 2 plays = 0.5
300
+ _entry(
301
+ run_id="r3",
302
+ cast={"alice": CastBinding(model_endpoint="google/gemma-12B")},
303
+ winner="alice",
304
+ winning_model="google/gemma-12B",
305
+ ),
306
+ _entry(
307
+ run_id="r4",
308
+ cast={
309
+ "alice": CastBinding(model_endpoint="google/gemma-12B"),
310
+ "bob": CastBinding(model_endpoint="openai/openbmb/MiniCPM-4B"),
311
+ },
312
+ winner="bob",
313
+ winning_model="openai/openbmb/MiniCPM-4B", # MiniCPM-4B wins here
314
+ ),
315
+ # MiniCPM-4B: 0 wins / 2 plays = 0.0 (plays in r4 winning, but that's a different calc)
316
+ _entry(
317
+ run_id="r5",
318
+ cast={
319
+ "alice": CastBinding(model_endpoint="openai/openbmb/MiniCPM-4B"),
320
+ "bob": CastBinding(model_endpoint="meta/llama-7B"),
321
+ },
322
+ winner="bob",
323
+ winning_model="meta/llama-7B",
324
+ ),
325
+ _entry(
326
+ run_id="r6",
327
+ cast={
328
+ "alice": CastBinding(model_endpoint="openai/openbmb/MiniCPM-4B"),
329
+ "bob": CastBinding(model_endpoint="meta/llama-7B"),
330
+ },
331
+ winner="alice", # MiniCPM-4B wins (but alice is the seat, need to check...)
332
+ winning_model="openai/openbmb/MiniCPM-4B",
333
+ ),
334
+ ]
335
+ result = model_table(entries)
336
+ result_models = [r.model for r in result]
337
+ assert result_models[0] == "openai/openbmb/MiniCPM-8B" # 1.0, 2 wins, 2 plays
338
+ # MiniCPM-4B: 2 wins / 3 plays β‰ˆ 0.67, Gemma: 1 win / 2 plays = 0.5, Llama: 1 win / 2 plays = 0.5
339
+ # Sort: MiniCPM-8B (1.0) > MiniCPM-4B (0.67) > Gemma (0.5, "gemma" < "llama") > Llama (0.5)
340
+ assert result_models[1] == "openai/openbmb/MiniCPM-4B" # 0.67, 2 wins
341
+ assert result_models[2] == "google/gemma-12B" # 0.5, 1 win
342
+ assert result_models[3] == "meta/llama-7B" # 0.5, 1 win
343
+
344
+ def test_winning_models_includes_all_credited_endpoints(self):
345
+ """A run with winning_models=[a,b] credits both with a win."""
346
+ entry = _entry(
347
+ run_id="r1",
348
+ scenario="Debate Duel",
349
+ cast={
350
+ "alice": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B"),
351
+ "bob": CastBinding(model_endpoint="google/gemma-12B"),
352
+ },
353
+ winner="team_a",
354
+ winner_kind="team",
355
+ winning_model=None,
356
+ winning_models=["openai/openbmb/MiniCPM-8B", "google/gemma-12B"],
357
+ symmetric_seats=["seat_a", "seat_b"],
358
+ )
359
+ result = model_table([entry])
360
+ assert len(result) == 2
361
+ m8b = next((r for r in result if r.model == "openai/openbmb/MiniCPM-8B"), None)
362
+ assert m8b is not None and m8b.wins == 1
363
+ gemma = next((r for r in result if r.model == "google/gemma-12B"), None)
364
+ assert gemma is not None and gemma.wins == 1
365
+
366
+
367
+ # ── Tests: agent_table (per-scenario, per-persona stats) ────────────────────────────
368
+
369
+
370
+ class TestAgentTable:
371
+ """Verify agent attribution: plays, wins, seat_type, model_endpoints, sort."""
372
+
373
+ def test_empty_entries_returns_empty_list(self):
374
+ """No entries β†’ no rows."""
375
+ result = agent_table([], "Debate Duel")
376
+ assert result == []
377
+
378
+ def test_agent_filtered_by_scenario(self):
379
+ """agent_table only includes agents from the named scenario."""
380
+ entries = [
381
+ _entry(
382
+ run_id="r1",
383
+ scenario="Debate Duel",
384
+ cast={"alice": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B")},
385
+ winner="alice",
386
+ ),
387
+ _entry(
388
+ run_id="r2",
389
+ scenario="Other Scenario",
390
+ cast={"bob": CastBinding(model_endpoint="google/gemma-12B")},
391
+ winner="bob",
392
+ ),
393
+ ]
394
+ result = agent_table(entries, "Debate Duel")
395
+ agents = [r.agent for r in result]
396
+ assert agents == ["alice"]
397
+
398
+ def test_agent_single_play_single_win(self):
399
+ """An agent with one win: plays=1, wins=1, win_rate=1.0."""
400
+ entry = _entry(
401
+ scenario="Debate Duel",
402
+ cast={"alice": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B")},
403
+ winner="alice",
404
+ winning_model="openai/openbmb/MiniCPM-8B",
405
+ symmetric_seats=["debater_a"],
406
+ )
407
+ result = agent_table([entry], "Debate Duel")
408
+ assert len(result) == 1
409
+ assert result[0].agent == "alice"
410
+ assert result[0].plays == 1
411
+ assert result[0].wins == 1
412
+ assert result[0].win_rate == 1.0
413
+
414
+ def test_agent_seat_type_from_symmetric_seats(self):
415
+ """In symmetric-seat scenarios, agent's seat_type is the seat name."""
416
+ entry = _entry(
417
+ scenario="Debate Duel",
418
+ cast={"debater_a": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B")},
419
+ winner="debater_a",
420
+ winning_model="openai/openbmb/MiniCPM-8B",
421
+ symmetric_seats=["debater_a", "debater_b"],
422
+ )
423
+ result = agent_table([entry], "Debate Duel")
424
+ assert result[0].agent == "debater_a"
425
+ assert result[0].seat_type == "debater_a"
426
+
427
+ def test_agent_seat_type_from_teams(self):
428
+ """In team scenarios, agent's seat_type is the team label they belong to."""
429
+ entry = _entry(
430
+ scenario="Team Game",
431
+ cast={
432
+ "alice": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B"),
433
+ "bob": CastBinding(model_endpoint="google/gemma-12B"),
434
+ },
435
+ teams={"team_a": ["alice", "charlie"], "team_b": ["bob"]},
436
+ winner="alice",
437
+ winning_model="openai/openbmb/MiniCPM-8B",
438
+ winner_kind="agent",
439
+ competition_kind="versus",
440
+ )
441
+ result = agent_table([entry], "Team Game")
442
+ alice = next((r for r in result if r.agent == "alice"), None)
443
+ bob = next((r for r in result if r.agent == "bob"), None)
444
+ assert alice is not None and alice.seat_type == "team_a"
445
+ assert bob is not None and bob.seat_type == "team_b"
446
+
447
+ def test_agent_seat_type_empty_when_unmapped(self):
448
+ """An agent not in teams or symmetric_seats (e.g., judge) has seat_type=''."""
449
+ entry = _entry(
450
+ scenario="Debate Duel",
451
+ cast={
452
+ "debater_a": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B"),
453
+ "judge": CastBinding(model_endpoint=None),
454
+ },
455
+ winner="debater_a",
456
+ winning_model="openai/openbmb/MiniCPM-8B",
457
+ symmetric_seats=["debater_a"],
458
+ competition_kind="judged",
459
+ )
460
+ result = agent_table([entry], "Debate Duel")
461
+ judge_row = next((r for r in result if r.agent == "judge"), None)
462
+ assert judge_row is not None
463
+ assert judge_row.seat_type == ""
464
+
465
+ def test_agent_model_endpoints_deduped_sorted(self):
466
+ """An agent's model_endpoints is sorted, distinct models that filled the seat."""
467
+ entries = [
468
+ _entry(
469
+ run_id="r1",
470
+ scenario="Debate Duel",
471
+ cast={"alice": CastBinding(model_endpoint="google/gemma-12B")},
472
+ winner="alice",
473
+ winning_model="google/gemma-12B",
474
+ ),
475
+ _entry(
476
+ run_id="r2",
477
+ scenario="Debate Duel",
478
+ cast={"alice": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B")},
479
+ winner="alice",
480
+ winning_model="openai/openbmb/MiniCPM-8B",
481
+ ),
482
+ _entry(
483
+ run_id="r3",
484
+ scenario="Debate Duel",
485
+ cast={"alice": CastBinding(model_endpoint="google/gemma-12B")}, # repeat
486
+ winner="alice",
487
+ winning_model="google/gemma-12B",
488
+ ),
489
+ ]
490
+ result = agent_table(entries, "Debate Duel")
491
+ assert len(result) == 1
492
+ assert result[0].model_endpoints == ["google/gemma-12B", "openai/openbmb/MiniCPM-8B"]
493
+
494
+ def test_team_win_credits_all_team_members(self):
495
+ """A team win credits every member of the winning team with a win."""
496
+ entry = _entry(
497
+ scenario="Team Game",
498
+ cast={
499
+ "alice": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B"),
500
+ "bob": CastBinding(model_endpoint="google/gemma-12B"),
501
+ },
502
+ teams={"team_a": ["alice"], "team_b": ["bob"]},
503
+ winner="team_a",
504
+ winner_kind="team",
505
+ winning_models=["openai/openbmb/MiniCPM-8B"],
506
+ winning_model=None,
507
+ competition_kind="versus",
508
+ )
509
+ result = agent_table([entry], "Team Game")
510
+ alice = next((r for r in result if r.agent == "alice"), None)
511
+ bob = next((r for r in result if r.agent == "bob"), None)
512
+ assert alice is not None and alice.wins == 1
513
+ assert bob is not None and bob.wins == 0
514
+
515
+
516
+ # ── Tests: fairness_table (seat-type aggregation) ─────────────────────────────────
517
+
518
+
519
+ class TestFairnessTable:
520
+ """Verify fairness_table: seat-type aggregation, unmapped excluded, sort."""
521
+
522
+ def test_empty_entries_returns_empty_list(self):
523
+ """No entries β†’ no rows."""
524
+ result = fairness_table([], "Debate Duel")
525
+ assert result == []
526
+
527
+ def test_only_declared_seat_types_appear(self):
528
+ """Only declared seat_types (teams or symmetric_seats) appear."""
529
+ entry = _entry(
530
+ scenario="Debate Duel",
531
+ cast={
532
+ "debater_a": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B"),
533
+ "judge": CastBinding(model_endpoint=None),
534
+ },
535
+ winner="debater_a",
536
+ winning_model="openai/openbmb/MiniCPM-8B",
537
+ symmetric_seats=["debater_a"],
538
+ competition_kind="judged",
539
+ )
540
+ result = fairness_table([entry], "Debate Duel")
541
+ seat_types = [r.seat_type for r in result]
542
+ # Only "debater_a" is declared; "judge" is unmapped.
543
+ assert seat_types == ["debater_a"]
544
+
545
+ def test_symmetric_seat_plays_and_wins(self):
546
+ """In symmetric-seat scenarios, each seat contributes one play per run."""
547
+ entry = _entry(
548
+ scenario="Debate Duel",
549
+ cast={
550
+ "debater_a": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B"),
551
+ "debater_b": CastBinding(model_endpoint="google/gemma-12B"),
552
+ },
553
+ winner="debater_a",
554
+ winning_model="openai/openbmb/MiniCPM-8B",
555
+ symmetric_seats=["debater_a", "debater_b"],
556
+ )
557
+ result = fairness_table([entry], "Debate Duel")
558
+ debater_a = next((r for r in result if r.seat_type == "debater_a"), None)
559
+ debater_b = next((r for r in result if r.seat_type == "debater_b"), None)
560
+ assert debater_a is not None and debater_a.plays == 1 and debater_a.wins == 1
561
+ assert debater_b is not None and debater_b.plays == 1 and debater_b.wins == 0
562
+
563
+ def test_team_seats_aggregation(self):
564
+ """In team scenarios, each team contributes one play; wins go to winning team."""
565
+ entry = _entry(
566
+ scenario="Team Game",
567
+ cast={
568
+ "alice": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B"),
569
+ "bob": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B"),
570
+ "charlie": CastBinding(model_endpoint="google/gemma-12B"),
571
+ },
572
+ teams={"team_a": ["alice", "bob"], "team_b": ["charlie"]},
573
+ winner="team_a",
574
+ winner_kind="team",
575
+ winning_models=["openai/openbmb/MiniCPM-8B"],
576
+ winning_model=None,
577
+ competition_kind="versus",
578
+ )
579
+ result = fairness_table([entry], "Team Game")
580
+ team_a = next((r for r in result if r.seat_type == "team_a"), None)
581
+ team_b = next((r for r in result if r.seat_type == "team_b"), None)
582
+ assert team_a is not None and team_a.plays == 1 and team_a.wins == 1
583
+ assert team_b is not None and team_b.plays == 1 and team_b.wins == 0
584
+
585
+ def test_judge_not_counted_in_fairness(self):
586
+ """A judge (unmapped cast member) does not appear in fairness_table."""
587
+ entry = _entry(
588
+ scenario="Debate Duel",
589
+ cast={
590
+ "debater_a": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B"),
591
+ "judge": CastBinding(model_endpoint=None),
592
+ },
593
+ winner="debater_a",
594
+ winning_model="openai/openbmb/MiniCPM-8B",
595
+ symmetric_seats=["debater_a"],
596
+ competition_kind="judged",
597
+ )
598
+ result = fairness_table([entry], "Debate Duel")
599
+ assert len(result) == 1
600
+ assert result[0].seat_type == "debater_a"
601
+
602
+ def test_win_rate_calculated_per_seat_type(self):
603
+ """A seat winning 1 of 2 runs has win_rate=0.5."""
604
+ entries = [
605
+ _entry(
606
+ run_id="r1",
607
+ scenario="Debate Duel",
608
+ cast={
609
+ "debater_a": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B"),
610
+ "debater_b": CastBinding(model_endpoint="google/gemma-12B"),
611
+ },
612
+ winner="debater_a",
613
+ winning_model="openai/openbmb/MiniCPM-8B",
614
+ symmetric_seats=["debater_a", "debater_b"],
615
+ ),
616
+ _entry(
617
+ run_id="r2",
618
+ scenario="Debate Duel",
619
+ cast={
620
+ "debater_a": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B"),
621
+ "debater_b": CastBinding(model_endpoint="google/gemma-12B"),
622
+ },
623
+ winner="debater_b",
624
+ winning_model="google/gemma-12B",
625
+ symmetric_seats=["debater_a", "debater_b"],
626
+ ),
627
+ ]
628
+ result = fairness_table(entries, "Debate Duel")
629
+ debater_a = next((r for r in result if r.seat_type == "debater_a"), None)
630
+ debater_b = next((r for r in result if r.seat_type == "debater_b"), None)
631
+ assert debater_a is not None and abs(debater_a.win_rate - 0.5) < 0.001
632
+ assert debater_b is not None and abs(debater_b.win_rate - 0.5) < 0.001
633
+
634
+
635
+ # ── Tests: headline (model-vs-model narrative) ─────────────────────────────────────
636
+
637
+
638
+ class TestHeadline:
639
+ """Verify headline: symmetric seats + β‰₯2 models with β‰₯1 win each."""
640
+
641
+ def test_empty_entries_returns_none(self):
642
+ """No entries β†’ no headline."""
643
+ result = headline([])
644
+ assert result is None
645
+
646
+ def test_no_symmetric_seat_scenarios_returns_none(self):
647
+ """Headline needs symmetric-seat scenario (model-vs-model); team scenarios excluded."""
648
+ entry = _entry(
649
+ scenario="Team Game",
650
+ cast={
651
+ "alice": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B"),
652
+ "bob": CastBinding(model_endpoint="google/gemma-12B"),
653
+ },
654
+ teams={"team_a": ["alice"], "team_b": ["bob"]},
655
+ winner="team_a",
656
+ winner_kind="team",
657
+ winning_models=["openai/openbmb/MiniCPM-8B"],
658
+ winning_model=None,
659
+ competition_kind="versus",
660
+ )
661
+ result = headline([entry])
662
+ assert result is None
663
+
664
+ def test_only_one_model_in_scenario_returns_none(self):
665
+ """A scenario with only one model (even if winning) doesn't qualify."""
666
+ entry = _entry(
667
+ scenario="Debate Duel",
668
+ cast={
669
+ "debater_a": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B"),
670
+ "debater_b": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B"),
671
+ },
672
+ winner="debater_a",
673
+ winning_model="openai/openbmb/MiniCPM-8B",
674
+ symmetric_seats=["debater_a", "debater_b"],
675
+ )
676
+ result = headline([entry])
677
+ assert result is None # only 1 distinct model
678
+
679
+ def test_two_models_but_loser_has_zero_wins_returns_none(self):
680
+ """Headline needs β‰₯2 models that have each won β‰₯1."""
681
+ entry = _entry(
682
+ scenario="Debate Duel",
683
+ cast={
684
+ "debater_a": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B"),
685
+ "debater_b": CastBinding(model_endpoint="google/gemma-12B"),
686
+ },
687
+ winner="debater_a",
688
+ winning_model="openai/openbmb/MiniCPM-8B",
689
+ winning_models=["openai/openbmb/MiniCPM-8B"],
690
+ symmetric_seats=["debater_a", "debater_b"],
691
+ )
692
+ result = headline([entry])
693
+ assert result is None # gemma has 0 wins
694
+
695
+ def test_headline_with_two_models_both_winning(self):
696
+ """Headline is generated when β‰₯2 models have each won β‰₯1 in symmetric scenario."""
697
+ entries = [
698
+ _entry(
699
+ run_id="r1",
700
+ scenario="Debate Duel",
701
+ cast={
702
+ "debater_a": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B"),
703
+ "debater_b": CastBinding(model_endpoint="google/gemma-12B"),
704
+ },
705
+ winner="debater_a",
706
+ winning_model="openai/openbmb/MiniCPM-8B",
707
+ winning_models=["openai/openbmb/MiniCPM-8B"],
708
+ symmetric_seats=["debater_a", "debater_b"],
709
+ ),
710
+ _entry(
711
+ run_id="r2",
712
+ scenario="Debate Duel",
713
+ cast={
714
+ "debater_a": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B"),
715
+ "debater_b": CastBinding(model_endpoint="google/gemma-12B"),
716
+ },
717
+ winner="debater_b",
718
+ winning_model="google/gemma-12B",
719
+ winning_models=["google/gemma-12B"],
720
+ symmetric_seats=["debater_a", "debater_b"],
721
+ ),
722
+ ]
723
+ result = headline(entries)
724
+ assert result is not None
725
+ assert "MiniCPM" in result
726
+ assert "gemma" in result.lower()
727
+ assert "Debate Duel" in result
728
+ assert "1-1" in result
729
+
730
+ def test_headline_picks_most_played_scenario(self):
731
+ """When multiple scenarios qualify, headline picks the one with most games decided."""
732
+ entries = [
733
+ # Debate Duel: 1 game
734
+ _entry(
735
+ run_id="r1",
736
+ scenario="Debate Duel",
737
+ cast={
738
+ "debater_a": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B"),
739
+ "debater_b": CastBinding(model_endpoint="google/gemma-12B"),
740
+ },
741
+ winner="debater_a",
742
+ winning_model="openai/openbmb/MiniCPM-8B",
743
+ symmetric_seats=["debater_a", "debater_b"],
744
+ ),
745
+ # Trivia Night: 3 games (most)
746
+ _entry(
747
+ run_id="r2",
748
+ scenario="Trivia Night",
749
+ cast={
750
+ "player_a": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B"),
751
+ "player_b": CastBinding(model_endpoint="google/gemma-12B"),
752
+ },
753
+ winner="player_a",
754
+ winning_model="openai/openbmb/MiniCPM-8B",
755
+ symmetric_seats=["player_a", "player_b"],
756
+ ),
757
+ _entry(
758
+ run_id="r3",
759
+ scenario="Trivia Night",
760
+ cast={
761
+ "player_a": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B"),
762
+ "player_b": CastBinding(model_endpoint="google/gemma-12B"),
763
+ },
764
+ winner="player_b",
765
+ winning_model="google/gemma-12B",
766
+ symmetric_seats=["player_a", "player_b"],
767
+ ),
768
+ _entry(
769
+ run_id="r4",
770
+ scenario="Trivia Night",
771
+ cast={
772
+ "player_a": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B"),
773
+ "player_b": CastBinding(model_endpoint="google/gemma-12B"),
774
+ },
775
+ winner="player_b",
776
+ winning_model="google/gemma-12B",
777
+ symmetric_seats=["player_a", "player_b"],
778
+ ),
779
+ ]
780
+ result = headline(entries)
781
+ assert result is not None
782
+ assert "Trivia Night" in result
783
+
784
+ def test_headline_alphabetical_tiebreak_when_same_wins(self):
785
+ """When scenarios tie on wins, pick the alphabetically-first scenario name."""
786
+ entries = [
787
+ # "Aardvark": 1 win for MiniCPM, 1 for Gemma
788
+ _entry(
789
+ run_id="a1",
790
+ scenario="Aardvark Duel",
791
+ cast={
792
+ "p1": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B"),
793
+ "p2": CastBinding(model_endpoint="google/gemma-12B"),
794
+ },
795
+ winner="p1",
796
+ winning_model="openai/openbmb/MiniCPM-8B",
797
+ symmetric_seats=["p1", "p2"],
798
+ ),
799
+ _entry(
800
+ run_id="a2",
801
+ scenario="Aardvark Duel",
802
+ cast={
803
+ "p1": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B"),
804
+ "p2": CastBinding(model_endpoint="google/gemma-12B"),
805
+ },
806
+ winner="p2",
807
+ winning_model="google/gemma-12B",
808
+ symmetric_seats=["p1", "p2"],
809
+ ),
810
+ # "Zebra": same record, but alphabetically later
811
+ _entry(
812
+ run_id="z1",
813
+ scenario="Zebra Duel",
814
+ cast={
815
+ "p1": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B"),
816
+ "p2": CastBinding(model_endpoint="google/gemma-12B"),
817
+ },
818
+ winner="p1",
819
+ winning_model="openai/openbmb/MiniCPM-8B",
820
+ symmetric_seats=["p1", "p2"],
821
+ ),
822
+ _entry(
823
+ run_id="z2",
824
+ scenario="Zebra Duel",
825
+ cast={
826
+ "p1": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B"),
827
+ "p2": CastBinding(model_endpoint="google/gemma-12B"),
828
+ },
829
+ winner="p2",
830
+ winning_model="google/gemma-12B",
831
+ symmetric_seats=["p1", "p2"],
832
+ ),
833
+ ]
834
+ result = headline(entries)
835
+ assert result is not None
836
+ assert "Aardvark Duel" in result # alphabetically first
837
+
838
+ def test_headline_format(self):
839
+ """Headline format: 'ModelA beats ModelB Β· X-Y at Scenario'."""
840
+ entries = [
841
+ _entry(
842
+ run_id="r1",
843
+ scenario="Debate Duel",
844
+ cast={
845
+ "debater_a": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B"),
846
+ "debater_b": CastBinding(model_endpoint="google/gemma-12B"),
847
+ },
848
+ winner="debater_a",
849
+ winning_model="openai/openbmb/MiniCPM-8B",
850
+ symmetric_seats=["debater_a", "debater_b"],
851
+ ),
852
+ _entry(
853
+ run_id="r2",
854
+ scenario="Debate Duel",
855
+ cast={
856
+ "debater_a": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B"),
857
+ "debater_b": CastBinding(model_endpoint="google/gemma-12B"),
858
+ },
859
+ winner="debater_a",
860
+ winning_model="openai/openbmb/MiniCPM-8B",
861
+ symmetric_seats=["debater_a", "debater_b"],
862
+ ),
863
+ _entry(
864
+ run_id="r3",
865
+ scenario="Debate Duel",
866
+ cast={
867
+ "debater_a": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B"),
868
+ "debater_b": CastBinding(model_endpoint="google/gemma-12B"),
869
+ },
870
+ winner="debater_b",
871
+ winning_model="google/gemma-12B",
872
+ symmetric_seats=["debater_a", "debater_b"],
873
+ ),
874
+ ]
875
+ result = headline(entries)
876
+ assert result is not None
877
+ assert "beats" in result
878
+ assert "Β·" in result
879
+ assert "Debate Duel" in result
880
+ assert "2-1" in result # MiniCPM beats Gemma 2-1
881
+
882
+
883
+ # ── Tests: Purity (determinism and idempotency) ────────────────────────────────────
884
+
885
+
886
+ class TestProjectionPurity:
887
+ """Verify that projections are deterministic regardless of entry order."""
888
+
889
+ def test_scenario_sessions_idempotent(self):
890
+ """Calling scenario_sessions twice returns the same result."""
891
+ entry = _entry(scenario="Debate Duel")
892
+ result1 = scenario_sessions([entry], "Debate Duel")
893
+ result2 = scenario_sessions([entry], "Debate Duel")
894
+ assert result1 == result2
895
+
896
+ def test_model_table_idempotent(self):
897
+ """Calling model_table twice returns the same result."""
898
+ entry = _entry()
899
+ result1 = model_table([entry])
900
+ result2 = model_table([entry])
901
+ assert result1 == result2
902
+
903
+ def test_agent_table_deterministic_order_independent(self):
904
+ """agent_table produces the same result regardless of entry order."""
905
+ entries = [
906
+ _entry(
907
+ run_id="r1",
908
+ scenario="Debate Duel",
909
+ cast={"alice": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B")},
910
+ winner="alice",
911
+ ),
912
+ _entry(
913
+ run_id="r2",
914
+ scenario="Debate Duel",
915
+ cast={"bob": CastBinding(model_endpoint="google/gemma-12B")},
916
+ winner="bob",
917
+ ),
918
+ ]
919
+ result1 = agent_table(entries, "Debate Duel")
920
+ result2 = agent_table(list(reversed(entries)), "Debate Duel")
921
+ assert result1 == result2
922
+
923
+ def test_fairness_table_deterministic_order_independent(self):
924
+ """fairness_table produces the same result regardless of entry order."""
925
+ entries = [
926
+ _entry(
927
+ run_id="r1",
928
+ scenario="Debate Duel",
929
+ cast={
930
+ "debater_a": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B"),
931
+ "debater_b": CastBinding(model_endpoint="google/gemma-12B"),
932
+ },
933
+ winner="debater_a",
934
+ symmetric_seats=["debater_a", "debater_b"],
935
+ ),
936
+ _entry(
937
+ run_id="r2",
938
+ scenario="Debate Duel",
939
+ cast={
940
+ "debater_a": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B"),
941
+ "debater_b": CastBinding(model_endpoint="google/gemma-12B"),
942
+ },
943
+ winner="debater_b",
944
+ symmetric_seats=["debater_a", "debater_b"],
945
+ ),
946
+ ]
947
+ result1 = fairness_table(entries, "Debate Duel")
948
+ result2 = fairness_table(list(reversed(entries)), "Debate Duel")
949
+ assert result1 == result2
950
+
951
+ def test_headline_deterministic_order_independent(self):
952
+ """headline produces the same result regardless of entry order."""
953
+ entries = [
954
+ _entry(
955
+ run_id="r1",
956
+ scenario="Debate Duel",
957
+ cast={
958
+ "debater_a": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B"),
959
+ "debater_b": CastBinding(model_endpoint="google/gemma-12B"),
960
+ },
961
+ winner="debater_a",
962
+ symmetric_seats=["debater_a", "debater_b"],
963
+ ),
964
+ _entry(
965
+ run_id="r2",
966
+ scenario="Debate Duel",
967
+ cast={
968
+ "debater_a": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B"),
969
+ "debater_b": CastBinding(model_endpoint="google/gemma-12B"),
970
+ },
971
+ winner="debater_b",
972
+ symmetric_seats=["debater_a", "debater_b"],
973
+ ),
974
+ ]
975
+ result1 = headline(entries)
976
+ result2 = headline(list(reversed(entries)))
977
+ assert result1 == result2
tests/test_leaderboard_store.py ADDED
@@ -0,0 +1,531 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Leaderboard store tests β€” persistence, round-trip, idempotence, E2E write path.
2
+
3
+ This suite verifies:
4
+ - LeaderboardStore round-trip: record β†’ read back via .entries() and .entries_for_scenario()
5
+ - Idempotent upsert on run_id (no duplicates, corrective records replace)
6
+ - Ordering: newest finished_at first, run_id tiebreak
7
+ - build_entry gating: only finished + winner + winning_model + competitive runs recorded
8
+ - Separate-table isolation: leaderboard_entries β‰  events (independent tables)
9
+ - End-to-end write path: FishbowlSession drive + finalize + leaderboard.recorded assertion
10
+
11
+ Test strategy: Build entries directly for unit tests; drive a real FishbowlSession offline
12
+ (deterministic stub) for the E2E test, reaching a verdict, calling finalize, and asserting
13
+ the scoreboard row landed in the store.
14
+ """
15
+
16
+ from __future__ import annotations
17
+
18
+ from datetime import datetime, timezone
19
+
20
+ import pytest
21
+
22
+ from src.core.leaderboard_store import (
23
+ LeaderboardEntry,
24
+ LeaderboardStore,
25
+ _reset_store_cache,
26
+ build_entry,
27
+ make_leaderboard_store,
28
+ )
29
+ from src.core.run_index import CastBinding, RunSummary
30
+
31
+
32
+ @pytest.fixture(autouse=True)
33
+ def _reset_leaderboard_store():
34
+ """Reset the memoised store cache before and after each test to prevent cross-test leakage."""
35
+ _reset_store_cache()
36
+ yield
37
+ _reset_store_cache()
38
+
39
+
40
+ def _entry(
41
+ run_id: str = "r1",
42
+ scenario: str = "Debate Duel",
43
+ seed: str = "seed123",
44
+ session_id: str | None = None,
45
+ competition_kind: str = "versus",
46
+ teams: dict[str, list[str]] | None = None,
47
+ symmetric_seats: list[str] | None = None,
48
+ cast: dict[str, CastBinding] | None = None,
49
+ winner: str | None = "alice",
50
+ winner_kind: str | None = "agent",
51
+ winning_model: str | None = "openai/openbmb/MiniCPM-8B",
52
+ winning_models: list[str] | None = None,
53
+ reason: str | None = "verdict",
54
+ turns: int = 5,
55
+ tokens: int = 200,
56
+ started_at: datetime | None = None,
57
+ finished_at: datetime | None = None,
58
+ ) -> LeaderboardEntry:
59
+ """Build a minimal LeaderboardEntry for testing."""
60
+ if started_at is None:
61
+ started_at = datetime(2025, 6, 14, 10, 0, 0, tzinfo=timezone.utc)
62
+ if finished_at is None:
63
+ finished_at = started_at
64
+ if cast is None:
65
+ cast = {"alice": CastBinding(model_endpoint=winning_model)}
66
+ if winning_models is None:
67
+ winning_models = [winning_model] if winning_model else []
68
+ return LeaderboardEntry(
69
+ run_id=run_id,
70
+ session_id=session_id,
71
+ scenario=scenario,
72
+ seed=seed,
73
+ competition_kind=competition_kind,
74
+ teams=teams,
75
+ symmetric_seats=symmetric_seats,
76
+ cast=cast,
77
+ winner=winner,
78
+ winner_kind=winner_kind,
79
+ winning_model=winning_model,
80
+ winning_models=winning_models,
81
+ reason=reason,
82
+ turns=turns,
83
+ tokens=tokens,
84
+ started_at=started_at,
85
+ finished_at=finished_at,
86
+ )
87
+
88
+
89
+ # ── Tests: LeaderboardStore round-trip and ordering ────────────────────────────────
90
+
91
+
92
+ class TestLeaderboardStoreRoundTrip:
93
+ """Verify record β†’ read-back: all fields survive serialization."""
94
+
95
+ def test_record_and_entries_round_trip(self):
96
+ """Recording an entry and reading back returns all fields intact."""
97
+ store = make_leaderboard_store(url="sqlite://")
98
+ entry = _entry(
99
+ run_id="r1",
100
+ scenario="Debate Duel",
101
+ seed="abc123",
102
+ cast={
103
+ "alice": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B", model_profile="large"),
104
+ "bob": CastBinding(model_endpoint="google/gemma-12B", model_profile="medium"),
105
+ },
106
+ teams={"team_a": ["alice"], "team_b": ["bob"]},
107
+ symmetric_seats=["seat_a", "seat_b"],
108
+ winner="alice",
109
+ winner_kind="agent",
110
+ winning_model="openai/openbmb/MiniCPM-8B",
111
+ winning_models=["openai/openbmb/MiniCPM-8B"],
112
+ )
113
+ store.record(entry)
114
+ entries = store.entries()
115
+ assert len(entries) == 1
116
+ row = entries[0]
117
+ assert row.run_id == "r1"
118
+ assert row.scenario == "Debate Duel"
119
+ assert row.seed == "abc123"
120
+ assert row.winner == "alice"
121
+ assert row.winner_kind == "agent"
122
+ assert row.winning_model == "openai/openbmb/MiniCPM-8B"
123
+ assert row.winning_models == ["openai/openbmb/MiniCPM-8B"]
124
+ assert "alice" in row.cast
125
+ assert row.cast["alice"].model_endpoint == "openai/openbmb/MiniCPM-8B"
126
+ assert row.cast["alice"].model_profile == "large"
127
+ assert row.teams == {"team_a": ["alice"], "team_b": ["bob"]}
128
+ assert row.symmetric_seats == ["seat_a", "seat_b"]
129
+
130
+ def test_entries_for_scenario_filters_correctly(self):
131
+ """entries_for_scenario returns only entries from that scenario."""
132
+ store = make_leaderboard_store(url="sqlite://")
133
+ store.record(_entry(run_id="r1", scenario="Debate Duel"))
134
+ store.record(_entry(run_id="r2", scenario="Trivia Night"))
135
+ store.record(_entry(run_id="r3", scenario="Debate Duel"))
136
+ debate_entries = store.entries_for_scenario("Debate Duel")
137
+ assert len(debate_entries) == 2
138
+ assert all(e.scenario == "Debate Duel" for e in debate_entries)
139
+ trivia_entries = store.entries_for_scenario("Trivia Night")
140
+ assert len(trivia_entries) == 1
141
+ assert trivia_entries[0].scenario == "Trivia Night"
142
+
143
+ def test_recorded_at_stamped_when_unset(self):
144
+ """recorded_at is stamped to UTC now if unset (inside record)."""
145
+ store = make_leaderboard_store(url="sqlite://")
146
+ entry = _entry(run_id="r1")
147
+ # Entry has recorded_at=None (from _entry default)
148
+ assert entry.recorded_at is None
149
+ recorded = store.record(entry)
150
+ # After recording, recorded_at is stamped
151
+ assert recorded.recorded_at is not None
152
+ assert recorded.recorded_at.tzinfo is not None
153
+ # Read back should have the same timestamp
154
+ entries = store.entries()
155
+ assert entries[0].recorded_at is not None
156
+ assert entries[0].recorded_at.tzinfo is not None
157
+
158
+
159
+ # ── Tests: Idempotent upsert on run_id ────────────────────────────────────────────
160
+
161
+
162
+ class TestLeaderboardStoreIdempotence:
163
+ """Verify that recording the same run_id twice produces one row, and second replaces."""
164
+
165
+ def test_recording_same_run_id_twice_produces_one_row(self):
166
+ """Recording the same run_id twice yields exactly one row (no duplicate)."""
167
+ store = make_leaderboard_store(url="sqlite://")
168
+ entry1 = _entry(run_id="r1", winner="alice", winning_model="ModelA")
169
+ store.record(entry1)
170
+ entry2 = _entry(run_id="r1", winner="bob", winning_model="ModelB") # same run_id, different winner
171
+ store.record(entry2)
172
+ entries = store.entries()
173
+ assert len(entries) == 1
174
+ assert entries[0].run_id == "r1"
175
+
176
+ def test_second_record_replaces_winner(self):
177
+ """A second record with the same run_id replaces the row (e.g., verdict supersedes budget close)."""
178
+ store = make_leaderboard_store(url="sqlite://")
179
+ entry1 = _entry(run_id="r1", winner="alice", winning_model="ModelA", reason="budget")
180
+ store.record(entry1)
181
+ entry2 = _entry(run_id="r1", winner="bob", winning_model="ModelB", reason="verdict") # corrective
182
+ store.record(entry2)
183
+ entries = store.entries()
184
+ assert len(entries) == 1
185
+ assert entries[0].winner == "bob"
186
+ assert entries[0].winning_model == "ModelB"
187
+ assert entries[0].reason == "verdict"
188
+
189
+ def test_second_record_same_data_is_idempotent(self):
190
+ """Recording the same entry twice is idempotent (no changes)."""
191
+ store = make_leaderboard_store(url="sqlite://")
192
+ entry = _entry(run_id="r1")
193
+ store.record(entry)
194
+ entries1 = store.entries()
195
+ store.record(entry)
196
+ entries2 = store.entries()
197
+ assert len(entries1) == len(entries2) == 1
198
+ assert entries1[0].run_id == entries2[0].run_id
199
+
200
+
201
+ # ── Tests: Ordering (newest finished_at first, run_id tiebreak) ───────────────────
202
+
203
+
204
+ class TestLeaderboardStoreOrdering:
205
+ """Verify .entries() and .entries_for_scenario() ordering."""
206
+
207
+ def test_entries_newest_first(self):
208
+ """entries() returns newest finished_at first."""
209
+ store = make_leaderboard_store(url="sqlite://")
210
+ old = datetime(2025, 6, 1, 10, 0, 0, tzinfo=timezone.utc)
211
+ new = datetime(2025, 6, 14, 10, 0, 0, tzinfo=timezone.utc)
212
+ store.record(_entry(run_id="r1", finished_at=old))
213
+ store.record(_entry(run_id="r2", finished_at=new))
214
+ entries = store.entries()
215
+ assert len(entries) == 2
216
+ assert entries[0].run_id == "r2" # newer
217
+ assert entries[1].run_id == "r1" # older
218
+
219
+ def test_entries_run_id_tiebreak_when_finished_at_same(self):
220
+ """When finished_at is equal, run_id breaks ties (ascending lexicographically)."""
221
+ store = make_leaderboard_store(url="sqlite://")
222
+ same_time = datetime(2025, 6, 14, 10, 0, 0, tzinfo=timezone.utc)
223
+ store.record(_entry(run_id="r2", finished_at=same_time))
224
+ store.record(_entry(run_id="r1", finished_at=same_time))
225
+ entries = store.entries()
226
+ assert len(entries) == 2
227
+ # Both have same finished_at; run_id breaks ties in ascending order (r1 < r2)
228
+ assert entries[0].run_id == "r1"
229
+ assert entries[1].run_id == "r2"
230
+
231
+ def test_entries_for_scenario_ordering(self):
232
+ """entries_for_scenario also respects newest-first ordering."""
233
+ store = make_leaderboard_store(url="sqlite://")
234
+ old = datetime(2025, 6, 1, 10, 0, 0, tzinfo=timezone.utc)
235
+ new = datetime(2025, 6, 14, 10, 0, 0, tzinfo=timezone.utc)
236
+ store.record(_entry(run_id="r1", scenario="Debate Duel", finished_at=old))
237
+ store.record(_entry(run_id="r2", scenario="Debate Duel", finished_at=new))
238
+ store.record(_entry(run_id="r3", scenario="Trivia Night", finished_at=new))
239
+ debate = store.entries_for_scenario("Debate Duel")
240
+ assert len(debate) == 2
241
+ assert debate[0].run_id == "r2" # newer
242
+ assert debate[1].run_id == "r1" # older
243
+
244
+
245
+ # ── Tests: build_entry gating (operator's requirement) ──────────────────────────────
246
+
247
+
248
+ class TestBuildEntryGating:
249
+ """Verify build_entry returns None unless finished + winner + winning_model + competitive."""
250
+
251
+ def test_build_entry_returns_none_when_not_finished(self):
252
+ """build_entry returns None when finished_at is None."""
253
+ summary = RunSummary(
254
+ run_id="r1",
255
+ scenario="Debate Duel",
256
+ winner="alice",
257
+ winning_model="ModelA",
258
+ finished_at=None,
259
+ )
260
+
261
+ class MockCompetition:
262
+ kind = "versus"
263
+ teams = None
264
+ symmetric_seats = None
265
+
266
+ result = build_entry(summary, MockCompetition())
267
+ assert result is None
268
+
269
+ def test_build_entry_returns_none_when_no_winner(self):
270
+ """build_entry returns None when winner is None or empty."""
271
+ summary = RunSummary(
272
+ run_id="r1",
273
+ scenario="Debate Duel",
274
+ winner=None,
275
+ winning_model="ModelA",
276
+ finished_at=datetime(2025, 6, 14, 10, 0, 0, tzinfo=timezone.utc),
277
+ )
278
+
279
+ class MockCompetition:
280
+ kind = "versus"
281
+ teams = None
282
+ symmetric_seats = None
283
+
284
+ result = build_entry(summary, MockCompetition())
285
+ assert result is None
286
+
287
+ def test_build_entry_returns_none_when_no_winning_model(self):
288
+ """build_entry returns None when neither winning_model nor winning_models has a value."""
289
+ summary = RunSummary(
290
+ run_id="r1",
291
+ scenario="Debate Duel",
292
+ winner="alice",
293
+ winning_model=None,
294
+ winning_models=[],
295
+ finished_at=datetime(2025, 6, 14, 10, 0, 0, tzinfo=timezone.utc),
296
+ )
297
+
298
+ class MockCompetition:
299
+ kind = "versus"
300
+ teams = None
301
+ symmetric_seats = None
302
+
303
+ result = build_entry(summary, MockCompetition())
304
+ assert result is None
305
+
306
+ def test_build_entry_returns_none_when_competition_kind_is_none(self):
307
+ """build_entry returns None when competition.kind == 'none' (non-competitive)."""
308
+ summary = RunSummary(
309
+ run_id="r1",
310
+ scenario="Exploratory Run",
311
+ winner="alice",
312
+ winning_model="ModelA",
313
+ finished_at=datetime(2025, 6, 14, 10, 0, 0, tzinfo=timezone.utc),
314
+ )
315
+
316
+ class MockCompetition:
317
+ kind = "none"
318
+ teams = None
319
+ symmetric_seats = None
320
+
321
+ result = build_entry(summary, MockCompetition())
322
+ assert result is None
323
+
324
+ def test_build_entry_returns_entry_when_all_gates_pass(self):
325
+ """build_entry returns a populated entry when finished + winner + winning_model + competitive."""
326
+ summary = RunSummary(
327
+ run_id="r1",
328
+ scenario="Debate Duel",
329
+ seed="seed123",
330
+ winner="alice",
331
+ winner_kind="agent",
332
+ winning_model="openai/openbmb/MiniCPM-8B",
333
+ winning_models=[],
334
+ finished_at=datetime(2025, 6, 14, 10, 0, 0, tzinfo=timezone.utc),
335
+ started_at=datetime(2025, 6, 14, 9, 55, 0, tzinfo=timezone.utc),
336
+ cast={"alice": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B")},
337
+ turns=5,
338
+ tokens=200,
339
+ )
340
+
341
+ class MockCompetition:
342
+ kind = "versus"
343
+ teams = None
344
+ symmetric_seats = ["debater_a"]
345
+
346
+ entry = build_entry(summary, MockCompetition())
347
+ assert entry is not None
348
+ assert entry.run_id == "r1"
349
+ assert entry.scenario == "Debate Duel"
350
+ assert entry.winner == "alice"
351
+ assert entry.winning_model == "openai/openbmb/MiniCPM-8B"
352
+ assert entry.competition_kind == "versus"
353
+
354
+ def test_build_entry_merges_winning_model_into_winning_models(self):
355
+ """build_entry includes both winning_model and winning_models in the merged list."""
356
+ summary = RunSummary(
357
+ run_id="r1",
358
+ scenario="Debate Duel",
359
+ winner="alice",
360
+ winning_model="ModelA",
361
+ winning_models=["ModelB"],
362
+ finished_at=datetime(2025, 6, 14, 10, 0, 0, tzinfo=timezone.utc),
363
+ )
364
+
365
+ class MockCompetition:
366
+ kind = "versus"
367
+ teams = None
368
+ symmetric_seats = None
369
+
370
+ entry = build_entry(summary, MockCompetition())
371
+ assert entry is not None
372
+ assert entry.winning_model == "ModelA"
373
+ assert set(entry.winning_models) == {"ModelA", "ModelB"}
374
+
375
+
376
+ # ── Tests: Separate-table isolation (leaderboard_entries β‰  events) ────────────────
377
+
378
+
379
+ class TestLeaderboardStoreTableIsolation:
380
+ """Verify leaderboard_entries and events tables are independent."""
381
+
382
+ def test_leaderboard_store_creates_leaderboard_entries_table(self):
383
+ """LeaderboardStore creates a dedicated leaderboard_entries table, not events."""
384
+ store = make_leaderboard_store(url="sqlite://")
385
+ entry = _entry(run_id="r1")
386
+ store.record(entry)
387
+ # Query the store's table to ensure it's leaderboard_entries
388
+ entries = store.entries()
389
+ assert len(entries) == 1
390
+ assert entries[0].run_id == "r1"
391
+
392
+ def test_multiple_databases_do_not_share_store(self):
393
+ """Two stores on different SQLite files have independent data."""
394
+ store1 = LeaderboardStore(url="sqlite:///:memory:")
395
+ store2 = LeaderboardStore(url="sqlite:///:memory:") # different in-memory DB
396
+ store1.record(_entry(run_id="r1"))
397
+ entries1 = store1.entries()
398
+ entries2 = store2.entries()
399
+ assert len(entries1) == 1
400
+ assert len(entries2) == 0 # store2 was never written to
401
+
402
+
403
+ # ── Tests: E2E write path (FishbowlSession β†’ finalize β†’ store) ────────────────────
404
+
405
+
406
+ class TestLeaderboardE2E:
407
+ """End-to-end: drive a FishbowlSession, finalize, assert leaderboard row recorded."""
408
+
409
+ def test_e2e_write_path_via_build_entry(self):
410
+ """Test the write path: build_entry β†’ store.record β†’ leaderboard_entries table."""
411
+ from src.core.leaderboard_store import build_entry
412
+
413
+ store = make_leaderboard_store(url="sqlite://")
414
+
415
+ # Simulate a finished competitive run with a winner + winning_models
416
+ summary = RunSummary(
417
+ run_id="e2e-r1",
418
+ scenario="twenty-sprouts",
419
+ seed="test_seed_e2e",
420
+ winner="keeper", # Team winner
421
+ winner_kind="team",
422
+ winning_model=None,
423
+ winning_models=["stub:balanced", "stub:fast"],
424
+ finished_at=datetime(2025, 6, 14, 10, 5, 0, tzinfo=timezone.utc),
425
+ started_at=datetime(2025, 6, 14, 10, 0, 0, tzinfo=timezone.utc),
426
+ cast={
427
+ "secret-keeper": CastBinding(model_endpoint="stub:balanced", model_profile="balanced"),
428
+ "sprout-guesser": CastBinding(model_endpoint="stub:fast", model_profile="fast"),
429
+ },
430
+ turns=10,
431
+ tokens=5000,
432
+ )
433
+
434
+ # Mock competition config
435
+ class MockCompetition:
436
+ kind = "versus"
437
+ teams = {"guesser": ["sprout-guesser"], "keeper": ["secret-keeper"]}
438
+ symmetric_seats = None
439
+
440
+ # Build and record the entry
441
+ entry = build_entry(summary, MockCompetition())
442
+ assert entry is not None, "Entry should pass the eligibility gate"
443
+
444
+ recorded = store.record(entry)
445
+ assert recorded.run_id == "e2e-r1"
446
+
447
+ # Read back from store
448
+ entries = store.entries()
449
+ assert len(entries) == 1
450
+ row = entries[0]
451
+ assert row.run_id == "e2e-r1"
452
+ assert row.scenario == "twenty-sprouts"
453
+ assert row.winner == "keeper"
454
+ assert row.winner_kind == "team"
455
+ assert row.winning_models == ["stub:balanced", "stub:fast"]
456
+ assert row.competition_kind == "versus"
457
+ assert row.finished_at is not None
458
+
459
+ def test_e2e_non_competitive_scenario_not_recorded(self):
460
+ """Drive a non-competitive scenario (if any) and verify no row is recorded."""
461
+ from src.ui.fishbowl.session import FishbowlSession
462
+
463
+ # "thousand-token-wood" is non-competitive (kind="none")
464
+ store = make_leaderboard_store(url="sqlite://")
465
+ session = FishbowlSession("thousand-token-wood")
466
+ session._leaderboard = store
467
+
468
+ session.reset(seed="test_seed_456")
469
+ session.step(5)
470
+
471
+ # Finalize without forcing verdict (or with a budget close)
472
+ session.finalize("budget")
473
+
474
+ # No row should be recorded for a non-competitive run
475
+ entries = store.entries()
476
+ assert len(entries) == 0
477
+
478
+ def test_e2e_abandoned_run_not_recorded(self):
479
+ """Drive a run, finalize with no winner, verify no row recorded."""
480
+ from src.ui.fishbowl.session import FishbowlSession
481
+
482
+ store = make_leaderboard_store(url="sqlite://")
483
+ session = FishbowlSession("twenty-sprouts")
484
+ session._leaderboard = store
485
+
486
+ session.reset(seed="test_seed_789")
487
+ session.step(2)
488
+
489
+ # Finalize without a verdict (user stops, no winner)
490
+ session.finalize("user_stop")
491
+
492
+ # No row should be recorded
493
+ entries = store.entries()
494
+ assert len(entries) == 0
495
+
496
+ def test_e2e_competitive_verdict_records_row(self):
497
+ """Positive E2E: a real FishbowlSession reaching a verdict writes exactly one row.
498
+
499
+ Drives "twenty-sprouts" (a versus, code-decided ground-truth game) offline with the
500
+ deterministic stub. Each seat is bound to an explicit ``model_endpoint`` (what the
501
+ Lab does via ``cast_models``) so the run.started cast map carries endpoints and the
502
+ "winning model selected" gate is satisfied β€” without that, the run is finished and
503
+ won but has no concrete winning model, so no row is recorded (see
504
+ ``test_e2e_abandoned_run_not_recorded`` for the no-winner case). Exercises the full
505
+ ``finalize β†’ _record_leaderboard β†’ build_entry β†’ store.record`` glue.
506
+ """
507
+ from src.ui.fishbowl.session import FishbowlSession
508
+
509
+ store = make_leaderboard_store(url="sqlite://")
510
+ session = FishbowlSession("twenty-sprouts")
511
+ session._leaderboard = store
512
+ # Bind a distinct endpoint to every seat (the Lab's cast_models, in miniature).
513
+ for agent in session.conductor.scenario.agents:
514
+ agent.manifest = agent.manifest.model_copy(update={"model_endpoint": f"openai/test/{agent.manifest.name}"})
515
+
516
+ session.reset(seed="seed-e2e")
517
+ for _ in range(12):
518
+ session.step(1)
519
+ ruled = session.force_verdict() # lands judge.verdict + finalize("verdict")
520
+ assert ruled is True
521
+
522
+ rows = [r for r in store.entries() if r.run_id == session.conductor.run_id]
523
+ assert len(rows) == 1
524
+ row = rows[0]
525
+ assert row.scenario == "twenty-sprouts"
526
+ assert row.competition_kind == "versus"
527
+ assert row.reason == "verdict"
528
+ assert row.winner # a real seat was crowned by the code-decided judge
529
+ # The winner's bound endpoint is credited as the winning model.
530
+ assert row.winning_models == [f"openai/test/{row.winner}"]
531
+ assert row.winning_model == f"openai/test/{row.winner}"