Merge pull request #22 from abducodez/feat/hall-of-fame-leaderboard
Browse filesfeat: Add Hall of Fame leaderboard backed by a dedicated results table
- docs/adr/0035-hall-of-fame-leaderboard.md +248 -0
- docs/architecture/fishbowl-ui.md +38 -4
- docs/architecture/next-steps/arena-roadmap.md +10 -2
- src/core/leaderboard.py +372 -0
- src/core/leaderboard_store.py +380 -0
- src/ui/fishbowl/app.py +31 -0
- src/ui/fishbowl/hall_of_fame.py +798 -0
- src/ui/fishbowl/session.py +38 -0
- tests/test_leaderboard.py +977 -0
- tests/test_leaderboard_store.py +531 -0
docs/adr/0035-hall-of-fame-leaderboard.md
ADDED
|
@@ -0,0 +1,248 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# ADR-0035: Hall of Fame β Dedicated Scoreboard Table, Detached from the Event Ledger
|
| 2 |
+
|
| 3 |
+
**Status:** Accepted
|
| 4 |
+
**Date:** 2026-06-14
|
| 5 |
+
|
| 6 |
+
## Context
|
| 7 |
+
|
| 8 |
+
Every competitive run now crowns a winner ([ADR-0029](0029-structured-verdicts.md),
|
| 9 |
+
[ADR-0030](0030-arena-scenarios-and-offline-winners.md)). What was missing was a
|
| 10 |
+
place to *remember*. Without a leaderboard the question "which model wins Debate Duel
|
| 11 |
+
the most?" has no answer β wins happen, are faithfully recorded on the ledger, and
|
| 12 |
+
then vanish from view the moment the next run starts.
|
| 13 |
+
|
| 14 |
+
The original design (the initial version of this ADR) proposed the leaderboard as a
|
| 15 |
+
*pure projection over the event stream* β a read-only fold with no new storage. That
|
| 16 |
+
approach was replaced, at the operator's explicit request, with the dedicated-table
|
| 17 |
+
design described here. The core trade-off: pure projections are elegant and
|
| 18 |
+
philosophically clean, but at page-load time they must fold the *entire* events log
|
| 19 |
+
β every utterance, thought, and verdict in every run β just to count wins. As the
|
| 20 |
+
ledger grows, that becomes an O(all-events) query on every Hall of Fame tab refresh.
|
| 21 |
+
A dedicated scoreboard table turns that into a cheap `SELECT` over a small,
|
| 22 |
+
already-decided set of rows.
|
| 23 |
+
|
| 24 |
+
Four constraints shaped the final design:
|
| 25 |
+
|
| 26 |
+
1. **One row per decided run, written once, at finish.** The scoreboard is not a live
|
| 27 |
+
aggregate; it is a durable receipt. Writing at finalization time, under an explicit
|
| 28 |
+
eligibility gate, keeps the table small and its rows trustworthy.
|
| 29 |
+
2. **Detached from the event ledger, linked by `run_id`.** The `events` table stays
|
| 30 |
+
the source of truth for *what happened* in a run (the full trace β every utterance,
|
| 31 |
+
thought, verdict, and poke). The `leaderboard_entries` table is a separate,
|
| 32 |
+
denormalised record of *who won*. They share a database instance (the same
|
| 33 |
+
`DATABASE_URL` / Postgres instance) but never share rows. The `run_id` foreign key
|
| 34 |
+
on every leaderboard entry links back to the full trace for replay β so the
|
| 35 |
+
separation is clean without being lossy.
|
| 36 |
+
3. **An explicit write gate.** A row is recorded only when the run is FINISHED, has a
|
| 37 |
+
WINNER, carries at least one concrete WINNING MODEL endpoint, and is COMPETITIVE
|
| 38 |
+
(`competition.kind != "none"`). This gate is the single choke point that keeps
|
| 39 |
+
the scoreboard honest: abandoned runs, budget closures with no verdict, and
|
| 40 |
+
exhibition-mode (`kind: none`) sessions never produce a row.
|
| 41 |
+
4. **Fairness is first-class.** Raw win counts mislead when seats are not symmetric.
|
| 42 |
+
The `LeaderboardEntry` row carries the full competition shape (`kind`, `teams`,
|
| 43 |
+
`symmetric_seats`), so the fairness rollup in `leaderboard.py` needs no registry
|
| 44 |
+
lookup and no event replay.
|
| 45 |
+
|
| 46 |
+
Workstream 6 of the [arena roadmap](../architecture/next-steps/arena-roadmap.md)
|
| 47 |
+
specified the "Hall of Fame" UI tab. This ADR records what shipped.
|
| 48 |
+
|
| 49 |
+
## Decision
|
| 50 |
+
|
| 51 |
+
### 1. `src/core/leaderboard_store.py` β the dedicated, durable table
|
| 52 |
+
|
| 53 |
+
`LeaderboardEntry` (Pydantic v2) is the denormalised scoreboard row β one per decided
|
| 54 |
+
competitive run. It carries:
|
| 55 |
+
|
| 56 |
+
- `run_id` β links back to `ledger.events_for_run(run_id)` for replay (ADR-0027).
|
| 57 |
+
- `scenario`, `seed`, `cast` (castβmodel bindings from the `run.started` stamp,
|
| 58 |
+
ADR-0030) β everything needed to describe the run without touching the events log.
|
| 59 |
+
- `winner`, `winner_kind`, `winning_model`, `winning_models` β the structured verdict
|
| 60 |
+
fields from ADR-0029.
|
| 61 |
+
- `competition_kind`, `teams`, `symmetric_seats` β the competition shape from
|
| 62 |
+
ADR-0030, needed for per-seat fairness rollups.
|
| 63 |
+
- `reason`, `turns`, `tokens`, `started_at`, `finished_at`, `recorded_at`.
|
| 64 |
+
|
| 65 |
+
`LeaderboardStore` (SQLAlchemy, mirroring `SqlAlchemyLedger`'s lazy-import pattern)
|
| 66 |
+
owns the `leaderboard_entries` table. Its `record()` method is an idempotent upsert
|
| 67 |
+
keyed on `run_id` β delete-then-insert in one transaction, so a verdict that
|
| 68 |
+
supersedes a budget close replaces the row rather than duplicating it. `entries()` and
|
| 69 |
+
`entries_for_scenario()` are the two read paths.
|
| 70 |
+
|
| 71 |
+
`make_leaderboard_store()` is a memoised factory that resolves the same `DATABASE_URL`
|
| 72 |
+
as `ledger_factory.make_ledger`. No extra configuration is needed: the store is a
|
| 73 |
+
separate table in the same Postgres instance. The in-memory (`sqlite://`) stage demo
|
| 74 |
+
works end-to-end because the factory memoises by URL β the write at `finalize` and the
|
| 75 |
+
Hall of Fame read share the same in-memory engine.
|
| 76 |
+
|
| 77 |
+
`build_entry(summary, competition)` is the eligibility gate. It accepts a `RunSummary`
|
| 78 |
+
(folded from the run's events by `index_runs`) and the scenario's `CompetitionConfig`.
|
| 79 |
+
It returns `None` β producing no row β unless all four conditions are met: `finished_at`
|
| 80 |
+
is set, `winner` is non-empty, at least one winning model endpoint is non-empty, and
|
| 81 |
+
`competition.kind != "none"`.
|
| 82 |
+
|
| 83 |
+
### 2. The write path β `FishbowlSession._record_leaderboard()`
|
| 84 |
+
|
| 85 |
+
The write happens exactly once per run, inside `FishbowlSession.finalize()`, called
|
| 86 |
+
when the run reaches `"verdict"` or `"budget"`:
|
| 87 |
+
|
| 88 |
+
```python
|
| 89 |
+
def _record_leaderboard(self) -> None:
|
| 90 |
+
# Folds the run's events into a RunSummary, checks eligibility via build_entry,
|
| 91 |
+
# and writes one row to leaderboard_entries. Idempotent. Fully defensive β
|
| 92 |
+
# any failure is swallowed so a store hiccup never breaks the show.
|
| 93 |
+
run_events = self.conductor.ledger.events_for_run(self.conductor.run_id)
|
| 94 |
+
summary = next((s for s in index_runs(run_events) if s.run_id == ...), None)
|
| 95 |
+
entry = build_entry(summary, getattr(scenario, "competition", None))
|
| 96 |
+
if entry is not None:
|
| 97 |
+
store.record(entry)
|
| 98 |
+
```
|
| 99 |
+
|
| 100 |
+
The method is fully defensive: any store failure is logged and swallowed. A
|
| 101 |
+
leaderboard write hiccup never breaks a live run.
|
| 102 |
+
|
| 103 |
+
### 3. `src/core/leaderboard.py` β aggregations over `LeaderboardEntry` rows
|
| 104 |
+
|
| 105 |
+
The five public aggregation functions take `Sequence[LeaderboardEntry]` (not an event
|
| 106 |
+
stream). They never touch the `events` table.
|
| 107 |
+
|
| 108 |
+
| Function | Returns | Sort order |
|
| 109 |
+
|---|---|---|
|
| 110 |
+
| `scenario_sessions(entries, scenario_name)` | `list[LeaderboardEntry]` | newest-first by `finished_at`, `run_id` tiebreak |
|
| 111 |
+
| `model_table(entries)` | `list[ModelRow]` | `(-win_rate, -wins, -plays, model)` |
|
| 112 |
+
| `agent_table(entries, scenario_name)` | `list[AgentRow]` | `(-win_rate, -wins, -plays, agent)` |
|
| 113 |
+
| `fairness_table(entries, scenario_name)` | `list[SeatRow]` | `(-win_rate, -wins, -plays, seat_type)` |
|
| 114 |
+
| `headline(entries)` | `str \| None` | single summary sentence |
|
| 115 |
+
|
| 116 |
+
`headline` considers only symmetric-seat scenarios, requires β₯2 distinct models that
|
| 117 |
+
have each won β₯1 game, picks the scenario with the most decided games, and shortens
|
| 118 |
+
model endpoint slugs to the segment after the last `/`. It returns `None` when the
|
| 119 |
+
table holds no qualifying data β safe at app start.
|
| 120 |
+
|
| 121 |
+
Attribution rules are identical to the previous pure-projection design: a model earns
|
| 122 |
+
one play per run (endpoint set deduplicated); win credit goes to every endpoint in
|
| 123 |
+
`winning_models βͺ {winning_model}`. `fairness_table` counts only declared seats
|
| 124 |
+
(team members and `symmetric_seats` members); cast members with no seat type (judges,
|
| 125 |
+
narrators) are excluded so they never appear as zero-percent rows.
|
| 126 |
+
|
| 127 |
+
### 4. "Hall of Fame" Gradio tab in `src/ui/fishbowl/hall_of_fame.py`
|
| 128 |
+
|
| 129 |
+
The tab calls `make_leaderboard_store().entries()` on render, passes the resulting
|
| 130 |
+
`list[LeaderboardEntry]` to the five aggregation functions, and renders HTML. It never
|
| 131 |
+
touches the event ledger or the live `Conductor`. The per-row **Replay** button calls
|
| 132 |
+
`load_replay(run_id)` from `src/ui/fishbowl/archive.py` (ADR-0027) β the full event
|
| 133 |
+
trace is still in the ledger, accessible via `run_id`, so replay requires no change to
|
| 134 |
+
the transport mechanism.
|
| 135 |
+
|
| 136 |
+
## Consequences
|
| 137 |
+
|
| 138 |
+
### Positive
|
| 139 |
+
|
| 140 |
+
- **Cheap reads.** Hall of Fame renders are a `SELECT * FROM leaderboard_entries`,
|
| 141 |
+
not an O(all-events) fold of the event log. The table stays small by design: only
|
| 142 |
+
finished, won, competitive runs produce a row.
|
| 143 |
+
- **Clean separation of trace and scoreboard.** The `events` table is the authoritative
|
| 144 |
+
record of what every agent said and thought. The `leaderboard_entries` table is the
|
| 145 |
+
permanent record of who won. Neither needs the other to do its job.
|
| 146 |
+
- **Rebuildable.** The scoreboard is derived from events: every `LeaderboardEntry` can
|
| 147 |
+
be reconstructed by replaying `events_for_run(run_id)` through `index_runs` and
|
| 148 |
+
`build_entry`. The table is not a competing source of truth; it is a materialised
|
| 149 |
+
view that happens to live in a dedicated table. If the table is lost or corrupted, a
|
| 150 |
+
rebuild script using ADR-0026's `index_runs` can regenerate it.
|
| 151 |
+
- **Idempotent write.** The upsert-on-`run_id` means a corrective re-finalize (a
|
| 152 |
+
budget close later superseded by a verdict β the ADR-0026 "last-wins" pattern)
|
| 153 |
+
produces exactly one, current row. No duplicates; no gaps.
|
| 154 |
+
- **Fairness is first-class.** Per-seat win rates are built into the row schema, not
|
| 155 |
+
bolted on later. The competition shape travels with the result, so no registry lookup
|
| 156 |
+
is needed at render time.
|
| 157 |
+
- **Replay falls out for free.** `run_id` links every scoreboard row back to its full
|
| 158 |
+
trace. ADR-0027's `load_replay` / `ReplaySession` already knows how to feed
|
| 159 |
+
historical events into The Show; the Hall of Fame is just a new call site.
|
| 160 |
+
- **Deterministic tiebreaks.** Every sort ends on a string column (`model`, `agent`,
|
| 161 |
+
`seat_type`, `run_id`) so table order is reproducible across re-renders.
|
| 162 |
+
|
| 163 |
+
### Negative / trade-offs
|
| 164 |
+
|
| 165 |
+
- **A second table can drift from the ledger.** The events log and the scoreboard are
|
| 166 |
+
separate tables. If a run's events are manually edited or a schema migration changes
|
| 167 |
+
the `run.finished` payload, the scoreboard row for that run will not update
|
| 168 |
+
automatically. The `run_id` link and the idempotent upsert mitigate this: a
|
| 169 |
+
re-finalize call regenerates the row from the current event state, and a rebuild
|
| 170 |
+
script can do the same in bulk. But the drift risk is real and was not present in a
|
| 171 |
+
pure-projection design.
|
| 172 |
+
- **Write-time eligibility is a point-in-time decision.** `build_entry` evaluates
|
| 173 |
+
eligibility at `finalize` time, against the competition config that was loaded into
|
| 174 |
+
the registry at that moment. A scenario YAML change that retroactively alters
|
| 175 |
+
`competition.kind` will not affect already-written rows. This is generally the
|
| 176 |
+
correct behaviour (the row reflects the game that was actually played), but operators
|
| 177 |
+
should be aware of it.
|
| 178 |
+
- **SQLAlchemy is now required on the write path.** The store's lazy-import pattern
|
| 179 |
+
(`SQLAlchemy` imported inside `__init__`) keeps the offline/stub path import-clean,
|
| 180 |
+
but any production `finalize` call on a Postgres deployment requires the driver.
|
| 181 |
+
This was already true of the `SqlAlchemyLedger` (ADR-0014); this ADR extends that
|
| 182 |
+
dependency to the write side of the Hall of Fame.
|
| 183 |
+
|
| 184 |
+
### Neutral
|
| 185 |
+
|
| 186 |
+
- `headline` returns `None` at startup (no qualifying data). The UI renders a
|
| 187 |
+
cheerful empty state; nothing is fabricated.
|
| 188 |
+
- Sort order is stable but opinionated: win-rate before raw wins, then plays, then
|
| 189 |
+
name. A future "sort by column" affordance can override this without changing the
|
| 190 |
+
aggregations.
|
| 191 |
+
- The store is co-located with the ledger (same `DATABASE_URL`) but is architecturally
|
| 192 |
+
independent: it could be moved to a separate database by changing the URL passed to
|
| 193 |
+
`make_leaderboard_store()`.
|
| 194 |
+
|
| 195 |
+
## Alternatives considered
|
| 196 |
+
|
| 197 |
+
- **Pure ledger projection (the original design).** The first version of this ADR
|
| 198 |
+
described five pure functions over `Iterable[Event]` β no new table, no new storage,
|
| 199 |
+
a perfect fit for ADR-0001's "ledger as source of truth" axiom. This was the
|
| 200 |
+
correct starting point. It was set aside because the O(all-events) cost per render
|
| 201 |
+
becomes perceptible as the ledger grows, and because the operator's explicit
|
| 202 |
+
requirement was a dedicated, durable scoreboard row rather than a live computation.
|
| 203 |
+
The pure-Python functions in `leaderboard.py` survive as aggregations over the
|
| 204 |
+
dedicated table's rows, so the computation model is preserved β only the input
|
| 205 |
+
changed from raw events to pre-decided `LeaderboardEntry` rows.
|
| 206 |
+
|
| 207 |
+
- **Extend `RunSummary` to carry the competition block.** This would have let a
|
| 208 |
+
pure-projection leaderboard reuse `index_runs` output cleanly (the original design's
|
| 209 |
+
main awkwardness). It was deferred rather than rejected: the change is mechanical but
|
| 210 |
+
touches ADR-0026's public surface. With the dedicated-table design, the competition
|
| 211 |
+
shape travels on the `LeaderboardEntry` row itself, making this less urgent.
|
| 212 |
+
|
| 213 |
+
- **SQL aggregate query as the primary path.** Faster at scale than folding Python
|
| 214 |
+
objects, but adds a backend-specific code path and a `SqlAlchemyLedger`-only
|
| 215 |
+
dependency into the aggregation module. The dedicated table achieves the same
|
| 216 |
+
read-performance goal with a simpler interface (fold a small list of Pydantic
|
| 217 |
+
objects). Revisit if the leaderboard's aggregation logic becomes a bottleneck.
|
| 218 |
+
|
| 219 |
+
- **Elo instead of win rate.** More statistically meaningful once session counts
|
| 220 |
+
grow, but requires at least ~30 decided games per model pair to be trustworthy.
|
| 221 |
+
Raw win rate is honest at small N and does not mislead when shown alongside the
|
| 222 |
+
play count. Elo is noted in the arena roadmap's "Beyond" section.
|
| 223 |
+
|
| 224 |
+
## References
|
| 225 |
+
|
| 226 |
+
- [ADR-0001](0001-event-ledger.md) β event ledger as the single source of truth for
|
| 227 |
+
*the trace*. The scoreboard is a derived, rebuildable materialisation keyed by
|
| 228 |
+
`run_id`, not a competing source of truth.
|
| 229 |
+
- [ADR-0014](0014-postgres-event-store.md) β durable Postgres store behind the ledger
|
| 230 |
+
interface; the `DATABASE_URL` the leaderboard store shares.
|
| 231 |
+
- [ADR-0026](0026-run-lifecycle-and-history.md) β run lifecycle; `run.started` /
|
| 232 |
+
`run.finished`; `RunSummary`; `index_runs` β the fold `build_entry` calls.
|
| 233 |
+
- [ADR-0027](0027-per-user-sessions-and-archive-replay.md) β `ReplaySession` and
|
| 234 |
+
`load_replay`; the replay transport the Hall of Fame Replay button reuses via `run_id`.
|
| 235 |
+
- [ADR-0029](0029-structured-verdicts.md) β structured `winner` / `winner_kind` /
|
| 236 |
+
`winning_models` on `run.finished`; the source of the scoreboard's attribution fields.
|
| 237 |
+
- [ADR-0030](0030-arena-scenarios-and-offline-winners.md) β `competition` block
|
| 238 |
+
stamped on `run.started`; `symmetric_seats`; the eight arena-grade scenarios. The
|
| 239 |
+
competition shape travels onto every `LeaderboardEntry` row.
|
| 240 |
+
- `src/core/leaderboard_store.py` β `LeaderboardEntry`, `LeaderboardStore`,
|
| 241 |
+
`build_entry`, `make_leaderboard_store`.
|
| 242 |
+
- `src/core/leaderboard.py` β the five aggregation functions over `LeaderboardEntry`
|
| 243 |
+
rows.
|
| 244 |
+
- `src/ui/fishbowl/session.py` β `FishbowlSession._record_leaderboard()`, the write
|
| 245 |
+
path.
|
| 246 |
+
- `src/ui/fishbowl/hall_of_fame.py` β the Gradio tab.
|
| 247 |
+
- [arena roadmap Β§ Workstream 6](../architecture/next-steps/arena-roadmap.md) β the
|
| 248 |
+
original specification these projections realise.
|
docs/architecture/fishbowl-ui.md
CHANGED
|
@@ -38,11 +38,11 @@ app.py (repo root) thin shim β from src.ui.fishbowl.app import demo
|
|
| 38 |
The dependency arrow is one-way: `src/ui/fishbowl/` reads the engine's public surface;
|
| 39 |
**the engine never imports the UI** (test-enforced by `tests/test_modularity.py`).
|
| 40 |
|
| 41 |
-
## The
|
| 42 |
|
| 43 |
-
Fishbowl is a `gr.Blocks` with a top bar and a `gr.Tabs` holding **The Lab**
|
| 44 |
-
**The Show**. A per-session `gr.State` carries the live
|
| 45 |
-
play-head, so concurrent visitors never share a world.
|
| 46 |
|
| 47 |
### The Lab β compose a run
|
| 48 |
|
|
@@ -73,6 +73,37 @@ Split); a **"Read their minds" `gr.Checkbox`**; a transport (scrubber `gr.Slider
|
|
| 73 |
disturbances. `build_show` returns component handles only β every callback is wired in
|
| 74 |
`app.py`.
|
| 75 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 76 |
## The render loop β `gr.HTML` + `gr.Timer`
|
| 77 |
|
| 78 |
The Show does not stream component diffs; it **re-renders one `gr.HTML` block** each
|
|
@@ -265,3 +296,6 @@ stays green by construction.
|
|
| 265 |
- [observer-pattern.md](observer-pattern.md) β the decoupled-rendering contract this
|
| 266 |
realizes.
|
| 267 |
- [ADR-0002](../adr/0002-gradio-first.md) β chose Gradio and anticipated this migration.
|
|
|
|
|
|
|
|
|
|
|
|
| 38 |
The dependency arrow is one-way: `src/ui/fishbowl/` reads the engine's public surface;
|
| 39 |
**the engine never imports the UI** (test-enforced by `tests/test_modularity.py`).
|
| 40 |
|
| 41 |
+
## The three tabs
|
| 42 |
|
| 43 |
+
Fishbowl is a `gr.Blocks` with a top bar and a `gr.Tabs` holding **The Lab**,
|
| 44 |
+
**The Show**, and **Hall of Fame**. A per-session `gr.State` carries the live
|
| 45 |
+
`FishbowlSession` and the play-head, so concurrent visitors never share a world.
|
| 46 |
|
| 47 |
### The Lab β compose a run
|
| 48 |
|
|
|
|
| 73 |
disturbances. `build_show` returns component handles only β every callback is wired in
|
| 74 |
`app.py`.
|
| 75 |
|
| 76 |
+
### Hall of Fame β the permanent record
|
| 77 |
+
|
| 78 |
+
`hall_of_fame.py` (`build_hall_of_fame`) is a read-only tab backed by the dedicated
|
| 79 |
+
`leaderboard_entries` table ([ADR-0035](../adr/0035-hall-of-fame-leaderboard.md)).
|
| 80 |
+
On each render it calls `make_leaderboard_store().entries()` to fetch the rows, then
|
| 81 |
+
passes them to the five aggregation functions in `src/core/leaderboard.py`. It never
|
| 82 |
+
reads the event ledger and never touches the live `Conductor` or the session state.
|
| 83 |
+
The `events` log stays the trace; this table is the scoreboard β linked back via
|
| 84 |
+
`run_id` for replay, not duplicated.
|
| 85 |
+
|
| 86 |
+
- **Scenario picker** (`gr.Dropdown`) filters all tables to one scenario. The "All
|
| 87 |
+
scenarios" view is the model leaderboard's default β the cross-scenario headline.
|
| 88 |
+
- **Sessions table** (`gr.Dataframe` of `SessionRow`) lists every finished competitive
|
| 89 |
+
run: date, cast summary, winner, winning model endpoint, turns, tokens, and end
|
| 90 |
+
reason. Newest first.
|
| 91 |
+
- **Replay button** β each row carries a run id; clicking it calls `load_replay` from
|
| 92 |
+
`src/ui/fishbowl/archive.py` (ADR-0027) and hands the `ReplaySession` to The Show.
|
| 93 |
+
No new transport mechanism; replay is the same path the Archive drawer uses.
|
| 94 |
+
- **Model leaderboard** (`gr.Dataframe` of `ModelRow`) β plays, wins, win rate, and
|
| 95 |
+
scenarios per model endpoint. This is the headline demo artifact: a single ledger
|
| 96 |
+
fold turns "MiniCPM-8B has beaten Gemma-12B 7β3 at Debate Duel" into a table cell.
|
| 97 |
+
- **Agent / fairness tables** (`gr.Dataframe` of `AgentRow` and `SeatRow`) β per-seat
|
| 98 |
+
win rates for the selected scenario. Judges and other non-seat cast members are
|
| 99 |
+
excluded from `SeatRow` counts so the asymmetry is honest, not hidden.
|
| 100 |
+
- **Headline** β the `headline()` projection renders a single prose sentence above the
|
| 101 |
+
tables when the ledger holds β₯1 symmetric-seat scenario with β₯2 models having won.
|
| 102 |
+
Returns `None` at app start (no runs yet) and renders nothing; the UI handles both.
|
| 103 |
+
|
| 104 |
+
The Hall of Fame refreshes on tab focus, not on a timer. It holds no derived state β
|
| 105 |
+
every render is a fresh projection fold.
|
| 106 |
+
|
| 107 |
## The render loop β `gr.HTML` + `gr.Timer`
|
| 108 |
|
| 109 |
The Show does not stream component diffs; it **re-renders one `gr.HTML` block** each
|
|
|
|
| 296 |
- [observer-pattern.md](observer-pattern.md) β the decoupled-rendering contract this
|
| 297 |
realizes.
|
| 298 |
- [ADR-0002](../adr/0002-gradio-first.md) β chose Gradio and anticipated this migration.
|
| 299 |
+
- [ADR-0035](../adr/0035-hall-of-fame-leaderboard.md) β Hall of Fame: dedicated
|
| 300 |
+
`leaderboard_entries` table detached from the event ledger; the five aggregation
|
| 301 |
+
functions in `leaderboard.py` and the replay reuse via `run_id`.
|
docs/architecture/next-steps/arena-roadmap.md
CHANGED
|
@@ -257,8 +257,16 @@ here.
|
|
| 257 |
|
| 258 |
## Workstream 6 β Hall of Fame (leaderboard)
|
| 259 |
|
| 260 |
-
|
| 261 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 262 |
|
| 263 |
### 6.1 `src/core/leaderboard.py` β pure functions
|
| 264 |
|
|
|
|
| 257 |
|
| 258 |
## Workstream 6 β Hall of Fame (leaderboard)
|
| 259 |
|
| 260 |
+
> β
*Shipped β 2026-06-14.* The dedicated `leaderboard_entries` table, the five
|
| 261 |
+
> aggregation functions, and the Gradio tab are live.
|
| 262 |
+
> See [ADR-0035](../../adr/0035-hall-of-fame-leaderboard.md) for the as-built record.
|
| 263 |
+
> The plan below is preserved as the original spec.
|
| 264 |
+
|
| 265 |
+
Dedicated detached table β not a pure projection. A separate `leaderboard_entries`
|
| 266 |
+
table (same Postgres instance as the event ledger, never sharing rows with `events`)
|
| 267 |
+
stores one denormalised result row per decided competitive run. The Hall of Fame reads
|
| 268 |
+
this small table directly; the `events` log stays the trace. `run_id` on every row
|
| 269 |
+
links back to the full trace for replay. Requires W1 (run lifecycle) + W2 (structured winner).
|
| 270 |
|
| 271 |
### 6.1 `src/core/leaderboard.py` β pure functions
|
| 272 |
|
src/core/leaderboard.py
ADDED
|
@@ -0,0 +1,372 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Leaderboard β competitive-results aggregations over the dedicated scoreboard table.
|
| 2 |
+
|
| 3 |
+
This module is the *read model* of the Hall of Fame. It is deliberately **detached from
|
| 4 |
+
the event ledger**: it folds a list of :class:`~src.core.leaderboard_store.LeaderboardEntry`
|
| 5 |
+
rows β the materialised scoreboard persisted in the ``leaderboard_entries`` table β into
|
| 6 |
+
the model / agent / fairness tables the UI renders. It never touches the ``events`` log.
|
| 7 |
+
|
| 8 |
+
The split, in one line: :mod:`src.core.leaderboard_store` owns *persistence* (one durable
|
| 9 |
+
row per decided run, written at finish), and this module owns *aggregation* (cheap folds
|
| 10 |
+
over those rows). Because a row is only ever written for a finished, won, competitive run
|
| 11 |
+
(see ``build_entry`` and ``FishbowlSession.finalize``), the functions here can trust every
|
| 12 |
+
entry they receive is already "ranked" β they re-check ``winner`` only defensively.
|
| 13 |
+
|
| 14 |
+
Each entry is self-describing: it carries the castβmodel bindings *and* the competition
|
| 15 |
+
shape (``competition_kind`` / ``teams`` / ``symmetric_seats``), so the per-seat fairness
|
| 16 |
+
rollup needs no registry lookup and no event replay.
|
| 17 |
+
|
| 18 |
+
Schema is additive only; ``schema_version`` is unaffected (this reads a separate table).
|
| 19 |
+
"""
|
| 20 |
+
|
| 21 |
+
from __future__ import annotations
|
| 22 |
+
|
| 23 |
+
from collections import defaultdict
|
| 24 |
+
from datetime import datetime
|
| 25 |
+
from typing import Iterable, Sequence
|
| 26 |
+
|
| 27 |
+
from pydantic import BaseModel, ConfigDict, Field
|
| 28 |
+
|
| 29 |
+
from src.core.leaderboard_store import LeaderboardEntry
|
| 30 |
+
|
| 31 |
+
|
| 32 |
+
# ββ competition shape (built per entry for seat mapping) βββββββββββββββββββββββββ
|
| 33 |
+
|
| 34 |
+
|
| 35 |
+
class CompetitionBlock(BaseModel):
|
| 36 |
+
"""The competition shape of one run (built from a :class:`LeaderboardEntry`).
|
| 37 |
+
|
| 38 |
+
``kind`` is ``"versus"`` / ``"judged"`` / ``"none"``; ``teams`` (versus only) maps a
|
| 39 |
+
team label β member agent names; ``symmetric_seats`` (versus only) lists identical
|
| 40 |
+
seats that differ only by model.
|
| 41 |
+
"""
|
| 42 |
+
|
| 43 |
+
model_config = ConfigDict(extra="ignore")
|
| 44 |
+
|
| 45 |
+
kind: str = "none"
|
| 46 |
+
teams: dict[str, list[str]] | None = None
|
| 47 |
+
symmetric_seats: list[str] | None = None
|
| 48 |
+
|
| 49 |
+
|
| 50 |
+
def _block(entry: LeaderboardEntry) -> CompetitionBlock:
|
| 51 |
+
"""The competition shape carried by *entry* (no registry / ledger lookup)."""
|
| 52 |
+
return CompetitionBlock(
|
| 53 |
+
kind=entry.competition_kind,
|
| 54 |
+
teams=entry.teams,
|
| 55 |
+
symmetric_seats=entry.symmetric_seats,
|
| 56 |
+
)
|
| 57 |
+
|
| 58 |
+
|
| 59 |
+
# ββ row models βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
|
| 60 |
+
|
| 61 |
+
|
| 62 |
+
class ModelRow(BaseModel):
|
| 63 |
+
"""A model endpoint's record across *all* competitive decided runs."""
|
| 64 |
+
|
| 65 |
+
model_config = ConfigDict(extra="forbid")
|
| 66 |
+
|
| 67 |
+
model: str
|
| 68 |
+
plays: int = 0
|
| 69 |
+
"""Decided competitive runs whose cast contained this endpoint."""
|
| 70 |
+
wins: int = 0
|
| 71 |
+
"""Of those, how many this endpoint was credited with winning."""
|
| 72 |
+
win_rate: float = 0.0
|
| 73 |
+
"""``wins / plays`` (0.0 when ``plays == 0``)."""
|
| 74 |
+
scenarios: list[str] = Field(default_factory=list)
|
| 75 |
+
"""Sorted distinct scenario names this endpoint appeared in."""
|
| 76 |
+
|
| 77 |
+
|
| 78 |
+
class AgentRow(BaseModel):
|
| 79 |
+
"""A persona's (cast seat *name*) record within a single scenario."""
|
| 80 |
+
|
| 81 |
+
model_config = ConfigDict(extra="forbid")
|
| 82 |
+
|
| 83 |
+
agent: str
|
| 84 |
+
"""The cast member name β the seat, not the model."""
|
| 85 |
+
seat_type: str = ""
|
| 86 |
+
"""Team label, symmetric-seat name, or ``""`` when the seat maps to neither."""
|
| 87 |
+
plays: int = 0
|
| 88 |
+
wins: int = 0
|
| 89 |
+
win_rate: float = 0.0
|
| 90 |
+
model_endpoints: list[str] = Field(default_factory=list)
|
| 91 |
+
"""Sorted distinct model endpoints that filled this seat."""
|
| 92 |
+
|
| 93 |
+
|
| 94 |
+
class SeatRow(BaseModel):
|
| 95 |
+
"""Win rate per *seat type* within a scenario (the 6.3 fairness footnote).
|
| 96 |
+
|
| 97 |
+
Surfaces structural asymmetry: spy vs herd, debater-a vs debater-b, the judge that
|
| 98 |
+
never wins. ``seat_type`` is a team label (versus-teams) or a symmetric-seat name.
|
| 99 |
+
"""
|
| 100 |
+
|
| 101 |
+
model_config = ConfigDict(extra="forbid")
|
| 102 |
+
|
| 103 |
+
seat_type: str
|
| 104 |
+
plays: int = 0
|
| 105 |
+
wins: int = 0
|
| 106 |
+
win_rate: float = 0.0
|
| 107 |
+
|
| 108 |
+
|
| 109 |
+
# ββ internal helpers βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
|
| 110 |
+
|
| 111 |
+
|
| 112 |
+
def _win_rate(wins: int, plays: int) -> float:
|
| 113 |
+
"""``wins / plays``, or ``0.0`` when *plays* is zero (never divide by zero)."""
|
| 114 |
+
return (wins / plays) if plays else 0.0
|
| 115 |
+
|
| 116 |
+
|
| 117 |
+
def _ranked(entries: Iterable[LeaderboardEntry]) -> list[LeaderboardEntry]:
|
| 118 |
+
"""Defensive gate: keep only entries that actually name a winner.
|
| 119 |
+
|
| 120 |
+
The store only ever persists finished + won + competitive runs, so this is belt-and-
|
| 121 |
+
suspenders β it drops any malformed/empty row rather than crediting a phantom win.
|
| 122 |
+
"""
|
| 123 |
+
return [e for e in entries if e and e.winner]
|
| 124 |
+
|
| 125 |
+
|
| 126 |
+
def _seat_type_for(agent: str, block: CompetitionBlock) -> str:
|
| 127 |
+
"""Map a cast member *name* to its seat type within *block*.
|
| 128 |
+
|
| 129 |
+
For ``versus`` teams the seat type is the team label the agent belongs to. For
|
| 130 |
+
``symmetric_seats`` each named seat is its own type (the agent name == the seat). A
|
| 131 |
+
cast member that belongs to neither (e.g. a judge, a narrator) maps to ``""``.
|
| 132 |
+
"""
|
| 133 |
+
if block.teams:
|
| 134 |
+
for label, members in block.teams.items():
|
| 135 |
+
if agent in (members or []):
|
| 136 |
+
return label
|
| 137 |
+
if block.symmetric_seats and agent in block.symmetric_seats:
|
| 138 |
+
return agent
|
| 139 |
+
return ""
|
| 140 |
+
|
| 141 |
+
|
| 142 |
+
def _winning_seat_types(entry: LeaderboardEntry, block: CompetitionBlock) -> set[str]:
|
| 143 |
+
"""The seat type(s) credited with the win for fairness accounting."""
|
| 144 |
+
winner = entry.winner or ""
|
| 145 |
+
if not winner:
|
| 146 |
+
return set()
|
| 147 |
+
if block.teams and winner in block.teams:
|
| 148 |
+
return {winner}
|
| 149 |
+
seat = _seat_type_for(winner, block)
|
| 150 |
+
return {seat} if seat else set()
|
| 151 |
+
|
| 152 |
+
|
| 153 |
+
def _winning_agents(entry: LeaderboardEntry, block: CompetitionBlock) -> set[str]:
|
| 154 |
+
"""Cast member name(s) credited with the win, for per-seat accounting.
|
| 155 |
+
|
| 156 |
+
A ``team`` winner credits every member of the winning team; an ``agent`` winner
|
| 157 |
+
(judged pick / symmetric-seat winner) credits just that name. Falls back to treating
|
| 158 |
+
``winner`` as a bare agent name when ``winner_kind`` is absent.
|
| 159 |
+
"""
|
| 160 |
+
winner = entry.winner or ""
|
| 161 |
+
if not winner:
|
| 162 |
+
return set()
|
| 163 |
+
if entry.winner_kind == "team" or (block.teams and winner in block.teams):
|
| 164 |
+
return set((block.teams or {}).get(winner) or [])
|
| 165 |
+
return {winner}
|
| 166 |
+
|
| 167 |
+
|
| 168 |
+
def _credited_models(entry: LeaderboardEntry) -> set[str]:
|
| 169 |
+
"""The model endpoint(s) credited with this run's win (``winning_models`` βͺ single)."""
|
| 170 |
+
credited = {m for m in entry.winning_models if m}
|
| 171 |
+
if entry.winning_model:
|
| 172 |
+
credited.add(entry.winning_model)
|
| 173 |
+
return credited
|
| 174 |
+
|
| 175 |
+
|
| 176 |
+
# ββ public aggregations (fold the scoreboard rows) βββββββββββββββββββββββββββββββ
|
| 177 |
+
|
| 178 |
+
|
| 179 |
+
def scenario_sessions(entries: Sequence[LeaderboardEntry], scenario_name: str) -> list[LeaderboardEntry]:
|
| 180 |
+
"""The decided sessions of *scenario_name*, newest first.
|
| 181 |
+
|
| 182 |
+
A thin filter + deterministic sort over the stored rows: by ``finished_at``
|
| 183 |
+
descending (newest first), runs missing a finish time sorted last, ``run_id`` breaking
|
| 184 |
+
ties. Returns the :class:`LeaderboardEntry` rows themselves β they already carry the
|
| 185 |
+
winner, castβmodel bindings and cost the sessions table renders.
|
| 186 |
+
"""
|
| 187 |
+
rows = [e for e in _ranked(entries) if e.scenario == scenario_name]
|
| 188 |
+
rows.sort(key=lambda e: (e.finished_at is None, _neg_ts(e.finished_at), e.run_id))
|
| 189 |
+
return rows
|
| 190 |
+
|
| 191 |
+
|
| 192 |
+
def model_table(entries: Sequence[LeaderboardEntry]) -> list[ModelRow]:
|
| 193 |
+
"""One :class:`ModelRow` per model endpoint across *all* decided competitive runs.
|
| 194 |
+
|
| 195 |
+
A model *plays* a run when its endpoint appears in that run's cast; it *wins* when its
|
| 196 |
+
endpoint is among the run's credited winners (``winning_models`` / ``winning_model``).
|
| 197 |
+
``scenarios`` lists the distinct scenario names the model appeared in (sorted). Sorted
|
| 198 |
+
by ``win_rate`` desc, then ``wins`` desc, then ``plays`` desc, then ``model`` asc.
|
| 199 |
+
"""
|
| 200 |
+
plays: dict[str, int] = defaultdict(int)
|
| 201 |
+
wins: dict[str, int] = defaultdict(int)
|
| 202 |
+
scenarios: dict[str, set[str]] = defaultdict(set)
|
| 203 |
+
for entry in _ranked(entries):
|
| 204 |
+
credited = _credited_models(entry)
|
| 205 |
+
seen: set[str] = set()
|
| 206 |
+
for binding in entry.cast.values():
|
| 207 |
+
endpoint = binding.model_endpoint
|
| 208 |
+
if not endpoint or endpoint in seen:
|
| 209 |
+
continue # one play per endpoint per run, even if it fills two seats
|
| 210 |
+
seen.add(endpoint)
|
| 211 |
+
plays[endpoint] += 1
|
| 212 |
+
scenarios[endpoint].add(entry.scenario)
|
| 213 |
+
if endpoint in credited:
|
| 214 |
+
wins[endpoint] += 1
|
| 215 |
+
rows = [
|
| 216 |
+
ModelRow(
|
| 217 |
+
model=endpoint,
|
| 218 |
+
plays=plays[endpoint],
|
| 219 |
+
wins=wins[endpoint],
|
| 220 |
+
win_rate=_win_rate(wins[endpoint], plays[endpoint]),
|
| 221 |
+
scenarios=sorted(scenarios[endpoint]),
|
| 222 |
+
)
|
| 223 |
+
for endpoint in plays
|
| 224 |
+
]
|
| 225 |
+
rows.sort(key=lambda r: (-r.win_rate, -r.wins, -r.plays, r.model))
|
| 226 |
+
return rows
|
| 227 |
+
|
| 228 |
+
|
| 229 |
+
def agent_table(entries: Sequence[LeaderboardEntry], scenario_name: str) -> list[AgentRow]:
|
| 230 |
+
"""Per-persona (cast seat *name*) wins within *scenario_name*.
|
| 231 |
+
|
| 232 |
+
One :class:`AgentRow` per cast member name that appears in a decided run of the
|
| 233 |
+
scenario. A seat *plays* a run when its name is in the cast and *wins* when it is the
|
| 234 |
+
run's winning agent, or a member of the winning team. ``seat_type`` is the seat's team
|
| 235 |
+
label / symmetric-seat name (or ``""``); ``model_endpoints`` lists the distinct models
|
| 236 |
+
that filled the seat. Deterministic sort matching :func:`model_table`.
|
| 237 |
+
"""
|
| 238 |
+
plays: dict[str, int] = defaultdict(int)
|
| 239 |
+
wins: dict[str, int] = defaultdict(int)
|
| 240 |
+
seat_types: dict[str, str] = {}
|
| 241 |
+
endpoints: dict[str, set[str]] = defaultdict(set)
|
| 242 |
+
for entry in _ranked(entries):
|
| 243 |
+
if entry.scenario != scenario_name:
|
| 244 |
+
continue
|
| 245 |
+
block = _block(entry)
|
| 246 |
+
winners = _winning_agents(entry, block)
|
| 247 |
+
for name, binding in entry.cast.items():
|
| 248 |
+
plays[name] += 1
|
| 249 |
+
seat_types.setdefault(name, _seat_type_for(name, block))
|
| 250 |
+
if binding.model_endpoint:
|
| 251 |
+
endpoints[name].add(binding.model_endpoint)
|
| 252 |
+
if name in winners:
|
| 253 |
+
wins[name] += 1
|
| 254 |
+
rows = [
|
| 255 |
+
AgentRow(
|
| 256 |
+
agent=name,
|
| 257 |
+
seat_type=seat_types.get(name, ""),
|
| 258 |
+
plays=plays[name],
|
| 259 |
+
wins=wins[name],
|
| 260 |
+
win_rate=_win_rate(wins[name], plays[name]),
|
| 261 |
+
model_endpoints=sorted(endpoints[name]),
|
| 262 |
+
)
|
| 263 |
+
for name in plays
|
| 264 |
+
]
|
| 265 |
+
rows.sort(key=lambda r: (-r.win_rate, -r.wins, -r.plays, r.agent))
|
| 266 |
+
return rows
|
| 267 |
+
|
| 268 |
+
|
| 269 |
+
def fairness_table(entries: Sequence[LeaderboardEntry], scenario_name: str) -> list[SeatRow]:
|
| 270 |
+
"""Win rate per *seat type* within *scenario_name* β the 6.3 fairness footnote.
|
| 271 |
+
|
| 272 |
+
Aggregates the per-persona view up to seat types so structural asymmetry is visible:
|
| 273 |
+
spy vs herd, debater-a vs debater-b, a judge that never wins. Seat membership comes
|
| 274 |
+
from each entry's stored competition shape (``teams`` β label per member;
|
| 275 |
+
``symmetric_seats`` β each seat its own type). A run contributes one *play* to each
|
| 276 |
+
seat type present in its cast, and one *win* to whichever seat type the winner maps to.
|
| 277 |
+
Unmapped cast members (``seat_type == ""``) are not counted β only declared seats
|
| 278 |
+
appear. Sorted by ``win_rate`` desc, ``wins`` desc, ``plays`` desc, ``seat_type`` asc.
|
| 279 |
+
"""
|
| 280 |
+
plays: dict[str, int] = defaultdict(int)
|
| 281 |
+
wins: dict[str, int] = defaultdict(int)
|
| 282 |
+
for entry in _ranked(entries):
|
| 283 |
+
if entry.scenario != scenario_name:
|
| 284 |
+
continue
|
| 285 |
+
block = _block(entry)
|
| 286 |
+
seats_present = {st for st in (_seat_type_for(n, block) for n in entry.cast) if st}
|
| 287 |
+
for seat in seats_present:
|
| 288 |
+
plays[seat] += 1
|
| 289 |
+
for seat in _winning_seat_types(entry, block):
|
| 290 |
+
if seat in seats_present:
|
| 291 |
+
wins[seat] += 1
|
| 292 |
+
rows = [
|
| 293 |
+
SeatRow(
|
| 294 |
+
seat_type=seat,
|
| 295 |
+
plays=plays[seat],
|
| 296 |
+
wins=wins[seat],
|
| 297 |
+
win_rate=_win_rate(wins[seat], plays[seat]),
|
| 298 |
+
)
|
| 299 |
+
for seat in plays
|
| 300 |
+
]
|
| 301 |
+
rows.sort(key=lambda r: (-r.win_rate, -r.wins, -r.plays, r.seat_type))
|
| 302 |
+
return rows
|
| 303 |
+
|
| 304 |
+
|
| 305 |
+
def headline(entries: Sequence[LeaderboardEntry]) -> str | None:
|
| 306 |
+
"""The killer demo line, or ``None`` when there isn't enough data.
|
| 307 |
+
|
| 308 |
+
Looks for the most-played *symmetric-seat* scenario (the "which model argues better"
|
| 309 |
+
comparison) and, within it, the two models with the most head-to-head wins. Renders
|
| 310 |
+
e.g. ``"MiniCPM-8B beats Gemma-12B Β· 7-3 at Debate Duel"``. Returns ``None`` when no
|
| 311 |
+
competitive symmetric scenario has at least two distinct models that have each won at
|
| 312 |
+
least once (so the line is never a hollow "0-0").
|
| 313 |
+
"""
|
| 314 |
+
ranked = _ranked(entries)
|
| 315 |
+
if not ranked:
|
| 316 |
+
return None
|
| 317 |
+
|
| 318 |
+
by_scenario: dict[str, list[LeaderboardEntry]] = defaultdict(list)
|
| 319 |
+
for entry in ranked:
|
| 320 |
+
if entry.symmetric_seats: # the model-vs-model comparison only
|
| 321 |
+
by_scenario[entry.scenario].append(entry)
|
| 322 |
+
if not by_scenario:
|
| 323 |
+
return None
|
| 324 |
+
|
| 325 |
+
best_line: str | None = None
|
| 326 |
+
best_key: tuple[int, int] = (-1, -1)
|
| 327 |
+
for scenario in sorted(by_scenario): # ascending scan: ties resolve to the first (alphabetical) name
|
| 328 |
+
runs = by_scenario[scenario]
|
| 329 |
+
wins: dict[str, int] = defaultdict(int)
|
| 330 |
+
plays: dict[str, int] = defaultdict(int)
|
| 331 |
+
for entry in runs:
|
| 332 |
+
credited = _credited_models(entry)
|
| 333 |
+
for endpoint in {b.model_endpoint for b in entry.cast.values() if b.model_endpoint}:
|
| 334 |
+
plays[endpoint] += 1
|
| 335 |
+
if endpoint in credited:
|
| 336 |
+
wins[endpoint] += 1
|
| 337 |
+
winners = sorted((m for m in plays if wins[m] > 0), key=lambda m: (-wins[m], -plays[m], m))
|
| 338 |
+
if len(winners) < 2:
|
| 339 |
+
continue
|
| 340 |
+
top, runner = winners[0], winners[1]
|
| 341 |
+
decided = wins[top] + wins[runner]
|
| 342 |
+
candidate_key = (decided, wins[top])
|
| 343 |
+
if candidate_key > best_key: # strict: a full tie keeps the earlier (alphabetical) scenario
|
| 344 |
+
best_key = candidate_key
|
| 345 |
+
best_line = f"{_short(top)} beats {_short(runner)} Β· {wins[top]}-{wins[runner]} at {scenario}"
|
| 346 |
+
return best_line
|
| 347 |
+
|
| 348 |
+
|
| 349 |
+
# ββ tiny formatting helpers βββββββββββββββββββββββββββββββββββββββββββββββββββββ
|
| 350 |
+
|
| 351 |
+
|
| 352 |
+
def _neg_ts(value: datetime | None) -> float:
|
| 353 |
+
"""Negated POSIX timestamp for descending sort; ``0.0`` for ``None`` (sorted last)."""
|
| 354 |
+
return -value.timestamp() if value is not None else 0.0
|
| 355 |
+
|
| 356 |
+
|
| 357 |
+
def _short(endpoint: str) -> str:
|
| 358 |
+
"""Compact a model endpoint for the headline (``"openai/openbmb/X"`` β ``"X"``)."""
|
| 359 |
+
return endpoint.rsplit("/", 1)[-1] if endpoint else endpoint
|
| 360 |
+
|
| 361 |
+
|
| 362 |
+
__all__ = [
|
| 363 |
+
"AgentRow",
|
| 364 |
+
"CompetitionBlock",
|
| 365 |
+
"ModelRow",
|
| 366 |
+
"SeatRow",
|
| 367 |
+
"agent_table",
|
| 368 |
+
"fairness_table",
|
| 369 |
+
"headline",
|
| 370 |
+
"model_table",
|
| 371 |
+
"scenario_sessions",
|
| 372 |
+
]
|
src/core/leaderboard_store.py
ADDED
|
@@ -0,0 +1,380 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Leaderboard store β a dedicated, durable table for competitive run results.
|
| 2 |
+
|
| 3 |
+
Unlike :mod:`src.core.leaderboard` (which *aggregates* results), this module owns the
|
| 4 |
+
**persistence** of one row per decided competitive run. It is deliberately **detached
|
| 5 |
+
from the event ledger**: the append-only ``events`` table stays the source of truth for
|
| 6 |
+
*what happened* in a run (every utterance, thought, verdict β the full logs), while this
|
| 7 |
+
``leaderboard_entries`` table is a separate, denormalised record of *who won* β a
|
| 8 |
+
materialised scoreboard the Hall of Fame reads without ever folding the event log.
|
| 9 |
+
|
| 10 |
+
Why a separate table (not a pure projection)
|
| 11 |
+
---------------------------------------------
|
| 12 |
+
A run's result is written **once**, at finish, only when it is genuinely decided β so the
|
| 13 |
+
Hall of Fame is a cheap ``SELECT`` over a small table rather than an O(all-events) fold of
|
| 14 |
+
the whole ledger on every page load. The two tables share one database (the same
|
| 15 |
+
``DATABASE_URL`` / Postgres instance) but never share rows: ``events`` is the trace,
|
| 16 |
+
``leaderboard_entries`` is the scoreboard. The ``run_id`` on each entry links a row back
|
| 17 |
+
to its full trace (``ledger.events_for_run(run_id)``) for replay.
|
| 18 |
+
|
| 19 |
+
The write gate (the operator's explicit requirement)
|
| 20 |
+
----------------------------------------------------
|
| 21 |
+
A row is recorded **only if** the run is finished AND a winner with a concrete winning
|
| 22 |
+
model is selected. Unfinished runs, abandoned runs (a budget close with no verdict), and
|
| 23 |
+
runs whose winner has no bound model endpoint never produce a row β see
|
| 24 |
+
:func:`build_entry` and the call site in ``FishbowlSession.finalize``.
|
| 25 |
+
|
| 26 |
+
SQLAlchemy is imported lazily inside ``LeaderboardStore.__init__`` so importing this
|
| 27 |
+
module β and therefore ``src.core.*`` β never requires SQLAlchemy or a DB driver. The
|
| 28 |
+
:class:`LeaderboardEntry` Pydantic model is import-clean and is reused by both the store
|
| 29 |
+
(persistence) and :mod:`src.core.leaderboard` (aggregation).
|
| 30 |
+
"""
|
| 31 |
+
|
| 32 |
+
from __future__ import annotations
|
| 33 |
+
|
| 34 |
+
import json
|
| 35 |
+
import os
|
| 36 |
+
from datetime import datetime, timezone
|
| 37 |
+
from pathlib import Path
|
| 38 |
+
from typing import TYPE_CHECKING, Any
|
| 39 |
+
|
| 40 |
+
from pydantic import BaseModel, ConfigDict, Field
|
| 41 |
+
|
| 42 |
+
from src.core.run_index import CastBinding, RunSummary
|
| 43 |
+
|
| 44 |
+
if TYPE_CHECKING: # types only β never imported at runtime on the offline path
|
| 45 |
+
from sqlalchemy import Engine, MetaData, Table
|
| 46 |
+
|
| 47 |
+
from src.scenarios.base import CompetitionConfig
|
| 48 |
+
|
| 49 |
+
|
| 50 |
+
# ββ the row model (shared by the store and the aggregations) βββββββββββββββββββββ
|
| 51 |
+
|
| 52 |
+
|
| 53 |
+
class LeaderboardEntry(BaseModel):
|
| 54 |
+
"""One decided competitive run β the durable scoreboard row.
|
| 55 |
+
|
| 56 |
+
Carries everything the Hall of Fame needs without touching the event ledger: the
|
| 57 |
+
castβmodel bindings, the winner and the model(s) behind it, the competition shape
|
| 58 |
+
(``kind`` / ``teams`` / ``symmetric_seats`` β needed for per-seat fairness), and the
|
| 59 |
+
run's cost/timing. ``run_id`` links the row back to its full trace for replay.
|
| 60 |
+
"""
|
| 61 |
+
|
| 62 |
+
model_config = ConfigDict(extra="forbid")
|
| 63 |
+
|
| 64 |
+
run_id: str
|
| 65 |
+
session_id: str | None = None
|
| 66 |
+
scenario: str = ""
|
| 67 |
+
seed: str = ""
|
| 68 |
+
|
| 69 |
+
# Competition shape (mirrors CompetitionConfig β needed for seat/fairness rollups).
|
| 70 |
+
competition_kind: str = "none"
|
| 71 |
+
teams: dict[str, list[str]] | None = None
|
| 72 |
+
symmetric_seats: list[str] | None = None
|
| 73 |
+
|
| 74 |
+
cast: dict[str, CastBinding] = Field(default_factory=dict)
|
| 75 |
+
|
| 76 |
+
winner: str | None = None
|
| 77 |
+
winner_kind: str | None = None
|
| 78 |
+
winning_model: str | None = None
|
| 79 |
+
winning_models: list[str] = Field(default_factory=list)
|
| 80 |
+
|
| 81 |
+
reason: str | None = None
|
| 82 |
+
turns: int = 0
|
| 83 |
+
tokens: int = 0
|
| 84 |
+
|
| 85 |
+
started_at: datetime | None = None
|
| 86 |
+
finished_at: datetime | None = None
|
| 87 |
+
recorded_at: datetime | None = None
|
| 88 |
+
"""When this scoreboard row was written (insert time)."""
|
| 89 |
+
|
| 90 |
+
|
| 91 |
+
def build_entry(summary: RunSummary, competition: "CompetitionConfig | None") -> LeaderboardEntry | None:
|
| 92 |
+
"""Build a :class:`LeaderboardEntry` from a finished run, or ``None`` if not eligible.
|
| 93 |
+
|
| 94 |
+
The single eligibility gate (the operator's requirement): the run must be **finished**
|
| 95 |
+
(``finished_at`` set), have a **winner**, carry at least one **winning model**, and be
|
| 96 |
+
**competitive** (``competition.kind != "none"``). Returns ``None`` otherwise, so the
|
| 97 |
+
caller simply doesn't record a row.
|
| 98 |
+
|
| 99 |
+
``summary`` is the :class:`~src.core.run_index.RunSummary` folded from the run's events
|
| 100 |
+
(it already resolves winner / winner_kind / winning_model(s) / cast / turns / tokens);
|
| 101 |
+
``competition`` is the scenario's config (for the seat/team shape). This function adds
|
| 102 |
+
no new attribution β it only decides eligibility and copies the fields across.
|
| 103 |
+
"""
|
| 104 |
+
kind = getattr(competition, "kind", "none") or "none"
|
| 105 |
+
winning_models = [m for m in (summary.winning_models or []) if m]
|
| 106 |
+
if summary.winning_model and summary.winning_model not in winning_models:
|
| 107 |
+
winning_models.append(summary.winning_model)
|
| 108 |
+
|
| 109 |
+
finished = summary.finished_at is not None
|
| 110 |
+
if not finished or not summary.winner or not winning_models or kind == "none":
|
| 111 |
+
return None
|
| 112 |
+
|
| 113 |
+
return LeaderboardEntry(
|
| 114 |
+
run_id=summary.run_id,
|
| 115 |
+
session_id=summary.session_id,
|
| 116 |
+
scenario=summary.scenario,
|
| 117 |
+
seed=summary.seed,
|
| 118 |
+
competition_kind=kind,
|
| 119 |
+
teams=getattr(competition, "teams", None),
|
| 120 |
+
symmetric_seats=getattr(competition, "symmetric_seats", None),
|
| 121 |
+
cast=dict(summary.cast),
|
| 122 |
+
winner=summary.winner,
|
| 123 |
+
winner_kind=summary.winner_kind,
|
| 124 |
+
winning_model=summary.winning_model,
|
| 125 |
+
winning_models=winning_models,
|
| 126 |
+
reason=summary.reason,
|
| 127 |
+
turns=summary.turns,
|
| 128 |
+
tokens=summary.tokens,
|
| 129 |
+
started_at=summary.started_at,
|
| 130 |
+
finished_at=summary.finished_at,
|
| 131 |
+
)
|
| 132 |
+
|
| 133 |
+
|
| 134 |
+
# ββ the durable store (mirrors SqlAlchemyLedger's lazy-SQLAlchemy pattern) ββββββββ
|
| 135 |
+
|
| 136 |
+
|
| 137 |
+
def _normalise_url(url: str | Path) -> str:
|
| 138 |
+
"""Accept a SQLAlchemy URL or a bare filesystem path (mirrors the ledger)."""
|
| 139 |
+
text = str(url)
|
| 140 |
+
if "://" in text:
|
| 141 |
+
return text
|
| 142 |
+
if text == ":memory:":
|
| 143 |
+
return "sqlite://"
|
| 144 |
+
return f"sqlite:///{text}"
|
| 145 |
+
|
| 146 |
+
|
| 147 |
+
class LeaderboardStore:
|
| 148 |
+
"""Durable store for :class:`LeaderboardEntry` rows in a dedicated table.
|
| 149 |
+
|
| 150 |
+
Backed by SQLAlchemy (Postgres or SQLite), exactly like ``SqlAlchemyLedger`` β but it
|
| 151 |
+
owns the ``leaderboard_entries`` table, never ``events``. ``record`` is an idempotent
|
| 152 |
+
upsert keyed on ``run_id`` (a re-finalised run β e.g. a budget close later superseded
|
| 153 |
+
by a verdict β replaces its row rather than duplicating it).
|
| 154 |
+
"""
|
| 155 |
+
|
| 156 |
+
def __init__(self, url: str | Path = ":memory:") -> None:
|
| 157 |
+
# Lazy import: keeps src.core.* importable without SQLAlchemy installed.
|
| 158 |
+
from sqlalchemy import (
|
| 159 |
+
Column,
|
| 160 |
+
DateTime,
|
| 161 |
+
Integer,
|
| 162 |
+
MetaData,
|
| 163 |
+
String,
|
| 164 |
+
Table,
|
| 165 |
+
Text,
|
| 166 |
+
create_engine,
|
| 167 |
+
)
|
| 168 |
+
|
| 169 |
+
self._url = _normalise_url(url)
|
| 170 |
+
self._engine: Engine = create_engine(self._url, pool_pre_ping=True)
|
| 171 |
+
self._metadata: MetaData = MetaData()
|
| 172 |
+
self._table: Table = Table(
|
| 173 |
+
"leaderboard_entries",
|
| 174 |
+
self._metadata,
|
| 175 |
+
Column("offset", Integer, primary_key=True, autoincrement=True),
|
| 176 |
+
Column("run_id", String(64), unique=True, nullable=False, index=True),
|
| 177 |
+
Column("session_id", String, nullable=True, index=True),
|
| 178 |
+
Column("scenario", String, nullable=False, index=True),
|
| 179 |
+
Column("seed", String, nullable=False, server_default=""),
|
| 180 |
+
Column("competition_kind", String, nullable=False, server_default="none"),
|
| 181 |
+
Column("teams", Text, nullable=True),
|
| 182 |
+
Column("symmetric_seats", Text, nullable=True),
|
| 183 |
+
Column("cast", Text, nullable=False, server_default="{}"),
|
| 184 |
+
Column("winner", String, nullable=True),
|
| 185 |
+
Column("winner_kind", String, nullable=True),
|
| 186 |
+
Column("winning_model", String, nullable=True),
|
| 187 |
+
Column("winning_models", Text, nullable=False, server_default="[]"),
|
| 188 |
+
Column("reason", String, nullable=True),
|
| 189 |
+
Column("turns", Integer, nullable=False, server_default="0"),
|
| 190 |
+
Column("tokens", Integer, nullable=False, server_default="0"),
|
| 191 |
+
Column("started_at", DateTime(timezone=True), nullable=True),
|
| 192 |
+
Column("finished_at", DateTime(timezone=True), nullable=True),
|
| 193 |
+
Column("recorded_at", DateTime(timezone=True), nullable=False),
|
| 194 |
+
)
|
| 195 |
+
self._metadata.create_all(self._engine)
|
| 196 |
+
|
| 197 |
+
# ββ write ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
|
| 198 |
+
|
| 199 |
+
def record(self, entry: LeaderboardEntry) -> LeaderboardEntry:
|
| 200 |
+
"""Persist *entry*, upserting on ``run_id`` (idempotent β never duplicates a run).
|
| 201 |
+
|
| 202 |
+
Stamps ``recorded_at`` if unset. A pre-existing row for the same ``run_id`` is
|
| 203 |
+
replaced (delete-then-insert in one transaction), so a corrective re-finalise
|
| 204 |
+
keeps exactly one, current scoreboard row per run.
|
| 205 |
+
"""
|
| 206 |
+
recorded = entry.model_copy(update={"recorded_at": entry.recorded_at or datetime.now(timezone.utc)})
|
| 207 |
+
values = self._to_row(recorded)
|
| 208 |
+
with self._engine.begin() as conn:
|
| 209 |
+
conn.execute(self._table.delete().where(self._table.c.run_id == recorded.run_id))
|
| 210 |
+
conn.execute(self._table.insert().values(**values))
|
| 211 |
+
return recorded
|
| 212 |
+
|
| 213 |
+
# ββ read βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
|
| 214 |
+
|
| 215 |
+
def entries(self) -> list[LeaderboardEntry]:
|
| 216 |
+
"""Every recorded entry, newest finish first (``run_id`` breaks ties)."""
|
| 217 |
+
from sqlalchemy import select
|
| 218 |
+
|
| 219 |
+
t = self._table
|
| 220 |
+
stmt = select(t).order_by(t.c.offset)
|
| 221 |
+
with self._engine.connect() as conn:
|
| 222 |
+
rows = conn.execute(stmt).mappings().all()
|
| 223 |
+
out = [self._row_to_entry(row) for row in rows]
|
| 224 |
+
out.sort(key=lambda e: (e.finished_at is None, _neg_ts(e.finished_at), e.run_id))
|
| 225 |
+
return out
|
| 226 |
+
|
| 227 |
+
def entries_for_scenario(self, scenario: str) -> list[LeaderboardEntry]:
|
| 228 |
+
"""The recorded entries for one scenario (indexed query), newest finish first."""
|
| 229 |
+
from sqlalchemy import select
|
| 230 |
+
|
| 231 |
+
t = self._table
|
| 232 |
+
stmt = select(t).where(t.c.scenario == scenario).order_by(t.c.offset)
|
| 233 |
+
with self._engine.connect() as conn:
|
| 234 |
+
rows = conn.execute(stmt).mappings().all()
|
| 235 |
+
out = [self._row_to_entry(row) for row in rows]
|
| 236 |
+
out.sort(key=lambda e: (e.finished_at is None, _neg_ts(e.finished_at), e.run_id))
|
| 237 |
+
return out
|
| 238 |
+
|
| 239 |
+
def close(self) -> None:
|
| 240 |
+
self._engine.dispose()
|
| 241 |
+
|
| 242 |
+
# ββ internal (de)serialisation βββββββββββββββββββββββββββββββββββββββββββββββ
|
| 243 |
+
|
| 244 |
+
@staticmethod
|
| 245 |
+
def _to_row(entry: LeaderboardEntry) -> dict[str, Any]:
|
| 246 |
+
return {
|
| 247 |
+
"run_id": entry.run_id,
|
| 248 |
+
"session_id": entry.session_id,
|
| 249 |
+
"scenario": entry.scenario,
|
| 250 |
+
"seed": entry.seed,
|
| 251 |
+
"competition_kind": entry.competition_kind,
|
| 252 |
+
"teams": json.dumps(entry.teams) if entry.teams is not None else None,
|
| 253 |
+
"symmetric_seats": (json.dumps(entry.symmetric_seats) if entry.symmetric_seats is not None else None),
|
| 254 |
+
"cast": json.dumps({name: binding.model_dump() for name, binding in entry.cast.items()}),
|
| 255 |
+
"winner": entry.winner,
|
| 256 |
+
"winner_kind": entry.winner_kind,
|
| 257 |
+
"winning_model": entry.winning_model,
|
| 258 |
+
"winning_models": json.dumps(list(entry.winning_models)),
|
| 259 |
+
"reason": entry.reason,
|
| 260 |
+
"turns": int(entry.turns),
|
| 261 |
+
"tokens": int(entry.tokens),
|
| 262 |
+
"started_at": _aware(entry.started_at),
|
| 263 |
+
"finished_at": _aware(entry.finished_at),
|
| 264 |
+
"recorded_at": _aware(entry.recorded_at) or datetime.now(timezone.utc),
|
| 265 |
+
}
|
| 266 |
+
|
| 267 |
+
@staticmethod
|
| 268 |
+
def _row_to_entry(row: Any) -> LeaderboardEntry:
|
| 269 |
+
cast_raw = _loads(row["cast"], {})
|
| 270 |
+
cast = {name: CastBinding(**(binding or {})) for name, binding in cast_raw.items()}
|
| 271 |
+
return LeaderboardEntry(
|
| 272 |
+
run_id=row["run_id"],
|
| 273 |
+
session_id=row.get("session_id"),
|
| 274 |
+
scenario=row["scenario"],
|
| 275 |
+
seed=row["seed"] or "",
|
| 276 |
+
competition_kind=row["competition_kind"] or "none",
|
| 277 |
+
teams=_loads(row.get("teams"), None),
|
| 278 |
+
symmetric_seats=_loads(row.get("symmetric_seats"), None),
|
| 279 |
+
cast=cast,
|
| 280 |
+
winner=row.get("winner"),
|
| 281 |
+
winner_kind=row.get("winner_kind"),
|
| 282 |
+
winning_model=row.get("winning_model"),
|
| 283 |
+
winning_models=_loads(row.get("winning_models"), []) or [],
|
| 284 |
+
reason=row.get("reason"),
|
| 285 |
+
turns=row["turns"] or 0,
|
| 286 |
+
tokens=row["tokens"] or 0,
|
| 287 |
+
started_at=_aware(_parse_dt(row.get("started_at"))),
|
| 288 |
+
finished_at=_aware(_parse_dt(row.get("finished_at"))),
|
| 289 |
+
recorded_at=_aware(_parse_dt(row.get("recorded_at"))),
|
| 290 |
+
)
|
| 291 |
+
|
| 292 |
+
|
| 293 |
+
# ββ factory (mirrors src.core.ledger_factory β same DATABASE_URL, separate table) β
|
| 294 |
+
|
| 295 |
+
# Memoise one store per resolved URL. This is what makes the no-key, in-memory
|
| 296 |
+
# (``sqlite://``) stage demo work end-to-end within a process: the write at finalize and
|
| 297 |
+
# the Hall-of-Fame read share the *same* engine (a fresh in-memory engine would be a
|
| 298 |
+
# different, empty database). For a file/Postgres URL it just avoids re-opening engines.
|
| 299 |
+
_STORES: dict[str, LeaderboardStore] = {}
|
| 300 |
+
|
| 301 |
+
|
| 302 |
+
def make_leaderboard_store(url: str | None = None) -> LeaderboardStore:
|
| 303 |
+
"""Construct (or reuse) the leaderboard store for the configured database.
|
| 304 |
+
|
| 305 |
+
Resolves the same ``DATABASE_URL`` as the event ledger (a separate *table* in the
|
| 306 |
+
same database), so no extra configuration is needed. *url* overrides it (tests pass
|
| 307 |
+
``"sqlite://"``). Raises :class:`RuntimeError` when neither is set β like the ledger,
|
| 308 |
+
the durable store is not optional.
|
| 309 |
+
"""
|
| 310 |
+
resolved = url or os.getenv("DATABASE_URL")
|
| 311 |
+
if not resolved:
|
| 312 |
+
raise RuntimeError(
|
| 313 |
+
"DATABASE_URL is required for the leaderboard store β it lives in the same "
|
| 314 |
+
"database as the event ledger (a separate table), so set DATABASE_URL or pass "
|
| 315 |
+
"an explicit url to make_leaderboard_store()."
|
| 316 |
+
)
|
| 317 |
+
from src.core.ledger_factory import _normalize_db_url
|
| 318 |
+
|
| 319 |
+
key = _normalize_db_url(resolved)
|
| 320 |
+
store = _STORES.get(key)
|
| 321 |
+
if store is None:
|
| 322 |
+
store = LeaderboardStore(key)
|
| 323 |
+
_STORES[key] = store
|
| 324 |
+
return store
|
| 325 |
+
|
| 326 |
+
|
| 327 |
+
def _reset_store_cache() -> None:
|
| 328 |
+
"""Drop memoised stores (test hook β production never needs this)."""
|
| 329 |
+
for store in _STORES.values():
|
| 330 |
+
try:
|
| 331 |
+
store.close()
|
| 332 |
+
except Exception: # pragma: no cover - best-effort cleanup
|
| 333 |
+
pass
|
| 334 |
+
_STORES.clear()
|
| 335 |
+
|
| 336 |
+
|
| 337 |
+
# ββ tiny datetime helpers (mirror the ledger's tz coercion) ββββββββββββββββββββββ
|
| 338 |
+
|
| 339 |
+
|
| 340 |
+
def _aware(dt: datetime | None) -> datetime | None:
|
| 341 |
+
"""Coerce a datetime to tz-aware UTC; pass ``None`` through (SQLite drops tzinfo)."""
|
| 342 |
+
if dt is None:
|
| 343 |
+
return None
|
| 344 |
+
if dt.tzinfo is None:
|
| 345 |
+
return dt.replace(tzinfo=timezone.utc)
|
| 346 |
+
return dt
|
| 347 |
+
|
| 348 |
+
|
| 349 |
+
def _parse_dt(value: Any) -> datetime | None:
|
| 350 |
+
if value is None or isinstance(value, datetime):
|
| 351 |
+
return value
|
| 352 |
+
try:
|
| 353 |
+
return datetime.fromisoformat(str(value))
|
| 354 |
+
except (ValueError, TypeError): # pragma: no cover - defensive
|
| 355 |
+
return None
|
| 356 |
+
|
| 357 |
+
|
| 358 |
+
def _neg_ts(value: datetime | None) -> float:
|
| 359 |
+
"""Negated POSIX timestamp for descending sort; ``0.0`` for ``None`` (sorted last)."""
|
| 360 |
+
return -value.timestamp() if value is not None else 0.0
|
| 361 |
+
|
| 362 |
+
|
| 363 |
+
def _loads(value: Any, default: Any) -> Any:
|
| 364 |
+
"""JSON-decode a stored text column, tolerating already-decoded values / nulls."""
|
| 365 |
+
if value is None:
|
| 366 |
+
return default
|
| 367 |
+
if isinstance(value, (dict, list)):
|
| 368 |
+
return value
|
| 369 |
+
try:
|
| 370 |
+
return json.loads(value)
|
| 371 |
+
except (ValueError, TypeError): # pragma: no cover - defensive
|
| 372 |
+
return default
|
| 373 |
+
|
| 374 |
+
|
| 375 |
+
__all__ = [
|
| 376 |
+
"LeaderboardEntry",
|
| 377 |
+
"LeaderboardStore",
|
| 378 |
+
"build_entry",
|
| 379 |
+
"make_leaderboard_store",
|
| 380 |
+
]
|
src/ui/fishbowl/app.py
CHANGED
|
@@ -49,6 +49,18 @@ except Exception: # pragma: no cover - degrade gracefully if the archive unit i
|
|
| 49 |
return "βΆ (run)"
|
| 50 |
|
| 51 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 52 |
# ββ loop-safety backstop ββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
|
| 53 |
# Belt-and-suspenders against a runaway autoplay loop: even when the governor never
|
| 54 |
# trips (e.g. a generous budget) the timer halts after this many consecutive auto-ticks
|
|
@@ -709,6 +721,8 @@ def build_app() -> gr.Blocks:
|
|
| 709 |
show_handles = build_show()
|
| 710 |
with gr.Tab("Telemetry", id="telemetry"):
|
| 711 |
build_telemetry()
|
|
|
|
|
|
|
| 712 |
|
| 713 |
# CRT foreground layers (scanlines + vignette, above content, click-through).
|
| 714 |
gr.HTML(_CRT_FG_HTML)
|
|
@@ -717,6 +731,23 @@ def build_app() -> gr.Blocks:
|
|
| 717 |
# exists, so it reads the live panes through this ref.
|
| 718 |
archive_refs["show_handles"] = show_handles or {}
|
| 719 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 720 |
_wire(
|
| 721 |
tabs=tabs,
|
| 722 |
lab_handles=lab_handles or {},
|
|
|
|
| 49 |
return "βΆ (run)"
|
| 50 |
|
| 51 |
|
| 52 |
+
try: # hall of fame: the competitive high-score board (Workstream 6.2 + 6.3)
|
| 53 |
+
from src.ui.fishbowl.hall_of_fame import build_hall_of_fame, wire_sessions_render
|
| 54 |
+
except Exception: # pragma: no cover - degrade gracefully if the unit is absent
|
| 55 |
+
|
| 56 |
+
def build_hall_of_fame() -> dict:
|
| 57 |
+
gr.HTML('<div class="fishbowl"><div class="fishbowl-placeholder">π Hall of Fame β coming soon.</div></div>')
|
| 58 |
+
return {}
|
| 59 |
+
|
| 60 |
+
def wire_sessions_render(*_args, **_kwargs) -> None:
|
| 61 |
+
return None
|
| 62 |
+
|
| 63 |
+
|
| 64 |
# ββ loop-safety backstop ββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
|
| 65 |
# Belt-and-suspenders against a runaway autoplay loop: even when the governor never
|
| 66 |
# trips (e.g. a generous budget) the timer halts after this many consecutive auto-ticks
|
|
|
|
| 721 |
show_handles = build_show()
|
| 722 |
with gr.Tab("Telemetry", id="telemetry"):
|
| 723 |
build_telemetry()
|
| 724 |
+
with gr.Tab("π Hall of Fame", id="hall"):
|
| 725 |
+
hall_handles = build_hall_of_fame()
|
| 726 |
|
| 727 |
# CRT foreground layers (scanlines + vignette, above content, click-through).
|
| 728 |
gr.HTML(_CRT_FG_HTML)
|
|
|
|
| 731 |
# exists, so it reads the live panes through this ref.
|
| 732 |
archive_refs["show_handles"] = show_handles or {}
|
| 733 |
|
| 734 |
+
# The Hall of Fame's Replay buttons target the Show panes; wire them after all
|
| 735 |
+
# tabs build (mirrors the Archive's deferred show_handles trick at :644).
|
| 736 |
+
wire_sessions_render(
|
| 737 |
+
hall_handles or {},
|
| 738 |
+
refs=archive_refs,
|
| 739 |
+
tabs=tabs,
|
| 740 |
+
states={
|
| 741 |
+
"session": session_state,
|
| 742 |
+
"k": k_state,
|
| 743 |
+
"scenario": scenario_state,
|
| 744 |
+
"layout": layout_state,
|
| 745 |
+
"mind": mind_reader_state,
|
| 746 |
+
"stopped": stopped_state,
|
| 747 |
+
"ticks": tick_count_state,
|
| 748 |
+
},
|
| 749 |
+
)
|
| 750 |
+
|
| 751 |
_wire(
|
| 752 |
tabs=tabs,
|
| 753 |
lab_handles=lab_handles or {},
|
src/ui/fishbowl/hall_of_fame.py
ADDED
|
@@ -0,0 +1,798 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""HALL OF FAME β the phosphor-arcade high-score board (Workstream 6.2 + 6.3).
|
| 2 |
+
|
| 3 |
+
A cheerful, screenshot-worthy scoreboard for the competitive results the engine has
|
| 4 |
+
recorded. It is a *pure read surface* over the dedicated leaderboard table (ADR-0035):
|
| 5 |
+
every number on the board comes from the aggregations in :mod:`src.core.leaderboard`
|
| 6 |
+
(``headline`` / ``model_table`` / ``scenario_sessions`` / ``fairness_table``), which fold
|
| 7 |
+
the :class:`~src.core.leaderboard_store.LeaderboardEntry` rows persisted in
|
| 8 |
+
``leaderboard_entries`` β a table *detached from the event ledger* (the ``events`` log
|
| 9 |
+
stays the trace; this is the materialised scoreboard).
|
| 10 |
+
|
| 11 |
+
Design β "Phosphor Arcade High-Score Board"
|
| 12 |
+
-------------------------------------------
|
| 13 |
+
The tab lives inside the existing ``.fishbowl`` CRT-phosphor scope and reuses the
|
| 14 |
+
theme's design tokens, fonts (Martian Mono display, IBM Plex Mono body) and existing
|
| 15 |
+
classes (``eyebrow`` / ``chip`` / ``panel`` / verdict + winner chrome). Champions glow
|
| 16 |
+
**gold/amber** against the cool teal CRT β that warm/cool contrast is the whole
|
| 17 |
+
identity. Only two scoped vars are introduced (``--gold`` / ``--gold-soft``) plus the
|
| 18 |
+
Hall-of-Fame layout classes, all under ``.fishbowl.fb-hall``.
|
| 19 |
+
|
| 20 |
+
Everything here is **offline-safe**: pure HTML strings, an inlined ``<style>`` block,
|
| 21 |
+
no network calls and no external assets. When the store is empty or unconfigured (the
|
| 22 |
+
deterministic stub path before any competitive run finishes) every pane degrades to a
|
| 23 |
+
cheerful empty state rather than crashing.
|
| 24 |
+
|
| 25 |
+
The per-row **Replay** button mirrors the Lab's Archive drawer exactly: it turns a run
|
| 26 |
+
into a read-only :class:`~src.ui.fishbowl.session.ReplaySession` via
|
| 27 |
+
:func:`src.ui.fishbowl.archive.load_replay` and pushes it into the Show transport
|
| 28 |
+
through the same ``gr.render`` Load-button contract (see ``build_hall_of_fame``).
|
| 29 |
+
"""
|
| 30 |
+
|
| 31 |
+
from __future__ import annotations
|
| 32 |
+
|
| 33 |
+
import html as _html
|
| 34 |
+
from datetime import datetime
|
| 35 |
+
|
| 36 |
+
import gradio as gr
|
| 37 |
+
|
| 38 |
+
from src.core.leaderboard import (
|
| 39 |
+
ModelRow,
|
| 40 |
+
SeatRow,
|
| 41 |
+
fairness_table,
|
| 42 |
+
headline,
|
| 43 |
+
model_table,
|
| 44 |
+
scenario_sessions,
|
| 45 |
+
)
|
| 46 |
+
from src.core.leaderboard_store import LeaderboardEntry
|
| 47 |
+
from src.core.registry import default_registry
|
| 48 |
+
|
| 49 |
+
|
| 50 |
+
# ββ data access (defensive: empty store / no backend β empty projections) ββββββββ
|
| 51 |
+
|
| 52 |
+
|
| 53 |
+
def _entries() -> list[LeaderboardEntry]:
|
| 54 |
+
"""Every scoreboard row from the dedicated leaderboard table, or ``[]`` if unavailable.
|
| 55 |
+
|
| 56 |
+
Reads the ``leaderboard_entries`` table (ADR-0035) β *detached from the event ledger*
|
| 57 |
+
β via :func:`make_leaderboard_store`. Defensive on every axis: a missing/unconfigured
|
| 58 |
+
store or any read error degrades to an empty list so the board shows its cheerful empty
|
| 59 |
+
state instead of raising.
|
| 60 |
+
"""
|
| 61 |
+
try:
|
| 62 |
+
from src.core.leaderboard_store import make_leaderboard_store
|
| 63 |
+
|
| 64 |
+
return list(make_leaderboard_store().entries())
|
| 65 |
+
except Exception: # pragma: no cover - no store configured / read error (defensive)
|
| 66 |
+
return []
|
| 67 |
+
|
| 68 |
+
|
| 69 |
+
def competitive_scenarios() -> list[tuple[str, str]]:
|
| 70 |
+
"""``(title, internal_name)`` pairs for scenarios whose ``competition.kind != none``.
|
| 71 |
+
|
| 72 |
+
These are the only worlds that can ever crown a champion, so the scenario picker
|
| 73 |
+
is scoped to them. Returns ``[]`` when the registry can't be read.
|
| 74 |
+
"""
|
| 75 |
+
try:
|
| 76 |
+
registry = default_registry()
|
| 77 |
+
except Exception: # pragma: no cover - registry unavailable (defensive)
|
| 78 |
+
return []
|
| 79 |
+
out: list[tuple[str, str]] = []
|
| 80 |
+
for name in sorted(registry.scenarios):
|
| 81 |
+
cfg = registry.scenarios[name]
|
| 82 |
+
comp = getattr(cfg, "competition", None)
|
| 83 |
+
if getattr(comp, "kind", "none") not in ("none", None):
|
| 84 |
+
out.append(((getattr(cfg, "title", "") or name), name))
|
| 85 |
+
return out
|
| 86 |
+
|
| 87 |
+
|
| 88 |
+
# ββ tiny formatting helpers ββββββββββββββββββββββββββββββββββββββββββββββββββββββ
|
| 89 |
+
|
| 90 |
+
|
| 91 |
+
def _esc(text: object) -> str:
|
| 92 |
+
"""HTML-escape any value (defensive β inputs may be ``None`` / non-str)."""
|
| 93 |
+
return _html.escape(str(text if text is not None else ""))
|
| 94 |
+
|
| 95 |
+
|
| 96 |
+
def _short_model(endpoint: str) -> str:
|
| 97 |
+
"""Compact a model endpoint to its last path segment (``a/b/Model`` β ``Model``)."""
|
| 98 |
+
if not endpoint:
|
| 99 |
+
return "β"
|
| 100 |
+
return endpoint.rsplit("/", 1)[-1] or endpoint
|
| 101 |
+
|
| 102 |
+
|
| 103 |
+
def _pct(rate: float) -> str:
|
| 104 |
+
"""A win-rate float as a whole-percent label (``0.7`` β ``"70%"``)."""
|
| 105 |
+
try:
|
| 106 |
+
return f"{round(rate * 100)}%"
|
| 107 |
+
except Exception: # pragma: no cover - non-numeric (defensive)
|
| 108 |
+
return "0%"
|
| 109 |
+
|
| 110 |
+
|
| 111 |
+
def _bar_width(rate: float) -> float:
|
| 112 |
+
"""Clamp a win-rate to a 0β100 bar width."""
|
| 113 |
+
try:
|
| 114 |
+
return max(0.0, min(100.0, float(rate) * 100.0))
|
| 115 |
+
except Exception: # pragma: no cover - non-numeric (defensive)
|
| 116 |
+
return 0.0
|
| 117 |
+
|
| 118 |
+
|
| 119 |
+
def _short_id(run_id: str) -> str:
|
| 120 |
+
"""A glanceable run handle β last 4 of its uuid (matches the Archive)."""
|
| 121 |
+
tail = run_id.replace("-", "")[-4:] if run_id else "????"
|
| 122 |
+
return f"#{tail}"
|
| 123 |
+
|
| 124 |
+
|
| 125 |
+
def _fmt_when(value: datetime | None) -> str:
|
| 126 |
+
"""A compact ``Jun 12 Β· 14:03`` timestamp, ``β`` when missing."""
|
| 127 |
+
if not isinstance(value, datetime):
|
| 128 |
+
return "β"
|
| 129 |
+
try:
|
| 130 |
+
return value.strftime("%b %-d Β· %H:%M")
|
| 131 |
+
except ValueError: # pragma: no cover - platforms without %-d
|
| 132 |
+
return value.strftime("%b %d Β· %H:%M")
|
| 133 |
+
|
| 134 |
+
|
| 135 |
+
def _fmt_tokens(tokens: int) -> str:
|
| 136 |
+
try:
|
| 137 |
+
tokens = int(tokens)
|
| 138 |
+
except Exception: # pragma: no cover - non-numeric (defensive)
|
| 139 |
+
return "0"
|
| 140 |
+
if tokens >= 1000:
|
| 141 |
+
return f"{tokens / 1000:.1f}k"
|
| 142 |
+
return str(tokens)
|
| 143 |
+
|
| 144 |
+
|
| 145 |
+
_REASON_LABEL = {
|
| 146 |
+
"verdict": "verdict",
|
| 147 |
+
"budget": "budget",
|
| 148 |
+
"tick_cap": "tick-cap",
|
| 149 |
+
"user_stop": "stopped",
|
| 150 |
+
}
|
| 151 |
+
|
| 152 |
+
|
| 153 |
+
def _cast_models(row: LeaderboardEntry) -> str:
|
| 154 |
+
"""A compact roster of the distinct models that played a session."""
|
| 155 |
+
seen: list[str] = []
|
| 156 |
+
for binding in (row.cast or {}).values():
|
| 157 |
+
endpoint = getattr(binding, "model_endpoint", None)
|
| 158 |
+
if endpoint:
|
| 159 |
+
short = _short_model(endpoint)
|
| 160 |
+
if short not in seen:
|
| 161 |
+
seen.append(short)
|
| 162 |
+
return ", ".join(seen) if seen else "β"
|
| 163 |
+
|
| 164 |
+
|
| 165 |
+
# ββ scoped stylesheet (offline; reuses theme tokens, adds only --gold) ββββββββββββ
|
| 166 |
+
|
| 167 |
+
_HALL_STYLE = """
|
| 168 |
+
<style>
|
| 169 |
+
.fishbowl.fb-hall { display:block; }
|
| 170 |
+
.fishbowl.fb-hall {
|
| 171 |
+
--gold:#ffcf6b; /* reuse the theme's --amber gold token */
|
| 172 |
+
--gold-soft:rgba(255,207,107,0.18);
|
| 173 |
+
}
|
| 174 |
+
.fishbowl.fb-hall .hall-wrap { display:flex; flex-direction:column; gap:18px; }
|
| 175 |
+
|
| 176 |
+
/* Headline marquee β the killer demo line, big and glowing. */
|
| 177 |
+
.fishbowl.fb-hall .hall-headline {
|
| 178 |
+
position:relative; overflow:hidden;
|
| 179 |
+
padding:20px 26px; border-radius:16px;
|
| 180 |
+
border:1px solid rgba(255,207,107,0.42);
|
| 181 |
+
background:
|
| 182 |
+
radial-gradient(120% 120% at 50% -20%, var(--gold-soft), transparent 60%),
|
| 183 |
+
rgba(6,20,27,0.96);
|
| 184 |
+
box-shadow:0 0 44px rgba(255,207,107,0.22), inset 0 1px 0 rgba(255,207,107,0.12);
|
| 185 |
+
}
|
| 186 |
+
.fishbowl.fb-hall .hall-headline .eyebrow { color:var(--gold); margin-bottom:8px; }
|
| 187 |
+
.fishbowl.fb-hall .hall-line {
|
| 188 |
+
font-family:var(--font-display); font-weight:800;
|
| 189 |
+
font-size:clamp(20px, 3vw, 34px); line-height:1.18;
|
| 190 |
+
color:var(--ink);
|
| 191 |
+
text-shadow:0 0 18px rgba(255,207,107,0.45);
|
| 192 |
+
}
|
| 193 |
+
.fishbowl.fb-hall .hall-line .hl-gold { color:var(--gold); }
|
| 194 |
+
/* A shimmer sweep across the headline (CSS-only). */
|
| 195 |
+
.fishbowl.fb-hall .hall-headline::after {
|
| 196 |
+
content:""; position:absolute; inset:0; pointer-events:none;
|
| 197 |
+
background:linear-gradient(105deg, transparent 30%,
|
| 198 |
+
rgba(255,207,107,0.16) 48%, transparent 66%);
|
| 199 |
+
transform:translateX(-120%);
|
| 200 |
+
animation:hallShimmer 5.5s ease-in-out infinite;
|
| 201 |
+
}
|
| 202 |
+
@keyframes hallShimmer { 0%,55% { transform:translateX(-120%);} 78%,100% { transform:translateX(120%);} }
|
| 203 |
+
|
| 204 |
+
/* Empty states β cheerful, never a dead screen. */
|
| 205 |
+
.fishbowl.fb-hall .hall-empty {
|
| 206 |
+
padding:22px 26px; border-radius:14px;
|
| 207 |
+
border:1px dashed var(--line); background:var(--panel);
|
| 208 |
+
}
|
| 209 |
+
.fishbowl.fb-hall .hall-empty .eyebrow { color:var(--gold); margin-bottom:8px; }
|
| 210 |
+
.fishbowl.fb-hall .hall-empty .he-body { color:var(--ink-mid); line-height:1.55; }
|
| 211 |
+
|
| 212 |
+
/* Section panels reuse the theme's card chrome. */
|
| 213 |
+
.fishbowl.fb-hall .hall-panel {
|
| 214 |
+
padding:18px 20px 20px; border-radius:var(--r-lg);
|
| 215 |
+
border:1px solid var(--line); background:var(--panel);
|
| 216 |
+
box-shadow:inset 0 1px 0 rgba(120,222,214,0.07), 0 16px 38px -32px rgba(0,0,0,0.9);
|
| 217 |
+
}
|
| 218 |
+
.fishbowl.fb-hall .hall-panel > .eyebrow { margin-bottom:14px; }
|
| 219 |
+
|
| 220 |
+
/* ββ Podium (top 3): gold / silver / bronze phosphor ββ */
|
| 221 |
+
.fishbowl.fb-hall .podium {
|
| 222 |
+
display:grid; grid-template-columns:1fr 1.2fr 1fr; gap:14px; align-items:end;
|
| 223 |
+
margin-bottom:18px;
|
| 224 |
+
}
|
| 225 |
+
.fishbowl.fb-hall .podium-slot {
|
| 226 |
+
display:flex; flex-direction:column; align-items:center; gap:8px;
|
| 227 |
+
padding:16px 12px; border-radius:14px; text-align:center;
|
| 228 |
+
border:1px solid var(--line); background:rgba(6,20,27,0.7);
|
| 229 |
+
animation:hallRise 0.5s cubic-bezier(0.2,0.9,0.3,1) both;
|
| 230 |
+
}
|
| 231 |
+
.fishbowl.fb-hall .podium-slot .ps-medal { font-size:30px; line-height:1; }
|
| 232 |
+
.fishbowl.fb-hall .podium-slot .ps-name {
|
| 233 |
+
font-family:var(--font-display); font-weight:700; font-size:14px; color:var(--ink);
|
| 234 |
+
word-break:break-word;
|
| 235 |
+
}
|
| 236 |
+
.fishbowl.fb-hall .podium-slot .ps-rate {
|
| 237 |
+
font-family:var(--font-display); font-weight:800; font-variant-numeric:tabular-nums;
|
| 238 |
+
font-size:22px;
|
| 239 |
+
}
|
| 240 |
+
.fishbowl.fb-hall .podium-slot .ps-sub { font-size:11px; color:var(--ink-dim); }
|
| 241 |
+
/* #1 β gold, taller, with a soft pulse. */
|
| 242 |
+
.fishbowl.fb-hall .podium-slot.rank-1 {
|
| 243 |
+
border-color:rgba(255,207,107,0.5);
|
| 244 |
+
background:radial-gradient(120% 120% at 50% -10%, var(--gold-soft), transparent 60%), rgba(6,20,27,0.92);
|
| 245 |
+
box-shadow:0 0 40px rgba(255,207,107,0.3); padding-top:26px; animation-delay:0.05s;
|
| 246 |
+
}
|
| 247 |
+
.fishbowl.fb-hall .podium-slot.rank-1 .ps-rate { color:var(--gold); text-shadow:0 0 14px rgba(255,207,107,0.6); }
|
| 248 |
+
.fishbowl.fb-hall .podium-slot.rank-1 .ps-medal { animation:hallChampPulse 2.4s ease-in-out infinite; }
|
| 249 |
+
.fishbowl.fb-hall .podium-slot.rank-2 {
|
| 250 |
+
border-color:rgba(159,220,210,0.45); animation-delay:0.16s;
|
| 251 |
+
}
|
| 252 |
+
.fishbowl.fb-hall .podium-slot.rank-2 .ps-rate { color:var(--ink-mid); }
|
| 253 |
+
.fishbowl.fb-hall .podium-slot.rank-3 {
|
| 254 |
+
border-color:rgba(255,143,125,0.4); animation-delay:0.27s;
|
| 255 |
+
}
|
| 256 |
+
.fishbowl.fb-hall .podium-slot.rank-3 .ps-rate { color:var(--coral); }
|
| 257 |
+
|
| 258 |
+
/* ββ Ranked table with glowing win-rate bars ββ */
|
| 259 |
+
.fishbowl.fb-hall .hall-table { display:flex; flex-direction:column; gap:2px; }
|
| 260 |
+
.fishbowl.fb-hall .ht-head, .fishbowl.fb-hall .ht-row {
|
| 261 |
+
display:grid; grid-template-columns:46px 1.6fr 0.7fr 0.7fr 1.5fr; gap:12px;
|
| 262 |
+
align-items:center; padding:9px 12px; border-radius:var(--r);
|
| 263 |
+
}
|
| 264 |
+
.fishbowl.fb-hall .ht-head {
|
| 265 |
+
font-family:var(--font-display); font-size:8.5px; letter-spacing:0.2em;
|
| 266 |
+
text-transform:uppercase; color:var(--ink-faint);
|
| 267 |
+
border-bottom:1px solid var(--line-soft); border-radius:0;
|
| 268 |
+
}
|
| 269 |
+
.fishbowl.fb-hall .ht-row { animation:hallRise 0.45s ease both; }
|
| 270 |
+
.fishbowl.fb-hall .ht-row:hover { background:var(--glass); }
|
| 271 |
+
.fishbowl.fb-hall .ht-row .ht-rank {
|
| 272 |
+
font-family:var(--font-display); font-weight:700; color:var(--ink-dim);
|
| 273 |
+
font-variant-numeric:tabular-nums;
|
| 274 |
+
}
|
| 275 |
+
.fishbowl.fb-hall .ht-row.is-champ .ht-rank { color:var(--gold); }
|
| 276 |
+
.fishbowl.fb-hall .ht-row .ht-model {
|
| 277 |
+
font-family:var(--font-display); font-weight:600; color:var(--ink); word-break:break-word;
|
| 278 |
+
}
|
| 279 |
+
.fishbowl.fb-hall .ht-row .ht-num { font-variant-numeric:tabular-nums; color:var(--ink-mid); }
|
| 280 |
+
.fishbowl.fb-hall .ht-bar {
|
| 281 |
+
position:relative; height:14px; border-radius:999px;
|
| 282 |
+
background:rgba(120,222,214,0.08); overflow:hidden;
|
| 283 |
+
}
|
| 284 |
+
.fishbowl.fb-hall .ht-bar .ht-fill {
|
| 285 |
+
position:absolute; inset:0 auto 0 0; height:100%; border-radius:999px;
|
| 286 |
+
background:linear-gradient(90deg, rgba(255,207,107,0.55), var(--gold));
|
| 287 |
+
box-shadow:0 0 12px rgba(255,207,107,0.45);
|
| 288 |
+
}
|
| 289 |
+
.fishbowl.fb-hall .ht-bar .ht-bar-label {
|
| 290 |
+
position:absolute; right:8px; top:50%; transform:translateY(-50%);
|
| 291 |
+
font-family:var(--font-display); font-size:9px; font-weight:700;
|
| 292 |
+
color:var(--bg-0); text-shadow:0 0 3px rgba(255,207,107,0.8);
|
| 293 |
+
font-variant-numeric:tabular-nums;
|
| 294 |
+
}
|
| 295 |
+
|
| 296 |
+
/* ββ Sessions table ββ */
|
| 297 |
+
.fishbowl.fb-hall .sess-head, .fishbowl.fb-hall .sess-row {
|
| 298 |
+
display:grid; grid-template-columns:0.7fr 1.4fr 1.1fr 0.9fr 0.7fr 0.9fr;
|
| 299 |
+
gap:10px; align-items:center; padding:9px 12px; border-radius:var(--r);
|
| 300 |
+
}
|
| 301 |
+
.fishbowl.fb-hall .sess-head {
|
| 302 |
+
font-family:var(--font-display); font-size:8.5px; letter-spacing:0.2em;
|
| 303 |
+
text-transform:uppercase; color:var(--ink-faint);
|
| 304 |
+
border-bottom:1px solid var(--line-soft); border-radius:0;
|
| 305 |
+
}
|
| 306 |
+
.fishbowl.fb-hall .sess-row:hover { background:var(--glass); }
|
| 307 |
+
.fishbowl.fb-hall .sess-row .sr-id {
|
| 308 |
+
font-family:var(--font-display); color:var(--ink-dim); font-size:12px;
|
| 309 |
+
}
|
| 310 |
+
.fishbowl.fb-hall .sess-row .sr-cast { color:var(--ink-mid); font-size:12px; word-break:break-word; }
|
| 311 |
+
.fishbowl.fb-hall .sess-row .sr-num { font-variant-numeric:tabular-nums; color:var(--ink-mid); }
|
| 312 |
+
.fishbowl.fb-hall .sr-won {
|
| 313 |
+
display:inline-flex; align-items:center; gap:5px;
|
| 314 |
+
font-family:var(--font-display); font-size:11px; font-weight:700;
|
| 315 |
+
text-transform:uppercase; letter-spacing:0.08em;
|
| 316 |
+
color:var(--gold); text-shadow:0 0 10px rgba(255,207,107,0.5);
|
| 317 |
+
}
|
| 318 |
+
.fishbowl.fb-hall .sr-reason { color:var(--ink-faint); font-size:11px; }
|
| 319 |
+
|
| 320 |
+
/* ββ Fairness footnote panel ββ */
|
| 321 |
+
.fishbowl.fb-hall .fair-grid {
|
| 322 |
+
display:grid; grid-template-columns:1.4fr 0.7fr 0.7fr 1.5fr; gap:10px;
|
| 323 |
+
align-items:center; padding:8px 12px; border-radius:var(--r);
|
| 324 |
+
}
|
| 325 |
+
.fishbowl.fb-hall .fair-head {
|
| 326 |
+
font-family:var(--font-display); font-size:8.5px; letter-spacing:0.2em;
|
| 327 |
+
text-transform:uppercase; color:var(--ink-faint);
|
| 328 |
+
border-bottom:1px solid var(--line-soft); border-radius:0;
|
| 329 |
+
}
|
| 330 |
+
.fishbowl.fb-hall .fair-grid:hover { background:var(--glass); }
|
| 331 |
+
.fishbowl.fb-hall .fair-seat { font-family:var(--font-display); font-weight:600; color:var(--ink); }
|
| 332 |
+
.fishbowl.fb-hall .fair-num { font-variant-numeric:tabular-nums; color:var(--ink-mid); }
|
| 333 |
+
.fishbowl.fb-hall .fair-bar { position:relative; height:10px; border-radius:999px;
|
| 334 |
+
background:rgba(120,222,214,0.08); overflow:hidden; }
|
| 335 |
+
.fishbowl.fb-hall .fair-bar .fair-fill { position:absolute; inset:0 auto 0 0; height:100%;
|
| 336 |
+
border-radius:999px; background:linear-gradient(90deg, var(--cyan), var(--teal));
|
| 337 |
+
box-shadow:0 0 8px rgba(79,230,210,0.4); }
|
| 338 |
+
.fishbowl.fb-hall .fair-note {
|
| 339 |
+
margin-top:14px; padding:10px 12px; border-radius:var(--r);
|
| 340 |
+
border:1px solid var(--line-soft); background:rgba(255,207,107,0.06);
|
| 341 |
+
color:var(--ink-dim); font-size:11px; line-height:1.5;
|
| 342 |
+
}
|
| 343 |
+
|
| 344 |
+
@keyframes hallRise { from { opacity:0; transform:translateY(8px);} to { opacity:1; transform:none;} }
|
| 345 |
+
@keyframes hallChampPulse {
|
| 346 |
+
0%,100% { transform:scale(1); filter:drop-shadow(0 0 6px rgba(255,207,107,0.5)); }
|
| 347 |
+
50% { transform:scale(1.12); filter:drop-shadow(0 0 14px rgba(255,207,107,0.85)); }
|
| 348 |
+
}
|
| 349 |
+
@media (prefers-reduced-motion: reduce) {
|
| 350 |
+
.fishbowl.fb-hall .hall-headline::after,
|
| 351 |
+
.fishbowl.fb-hall .podium-slot,
|
| 352 |
+
.fishbowl.fb-hall .podium-slot.rank-1 .ps-medal,
|
| 353 |
+
.fishbowl.fb-hall .ht-row { animation:none !important; }
|
| 354 |
+
}
|
| 355 |
+
</style>
|
| 356 |
+
"""
|
| 357 |
+
|
| 358 |
+
|
| 359 |
+
# ββ render functions (pure HTML strings, defensive, .fishbowl-scoped) βββββββββββββ
|
| 360 |
+
|
| 361 |
+
|
| 362 |
+
def _wrap(inner: str) -> str:
|
| 363 |
+
"""Wrap a Hall-of-Fame pane in the ``.fishbowl.fb-hall`` scope root.
|
| 364 |
+
|
| 365 |
+
Mirrors :func:`src.ui.fishbowl.app._fishbowl` so the theater stylesheet (and the
|
| 366 |
+
inlined Hall-of-Fame ``<style>``) win over Gradio's cascade.
|
| 367 |
+
"""
|
| 368 |
+
return f'<div class="fishbowl fb-hall">{_HALL_STYLE}{inner}</div>'
|
| 369 |
+
|
| 370 |
+
|
| 371 |
+
def render_headline(entries) -> str:
|
| 372 |
+
"""The big glowing demo line from :func:`leaderboard.headline`, or an empty state.
|
| 373 |
+
|
| 374 |
+
Renders e.g. ``MiniCPM-8B beats Gemma-12B Β· 7-3 at Debate Duel`` with the two model
|
| 375 |
+
names gilded. When ``headline`` returns ``None`` (no symmetric scenario with two
|
| 376 |
+
models that have each won), shows a cheerful "no champions yet" call to action.
|
| 377 |
+
"""
|
| 378 |
+
try:
|
| 379 |
+
line = headline(entries)
|
| 380 |
+
except Exception: # pragma: no cover - projection error (defensive)
|
| 381 |
+
line = None
|
| 382 |
+
if not line:
|
| 383 |
+
return _wrap(
|
| 384 |
+
'<div class="hall-empty">'
|
| 385 |
+
'<div class="eyebrow">🏆 Hall of Fame</div>'
|
| 386 |
+
'<div class="he-body">No champions crowned yet β run a competitive scenario '
|
| 387 |
+
"(a versus duel or a judged contest) to fill the Hall.</div>"
|
| 388 |
+
"</div>"
|
| 389 |
+
)
|
| 390 |
+
# Gild the two model names: the line is "<A> beats <B> Β· X-Y at <Scenario>".
|
| 391 |
+
safe = _esc(line)
|
| 392 |
+
if " beats " in line:
|
| 393 |
+
left, _, rest = line.partition(" beats ")
|
| 394 |
+
runner, _, tail = rest.partition(" Β· ")
|
| 395 |
+
safe = (
|
| 396 |
+
f'<span class="hl-gold">{_esc(left)}</span> beats '
|
| 397 |
+
f'<span class="hl-gold">{_esc(runner)}</span> Β· {_esc(tail)}'
|
| 398 |
+
)
|
| 399 |
+
return _wrap(
|
| 400 |
+
'<div class="hall-headline">'
|
| 401 |
+
'<div class="eyebrow">🏆 Today\'s Champion</div>'
|
| 402 |
+
f'<div class="hall-line">{safe}</div>'
|
| 403 |
+
"</div>"
|
| 404 |
+
)
|
| 405 |
+
|
| 406 |
+
|
| 407 |
+
def _podium_html(rows: list[ModelRow]) -> str:
|
| 408 |
+
"""The gold/silver/bronze top-3 podium (visual order: 2nd Β· 1st Β· 3rd)."""
|
| 409 |
+
if not rows:
|
| 410 |
+
return ""
|
| 411 |
+
top = rows[:3]
|
| 412 |
+
medals = {0: "🥇", 1: "🥈", 2: "🥉"} # π₯ π₯ π₯
|
| 413 |
+
|
| 414 |
+
def slot(idx: int) -> str:
|
| 415 |
+
if idx >= len(top):
|
| 416 |
+
return '<div class="podium-slot" style="visibility:hidden"></div>'
|
| 417 |
+
row = top[idx]
|
| 418 |
+
return (
|
| 419 |
+
f'<div class="podium-slot rank-{idx + 1}">'
|
| 420 |
+
f'<div class="ps-medal">{medals.get(idx, "")}</div>'
|
| 421 |
+
f'<div class="ps-name">{_esc(_short_model(row.model))}</div>'
|
| 422 |
+
f'<div class="ps-rate">{_pct(row.win_rate)}</div>'
|
| 423 |
+
f'<div class="ps-sub">{row.wins}/{row.plays} won</div>'
|
| 424 |
+
"</div>"
|
| 425 |
+
)
|
| 426 |
+
|
| 427 |
+
# Visual order puts #1 in the centre, raised.
|
| 428 |
+
order = [1, 0, 2]
|
| 429 |
+
return '<div class="podium">' + "".join(slot(i) for i in order) + "</div>"
|
| 430 |
+
|
| 431 |
+
|
| 432 |
+
def render_model_board(entries) -> str:
|
| 433 |
+
"""The model podium + ranked table with glowing gold win-rate bars.
|
| 434 |
+
|
| 435 |
+
Folds :func:`leaderboard.model_table` (already sorted by win-rate) into the podium
|
| 436 |
+
for the top 3 and a ranked table below. Each row carries rank, model, plays, wins
|
| 437 |
+
and a horizontal bar whose width is the win-rate. Empty store β empty state.
|
| 438 |
+
"""
|
| 439 |
+
try:
|
| 440 |
+
rows = model_table(entries)
|
| 441 |
+
except Exception: # pragma: no cover - projection error (defensive)
|
| 442 |
+
rows = []
|
| 443 |
+
if not rows:
|
| 444 |
+
return _wrap(
|
| 445 |
+
'<div class="hall-empty">'
|
| 446 |
+
'<div class="eyebrow">📊 Model Leaderboard</div>'
|
| 447 |
+
'<div class="he-body">No decided competitive runs yet. Once a versus or judged '
|
| 448 |
+
"scenario crowns a winner, the models climb the board here.</div>"
|
| 449 |
+
"</div>"
|
| 450 |
+
)
|
| 451 |
+
head = '<div class="ht-head"><div>#</div><div>Model</div><div>Plays</div><div>Wins</div><div>Win rate</div></div>'
|
| 452 |
+
body_rows: list[str] = []
|
| 453 |
+
for idx, row in enumerate(rows):
|
| 454 |
+
champ = " is-champ" if idx == 0 else ""
|
| 455 |
+
width = _bar_width(row.win_rate)
|
| 456 |
+
body_rows.append(
|
| 457 |
+
f'<div class="ht-row{champ}" style="animation-delay:{min(idx, 12) * 0.04:.2f}s">'
|
| 458 |
+
f'<div class="ht-rank">{idx + 1}</div>'
|
| 459 |
+
f'<div class="ht-model">{_esc(_short_model(row.model))}</div>'
|
| 460 |
+
f'<div class="ht-num">{row.plays}</div>'
|
| 461 |
+
f'<div class="ht-num">{row.wins}</div>'
|
| 462 |
+
'<div class="ht-bar">'
|
| 463 |
+
f'<div class="ht-fill" style="width:{width:.1f}%"></div>'
|
| 464 |
+
f'<div class="ht-bar-label">{_pct(row.win_rate)}</div>'
|
| 465 |
+
"</div>"
|
| 466 |
+
"</div>"
|
| 467 |
+
)
|
| 468 |
+
return _wrap(
|
| 469 |
+
'<div class="hall-panel">'
|
| 470 |
+
'<div class="eyebrow">📊 Model Leaderboard</div>'
|
| 471 |
+
+ _podium_html(rows)
|
| 472 |
+
+ '<div class="hall-table">'
|
| 473 |
+
+ head
|
| 474 |
+
+ "".join(body_rows)
|
| 475 |
+
+ "</div></div>"
|
| 476 |
+
)
|
| 477 |
+
|
| 478 |
+
|
| 479 |
+
def render_sessions(entries, scenario_name: str) -> str:
|
| 480 |
+
"""The sessions table for *scenario_name* (without the Replay buttons).
|
| 481 |
+
|
| 482 |
+
Lists each finished, won, competitive run newest-first: a short id, the cast's
|
| 483 |
+
models, the gold WON badge, why it ended, turns/tokens and the date. The Replay
|
| 484 |
+
buttons themselves are real Gradio components rendered alongside this HTML by
|
| 485 |
+
``build_hall_of_fame`` (gr.HTML can't host working buttons); this pane is the
|
| 486 |
+
glanceable record. Empty / non-competitive scenario β empty state.
|
| 487 |
+
"""
|
| 488 |
+
if not scenario_name:
|
| 489 |
+
return _wrap(
|
| 490 |
+
'<div class="hall-empty">'
|
| 491 |
+
'<div class="eyebrow">▶ Sessions</div>'
|
| 492 |
+
'<div class="he-body">Pick a competitive world above to see its decided '
|
| 493 |
+
"matches and replay any of them.</div>"
|
| 494 |
+
"</div>"
|
| 495 |
+
)
|
| 496 |
+
try:
|
| 497 |
+
rows = scenario_sessions(entries, scenario_name)
|
| 498 |
+
except Exception: # pragma: no cover - projection error (defensive)
|
| 499 |
+
rows = []
|
| 500 |
+
title = _esc(_scenario_title(scenario_name))
|
| 501 |
+
if not rows:
|
| 502 |
+
return _wrap(
|
| 503 |
+
'<div class="hall-panel">'
|
| 504 |
+
f'<div class="eyebrow">▶ {title} Β· Sessions</div>'
|
| 505 |
+
'<div class="hall-empty" style="border:none;background:transparent;padding:8px 0">'
|
| 506 |
+
'<div class="he-body">No decided matches in this world yet. Run it as a '
|
| 507 |
+
"competition and the results land here, replayable.</div>"
|
| 508 |
+
"</div></div>"
|
| 509 |
+
)
|
| 510 |
+
head = (
|
| 511 |
+
'<div class="sess-head">'
|
| 512 |
+
"<div>Run</div><div>Cast models</div><div>Winner</div>"
|
| 513 |
+
"<div>Why</div><div>Turns</div><div>When</div>"
|
| 514 |
+
"</div>"
|
| 515 |
+
)
|
| 516 |
+
body_rows = "".join(_session_row_html(row) for row in rows)
|
| 517 |
+
return _wrap(
|
| 518 |
+
'<div class="hall-panel">'
|
| 519 |
+
f'<div class="eyebrow">▶ {title} Β· Sessions</div>'
|
| 520 |
+
f'<div class="hall-table">{head}{body_rows}</div>'
|
| 521 |
+
"</div>"
|
| 522 |
+
)
|
| 523 |
+
|
| 524 |
+
|
| 525 |
+
def _session_row_html(row: LeaderboardEntry) -> str:
|
| 526 |
+
"""One sessions-table row (glanceable; the Replay button sits beside it)."""
|
| 527 |
+
reason = _REASON_LABEL.get(row.reason or "", row.reason or "β")
|
| 528 |
+
won = (
|
| 529 |
+
f'<span class="sr-won">🏆 {_esc(row.winner)}</span>'
|
| 530 |
+
if row.winner
|
| 531 |
+
else '<span class="sr-reason">β</span>'
|
| 532 |
+
)
|
| 533 |
+
return (
|
| 534 |
+
'<div class="sess-row">'
|
| 535 |
+
f'<div class="sr-id">{_esc(_short_id(row.run_id))}</div>'
|
| 536 |
+
f'<div class="sr-cast">{_esc(_cast_models(row))}</div>'
|
| 537 |
+
f"<div>{won}</div>"
|
| 538 |
+
f'<div class="sr-reason">{_esc(reason)}</div>'
|
| 539 |
+
f'<div class="sr-num">{row.turns}t Β· {_esc(_fmt_tokens(row.tokens))}</div>'
|
| 540 |
+
f'<div class="sr-num">{_esc(_fmt_when(row.finished_at))}</div>'
|
| 541 |
+
"</div>"
|
| 542 |
+
)
|
| 543 |
+
|
| 544 |
+
|
| 545 |
+
def render_fairness(entries, scenario_name: str) -> str:
|
| 546 |
+
"""The 6.3 fairness footnote: win rate per seat type, with the asymmetry note.
|
| 547 |
+
|
| 548 |
+
Folds :func:`leaderboard.fairness_table` (only *declared* seats β teams /
|
| 549 |
+
symmetric seats; judges and other unmapped seats are excluded by design). A
|
| 550 |
+
footnote reminds viewers that asymmetric seats (spy vs herd) skew raw win rates.
|
| 551 |
+
Empty β a quiet empty state.
|
| 552 |
+
"""
|
| 553 |
+
if not scenario_name:
|
| 554 |
+
return ""
|
| 555 |
+
try:
|
| 556 |
+
rows = fairness_table(entries, scenario_name)
|
| 557 |
+
except Exception: # pragma: no cover - projection error (defensive)
|
| 558 |
+
rows = []
|
| 559 |
+
if not rows:
|
| 560 |
+
return _wrap(
|
| 561 |
+
'<div class="hall-panel">'
|
| 562 |
+
'<div class="eyebrow">⚖️ Seat Fairness</div>'
|
| 563 |
+
'<div class="he-body" style="color:var(--ink-mid)">No per-seat data yet β '
|
| 564 |
+
"this footnote fills in once this world has decided competitive runs.</div>"
|
| 565 |
+
"</div>"
|
| 566 |
+
)
|
| 567 |
+
head = (
|
| 568 |
+
'<div class="fair-grid fair-head"><div>Seat type</div><div>Plays</div><div>Wins</div><div>Win rate</div></div>'
|
| 569 |
+
)
|
| 570 |
+
body = "".join(_fairness_row_html(row) for row in rows)
|
| 571 |
+
note = (
|
| 572 |
+
'<div class="fair-note">⚠️ Seats are not always symmetric β a spy '
|
| 573 |
+
"faces a whole herd, a lone debater a panel. Raw seat win rates show the structural "
|
| 574 |
+
"tilt of the game, not just who played well.</div>"
|
| 575 |
+
)
|
| 576 |
+
return _wrap(
|
| 577 |
+
'<div class="hall-panel">'
|
| 578 |
+
'<div class="eyebrow">⚖️ Seat Fairness</div>'
|
| 579 |
+
f'<div class="hall-table">{head}{body}</div>'
|
| 580 |
+
f"{note}</div>"
|
| 581 |
+
)
|
| 582 |
+
|
| 583 |
+
|
| 584 |
+
def _fairness_row_html(row: SeatRow) -> str:
|
| 585 |
+
"""One fairness row: seat type, plays, wins, and a cyan win-rate bar."""
|
| 586 |
+
width = _bar_width(row.win_rate)
|
| 587 |
+
return (
|
| 588 |
+
'<div class="fair-grid">'
|
| 589 |
+
f'<div class="fair-seat">{_esc(row.seat_type)}</div>'
|
| 590 |
+
f'<div class="fair-num">{row.plays}</div>'
|
| 591 |
+
f'<div class="fair-num">{row.wins}</div>'
|
| 592 |
+
'<div class="fair-bar">'
|
| 593 |
+
f'<div class="fair-fill" style="width:{width:.1f}%"></div>'
|
| 594 |
+
"</div>"
|
| 595 |
+
"</div>"
|
| 596 |
+
)
|
| 597 |
+
|
| 598 |
+
|
| 599 |
+
def _scenario_title(scenario_name: str) -> str:
|
| 600 |
+
"""The display title for an internal scenario name, falling back to the name."""
|
| 601 |
+
try:
|
| 602 |
+
cfg = default_registry().scenarios.get(scenario_name)
|
| 603 |
+
return (getattr(cfg, "title", "") or scenario_name) if cfg else scenario_name
|
| 604 |
+
except Exception: # pragma: no cover - registry unavailable (defensive)
|
| 605 |
+
return scenario_name
|
| 606 |
+
|
| 607 |
+
|
| 608 |
+
# ββ builder ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
|
| 609 |
+
|
| 610 |
+
|
| 611 |
+
def build_hall_of_fame() -> dict:
|
| 612 |
+
"""Lay out the Hall of Fame tab and return a handles dict for the app to wire.
|
| 613 |
+
|
| 614 |
+
Layout (top β bottom): the glowing **headline marquee**, a **scenario picker**
|
| 615 |
+
(only competitive worlds), the **model podium + ranked table**, the per-scenario
|
| 616 |
+
**sessions area** (an HTML record + a ``gr.render`` of per-run Replay buttons),
|
| 617 |
+
the **fairness footnote**, and a **refresh** control.
|
| 618 |
+
|
| 619 |
+
Returns a handles dict the app wires into the Show transport:
|
| 620 |
+
|
| 621 |
+
``{"scenario_dd", "refresh", "headline_html", "model_html", "sessions_html",
|
| 622 |
+
"fairness_html", "sessions_render_anchor", "default_scenario"}``
|
| 623 |
+
|
| 624 |
+
The Replay buttons are *not* built here β they need the Show's pane handles, which
|
| 625 |
+
only exist after the Show tab builds. The app calls :func:`wire_sessions_render`
|
| 626 |
+
(below) with the shared ``archive_refs`` + states after all tabs build, mirroring
|
| 627 |
+
how ``_build_archive_drawer`` defers its ``show_handles`` lookup.
|
| 628 |
+
"""
|
| 629 |
+
scenarios = competitive_scenarios()
|
| 630 |
+
choices = [(title, name) for title, name in scenarios]
|
| 631 |
+
default_scenario = choices[0][1] if choices else None
|
| 632 |
+
entries = _entries()
|
| 633 |
+
|
| 634 |
+
handles: dict = {"default_scenario": default_scenario}
|
| 635 |
+
|
| 636 |
+
with gr.Column(elem_classes=["hall-wrap"]):
|
| 637 |
+
handles["headline_html"] = gr.HTML(render_headline(entries))
|
| 638 |
+
|
| 639 |
+
handles["scenario_dd"] = gr.Dropdown(
|
| 640 |
+
choices=choices,
|
| 641 |
+
value=default_scenario,
|
| 642 |
+
label="World",
|
| 643 |
+
elem_classes=["hall-scenario"],
|
| 644 |
+
interactive=True,
|
| 645 |
+
visible=bool(choices),
|
| 646 |
+
)
|
| 647 |
+
handles["refresh"] = gr.Button("β³ refresh", size="sm", scale=0)
|
| 648 |
+
|
| 649 |
+
handles["model_html"] = gr.HTML(render_model_board(entries))
|
| 650 |
+
|
| 651 |
+
handles["sessions_html"] = gr.HTML(render_sessions(entries, default_scenario))
|
| 652 |
+
# The per-run Replay buttons live in this column; the app fills it with a
|
| 653 |
+
# gr.render (it needs the Show panes, populated after all tabs build).
|
| 654 |
+
handles["sessions_render_anchor"] = gr.Column(elem_classes=["hall-replays"])
|
| 655 |
+
|
| 656 |
+
handles["fairness_html"] = gr.HTML(render_fairness(entries, default_scenario))
|
| 657 |
+
|
| 658 |
+
return handles
|
| 659 |
+
|
| 660 |
+
|
| 661 |
+
def wire_sessions_render(handles: dict, *, refs: dict, tabs, states: dict) -> None:
|
| 662 |
+
"""Wire the Hall's data refresh + the per-run Replay buttons into the Show.
|
| 663 |
+
|
| 664 |
+
Called by the app **after all tabs build** so ``refs["show_handles"]`` is live
|
| 665 |
+
(mirrors ``_build_archive_drawer``'s deferred-ref trick). Three behaviours:
|
| 666 |
+
|
| 667 |
+
* Changing the scenario picker (or refresh) re-reads the ledger and repaints the
|
| 668 |
+
model board, sessions table and fairness panel.
|
| 669 |
+
* A ``gr.render`` keyed on (scenario, refresh) lists per-run **Replay** buttons.
|
| 670 |
+
* Each Replay click loads the run via :func:`load_replay` and pushes it into the
|
| 671 |
+
Show β the *exact* output contract from ``_build_archive_drawer`` (session, k,
|
| 672 |
+
scenario, switch Tabs to "show", repaint panes, stopped=False, ticks=0).
|
| 673 |
+
|
| 674 |
+
Defensive throughout: missing handles or an unavailable store degrade to no-ops /
|
| 675 |
+
empty states rather than raising.
|
| 676 |
+
"""
|
| 677 |
+
if not handles:
|
| 678 |
+
return
|
| 679 |
+
# Late imports of the app's render/replay helpers β they live in app.py and importing
|
| 680 |
+
# them at module top would create an import cycle (app imports this module).
|
| 681 |
+
try:
|
| 682 |
+
from src.ui.fishbowl.app import (
|
| 683 |
+
_pad_values,
|
| 684 |
+
_registry,
|
| 685 |
+
_render_at,
|
| 686 |
+
_show_outs,
|
| 687 |
+
_title_for,
|
| 688 |
+
_tools,
|
| 689 |
+
load_replay,
|
| 690 |
+
)
|
| 691 |
+
except Exception: # pragma: no cover - app helpers unavailable (defensive)
|
| 692 |
+
return
|
| 693 |
+
|
| 694 |
+
scenario_dd = handles.get("scenario_dd")
|
| 695 |
+
refresh = handles.get("refresh")
|
| 696 |
+
model_html = handles.get("model_html")
|
| 697 |
+
sessions_html = handles.get("sessions_html")
|
| 698 |
+
fairness_html = handles.get("fairness_html")
|
| 699 |
+
anchor = handles.get("sessions_render_anchor")
|
| 700 |
+
if scenario_dd is None:
|
| 701 |
+
return
|
| 702 |
+
|
| 703 |
+
# ββ refresh the boards on scenario change / refresh click ββββββββββββββββββββββ
|
| 704 |
+
def _repaint(scenario_name):
|
| 705 |
+
entries = _entries()
|
| 706 |
+
return (
|
| 707 |
+
render_model_board(entries),
|
| 708 |
+
render_sessions(entries, scenario_name),
|
| 709 |
+
render_fairness(entries, scenario_name),
|
| 710 |
+
)
|
| 711 |
+
|
| 712 |
+
repaint_outputs = [c for c in (model_html, sessions_html, fairness_html) if c is not None]
|
| 713 |
+
if repaint_outputs:
|
| 714 |
+
# Always emit a full tuple, but only route to present panes (defensive padding).
|
| 715 |
+
def _repaint_present(scenario_name):
|
| 716 |
+
full = _repaint(scenario_name)
|
| 717 |
+
picked = []
|
| 718 |
+
for comp, value in zip((model_html, sessions_html, fairness_html), full):
|
| 719 |
+
if comp is not None:
|
| 720 |
+
picked.append(value)
|
| 721 |
+
return tuple(picked) if len(picked) != 1 else picked[0]
|
| 722 |
+
|
| 723 |
+
scenario_dd.change(_repaint_present, inputs=[scenario_dd], outputs=repaint_outputs)
|
| 724 |
+
if refresh is not None:
|
| 725 |
+
refresh.click(_repaint_present, inputs=[scenario_dd], outputs=repaint_outputs)
|
| 726 |
+
|
| 727 |
+
# ββ per-run Replay buttons (gr.render β Show transport) ββββββββββββββββββββββββ
|
| 728 |
+
if anchor is None:
|
| 729 |
+
return
|
| 730 |
+
|
| 731 |
+
with anchor:
|
| 732 |
+
|
| 733 |
+
@gr.render(
|
| 734 |
+
inputs=[scenario_dd],
|
| 735 |
+
triggers=[scenario_dd.change] + ([refresh.click] if refresh is not None else []),
|
| 736 |
+
)
|
| 737 |
+
def _render_replays(scenario_name):
|
| 738 |
+
if not scenario_name:
|
| 739 |
+
return
|
| 740 |
+
entries = _entries()
|
| 741 |
+
try:
|
| 742 |
+
rows = scenario_sessions(entries, scenario_name)
|
| 743 |
+
except Exception: # pragma: no cover - projection error (defensive)
|
| 744 |
+
rows = []
|
| 745 |
+
if not rows:
|
| 746 |
+
return
|
| 747 |
+
|
| 748 |
+
show_outs = _show_outs(refs.get("show_handles") or {})
|
| 749 |
+
n_out = 6 + len(show_outs) # session, k, scenario, tabs, *panes, stopped, ticks
|
| 750 |
+
|
| 751 |
+
def _loader(run_id: str):
|
| 752 |
+
def _load(layout, mind_reader):
|
| 753 |
+
session = load_replay(run_id, registry=_registry, tools=_tools)
|
| 754 |
+
if session is None:
|
| 755 |
+
return tuple(gr.update() for _ in range(n_out))
|
| 756 |
+
k = session.head # land on the full discussion; βΆ replays it
|
| 757 |
+
out = _render_at(session, k, layout=layout, mind_reader=mind_reader)
|
| 758 |
+
return (
|
| 759 |
+
session,
|
| 760 |
+
k,
|
| 761 |
+
_title_for(session.scenario_name),
|
| 762 |
+
gr.update(selected="show"),
|
| 763 |
+
*_pad_values(out, show_outs),
|
| 764 |
+
False,
|
| 765 |
+
0,
|
| 766 |
+
)
|
| 767 |
+
|
| 768 |
+
return _load
|
| 769 |
+
|
| 770 |
+
for row in rows:
|
| 771 |
+
label = f"βΆ Replay {_short_id(row.run_id)}"
|
| 772 |
+
if row.winner:
|
| 773 |
+
label += f" Β· WON {row.winner}"
|
| 774 |
+
btn = gr.Button(label, elem_classes=["archive-card"], size="sm")
|
| 775 |
+
btn.click(
|
| 776 |
+
_loader(row.run_id),
|
| 777 |
+
inputs=[states["layout"], states["mind"]],
|
| 778 |
+
outputs=[
|
| 779 |
+
states["session"],
|
| 780 |
+
states["k"],
|
| 781 |
+
states["scenario"],
|
| 782 |
+
tabs,
|
| 783 |
+
*show_outs,
|
| 784 |
+
states["stopped"],
|
| 785 |
+
states["ticks"],
|
| 786 |
+
],
|
| 787 |
+
)
|
| 788 |
+
|
| 789 |
+
|
| 790 |
+
__all__ = [
|
| 791 |
+
"build_hall_of_fame",
|
| 792 |
+
"competitive_scenarios",
|
| 793 |
+
"render_fairness",
|
| 794 |
+
"render_headline",
|
| 795 |
+
"render_model_board",
|
| 796 |
+
"render_sessions",
|
| 797 |
+
"wire_sessions_render",
|
| 798 |
+
]
|
src/ui/fishbowl/session.py
CHANGED
|
@@ -15,8 +15,10 @@ from __future__ import annotations
|
|
| 15 |
from src import observability as obs
|
| 16 |
from src.core.conductor import Conductor
|
| 17 |
from src.core.ledger_factory import make_ledger
|
|
|
|
| 18 |
from src.core.manifest import AgentManifest
|
| 19 |
from src.core.registry import Registry, default_registry
|
|
|
|
| 20 |
from src.tools.builtins import default_tool_registry
|
| 21 |
from src.ui.fishbowl.view_model import view_model_at
|
| 22 |
|
|
@@ -50,6 +52,14 @@ class FishbowlSession:
|
|
| 50 |
governor=self._registry.governor_for(scenario_name),
|
| 51 |
ledger=make_ledger(),
|
| 52 |
)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 53 |
obs.log(
|
| 54 |
"session.created",
|
| 55 |
scenario=scenario_name,
|
|
@@ -182,6 +192,34 @@ class FishbowlSession:
|
|
| 182 |
winning_model=winning_model,
|
| 183 |
winning_models=winning_models,
|
| 184 |
)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 185 |
|
| 186 |
@property
|
| 187 |
def cast(self) -> list[AgentManifest]:
|
|
|
|
| 15 |
from src import observability as obs
|
| 16 |
from src.core.conductor import Conductor
|
| 17 |
from src.core.ledger_factory import make_ledger
|
| 18 |
+
from src.core.leaderboard_store import build_entry, make_leaderboard_store
|
| 19 |
from src.core.manifest import AgentManifest
|
| 20 |
from src.core.registry import Registry, default_registry
|
| 21 |
+
from src.core.run_index import index_runs
|
| 22 |
from src.tools.builtins import default_tool_registry
|
| 23 |
from src.ui.fishbowl.view_model import view_model_at
|
| 24 |
|
|
|
|
| 52 |
governor=self._registry.governor_for(scenario_name),
|
| 53 |
ledger=make_ledger(),
|
| 54 |
)
|
| 55 |
+
# The leaderboard's dedicated scoreboard table (ADR-0035) β a *separate* table in
|
| 56 |
+
# the same database as the event ledger, never the events log. Built defensively
|
| 57 |
+
# so a store/config hiccup can never break a live run; a None store just means no
|
| 58 |
+
# scoreboard row is recorded.
|
| 59 |
+
try:
|
| 60 |
+
self._leaderboard = make_leaderboard_store()
|
| 61 |
+
except Exception: # pragma: no cover - no store configured (defensive)
|
| 62 |
+
self._leaderboard = None
|
| 63 |
obs.log(
|
| 64 |
"session.created",
|
| 65 |
scenario=scenario_name,
|
|
|
|
| 192 |
winning_model=winning_model,
|
| 193 |
winning_models=winning_models,
|
| 194 |
)
|
| 195 |
+
self._record_leaderboard()
|
| 196 |
+
|
| 197 |
+
def _record_leaderboard(self) -> None:
|
| 198 |
+
"""Write this run's scoreboard row to the dedicated leaderboard table (ADR-0035).
|
| 199 |
+
|
| 200 |
+
Detached from the event ledger: the run's events stay the source of truth for the
|
| 201 |
+
trace, while this persists one denormalised result row to ``leaderboard_entries``
|
| 202 |
+
β but only when :func:`build_entry` deems the run eligible (finished, a winner, a
|
| 203 |
+
concrete winning model, and a competitive scenario). Idempotent via the store's
|
| 204 |
+
upsert-on-``run_id``, so a verdict that supersedes a budget close replaces the row.
|
| 205 |
+
|
| 206 |
+
Fully defensive: any failure is logged and swallowed so a leaderboard hiccup never
|
| 207 |
+
breaks the show."""
|
| 208 |
+
store = getattr(self, "_leaderboard", None)
|
| 209 |
+
if store is None:
|
| 210 |
+
return
|
| 211 |
+
try:
|
| 212 |
+
run_events = self.conductor.ledger.events_for_run(self.conductor.run_id)
|
| 213 |
+
summary = next((s for s in index_runs(run_events) if s.run_id == self.conductor.run_id), None)
|
| 214 |
+
if summary is None:
|
| 215 |
+
return
|
| 216 |
+
scenario = self._registry.scenarios.get(self._scenario_name)
|
| 217 |
+
entry = build_entry(summary, getattr(scenario, "competition", None))
|
| 218 |
+
if entry is not None:
|
| 219 |
+
store.record(entry)
|
| 220 |
+
obs.log("leaderboard.recorded", run_id=entry.run_id, scenario=entry.scenario, winner=entry.winner)
|
| 221 |
+
except Exception: # pragma: no cover - never let a scoreboard write break a run
|
| 222 |
+
obs.log("leaderboard.record_failed", level="warning", run_id=self.conductor.run_id)
|
| 223 |
|
| 224 |
@property
|
| 225 |
def cast(self) -> list[AgentManifest]:
|
tests/test_leaderboard.py
ADDED
|
@@ -0,0 +1,977 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Leaderboard aggregations over LeaderboardEntry rows β zero-mock, deterministic.
|
| 2 |
+
|
| 3 |
+
This suite verifies the leaderboard's read-model invariants: that aggregations over a
|
| 4 |
+
list of LeaderboardEntry objects produce correct tables, win_rate math, sorting, and that
|
| 5 |
+
projections are deterministic regardless of input order.
|
| 6 |
+
|
| 7 |
+
Test strategy: Build LeaderboardEntry objects directly (no events, no ledger), call the
|
| 8 |
+
public aggregation functions, and assert on the resulting rows. This guards against:
|
| 9 |
+
|
| 10 |
+
- Incorrect model endpoint deduplication (one play per endpoint per run, even when
|
| 11 |
+
filling multiple seats).
|
| 12 |
+
- Team wins crediting every member's model vs single-agent wins.
|
| 13 |
+
- Judges and other unmapped cast members affecting fairness and agent tables.
|
| 14 |
+
- Sorting regressions (determinism is a contract).
|
| 15 |
+
- Headline generation requiring symmetric seats + β₯2 models with β₯1 win each.
|
| 16 |
+
- Edge cases: empty input, zero plays, missing fields, out-of-order entries.
|
| 17 |
+
"""
|
| 18 |
+
|
| 19 |
+
from __future__ import annotations
|
| 20 |
+
|
| 21 |
+
from datetime import datetime, timedelta, timezone
|
| 22 |
+
|
| 23 |
+
from src.core.leaderboard import (
|
| 24 |
+
agent_table,
|
| 25 |
+
fairness_table,
|
| 26 |
+
headline,
|
| 27 |
+
model_table,
|
| 28 |
+
scenario_sessions,
|
| 29 |
+
)
|
| 30 |
+
from src.core.leaderboard_store import LeaderboardEntry
|
| 31 |
+
from src.core.run_index import CastBinding
|
| 32 |
+
|
| 33 |
+
|
| 34 |
+
# ββ LeaderboardEntry builders ββββββββββββββββββββββββββββββββββββββββββββββββββββββ
|
| 35 |
+
|
| 36 |
+
|
| 37 |
+
def _entry(
|
| 38 |
+
run_id: str = "r1",
|
| 39 |
+
scenario: str = "Debate Duel",
|
| 40 |
+
seed: str = "seed123",
|
| 41 |
+
session_id: str | None = None,
|
| 42 |
+
competition_kind: str = "versus",
|
| 43 |
+
teams: dict[str, list[str]] | None = None,
|
| 44 |
+
symmetric_seats: list[str] | None = None,
|
| 45 |
+
cast: dict[str, CastBinding] | None = None,
|
| 46 |
+
winner: str | None = "alice",
|
| 47 |
+
winner_kind: str | None = "agent",
|
| 48 |
+
winning_model: str | None = "openai/openbmb/MiniCPM-8B",
|
| 49 |
+
winning_models: list[str] | None = None,
|
| 50 |
+
reason: str | None = "verdict",
|
| 51 |
+
turns: int = 5,
|
| 52 |
+
tokens: int = 200,
|
| 53 |
+
started_at: datetime | None = None,
|
| 54 |
+
finished_at: datetime | None = None,
|
| 55 |
+
) -> LeaderboardEntry:
|
| 56 |
+
"""Build a minimal LeaderboardEntry for testing."""
|
| 57 |
+
if started_at is None:
|
| 58 |
+
started_at = datetime(2025, 6, 14, 10, 0, 0, tzinfo=timezone.utc)
|
| 59 |
+
if finished_at is None:
|
| 60 |
+
finished_at = started_at + timedelta(minutes=5)
|
| 61 |
+
if cast is None:
|
| 62 |
+
cast = {"alice": CastBinding(model_endpoint=winning_model)}
|
| 63 |
+
if winning_models is None:
|
| 64 |
+
winning_models = [winning_model] if winning_model else []
|
| 65 |
+
return LeaderboardEntry(
|
| 66 |
+
run_id=run_id,
|
| 67 |
+
session_id=session_id,
|
| 68 |
+
scenario=scenario,
|
| 69 |
+
seed=seed,
|
| 70 |
+
competition_kind=competition_kind,
|
| 71 |
+
teams=teams,
|
| 72 |
+
symmetric_seats=symmetric_seats,
|
| 73 |
+
cast=cast,
|
| 74 |
+
winner=winner,
|
| 75 |
+
winner_kind=winner_kind,
|
| 76 |
+
winning_model=winning_model,
|
| 77 |
+
winning_models=winning_models,
|
| 78 |
+
reason=reason,
|
| 79 |
+
turns=turns,
|
| 80 |
+
tokens=tokens,
|
| 81 |
+
started_at=started_at,
|
| 82 |
+
finished_at=finished_at,
|
| 83 |
+
)
|
| 84 |
+
|
| 85 |
+
|
| 86 |
+
# ββ Tests: scenario_sessions (newest-first filtering) βββββββββββββββββββββββββββββ
|
| 87 |
+
|
| 88 |
+
|
| 89 |
+
class TestScenarioSessions:
|
| 90 |
+
"""Verify scenario_sessions filters, orders, and projects correctly."""
|
| 91 |
+
|
| 92 |
+
def test_empty_entries_returns_empty_list(self):
|
| 93 |
+
"""Empty entry list returns no sessions."""
|
| 94 |
+
result = scenario_sessions([], "Debate Duel")
|
| 95 |
+
assert result == []
|
| 96 |
+
|
| 97 |
+
def test_no_winner_entry_excluded(self):
|
| 98 |
+
"""An entry with no winner is dropped (defensive gate)."""
|
| 99 |
+
entry = _entry(winner=None)
|
| 100 |
+
result = scenario_sessions([entry], "Debate Duel")
|
| 101 |
+
assert result == []
|
| 102 |
+
|
| 103 |
+
def test_scenario_filter_excludes_other_scenarios(self):
|
| 104 |
+
"""Sessions from other scenarios are not returned."""
|
| 105 |
+
entry = _entry(scenario="Debate Duel")
|
| 106 |
+
result = scenario_sessions([entry], "Trivia Night")
|
| 107 |
+
assert result == []
|
| 108 |
+
|
| 109 |
+
def test_one_entry_has_all_fields(self):
|
| 110 |
+
"""A single entry is projected with all fields intact."""
|
| 111 |
+
cast = {
|
| 112 |
+
"alice": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B", model_profile="large"),
|
| 113 |
+
"bob": CastBinding(model_endpoint="google/gemma-12B", model_profile="medium"),
|
| 114 |
+
}
|
| 115 |
+
started_at = datetime(2025, 6, 14, 10, 0, 0, tzinfo=timezone.utc)
|
| 116 |
+
finished_at = started_at + timedelta(minutes=5)
|
| 117 |
+
entry = _entry(
|
| 118 |
+
run_id="r1",
|
| 119 |
+
scenario="Debate Duel",
|
| 120 |
+
seed="abc123",
|
| 121 |
+
cast=cast,
|
| 122 |
+
symmetric_seats=["debater_a", "debater_b"],
|
| 123 |
+
winner="alice",
|
| 124 |
+
winner_kind="agent",
|
| 125 |
+
winning_model="openai/openbmb/MiniCPM-8B",
|
| 126 |
+
winning_models=["openai/openbmb/MiniCPM-8B"],
|
| 127 |
+
turns=7,
|
| 128 |
+
tokens=320,
|
| 129 |
+
started_at=started_at,
|
| 130 |
+
finished_at=finished_at,
|
| 131 |
+
)
|
| 132 |
+
result = scenario_sessions([entry], "Debate Duel")
|
| 133 |
+
assert len(result) == 1
|
| 134 |
+
row = result[0]
|
| 135 |
+
assert row.run_id == "r1"
|
| 136 |
+
assert row.scenario == "Debate Duel"
|
| 137 |
+
assert row.seed == "abc123"
|
| 138 |
+
assert row.winner == "alice"
|
| 139 |
+
assert row.winner_kind == "agent"
|
| 140 |
+
assert row.turns == 7
|
| 141 |
+
assert row.tokens == 320
|
| 142 |
+
assert row.started_at == started_at
|
| 143 |
+
assert row.finished_at == finished_at
|
| 144 |
+
assert "alice" in row.cast
|
| 145 |
+
assert row.cast["alice"].model_endpoint == "openai/openbmb/MiniCPM-8B"
|
| 146 |
+
|
| 147 |
+
def test_newest_first_order(self):
|
| 148 |
+
"""Sessions are sorted newest first by finished_at."""
|
| 149 |
+
old = datetime(2025, 6, 1, 10, 0, 0, tzinfo=timezone.utc)
|
| 150 |
+
new = datetime(2025, 6, 14, 10, 0, 0, tzinfo=timezone.utc)
|
| 151 |
+
entries = [
|
| 152 |
+
_entry(run_id="r1", finished_at=old + timedelta(minutes=1)),
|
| 153 |
+
_entry(run_id="r2", finished_at=new + timedelta(minutes=1)),
|
| 154 |
+
]
|
| 155 |
+
result = scenario_sessions(entries, "Debate Duel")
|
| 156 |
+
assert len(result) == 2
|
| 157 |
+
assert result[0].run_id == "r2" # newer first
|
| 158 |
+
assert result[1].run_id == "r1"
|
| 159 |
+
|
| 160 |
+
def test_run_id_tiebreak_when_finished_at_same(self):
|
| 161 |
+
"""When finished_at is equal, run_id is the tiebreaker (ascending alphabetical)."""
|
| 162 |
+
same_time = datetime(2025, 6, 14, 10, 0, 0, tzinfo=timezone.utc)
|
| 163 |
+
entries = [
|
| 164 |
+
_entry(run_id="r2", finished_at=same_time + timedelta(minutes=1)),
|
| 165 |
+
_entry(run_id="r1", finished_at=same_time + timedelta(minutes=1)),
|
| 166 |
+
]
|
| 167 |
+
result = scenario_sessions(entries, "Debate Duel")
|
| 168 |
+
assert len(result) == 2
|
| 169 |
+
# Same finished_at; run_id breaks ties in ascending order (r1 < r2)
|
| 170 |
+
assert result[0].run_id == "r1"
|
| 171 |
+
assert result[1].run_id == "r2"
|
| 172 |
+
|
| 173 |
+
|
| 174 |
+
# ββ Tests: model_table (endpoint-level stats across all scenarios) ββββββββββββββββββ
|
| 175 |
+
|
| 176 |
+
|
| 177 |
+
class TestModelTable:
|
| 178 |
+
"""Verify model aggregation: plays, wins, win_rate, scenarios, deterministic sort."""
|
| 179 |
+
|
| 180 |
+
def test_empty_entries_returns_empty_list(self):
|
| 181 |
+
"""No entries β no rows."""
|
| 182 |
+
result = model_table([])
|
| 183 |
+
assert result == []
|
| 184 |
+
|
| 185 |
+
def test_no_winner_entry_not_counted(self):
|
| 186 |
+
"""An entry with no winner contributes no stats."""
|
| 187 |
+
entry = _entry(winner=None)
|
| 188 |
+
result = model_table([entry])
|
| 189 |
+
assert result == []
|
| 190 |
+
|
| 191 |
+
def test_one_model_one_play_one_win(self):
|
| 192 |
+
"""A single-entry win: plays=1, wins=1, win_rate=1.0."""
|
| 193 |
+
entry = _entry(
|
| 194 |
+
scenario="Debate Duel",
|
| 195 |
+
cast={"alice": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B")},
|
| 196 |
+
winner="alice",
|
| 197 |
+
winning_model="openai/openbmb/MiniCPM-8B",
|
| 198 |
+
winning_models=["openai/openbmb/MiniCPM-8B"],
|
| 199 |
+
)
|
| 200 |
+
result = model_table([entry])
|
| 201 |
+
assert len(result) == 1
|
| 202 |
+
assert result[0].model == "openai/openbmb/MiniCPM-8B"
|
| 203 |
+
assert result[0].plays == 1
|
| 204 |
+
assert result[0].wins == 1
|
| 205 |
+
assert result[0].win_rate == 1.0
|
| 206 |
+
assert result[0].scenarios == ["Debate Duel"]
|
| 207 |
+
|
| 208 |
+
def test_one_model_mixed_wins_and_losses(self):
|
| 209 |
+
"""A model with 2 wins / 3 plays has win_rate β 0.667."""
|
| 210 |
+
entries = [
|
| 211 |
+
_entry(
|
| 212 |
+
run_id="r1",
|
| 213 |
+
scenario="Debate Duel",
|
| 214 |
+
cast={"alice": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B")},
|
| 215 |
+
winner="alice",
|
| 216 |
+
winning_model="openai/openbmb/MiniCPM-8B",
|
| 217 |
+
),
|
| 218 |
+
_entry(
|
| 219 |
+
run_id="r2",
|
| 220 |
+
scenario="Debate Duel",
|
| 221 |
+
cast={"alice": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B")},
|
| 222 |
+
winner="alice",
|
| 223 |
+
winning_model="openai/openbmb/MiniCPM-8B",
|
| 224 |
+
),
|
| 225 |
+
_entry(
|
| 226 |
+
run_id="r3",
|
| 227 |
+
scenario="Debate Duel",
|
| 228 |
+
cast={
|
| 229 |
+
"alice": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B"),
|
| 230 |
+
"bob": CastBinding(model_endpoint="google/gemma-12B"),
|
| 231 |
+
},
|
| 232 |
+
winner="bob",
|
| 233 |
+
winning_model="google/gemma-12B",
|
| 234 |
+
),
|
| 235 |
+
]
|
| 236 |
+
result = model_table(entries)
|
| 237 |
+
m8b = next((r for r in result if r.model == "openai/openbmb/MiniCPM-8B"), None)
|
| 238 |
+
assert m8b is not None
|
| 239 |
+
assert m8b.plays == 3
|
| 240 |
+
assert m8b.wins == 2
|
| 241 |
+
assert abs(m8b.win_rate - (2 / 3)) < 0.001
|
| 242 |
+
|
| 243 |
+
def test_model_endpoint_deduplication_one_play_per_endpoint_per_run(self):
|
| 244 |
+
"""A model filling two seats in one run counts as one play (dedup by endpoint)."""
|
| 245 |
+
entry = _entry(
|
| 246 |
+
run_id="r1",
|
| 247 |
+
scenario="Some Scenario",
|
| 248 |
+
cast={
|
| 249 |
+
"alice": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B"),
|
| 250 |
+
"bob": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B"), # same endpoint
|
| 251 |
+
},
|
| 252 |
+
winner="alice",
|
| 253 |
+
winning_model="openai/openbmb/MiniCPM-8B",
|
| 254 |
+
symmetric_seats=["seat_a", "seat_b"],
|
| 255 |
+
)
|
| 256 |
+
result = model_table([entry])
|
| 257 |
+
assert len(result) == 1
|
| 258 |
+
assert result[0].plays == 1 # not 2!
|
| 259 |
+
assert result[0].wins == 1
|
| 260 |
+
|
| 261 |
+
def test_scenarios_lists_all_distinct_scenarios_sorted(self):
|
| 262 |
+
"""A model in multiple scenarios lists all of them (sorted)."""
|
| 263 |
+
entries = [
|
| 264 |
+
_entry(
|
| 265 |
+
run_id="r1",
|
| 266 |
+
scenario="Zebra Debate",
|
| 267 |
+
cast={"alice": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B")},
|
| 268 |
+
winner="alice",
|
| 269 |
+
winning_model="openai/openbmb/MiniCPM-8B",
|
| 270 |
+
),
|
| 271 |
+
_entry(
|
| 272 |
+
run_id="r2",
|
| 273 |
+
scenario="Alpha Trivia",
|
| 274 |
+
cast={"alice": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B")},
|
| 275 |
+
winner="alice",
|
| 276 |
+
winning_model="openai/openbmb/MiniCPM-8B",
|
| 277 |
+
),
|
| 278 |
+
]
|
| 279 |
+
result = model_table(entries)
|
| 280 |
+
assert len(result) == 1
|
| 281 |
+
assert result[0].scenarios == ["Alpha Trivia", "Zebra Debate"]
|
| 282 |
+
|
| 283 |
+
def test_deterministic_sort_win_rate_then_wins_then_plays_then_model(self):
|
| 284 |
+
"""model_table sorts by (-win_rate, -wins, -plays, model asc)."""
|
| 285 |
+
entries = [
|
| 286 |
+
# MiniCPM-8B: 2 wins / 2 plays = 1.0
|
| 287 |
+
_entry(
|
| 288 |
+
run_id="r1",
|
| 289 |
+
cast={"alice": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B")},
|
| 290 |
+
winner="alice",
|
| 291 |
+
winning_model="openai/openbmb/MiniCPM-8B",
|
| 292 |
+
),
|
| 293 |
+
_entry(
|
| 294 |
+
run_id="r2",
|
| 295 |
+
cast={"alice": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B")},
|
| 296 |
+
winner="alice",
|
| 297 |
+
winning_model="openai/openbmb/MiniCPM-8B",
|
| 298 |
+
),
|
| 299 |
+
# Gemma: 1 win / 2 plays = 0.5
|
| 300 |
+
_entry(
|
| 301 |
+
run_id="r3",
|
| 302 |
+
cast={"alice": CastBinding(model_endpoint="google/gemma-12B")},
|
| 303 |
+
winner="alice",
|
| 304 |
+
winning_model="google/gemma-12B",
|
| 305 |
+
),
|
| 306 |
+
_entry(
|
| 307 |
+
run_id="r4",
|
| 308 |
+
cast={
|
| 309 |
+
"alice": CastBinding(model_endpoint="google/gemma-12B"),
|
| 310 |
+
"bob": CastBinding(model_endpoint="openai/openbmb/MiniCPM-4B"),
|
| 311 |
+
},
|
| 312 |
+
winner="bob",
|
| 313 |
+
winning_model="openai/openbmb/MiniCPM-4B", # MiniCPM-4B wins here
|
| 314 |
+
),
|
| 315 |
+
# MiniCPM-4B: 0 wins / 2 plays = 0.0 (plays in r4 winning, but that's a different calc)
|
| 316 |
+
_entry(
|
| 317 |
+
run_id="r5",
|
| 318 |
+
cast={
|
| 319 |
+
"alice": CastBinding(model_endpoint="openai/openbmb/MiniCPM-4B"),
|
| 320 |
+
"bob": CastBinding(model_endpoint="meta/llama-7B"),
|
| 321 |
+
},
|
| 322 |
+
winner="bob",
|
| 323 |
+
winning_model="meta/llama-7B",
|
| 324 |
+
),
|
| 325 |
+
_entry(
|
| 326 |
+
run_id="r6",
|
| 327 |
+
cast={
|
| 328 |
+
"alice": CastBinding(model_endpoint="openai/openbmb/MiniCPM-4B"),
|
| 329 |
+
"bob": CastBinding(model_endpoint="meta/llama-7B"),
|
| 330 |
+
},
|
| 331 |
+
winner="alice", # MiniCPM-4B wins (but alice is the seat, need to check...)
|
| 332 |
+
winning_model="openai/openbmb/MiniCPM-4B",
|
| 333 |
+
),
|
| 334 |
+
]
|
| 335 |
+
result = model_table(entries)
|
| 336 |
+
result_models = [r.model for r in result]
|
| 337 |
+
assert result_models[0] == "openai/openbmb/MiniCPM-8B" # 1.0, 2 wins, 2 plays
|
| 338 |
+
# MiniCPM-4B: 2 wins / 3 plays β 0.67, Gemma: 1 win / 2 plays = 0.5, Llama: 1 win / 2 plays = 0.5
|
| 339 |
+
# Sort: MiniCPM-8B (1.0) > MiniCPM-4B (0.67) > Gemma (0.5, "gemma" < "llama") > Llama (0.5)
|
| 340 |
+
assert result_models[1] == "openai/openbmb/MiniCPM-4B" # 0.67, 2 wins
|
| 341 |
+
assert result_models[2] == "google/gemma-12B" # 0.5, 1 win
|
| 342 |
+
assert result_models[3] == "meta/llama-7B" # 0.5, 1 win
|
| 343 |
+
|
| 344 |
+
def test_winning_models_includes_all_credited_endpoints(self):
|
| 345 |
+
"""A run with winning_models=[a,b] credits both with a win."""
|
| 346 |
+
entry = _entry(
|
| 347 |
+
run_id="r1",
|
| 348 |
+
scenario="Debate Duel",
|
| 349 |
+
cast={
|
| 350 |
+
"alice": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B"),
|
| 351 |
+
"bob": CastBinding(model_endpoint="google/gemma-12B"),
|
| 352 |
+
},
|
| 353 |
+
winner="team_a",
|
| 354 |
+
winner_kind="team",
|
| 355 |
+
winning_model=None,
|
| 356 |
+
winning_models=["openai/openbmb/MiniCPM-8B", "google/gemma-12B"],
|
| 357 |
+
symmetric_seats=["seat_a", "seat_b"],
|
| 358 |
+
)
|
| 359 |
+
result = model_table([entry])
|
| 360 |
+
assert len(result) == 2
|
| 361 |
+
m8b = next((r for r in result if r.model == "openai/openbmb/MiniCPM-8B"), None)
|
| 362 |
+
assert m8b is not None and m8b.wins == 1
|
| 363 |
+
gemma = next((r for r in result if r.model == "google/gemma-12B"), None)
|
| 364 |
+
assert gemma is not None and gemma.wins == 1
|
| 365 |
+
|
| 366 |
+
|
| 367 |
+
# ββ Tests: agent_table (per-scenario, per-persona stats) ββββββββββββββββββββββββββββ
|
| 368 |
+
|
| 369 |
+
|
| 370 |
+
class TestAgentTable:
|
| 371 |
+
"""Verify agent attribution: plays, wins, seat_type, model_endpoints, sort."""
|
| 372 |
+
|
| 373 |
+
def test_empty_entries_returns_empty_list(self):
|
| 374 |
+
"""No entries β no rows."""
|
| 375 |
+
result = agent_table([], "Debate Duel")
|
| 376 |
+
assert result == []
|
| 377 |
+
|
| 378 |
+
def test_agent_filtered_by_scenario(self):
|
| 379 |
+
"""agent_table only includes agents from the named scenario."""
|
| 380 |
+
entries = [
|
| 381 |
+
_entry(
|
| 382 |
+
run_id="r1",
|
| 383 |
+
scenario="Debate Duel",
|
| 384 |
+
cast={"alice": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B")},
|
| 385 |
+
winner="alice",
|
| 386 |
+
),
|
| 387 |
+
_entry(
|
| 388 |
+
run_id="r2",
|
| 389 |
+
scenario="Other Scenario",
|
| 390 |
+
cast={"bob": CastBinding(model_endpoint="google/gemma-12B")},
|
| 391 |
+
winner="bob",
|
| 392 |
+
),
|
| 393 |
+
]
|
| 394 |
+
result = agent_table(entries, "Debate Duel")
|
| 395 |
+
agents = [r.agent for r in result]
|
| 396 |
+
assert agents == ["alice"]
|
| 397 |
+
|
| 398 |
+
def test_agent_single_play_single_win(self):
|
| 399 |
+
"""An agent with one win: plays=1, wins=1, win_rate=1.0."""
|
| 400 |
+
entry = _entry(
|
| 401 |
+
scenario="Debate Duel",
|
| 402 |
+
cast={"alice": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B")},
|
| 403 |
+
winner="alice",
|
| 404 |
+
winning_model="openai/openbmb/MiniCPM-8B",
|
| 405 |
+
symmetric_seats=["debater_a"],
|
| 406 |
+
)
|
| 407 |
+
result = agent_table([entry], "Debate Duel")
|
| 408 |
+
assert len(result) == 1
|
| 409 |
+
assert result[0].agent == "alice"
|
| 410 |
+
assert result[0].plays == 1
|
| 411 |
+
assert result[0].wins == 1
|
| 412 |
+
assert result[0].win_rate == 1.0
|
| 413 |
+
|
| 414 |
+
def test_agent_seat_type_from_symmetric_seats(self):
|
| 415 |
+
"""In symmetric-seat scenarios, agent's seat_type is the seat name."""
|
| 416 |
+
entry = _entry(
|
| 417 |
+
scenario="Debate Duel",
|
| 418 |
+
cast={"debater_a": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B")},
|
| 419 |
+
winner="debater_a",
|
| 420 |
+
winning_model="openai/openbmb/MiniCPM-8B",
|
| 421 |
+
symmetric_seats=["debater_a", "debater_b"],
|
| 422 |
+
)
|
| 423 |
+
result = agent_table([entry], "Debate Duel")
|
| 424 |
+
assert result[0].agent == "debater_a"
|
| 425 |
+
assert result[0].seat_type == "debater_a"
|
| 426 |
+
|
| 427 |
+
def test_agent_seat_type_from_teams(self):
|
| 428 |
+
"""In team scenarios, agent's seat_type is the team label they belong to."""
|
| 429 |
+
entry = _entry(
|
| 430 |
+
scenario="Team Game",
|
| 431 |
+
cast={
|
| 432 |
+
"alice": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B"),
|
| 433 |
+
"bob": CastBinding(model_endpoint="google/gemma-12B"),
|
| 434 |
+
},
|
| 435 |
+
teams={"team_a": ["alice", "charlie"], "team_b": ["bob"]},
|
| 436 |
+
winner="alice",
|
| 437 |
+
winning_model="openai/openbmb/MiniCPM-8B",
|
| 438 |
+
winner_kind="agent",
|
| 439 |
+
competition_kind="versus",
|
| 440 |
+
)
|
| 441 |
+
result = agent_table([entry], "Team Game")
|
| 442 |
+
alice = next((r for r in result if r.agent == "alice"), None)
|
| 443 |
+
bob = next((r for r in result if r.agent == "bob"), None)
|
| 444 |
+
assert alice is not None and alice.seat_type == "team_a"
|
| 445 |
+
assert bob is not None and bob.seat_type == "team_b"
|
| 446 |
+
|
| 447 |
+
def test_agent_seat_type_empty_when_unmapped(self):
|
| 448 |
+
"""An agent not in teams or symmetric_seats (e.g., judge) has seat_type=''."""
|
| 449 |
+
entry = _entry(
|
| 450 |
+
scenario="Debate Duel",
|
| 451 |
+
cast={
|
| 452 |
+
"debater_a": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B"),
|
| 453 |
+
"judge": CastBinding(model_endpoint=None),
|
| 454 |
+
},
|
| 455 |
+
winner="debater_a",
|
| 456 |
+
winning_model="openai/openbmb/MiniCPM-8B",
|
| 457 |
+
symmetric_seats=["debater_a"],
|
| 458 |
+
competition_kind="judged",
|
| 459 |
+
)
|
| 460 |
+
result = agent_table([entry], "Debate Duel")
|
| 461 |
+
judge_row = next((r for r in result if r.agent == "judge"), None)
|
| 462 |
+
assert judge_row is not None
|
| 463 |
+
assert judge_row.seat_type == ""
|
| 464 |
+
|
| 465 |
+
def test_agent_model_endpoints_deduped_sorted(self):
|
| 466 |
+
"""An agent's model_endpoints is sorted, distinct models that filled the seat."""
|
| 467 |
+
entries = [
|
| 468 |
+
_entry(
|
| 469 |
+
run_id="r1",
|
| 470 |
+
scenario="Debate Duel",
|
| 471 |
+
cast={"alice": CastBinding(model_endpoint="google/gemma-12B")},
|
| 472 |
+
winner="alice",
|
| 473 |
+
winning_model="google/gemma-12B",
|
| 474 |
+
),
|
| 475 |
+
_entry(
|
| 476 |
+
run_id="r2",
|
| 477 |
+
scenario="Debate Duel",
|
| 478 |
+
cast={"alice": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B")},
|
| 479 |
+
winner="alice",
|
| 480 |
+
winning_model="openai/openbmb/MiniCPM-8B",
|
| 481 |
+
),
|
| 482 |
+
_entry(
|
| 483 |
+
run_id="r3",
|
| 484 |
+
scenario="Debate Duel",
|
| 485 |
+
cast={"alice": CastBinding(model_endpoint="google/gemma-12B")}, # repeat
|
| 486 |
+
winner="alice",
|
| 487 |
+
winning_model="google/gemma-12B",
|
| 488 |
+
),
|
| 489 |
+
]
|
| 490 |
+
result = agent_table(entries, "Debate Duel")
|
| 491 |
+
assert len(result) == 1
|
| 492 |
+
assert result[0].model_endpoints == ["google/gemma-12B", "openai/openbmb/MiniCPM-8B"]
|
| 493 |
+
|
| 494 |
+
def test_team_win_credits_all_team_members(self):
|
| 495 |
+
"""A team win credits every member of the winning team with a win."""
|
| 496 |
+
entry = _entry(
|
| 497 |
+
scenario="Team Game",
|
| 498 |
+
cast={
|
| 499 |
+
"alice": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B"),
|
| 500 |
+
"bob": CastBinding(model_endpoint="google/gemma-12B"),
|
| 501 |
+
},
|
| 502 |
+
teams={"team_a": ["alice"], "team_b": ["bob"]},
|
| 503 |
+
winner="team_a",
|
| 504 |
+
winner_kind="team",
|
| 505 |
+
winning_models=["openai/openbmb/MiniCPM-8B"],
|
| 506 |
+
winning_model=None,
|
| 507 |
+
competition_kind="versus",
|
| 508 |
+
)
|
| 509 |
+
result = agent_table([entry], "Team Game")
|
| 510 |
+
alice = next((r for r in result if r.agent == "alice"), None)
|
| 511 |
+
bob = next((r for r in result if r.agent == "bob"), None)
|
| 512 |
+
assert alice is not None and alice.wins == 1
|
| 513 |
+
assert bob is not None and bob.wins == 0
|
| 514 |
+
|
| 515 |
+
|
| 516 |
+
# ββ Tests: fairness_table (seat-type aggregation) βββββββββββββββββββββββββββββββββ
|
| 517 |
+
|
| 518 |
+
|
| 519 |
+
class TestFairnessTable:
|
| 520 |
+
"""Verify fairness_table: seat-type aggregation, unmapped excluded, sort."""
|
| 521 |
+
|
| 522 |
+
def test_empty_entries_returns_empty_list(self):
|
| 523 |
+
"""No entries β no rows."""
|
| 524 |
+
result = fairness_table([], "Debate Duel")
|
| 525 |
+
assert result == []
|
| 526 |
+
|
| 527 |
+
def test_only_declared_seat_types_appear(self):
|
| 528 |
+
"""Only declared seat_types (teams or symmetric_seats) appear."""
|
| 529 |
+
entry = _entry(
|
| 530 |
+
scenario="Debate Duel",
|
| 531 |
+
cast={
|
| 532 |
+
"debater_a": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B"),
|
| 533 |
+
"judge": CastBinding(model_endpoint=None),
|
| 534 |
+
},
|
| 535 |
+
winner="debater_a",
|
| 536 |
+
winning_model="openai/openbmb/MiniCPM-8B",
|
| 537 |
+
symmetric_seats=["debater_a"],
|
| 538 |
+
competition_kind="judged",
|
| 539 |
+
)
|
| 540 |
+
result = fairness_table([entry], "Debate Duel")
|
| 541 |
+
seat_types = [r.seat_type for r in result]
|
| 542 |
+
# Only "debater_a" is declared; "judge" is unmapped.
|
| 543 |
+
assert seat_types == ["debater_a"]
|
| 544 |
+
|
| 545 |
+
def test_symmetric_seat_plays_and_wins(self):
|
| 546 |
+
"""In symmetric-seat scenarios, each seat contributes one play per run."""
|
| 547 |
+
entry = _entry(
|
| 548 |
+
scenario="Debate Duel",
|
| 549 |
+
cast={
|
| 550 |
+
"debater_a": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B"),
|
| 551 |
+
"debater_b": CastBinding(model_endpoint="google/gemma-12B"),
|
| 552 |
+
},
|
| 553 |
+
winner="debater_a",
|
| 554 |
+
winning_model="openai/openbmb/MiniCPM-8B",
|
| 555 |
+
symmetric_seats=["debater_a", "debater_b"],
|
| 556 |
+
)
|
| 557 |
+
result = fairness_table([entry], "Debate Duel")
|
| 558 |
+
debater_a = next((r for r in result if r.seat_type == "debater_a"), None)
|
| 559 |
+
debater_b = next((r for r in result if r.seat_type == "debater_b"), None)
|
| 560 |
+
assert debater_a is not None and debater_a.plays == 1 and debater_a.wins == 1
|
| 561 |
+
assert debater_b is not None and debater_b.plays == 1 and debater_b.wins == 0
|
| 562 |
+
|
| 563 |
+
def test_team_seats_aggregation(self):
|
| 564 |
+
"""In team scenarios, each team contributes one play; wins go to winning team."""
|
| 565 |
+
entry = _entry(
|
| 566 |
+
scenario="Team Game",
|
| 567 |
+
cast={
|
| 568 |
+
"alice": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B"),
|
| 569 |
+
"bob": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B"),
|
| 570 |
+
"charlie": CastBinding(model_endpoint="google/gemma-12B"),
|
| 571 |
+
},
|
| 572 |
+
teams={"team_a": ["alice", "bob"], "team_b": ["charlie"]},
|
| 573 |
+
winner="team_a",
|
| 574 |
+
winner_kind="team",
|
| 575 |
+
winning_models=["openai/openbmb/MiniCPM-8B"],
|
| 576 |
+
winning_model=None,
|
| 577 |
+
competition_kind="versus",
|
| 578 |
+
)
|
| 579 |
+
result = fairness_table([entry], "Team Game")
|
| 580 |
+
team_a = next((r for r in result if r.seat_type == "team_a"), None)
|
| 581 |
+
team_b = next((r for r in result if r.seat_type == "team_b"), None)
|
| 582 |
+
assert team_a is not None and team_a.plays == 1 and team_a.wins == 1
|
| 583 |
+
assert team_b is not None and team_b.plays == 1 and team_b.wins == 0
|
| 584 |
+
|
| 585 |
+
def test_judge_not_counted_in_fairness(self):
|
| 586 |
+
"""A judge (unmapped cast member) does not appear in fairness_table."""
|
| 587 |
+
entry = _entry(
|
| 588 |
+
scenario="Debate Duel",
|
| 589 |
+
cast={
|
| 590 |
+
"debater_a": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B"),
|
| 591 |
+
"judge": CastBinding(model_endpoint=None),
|
| 592 |
+
},
|
| 593 |
+
winner="debater_a",
|
| 594 |
+
winning_model="openai/openbmb/MiniCPM-8B",
|
| 595 |
+
symmetric_seats=["debater_a"],
|
| 596 |
+
competition_kind="judged",
|
| 597 |
+
)
|
| 598 |
+
result = fairness_table([entry], "Debate Duel")
|
| 599 |
+
assert len(result) == 1
|
| 600 |
+
assert result[0].seat_type == "debater_a"
|
| 601 |
+
|
| 602 |
+
def test_win_rate_calculated_per_seat_type(self):
|
| 603 |
+
"""A seat winning 1 of 2 runs has win_rate=0.5."""
|
| 604 |
+
entries = [
|
| 605 |
+
_entry(
|
| 606 |
+
run_id="r1",
|
| 607 |
+
scenario="Debate Duel",
|
| 608 |
+
cast={
|
| 609 |
+
"debater_a": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B"),
|
| 610 |
+
"debater_b": CastBinding(model_endpoint="google/gemma-12B"),
|
| 611 |
+
},
|
| 612 |
+
winner="debater_a",
|
| 613 |
+
winning_model="openai/openbmb/MiniCPM-8B",
|
| 614 |
+
symmetric_seats=["debater_a", "debater_b"],
|
| 615 |
+
),
|
| 616 |
+
_entry(
|
| 617 |
+
run_id="r2",
|
| 618 |
+
scenario="Debate Duel",
|
| 619 |
+
cast={
|
| 620 |
+
"debater_a": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B"),
|
| 621 |
+
"debater_b": CastBinding(model_endpoint="google/gemma-12B"),
|
| 622 |
+
},
|
| 623 |
+
winner="debater_b",
|
| 624 |
+
winning_model="google/gemma-12B",
|
| 625 |
+
symmetric_seats=["debater_a", "debater_b"],
|
| 626 |
+
),
|
| 627 |
+
]
|
| 628 |
+
result = fairness_table(entries, "Debate Duel")
|
| 629 |
+
debater_a = next((r for r in result if r.seat_type == "debater_a"), None)
|
| 630 |
+
debater_b = next((r for r in result if r.seat_type == "debater_b"), None)
|
| 631 |
+
assert debater_a is not None and abs(debater_a.win_rate - 0.5) < 0.001
|
| 632 |
+
assert debater_b is not None and abs(debater_b.win_rate - 0.5) < 0.001
|
| 633 |
+
|
| 634 |
+
|
| 635 |
+
# ββ Tests: headline (model-vs-model narrative) βββββββββββββββββββββββββββββββββββββ
|
| 636 |
+
|
| 637 |
+
|
| 638 |
+
class TestHeadline:
|
| 639 |
+
"""Verify headline: symmetric seats + β₯2 models with β₯1 win each."""
|
| 640 |
+
|
| 641 |
+
def test_empty_entries_returns_none(self):
|
| 642 |
+
"""No entries β no headline."""
|
| 643 |
+
result = headline([])
|
| 644 |
+
assert result is None
|
| 645 |
+
|
| 646 |
+
def test_no_symmetric_seat_scenarios_returns_none(self):
|
| 647 |
+
"""Headline needs symmetric-seat scenario (model-vs-model); team scenarios excluded."""
|
| 648 |
+
entry = _entry(
|
| 649 |
+
scenario="Team Game",
|
| 650 |
+
cast={
|
| 651 |
+
"alice": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B"),
|
| 652 |
+
"bob": CastBinding(model_endpoint="google/gemma-12B"),
|
| 653 |
+
},
|
| 654 |
+
teams={"team_a": ["alice"], "team_b": ["bob"]},
|
| 655 |
+
winner="team_a",
|
| 656 |
+
winner_kind="team",
|
| 657 |
+
winning_models=["openai/openbmb/MiniCPM-8B"],
|
| 658 |
+
winning_model=None,
|
| 659 |
+
competition_kind="versus",
|
| 660 |
+
)
|
| 661 |
+
result = headline([entry])
|
| 662 |
+
assert result is None
|
| 663 |
+
|
| 664 |
+
def test_only_one_model_in_scenario_returns_none(self):
|
| 665 |
+
"""A scenario with only one model (even if winning) doesn't qualify."""
|
| 666 |
+
entry = _entry(
|
| 667 |
+
scenario="Debate Duel",
|
| 668 |
+
cast={
|
| 669 |
+
"debater_a": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B"),
|
| 670 |
+
"debater_b": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B"),
|
| 671 |
+
},
|
| 672 |
+
winner="debater_a",
|
| 673 |
+
winning_model="openai/openbmb/MiniCPM-8B",
|
| 674 |
+
symmetric_seats=["debater_a", "debater_b"],
|
| 675 |
+
)
|
| 676 |
+
result = headline([entry])
|
| 677 |
+
assert result is None # only 1 distinct model
|
| 678 |
+
|
| 679 |
+
def test_two_models_but_loser_has_zero_wins_returns_none(self):
|
| 680 |
+
"""Headline needs β₯2 models that have each won β₯1."""
|
| 681 |
+
entry = _entry(
|
| 682 |
+
scenario="Debate Duel",
|
| 683 |
+
cast={
|
| 684 |
+
"debater_a": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B"),
|
| 685 |
+
"debater_b": CastBinding(model_endpoint="google/gemma-12B"),
|
| 686 |
+
},
|
| 687 |
+
winner="debater_a",
|
| 688 |
+
winning_model="openai/openbmb/MiniCPM-8B",
|
| 689 |
+
winning_models=["openai/openbmb/MiniCPM-8B"],
|
| 690 |
+
symmetric_seats=["debater_a", "debater_b"],
|
| 691 |
+
)
|
| 692 |
+
result = headline([entry])
|
| 693 |
+
assert result is None # gemma has 0 wins
|
| 694 |
+
|
| 695 |
+
def test_headline_with_two_models_both_winning(self):
|
| 696 |
+
"""Headline is generated when β₯2 models have each won β₯1 in symmetric scenario."""
|
| 697 |
+
entries = [
|
| 698 |
+
_entry(
|
| 699 |
+
run_id="r1",
|
| 700 |
+
scenario="Debate Duel",
|
| 701 |
+
cast={
|
| 702 |
+
"debater_a": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B"),
|
| 703 |
+
"debater_b": CastBinding(model_endpoint="google/gemma-12B"),
|
| 704 |
+
},
|
| 705 |
+
winner="debater_a",
|
| 706 |
+
winning_model="openai/openbmb/MiniCPM-8B",
|
| 707 |
+
winning_models=["openai/openbmb/MiniCPM-8B"],
|
| 708 |
+
symmetric_seats=["debater_a", "debater_b"],
|
| 709 |
+
),
|
| 710 |
+
_entry(
|
| 711 |
+
run_id="r2",
|
| 712 |
+
scenario="Debate Duel",
|
| 713 |
+
cast={
|
| 714 |
+
"debater_a": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B"),
|
| 715 |
+
"debater_b": CastBinding(model_endpoint="google/gemma-12B"),
|
| 716 |
+
},
|
| 717 |
+
winner="debater_b",
|
| 718 |
+
winning_model="google/gemma-12B",
|
| 719 |
+
winning_models=["google/gemma-12B"],
|
| 720 |
+
symmetric_seats=["debater_a", "debater_b"],
|
| 721 |
+
),
|
| 722 |
+
]
|
| 723 |
+
result = headline(entries)
|
| 724 |
+
assert result is not None
|
| 725 |
+
assert "MiniCPM" in result
|
| 726 |
+
assert "gemma" in result.lower()
|
| 727 |
+
assert "Debate Duel" in result
|
| 728 |
+
assert "1-1" in result
|
| 729 |
+
|
| 730 |
+
def test_headline_picks_most_played_scenario(self):
|
| 731 |
+
"""When multiple scenarios qualify, headline picks the one with most games decided."""
|
| 732 |
+
entries = [
|
| 733 |
+
# Debate Duel: 1 game
|
| 734 |
+
_entry(
|
| 735 |
+
run_id="r1",
|
| 736 |
+
scenario="Debate Duel",
|
| 737 |
+
cast={
|
| 738 |
+
"debater_a": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B"),
|
| 739 |
+
"debater_b": CastBinding(model_endpoint="google/gemma-12B"),
|
| 740 |
+
},
|
| 741 |
+
winner="debater_a",
|
| 742 |
+
winning_model="openai/openbmb/MiniCPM-8B",
|
| 743 |
+
symmetric_seats=["debater_a", "debater_b"],
|
| 744 |
+
),
|
| 745 |
+
# Trivia Night: 3 games (most)
|
| 746 |
+
_entry(
|
| 747 |
+
run_id="r2",
|
| 748 |
+
scenario="Trivia Night",
|
| 749 |
+
cast={
|
| 750 |
+
"player_a": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B"),
|
| 751 |
+
"player_b": CastBinding(model_endpoint="google/gemma-12B"),
|
| 752 |
+
},
|
| 753 |
+
winner="player_a",
|
| 754 |
+
winning_model="openai/openbmb/MiniCPM-8B",
|
| 755 |
+
symmetric_seats=["player_a", "player_b"],
|
| 756 |
+
),
|
| 757 |
+
_entry(
|
| 758 |
+
run_id="r3",
|
| 759 |
+
scenario="Trivia Night",
|
| 760 |
+
cast={
|
| 761 |
+
"player_a": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B"),
|
| 762 |
+
"player_b": CastBinding(model_endpoint="google/gemma-12B"),
|
| 763 |
+
},
|
| 764 |
+
winner="player_b",
|
| 765 |
+
winning_model="google/gemma-12B",
|
| 766 |
+
symmetric_seats=["player_a", "player_b"],
|
| 767 |
+
),
|
| 768 |
+
_entry(
|
| 769 |
+
run_id="r4",
|
| 770 |
+
scenario="Trivia Night",
|
| 771 |
+
cast={
|
| 772 |
+
"player_a": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B"),
|
| 773 |
+
"player_b": CastBinding(model_endpoint="google/gemma-12B"),
|
| 774 |
+
},
|
| 775 |
+
winner="player_b",
|
| 776 |
+
winning_model="google/gemma-12B",
|
| 777 |
+
symmetric_seats=["player_a", "player_b"],
|
| 778 |
+
),
|
| 779 |
+
]
|
| 780 |
+
result = headline(entries)
|
| 781 |
+
assert result is not None
|
| 782 |
+
assert "Trivia Night" in result
|
| 783 |
+
|
| 784 |
+
def test_headline_alphabetical_tiebreak_when_same_wins(self):
|
| 785 |
+
"""When scenarios tie on wins, pick the alphabetically-first scenario name."""
|
| 786 |
+
entries = [
|
| 787 |
+
# "Aardvark": 1 win for MiniCPM, 1 for Gemma
|
| 788 |
+
_entry(
|
| 789 |
+
run_id="a1",
|
| 790 |
+
scenario="Aardvark Duel",
|
| 791 |
+
cast={
|
| 792 |
+
"p1": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B"),
|
| 793 |
+
"p2": CastBinding(model_endpoint="google/gemma-12B"),
|
| 794 |
+
},
|
| 795 |
+
winner="p1",
|
| 796 |
+
winning_model="openai/openbmb/MiniCPM-8B",
|
| 797 |
+
symmetric_seats=["p1", "p2"],
|
| 798 |
+
),
|
| 799 |
+
_entry(
|
| 800 |
+
run_id="a2",
|
| 801 |
+
scenario="Aardvark Duel",
|
| 802 |
+
cast={
|
| 803 |
+
"p1": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B"),
|
| 804 |
+
"p2": CastBinding(model_endpoint="google/gemma-12B"),
|
| 805 |
+
},
|
| 806 |
+
winner="p2",
|
| 807 |
+
winning_model="google/gemma-12B",
|
| 808 |
+
symmetric_seats=["p1", "p2"],
|
| 809 |
+
),
|
| 810 |
+
# "Zebra": same record, but alphabetically later
|
| 811 |
+
_entry(
|
| 812 |
+
run_id="z1",
|
| 813 |
+
scenario="Zebra Duel",
|
| 814 |
+
cast={
|
| 815 |
+
"p1": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B"),
|
| 816 |
+
"p2": CastBinding(model_endpoint="google/gemma-12B"),
|
| 817 |
+
},
|
| 818 |
+
winner="p1",
|
| 819 |
+
winning_model="openai/openbmb/MiniCPM-8B",
|
| 820 |
+
symmetric_seats=["p1", "p2"],
|
| 821 |
+
),
|
| 822 |
+
_entry(
|
| 823 |
+
run_id="z2",
|
| 824 |
+
scenario="Zebra Duel",
|
| 825 |
+
cast={
|
| 826 |
+
"p1": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B"),
|
| 827 |
+
"p2": CastBinding(model_endpoint="google/gemma-12B"),
|
| 828 |
+
},
|
| 829 |
+
winner="p2",
|
| 830 |
+
winning_model="google/gemma-12B",
|
| 831 |
+
symmetric_seats=["p1", "p2"],
|
| 832 |
+
),
|
| 833 |
+
]
|
| 834 |
+
result = headline(entries)
|
| 835 |
+
assert result is not None
|
| 836 |
+
assert "Aardvark Duel" in result # alphabetically first
|
| 837 |
+
|
| 838 |
+
def test_headline_format(self):
|
| 839 |
+
"""Headline format: 'ModelA beats ModelB Β· X-Y at Scenario'."""
|
| 840 |
+
entries = [
|
| 841 |
+
_entry(
|
| 842 |
+
run_id="r1",
|
| 843 |
+
scenario="Debate Duel",
|
| 844 |
+
cast={
|
| 845 |
+
"debater_a": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B"),
|
| 846 |
+
"debater_b": CastBinding(model_endpoint="google/gemma-12B"),
|
| 847 |
+
},
|
| 848 |
+
winner="debater_a",
|
| 849 |
+
winning_model="openai/openbmb/MiniCPM-8B",
|
| 850 |
+
symmetric_seats=["debater_a", "debater_b"],
|
| 851 |
+
),
|
| 852 |
+
_entry(
|
| 853 |
+
run_id="r2",
|
| 854 |
+
scenario="Debate Duel",
|
| 855 |
+
cast={
|
| 856 |
+
"debater_a": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B"),
|
| 857 |
+
"debater_b": CastBinding(model_endpoint="google/gemma-12B"),
|
| 858 |
+
},
|
| 859 |
+
winner="debater_a",
|
| 860 |
+
winning_model="openai/openbmb/MiniCPM-8B",
|
| 861 |
+
symmetric_seats=["debater_a", "debater_b"],
|
| 862 |
+
),
|
| 863 |
+
_entry(
|
| 864 |
+
run_id="r3",
|
| 865 |
+
scenario="Debate Duel",
|
| 866 |
+
cast={
|
| 867 |
+
"debater_a": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B"),
|
| 868 |
+
"debater_b": CastBinding(model_endpoint="google/gemma-12B"),
|
| 869 |
+
},
|
| 870 |
+
winner="debater_b",
|
| 871 |
+
winning_model="google/gemma-12B",
|
| 872 |
+
symmetric_seats=["debater_a", "debater_b"],
|
| 873 |
+
),
|
| 874 |
+
]
|
| 875 |
+
result = headline(entries)
|
| 876 |
+
assert result is not None
|
| 877 |
+
assert "beats" in result
|
| 878 |
+
assert "Β·" in result
|
| 879 |
+
assert "Debate Duel" in result
|
| 880 |
+
assert "2-1" in result # MiniCPM beats Gemma 2-1
|
| 881 |
+
|
| 882 |
+
|
| 883 |
+
# ββ Tests: Purity (determinism and idempotency) ββββββββββββββββββββββββββββββββββββ
|
| 884 |
+
|
| 885 |
+
|
| 886 |
+
class TestProjectionPurity:
|
| 887 |
+
"""Verify that projections are deterministic regardless of entry order."""
|
| 888 |
+
|
| 889 |
+
def test_scenario_sessions_idempotent(self):
|
| 890 |
+
"""Calling scenario_sessions twice returns the same result."""
|
| 891 |
+
entry = _entry(scenario="Debate Duel")
|
| 892 |
+
result1 = scenario_sessions([entry], "Debate Duel")
|
| 893 |
+
result2 = scenario_sessions([entry], "Debate Duel")
|
| 894 |
+
assert result1 == result2
|
| 895 |
+
|
| 896 |
+
def test_model_table_idempotent(self):
|
| 897 |
+
"""Calling model_table twice returns the same result."""
|
| 898 |
+
entry = _entry()
|
| 899 |
+
result1 = model_table([entry])
|
| 900 |
+
result2 = model_table([entry])
|
| 901 |
+
assert result1 == result2
|
| 902 |
+
|
| 903 |
+
def test_agent_table_deterministic_order_independent(self):
|
| 904 |
+
"""agent_table produces the same result regardless of entry order."""
|
| 905 |
+
entries = [
|
| 906 |
+
_entry(
|
| 907 |
+
run_id="r1",
|
| 908 |
+
scenario="Debate Duel",
|
| 909 |
+
cast={"alice": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B")},
|
| 910 |
+
winner="alice",
|
| 911 |
+
),
|
| 912 |
+
_entry(
|
| 913 |
+
run_id="r2",
|
| 914 |
+
scenario="Debate Duel",
|
| 915 |
+
cast={"bob": CastBinding(model_endpoint="google/gemma-12B")},
|
| 916 |
+
winner="bob",
|
| 917 |
+
),
|
| 918 |
+
]
|
| 919 |
+
result1 = agent_table(entries, "Debate Duel")
|
| 920 |
+
result2 = agent_table(list(reversed(entries)), "Debate Duel")
|
| 921 |
+
assert result1 == result2
|
| 922 |
+
|
| 923 |
+
def test_fairness_table_deterministic_order_independent(self):
|
| 924 |
+
"""fairness_table produces the same result regardless of entry order."""
|
| 925 |
+
entries = [
|
| 926 |
+
_entry(
|
| 927 |
+
run_id="r1",
|
| 928 |
+
scenario="Debate Duel",
|
| 929 |
+
cast={
|
| 930 |
+
"debater_a": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B"),
|
| 931 |
+
"debater_b": CastBinding(model_endpoint="google/gemma-12B"),
|
| 932 |
+
},
|
| 933 |
+
winner="debater_a",
|
| 934 |
+
symmetric_seats=["debater_a", "debater_b"],
|
| 935 |
+
),
|
| 936 |
+
_entry(
|
| 937 |
+
run_id="r2",
|
| 938 |
+
scenario="Debate Duel",
|
| 939 |
+
cast={
|
| 940 |
+
"debater_a": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B"),
|
| 941 |
+
"debater_b": CastBinding(model_endpoint="google/gemma-12B"),
|
| 942 |
+
},
|
| 943 |
+
winner="debater_b",
|
| 944 |
+
symmetric_seats=["debater_a", "debater_b"],
|
| 945 |
+
),
|
| 946 |
+
]
|
| 947 |
+
result1 = fairness_table(entries, "Debate Duel")
|
| 948 |
+
result2 = fairness_table(list(reversed(entries)), "Debate Duel")
|
| 949 |
+
assert result1 == result2
|
| 950 |
+
|
| 951 |
+
def test_headline_deterministic_order_independent(self):
|
| 952 |
+
"""headline produces the same result regardless of entry order."""
|
| 953 |
+
entries = [
|
| 954 |
+
_entry(
|
| 955 |
+
run_id="r1",
|
| 956 |
+
scenario="Debate Duel",
|
| 957 |
+
cast={
|
| 958 |
+
"debater_a": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B"),
|
| 959 |
+
"debater_b": CastBinding(model_endpoint="google/gemma-12B"),
|
| 960 |
+
},
|
| 961 |
+
winner="debater_a",
|
| 962 |
+
symmetric_seats=["debater_a", "debater_b"],
|
| 963 |
+
),
|
| 964 |
+
_entry(
|
| 965 |
+
run_id="r2",
|
| 966 |
+
scenario="Debate Duel",
|
| 967 |
+
cast={
|
| 968 |
+
"debater_a": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B"),
|
| 969 |
+
"debater_b": CastBinding(model_endpoint="google/gemma-12B"),
|
| 970 |
+
},
|
| 971 |
+
winner="debater_b",
|
| 972 |
+
symmetric_seats=["debater_a", "debater_b"],
|
| 973 |
+
),
|
| 974 |
+
]
|
| 975 |
+
result1 = headline(entries)
|
| 976 |
+
result2 = headline(list(reversed(entries)))
|
| 977 |
+
assert result1 == result2
|
tests/test_leaderboard_store.py
ADDED
|
@@ -0,0 +1,531 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Leaderboard store tests β persistence, round-trip, idempotence, E2E write path.
|
| 2 |
+
|
| 3 |
+
This suite verifies:
|
| 4 |
+
- LeaderboardStore round-trip: record β read back via .entries() and .entries_for_scenario()
|
| 5 |
+
- Idempotent upsert on run_id (no duplicates, corrective records replace)
|
| 6 |
+
- Ordering: newest finished_at first, run_id tiebreak
|
| 7 |
+
- build_entry gating: only finished + winner + winning_model + competitive runs recorded
|
| 8 |
+
- Separate-table isolation: leaderboard_entries β events (independent tables)
|
| 9 |
+
- End-to-end write path: FishbowlSession drive + finalize + leaderboard.recorded assertion
|
| 10 |
+
|
| 11 |
+
Test strategy: Build entries directly for unit tests; drive a real FishbowlSession offline
|
| 12 |
+
(deterministic stub) for the E2E test, reaching a verdict, calling finalize, and asserting
|
| 13 |
+
the scoreboard row landed in the store.
|
| 14 |
+
"""
|
| 15 |
+
|
| 16 |
+
from __future__ import annotations
|
| 17 |
+
|
| 18 |
+
from datetime import datetime, timezone
|
| 19 |
+
|
| 20 |
+
import pytest
|
| 21 |
+
|
| 22 |
+
from src.core.leaderboard_store import (
|
| 23 |
+
LeaderboardEntry,
|
| 24 |
+
LeaderboardStore,
|
| 25 |
+
_reset_store_cache,
|
| 26 |
+
build_entry,
|
| 27 |
+
make_leaderboard_store,
|
| 28 |
+
)
|
| 29 |
+
from src.core.run_index import CastBinding, RunSummary
|
| 30 |
+
|
| 31 |
+
|
| 32 |
+
@pytest.fixture(autouse=True)
|
| 33 |
+
def _reset_leaderboard_store():
|
| 34 |
+
"""Reset the memoised store cache before and after each test to prevent cross-test leakage."""
|
| 35 |
+
_reset_store_cache()
|
| 36 |
+
yield
|
| 37 |
+
_reset_store_cache()
|
| 38 |
+
|
| 39 |
+
|
| 40 |
+
def _entry(
|
| 41 |
+
run_id: str = "r1",
|
| 42 |
+
scenario: str = "Debate Duel",
|
| 43 |
+
seed: str = "seed123",
|
| 44 |
+
session_id: str | None = None,
|
| 45 |
+
competition_kind: str = "versus",
|
| 46 |
+
teams: dict[str, list[str]] | None = None,
|
| 47 |
+
symmetric_seats: list[str] | None = None,
|
| 48 |
+
cast: dict[str, CastBinding] | None = None,
|
| 49 |
+
winner: str | None = "alice",
|
| 50 |
+
winner_kind: str | None = "agent",
|
| 51 |
+
winning_model: str | None = "openai/openbmb/MiniCPM-8B",
|
| 52 |
+
winning_models: list[str] | None = None,
|
| 53 |
+
reason: str | None = "verdict",
|
| 54 |
+
turns: int = 5,
|
| 55 |
+
tokens: int = 200,
|
| 56 |
+
started_at: datetime | None = None,
|
| 57 |
+
finished_at: datetime | None = None,
|
| 58 |
+
) -> LeaderboardEntry:
|
| 59 |
+
"""Build a minimal LeaderboardEntry for testing."""
|
| 60 |
+
if started_at is None:
|
| 61 |
+
started_at = datetime(2025, 6, 14, 10, 0, 0, tzinfo=timezone.utc)
|
| 62 |
+
if finished_at is None:
|
| 63 |
+
finished_at = started_at
|
| 64 |
+
if cast is None:
|
| 65 |
+
cast = {"alice": CastBinding(model_endpoint=winning_model)}
|
| 66 |
+
if winning_models is None:
|
| 67 |
+
winning_models = [winning_model] if winning_model else []
|
| 68 |
+
return LeaderboardEntry(
|
| 69 |
+
run_id=run_id,
|
| 70 |
+
session_id=session_id,
|
| 71 |
+
scenario=scenario,
|
| 72 |
+
seed=seed,
|
| 73 |
+
competition_kind=competition_kind,
|
| 74 |
+
teams=teams,
|
| 75 |
+
symmetric_seats=symmetric_seats,
|
| 76 |
+
cast=cast,
|
| 77 |
+
winner=winner,
|
| 78 |
+
winner_kind=winner_kind,
|
| 79 |
+
winning_model=winning_model,
|
| 80 |
+
winning_models=winning_models,
|
| 81 |
+
reason=reason,
|
| 82 |
+
turns=turns,
|
| 83 |
+
tokens=tokens,
|
| 84 |
+
started_at=started_at,
|
| 85 |
+
finished_at=finished_at,
|
| 86 |
+
)
|
| 87 |
+
|
| 88 |
+
|
| 89 |
+
# ββ Tests: LeaderboardStore round-trip and ordering ββββββββββββββββββββββββββββββββ
|
| 90 |
+
|
| 91 |
+
|
| 92 |
+
class TestLeaderboardStoreRoundTrip:
|
| 93 |
+
"""Verify record β read-back: all fields survive serialization."""
|
| 94 |
+
|
| 95 |
+
def test_record_and_entries_round_trip(self):
|
| 96 |
+
"""Recording an entry and reading back returns all fields intact."""
|
| 97 |
+
store = make_leaderboard_store(url="sqlite://")
|
| 98 |
+
entry = _entry(
|
| 99 |
+
run_id="r1",
|
| 100 |
+
scenario="Debate Duel",
|
| 101 |
+
seed="abc123",
|
| 102 |
+
cast={
|
| 103 |
+
"alice": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B", model_profile="large"),
|
| 104 |
+
"bob": CastBinding(model_endpoint="google/gemma-12B", model_profile="medium"),
|
| 105 |
+
},
|
| 106 |
+
teams={"team_a": ["alice"], "team_b": ["bob"]},
|
| 107 |
+
symmetric_seats=["seat_a", "seat_b"],
|
| 108 |
+
winner="alice",
|
| 109 |
+
winner_kind="agent",
|
| 110 |
+
winning_model="openai/openbmb/MiniCPM-8B",
|
| 111 |
+
winning_models=["openai/openbmb/MiniCPM-8B"],
|
| 112 |
+
)
|
| 113 |
+
store.record(entry)
|
| 114 |
+
entries = store.entries()
|
| 115 |
+
assert len(entries) == 1
|
| 116 |
+
row = entries[0]
|
| 117 |
+
assert row.run_id == "r1"
|
| 118 |
+
assert row.scenario == "Debate Duel"
|
| 119 |
+
assert row.seed == "abc123"
|
| 120 |
+
assert row.winner == "alice"
|
| 121 |
+
assert row.winner_kind == "agent"
|
| 122 |
+
assert row.winning_model == "openai/openbmb/MiniCPM-8B"
|
| 123 |
+
assert row.winning_models == ["openai/openbmb/MiniCPM-8B"]
|
| 124 |
+
assert "alice" in row.cast
|
| 125 |
+
assert row.cast["alice"].model_endpoint == "openai/openbmb/MiniCPM-8B"
|
| 126 |
+
assert row.cast["alice"].model_profile == "large"
|
| 127 |
+
assert row.teams == {"team_a": ["alice"], "team_b": ["bob"]}
|
| 128 |
+
assert row.symmetric_seats == ["seat_a", "seat_b"]
|
| 129 |
+
|
| 130 |
+
def test_entries_for_scenario_filters_correctly(self):
|
| 131 |
+
"""entries_for_scenario returns only entries from that scenario."""
|
| 132 |
+
store = make_leaderboard_store(url="sqlite://")
|
| 133 |
+
store.record(_entry(run_id="r1", scenario="Debate Duel"))
|
| 134 |
+
store.record(_entry(run_id="r2", scenario="Trivia Night"))
|
| 135 |
+
store.record(_entry(run_id="r3", scenario="Debate Duel"))
|
| 136 |
+
debate_entries = store.entries_for_scenario("Debate Duel")
|
| 137 |
+
assert len(debate_entries) == 2
|
| 138 |
+
assert all(e.scenario == "Debate Duel" for e in debate_entries)
|
| 139 |
+
trivia_entries = store.entries_for_scenario("Trivia Night")
|
| 140 |
+
assert len(trivia_entries) == 1
|
| 141 |
+
assert trivia_entries[0].scenario == "Trivia Night"
|
| 142 |
+
|
| 143 |
+
def test_recorded_at_stamped_when_unset(self):
|
| 144 |
+
"""recorded_at is stamped to UTC now if unset (inside record)."""
|
| 145 |
+
store = make_leaderboard_store(url="sqlite://")
|
| 146 |
+
entry = _entry(run_id="r1")
|
| 147 |
+
# Entry has recorded_at=None (from _entry default)
|
| 148 |
+
assert entry.recorded_at is None
|
| 149 |
+
recorded = store.record(entry)
|
| 150 |
+
# After recording, recorded_at is stamped
|
| 151 |
+
assert recorded.recorded_at is not None
|
| 152 |
+
assert recorded.recorded_at.tzinfo is not None
|
| 153 |
+
# Read back should have the same timestamp
|
| 154 |
+
entries = store.entries()
|
| 155 |
+
assert entries[0].recorded_at is not None
|
| 156 |
+
assert entries[0].recorded_at.tzinfo is not None
|
| 157 |
+
|
| 158 |
+
|
| 159 |
+
# ββ Tests: Idempotent upsert on run_id ββββββββββββββββββββββββββββββββββββββββββββ
|
| 160 |
+
|
| 161 |
+
|
| 162 |
+
class TestLeaderboardStoreIdempotence:
|
| 163 |
+
"""Verify that recording the same run_id twice produces one row, and second replaces."""
|
| 164 |
+
|
| 165 |
+
def test_recording_same_run_id_twice_produces_one_row(self):
|
| 166 |
+
"""Recording the same run_id twice yields exactly one row (no duplicate)."""
|
| 167 |
+
store = make_leaderboard_store(url="sqlite://")
|
| 168 |
+
entry1 = _entry(run_id="r1", winner="alice", winning_model="ModelA")
|
| 169 |
+
store.record(entry1)
|
| 170 |
+
entry2 = _entry(run_id="r1", winner="bob", winning_model="ModelB") # same run_id, different winner
|
| 171 |
+
store.record(entry2)
|
| 172 |
+
entries = store.entries()
|
| 173 |
+
assert len(entries) == 1
|
| 174 |
+
assert entries[0].run_id == "r1"
|
| 175 |
+
|
| 176 |
+
def test_second_record_replaces_winner(self):
|
| 177 |
+
"""A second record with the same run_id replaces the row (e.g., verdict supersedes budget close)."""
|
| 178 |
+
store = make_leaderboard_store(url="sqlite://")
|
| 179 |
+
entry1 = _entry(run_id="r1", winner="alice", winning_model="ModelA", reason="budget")
|
| 180 |
+
store.record(entry1)
|
| 181 |
+
entry2 = _entry(run_id="r1", winner="bob", winning_model="ModelB", reason="verdict") # corrective
|
| 182 |
+
store.record(entry2)
|
| 183 |
+
entries = store.entries()
|
| 184 |
+
assert len(entries) == 1
|
| 185 |
+
assert entries[0].winner == "bob"
|
| 186 |
+
assert entries[0].winning_model == "ModelB"
|
| 187 |
+
assert entries[0].reason == "verdict"
|
| 188 |
+
|
| 189 |
+
def test_second_record_same_data_is_idempotent(self):
|
| 190 |
+
"""Recording the same entry twice is idempotent (no changes)."""
|
| 191 |
+
store = make_leaderboard_store(url="sqlite://")
|
| 192 |
+
entry = _entry(run_id="r1")
|
| 193 |
+
store.record(entry)
|
| 194 |
+
entries1 = store.entries()
|
| 195 |
+
store.record(entry)
|
| 196 |
+
entries2 = store.entries()
|
| 197 |
+
assert len(entries1) == len(entries2) == 1
|
| 198 |
+
assert entries1[0].run_id == entries2[0].run_id
|
| 199 |
+
|
| 200 |
+
|
| 201 |
+
# ββ Tests: Ordering (newest finished_at first, run_id tiebreak) βββββββββββββββββββ
|
| 202 |
+
|
| 203 |
+
|
| 204 |
+
class TestLeaderboardStoreOrdering:
|
| 205 |
+
"""Verify .entries() and .entries_for_scenario() ordering."""
|
| 206 |
+
|
| 207 |
+
def test_entries_newest_first(self):
|
| 208 |
+
"""entries() returns newest finished_at first."""
|
| 209 |
+
store = make_leaderboard_store(url="sqlite://")
|
| 210 |
+
old = datetime(2025, 6, 1, 10, 0, 0, tzinfo=timezone.utc)
|
| 211 |
+
new = datetime(2025, 6, 14, 10, 0, 0, tzinfo=timezone.utc)
|
| 212 |
+
store.record(_entry(run_id="r1", finished_at=old))
|
| 213 |
+
store.record(_entry(run_id="r2", finished_at=new))
|
| 214 |
+
entries = store.entries()
|
| 215 |
+
assert len(entries) == 2
|
| 216 |
+
assert entries[0].run_id == "r2" # newer
|
| 217 |
+
assert entries[1].run_id == "r1" # older
|
| 218 |
+
|
| 219 |
+
def test_entries_run_id_tiebreak_when_finished_at_same(self):
|
| 220 |
+
"""When finished_at is equal, run_id breaks ties (ascending lexicographically)."""
|
| 221 |
+
store = make_leaderboard_store(url="sqlite://")
|
| 222 |
+
same_time = datetime(2025, 6, 14, 10, 0, 0, tzinfo=timezone.utc)
|
| 223 |
+
store.record(_entry(run_id="r2", finished_at=same_time))
|
| 224 |
+
store.record(_entry(run_id="r1", finished_at=same_time))
|
| 225 |
+
entries = store.entries()
|
| 226 |
+
assert len(entries) == 2
|
| 227 |
+
# Both have same finished_at; run_id breaks ties in ascending order (r1 < r2)
|
| 228 |
+
assert entries[0].run_id == "r1"
|
| 229 |
+
assert entries[1].run_id == "r2"
|
| 230 |
+
|
| 231 |
+
def test_entries_for_scenario_ordering(self):
|
| 232 |
+
"""entries_for_scenario also respects newest-first ordering."""
|
| 233 |
+
store = make_leaderboard_store(url="sqlite://")
|
| 234 |
+
old = datetime(2025, 6, 1, 10, 0, 0, tzinfo=timezone.utc)
|
| 235 |
+
new = datetime(2025, 6, 14, 10, 0, 0, tzinfo=timezone.utc)
|
| 236 |
+
store.record(_entry(run_id="r1", scenario="Debate Duel", finished_at=old))
|
| 237 |
+
store.record(_entry(run_id="r2", scenario="Debate Duel", finished_at=new))
|
| 238 |
+
store.record(_entry(run_id="r3", scenario="Trivia Night", finished_at=new))
|
| 239 |
+
debate = store.entries_for_scenario("Debate Duel")
|
| 240 |
+
assert len(debate) == 2
|
| 241 |
+
assert debate[0].run_id == "r2" # newer
|
| 242 |
+
assert debate[1].run_id == "r1" # older
|
| 243 |
+
|
| 244 |
+
|
| 245 |
+
# ββ Tests: build_entry gating (operator's requirement) ββββββββββββββββββββββββββββββ
|
| 246 |
+
|
| 247 |
+
|
| 248 |
+
class TestBuildEntryGating:
|
| 249 |
+
"""Verify build_entry returns None unless finished + winner + winning_model + competitive."""
|
| 250 |
+
|
| 251 |
+
def test_build_entry_returns_none_when_not_finished(self):
|
| 252 |
+
"""build_entry returns None when finished_at is None."""
|
| 253 |
+
summary = RunSummary(
|
| 254 |
+
run_id="r1",
|
| 255 |
+
scenario="Debate Duel",
|
| 256 |
+
winner="alice",
|
| 257 |
+
winning_model="ModelA",
|
| 258 |
+
finished_at=None,
|
| 259 |
+
)
|
| 260 |
+
|
| 261 |
+
class MockCompetition:
|
| 262 |
+
kind = "versus"
|
| 263 |
+
teams = None
|
| 264 |
+
symmetric_seats = None
|
| 265 |
+
|
| 266 |
+
result = build_entry(summary, MockCompetition())
|
| 267 |
+
assert result is None
|
| 268 |
+
|
| 269 |
+
def test_build_entry_returns_none_when_no_winner(self):
|
| 270 |
+
"""build_entry returns None when winner is None or empty."""
|
| 271 |
+
summary = RunSummary(
|
| 272 |
+
run_id="r1",
|
| 273 |
+
scenario="Debate Duel",
|
| 274 |
+
winner=None,
|
| 275 |
+
winning_model="ModelA",
|
| 276 |
+
finished_at=datetime(2025, 6, 14, 10, 0, 0, tzinfo=timezone.utc),
|
| 277 |
+
)
|
| 278 |
+
|
| 279 |
+
class MockCompetition:
|
| 280 |
+
kind = "versus"
|
| 281 |
+
teams = None
|
| 282 |
+
symmetric_seats = None
|
| 283 |
+
|
| 284 |
+
result = build_entry(summary, MockCompetition())
|
| 285 |
+
assert result is None
|
| 286 |
+
|
| 287 |
+
def test_build_entry_returns_none_when_no_winning_model(self):
|
| 288 |
+
"""build_entry returns None when neither winning_model nor winning_models has a value."""
|
| 289 |
+
summary = RunSummary(
|
| 290 |
+
run_id="r1",
|
| 291 |
+
scenario="Debate Duel",
|
| 292 |
+
winner="alice",
|
| 293 |
+
winning_model=None,
|
| 294 |
+
winning_models=[],
|
| 295 |
+
finished_at=datetime(2025, 6, 14, 10, 0, 0, tzinfo=timezone.utc),
|
| 296 |
+
)
|
| 297 |
+
|
| 298 |
+
class MockCompetition:
|
| 299 |
+
kind = "versus"
|
| 300 |
+
teams = None
|
| 301 |
+
symmetric_seats = None
|
| 302 |
+
|
| 303 |
+
result = build_entry(summary, MockCompetition())
|
| 304 |
+
assert result is None
|
| 305 |
+
|
| 306 |
+
def test_build_entry_returns_none_when_competition_kind_is_none(self):
|
| 307 |
+
"""build_entry returns None when competition.kind == 'none' (non-competitive)."""
|
| 308 |
+
summary = RunSummary(
|
| 309 |
+
run_id="r1",
|
| 310 |
+
scenario="Exploratory Run",
|
| 311 |
+
winner="alice",
|
| 312 |
+
winning_model="ModelA",
|
| 313 |
+
finished_at=datetime(2025, 6, 14, 10, 0, 0, tzinfo=timezone.utc),
|
| 314 |
+
)
|
| 315 |
+
|
| 316 |
+
class MockCompetition:
|
| 317 |
+
kind = "none"
|
| 318 |
+
teams = None
|
| 319 |
+
symmetric_seats = None
|
| 320 |
+
|
| 321 |
+
result = build_entry(summary, MockCompetition())
|
| 322 |
+
assert result is None
|
| 323 |
+
|
| 324 |
+
def test_build_entry_returns_entry_when_all_gates_pass(self):
|
| 325 |
+
"""build_entry returns a populated entry when finished + winner + winning_model + competitive."""
|
| 326 |
+
summary = RunSummary(
|
| 327 |
+
run_id="r1",
|
| 328 |
+
scenario="Debate Duel",
|
| 329 |
+
seed="seed123",
|
| 330 |
+
winner="alice",
|
| 331 |
+
winner_kind="agent",
|
| 332 |
+
winning_model="openai/openbmb/MiniCPM-8B",
|
| 333 |
+
winning_models=[],
|
| 334 |
+
finished_at=datetime(2025, 6, 14, 10, 0, 0, tzinfo=timezone.utc),
|
| 335 |
+
started_at=datetime(2025, 6, 14, 9, 55, 0, tzinfo=timezone.utc),
|
| 336 |
+
cast={"alice": CastBinding(model_endpoint="openai/openbmb/MiniCPM-8B")},
|
| 337 |
+
turns=5,
|
| 338 |
+
tokens=200,
|
| 339 |
+
)
|
| 340 |
+
|
| 341 |
+
class MockCompetition:
|
| 342 |
+
kind = "versus"
|
| 343 |
+
teams = None
|
| 344 |
+
symmetric_seats = ["debater_a"]
|
| 345 |
+
|
| 346 |
+
entry = build_entry(summary, MockCompetition())
|
| 347 |
+
assert entry is not None
|
| 348 |
+
assert entry.run_id == "r1"
|
| 349 |
+
assert entry.scenario == "Debate Duel"
|
| 350 |
+
assert entry.winner == "alice"
|
| 351 |
+
assert entry.winning_model == "openai/openbmb/MiniCPM-8B"
|
| 352 |
+
assert entry.competition_kind == "versus"
|
| 353 |
+
|
| 354 |
+
def test_build_entry_merges_winning_model_into_winning_models(self):
|
| 355 |
+
"""build_entry includes both winning_model and winning_models in the merged list."""
|
| 356 |
+
summary = RunSummary(
|
| 357 |
+
run_id="r1",
|
| 358 |
+
scenario="Debate Duel",
|
| 359 |
+
winner="alice",
|
| 360 |
+
winning_model="ModelA",
|
| 361 |
+
winning_models=["ModelB"],
|
| 362 |
+
finished_at=datetime(2025, 6, 14, 10, 0, 0, tzinfo=timezone.utc),
|
| 363 |
+
)
|
| 364 |
+
|
| 365 |
+
class MockCompetition:
|
| 366 |
+
kind = "versus"
|
| 367 |
+
teams = None
|
| 368 |
+
symmetric_seats = None
|
| 369 |
+
|
| 370 |
+
entry = build_entry(summary, MockCompetition())
|
| 371 |
+
assert entry is not None
|
| 372 |
+
assert entry.winning_model == "ModelA"
|
| 373 |
+
assert set(entry.winning_models) == {"ModelA", "ModelB"}
|
| 374 |
+
|
| 375 |
+
|
| 376 |
+
# ββ Tests: Separate-table isolation (leaderboard_entries β events) ββββββββββββββββ
|
| 377 |
+
|
| 378 |
+
|
| 379 |
+
class TestLeaderboardStoreTableIsolation:
|
| 380 |
+
"""Verify leaderboard_entries and events tables are independent."""
|
| 381 |
+
|
| 382 |
+
def test_leaderboard_store_creates_leaderboard_entries_table(self):
|
| 383 |
+
"""LeaderboardStore creates a dedicated leaderboard_entries table, not events."""
|
| 384 |
+
store = make_leaderboard_store(url="sqlite://")
|
| 385 |
+
entry = _entry(run_id="r1")
|
| 386 |
+
store.record(entry)
|
| 387 |
+
# Query the store's table to ensure it's leaderboard_entries
|
| 388 |
+
entries = store.entries()
|
| 389 |
+
assert len(entries) == 1
|
| 390 |
+
assert entries[0].run_id == "r1"
|
| 391 |
+
|
| 392 |
+
def test_multiple_databases_do_not_share_store(self):
|
| 393 |
+
"""Two stores on different SQLite files have independent data."""
|
| 394 |
+
store1 = LeaderboardStore(url="sqlite:///:memory:")
|
| 395 |
+
store2 = LeaderboardStore(url="sqlite:///:memory:") # different in-memory DB
|
| 396 |
+
store1.record(_entry(run_id="r1"))
|
| 397 |
+
entries1 = store1.entries()
|
| 398 |
+
entries2 = store2.entries()
|
| 399 |
+
assert len(entries1) == 1
|
| 400 |
+
assert len(entries2) == 0 # store2 was never written to
|
| 401 |
+
|
| 402 |
+
|
| 403 |
+
# ββ Tests: E2E write path (FishbowlSession β finalize β store) ββββββββββββββββββββ
|
| 404 |
+
|
| 405 |
+
|
| 406 |
+
class TestLeaderboardE2E:
|
| 407 |
+
"""End-to-end: drive a FishbowlSession, finalize, assert leaderboard row recorded."""
|
| 408 |
+
|
| 409 |
+
def test_e2e_write_path_via_build_entry(self):
|
| 410 |
+
"""Test the write path: build_entry β store.record β leaderboard_entries table."""
|
| 411 |
+
from src.core.leaderboard_store import build_entry
|
| 412 |
+
|
| 413 |
+
store = make_leaderboard_store(url="sqlite://")
|
| 414 |
+
|
| 415 |
+
# Simulate a finished competitive run with a winner + winning_models
|
| 416 |
+
summary = RunSummary(
|
| 417 |
+
run_id="e2e-r1",
|
| 418 |
+
scenario="twenty-sprouts",
|
| 419 |
+
seed="test_seed_e2e",
|
| 420 |
+
winner="keeper", # Team winner
|
| 421 |
+
winner_kind="team",
|
| 422 |
+
winning_model=None,
|
| 423 |
+
winning_models=["stub:balanced", "stub:fast"],
|
| 424 |
+
finished_at=datetime(2025, 6, 14, 10, 5, 0, tzinfo=timezone.utc),
|
| 425 |
+
started_at=datetime(2025, 6, 14, 10, 0, 0, tzinfo=timezone.utc),
|
| 426 |
+
cast={
|
| 427 |
+
"secret-keeper": CastBinding(model_endpoint="stub:balanced", model_profile="balanced"),
|
| 428 |
+
"sprout-guesser": CastBinding(model_endpoint="stub:fast", model_profile="fast"),
|
| 429 |
+
},
|
| 430 |
+
turns=10,
|
| 431 |
+
tokens=5000,
|
| 432 |
+
)
|
| 433 |
+
|
| 434 |
+
# Mock competition config
|
| 435 |
+
class MockCompetition:
|
| 436 |
+
kind = "versus"
|
| 437 |
+
teams = {"guesser": ["sprout-guesser"], "keeper": ["secret-keeper"]}
|
| 438 |
+
symmetric_seats = None
|
| 439 |
+
|
| 440 |
+
# Build and record the entry
|
| 441 |
+
entry = build_entry(summary, MockCompetition())
|
| 442 |
+
assert entry is not None, "Entry should pass the eligibility gate"
|
| 443 |
+
|
| 444 |
+
recorded = store.record(entry)
|
| 445 |
+
assert recorded.run_id == "e2e-r1"
|
| 446 |
+
|
| 447 |
+
# Read back from store
|
| 448 |
+
entries = store.entries()
|
| 449 |
+
assert len(entries) == 1
|
| 450 |
+
row = entries[0]
|
| 451 |
+
assert row.run_id == "e2e-r1"
|
| 452 |
+
assert row.scenario == "twenty-sprouts"
|
| 453 |
+
assert row.winner == "keeper"
|
| 454 |
+
assert row.winner_kind == "team"
|
| 455 |
+
assert row.winning_models == ["stub:balanced", "stub:fast"]
|
| 456 |
+
assert row.competition_kind == "versus"
|
| 457 |
+
assert row.finished_at is not None
|
| 458 |
+
|
| 459 |
+
def test_e2e_non_competitive_scenario_not_recorded(self):
|
| 460 |
+
"""Drive a non-competitive scenario (if any) and verify no row is recorded."""
|
| 461 |
+
from src.ui.fishbowl.session import FishbowlSession
|
| 462 |
+
|
| 463 |
+
# "thousand-token-wood" is non-competitive (kind="none")
|
| 464 |
+
store = make_leaderboard_store(url="sqlite://")
|
| 465 |
+
session = FishbowlSession("thousand-token-wood")
|
| 466 |
+
session._leaderboard = store
|
| 467 |
+
|
| 468 |
+
session.reset(seed="test_seed_456")
|
| 469 |
+
session.step(5)
|
| 470 |
+
|
| 471 |
+
# Finalize without forcing verdict (or with a budget close)
|
| 472 |
+
session.finalize("budget")
|
| 473 |
+
|
| 474 |
+
# No row should be recorded for a non-competitive run
|
| 475 |
+
entries = store.entries()
|
| 476 |
+
assert len(entries) == 0
|
| 477 |
+
|
| 478 |
+
def test_e2e_abandoned_run_not_recorded(self):
|
| 479 |
+
"""Drive a run, finalize with no winner, verify no row recorded."""
|
| 480 |
+
from src.ui.fishbowl.session import FishbowlSession
|
| 481 |
+
|
| 482 |
+
store = make_leaderboard_store(url="sqlite://")
|
| 483 |
+
session = FishbowlSession("twenty-sprouts")
|
| 484 |
+
session._leaderboard = store
|
| 485 |
+
|
| 486 |
+
session.reset(seed="test_seed_789")
|
| 487 |
+
session.step(2)
|
| 488 |
+
|
| 489 |
+
# Finalize without a verdict (user stops, no winner)
|
| 490 |
+
session.finalize("user_stop")
|
| 491 |
+
|
| 492 |
+
# No row should be recorded
|
| 493 |
+
entries = store.entries()
|
| 494 |
+
assert len(entries) == 0
|
| 495 |
+
|
| 496 |
+
def test_e2e_competitive_verdict_records_row(self):
|
| 497 |
+
"""Positive E2E: a real FishbowlSession reaching a verdict writes exactly one row.
|
| 498 |
+
|
| 499 |
+
Drives "twenty-sprouts" (a versus, code-decided ground-truth game) offline with the
|
| 500 |
+
deterministic stub. Each seat is bound to an explicit ``model_endpoint`` (what the
|
| 501 |
+
Lab does via ``cast_models``) so the run.started cast map carries endpoints and the
|
| 502 |
+
"winning model selected" gate is satisfied β without that, the run is finished and
|
| 503 |
+
won but has no concrete winning model, so no row is recorded (see
|
| 504 |
+
``test_e2e_abandoned_run_not_recorded`` for the no-winner case). Exercises the full
|
| 505 |
+
``finalize β _record_leaderboard β build_entry β store.record`` glue.
|
| 506 |
+
"""
|
| 507 |
+
from src.ui.fishbowl.session import FishbowlSession
|
| 508 |
+
|
| 509 |
+
store = make_leaderboard_store(url="sqlite://")
|
| 510 |
+
session = FishbowlSession("twenty-sprouts")
|
| 511 |
+
session._leaderboard = store
|
| 512 |
+
# Bind a distinct endpoint to every seat (the Lab's cast_models, in miniature).
|
| 513 |
+
for agent in session.conductor.scenario.agents:
|
| 514 |
+
agent.manifest = agent.manifest.model_copy(update={"model_endpoint": f"openai/test/{agent.manifest.name}"})
|
| 515 |
+
|
| 516 |
+
session.reset(seed="seed-e2e")
|
| 517 |
+
for _ in range(12):
|
| 518 |
+
session.step(1)
|
| 519 |
+
ruled = session.force_verdict() # lands judge.verdict + finalize("verdict")
|
| 520 |
+
assert ruled is True
|
| 521 |
+
|
| 522 |
+
rows = [r for r in store.entries() if r.run_id == session.conductor.run_id]
|
| 523 |
+
assert len(rows) == 1
|
| 524 |
+
row = rows[0]
|
| 525 |
+
assert row.scenario == "twenty-sprouts"
|
| 526 |
+
assert row.competition_kind == "versus"
|
| 527 |
+
assert row.reason == "verdict"
|
| 528 |
+
assert row.winner # a real seat was crowned by the code-decided judge
|
| 529 |
+
# The winner's bound endpoint is credited as the winning model.
|
| 530 |
+
assert row.winning_models == [f"openai/test/{row.winner}"]
|
| 531 |
+
assert row.winning_model == f"openai/test/{row.winner}"
|