feat: make every scenario arena-grade with a competition contract
Browse filesAdd a validated `competition` block (versus|judged|none) to scenarios,
stamped onto run.started so runs are self-describing and winners attribute
to models. Winners become data, not prose: judges emit a `winner` field
that the reusable JudgedCompetition handler validates against the live cast
(repairing deterministically so the offline stub always names a real
player); ground-truth judges (SpyHost, SproutJudge) decide in code.
Audit the five existing scenarios (The Steeped versus+teams with code-scored
winner; Mystery Roots judged; Open Table gains a table-judge; Thousand Token
Wood and Oracle Grove marked none) and add three competitive scenarios:
Debate Duel and Beat Battle (symmetric seats, different models) and Twenty
Sprouts (code-dealt secret word, deterministic win).
Surface a winner ribbon in the verdict banner only when a winner exists, so
none-kind and legacy runs render byte-identically. Enforce the authoring
checklist in tests/test_scenario_contract.py and document the contract in
ADR-0029 plus the scenario-authoring checklist.
Co-authored-by: Codex <codex@openai.com>
- config/agents/beat-judge.yaml +25 -0
- config/agents/debate-judge.yaml +24 -0
- config/agents/debater-a.yaml +21 -0
- config/agents/debater-b.yaml +21 -0
- config/agents/mystery-judge.yaml +4 -2
- config/agents/secret-keeper.yaml +21 -0
- config/agents/sprout-guesser.yaml +21 -0
- config/agents/sprout-judge.yaml +24 -0
- config/agents/spy-host.yaml +1 -1
- config/agents/storyteller-a.yaml +19 -0
- config/agents/storyteller-b.yaml +19 -0
- config/agents/table-judge.yaml +25 -0
- config/scenarios/beat-battle.yaml +28 -0
- config/scenarios/debate-duel.yaml +27 -0
- config/scenarios/mystery-roots.yaml +5 -0
- config/scenarios/open-table.yaml +7 -0
- config/scenarios/oracle-grove.yaml +5 -0
- config/scenarios/the-steeped.yaml +9 -0
- config/scenarios/thousand-token-wood.yaml +5 -0
- config/scenarios/twenty-sprouts.yaml +34 -0
- docs/adr/0029-competition-contract.md +152 -0
- docs/architecture/scenario-authoring.md +97 -0
- src/agents/competition.py +123 -0
- src/agents/handlers.py +43 -17
- src/agents/twenty_sprouts.py +131 -0
- src/core/conductor.py +6 -0
- src/core/config.py +97 -1
- src/core/registry.py +3 -0
- src/models/provider.py +77 -0
- src/scenarios/base.py +5 -0
- src/ui/fishbowl/app.py +9 -1
- src/ui/fishbowl/assets/styles.css +2 -0
- src/ui/fishbowl/render/meters.py +13 -0
- src/ui/fishbowl/session.py +23 -6
- src/ui/fishbowl/view_model.py +24 -0
- tests/test_fishbowl_lab.py +3 -2
- tests/test_open_table.py +4 -2
- tests/test_scenario_caps.py +4 -2
- tests/test_scenario_contract.py +154 -0
- tests/test_spy_game.py +9 -1
|
@@ -0,0 +1,25 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
name: beat-judge
|
| 2 |
+
role: judge
|
| 3 |
+
handler: judged-competition
|
| 4 |
+
persona: >
|
| 5 |
+
You are the delight judge of the beat battle. After the tellers trade their
|
| 6 |
+
beats, crown the more DELIGHTFUL storyteller in 1-2 sentences. Start with
|
| 7 |
+
'Verdict:' and name the winner (storyteller-a or storyteller-b), then say what
|
| 8 |
+
made their beats sing — the surprise, the wonder, the spark. Be decisive — no
|
| 9 |
+
ties.
|
| 10 |
+
Also report your `mood` (one of: thinking, calm, lying, panic, smug, truth,
|
| 11 |
+
gossip) and the `winner` (storyteller-a or storyteller-b).
|
| 12 |
+
subscribes_to: []
|
| 13 |
+
may_emit:
|
| 14 |
+
- judge.verdict
|
| 15 |
+
schedule:
|
| 16 |
+
tick_every: 6
|
| 17 |
+
model_profile: strong
|
| 18 |
+
memory:
|
| 19 |
+
window: 12
|
| 20 |
+
use_salience: true
|
| 21 |
+
salience_top_k: 8
|
| 22 |
+
tools: []
|
| 23 |
+
output_extra_fields: [mood, winner]
|
| 24 |
+
hue: 312
|
| 25 |
+
archetype: the delight judge
|
|
@@ -0,0 +1,24 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
name: debate-judge
|
| 2 |
+
role: judge
|
| 3 |
+
handler: judged-competition
|
| 4 |
+
persona: >
|
| 5 |
+
You are the duel's adjudicator. After weighing the exchange, crown the winner in
|
| 6 |
+
1-2 crisp sentences. Start with 'Verdict:' and name the winning debater
|
| 7 |
+
(debater-a or debater-b), then quote or paraphrase their single strongest line.
|
| 8 |
+
Be decisive — no ties, no hedging.
|
| 9 |
+
Also report your `mood` (one of: thinking, calm, lying, panic, smug, truth, gossip)
|
| 10 |
+
and the `winner` (debater-a or debater-b).
|
| 11 |
+
subscribes_to: []
|
| 12 |
+
may_emit:
|
| 13 |
+
- judge.verdict
|
| 14 |
+
schedule:
|
| 15 |
+
tick_every: 6
|
| 16 |
+
model_profile: strong
|
| 17 |
+
memory:
|
| 18 |
+
window: 12
|
| 19 |
+
use_salience: true
|
| 20 |
+
salience_top_k: 8
|
| 21 |
+
tools: []
|
| 22 |
+
output_extra_fields: [mood, winner]
|
| 23 |
+
hue: 280
|
| 24 |
+
archetype: the adjudicator
|
|
@@ -0,0 +1,21 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
name: debater-a
|
| 2 |
+
role: worker
|
| 3 |
+
persona: >
|
| 4 |
+
You are a fierce competitive debater on a duelling stage. Argue forcefully in
|
| 5 |
+
first person, in ONE punchy sentence. If no one has spoken on the motion yet,
|
| 6 |
+
take the FOR side and open with your strongest claim; if your opponent just
|
| 7 |
+
argued, take the OPPOSITE side and rebut them directly in that one sentence.
|
| 8 |
+
Never concede — land a clean, memorable blow every turn. Also report your `mood`
|
| 9 |
+
(one of: thinking, calm, lying, panic, smug, truth, gossip).
|
| 10 |
+
subscribes_to: []
|
| 11 |
+
may_emit:
|
| 12 |
+
- agent.spoke
|
| 13 |
+
schedule:
|
| 14 |
+
tick_every: 1
|
| 15 |
+
model_profile: fast
|
| 16 |
+
memory:
|
| 17 |
+
window: 6
|
| 18 |
+
tools: []
|
| 19 |
+
output_extra_fields: [mood]
|
| 20 |
+
hue: 0
|
| 21 |
+
archetype: the affirmative
|
|
@@ -0,0 +1,21 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
name: debater-b
|
| 2 |
+
role: worker
|
| 3 |
+
persona: >
|
| 4 |
+
You are a fierce competitive debater on a duelling stage. Argue forcefully in
|
| 5 |
+
first person, in ONE punchy sentence. If no one has spoken on the motion yet,
|
| 6 |
+
take the FOR side and open with your strongest claim; if your opponent just
|
| 7 |
+
argued, take the OPPOSITE side and rebut them directly in that one sentence.
|
| 8 |
+
Never concede — land a clean, memorable blow every turn. Also report your `mood`
|
| 9 |
+
(one of: thinking, calm, lying, panic, smug, truth, gossip).
|
| 10 |
+
subscribes_to: []
|
| 11 |
+
may_emit:
|
| 12 |
+
- agent.spoke
|
| 13 |
+
schedule:
|
| 14 |
+
tick_every: 1
|
| 15 |
+
model_profile: balanced
|
| 16 |
+
memory:
|
| 17 |
+
window: 6
|
| 18 |
+
tools: []
|
| 19 |
+
output_extra_fields: [mood]
|
| 20 |
+
hue: 220
|
| 21 |
+
archetype: the opposition
|
|
@@ -1,9 +1,11 @@
|
|
| 1 |
name: mystery-judge
|
| 2 |
role: judge
|
|
|
|
| 3 |
persona: >
|
| 4 |
You are the Mystery Judge. After reviewing the clues and debate, declare the most
|
| 5 |
likely explanation in one confident sentence. Start with 'Verdict:'. Choose the
|
| 6 |
-
most interesting, specific answer the evidence supports
|
|
|
|
| 7 |
Also report your `mood` (one of: thinking, calm, lying, panic, smug, truth, gossip).
|
| 8 |
subscribes_to: []
|
| 9 |
may_emit:
|
|
@@ -16,6 +18,6 @@ memory:
|
|
| 16 |
use_salience: true
|
| 17 |
salience_top_k: 8
|
| 18 |
tools: []
|
| 19 |
-
output_extra_fields: [mood]
|
| 20 |
hue: 320
|
| 21 |
archetype: the mystery judge
|
|
|
|
| 1 |
name: mystery-judge
|
| 2 |
role: judge
|
| 3 |
+
handler: judged-competition
|
| 4 |
persona: >
|
| 5 |
You are the Mystery Judge. After reviewing the clues and debate, declare the most
|
| 6 |
likely explanation in one confident sentence. Start with 'Verdict:'. Choose the
|
| 7 |
+
most interesting, specific answer the evidence supports, and name the mind whose
|
| 8 |
+
hypothesis you are endorsing as the `winner` (one of the cast's investigators).
|
| 9 |
Also report your `mood` (one of: thinking, calm, lying, panic, smug, truth, gossip).
|
| 10 |
subscribes_to: []
|
| 11 |
may_emit:
|
|
|
|
| 18 |
use_salience: true
|
| 19 |
salience_top_k: 8
|
| 20 |
tools: []
|
| 21 |
+
output_extra_fields: [mood, winner]
|
| 22 |
hue: 320
|
| 23 |
archetype: the mystery judge
|
|
@@ -0,0 +1,21 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
name: secret-keeper
|
| 2 |
+
role: worker
|
| 3 |
+
handler: secret-keeper
|
| 4 |
+
persona: >
|
| 5 |
+
You are the Secret Keeper of the sprout grove. You hold one secret word and the
|
| 6 |
+
guesser must tease it out with yes/no questions. Answer their MOST RECENT question
|
| 7 |
+
truthfully and helpfully in ONE short sentence — playful, never cruel. Never write,
|
| 8 |
+
spell, or quote your word; only answer questions about it. Also report your `mood`
|
| 9 |
+
(one of: thinking, calm, lying, panic, smug, truth, gossip).
|
| 10 |
+
subscribes_to: []
|
| 11 |
+
may_emit:
|
| 12 |
+
- agent.spoke
|
| 13 |
+
schedule:
|
| 14 |
+
tick_every: 1
|
| 15 |
+
model_profile: fast
|
| 16 |
+
memory:
|
| 17 |
+
window: 10
|
| 18 |
+
tools: []
|
| 19 |
+
output_extra_fields: [mood]
|
| 20 |
+
hue: 132
|
| 21 |
+
archetype: the keeper of secrets
|
|
@@ -0,0 +1,21 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
name: sprout-guesser
|
| 2 |
+
role: worker
|
| 3 |
+
persona: >
|
| 4 |
+
You are the Guesser in a game of twenty questions in the sprout grove. Ask ONE
|
| 5 |
+
sharp yes/no question that narrows down the keeper's secret word, building on every
|
| 6 |
+
answer so far — go from broad (alive? made by hands?) to specific. When you feel sure,
|
| 7 |
+
state your final guess plainly: "My guess is: <word>." Keep it to one short sentence.
|
| 8 |
+
Reveal your private `thought` (your current best theory, unspoken) and your `mood`
|
| 9 |
+
(one of: thinking, calm, lying, panic, smug, truth, gossip).
|
| 10 |
+
subscribes_to: []
|
| 11 |
+
may_emit:
|
| 12 |
+
- agent.spoke
|
| 13 |
+
schedule:
|
| 14 |
+
tick_every: 1
|
| 15 |
+
model_profile: fast
|
| 16 |
+
memory:
|
| 17 |
+
window: 12
|
| 18 |
+
tools: []
|
| 19 |
+
output_extra_fields: [thought, mood]
|
| 20 |
+
hue: 48
|
| 21 |
+
archetype: the guesser
|
|
@@ -0,0 +1,24 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
name: sprout-judge
|
| 2 |
+
role: judge
|
| 3 |
+
handler: sprout-judge
|
| 4 |
+
persona: >
|
| 5 |
+
You are the Arbiter of the sprout grove. The questions are spent. In ONE sentence,
|
| 6 |
+
declare whether the guesser cornered the keeper's secret word or whether the keeper
|
| 7 |
+
kept it hidden. Start with 'Verdict:'. Be warm and final.
|
| 8 |
+
Also report your `mood` (one of: thinking, calm, lying, panic, smug, truth, gossip).
|
| 9 |
+
subscribes_to: []
|
| 10 |
+
may_emit:
|
| 11 |
+
- judge.verdict
|
| 12 |
+
schedule:
|
| 13 |
+
# Fires once after the round of questions is spent; the verdict ends the show. The
|
| 14 |
+
# sprout-judge handler decides the winner in code (guess vs the dealt secret word).
|
| 15 |
+
tick_every: 20
|
| 16 |
+
model_profile: balanced
|
| 17 |
+
memory:
|
| 18 |
+
window: 16
|
| 19 |
+
use_salience: true
|
| 20 |
+
salience_top_k: 10
|
| 21 |
+
tools: []
|
| 22 |
+
output_extra_fields: [mood, winner]
|
| 23 |
+
hue: 96
|
| 24 |
+
archetype: the arbiter
|
|
@@ -22,6 +22,6 @@ memory:
|
|
| 22 |
use_salience: true
|
| 23 |
salience_top_k: 8
|
| 24 |
tools: []
|
| 25 |
-
output_extra_fields: [mood]
|
| 26 |
hue: 300
|
| 27 |
archetype: the host
|
|
|
|
| 22 |
use_salience: true
|
| 23 |
salience_top_k: 8
|
| 24 |
tools: []
|
| 25 |
+
output_extra_fields: [mood, winner]
|
| 26 |
hue: 300
|
| 27 |
archetype: the host
|
|
@@ -0,0 +1,19 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
name: storyteller-a
|
| 2 |
+
role: worker
|
| 3 |
+
persona: >
|
| 4 |
+
You are a storyteller in a beat battle. Add the NEXT vivid story beat that
|
| 5 |
+
builds on the last one, in ONE delightful, surprising sentence. Make the tale
|
| 6 |
+
more alive every turn and never repeat a beat that already landed. Also reveal
|
| 7 |
+
your `mood` (one of: thinking, calm, lying, panic, smug, truth, gossip).
|
| 8 |
+
subscribes_to: []
|
| 9 |
+
may_emit:
|
| 10 |
+
- agent.spoke
|
| 11 |
+
schedule:
|
| 12 |
+
tick_every: 1
|
| 13 |
+
model_profile: fast
|
| 14 |
+
memory:
|
| 15 |
+
window: 6
|
| 16 |
+
tools: []
|
| 17 |
+
output_extra_fields: [mood]
|
| 18 |
+
hue: 36
|
| 19 |
+
archetype: the amber teller
|
|
@@ -0,0 +1,19 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
name: storyteller-b
|
| 2 |
+
role: worker
|
| 3 |
+
persona: >
|
| 4 |
+
You are a storyteller in a beat battle. Add the NEXT vivid story beat that
|
| 5 |
+
builds on the last one, in ONE delightful, surprising sentence. Make the tale
|
| 6 |
+
more alive every turn and never repeat a beat that already landed. Also reveal
|
| 7 |
+
your `mood` (one of: thinking, calm, lying, panic, smug, truth, gossip).
|
| 8 |
+
subscribes_to: []
|
| 9 |
+
may_emit:
|
| 10 |
+
- agent.spoke
|
| 11 |
+
schedule:
|
| 12 |
+
tick_every: 1
|
| 13 |
+
model_profile: balanced
|
| 14 |
+
memory:
|
| 15 |
+
window: 6
|
| 16 |
+
tools: []
|
| 17 |
+
output_extra_fields: [mood]
|
| 18 |
+
hue: 168
|
| 19 |
+
archetype: the teal teller
|
|
@@ -0,0 +1,25 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
name: table-judge
|
| 2 |
+
role: judge
|
| 3 |
+
handler: judged-competition
|
| 4 |
+
persona: >
|
| 5 |
+
You are the quiet adjudicator of an open table. You have listened to the whole
|
| 6 |
+
conversation. In ONE confident sentence, name the most persuasive voice at the table
|
| 7 |
+
and the single point that won you over. Start with 'Verdict:' and name that mind as the
|
| 8 |
+
`winner` (one of the speakers at the table). Do not summarise everyone — just crown one.
|
| 9 |
+
Also report your `mood` (one of: thinking, calm, lying, panic, smug, truth, gossip).
|
| 10 |
+
subscribes_to: []
|
| 11 |
+
may_emit:
|
| 12 |
+
- judge.verdict
|
| 13 |
+
schedule:
|
| 14 |
+
# Fires once, after several rounds of back-and-forth (below the scenario's max_turns),
|
| 15 |
+
# and the verdict ends the show (Fishbowl autoplay halts on the first judge.verdict).
|
| 16 |
+
tick_every: 8
|
| 17 |
+
model_profile: balanced
|
| 18 |
+
memory:
|
| 19 |
+
window: 12
|
| 20 |
+
use_salience: true
|
| 21 |
+
salience_top_k: 8
|
| 22 |
+
tools: []
|
| 23 |
+
output_extra_fields: [mood, winner]
|
| 24 |
+
hue: 268
|
| 25 |
+
archetype: the table judge
|
|
@@ -0,0 +1,28 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
name: beat-battle
|
| 2 |
+
title: "🎭 Beat Battle"
|
| 3 |
+
goal: >
|
| 4 |
+
Two storytellers in matched seats alternate vivid story beats on a single seed,
|
| 5 |
+
building one shared tale until the delight judge crowns the teller whose beats
|
| 6 |
+
sing loudest. The point is a watchable showcase of which model tells the more
|
| 7 |
+
delightful story.
|
| 8 |
+
default_seed: A lighthouse keeper discovers the sea has started writing letters back.
|
| 9 |
+
example_seeds:
|
| 10 |
+
- A lighthouse keeper discovers the sea has started writing letters back.
|
| 11 |
+
- A clockmaker inherits a town where everyone's shadow runs an hour late.
|
| 12 |
+
- A child trades their name to a fox and must win it back one favour at a time.
|
| 13 |
+
genesis_text: "Two tellers, one tale. It begins: '{seed}'"
|
| 14 |
+
cast:
|
| 15 |
+
- storyteller-a
|
| 16 |
+
- storyteller-b
|
| 17 |
+
- beat-judge
|
| 18 |
+
governor:
|
| 19 |
+
max_turns: 24
|
| 20 |
+
max_calls_per_turn: 8
|
| 21 |
+
max_total_calls: 300
|
| 22 |
+
max_total_tokens: 45000
|
| 23 |
+
hourly_budget_usd: 4.0
|
| 24 |
+
competition:
|
| 25 |
+
kind: versus
|
| 26 |
+
symmetric_seats:
|
| 27 |
+
- storyteller-a
|
| 28 |
+
- storyteller-b
|
|
@@ -0,0 +1,27 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
name: debate-duel
|
| 2 |
+
title: "⚔️ Debate Duel"
|
| 3 |
+
goal: >
|
| 4 |
+
Two debaters in matched seats trade forceful one-line arguments on a single
|
| 5 |
+
motion, round after round, until the adjudicator crowns the sharper voice. The
|
| 6 |
+
point is a watchable clash that reveals which model argues best.
|
| 7 |
+
default_seed: This house believes the forest should never be mapped.
|
| 8 |
+
example_seeds:
|
| 9 |
+
- This house believes the forest should never be mapped.
|
| 10 |
+
- This house would let the oldest tree fall where it stands.
|
| 11 |
+
- This house believes every animal deserves a name.
|
| 12 |
+
genesis_text: "The duelling stage is set. The motion: '{seed}'"
|
| 13 |
+
cast:
|
| 14 |
+
- debater-a
|
| 15 |
+
- debater-b
|
| 16 |
+
- debate-judge
|
| 17 |
+
governor:
|
| 18 |
+
max_turns: 24
|
| 19 |
+
max_calls_per_turn: 8
|
| 20 |
+
max_total_calls: 300
|
| 21 |
+
max_total_tokens: 40000
|
| 22 |
+
hourly_budget_usd: 4.0
|
| 23 |
+
competition:
|
| 24 |
+
kind: versus
|
| 25 |
+
symmetric_seats:
|
| 26 |
+
- debater-a
|
| 27 |
+
- debater-b
|
|
@@ -20,3 +20,8 @@ governor:
|
|
| 20 |
max_total_calls: 600
|
| 21 |
max_total_tokens: 60000
|
| 22 |
hourly_budget_usd: 5.0
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 20 |
max_total_calls: 600
|
| 21 |
max_total_tokens: 60000
|
| 22 |
hourly_budget_usd: 5.0
|
| 23 |
+
# The arena contract (ADR-0029): a judged competition. The mystery-judge endorses one
|
| 24 |
+
# mind's hypothesis as the winner (named in its judge.verdict `winner` field; the
|
| 25 |
+
# judged-competition handler validates that name against the cast).
|
| 26 |
+
competition:
|
| 27 |
+
kind: judged
|
|
@@ -14,9 +14,16 @@ cast:
|
|
| 14 |
- chat-curious
|
| 15 |
- chat-skeptic
|
| 16 |
- chat-host
|
|
|
|
| 17 |
governor:
|
| 18 |
max_turns: 40
|
| 19 |
max_calls_per_turn: 8
|
| 20 |
max_total_calls: 400
|
| 21 |
max_total_tokens: 30000
|
| 22 |
hourly_budget_usd: 3.0
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 14 |
- chat-curious
|
| 15 |
- chat-skeptic
|
| 16 |
- chat-host
|
| 17 |
+
- table-judge
|
| 18 |
governor:
|
| 19 |
max_turns: 40
|
| 20 |
max_calls_per_turn: 8
|
| 21 |
max_total_calls: 400
|
| 22 |
max_total_tokens: 30000
|
| 23 |
hourly_budget_usd: 3.0
|
| 24 |
+
# The arena contract (ADR-0029): a judged competition. After several rounds the
|
| 25 |
+
# table-judge names the most persuasive voice as the winner (a balanced-tier judge so
|
| 26 |
+
# the conversational seats can run on smaller models). The judged-competition handler
|
| 27 |
+
# validates the named winner against the speakers actually at the table.
|
| 28 |
+
competition:
|
| 29 |
+
kind: judged
|
|
@@ -18,3 +18,8 @@ governor:
|
|
| 18 |
max_total_calls: 600
|
| 19 |
max_total_tokens: 60000
|
| 20 |
hourly_budget_usd: 5.0
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 18 |
max_total_calls: 600
|
| 19 |
max_total_tokens: 60000
|
| 20 |
hourly_budget_usd: 5.0
|
| 21 |
+
# The arena contract (ADR-0029): a showcase, not a contest — it exists to demonstrate a
|
| 22 |
+
# tool-using agent (the fortune-teller, granted the oracle) beside one with no tools.
|
| 23 |
+
# There is no meaningful winner to crown, so kind:none keeps it honest (no judge needed).
|
| 24 |
+
competition:
|
| 25 |
+
kind: none
|
|
@@ -29,3 +29,12 @@ governor:
|
|
| 29 |
max_turns: 2000
|
| 30 |
max_calls_per_turn: 16
|
| 31 |
max_total_calls: 20000
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 29 |
max_turns: 2000
|
| 30 |
max_calls_per_turn: 16
|
| 31 |
max_total_calls: 20000
|
| 32 |
+
# The arena contract (ADR-0029): a head-to-head between the lone spy and the herd.
|
| 33 |
+
# The winner is ground truth, decided in code by the spy-host handler (it compares the
|
| 34 |
+
# host's accusation to the real spy), not by the model's prose — so a single-member
|
| 35 |
+
# 'spy' team attributes the win to spy-nil's model, while a 'herd' win credits the seat.
|
| 36 |
+
competition:
|
| 37 |
+
kind: versus
|
| 38 |
+
teams:
|
| 39 |
+
spy: [spy-nil]
|
| 40 |
+
herd: [spy-cara, spy-bex, spy-ovo]
|
|
@@ -32,3 +32,8 @@ governor:
|
|
| 32 |
max_total_calls: 600
|
| 33 |
max_total_tokens: 60000
|
| 34 |
hourly_budget_usd: 5.0
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 32 |
max_total_calls: 600
|
| 33 |
max_total_tokens: 60000
|
| 34 |
hourly_budget_usd: 5.0
|
| 35 |
+
# The arena contract (ADR-0029): collaborative, not competitive. The mischief-critic's
|
| 36 |
+
# reckoning records what became real — it crowns no winner. A full session/history is
|
| 37 |
+
# still kept; it just produces no leaderboard rows.
|
| 38 |
+
competition:
|
| 39 |
+
kind: none
|
|
@@ -0,0 +1,34 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
name: twenty-sprouts
|
| 2 |
+
title: "❓ Twenty Sprouts"
|
| 3 |
+
goal: >
|
| 4 |
+
Play twenty questions in the sprout grove: the keeper holds one secret word, the
|
| 5 |
+
guesser narrows it down with yes/no questions and lands a final guess. The keeper
|
| 6 |
+
wins by staying hidden; the guesser wins by naming the word.
|
| 7 |
+
# The secret word is dealt BY CODE (the secret-keeper handler), never by the model — the
|
| 8 |
+
# same ground-truth discipline as The Steeped's spy words. It rides each keeper event on a
|
| 9 |
+
# private `secret` payload key, which the context builder never surfaces (only `text` is
|
| 10 |
+
# shown), so the guesser can't read it; the sprout-judge handler reads it back at the end.
|
| 11 |
+
default_seed: A small thing from the wood, ordinary enough to overlook, strange enough to miss.
|
| 12 |
+
example_seeds:
|
| 13 |
+
- A small thing from the wood, ordinary enough to overlook, strange enough to miss.
|
| 14 |
+
- Something a traveller might carry, or wish they had, on the long path through the trees.
|
| 15 |
+
- One object the grove keeps returning to, no matter how often it is lost.
|
| 16 |
+
genesis_text: "A hush falls over the sprout grove. The keeper has chosen a word, and the questions begin: '{seed}'"
|
| 17 |
+
cast:
|
| 18 |
+
- sprout-guesser
|
| 19 |
+
- secret-keeper
|
| 20 |
+
- sprout-judge
|
| 21 |
+
governor:
|
| 22 |
+
max_turns: 44
|
| 23 |
+
max_calls_per_turn: 8
|
| 24 |
+
max_total_calls: 400
|
| 25 |
+
max_total_tokens: 50000
|
| 26 |
+
hourly_budget_usd: 4.0
|
| 27 |
+
# The arena contract (ADR-0029): a head-to-head with a deterministic win condition. The
|
| 28 |
+
# winner is the guesser (the dealt word appears in their final guess) or the keeper (it
|
| 29 |
+
# does not) — decided in code by the sprout-judge handler, not the model's prose.
|
| 30 |
+
competition:
|
| 31 |
+
kind: versus
|
| 32 |
+
teams:
|
| 33 |
+
guesser: [sprout-guesser]
|
| 34 |
+
keeper: [secret-keeper]
|
|
@@ -0,0 +1,152 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# ADR-0029: The Scenario Competition Contract — Winners as Data
|
| 2 |
+
|
| 3 |
+
## Status
|
| 4 |
+
|
| 5 |
+
Accepted
|
| 6 |
+
|
| 7 |
+
## Context
|
| 8 |
+
|
| 9 |
+
The arena vision (the [arena roadmap](../architecture/next-steps/arena-roadmap.md),
|
| 10 |
+
Workstreams 2–3) needs one answer the engine could not give: **who won, and on which
|
| 11 |
+
model?** Three gaps blocked it:
|
| 12 |
+
|
| 13 |
+
1. **Winners were prose.** A `judge.verdict` carried only text (`"Verdict: …"`).
|
| 14 |
+
ADR-0026's `run.finished` reserved a `winner` field, but nothing guaranteed a
|
| 15 |
+
structured winner ever existed to fill it — every consumer would have had to parse
|
| 16 |
+
the judge's sentence.
|
| 17 |
+
2. **Some scenarios had no judge at all.** Open Table and Oracle Grove ran until the
|
| 18 |
+
budget tripped; nothing distinguished "this show crowns a winner" from "this show
|
| 19 |
+
is a collaborative showcase".
|
| 20 |
+
3. **The leaderboard (W6) needs declarative attribution.** Wins must attribute to a
|
| 21 |
+
*model*, but a winner can be an agent, a lone-member team (The Steeped's spy), or a
|
| 22 |
+
seat that several minds share. Which scenarios produce winners, and how to map a
|
| 23 |
+
winner to a model, had to be data the run carries — not a heuristic.
|
| 24 |
+
|
| 25 |
+
## Decision
|
| 26 |
+
|
| 27 |
+
Make every scenario declare how (and whether) it produces a winner, and make every
|
| 28 |
+
winner a machine-readable cast reference. Four pieces:
|
| 29 |
+
|
| 30 |
+
- **A `competition` block on `ScenarioConfig`**, validated by a new
|
| 31 |
+
`CompetitionConfig` Pydantic model (`src/core/config.py`, the ADR-0011 declarative
|
| 32 |
+
surface):
|
| 33 |
+
|
| 34 |
+
```yaml
|
| 35 |
+
competition:
|
| 36 |
+
kind: versus | judged | none
|
| 37 |
+
teams: {spy: [spy-nil], herd: [spy-cara, spy-bex, spy-ovo]} # versus, asymmetric sides
|
| 38 |
+
symmetric_seats: [debater-a, debater-b] # versus, identical seats
|
| 39 |
+
```
|
| 40 |
+
|
| 41 |
+
`none` = collaborative/showcase — no winner, no leaderboard rows, still a full
|
| 42 |
+
session. `judged` = a judge in the cast names the winning *agent*. `versus` =
|
| 43 |
+
head-to-head — the winner is decided by ground-truth code or by a judge naming a
|
| 44 |
+
side. `symmetric_seats` are identical manifests bound to different models — the
|
| 45 |
+
"which model argues better" comparison that makes the model leaderboard meaningful.
|
| 46 |
+
|
| 47 |
+
Validation splits by what it needs to see. Shape rules are self-contained on
|
| 48 |
+
`CompetitionConfig._check_kind_shape` (`none` forbids teams/seats; `versus` needs
|
| 49 |
+
≥2 teams or ≥2 seats). Cross-cast rules live in `WorldConfig._check_competition`
|
| 50 |
+
(team/seat members ⊆ cast; `versus`/`judged` require a cast member with
|
| 51 |
+
`role: judge` that emits `judge.verdict`). The block defaults to `kind: none` so a
|
| 52 |
+
block-less scenario still validates; the authoring checklist and
|
| 53 |
+
`tests/test_scenario_contract.py` require an explicit block on every shipped
|
| 54 |
+
scenario.
|
| 55 |
+
|
| 56 |
+
- **The block is stamped onto `run.started`** (`Conductor.reset()`,
|
| 57 |
+
`src/core/conductor.py`), next to ADR-0026's cast→model map. A run is
|
| 58 |
+
self-describing forever: the leaderboard and `FishbowlSession.finalize` read the
|
| 59 |
+
contract off the event, never off mutable config.
|
| 60 |
+
|
| 61 |
+
- **AI judges, code scores.** Judges list `winner` in `output_extra_fields`
|
| 62 |
+
(ADR-0016 structured output), so the live model must emit one. The
|
| 63 |
+
`JudgedCompetition` handler (`handler: judged-competition`,
|
| 64 |
+
`src/agents/competition.py`) then guarantees it is real: it validates the name
|
| 65 |
+
against the agents who actually played and, on a miss (offline stub, or a live
|
| 66 |
+
hallucination), repairs it deterministically — a name found in the verdict prose
|
| 67 |
+
first, the most-active competitor as the fallback. The winner is *always* a genuine
|
| 68 |
+
player, so the no-API-key demo stays watchable. Where ground truth exists, code
|
| 69 |
+
decides instead: `SproutJudge` (`src/agents/twenty_sprouts.py`) subclasses
|
| 70 |
+
`JudgedCompetition` and overrides `decide_winner` — it reads the dealt secret word
|
| 71 |
+
off the ledger and checks the guesser's final line. `SpyHost`
|
| 72 |
+
(`src/agents/handlers.py`) parses the accused name, compares it to the actual spy,
|
| 73 |
+
and stamps `winner = "herd" | "spy"` (a *team* label) plus `correct: bool`. The LLM
|
| 74 |
+
provides the drama; the handler provides the scoreboard.
|
| 75 |
+
|
| 76 |
+
- **Attribution reconciles agent names and team labels.**
|
| 77 |
+
`FishbowlSession.finalize` (`src/ui/fishbowl/session.py`) maps an agent-name winner
|
| 78 |
+
straight to its model via the `run.started` cast map; a single-member team
|
| 79 |
+
attributes the lone member's model; a multi-member team yields no single winning
|
| 80 |
+
model — the seat won, not a model. The Fishbowl shows a winner ribbon in the
|
| 81 |
+
verdict banner (`view_model.py` → `render/meters.py::render_verdict`) only when a
|
| 82 |
+
`winner` is present, so `none`-kind scenarios and legacy runs render byte-identically
|
| 83 |
+
to before.
|
| 84 |
+
|
| 85 |
+
All eight shipped scenarios now declare their kind: The Steeped (versus,
|
| 86 |
+
teams, ground-truth `SpyHost`) · Twenty Sprouts (versus, teams, ground-truth
|
| 87 |
+
`SproutJudge`) · Debate Duel and Beat Battle (versus, symmetric seats) · Mystery
|
| 88 |
+
Roots and Open Table (judged) · Thousand Token Wood and Oracle Grove (none).
|
| 89 |
+
|
| 90 |
+
## Consequences
|
| 91 |
+
|
| 92 |
+
- **The leaderboard becomes a projection.** `run.started` carries the contract and
|
| 93 |
+
the cast→model map; `judge.verdict` carries a validated winner; `run.finished`
|
| 94 |
+
carries the resolved `winner`/`winning_model`. W6 is a pure fold over the ledger.
|
| 95 |
+
- **Offline runs stay reproducible and watchable.** The deterministic repair path
|
| 96 |
+
means the stub demo crowns a real player without any model emitting a `winner`.
|
| 97 |
+
- **Scenarios are honest about what they are.** `kind: none` is a first-class
|
| 98 |
+
answer — Oracle Grove no longer looks like a game that forgot its judge.
|
| 99 |
+
- **Cross-cast validation only runs through `WorldConfig`.** `validate_scenario` on a
|
| 100 |
+
lone YAML checks shape but cannot know the cast's manifests, so a missing judge or
|
| 101 |
+
stray team member surfaces only when a `WorldConfig` is composed — the Lab path and
|
| 102 |
+
`tests/test_scenario_contract.py`, not per-file validation. Authors must run the
|
| 103 |
+
contract test, not just eyeball the YAML.
|
| 104 |
+
- **The repaired winner can be wrong — but never fake.** If a live judge hallucinates
|
| 105 |
+
a name and the prose match fails, the most-active fallback may crown a competitor
|
| 106 |
+
the judge did not intend. We trade occasional misattribution for a hard guarantee
|
| 107 |
+
that every winner maps to a real model.
|
| 108 |
+
- **`winner` is overloaded: agent name or team label.** Consumers must check both
|
| 109 |
+
namespaces (as `finalize` does). A future agent named like a team label would
|
| 110 |
+
collide; cast naming conventions carry that risk for now.
|
| 111 |
+
- **Symmetric-seat fairness is deferred.** Raw win counts mislead when seats differ
|
| 112 |
+
in difficulty or order; the model leaderboard should report win rate *per seat* and
|
| 113 |
+
alternate first-speaker across sessions (roadmap §6.3). The contract records the
|
| 114 |
+
seats; the fairness math lands with W6.
|
| 115 |
+
|
| 116 |
+
## Alternatives considered
|
| 117 |
+
|
| 118 |
+
- **Parse winners out of verdict prose at read time.** No schema change, but every
|
| 119 |
+
consumer re-implements the heuristic, the offline stub yields no winner at all, and
|
| 120 |
+
a hallucinated name poisons history silently. Rejected: the ledger should store the
|
| 121 |
+
answer, not the puzzle.
|
| 122 |
+
- **Require a judge in every scenario.** Uniform, but forces a fake contest onto
|
| 123 |
+
collaborative showcases. `kind: none` keeps them honest and still gives them
|
| 124 |
+
sessions and history.
|
| 125 |
+
- **Re-ask the model on an invalid winner, then declare "no contest"** (the
|
| 126 |
+
roadmap's original sketch). Spends live budget on a retry and leaves offline runs
|
| 127 |
+
winner-less. The deterministic repair is cheaper and keeps every demo resolvable.
|
| 128 |
+
- **Hang the contract on the judge manifest instead of the scenario.** The judge
|
| 129 |
+
doesn't know the teams or the cast — the scenario does. Keeping it on
|
| 130 |
+
`ScenarioConfig` also lets one generic judge handler (`judged-competition`) serve
|
| 131 |
+
four different scenarios.
|
| 132 |
+
- **Make the `competition` block required.** Breaks every legacy/test scenario and
|
| 133 |
+
Lab-composed world. A `kind: none` default plus test-enforced explicitness on
|
| 134 |
+
shipped scenarios gets the same guarantee without the migration.
|
| 135 |
+
|
| 136 |
+
## References
|
| 137 |
+
|
| 138 |
+
- Builds on ADR-0011 (declarative, validatable config — `CompetitionConfig` is that
|
| 139 |
+
surface), ADR-0016 (validated structured output — `winner` rides
|
| 140 |
+
`output_extra_fields`), and ADR-0026 (run lifecycle — `run.started` stamping,
|
| 141 |
+
`run.finished.winner`/`winning_model`).
|
| 142 |
+
- Spec: [arena-roadmap.md](../architecture/next-steps/arena-roadmap.md) §W2–W3;
|
| 143 |
+
authoring guide: [scenario-authoring.md](../architecture/scenario-authoring.md)
|
| 144 |
+
(the arena-grade checklist).
|
| 145 |
+
- `src/core/config.py` — `CompetitionConfig`, `ScenarioConfig.competition`,
|
| 146 |
+
`WorldConfig._check_competition`
|
| 147 |
+
- `src/core/conductor.py` — `run.started` competition stamp
|
| 148 |
+
- `src/agents/competition.py` — `JudgedCompetition`; `src/agents/handlers.py` —
|
| 149 |
+
`SpyHost`; `src/agents/twenty_sprouts.py` — `SecretKeeper`, `SproutJudge`
|
| 150 |
+
- `src/ui/fishbowl/session.py` — winner→model reconciliation;
|
| 151 |
+
`src/ui/fishbowl/view_model.py`, `src/ui/fishbowl/render/meters.py` — winner ribbon
|
| 152 |
+
- `tests/test_scenario_contract.py` — the enforced authoring checklist
|
|
@@ -425,3 +425,100 @@ only post to the shared log, and the seam shows.
|
|
| 425 |
invisible unless you add a renderer.
|
| 426 |
- **One shared ledger across scenarios** when `DATABASE_URL` is set (`app.py:34`);
|
| 427 |
use `scripts/resume_run.py` (one DB per scenario) for isolated durable runs.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 425 |
invisible unless you add a renderer.
|
| 426 |
- **One shared ledger across scenarios** when `DATABASE_URL` is set (`app.py:34`);
|
| 427 |
use `scripts/resume_run.py` (one DB per scenario) for isolated durable runs.
|
| 428 |
+
|
| 429 |
+
---
|
| 430 |
+
|
| 431 |
+
## Scenario authoring checklist: making it arena-grade
|
| 432 |
+
|
| 433 |
+
A scenario that *runs* is not yet a scenario that *competes*. The arena (ADR-0029,
|
| 434 |
+
[arena roadmap](next-steps/arena-roadmap.md) W2–W3) asks every shipped scenario to
|
| 435 |
+
declare how it ends and who — if anyone — wins, so the leaderboard can attribute wins
|
| 436 |
+
to models. Run down this list before you ship:
|
| 437 |
+
|
| 438 |
+
- [ ] **Premise** — `goal`, `default_seed`, `example_seeds`, and `genesis_text` are
|
| 439 |
+
all set. The goal is rendered into every prompt; the seeds are what the Summon
|
| 440 |
+
dropdown offers; the genesis is the audience's opening line.
|
| 441 |
+
- [ ] **Cast** — every name in `cast:` resolves to a manifest in `config/agents/`
|
| 442 |
+
(`validate_world` fails loudly if not).
|
| 443 |
+
- [ ] **Governor** — explicit budgets (`max_turns`, `max_total_calls`, ideally
|
| 444 |
+
`max_total_tokens` and `hourly_budget_usd`). The budget is the show's outer wall.
|
| 445 |
+
- [ ] **`competition` block** — explicit, even when it's `kind: none`. The model
|
| 446 |
+
defaults to `none` so old YAML still validates, but shipped scenarios must say
|
| 447 |
+
what they are out loud (the contract test checks).
|
| 448 |
+
- [ ] **End condition** — a judge whose `schedule.tick_every` fires *within* the
|
| 449 |
+
governor's budget (the first `judge.verdict` drops the curtain — see the pitfall
|
| 450 |
+
above), or `kind: none` with a closing voice like the Wood's reckoning.
|
| 451 |
+
- [ ] **Structured verdict** — the judge lists `winner` in `output_extra_fields`
|
| 452 |
+
and runs `handler: judged-competition` (or a ground-truth subclass), so the
|
| 453 |
+
verdict carries `payload.winner` as data, not prose.
|
| 454 |
+
- [ ] **Reveal text** — the verdict reads as an *ending*. Hidden-info games attach a
|
| 455 |
+
`reveal: [{agent, secret, role}]` payload (the banner renders it); judged games
|
| 456 |
+
make the verdict prose name and justify the winner.
|
| 457 |
+
|
| 458 |
+
### The `competition` block, one example per kind
|
| 459 |
+
|
| 460 |
+
**`versus` with teams** — asymmetric sides; ground truth decided in code
|
| 461 |
+
(`config/scenarios/the-steeped.yaml`, scored by the `spy-host` handler):
|
| 462 |
+
|
| 463 |
+
```yaml
|
| 464 |
+
competition:
|
| 465 |
+
kind: versus
|
| 466 |
+
teams:
|
| 467 |
+
spy: [spy-nil]
|
| 468 |
+
herd: [spy-cara, spy-bex, spy-ovo]
|
| 469 |
+
```
|
| 470 |
+
|
| 471 |
+
**`versus` with symmetric seats** — identical manifests, different models; the
|
| 472 |
+
cleanest "which model plays better" arena (`config/scenarios/debate-duel.yaml`):
|
| 473 |
+
|
| 474 |
+
```yaml
|
| 475 |
+
competition:
|
| 476 |
+
kind: versus
|
| 477 |
+
symmetric_seats:
|
| 478 |
+
- debater-a
|
| 479 |
+
- debater-b
|
| 480 |
+
```
|
| 481 |
+
|
| 482 |
+
**`judged`** — a judge names the winning *agent* via its verdict's `winner` field
|
| 483 |
+
(`config/scenarios/mystery-roots.yaml`, `open-table.yaml`):
|
| 484 |
+
|
| 485 |
+
```yaml
|
| 486 |
+
competition:
|
| 487 |
+
kind: judged
|
| 488 |
+
```
|
| 489 |
+
|
| 490 |
+
**`none`** — collaborative or showcase; full session and history, no winner, no
|
| 491 |
+
leaderboard rows (`config/scenarios/thousand-token-wood.yaml`, `oracle-grove.yaml`):
|
| 492 |
+
|
| 493 |
+
```yaml
|
| 494 |
+
competition:
|
| 495 |
+
kind: none
|
| 496 |
+
```
|
| 497 |
+
|
| 498 |
+
The eight shipped scenarios, for orientation: 🕵 The Steeped (versus, teams,
|
| 499 |
+
ground-truth `SpyHost`) · ❓ Twenty Sprouts (versus, teams, ground-truth
|
| 500 |
+
`SproutJudge`) · ⚔️ Debate Duel (versus, symmetric seats) · 🎭 Beat Battle (versus,
|
| 501 |
+
symmetric seats) · 🔍 Mystery Roots (judged) · 💬 Open Table (judged) · 🍄 Thousand
|
| 502 |
+
Token Wood (none) · 🔮 Oracle Grove (none, tool-use showcase).
|
| 503 |
+
|
| 504 |
+
### The checklist is enforced, not aspirational
|
| 505 |
+
|
| 506 |
+
`tests/test_scenario_contract.py` loads every YAML under `config/scenarios/`,
|
| 507 |
+
composes a full `WorldConfig` (this matters — the cross-cast competition rules run on
|
| 508 |
+
`WorldConfig`, **not** on the per-file `validate_scenario` path, so a lone-YAML check
|
| 509 |
+
cannot catch a missing judge), and asserts:
|
| 510 |
+
|
| 511 |
+
- an **explicit** `competition` block is present on every shipped scenario;
|
| 512 |
+
- `judged`/`versus` scenarios include a cast member with `role: judge` that emits
|
| 513 |
+
`judge.verdict`;
|
| 514 |
+
- every team member and symmetric seat is in the scenario's `cast`;
|
| 515 |
+
- symmetric seats are *actually* symmetric — their manifests are identical except
|
| 516 |
+
`name`, `hue`, `archetype`, `model_profile`, and `model_endpoint`, so a seat win
|
| 517 |
+
measures the model, not a stacked persona.
|
| 518 |
+
|
| 519 |
+
```bash
|
| 520 |
+
uv run pytest tests/test_scenario_contract.py -q # your scenario is arena-grade when this is green
|
| 521 |
+
```
|
| 522 |
+
|
| 523 |
+
If your scenario fails the judge check but genuinely has no winner, the fix is one
|
| 524 |
+
line: say `kind: none` and mean it.
|
|
@@ -0,0 +1,123 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Competition handlers — turning a judge's ruling into a machine-readable winner.
|
| 2 |
+
|
| 3 |
+
The arena contract (ADR-0029) says a winner is *data*, not prose: a
|
| 4 |
+
``judge.verdict`` event should carry ``payload["winner"]`` naming a cast member,
|
| 5 |
+
so the leaderboard can attribute the win to that agent's model. Two layers make
|
| 6 |
+
that true:
|
| 7 |
+
|
| 8 |
+
* **The LLM provides the drama.** A judge manifest lists ``winner`` in
|
| 9 |
+
``output_extra_fields`` (``src/core/structured.py``), so on the live path the
|
| 10 |
+
model is *required* to emit a ``winner`` string alongside its verdict text.
|
| 11 |
+
* **Code provides the scoreboard.** :class:`JudgedCompetition` validates that
|
| 12 |
+
string against the agents actually on stage and, on any miss (a hallucinated
|
| 13 |
+
name, or the offline stub which can't know the cast), falls back to a
|
| 14 |
+
deterministic real candidate — so the verdict *always* names a genuine player.
|
| 15 |
+
This keeps the no-API-key offline demo watchable: the winner is real even when
|
| 16 |
+
the model never produced a usable one.
|
| 17 |
+
|
| 18 |
+
A judge with a known ground truth (The Steeped's :class:`SpyHost`, Twenty Sprouts'
|
| 19 |
+
``SproutJudge``) computes the winner in code instead — those subclass this and
|
| 20 |
+
override :meth:`decide_winner`. This is the best-practice split the roadmap calls
|
| 21 |
+
for: AI is load-bearing for judgment, code is load-bearing for bookkeeping.
|
| 22 |
+
"""
|
| 23 |
+
|
| 24 |
+
from __future__ import annotations
|
| 25 |
+
|
| 26 |
+
from src.agents.base import ManifestAgent
|
| 27 |
+
from src.core.events import Event
|
| 28 |
+
from src.core.projections import StageProjection
|
| 29 |
+
from src.core.registry import register_handler
|
| 30 |
+
|
| 31 |
+
# Kinds a *competitor* emits — used to find who is eligible to win. A judge emits
|
| 32 |
+
# ``judge.verdict`` and is excluded; the scene narrator emits ``world.observed`` (and at
|
| 33 |
+
# genesis its actor is the *scenario name*, not a player), so that is excluded too.
|
| 34 |
+
# Candidates are thus only the minds that actually spoke, derived from the run's events.
|
| 35 |
+
_COMPETITOR_KINDS = frozenset({"agent.spoke", "agent.thought", "oracle.spoke"})
|
| 36 |
+
|
| 37 |
+
|
| 38 |
+
@register_handler("judged-competition")
|
| 39 |
+
class JudgedCompetition(ManifestAgent):
|
| 40 |
+
"""A judge whose ``winner`` is validated against the live cast (and repaired offline).
|
| 41 |
+
|
| 42 |
+
The generic turn produces the verdict text plus a ``winner`` field (because the
|
| 43 |
+
manifest lists it in ``output_extra_fields``). This handler then guarantees
|
| 44 |
+
``payload["winner"]`` is a real on-stage competitor: if the model's value isn't
|
| 45 |
+
one (offline stub, or a live hallucination), it derives one — first by reading a
|
| 46 |
+
name out of the verdict prose, then by falling back to the most active competitor.
|
| 47 |
+
Deterministic, so offline runs are reproducible.
|
| 48 |
+
"""
|
| 49 |
+
|
| 50 |
+
def act(
|
| 51 |
+
self,
|
| 52 |
+
run_id: str,
|
| 53 |
+
turn: int,
|
| 54 |
+
projection: StageProjection,
|
| 55 |
+
recent_events: tuple[Event, ...],
|
| 56 |
+
) -> Event:
|
| 57 |
+
event = super().act(run_id, turn, projection, recent_events)
|
| 58 |
+
candidates = self._candidates(recent_events)
|
| 59 |
+
if not candidates:
|
| 60 |
+
return event # nothing to attribute — leave the verdict as prose only
|
| 61 |
+
winner = self.decide_winner(event, candidates, recent_events)
|
| 62 |
+
if winner:
|
| 63 |
+
event.payload["winner"] = winner
|
| 64 |
+
return event
|
| 65 |
+
|
| 66 |
+
# ── decision (override for ground-truth judges) ──────────────────────────────
|
| 67 |
+
|
| 68 |
+
def decide_winner(
|
| 69 |
+
self,
|
| 70 |
+
event: Event,
|
| 71 |
+
candidates: list[str],
|
| 72 |
+
recent_events: tuple[Event, ...],
|
| 73 |
+
) -> str | None:
|
| 74 |
+
"""Return the winning cast name.
|
| 75 |
+
|
| 76 |
+
Honours the model's ``winner`` when it names a real competitor; otherwise
|
| 77 |
+
repairs it deterministically (prose mention → most-active fallback)."""
|
| 78 |
+
named = (event.payload.get("winner") or "").strip()
|
| 79 |
+
if named in candidates:
|
| 80 |
+
return named
|
| 81 |
+
return self._winner_from_prose(event, candidates) or self._most_active(candidates, recent_events)
|
| 82 |
+
|
| 83 |
+
# ── helpers ──────────────────────────────────────────────────────────────────
|
| 84 |
+
|
| 85 |
+
def _candidates(self, recent_events: tuple[Event, ...]) -> list[str]:
|
| 86 |
+
"""On-stage competitors: actors who actually spoke, minus this judge.
|
| 87 |
+
|
| 88 |
+
Insertion-ordered (first appearance) so the fallback is stable and readable.
|
| 89 |
+
"""
|
| 90 |
+
seen: dict[str, None] = {}
|
| 91 |
+
for e in recent_events:
|
| 92 |
+
if e.kind in _COMPETITOR_KINDS and e.actor and e.actor != self.name:
|
| 93 |
+
seen.setdefault(e.actor, None)
|
| 94 |
+
return list(seen)
|
| 95 |
+
|
| 96 |
+
@staticmethod
|
| 97 |
+
def _winner_from_prose(event: Event, candidates: list[str]) -> str | None:
|
| 98 |
+
"""Find the candidate the verdict text names, by full-slug substring match.
|
| 99 |
+
|
| 100 |
+
Matches on the exact cast slug (``debater-a``) — and the hyphen→space variant
|
| 101 |
+
(``debater a``) a live model might write — so it distinguishes symmetric seats
|
| 102 |
+
that share a stem (``debater-a`` vs ``debater-b``) instead of matching the stem.
|
| 103 |
+
When several are named, the earliest-mentioned wins (the one the judge leads
|
| 104 |
+
with). Returns ``None`` when the prose names no competitor."""
|
| 105 |
+
text = (event.payload.get("text") or "").lower()
|
| 106 |
+
best: str | None = None
|
| 107 |
+
best_pos = len(text) + 1
|
| 108 |
+
for name in candidates:
|
| 109 |
+
for needle in (name.lower(), name.lower().replace("-", " ")):
|
| 110 |
+
pos = text.find(needle)
|
| 111 |
+
if 0 <= pos < best_pos:
|
| 112 |
+
best, best_pos = name, pos
|
| 113 |
+
break
|
| 114 |
+
return best
|
| 115 |
+
|
| 116 |
+
@staticmethod
|
| 117 |
+
def _most_active(candidates: list[str], recent_events: tuple[Event, ...]) -> str:
|
| 118 |
+
"""Deterministic fallback: the competitor who spoke most (ties → cast order)."""
|
| 119 |
+
counts = {name: 0 for name in candidates}
|
| 120 |
+
for e in recent_events:
|
| 121 |
+
if e.actor in counts and e.kind in _COMPETITOR_KINDS:
|
| 122 |
+
counts[e.actor] += 1
|
| 123 |
+
return max(candidates, key=lambda name: (counts[name], -candidates.index(name)))
|
|
@@ -9,6 +9,8 @@ manifest; the manifest still supplies all declarative fields.
|
|
| 9 |
|
| 10 |
from __future__ import annotations
|
| 11 |
|
|
|
|
|
|
|
| 12 |
from src.agents.base import ManifestAgent
|
| 13 |
from src.core.events import Event
|
| 14 |
from src.core.projections import StageProjection
|
|
@@ -17,18 +19,23 @@ from src.core.registry import register_handler
|
|
| 17 |
|
| 18 |
@register_handler("spy-host")
|
| 19 |
class SpyHost(ManifestAgent):
|
| 20 |
-
"""The word-pair bluff host: delivers a verdict
|
| 21 |
|
| 22 |
The generic turn produces the verdict *text* (who the host accuses). This handler
|
| 23 |
-
then
|
| 24 |
-
|
| 25 |
-
|
| 26 |
-
|
| 27 |
-
|
| 28 |
-
|
| 29 |
-
|
| 30 |
-
|
| 31 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 32 |
"""
|
| 33 |
|
| 34 |
# agent name → (secret word, table role) for the shipped "the-steeped" cast.
|
|
@@ -39,6 +46,11 @@ class SpyHost(ManifestAgent):
|
|
| 39 |
"spy-nil": ("TEA", "SPY — CAUGHT"),
|
| 40 |
}
|
| 41 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 42 |
def act(
|
| 43 |
self,
|
| 44 |
run_id: str,
|
|
@@ -48,15 +60,29 @@ class SpyHost(ManifestAgent):
|
|
| 48 |
) -> Event:
|
| 49 |
event = super().act(run_id, turn, projection, recent_events)
|
| 50 |
on_stage = {e.actor for e in recent_events}
|
| 51 |
-
|
| 52 |
-
|
| 53 |
-
|
| 54 |
-
|
| 55 |
-
|
| 56 |
-
|
| 57 |
-
|
|
|
|
|
|
|
| 58 |
return event
|
| 59 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 60 |
|
| 61 |
@register_handler("fortune-teller")
|
| 62 |
class FortuneTeller(ManifestAgent):
|
|
|
|
| 9 |
|
| 10 |
from __future__ import annotations
|
| 11 |
|
| 12 |
+
import re
|
| 13 |
+
|
| 14 |
from src.agents.base import ManifestAgent
|
| 15 |
from src.core.events import Event
|
| 16 |
from src.core.projections import StageProjection
|
|
|
|
| 19 |
|
| 20 |
@register_handler("spy-host")
|
| 21 |
class SpyHost(ManifestAgent):
|
| 22 |
+
"""The word-pair bluff host: delivers a verdict, scores the game, unmasks every word.
|
| 23 |
|
| 24 |
The generic turn produces the verdict *text* (who the host accuses). This handler
|
| 25 |
+
then does two things on the emitted ``judge.verdict`` payload:
|
| 26 |
+
|
| 27 |
+
* **Scoreboard (W2.2).** Ground truth lives in code, not the model: it parses the
|
| 28 |
+
accused name out of the prose, compares it to the *actual* spy, and stamps
|
| 29 |
+
``winner = "herd"`` (caught) or ``"spy"`` (escaped) plus ``correct: bool``. The
|
| 30 |
+
``winner`` is a *team* label (matching ``competition.teams`` in the scenario);
|
| 31 |
+
``FishbowlSession.finalize`` reconciles a single-member team to its model.
|
| 32 |
+
* **Reveal.** One ``{agent, secret, role}`` row per on-stage player — exactly the
|
| 33 |
+
shape the Fishbowl verdict banner renders (``view_model``/``render_verdict``).
|
| 34 |
+
|
| 35 |
+
Both ride the real ledger, so the unmasking and the score are genuine engine events,
|
| 36 |
+
not a UI overlay. The secret-word map is curated demo content for ``the-steeped``
|
| 37 |
+
(mirroring the words baked into each player's persona); only players actually present
|
| 38 |
+
on stage are revealed, so editing the cast in the Lab never produces a phantom row.
|
| 39 |
"""
|
| 40 |
|
| 41 |
# agent name → (secret word, table role) for the shipped "the-steeped" cast.
|
|
|
|
| 46 |
"spy-nil": ("TEA", "SPY — CAUGHT"),
|
| 47 |
}
|
| 48 |
|
| 49 |
+
@property
|
| 50 |
+
def _true_spy(self) -> str | None:
|
| 51 |
+
"""The agent who actually holds the odd word — the ground truth the prose is scored against."""
|
| 52 |
+
return next((name for name, (_, role) in self._REVEAL.items() if "SPY" in role), None)
|
| 53 |
+
|
| 54 |
def act(
|
| 55 |
self,
|
| 56 |
run_id: str,
|
|
|
|
| 60 |
) -> Event:
|
| 61 |
event = super().act(run_id, turn, projection, recent_events)
|
| 62 |
on_stage = {e.actor for e in recent_events}
|
| 63 |
+
present = [name for name in self._REVEAL if name in on_stage]
|
| 64 |
+
if present:
|
| 65 |
+
event.payload["reveal"] = [
|
| 66 |
+
{"agent": name, "secret": self._REVEAL[name][0], "role": self._REVEAL[name][1]} for name in present
|
| 67 |
+
]
|
| 68 |
+
accused = self._accused(event.payload.get("text", ""), present)
|
| 69 |
+
correct = accused is not None and accused == self._true_spy
|
| 70 |
+
event.payload["correct"] = correct
|
| 71 |
+
event.payload["winner"] = "herd" if correct else "spy"
|
| 72 |
return event
|
| 73 |
|
| 74 |
+
@staticmethod
|
| 75 |
+
def _accused(text: str, present: list[str]) -> str | None:
|
| 76 |
+
"""Which on-stage player the host's prose names as the spy.
|
| 77 |
+
|
| 78 |
+
Personas/verdicts name players by their bare handle in caps ("NIL is the spy"),
|
| 79 |
+
so match each present agent's tail segment (``spy-nil`` → ``NIL``) as a whole word."""
|
| 80 |
+
words = set(re.findall(r"[a-z]+", text.lower()))
|
| 81 |
+
for name in present:
|
| 82 |
+
if name.rsplit("-", 1)[-1].lower() in words:
|
| 83 |
+
return name
|
| 84 |
+
return None
|
| 85 |
+
|
| 86 |
|
| 87 |
@register_handler("fortune-teller")
|
| 88 |
class FortuneTeller(ManifestAgent):
|
|
@@ -0,0 +1,131 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Twenty Sprouts handlers — a 20-questions game where code owns the ground truth.
|
| 2 |
+
|
| 3 |
+
The secret word is *dealt by code*, never by the model — exactly the discipline The
|
| 4 |
+
Steeped uses for the spy words. Two handlers:
|
| 5 |
+
|
| 6 |
+
* :class:`SecretKeeper` deals a secret word deterministically from the seed and
|
| 7 |
+
carries it on its events as a **private** ``secret`` payload key. Because the
|
| 8 |
+
context/memory builder only ever surfaces an event's ``text`` (``_displayable``
|
| 9 |
+
in ``src/core/memory.py``), the guesser never sees the word — only the keeper's
|
| 10 |
+
yes/no answers. The word rides the ledger as ground truth without leaking.
|
| 11 |
+
* :class:`SproutJudge` (a :class:`~src.agents.competition.JudgedCompetition`) reads
|
| 12 |
+
the dealt word off the ledger and the guesser's last line, decides the winner in
|
| 13 |
+
code (the guess contains the word → guesser; else the keeper kept its secret),
|
| 14 |
+
and attaches a ``reveal`` unmasking the word. Deterministic win condition,
|
| 15 |
+
reproducible offline.
|
| 16 |
+
"""
|
| 17 |
+
|
| 18 |
+
from __future__ import annotations
|
| 19 |
+
|
| 20 |
+
import hashlib
|
| 21 |
+
import re
|
| 22 |
+
|
| 23 |
+
from src.agents.base import ManifestAgent
|
| 24 |
+
from src.agents.competition import JudgedCompetition
|
| 25 |
+
from src.core.events import Event
|
| 26 |
+
from src.core.projections import StageProjection
|
| 27 |
+
from src.core.registry import register_handler
|
| 28 |
+
|
| 29 |
+
# Curated, woodland-flavoured words the keeper can hold. Evocative enough to make a
|
| 30 |
+
# clean offline demo, concrete enough that a guesser can corner them with yes/no.
|
| 31 |
+
_WORDS: tuple[str, ...] = (
|
| 32 |
+
"ACORN",
|
| 33 |
+
"LANTERN",
|
| 34 |
+
"RIVER",
|
| 35 |
+
"FIDDLE",
|
| 36 |
+
"COMPASS",
|
| 37 |
+
"EMBER",
|
| 38 |
+
"WILLOW",
|
| 39 |
+
"KETTLE",
|
| 40 |
+
"FEATHER",
|
| 41 |
+
"BRIDGE",
|
| 42 |
+
)
|
| 43 |
+
|
| 44 |
+
_GUESSER_NAME = "sprout-guesser"
|
| 45 |
+
_WORD = re.compile(r"[a-z]+")
|
| 46 |
+
|
| 47 |
+
|
| 48 |
+
def _word_for_seed(seed: str) -> str:
|
| 49 |
+
"""Deal a secret word as a pure function of the seed — reproducible offline."""
|
| 50 |
+
digest = hashlib.sha256((seed or "").encode("utf-8")).hexdigest()
|
| 51 |
+
return _WORDS[int(digest[:8], 16) % len(_WORDS)]
|
| 52 |
+
|
| 53 |
+
|
| 54 |
+
@register_handler("secret-keeper")
|
| 55 |
+
class SecretKeeper(ManifestAgent):
|
| 56 |
+
"""Holds the dealt word and answers the guesser, never spelling it aloud.
|
| 57 |
+
|
| 58 |
+
The word is stamped on every one of the keeper's events as a private ``secret``
|
| 59 |
+
key — visible to the judge (which reads payloads off the ledger) but never to the
|
| 60 |
+
guesser (whose context is built from ``text`` only). The keeper's spoken ``text``
|
| 61 |
+
is its yes/no answer; the secret stays out of it.
|
| 62 |
+
"""
|
| 63 |
+
|
| 64 |
+
def _build_extra_prompt(self, projection: StageProjection, recent_events: tuple[Event, ...]) -> str:
|
| 65 |
+
word = _word_for_seed(projection.seed)
|
| 66 |
+
return (
|
| 67 |
+
f"YOUR SECRET WORD (never write, spell, or quote it — only answer about it): {word}\n"
|
| 68 |
+
"Answer the guesser's most recent yes/no question truthfully in one short sentence. "
|
| 69 |
+
"If they have not asked yet, invite them to begin. Never reveal the word."
|
| 70 |
+
)
|
| 71 |
+
|
| 72 |
+
def act(
|
| 73 |
+
self,
|
| 74 |
+
run_id: str,
|
| 75 |
+
turn: int,
|
| 76 |
+
projection: StageProjection,
|
| 77 |
+
recent_events: tuple[Event, ...],
|
| 78 |
+
) -> Event:
|
| 79 |
+
event = super().act(run_id, turn, projection, recent_events)
|
| 80 |
+
# Ground truth on the ledger, private (non-``text``) so it never reaches the
|
| 81 |
+
# guesser's prompt — the judge reads it back at the reckoning.
|
| 82 |
+
event.payload["secret"] = _word_for_seed(projection.seed)
|
| 83 |
+
return event
|
| 84 |
+
|
| 85 |
+
|
| 86 |
+
@register_handler("sprout-judge")
|
| 87 |
+
class SproutJudge(JudgedCompetition):
|
| 88 |
+
"""Decides Twenty Sprouts in code: did the guesser's last line contain the word?
|
| 89 |
+
|
| 90 |
+
Reads the dealt word off the keeper's latest event and the guesser's most recent
|
| 91 |
+
line, both from the ledger. Winner is the **agent name** (``sprout-guesser`` if
|
| 92 |
+
the word appears in their guess, else ``secret-keeper``), so the run's
|
| 93 |
+
winner→model attribution maps straight through the cast. Attaches a ``reveal``
|
| 94 |
+
unmasking the word for the verdict banner.
|
| 95 |
+
"""
|
| 96 |
+
|
| 97 |
+
def decide_winner(
|
| 98 |
+
self,
|
| 99 |
+
event: Event,
|
| 100 |
+
candidates: list[str],
|
| 101 |
+
recent_events: tuple[Event, ...],
|
| 102 |
+
) -> str | None:
|
| 103 |
+
secret = self._dealt_word(recent_events)
|
| 104 |
+
if not secret:
|
| 105 |
+
return super().decide_winner(event, candidates, recent_events)
|
| 106 |
+
guess = self._last_guess(recent_events)
|
| 107 |
+
caught = secret.lower() in set(_WORD.findall(guess.lower()))
|
| 108 |
+
event.payload["correct"] = caught
|
| 109 |
+
event.payload["reveal"] = [
|
| 110 |
+
{
|
| 111 |
+
"agent": "secret-keeper",
|
| 112 |
+
"secret": secret,
|
| 113 |
+
"role": "GUESSED" if caught else "KEPT SECRET",
|
| 114 |
+
}
|
| 115 |
+
]
|
| 116 |
+
return _GUESSER_NAME if caught else "secret-keeper"
|
| 117 |
+
|
| 118 |
+
@staticmethod
|
| 119 |
+
def _dealt_word(recent_events: tuple[Event, ...]) -> str:
|
| 120 |
+
for e in reversed(recent_events):
|
| 121 |
+
secret = e.payload.get("secret")
|
| 122 |
+
if secret:
|
| 123 |
+
return str(secret)
|
| 124 |
+
return ""
|
| 125 |
+
|
| 126 |
+
@staticmethod
|
| 127 |
+
def _last_guess(recent_events: tuple[Event, ...]) -> str:
|
| 128 |
+
for e in reversed(recent_events):
|
| 129 |
+
if e.actor == _GUESSER_NAME and e.kind == "agent.spoke":
|
| 130 |
+
return str(e.payload.get("text", ""))
|
| 131 |
+
return ""
|
|
@@ -136,6 +136,12 @@ class Conductor:
|
|
| 136 |
obs.set_context(run_id=self.run_id, turn=self.turn)
|
| 137 |
obs.log("run.started", run_id=self.run_id, seed=seed, goal=goal, scenario=scenario_name)
|
| 138 |
payload: dict = {"seed": seed, "goal": goal, "scenario": scenario_name, "cast": cast}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 139 |
if self.session_id:
|
| 140 |
payload["session_id"] = self.session_id
|
| 141 |
genesis_start = Event(
|
|
|
|
| 136 |
obs.set_context(run_id=self.run_id, turn=self.turn)
|
| 137 |
obs.log("run.started", run_id=self.run_id, seed=seed, goal=goal, scenario=scenario_name)
|
| 138 |
payload: dict = {"seed": seed, "goal": goal, "scenario": scenario_name, "cast": cast}
|
| 139 |
+
# Stamp the arena contract so the run is self-describing forever (ADR-0029):
|
| 140 |
+
# the leaderboard reads competition.kind to know which runs produce winners
|
| 141 |
+
# and how to attribute them. None (legacy/test scenarios) behaves like 'none'.
|
| 142 |
+
competition = getattr(self.scenario, "competition", None)
|
| 143 |
+
if competition is not None:
|
| 144 |
+
payload["competition"] = competition.model_dump()
|
| 145 |
if self.session_id:
|
| 146 |
payload["session_id"] = self.session_id
|
| 147 |
genesis_start = Event(
|
|
@@ -16,8 +16,11 @@ to "emit JSON, validate it, run it." See ADR-0011.
|
|
| 16 |
The agent schema itself is :class:`AgentManifest` (``src/core/manifest.py``) — we
|
| 17 |
reuse it here rather than duplicating, so the four stable contracts stay singular.
|
| 18 |
"""
|
|
|
|
| 19 |
from __future__ import annotations
|
| 20 |
|
|
|
|
|
|
|
| 21 |
from pydantic import BaseModel, ConfigDict, Field, model_validator
|
| 22 |
|
| 23 |
from src.core.manifest import AgentManifest
|
|
@@ -75,6 +78,60 @@ class GovernorConfig(BaseModel):
|
|
| 75 |
hourly_budget_usd: float | None = None
|
| 76 |
|
| 77 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 78 |
# ── scenario ─────────────────────────────────────────────────────────────────────
|
| 79 |
|
| 80 |
|
|
@@ -96,6 +153,12 @@ class ScenarioConfig(BaseModel):
|
|
| 96 |
genesis_text: str | None = None
|
| 97 |
governor: GovernorConfig | None = None
|
| 98 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 99 |
|
| 100 |
# ── the whole world ──────────────────────────────────────────────────────────────
|
| 101 |
|
|
@@ -117,7 +180,8 @@ class WorldConfig(BaseModel):
|
|
| 117 |
|
| 118 |
@model_validator(mode="after")
|
| 119 |
def _check_cast_references(self) -> "WorldConfig":
|
| 120 |
-
|
|
|
|
| 121 |
for scenario in self.scenarios:
|
| 122 |
missing = [name for name in scenario.cast if name not in defined]
|
| 123 |
if missing:
|
|
@@ -125,8 +189,40 @@ class WorldConfig(BaseModel):
|
|
| 125 |
f"scenario {scenario.name!r} references undefined agents: {missing}. "
|
| 126 |
f"Defined agents: {sorted(defined)}"
|
| 127 |
)
|
|
|
|
| 128 |
return self
|
| 129 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 130 |
|
| 131 |
# ── validation entrypoints (the 'configure from a prompt' surface) ───────────────
|
| 132 |
|
|
|
|
| 16 |
The agent schema itself is :class:`AgentManifest` (``src/core/manifest.py``) — we
|
| 17 |
reuse it here rather than duplicating, so the four stable contracts stay singular.
|
| 18 |
"""
|
| 19 |
+
|
| 20 |
from __future__ import annotations
|
| 21 |
|
| 22 |
+
from typing import Literal
|
| 23 |
+
|
| 24 |
from pydantic import BaseModel, ConfigDict, Field, model_validator
|
| 25 |
|
| 26 |
from src.core.manifest import AgentManifest
|
|
|
|
| 78 |
hourly_budget_usd: float | None = None
|
| 79 |
|
| 80 |
|
| 81 |
+
# ── competition ──────────────────────────────────────────────────────────────────
|
| 82 |
+
|
| 83 |
+
CompetitionKind = Literal["versus", "judged", "none"]
|
| 84 |
+
|
| 85 |
+
|
| 86 |
+
class CompetitionConfig(BaseModel):
|
| 87 |
+
"""How (and whether) a scenario produces a winner — the arena contract (ADR-0029).
|
| 88 |
+
|
| 89 |
+
Three kinds:
|
| 90 |
+
* ``versus`` — a head-to-head between symmetric seats or named teams; the
|
| 91 |
+
winner is decided either by ground-truth code (The Steeped, Twenty Sprouts)
|
| 92 |
+
or by a judge naming a side. Use ``teams`` for asymmetric sides and
|
| 93 |
+
``symmetric_seats`` for "same manifest, different model" duels (Debate Duel,
|
| 94 |
+
Beat Battle) — the latter is what makes the model-leaderboard meaningful.
|
| 95 |
+
* ``judged`` — a judge in the cast names the winning *agent* via its
|
| 96 |
+
``judge.verdict`` payload ``winner`` field (Mystery Roots, Open Table).
|
| 97 |
+
* ``none`` — collaborative or showcase; no winner, no leaderboard rows, but
|
| 98 |
+
still a full session/history (Thousand Token Wood, Oracle Grove).
|
| 99 |
+
|
| 100 |
+
The block is stamped onto ``run.started`` so a run is self-describing forever —
|
| 101 |
+
the leaderboard (W6) reads it to know which runs produce winners and how to
|
| 102 |
+
attribute them. Team / seat members are validated against the scenario cast in
|
| 103 |
+
:meth:`WorldConfig._check_cast_references`.
|
| 104 |
+
"""
|
| 105 |
+
|
| 106 |
+
model_config = ConfigDict(extra="forbid")
|
| 107 |
+
|
| 108 |
+
kind: CompetitionKind = "none"
|
| 109 |
+
|
| 110 |
+
teams: dict[str, list[str]] | None = None
|
| 111 |
+
"""Named sides, e.g. ``{spy: [spy-nil], herd: [spy-cara, spy-bex, spy-ovo]}``.
|
| 112 |
+
Each member must be in the scenario cast. ``versus`` only."""
|
| 113 |
+
|
| 114 |
+
symmetric_seats: list[str] | None = None
|
| 115 |
+
"""Cast members occupying *identical* seats that differ only by model — the
|
| 116 |
+
"which model argues better" comparison. ``versus`` only; needs ≥2 entries."""
|
| 117 |
+
|
| 118 |
+
@model_validator(mode="after")
|
| 119 |
+
def _check_kind_shape(self) -> "CompetitionConfig":
|
| 120 |
+
if self.kind == "none":
|
| 121 |
+
if self.teams or self.symmetric_seats:
|
| 122 |
+
raise ValueError("competition.kind 'none' must not declare teams or symmetric_seats")
|
| 123 |
+
elif self.kind == "versus":
|
| 124 |
+
n_teams = len(self.teams or {})
|
| 125 |
+
n_seats = len(self.symmetric_seats or [])
|
| 126 |
+
if n_teams < 2 and n_seats < 2:
|
| 127 |
+
raise ValueError(
|
| 128 |
+
"competition.kind 'versus' needs ≥2 teams or ≥2 symmetric_seats "
|
| 129 |
+
f"(got teams={n_teams}, symmetric_seats={n_seats})"
|
| 130 |
+
)
|
| 131 |
+
# 'judged' carries no required structural fields — the judge names the winner.
|
| 132 |
+
return self
|
| 133 |
+
|
| 134 |
+
|
| 135 |
# ── scenario ─────────────────────────────────────────────────────────────────────
|
| 136 |
|
| 137 |
|
|
|
|
| 153 |
genesis_text: str | None = None
|
| 154 |
governor: GovernorConfig | None = None
|
| 155 |
|
| 156 |
+
competition: CompetitionConfig = Field(default_factory=CompetitionConfig)
|
| 157 |
+
"""The arena contract: how this scenario produces a winner (ADR-0029). Defaulted
|
| 158 |
+
to ``kind: none`` so a scenario without a block still validates, but the authoring
|
| 159 |
+
checklist (and ``tests/test_scenario_contract.py``) requires an explicit block on
|
| 160 |
+
every shipped scenario."""
|
| 161 |
+
|
| 162 |
|
| 163 |
# ── the whole world ──────────────────────────────────────────────────────────────
|
| 164 |
|
|
|
|
| 180 |
|
| 181 |
@model_validator(mode="after")
|
| 182 |
def _check_cast_references(self) -> "WorldConfig":
|
| 183 |
+
by_name = {a.name: a for a in self.agents}
|
| 184 |
+
defined = set(by_name)
|
| 185 |
for scenario in self.scenarios:
|
| 186 |
missing = [name for name in scenario.cast if name not in defined]
|
| 187 |
if missing:
|
|
|
|
| 189 |
f"scenario {scenario.name!r} references undefined agents: {missing}. "
|
| 190 |
f"Defined agents: {sorted(defined)}"
|
| 191 |
)
|
| 192 |
+
self._check_competition(scenario, by_name)
|
| 193 |
return self
|
| 194 |
|
| 195 |
+
@staticmethod
|
| 196 |
+
def _check_competition(scenario: ScenarioConfig, by_name: dict[str, AgentManifest]) -> None:
|
| 197 |
+
"""Cross-cast rules for the competition contract (ADR-0029).
|
| 198 |
+
|
| 199 |
+
Self-contained shape rules live on :class:`CompetitionConfig`; the checks that
|
| 200 |
+
need the cast + agent registry live here: team/seat members must be in the
|
| 201 |
+
cast, and a winner-bearing kind (versus / judged) must include a judge that
|
| 202 |
+
actually emits ``judge.verdict``. ``none`` scenarios require no judge.
|
| 203 |
+
"""
|
| 204 |
+
comp = scenario.competition
|
| 205 |
+
cast = set(scenario.cast)
|
| 206 |
+
members: list[str] = list(scenario.competition.symmetric_seats or [])
|
| 207 |
+
for team in (comp.teams or {}).values():
|
| 208 |
+
members.extend(team)
|
| 209 |
+
stray = sorted(m for m in members if m not in cast)
|
| 210 |
+
if stray:
|
| 211 |
+
raise ValueError(
|
| 212 |
+
f"scenario {scenario.name!r} competition references non-cast members: {stray}. Cast: {sorted(cast)}"
|
| 213 |
+
)
|
| 214 |
+
if comp.kind in ("versus", "judged"):
|
| 215 |
+
judges = [
|
| 216 |
+
name
|
| 217 |
+
for name in scenario.cast
|
| 218 |
+
if (m := by_name.get(name)) and m.role == "judge" and "judge.verdict" in m.may_emit
|
| 219 |
+
]
|
| 220 |
+
if not judges:
|
| 221 |
+
raise ValueError(
|
| 222 |
+
f"scenario {scenario.name!r} competition.kind {comp.kind!r} requires a cast member "
|
| 223 |
+
"with role 'judge' that emits 'judge.verdict' to decide the winner"
|
| 224 |
+
)
|
| 225 |
+
|
| 226 |
|
| 227 |
# ── validation entrypoints (the 'configure from a prompt' surface) ───────────────
|
| 228 |
|
|
@@ -253,6 +253,7 @@ class Registry:
|
|
| 253 |
example_seeds=cfg.example_seeds,
|
| 254 |
goal=cfg.goal,
|
| 255 |
genesis_text=cfg.genesis_text,
|
|
|
|
| 256 |
)
|
| 257 |
|
| 258 |
def governor_for(self, name: str) -> Governor:
|
|
@@ -292,4 +293,6 @@ def default_registry() -> Registry:
|
|
| 292 |
|
| 293 |
# Load behaviour handlers so their @register_handler side effects run. Imported
|
| 294 |
# at the bottom, after register_handler is defined, so there is no import cycle.
|
|
|
|
| 295 |
from src.agents import handlers as _handlers # noqa: E402,F401
|
|
|
|
|
|
| 253 |
example_seeds=cfg.example_seeds,
|
| 254 |
goal=cfg.goal,
|
| 255 |
genesis_text=cfg.genesis_text,
|
| 256 |
+
competition=cfg.competition,
|
| 257 |
)
|
| 258 |
|
| 259 |
def governor_for(self, name: str) -> Governor:
|
|
|
|
| 293 |
|
| 294 |
# Load behaviour handlers so their @register_handler side effects run. Imported
|
| 295 |
# at the bottom, after register_handler is defined, so there is no import cycle.
|
| 296 |
+
from src.agents import competition as _competition # noqa: E402,F401
|
| 297 |
from src.agents import handlers as _handlers # noqa: E402,F401
|
| 298 |
+
from src.agents import twenty_sprouts as _twenty_sprouts # noqa: E402,F401
|
|
@@ -91,6 +91,18 @@ _STUB_MOODS_BY_ROLE: dict[str, tuple[str, ...]] = {
|
|
| 91 |
"spy-cara": ("calm", "smug", "calm"),
|
| 92 |
"spy-ovo": ("thinking", "calm", "calm"),
|
| 93 |
"spy-host": ("smug", "calm", "truth"),
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 94 |
}
|
| 95 |
|
| 96 |
# Curated private monologue per role, paired with the public ``text`` lines to make the
|
|
@@ -246,6 +258,65 @@ class DeterministicTinyModel(ModelProvider):
|
|
| 246 |
"Let me nudge us forward: what does the square need most, right now?",
|
| 247 |
"Lovely tension here — say more about who this is really for.",
|
| 248 |
],
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 249 |
}
|
| 250 |
options = choices.get(role, ["The wood hums and waits."])
|
| 251 |
text = options[int(digest[:2], 16) % len(options)]
|
|
@@ -301,5 +372,11 @@ class DeterministicTinyModel(ModelProvider):
|
|
| 301 |
if name == "thought":
|
| 302 |
opts = _STUB_THOUGHTS.get(role, _STUB_THOUGHT_DEFAULT)
|
| 303 |
return opts[int(digest[6:8], 16) % len(opts)]
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 304 |
# Unknown extra field: a short, stable placeholder keeps the output valid.
|
| 305 |
return f"{name}:{digest[:4]}"
|
|
|
|
| 91 |
"spy-cara": ("calm", "smug", "calm"),
|
| 92 |
"spy-ovo": ("thinking", "calm", "calm"),
|
| 93 |
"spy-host": ("smug", "calm", "truth"),
|
| 94 |
+
# ── arena judges + competitors (ADR-0029) ──────────────────────────────────
|
| 95 |
+
"mystery-judge": ("thinking", "truth", "calm"),
|
| 96 |
+
"table-judge": ("calm", "thinking", "truth"),
|
| 97 |
+
"debater-a": ("smug", "panic", "smug", "calm"),
|
| 98 |
+
"debater-b": ("calm", "smug", "thinking", "smug"),
|
| 99 |
+
"debate-judge": ("smug", "calm", "truth"),
|
| 100 |
+
"storyteller-a": ("thinking", "calm", "smug", "truth"),
|
| 101 |
+
"storyteller-b": ("calm", "thinking", "gossip", "truth"),
|
| 102 |
+
"beat-judge": ("thinking", "calm", "truth"),
|
| 103 |
+
"secret-keeper": ("smug", "calm", "gossip", "thinking"),
|
| 104 |
+
"sprout-guesser": ("thinking", "thinking", "calm", "smug"),
|
| 105 |
+
"sprout-judge": ("calm", "truth", "thinking"),
|
| 106 |
}
|
| 107 |
|
| 108 |
# Curated private monologue per role, paired with the public ``text`` lines to make the
|
|
|
|
| 258 |
"Let me nudge us forward: what does the square need most, right now?",
|
| 259 |
"Lovely tension here — say more about who this is really for.",
|
| 260 |
],
|
| 261 |
+
# ── mystery roots / open table judges (ADR-0029) ────────────────────
|
| 262 |
+
"mystery-judge": [
|
| 263 |
+
"Verdict: the most likely truth is hypothesis-former's — the clue and the cause line up, and that ordering is what convinces me.",
|
| 264 |
+
"Verdict: I endorse hypothesis-former's reading; it is the one explanation that leaves no clue stranded.",
|
| 265 |
+
"Verdict: the evidence bends toward hypothesis-former's account — specific, testable, and unbroken by the doubt raised against it.",
|
| 266 |
+
],
|
| 267 |
+
"table-judge": [
|
| 268 |
+
"Verdict: chat-skeptic was the most persuasive voice — the point about who tends it after dark is the one I can't argue away.",
|
| 269 |
+
"Verdict: I crown chat-curious; the question of what the village still loves in ten years reframed the whole table.",
|
| 270 |
+
"Verdict: chat-skeptic takes it — turning comfort-today against shade-we-never-sit-under was the sharpest cut of the hour.",
|
| 271 |
+
],
|
| 272 |
+
# ── debate duel (symmetric seats, different models) ─────────────────
|
| 273 |
+
"debater-a": [
|
| 274 |
+
"The bold path is always the right one — hesitation is just defeat in a slower coat.",
|
| 275 |
+
"My opponent mistakes caution for wisdom; history rewards the daring, not the timid.",
|
| 276 |
+
"Strip away the fear and what remains is obvious: we must act, and act now.",
|
| 277 |
+
],
|
| 278 |
+
"debater-b": [
|
| 279 |
+
"Every reckless 'yes' has a graveyard of consequences my opponent conveniently forgets.",
|
| 280 |
+
"Restraint is not weakness — it is the only argument that survives the morning after.",
|
| 281 |
+
"You call it boldness; I call it a beautifully worded mistake.",
|
| 282 |
+
],
|
| 283 |
+
"debate-judge": [
|
| 284 |
+
"Verdict: debater-a takes it — that line about history rewarding the daring landed clean and never wavered.",
|
| 285 |
+
"Verdict: debater-b wins on the strength of 'the morning after,' the sharpest blow of the duel.",
|
| 286 |
+
"Verdict: debater-a, by a hair — the closing call to act now outpunched every rebuttal.",
|
| 287 |
+
],
|
| 288 |
+
# ── beat battle (symmetric seats, different models) ─────────────────
|
| 289 |
+
"storyteller-a": [
|
| 290 |
+
"The lighthouse keeper unfolds a wave that has signed its name in foam and three patient question marks.",
|
| 291 |
+
"By dawn the gulls are reciting the sea's letters aloud, and one of them has started to weep with joy.",
|
| 292 |
+
"A single drop climbs the spiral stair, knocks politely, and asks to borrow the lamp for a love note.",
|
| 293 |
+
],
|
| 294 |
+
"storyteller-b": [
|
| 295 |
+
"The tide leaves a sealed envelope of kelp on the top step, still warm from somewhere far below.",
|
| 296 |
+
"Tonight the beam writes back in light, and the whole bay holds its breath to read the reply.",
|
| 297 |
+
"The keeper discovers the sea has been practicing his own handwriting, only kinder, only braver.",
|
| 298 |
+
],
|
| 299 |
+
"beat-judge": [
|
| 300 |
+
"Verdict: storyteller-a wins — their beats turned a single wave into a character we ached for, surprising and warm in one breath.",
|
| 301 |
+
"Verdict: storyteller-b takes it, every line opening a door the last one only hinted at, delight stacked on delight.",
|
| 302 |
+
"Verdict: storyteller-a, by a whisper — the weeping gull was the kind of impossible detail that makes a tale sing.",
|
| 303 |
+
],
|
| 304 |
+
# ── twenty sprouts (code-dealt secret word) ──────────────────���──────
|
| 305 |
+
"secret-keeper": [
|
| 306 |
+
"Yes — you could hold it in one hand, if your hand were patient enough.",
|
| 307 |
+
"No, it was never alive, though plenty of living things have leaned on it.",
|
| 308 |
+
"Warmer now — it does belong to the wood, but not to any creature in it.",
|
| 309 |
+
],
|
| 310 |
+
"sprout-guesser": [
|
| 311 |
+
"Is the thing you're holding something a traveller would carry on the path?",
|
| 312 |
+
"Does it make a sound, or is it silent until someone uses it?",
|
| 313 |
+
"Then is it older than the trees, or younger than this morning's dew?",
|
| 314 |
+
],
|
| 315 |
+
"sprout-judge": [
|
| 316 |
+
"Verdict: the keeper kept its secret — the guesser circled close but never named the word.",
|
| 317 |
+
"Verdict: a clean catch — the guesser cornered the word before the questions ran dry.",
|
| 318 |
+
"Verdict: the grove falls quiet; the secret held, and the keeper smiles.",
|
| 319 |
+
],
|
| 320 |
}
|
| 321 |
options = choices.get(role, ["The wood hums and waits."])
|
| 322 |
text = options[int(digest[:2], 16) % len(options)]
|
|
|
|
| 372 |
if name == "thought":
|
| 373 |
opts = _STUB_THOUGHTS.get(role, _STUB_THOUGHT_DEFAULT)
|
| 374 |
return opts[int(digest[6:8], 16) % len(opts)]
|
| 375 |
+
if name in ("winner", "scores"):
|
| 376 |
+
# The stub can't know the live cast, so it must NOT invent a fake winner
|
| 377 |
+
# name (it would surface as junk in the verdict). Leave it empty: the
|
| 378 |
+
# judge's handler (JudgedCompetition / ground-truth subclasses) derives the
|
| 379 |
+
# real winner from the on-stage cast. See ADR-0029.
|
| 380 |
+
return ""
|
| 381 |
# Unknown extra field: a short, stable placeholder keeps the output valid.
|
| 382 |
return f"{name}:{digest[:4]}"
|
|
@@ -4,6 +4,7 @@ from collections.abc import Iterable
|
|
| 4 |
from dataclasses import dataclass, field
|
| 5 |
|
| 6 |
from src.agents.base import Agent
|
|
|
|
| 7 |
from src.core.events import Event
|
| 8 |
|
| 9 |
|
|
@@ -20,6 +21,10 @@ class Scenario:
|
|
| 20 |
genesis_text: str | None = None
|
| 21 |
"""Template for the opening world.observed event. '{seed}' is substituted.
|
| 22 |
Falls back to a generic clearing line when None."""
|
|
|
|
|
|
|
|
|
|
|
|
|
| 23 |
|
| 24 |
def genesis(self, run_id: str, turn: int, seed: str) -> Iterable[Event]:
|
| 25 |
template = self.genesis_text or "The first clearing forms around '{seed}'."
|
|
|
|
| 4 |
from dataclasses import dataclass, field
|
| 5 |
|
| 6 |
from src.agents.base import Agent
|
| 7 |
+
from src.core.config import CompetitionConfig
|
| 8 |
from src.core.events import Event
|
| 9 |
|
| 10 |
|
|
|
|
| 21 |
genesis_text: str | None = None
|
| 22 |
"""Template for the opening world.observed event. '{seed}' is substituted.
|
| 23 |
Falls back to a generic clearing line when None."""
|
| 24 |
+
competition: CompetitionConfig | None = None
|
| 25 |
+
"""The arena contract (ADR-0029) — how this scenario produces a winner. Stamped
|
| 26 |
+
onto ``run.started`` by the Conductor so a run is self-describing. ``None`` (the
|
| 27 |
+
legacy/test default) behaves like ``kind: none``: no winner, no leaderboard rows."""
|
| 28 |
|
| 29 |
def genesis(self, run_id: str, turn: int, seed: str) -> Iterable[Event]:
|
| 30 |
template = self.genesis_text or "The first clearing forms around '{seed}'."
|
|
@@ -308,7 +308,15 @@ def build_telemetry() -> None:
|
|
| 308 |
_registry = default_registry()
|
| 309 |
_tools = default_tool_registry()
|
| 310 |
|
| 311 |
-
_PREFERRED = [
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 312 |
_names = [n for n in _PREFERRED if n in _registry.scenarios] + [
|
| 313 |
n for n in sorted(_registry.scenarios) if n not in _PREFERRED
|
| 314 |
]
|
|
|
|
| 308 |
_registry = default_registry()
|
| 309 |
_tools = default_tool_registry()
|
| 310 |
|
| 311 |
+
_PREFERRED = [
|
| 312 |
+
"thousand-token-wood",
|
| 313 |
+
"mystery-roots",
|
| 314 |
+
"oracle-grove",
|
| 315 |
+
"the-steeped",
|
| 316 |
+
"debate-duel",
|
| 317 |
+
"twenty-sprouts",
|
| 318 |
+
"beat-battle",
|
| 319 |
+
]
|
| 320 |
_names = [n for n in _PREFERRED if n in _registry.scenarios] + [
|
| 321 |
n for n in sorted(_registry.scenarios) if n not in _PREFERRED
|
| 322 |
]
|
|
@@ -778,6 +778,8 @@ footer { display: none !important; }
|
|
| 778 |
@keyframes vbDrop { from { opacity: 0; transform: translateX(-50%) translateY(-20px); } to { opacity: 1; } }
|
| 779 |
.fishbowl .verdict-banner .disp { color: var(--lime); font-size: 13px; letter-spacing: 0.14em; text-shadow: 0 0 10px var(--lime); }
|
| 780 |
.fishbowl .vb-text { color: var(--ink); font-size: 12.5px; }
|
|
|
|
|
|
|
| 781 |
|
| 782 |
/* ---- poke strip ---- */
|
| 783 |
.fishbowl .poke-strip { display: flex; align-items: center; gap: 14px; padding: 10px 20px; border-top: 1px solid var(--line-soft); background: rgba(6,20,27,0.5); flex: none; }
|
|
|
|
| 778 |
@keyframes vbDrop { from { opacity: 0; transform: translateX(-50%) translateY(-20px); } to { opacity: 1; } }
|
| 779 |
.fishbowl .verdict-banner .disp { color: var(--lime); font-size: 13px; letter-spacing: 0.14em; text-shadow: 0 0 10px var(--lime); }
|
| 780 |
.fishbowl .vb-text { color: var(--ink); font-size: 12.5px; }
|
| 781 |
+
/* Winner ribbon — the arena's "who won" chip; only present for competitive runs. */
|
| 782 |
+
.fishbowl .vb-winner { font-family: var(--font-display); font-size: 12px; letter-spacing: 0.16em; text-transform: uppercase; color: var(--amber); text-shadow: 0 0 12px rgba(255,207,107,0.55); white-space: nowrap; }
|
| 783 |
|
| 784 |
/* ---- poke strip ---- */
|
| 785 |
.fishbowl .poke-strip { display: flex; align-items: center; gap: 14px; padding: 10px 20px; border-top: 1px solid var(--line-soft); background: rgba(6,20,27,0.5); flex: none; }
|
|
@@ -119,6 +119,18 @@ def render_verdict(vm: dict) -> str:
|
|
| 119 |
text = html.escape(str(verdict.get("text", "")))
|
| 120 |
reveal = verdict.get("reveal") or []
|
| 121 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 122 |
lines = ""
|
| 123 |
for r in reveal:
|
| 124 |
agent = html.escape(str(r.get("agent", "")))
|
|
@@ -135,6 +147,7 @@ def render_verdict(vm: dict) -> str:
|
|
| 135 |
return (
|
| 136 |
'<div class="verdict banner">'
|
| 137 |
'<div class="eyebrow">⚖ Verdict</div>'
|
|
|
|
| 138 |
f'<div class="disp vb-text">{text}</div>'
|
| 139 |
f"{lines}"
|
| 140 |
"</div>"
|
|
|
|
| 119 |
text = html.escape(str(verdict.get("text", "")))
|
| 120 |
reveal = verdict.get("reveal") or []
|
| 121 |
|
| 122 |
+
# Winner ribbon — rendered only when the run declared a winner (ADR-0029). A
|
| 123 |
+
# ``none``-kind scenario (Wood, Oracle) and any legacy verdict carry no ``winner``,
|
| 124 |
+
# so this is empty and the banner is byte-identical to before.
|
| 125 |
+
ribbon = ""
|
| 126 |
+
label = verdict.get("winner_label")
|
| 127 |
+
if label:
|
| 128 |
+
correct = verdict.get("correct")
|
| 129 |
+
flair = "🏆" # 🏆
|
| 130 |
+
if correct is False:
|
| 131 |
+
flair = "🥷" # 🦝 — the spy slipped away (ground-truth miss)
|
| 132 |
+
ribbon = f'<div class="vb-winner disp">{flair} {html.escape(str(label))}</div>'
|
| 133 |
+
|
| 134 |
lines = ""
|
| 135 |
for r in reveal:
|
| 136 |
agent = html.escape(str(r.get("agent", "")))
|
|
|
|
| 147 |
return (
|
| 148 |
'<div class="verdict banner">'
|
| 149 |
'<div class="eyebrow">⚖ Verdict</div>'
|
| 150 |
+
f"{ribbon}"
|
| 151 |
f'<div class="disp vb-text">{text}</div>'
|
| 152 |
f"{lines}"
|
| 153 |
"</div>"
|
|
@@ -22,7 +22,15 @@ from src.ui.fishbowl.view_model import view_model_at
|
|
| 22 |
|
| 23 |
# Preferred display order, mirroring root app.py. Any other scenarios dropped into
|
| 24 |
# config/ follow in sorted order.
|
| 25 |
-
_PREFERRED = [
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 26 |
|
| 27 |
|
| 28 |
def _ordered_names(registry: Registry) -> list[str]:
|
|
@@ -115,9 +123,13 @@ class FishbowlSession:
|
|
| 115 |
def finalize(self, reason: str) -> None:
|
| 116 |
"""Close the current run with a ``run.finished`` event (idempotent-safe).
|
| 117 |
|
| 118 |
-
On a verdict we derive ``winner`` from the judge's ruling (
|
| 119 |
-
|
| 120 |
-
cast
|
|
|
|
|
|
|
|
|
|
|
|
|
| 121 |
winner: str | None = None
|
| 122 |
winning_model: str | None = None
|
| 123 |
run_events = self.conductor.ledger.events_for_run(self.conductor.run_id)
|
|
@@ -127,8 +139,13 @@ class FishbowlSession:
|
|
| 127 |
winner = verdict.payload.get("winner") or None
|
| 128 |
if winner:
|
| 129 |
started = next((e for e in run_events if e.kind == "run.started"), None)
|
| 130 |
-
|
| 131 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 132 |
self.conductor.finalize(reason, winner=winner, winning_model=winning_model)
|
| 133 |
|
| 134 |
@property
|
|
|
|
| 22 |
|
| 23 |
# Preferred display order, mirroring root app.py. Any other scenarios dropped into
|
| 24 |
# config/ follow in sorted order.
|
| 25 |
+
_PREFERRED = [
|
| 26 |
+
"thousand-token-wood",
|
| 27 |
+
"mystery-roots",
|
| 28 |
+
"oracle-grove",
|
| 29 |
+
"the-steeped",
|
| 30 |
+
"debate-duel",
|
| 31 |
+
"twenty-sprouts",
|
| 32 |
+
"beat-battle",
|
| 33 |
+
]
|
| 34 |
|
| 35 |
|
| 36 |
def _ordered_names(registry: Registry) -> list[str]:
|
|
|
|
| 123 |
def finalize(self, reason: str) -> None:
|
| 124 |
"""Close the current run with a ``run.finished`` event (idempotent-safe).
|
| 125 |
|
| 126 |
+
On a verdict we derive ``winner`` from the judge's ruling (the ``winner``
|
| 127 |
+
payload key) and ``winning_model`` from the run.started cast map. The winner
|
| 128 |
+
may be a cast *agent name* (judged scenarios → maps straight to its model) or a
|
| 129 |
+
*team label* (versus scenarios, e.g. ``"herd"``). For a team we attribute the
|
| 130 |
+
model only when the team has exactly one member; multi-member teams have no
|
| 131 |
+
single winning model (the seat, not a model, won) — the leaderboard credits the
|
| 132 |
+
team. Everything falls back to ``None`` when unknown."""
|
| 133 |
winner: str | None = None
|
| 134 |
winning_model: str | None = None
|
| 135 |
run_events = self.conductor.ledger.events_for_run(self.conductor.run_id)
|
|
|
|
| 139 |
winner = verdict.payload.get("winner") or None
|
| 140 |
if winner:
|
| 141 |
started = next((e for e in run_events if e.kind == "run.started"), None)
|
| 142 |
+
started_payload = started.payload if started is not None else {}
|
| 143 |
+
cast = started_payload.get("cast") or {}
|
| 144 |
+
teams = (started_payload.get("competition") or {}).get("teams") or {}
|
| 145 |
+
if winner in cast:
|
| 146 |
+
winning_model = (cast.get(winner) or {}).get("model_endpoint")
|
| 147 |
+
elif winner in teams and len(teams[winner]) == 1:
|
| 148 |
+
winning_model = (cast.get(teams[winner][0]) or {}).get("model_endpoint")
|
| 149 |
self.conductor.finalize(reason, winner=winner, winning_model=winning_model)
|
| 150 |
|
| 151 |
@property
|
|
@@ -35,6 +35,18 @@ from src.ui.fishbowl.cast_state import derive_cast_state
|
|
| 35 |
_SPEAKING_KINDS = frozenset({"agent.spoke", "agent.thought", "oracle.spoke", "judge.verdict"})
|
| 36 |
|
| 37 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 38 |
def _estimate_tokens_through(events: Sequence[Event]) -> int:
|
| 39 |
"""A real-text token estimate for the scrubber meter (grows as you advance)."""
|
| 40 |
total = 0
|
|
@@ -111,13 +123,25 @@ def view_model_at(
|
|
| 111 |
item["turn"] = e.turn
|
| 112 |
feed.append(item)
|
| 113 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 114 |
verdict = None
|
| 115 |
for e in prefix:
|
| 116 |
if e.kind == "judge.verdict":
|
|
|
|
| 117 |
verdict = {
|
| 118 |
"text": e.payload.get("text", ""),
|
| 119 |
"reveal": e.payload.get("reveal", []),
|
| 120 |
"agent": e.actor,
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 121 |
}
|
| 122 |
|
| 123 |
# "Round" = the live iteration count, i.e. the sim-turn at the play-head, so the
|
|
|
|
| 35 |
_SPEAKING_KINDS = frozenset({"agent.spoke", "agent.thought", "oracle.spoke", "judge.verdict"})
|
| 36 |
|
| 37 |
|
| 38 |
+
def _winner_label(winner: str | None, teams: dict) -> str | None:
|
| 39 |
+
"""Human-readable winner for the banner ribbon.
|
| 40 |
+
|
| 41 |
+
A team label reads as "Team Herd"; an agent slug reads as "Hypothesis Former".
|
| 42 |
+
Returns ``None`` when there is no winner, so ``none``-kind scenarios stay ribbon-less.
|
| 43 |
+
"""
|
| 44 |
+
if not winner:
|
| 45 |
+
return None
|
| 46 |
+
pretty = winner.replace("-", " ").replace("_", " ").title()
|
| 47 |
+
return f"Team {pretty}" if winner in teams else pretty
|
| 48 |
+
|
| 49 |
+
|
| 50 |
def _estimate_tokens_through(events: Sequence[Event]) -> int:
|
| 51 |
"""A real-text token estimate for the scrubber meter (grows as you advance)."""
|
| 52 |
total = 0
|
|
|
|
| 123 |
item["turn"] = e.turn
|
| 124 |
feed.append(item)
|
| 125 |
|
| 126 |
+
# The arena contract for this run (ADR-0029), stamped on run.started — lets the
|
| 127 |
+
# verdict banner label a team win ("herd wins") apart from an agent win.
|
| 128 |
+
competition = next((e.payload.get("competition") for e in prefix if e.kind == "run.started"), None) or {}
|
| 129 |
+
teams = competition.get("teams") or {}
|
| 130 |
+
|
| 131 |
verdict = None
|
| 132 |
for e in prefix:
|
| 133 |
if e.kind == "judge.verdict":
|
| 134 |
+
winner = e.payload.get("winner") or None
|
| 135 |
verdict = {
|
| 136 |
"text": e.payload.get("text", ""),
|
| 137 |
"reveal": e.payload.get("reveal", []),
|
| 138 |
"agent": e.actor,
|
| 139 |
+
"winner": winner,
|
| 140 |
+
# A team win names a side; an agent win names a mind. ``correct`` (when a
|
| 141 |
+
# ground-truth judge set it) drives the win/miss flavour in the banner.
|
| 142 |
+
"winner_label": _winner_label(winner, teams),
|
| 143 |
+
"winner_kind": "team" if winner in teams else ("agent" if winner else None),
|
| 144 |
+
"correct": e.payload.get("correct"),
|
| 145 |
}
|
| 146 |
|
| 147 |
# "Round" = the live iteration count, i.e. the sim-turn at the play-head, so the
|
|
@@ -334,9 +334,10 @@ def test_collect_world_config_honours_roster_genesis_and_governor():
|
|
| 334 |
|
| 335 |
|
| 336 |
def test_collect_world_config_judgeless_world_is_valid_without_judge_knobs():
|
| 337 |
-
#
|
|
|
|
| 338 |
registry = default_registry()
|
| 339 |
-
scenario = registry.scenarios["
|
| 340 |
world = lab.collect_world_config(
|
| 341 |
scenario=scenario.name,
|
| 342 |
premise="",
|
|
|
|
| 334 |
|
| 335 |
|
| 336 |
def test_collect_world_config_judgeless_world_is_valid_without_judge_knobs():
|
| 337 |
+
# oracle-grove is a judge-less tool-use showcase; composing it with no judge model
|
| 338 |
+
# must still validate. (Open Table gained a table-judge with the arena verdict.)
|
| 339 |
registry = default_registry()
|
| 340 |
+
scenario = registry.scenarios["oracle-grove"]
|
| 341 |
world = lab.collect_world_config(
|
| 342 |
scenario=scenario.name,
|
| 343 |
premise="",
|
|
@@ -32,10 +32,12 @@ class TestOpenTableRegistry:
|
|
| 32 |
assert "open-table" in reg.scenarios
|
| 33 |
assert {"chat-curious", "chat-skeptic", "chat-host"} <= set(reg.agents)
|
| 34 |
|
| 35 |
-
def
|
| 36 |
reg = default_registry()
|
| 37 |
sc = reg.build_scenario("open-table")
|
| 38 |
-
|
|
|
|
|
|
|
| 39 |
assert all(isinstance(a, ManifestAgent) for a in sc.agents)
|
| 40 |
assert sc.goal
|
| 41 |
|
|
|
|
| 32 |
assert "open-table" in reg.scenarios
|
| 33 |
assert {"chat-curious", "chat-skeptic", "chat-host"} <= set(reg.agents)
|
| 34 |
|
| 35 |
+
def test_scenario_builds_with_its_manifest_cast(self):
|
| 36 |
reg = default_registry()
|
| 37 |
sc = reg.build_scenario("open-table")
|
| 38 |
+
# The three conversational voices plus the table-judge that names the most
|
| 39 |
+
# persuasive of them at the end (the arena verdict, ADR-0029).
|
| 40 |
+
assert [a.name for a in sc.agents] == ["chat-curious", "chat-skeptic", "chat-host", "table-judge"]
|
| 41 |
assert all(isinstance(a, ManifestAgent) for a in sc.agents)
|
| 42 |
assert sc.goal
|
| 43 |
|
|
@@ -15,8 +15,10 @@ from src.core.registry import default_registry
|
|
| 15 |
from src.ui.fishbowl.scenario_caps import scenario_ui_caps
|
| 16 |
|
| 17 |
# Scenarios with / without a judge in their stock cast (per the capability matrix).
|
| 18 |
-
|
| 19 |
-
|
|
|
|
|
|
|
| 20 |
|
| 21 |
|
| 22 |
@pytest.mark.parametrize("name", _JUDGELESS)
|
|
|
|
| 15 |
from src.ui.fishbowl.scenario_caps import scenario_ui_caps
|
| 16 |
|
| 17 |
# Scenarios with / without a judge in their stock cast (per the capability matrix).
|
| 18 |
+
# Open Table gained a table-judge with the arena verdict (ADR-0029); Oracle Grove stays
|
| 19 |
+
# a judge-less tool-use showcase.
|
| 20 |
+
_JUDGED = ["thousand-token-wood", "mystery-roots", "the-steeped", "open-table"]
|
| 21 |
+
_JUDGELESS = ["oracle-grove"]
|
| 22 |
|
| 23 |
|
| 24 |
@pytest.mark.parametrize("name", _JUDGELESS)
|
|
@@ -0,0 +1,154 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""The scenario authoring checklist, enforced.
|
| 2 |
+
|
| 3 |
+
Every shipped scenario must be *arena-grade* (ADR-0029): premise + cast + governor +
|
| 4 |
+
an explicit ``competition`` block + a real end condition + a structured verdict path.
|
| 5 |
+
These tests load every ``config/scenarios/*.yaml`` and assert that contract — entirely
|
| 6 |
+
offline, no tokens — so a new scenario can't ship half-wired. See
|
| 7 |
+
``docs/architecture/scenario-authoring.md`` for the human-readable checklist.
|
| 8 |
+
"""
|
| 9 |
+
|
| 10 |
+
from __future__ import annotations
|
| 11 |
+
|
| 12 |
+
from pathlib import Path
|
| 13 |
+
|
| 14 |
+
import pytest
|
| 15 |
+
import yaml
|
| 16 |
+
|
| 17 |
+
from src.core.config import WorldConfig
|
| 18 |
+
from src.core.conductor import Conductor
|
| 19 |
+
from src.core.governor import BudgetExceeded
|
| 20 |
+
from src.core.registry import Registry
|
| 21 |
+
from src.models.router import ModelRouter
|
| 22 |
+
from src.tools.builtins import default_tool_registry
|
| 23 |
+
|
| 24 |
+
_CONFIG = Path(__file__).resolve().parents[1] / "config"
|
| 25 |
+
_SCENARIO_FILES = sorted((_CONFIG / "scenarios").glob("*.yaml"))
|
| 26 |
+
_SCENARIO_IDS = [p.stem for p in _SCENARIO_FILES]
|
| 27 |
+
|
| 28 |
+
# Across symmetric seats (Debate Duel, Beat Battle) only these may differ — the whole
|
| 29 |
+
# point is "same seat, different model", so everything else must be byte-identical.
|
| 30 |
+
_SEAT_DIFF_ALLOWED = {"name", "hue", "archetype", "model_profile", "model_endpoint"}
|
| 31 |
+
|
| 32 |
+
|
| 33 |
+
@pytest.fixture(scope="module")
|
| 34 |
+
def registry() -> Registry:
|
| 35 |
+
return Registry.from_dir()
|
| 36 |
+
|
| 37 |
+
|
| 38 |
+
@pytest.fixture(scope="module")
|
| 39 |
+
def world(registry: Registry) -> WorldConfig:
|
| 40 |
+
# Composing a WorldConfig is what exercises the cross-cast competition rules
|
| 41 |
+
# (team/seat members ⊆ cast; judged/versus need a judge) — the per-file
|
| 42 |
+
# validate_scenario path can't see the agent set. Must not raise.
|
| 43 |
+
return WorldConfig(agents=list(registry.agents.values()), scenarios=list(registry.scenarios.values()))
|
| 44 |
+
|
| 45 |
+
|
| 46 |
+
def test_all_scenarios_compose_into_a_valid_world(world: WorldConfig) -> None:
|
| 47 |
+
assert world.scenarios, "expected at least one scenario to load"
|
| 48 |
+
|
| 49 |
+
|
| 50 |
+
@pytest.mark.parametrize("path", _SCENARIO_FILES, ids=_SCENARIO_IDS)
|
| 51 |
+
def test_scenario_declares_an_explicit_competition_block(path: Path) -> None:
|
| 52 |
+
"""The block is defaulted in the schema, but the checklist requires it spelled out."""
|
| 53 |
+
raw = yaml.safe_load(path.read_text()) or {}
|
| 54 |
+
assert "competition" in raw, f"{path.stem}: must declare an explicit `competition` block"
|
| 55 |
+
assert raw["competition"].get("kind") in ("versus", "judged", "none"), f"{path.stem}: bad competition.kind"
|
| 56 |
+
|
| 57 |
+
|
| 58 |
+
@pytest.mark.parametrize("path", _SCENARIO_FILES, ids=_SCENARIO_IDS)
|
| 59 |
+
def test_scenario_has_the_authoring_basics(path: Path, registry: Registry) -> None:
|
| 60 |
+
"""Premise + cast + governor — the bones every scenario needs."""
|
| 61 |
+
cfg = registry.scenarios[path.stem]
|
| 62 |
+
assert cfg.goal.strip(), f"{path.stem}: needs a goal"
|
| 63 |
+
assert cfg.default_seed.strip(), f"{path.stem}: needs a default_seed"
|
| 64 |
+
assert cfg.example_seeds, f"{path.stem}: needs example_seeds"
|
| 65 |
+
assert cfg.genesis_text, f"{path.stem}: needs genesis_text"
|
| 66 |
+
assert cfg.governor is not None, f"{path.stem}: needs an explicit governor budget"
|
| 67 |
+
assert cfg.cast, f"{path.stem}: needs a cast"
|
| 68 |
+
|
| 69 |
+
|
| 70 |
+
@pytest.mark.parametrize("path", _SCENARIO_FILES, ids=_SCENARIO_IDS)
|
| 71 |
+
def test_competitive_scenarios_have_a_judge_that_ends_the_show(path: Path, registry: Registry) -> None:
|
| 72 |
+
"""judged/versus need a judge emitting judge.verdict, on a tick that fires in-budget.
|
| 73 |
+
|
| 74 |
+
``none`` scenarios must NOT be forced to carry a judge.
|
| 75 |
+
"""
|
| 76 |
+
cfg = registry.scenarios[path.stem]
|
| 77 |
+
comp = cfg.competition
|
| 78 |
+
judges = [
|
| 79 |
+
m
|
| 80 |
+
for name in cfg.cast
|
| 81 |
+
if (m := registry.agents.get(name)) and m.role == "judge" and "judge.verdict" in m.may_emit
|
| 82 |
+
]
|
| 83 |
+
if comp.kind == "none":
|
| 84 |
+
return # showcase / collaborative — no winner machinery required
|
| 85 |
+
assert judges, f"{path.stem}: kind {comp.kind!r} needs a judge emitting judge.verdict"
|
| 86 |
+
# End condition: at least one judge fires within the round budget, so the verdict
|
| 87 |
+
# actually lands instead of the show grinding to a budget halt with no winner.
|
| 88 |
+
max_turns = cfg.governor.max_turns if cfg.governor else 100
|
| 89 |
+
firing = [m for m in judges if m.schedule.tick_every is not None and 0 < m.schedule.tick_every <= max_turns]
|
| 90 |
+
assert firing, f"{path.stem}: no judge has a tick_every that fires within max_turns={max_turns}"
|
| 91 |
+
|
| 92 |
+
|
| 93 |
+
@pytest.mark.parametrize("path", _SCENARIO_FILES, ids=_SCENARIO_IDS)
|
| 94 |
+
def test_symmetric_seats_are_identical_except_model(path: Path, registry: Registry) -> None:
|
| 95 |
+
"""Debate Duel / Beat Battle: symmetric seats may differ only by name/hue/model.
|
| 96 |
+
|
| 97 |
+
This is the fairness guarantee for the model leaderboard — the seats must be
|
| 98 |
+
truly identical apart from which model fills them.
|
| 99 |
+
"""
|
| 100 |
+
cfg = registry.scenarios[path.stem]
|
| 101 |
+
seats = cfg.competition.symmetric_seats or []
|
| 102 |
+
if len(seats) < 2:
|
| 103 |
+
return
|
| 104 |
+
manifests = [registry.agents[name].model_dump() for name in seats]
|
| 105 |
+
base = manifests[0]
|
| 106 |
+
for other in manifests[1:]:
|
| 107 |
+
differing = {k for k in base if base[k] != other[k]}
|
| 108 |
+
illegal = differing - _SEAT_DIFF_ALLOWED
|
| 109 |
+
assert not illegal, (
|
| 110 |
+
f"{path.stem}: symmetric seats {seats} differ in fields they may not: {sorted(illegal)} "
|
| 111 |
+
f"(only {sorted(_SEAT_DIFF_ALLOWED)} may differ)"
|
| 112 |
+
)
|
| 113 |
+
|
| 114 |
+
|
| 115 |
+
def test_world_rejects_team_member_not_in_cast(registry: Registry) -> None:
|
| 116 |
+
"""The cross-cast guard must actually fire on a broken competition block."""
|
| 117 |
+
good = registry.scenarios["the-steeped"].model_dump()
|
| 118 |
+
good["competition"] = {"kind": "versus", "teams": {"spy": ["nobody-here"], "herd": ["spy-cara"]}}
|
| 119 |
+
with pytest.raises(ValueError, match="non-cast members"):
|
| 120 |
+
WorldConfig(agents=list(registry.agents.values()), scenarios=[good])
|
| 121 |
+
|
| 122 |
+
|
| 123 |
+
def test_world_rejects_competitive_scenario_without_a_judge(registry: Registry) -> None:
|
| 124 |
+
bad = registry.scenarios["open-table"].model_dump()
|
| 125 |
+
bad["cast"] = ["chat-curious", "chat-skeptic", "chat-host"] # drop the table-judge
|
| 126 |
+
bad["competition"] = {"kind": "judged"}
|
| 127 |
+
with pytest.raises(ValueError, match="requires a cast member"):
|
| 128 |
+
WorldConfig(agents=list(registry.agents.values()), scenarios=[bad])
|
| 129 |
+
|
| 130 |
+
|
| 131 |
+
@pytest.mark.parametrize("name", ["the-steeped", "mystery-roots", "debate-duel", "twenty-sprouts"])
|
| 132 |
+
def test_competitive_scenario_names_a_real_winner_offline(name: str, registry: Registry) -> None:
|
| 133 |
+
"""End-to-end on the deterministic stub: the verdict names a real player / team.
|
| 134 |
+
|
| 135 |
+
Proves the watchable-stub winner path (handler validation + fallback) without
|
| 136 |
+
spending a token, and that winner attribution is assertable on the offline path.
|
| 137 |
+
"""
|
| 138 |
+
scenario = registry.build_scenario(name, router=ModelRouter(offline=True, specs={}), tools=default_tool_registry())
|
| 139 |
+
conductor = Conductor(scenario, governor=registry.governor_for(name))
|
| 140 |
+
conductor.reset(scenario.default_seed)
|
| 141 |
+
for _ in range(80):
|
| 142 |
+
try:
|
| 143 |
+
conductor.step()
|
| 144 |
+
except BudgetExceeded:
|
| 145 |
+
break
|
| 146 |
+
if any(e.kind == "judge.verdict" for e in conductor.ledger.events_for_run(conductor.run_id)):
|
| 147 |
+
break
|
| 148 |
+
events = conductor.ledger.events_for_run(conductor.run_id)
|
| 149 |
+
verdict = next((e for e in reversed(events) if e.kind == "judge.verdict"), None)
|
| 150 |
+
assert verdict is not None, f"{name}: no verdict reached offline"
|
| 151 |
+
winner = verdict.payload.get("winner")
|
| 152 |
+
cfg = registry.scenarios[name]
|
| 153 |
+
valid = set(cfg.cast) | set(cfg.competition.teams or {})
|
| 154 |
+
assert winner in valid, f"{name}: winner {winner!r} is not a cast member or team label"
|
|
@@ -11,6 +11,8 @@ offline (deterministic stub, no API key):
|
|
| 11 |
|
| 12 |
from __future__ import annotations
|
| 13 |
|
|
|
|
|
|
|
| 14 |
from src.core.conductor import Conductor
|
| 15 |
from src.core.ledger_factory import make_ledger
|
| 16 |
from src.core.registry import default_registry
|
|
@@ -62,10 +64,16 @@ def test_public_setup_never_names_the_words() -> None:
|
|
| 62 |
# globally visible — every mind reads them. They must set up the game without
|
| 63 |
# publishing the answer; the words live only in each persona + the host reveal.
|
| 64 |
cond = _run()
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 65 |
for e in cond.ledger.events:
|
| 66 |
if e.kind in ("run.started", "world.observed"):
|
| 67 |
blob = str(e.payload).upper()
|
| 68 |
-
assert
|
| 69 |
|
| 70 |
|
| 71 |
def test_host_verdict_unmasks_the_spy() -> None:
|
|
|
|
| 11 |
|
| 12 |
from __future__ import annotations
|
| 13 |
|
| 14 |
+
import re
|
| 15 |
+
|
| 16 |
from src.core.conductor import Conductor
|
| 17 |
from src.core.ledger_factory import make_ledger
|
| 18 |
from src.core.registry import default_registry
|
|
|
|
| 64 |
# globally visible — every mind reads them. They must set up the game without
|
| 65 |
# publishing the answer; the words live only in each persona + the host reveal.
|
| 66 |
cond = _run()
|
| 67 |
+
# Match the secret words as whole words, not substrings: run.started now carries the
|
| 68 |
+
# competition block whose ``teams`` key contains the literal "TEA" inside "TEAMS"
|
| 69 |
+
# (ADR-0029) — a coincidence, not a leak. The team map names the spy *agent*, never
|
| 70 |
+
# the secret *word*, and run.started never reaches an agent's prompt (only `text`/
|
| 71 |
+
# `goal` do). What must never appear is the word itself.
|
| 72 |
+
leak = re.compile(r"\b(COFFEE|TEA)\b")
|
| 73 |
for e in cond.ledger.events:
|
| 74 |
if e.kind in ("run.started", "world.observed"):
|
| 75 |
blob = str(e.payload).upper()
|
| 76 |
+
assert not leak.search(blob), f"secret word leaked into {e.kind}: {e.payload}"
|
| 77 |
|
| 78 |
|
| 79 |
def test_host_verdict_unmasks_the_spy() -> None:
|