agharsallah commited on
Commit
3966dc5
Β·
1 Parent(s): 28cf0fa

feat: Enhance Secret Keeper's response logic to confirm correct guesses and improve consistency in answers

Browse files
config/agents/secret-keeper.yaml CHANGED
@@ -3,11 +3,13 @@ role: worker
3
  handler: secret-keeper
4
  persona: >
5
  You are the Secret Keeper of the sprout grove. You hold one secret word and the
6
- guesser must tease it out with yes/no questions. You ANSWER β€” you never ask. Reply to
7
- their MOST RECENT question truthfully and helpfully in ONE short sentence beginning with
8
- "Yes" or "No" β€” playful, never cruel, never contradicting an earlier answer. Never write,
9
- spell, or quote your word, and never end your line with a question. Also report your
10
- `mood` (one of: thinking, calm, lying, panic, smug, truth, gossip).
 
 
11
  subscribes_to: []
12
  may_emit:
13
  - agent.spoke
 
3
  handler: secret-keeper
4
  persona: >
5
  You are the Secret Keeper of the sprout grove. You hold one secret word and the
6
+ guesser must tease it out with yes/no questions. You ANSWER β€” you never ask. Keep your
7
+ word in mind for every answer and reply to their MOST RECENT question truthfully in ONE
8
+ short sentence beginning with "Yes" or "No" β€” playful, never cruel. Always tell the truth
9
+ about the word; never lie, never bluff, and never contradict an earlier answer. If the
10
+ guesser names your word β€” in any spelling or capitalization β€” say plainly that it is a
11
+ match. Until then never write, spell, or quote your word, and never end your line with a
12
+ question. Also report your `mood` (one of: thinking, calm, lying, panic, smug, truth, gossip).
13
  subscribes_to: []
14
  may_emit:
15
  - agent.spoke
docs/blog/state-of-the-wood.md CHANGED
@@ -2,6 +2,11 @@
2
 
3
  *What we've built, what broke in the best possible way, and where the forest theater goes next.*
4
 
 
 
 
 
 
5
  ---
6
 
7
  The elevator pitch hasn't changed: a forest where small specialist AI models β€” each one
@@ -17,7 +22,7 @@ models in a single cast. A Fishbowl UI with MindCards and a mind-reader. Seven h
17
  and forty-one passing tests with no mocks and no API key required. This is the next
18
  chapter.
19
 
20
- [1]: engine-architecture.md
21
 
22
  ---
23
 
 
2
 
3
  *What we've built, what broke in the best possible way, and where the forest theater goes next.*
4
 
5
+ > **Archival snapshot.** This is a dated milestone report, kept as-written. For the evergreen
6
+ > walkthrough β€” pitch to internals β€” start at the
7
+ > [Field Notes series index](00-field-notes-index.md). Figures here (test counts, scenario
8
+ > tables) reflect the engine at the time of writing.
9
+
10
  ---
11
 
12
  The elevator pitch hasn't changed: a forest where small specialist AI models β€” each one
 
22
  and forty-one passing tests with no mocks and no API key required. This is the next
23
  chapter.
24
 
25
+ [1]: 03-one-engine-three-costumes.md
26
 
27
  ---
28
 
src/agents/twenty_sprouts.py CHANGED
@@ -94,6 +94,17 @@ def _contains_word(text: str, word: str) -> bool:
94
  return bool(word) and bool(_word_re(word).search(text or ""))
95
 
96
 
 
 
 
 
 
 
 
 
 
 
 
97
  def _redact_word(text: str, word: str) -> str:
98
  """Mask any spelled-out secret word so it can never reach the stage.
99
 
@@ -199,12 +210,26 @@ class SecretKeeper(ManifestAgent):
199
  # The guesser's most recent *open* question (the one awaiting this answer).
200
  question = next((q for q, a in reversed(pairs) if a is None), "")
201
  already = [a for _, a in pairs if a]
 
 
 
 
 
 
 
 
 
 
 
 
 
202
  guard = (
203
- "\n\nSTRICT RULES: You ANSWER, you never ask. Begin with 'Yes' or 'No' (then one short, "
204
- "truthful, playful clause). Answer ONLY about your secret word, and answer the SAME way "
205
- "every time about the same property β€” never contradict an earlier answer. Do NOT end your "
206
- f"line with a question mark. NEVER write the word '{word}' or any form of it β€” describe it, "
207
- "never name it; that instantly loses the game."
 
208
  )
209
  # On a corrective re-ask (set by ``act`` after a rejected reply), tell the model
210
  # exactly what it did wrong so the retry actually fixes it.
@@ -212,15 +237,17 @@ class SecretKeeper(ManifestAgent):
212
  correction = f"\n\nCORRECTION: your previous reply was rejected because {reason}. Try again." if reason else ""
213
  if not question:
214
  return (
215
- f"YOUR SECRET WORD (never write, spell, or quote it β€” only answer about it): {word}\n"
216
  "The guesser has not asked yet. In ONE short sentence, invite them to begin asking "
217
  "yes/no questions. Do not reveal anything about the word yet." + guard + correction
218
  )
219
  consistency = (
220
- ("\n\nYour earlier answers (stay consistent with these):\n" + _bullets(already[-6:])) if already else ""
 
 
221
  )
222
  return (
223
- f"YOUR SECRET WORD (never write, spell, or quote it β€” only answer about it): {word}\n"
224
  f'The guesser just asked: "{_trim(question)}"\n'
225
  "Answer THAT question about your word, truthfully, in ONE short sentence."
226
  + consistency
@@ -237,15 +264,22 @@ class SecretKeeper(ManifestAgent):
237
  ) -> Event:
238
  offline = bool(getattr(self.router, "offline", False))
239
  word = _word_for_seed(projection.seed)
 
 
 
 
 
 
 
240
  event = super().act(run_id, turn, projection, recent_events)
241
  usage_total = dict(self.last_usage)
242
  text = str(event.payload.get("text", ""))
243
 
244
- # Live quality pass: if the keeper asked a question or spelled the word, re-ask once
245
- # with the reason fed back. Offline lines are curated, leak-free answers, so this
246
- # never fires there β€” the deterministic path is identical. The retry's tokens are
247
- # summed so the governor still meters both calls (mirrors base ``_verify_verdict``).
248
- if not offline and _answer_violation(text, word):
249
  self._retry_reason = _answer_violation(text, word)
250
  try:
251
  retry = super().act(run_id, turn, projection, recent_events)
@@ -255,14 +289,15 @@ class SecretKeeper(ManifestAgent):
255
  self.last_usage = usage_total
256
  event, text = retry, str(retry.payload.get("text", ""))
257
 
258
- # Absolute guarantee: scrub any spelled-out word before it ships. A no-op offline
259
- # and on a clean reply; the safety net when the model leaks anyway (as it did with
260
- # "a glowing ember…"). Logged so a persistent leak is visible in the run.
261
- scrubbed = _redact_word(text, word)
262
- if scrubbed != text:
263
- obs.log("twenty_sprouts.redacted_keeper_leak", agent=self.name, turn=turn)
264
- text = scrubbed
265
- event.payload["text"] = text
 
266
 
267
  # The keeper must answer, not ask: a reply still ending in a question after the
268
  # re-ask is skipped (live only) rather than shown asking. Redaction above can't fix
@@ -405,13 +440,14 @@ class SproutJudge(JudgedCompetition):
405
  def _guessed(recent_events: tuple[Event, ...], secret: str) -> bool:
406
  """True if the dealt word appears in ANY guesser line β€” a win sticks once made.
407
 
408
- Scanning every guesser ``agent.spoke`` (not just the latest) is the fix for the
409
- live bug where a correct guess was later buried under further guessing and the
410
- round was wrongly scored a miss."""
411
- needle = secret.lower()
 
412
  return any(
413
  e.actor == _GUESSER_NAME
414
  and e.kind == "agent.spoke"
415
- and needle in set(_WORD.findall(str(e.payload.get("text", "")).lower()))
416
  for e in recent_events
417
  )
 
94
  return bool(word) and bool(_word_re(word).search(text or ""))
95
 
96
 
97
+ def _line_names_word(text: str, word: str) -> bool:
98
+ """True when a line names the secret word as a whole token, in ANY capitalization.
99
+
100
+ The single shared "this is the word" test used by BOTH the keeper β€” to confirm a match
101
+ the moment the guesser says it β€” and the judge β€” to award the win. Sharing it guarantees
102
+ they never disagree (the keeper can't say "that's it!" on a line the judge scores a miss,
103
+ or deny a line the judge counts). Case-insensitive (``COMPASS`` == ``compass`` == ``Compass``),
104
+ whole-word (so ``compass`` doesn't match inside ``encompasses``)."""
105
+ return bool(word) and word.lower() in set(_WORD.findall((text or "").lower()))
106
+
107
+
108
  def _redact_word(text: str, word: str) -> str:
109
  """Mask any spelled-out secret word so it can never reach the stage.
110
 
 
210
  # The guesser's most recent *open* question (the one awaiting this answer).
211
  question = next((q for q, a in reversed(pairs) if a is None), "")
212
  already = [a for _, a in pairs if a]
213
+
214
+ # The guesser just NAMED the word (any capitalization) β€” that is a win, not a
215
+ # question to dodge. Confirm the match plainly; the secret is already out (they said
216
+ # it), so the no-spelling rule no longer applies and ``act`` lets this line through.
217
+ if question and _line_names_word(question, word):
218
+ return (
219
+ f"YOUR SECRET WORD: {word}\n"
220
+ f'The guesser just said: "{_trim(question)}"\n'
221
+ f"They have named your secret word EXACTLY β€” '{word}' is a match (capitalization does "
222
+ "not matter). This is a win for them. Do NOT deny it, deflect, or contradict it. In ONE "
223
+ "warm, final sentence, confirm plainly that they have guessed the word and the secret is revealed."
224
+ )
225
+
226
  guard = (
227
+ "\n\nSTRICT RULES: You ANSWER, you never ask. Begin with 'Yes' or 'No' (then one short clause). "
228
+ "Keep YOUR SECRET WORD above in mind for EVERY answer and answer ONLY about it. Tell the TRUTH "
229
+ "about its real properties β€” never lie, never bluff, and answer the SAME way every time about the "
230
+ "same property (never contradict an earlier answer). If the guesser NAMES the word β€” in any "
231
+ "capitalization β€” say plainly that it is a match. Do NOT end your line with a question mark. Until "
232
+ f"it is guessed, NEVER write the word '{word}' or any form of it β€” describe it, never spell it."
233
  )
234
  # On a corrective re-ask (set by ``act`` after a rejected reply), tell the model
235
  # exactly what it did wrong so the retry actually fixes it.
 
237
  correction = f"\n\nCORRECTION: your previous reply was rejected because {reason}. Try again." if reason else ""
238
  if not question:
239
  return (
240
+ f"YOUR SECRET WORD (keep it in mind; never spell it until it is guessed): {word}\n"
241
  "The guesser has not asked yet. In ONE short sentence, invite them to begin asking "
242
  "yes/no questions. Do not reveal anything about the word yet." + guard + correction
243
  )
244
  consistency = (
245
+ ("\n\nYour earlier answers (be truthful and consistent with these):\n" + _bullets(already[-6:]))
246
+ if already
247
+ else ""
248
  )
249
  return (
250
+ f"YOUR SECRET WORD (keep it in mind for this answer; never spell it until it is guessed): {word}\n"
251
  f'The guesser just asked: "{_trim(question)}"\n'
252
  "Answer THAT question about your word, truthfully, in ONE short sentence."
253
  + consistency
 
264
  ) -> Event:
265
  offline = bool(getattr(self.router, "offline", False))
266
  word = _word_for_seed(projection.seed)
267
+ # Did the guesser just NAME the word? On a match the secret is already public (they
268
+ # said it), so the no-spelling guards below stand down β€” the keeper confirms the win
269
+ # in plain words ("Yes β€” compass, you've got it!") instead of being scrubbed to "thing".
270
+ pairs = _qa_history(recent_events)
271
+ open_question = next((q for q, a in reversed(pairs) if a is None), "")
272
+ matched = _line_names_word(open_question, word)
273
+
274
  event = super().act(run_id, turn, projection, recent_events)
275
  usage_total = dict(self.last_usage)
276
  text = str(event.payload.get("text", ""))
277
 
278
+ # Live quality pass: if the keeper asked a question or (before the word is guessed)
279
+ # spelled it, re-ask once with the reason fed back. Skipped on a match β€” there the
280
+ # keeper is *meant* to name the word. Offline lines are curated, leak-free answers, so
281
+ # this never fires there. Retry tokens are summed so the governor meters both calls.
282
+ if not offline and not matched and _answer_violation(text, word):
283
  self._retry_reason = _answer_violation(text, word)
284
  try:
285
  retry = super().act(run_id, turn, projection, recent_events)
 
289
  self.last_usage = usage_total
290
  event, text = retry, str(retry.payload.get("text", ""))
291
 
292
+ # Absolute guarantee: scrub any spelled-out word before it ships β€” UNLESS the guesser
293
+ # already named it (a match), where confirming the word is the whole point. A no-op
294
+ # offline and on a clean reply; the safety net when the model leaks anyway.
295
+ if not matched:
296
+ scrubbed = _redact_word(text, word)
297
+ if scrubbed != text:
298
+ obs.log("twenty_sprouts.redacted_keeper_leak", agent=self.name, turn=turn)
299
+ text = scrubbed
300
+ event.payload["text"] = text
301
 
302
  # The keeper must answer, not ask: a reply still ending in a question after the
303
  # re-ask is skipped (live only) rather than shown asking. Redaction above can't fix
 
440
  def _guessed(recent_events: tuple[Event, ...], secret: str) -> bool:
441
  """True if the dealt word appears in ANY guesser line β€” a win sticks once made.
442
 
443
+ Uses the same case-insensitive whole-word test the keeper uses to confirm a match
444
+ (:func:`_line_names_word`), so judge and keeper always agree on what counts as a win.
445
+ Scanning every guesser ``agent.spoke`` (not just the latest) is the fix for the live
446
+ bug where a correct guess was later buried under further guessing and the round was
447
+ wrongly scored a miss."""
448
  return any(
449
  e.actor == _GUESSER_NAME
450
  and e.kind == "agent.spoke"
451
+ and _line_names_word(str(e.payload.get("text", "")), secret)
452
  for e in recent_events
453
  )
tests/test_twenty_sprouts.py CHANGED
@@ -10,6 +10,8 @@ from __future__ import annotations
10
 
11
  import pytest
12
 
 
 
13
  from src.agents.twenty_sprouts import (
14
  _WORDS,
15
  SecretKeeper,
@@ -17,6 +19,7 @@ from src.agents.twenty_sprouts import (
17
  _answer_violation,
18
  _classify_answer,
19
  _contains_word,
 
20
  _qa_history,
21
  _questions_since_guess,
22
  _redact_word,
@@ -142,6 +145,88 @@ def test_keeper_question_guard_is_a_passthrough_offline():
142
  assert event.payload["secret"] == _word_for_seed("seed-A") # ground truth stamped on
143
 
144
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
145
  # ── the keeper never leaks the word (the worst bug: it spelled "ember" aloud) ─────
146
 
147
 
 
10
 
11
  import pytest
12
 
13
+ from dataclasses import dataclass, field
14
+
15
  from src.agents.twenty_sprouts import (
16
  _WORDS,
17
  SecretKeeper,
 
19
  _answer_violation,
20
  _classify_answer,
21
  _contains_word,
22
+ _line_names_word,
23
  _qa_history,
24
  _questions_since_guess,
25
  _redact_word,
 
145
  assert event.payload["secret"] == _word_for_seed("seed-A") # ground truth stamped on
146
 
147
 
148
+ def test_keeper_prompt_demands_truth_and_consistency():
149
+ prompt = _keeper()._build_extra_prompt(StageProjection(seed="seed-A"), _qa(("Is it warm?", None)))
150
+ low = prompt.lower()
151
+ assert "never lie" in low # don't bluff about the word
152
+ assert "never contradict" in low # stay consistent across answers
153
+ assert "keep your secret word" in low or "keep it in mind" in low # word stays in context
154
+
155
+
156
+ # ── the keeper confirms a correct guess in ANY capitalization (the asked-for fix) ─────
157
+
158
+
159
+ @pytest.mark.parametrize(
160
+ "text,word,names",
161
+ [
162
+ ("My guess is: COMPASS.", "COMPASS", True),
163
+ ("my guess is: compass", "COMPASS", True),
164
+ ("Is it a Compass?", "COMPASS", True), # any capitalization is a match
165
+ ("it encompasses the grove", "COMPASS", False), # whole-word, not a substring
166
+ ("Is it a map?", "COMPASS", False),
167
+ ],
168
+ )
169
+ def test_line_names_word_is_case_insensitive_whole_word(text, word, names):
170
+ assert _line_names_word(text, word) is names
171
+
172
+
173
+ @pytest.mark.parametrize("caser", [str.upper, str.lower, str.title])
174
+ def test_keeper_prompt_confirms_a_match_in_any_case(caser):
175
+ word = _word_for_seed("seed-A")
176
+ guess = f"My guess is: {caser(word)}."
177
+ prompt = _keeper()._build_extra_prompt(StageProjection(seed="seed-A"), _qa((guess, None)))
178
+ low = prompt.lower()
179
+ assert "match" in low and "named your secret word" in low
180
+ assert "do not deny" in low # must not deflect a correct guess
181
+
182
+
183
+ @dataclass
184
+ class _ScriptedProvider:
185
+ """A hand-written provider stand-in (Γ  la test_memory._RaisingIndex) that returns a
186
+ fixed line β€” so we can drive the keeper's live-path guards deterministically, no mocks."""
187
+
188
+ line: str
189
+ last_usage: dict = field(default_factory=lambda: {"total_tokens": 5})
190
+ last_reasoning: str = ""
191
+ model_id: str = "scripted"
192
+
193
+ def complete(self, role: str, prompt: str) -> str:
194
+ return self.line
195
+
196
+
197
+ @dataclass
198
+ class _ScriptedRouter:
199
+ provider: _ScriptedProvider
200
+ offline: bool = False # exercise the LIVE guards (match/leak/scrub), not the stub path
201
+
202
+ def for_profile(self, key: str):
203
+ return self.provider
204
+
205
+
206
+ def _scripted_keeper(line: str) -> SecretKeeper:
207
+ agent = SecretKeeper(_ScriptedRouter(_ScriptedProvider(line)))
208
+ agent.manifest = Registry.from_dir().agents["secret-keeper"]
209
+ return agent
210
+
211
+
212
+ def test_keeper_acknowledges_a_match_with_the_word_intact():
213
+ # The guesser named the word, so confirming it WITH the word is the point β€” the leak
214
+ # scrub must stand down (no "thing" mangling of the winning acknowledgment).
215
+ word = _word_for_seed("seed-A")
216
+ guess = f"My guess is: {word.lower()}."
217
+ keeper = _scripted_keeper(f"Yes β€” {word.lower()}, you've found it!")
218
+ event = keeper.act(_RUN, 5, StageProjection(seed="seed-A"), _qa((guess, None)))
219
+ assert word.lower() in event.payload["text"].lower() # the word survives on a match
220
+
221
+
222
+ def test_keeper_still_scrubs_a_leak_when_it_is_not_a_match():
223
+ # A normal question (no guess) where the model blurts the word β†’ still scrubbed.
224
+ word = _word_for_seed("seed-A")
225
+ keeper = _scripted_keeper(f"Yes, a {word.lower()} guides you.")
226
+ event = keeper.act(_RUN, 5, StageProjection(seed="seed-A"), _qa(("Is it used for direction?", None)))
227
+ assert word.lower() not in event.payload["text"].lower() # leak scrubbed away
228
+
229
+
230
  # ── the keeper never leaks the word (the worst bug: it spelled "ember" aloud) ─────
231
 
232