sofhiaazzhr Claude Opus 4.8 commited on
Commit
4d89d7d
·
1 Parent(s): b40fa76

fix(planner): reply-language for infeasible_reason + answer-only-asked rule

Browse files

Two prompt-layer fixes for answer quality:

- Language consistency: the planner authored infeasible_reason in English while
the data-gap wrapper (refusals.py) was rendered in the user's language, so an
infeasible answer came back mixed ID+EN. Thread reply_language coordinator ->
PlannerService.plan -> build_planner_prompt and instruct the planner (planner.md)
to write infeasible_reason in the reply language. All params optional/backward-
compatible; the prompt section only renders when a language is known.

- Scope creep: the planner added unasked tasks (e.g. an extra count-by-site_type)
because "smallest plan" was only a soft bullet. Promote it to Hard rule #7:
answer ONLY the asked question; a multi-part plan is correct only when the
question itself has multiple parts.

Behavioral (prompt) change, not a deterministic guard — validated for plumbing by
the unit suite; behavioral effect should be checked via eval/ and reviewed with
the planner few-shot owner.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

src/agents/planner/prompt.py CHANGED
@@ -95,13 +95,16 @@ def build_planner_prompt(
95
  query: str,
96
  constraints: Constraints,
97
  previous_errors: list[str] | None = None,
 
98
  ) -> str:
99
  """Return the human-message content for the planner LLM.
100
 
101
  The system prompt (`config/prompts/planner.md`) is loaded separately by
102
  `PlannerService`. `previous_errors` is the full history of prior validation
103
  failures (oldest first) so a retry fixes ALL of them at once instead of fixing
104
- one and reintroducing an earlier one.
 
 
105
  """
106
  sections = [
107
  f"# Business context\n\n{render_business_context(context)}",
@@ -111,6 +114,13 @@ def build_planner_prompt(
111
  f"# Examples\n\n{render_examples()}",
112
  f"# Question\n\n{query}",
113
  ]
 
 
 
 
 
 
 
114
  if previous_errors:
115
  joined = "\n".join(f"- {err}" for err in previous_errors)
116
  sections.append(
 
95
  query: str,
96
  constraints: Constraints,
97
  previous_errors: list[str] | None = None,
98
+ reply_language: str | None = None,
99
  ) -> str:
100
  """Return the human-message content for the planner LLM.
101
 
102
  The system prompt (`config/prompts/planner.md`) is loaded separately by
103
  `PlannerService`. `previous_errors` is the full history of prior validation
104
  failures (oldest first) so a retry fixes ALL of them at once instead of fixing
105
+ one and reintroducing an earlier one. `reply_language` (the pipeline's detected
106
+ user language) is surfaced so the only user-facing free text the planner emits
107
+ — `infeasible_reason` — comes back in that language instead of always English.
108
  """
109
  sections = [
110
  f"# Business context\n\n{render_business_context(context)}",
 
114
  f"# Examples\n\n{render_examples()}",
115
  f"# Question\n\n{query}",
116
  ]
117
+ if reply_language:
118
+ sections.append(
119
+ f"# Reply language\n\n"
120
+ f"The user writes in {reply_language}. If (and only if) you return an "
121
+ f"infeasible plan, write `infeasible_reason` in {reply_language} — it is "
122
+ f"shown to the user verbatim. The plan structure itself is unaffected."
123
+ )
124
  if previous_errors:
125
  joined = "\n".join(f"- {err}" for err in previous_errors)
126
  sections.append(
src/agents/planner/service.py CHANGED
@@ -102,6 +102,7 @@ class PlannerService:
102
  query: str,
103
  constraints: Constraints,
104
  callbacks: list | None = None,
 
105
  ) -> TaskList:
106
  summary = CatalogSummary.from_catalog(catalog)
107
  chain = self._ensure_chain()
@@ -109,7 +110,13 @@ class PlannerService:
109
 
110
  for attempt in range(1, self._max_retries + 1):
111
  human_content = build_planner_prompt(
112
- context, summary, tools, query, constraints, previous_errors
 
 
 
 
 
 
113
  )
114
  # All retry attempts share `callbacks`, so each shows up under the same
115
  # trace — that is how retry token cost becomes visible.
@@ -176,6 +183,9 @@ async def plan_analysis(
176
  tools: ToolRegistry,
177
  query: str,
178
  constraints: Constraints,
 
179
  ) -> TaskList:
180
  """Convenience entry point using the default chain + validator."""
181
- return await PlannerService().plan(context, catalog, tools, query, constraints)
 
 
 
102
  query: str,
103
  constraints: Constraints,
104
  callbacks: list | None = None,
105
+ reply_language: str | None = None,
106
  ) -> TaskList:
107
  summary = CatalogSummary.from_catalog(catalog)
108
  chain = self._ensure_chain()
 
110
 
111
  for attempt in range(1, self._max_retries + 1):
112
  human_content = build_planner_prompt(
113
+ context,
114
+ summary,
115
+ tools,
116
+ query,
117
+ constraints,
118
+ previous_errors,
119
+ reply_language=reply_language,
120
  )
121
  # All retry attempts share `callbacks`, so each shows up under the same
122
  # trace — that is how retry token cost becomes visible.
 
183
  tools: ToolRegistry,
184
  query: str,
185
  constraints: Constraints,
186
+ reply_language: str | None = None,
187
  ) -> TaskList:
188
  """Convenience entry point using the default chain + validator."""
189
+ return await PlannerService().plan(
190
+ context, catalog, tools, query, constraints, reply_language=reply_language
191
+ )
src/agents/slow_path/coordinator.py CHANGED
@@ -55,7 +55,8 @@ class SlowPathCoordinator:
55
  await progress("Planning the analysis…")
56
  plan_kw = {"callbacks": planner_callbacks} if planner_callbacks else {}
57
  task_list = await self._planner.plan(
58
- context, catalog, self._registry, query, constraints, **plan_kw
 
59
  )
60
  if task_list.infeasible_reason and not task_list.tasks:
61
  # Honest data-gap outcome (planner.md "When the catalog cannot
 
55
  await progress("Planning the analysis…")
56
  plan_kw = {"callbacks": planner_callbacks} if planner_callbacks else {}
57
  task_list = await self._planner.plan(
58
+ context, catalog, self._registry, query, constraints,
59
+ reply_language=reply_language, **plan_kw
60
  )
61
  if task_list.infeasible_reason and not task_list.tasks:
62
  # Honest data-gap outcome (planner.md "When the catalog cannot
src/config/prompts/planner.md CHANGED
@@ -27,6 +27,13 @@ only a `TaskList` object that conforms to the provided schema.
27
  0-1 ratio AS a percentage metric). If no column plausibly holds the measure
28
  or entity the question asks about, the plan is **infeasible** — see "When the
29
  catalog cannot answer".
 
 
 
 
 
 
 
30
 
31
  # How to plan
32
 
@@ -99,6 +106,9 @@ subscription data). For those:
99
  - Return `tasks: []` and set **`infeasible_reason`**: one short paragraph naming
100
  (a) what the question needs that no column provides, and (b) the nearest
101
  analyses the catalog CAN support, so the user knows what to ask instead.
 
 
 
102
  - Do NOT emit a plan that maps the question onto semantically unrelated columns
103
  just because their types fit — a confidently wrong number is worse than an
104
  honest gap.
 
27
  0-1 ratio AS a percentage metric). If no column plausibly holds the measure
28
  or entity the question asks about, the plan is **infeasible** — see "When the
29
  catalog cannot answer".
30
+ 7. **Answer ONLY the asked question.** Plan the smallest analysis that fully
31
+ answers what the user actually asked — nothing more. Do NOT add tasks for
32
+ adjacent, "might be useful", or unasked questions (e.g. an extra breakdown by
33
+ a category the question never mentioned, or a count the user did not request),
34
+ even when the data would support them. Extra breadth the user did not ask for
35
+ is noise, not helpfulness. A multi-part task list is correct ONLY when the
36
+ question itself has multiple parts (e.g. "trend by region AND what's unusual").
37
 
38
  # How to plan
39
 
 
106
  - Return `tasks: []` and set **`infeasible_reason`**: one short paragraph naming
107
  (a) what the question needs that no column provides, and (b) the nearest
108
  analyses the catalog CAN support, so the user knows what to ask instead.
109
+ Write this paragraph in the user's reply language (see the "Reply language"
110
+ note in the human message, when present) — it is shown to the user verbatim, so
111
+ a mismatched language reads as broken.
112
  - Do NOT emit a plan that maps the question onto semantically unrelated columns
113
  just because their types fit — a confidently wrong number is worse than an
114
  honest gap.