fix(planner): reply-language for infeasible_reason + answer-only-asked rule
Browse filesTwo prompt-layer fixes for answer quality:
- Language consistency: the planner authored infeasible_reason in English while
the data-gap wrapper (refusals.py) was rendered in the user's language, so an
infeasible answer came back mixed ID+EN. Thread reply_language coordinator ->
PlannerService.plan -> build_planner_prompt and instruct the planner (planner.md)
to write infeasible_reason in the reply language. All params optional/backward-
compatible; the prompt section only renders when a language is known.
- Scope creep: the planner added unasked tasks (e.g. an extra count-by-site_type)
because "smallest plan" was only a soft bullet. Promote it to Hard rule #7:
answer ONLY the asked question; a multi-part plan is correct only when the
question itself has multiple parts.
Behavioral (prompt) change, not a deterministic guard — validated for plumbing by
the unit suite; behavioral effect should be checked via eval/ and reviewed with
the planner few-shot owner.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
@@ -95,13 +95,16 @@ def build_planner_prompt(
|
|
| 95 |
query: str,
|
| 96 |
constraints: Constraints,
|
| 97 |
previous_errors: list[str] | None = None,
|
|
|
|
| 98 |
) -> str:
|
| 99 |
"""Return the human-message content for the planner LLM.
|
| 100 |
|
| 101 |
The system prompt (`config/prompts/planner.md`) is loaded separately by
|
| 102 |
`PlannerService`. `previous_errors` is the full history of prior validation
|
| 103 |
failures (oldest first) so a retry fixes ALL of them at once instead of fixing
|
| 104 |
-
one and reintroducing an earlier one.
|
|
|
|
|
|
|
| 105 |
"""
|
| 106 |
sections = [
|
| 107 |
f"# Business context\n\n{render_business_context(context)}",
|
|
@@ -111,6 +114,13 @@ def build_planner_prompt(
|
|
| 111 |
f"# Examples\n\n{render_examples()}",
|
| 112 |
f"# Question\n\n{query}",
|
| 113 |
]
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 114 |
if previous_errors:
|
| 115 |
joined = "\n".join(f"- {err}" for err in previous_errors)
|
| 116 |
sections.append(
|
|
|
|
| 95 |
query: str,
|
| 96 |
constraints: Constraints,
|
| 97 |
previous_errors: list[str] | None = None,
|
| 98 |
+
reply_language: str | None = None,
|
| 99 |
) -> str:
|
| 100 |
"""Return the human-message content for the planner LLM.
|
| 101 |
|
| 102 |
The system prompt (`config/prompts/planner.md`) is loaded separately by
|
| 103 |
`PlannerService`. `previous_errors` is the full history of prior validation
|
| 104 |
failures (oldest first) so a retry fixes ALL of them at once instead of fixing
|
| 105 |
+
one and reintroducing an earlier one. `reply_language` (the pipeline's detected
|
| 106 |
+
user language) is surfaced so the only user-facing free text the planner emits
|
| 107 |
+
— `infeasible_reason` — comes back in that language instead of always English.
|
| 108 |
"""
|
| 109 |
sections = [
|
| 110 |
f"# Business context\n\n{render_business_context(context)}",
|
|
|
|
| 114 |
f"# Examples\n\n{render_examples()}",
|
| 115 |
f"# Question\n\n{query}",
|
| 116 |
]
|
| 117 |
+
if reply_language:
|
| 118 |
+
sections.append(
|
| 119 |
+
f"# Reply language\n\n"
|
| 120 |
+
f"The user writes in {reply_language}. If (and only if) you return an "
|
| 121 |
+
f"infeasible plan, write `infeasible_reason` in {reply_language} — it is "
|
| 122 |
+
f"shown to the user verbatim. The plan structure itself is unaffected."
|
| 123 |
+
)
|
| 124 |
if previous_errors:
|
| 125 |
joined = "\n".join(f"- {err}" for err in previous_errors)
|
| 126 |
sections.append(
|
|
@@ -102,6 +102,7 @@ class PlannerService:
|
|
| 102 |
query: str,
|
| 103 |
constraints: Constraints,
|
| 104 |
callbacks: list | None = None,
|
|
|
|
| 105 |
) -> TaskList:
|
| 106 |
summary = CatalogSummary.from_catalog(catalog)
|
| 107 |
chain = self._ensure_chain()
|
|
@@ -109,7 +110,13 @@ class PlannerService:
|
|
| 109 |
|
| 110 |
for attempt in range(1, self._max_retries + 1):
|
| 111 |
human_content = build_planner_prompt(
|
| 112 |
-
context,
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 113 |
)
|
| 114 |
# All retry attempts share `callbacks`, so each shows up under the same
|
| 115 |
# trace — that is how retry token cost becomes visible.
|
|
@@ -176,6 +183,9 @@ async def plan_analysis(
|
|
| 176 |
tools: ToolRegistry,
|
| 177 |
query: str,
|
| 178 |
constraints: Constraints,
|
|
|
|
| 179 |
) -> TaskList:
|
| 180 |
"""Convenience entry point using the default chain + validator."""
|
| 181 |
-
return await PlannerService().plan(
|
|
|
|
|
|
|
|
|
| 102 |
query: str,
|
| 103 |
constraints: Constraints,
|
| 104 |
callbacks: list | None = None,
|
| 105 |
+
reply_language: str | None = None,
|
| 106 |
) -> TaskList:
|
| 107 |
summary = CatalogSummary.from_catalog(catalog)
|
| 108 |
chain = self._ensure_chain()
|
|
|
|
| 110 |
|
| 111 |
for attempt in range(1, self._max_retries + 1):
|
| 112 |
human_content = build_planner_prompt(
|
| 113 |
+
context,
|
| 114 |
+
summary,
|
| 115 |
+
tools,
|
| 116 |
+
query,
|
| 117 |
+
constraints,
|
| 118 |
+
previous_errors,
|
| 119 |
+
reply_language=reply_language,
|
| 120 |
)
|
| 121 |
# All retry attempts share `callbacks`, so each shows up under the same
|
| 122 |
# trace — that is how retry token cost becomes visible.
|
|
|
|
| 183 |
tools: ToolRegistry,
|
| 184 |
query: str,
|
| 185 |
constraints: Constraints,
|
| 186 |
+
reply_language: str | None = None,
|
| 187 |
) -> TaskList:
|
| 188 |
"""Convenience entry point using the default chain + validator."""
|
| 189 |
+
return await PlannerService().plan(
|
| 190 |
+
context, catalog, tools, query, constraints, reply_language=reply_language
|
| 191 |
+
)
|
|
@@ -55,7 +55,8 @@ class SlowPathCoordinator:
|
|
| 55 |
await progress("Planning the analysis…")
|
| 56 |
plan_kw = {"callbacks": planner_callbacks} if planner_callbacks else {}
|
| 57 |
task_list = await self._planner.plan(
|
| 58 |
-
context, catalog, self._registry, query, constraints,
|
|
|
|
| 59 |
)
|
| 60 |
if task_list.infeasible_reason and not task_list.tasks:
|
| 61 |
# Honest data-gap outcome (planner.md "When the catalog cannot
|
|
|
|
| 55 |
await progress("Planning the analysis…")
|
| 56 |
plan_kw = {"callbacks": planner_callbacks} if planner_callbacks else {}
|
| 57 |
task_list = await self._planner.plan(
|
| 58 |
+
context, catalog, self._registry, query, constraints,
|
| 59 |
+
reply_language=reply_language, **plan_kw
|
| 60 |
)
|
| 61 |
if task_list.infeasible_reason and not task_list.tasks:
|
| 62 |
# Honest data-gap outcome (planner.md "When the catalog cannot
|
|
@@ -27,6 +27,13 @@ only a `TaskList` object that conforms to the provided schema.
|
|
| 27 |
0-1 ratio AS a percentage metric). If no column plausibly holds the measure
|
| 28 |
or entity the question asks about, the plan is **infeasible** — see "When the
|
| 29 |
catalog cannot answer".
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 30 |
|
| 31 |
# How to plan
|
| 32 |
|
|
@@ -99,6 +106,9 @@ subscription data). For those:
|
|
| 99 |
- Return `tasks: []` and set **`infeasible_reason`**: one short paragraph naming
|
| 100 |
(a) what the question needs that no column provides, and (b) the nearest
|
| 101 |
analyses the catalog CAN support, so the user knows what to ask instead.
|
|
|
|
|
|
|
|
|
|
| 102 |
- Do NOT emit a plan that maps the question onto semantically unrelated columns
|
| 103 |
just because their types fit — a confidently wrong number is worse than an
|
| 104 |
honest gap.
|
|
|
|
| 27 |
0-1 ratio AS a percentage metric). If no column plausibly holds the measure
|
| 28 |
or entity the question asks about, the plan is **infeasible** — see "When the
|
| 29 |
catalog cannot answer".
|
| 30 |
+
7. **Answer ONLY the asked question.** Plan the smallest analysis that fully
|
| 31 |
+
answers what the user actually asked — nothing more. Do NOT add tasks for
|
| 32 |
+
adjacent, "might be useful", or unasked questions (e.g. an extra breakdown by
|
| 33 |
+
a category the question never mentioned, or a count the user did not request),
|
| 34 |
+
even when the data would support them. Extra breadth the user did not ask for
|
| 35 |
+
is noise, not helpfulness. A multi-part task list is correct ONLY when the
|
| 36 |
+
question itself has multiple parts (e.g. "trend by region AND what's unusual").
|
| 37 |
|
| 38 |
# How to plan
|
| 39 |
|
|
|
|
| 106 |
- Return `tasks: []` and set **`infeasible_reason`**: one short paragraph naming
|
| 107 |
(a) what the question needs that no column provides, and (b) the nearest
|
| 108 |
analyses the catalog CAN support, so the user knows what to ask instead.
|
| 109 |
+
Write this paragraph in the user's reply language (see the "Reply language"
|
| 110 |
+
note in the human message, when present) — it is shown to the user verbatim, so
|
| 111 |
+
a mismatched language reads as broken.
|
| 112 |
- Do NOT emit a plan that maps the question onto semantically unrelated columns
|
| 113 |
just because their types fit — a confidently wrong number is worse than an
|
| 114 |
honest gap.
|