Rifqi Hafizuddin Claude Opus 4.8 commited on
Commit
f87e5ec
·
1 Parent(s): 3b27b6b

[KM-626][AI] Planner few-shot: make describe_source success_criteria checkable

Browse files

Live end-to-end test against real user data exposed a Planner robustness bug:
the validator requires success_criteria to contain a measurable signal word, but
the three describe_source examples modeled "describe_source returns the orders
table..." (no signal word). The LLM imitated that phrasing and failed its own
validation (3 retries exhausted). Reworded to "Produced the orders table
schema..." so the examples are consistent with the validator. After the fix the
same question plans in one shot (no retries).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Files changed (1) hide show
  1. src/agents/planner/examples.py +3 -3
src/agents/planner/examples.py CHANGED
@@ -38,7 +38,7 @@ _EXAMPLE_A = TaskList(
38
  objective="Confirm the sales source exposes category, revenue, and order date.",
39
  tool_calls=[ToolCall(tool="describe_source", args={"source_id": "src_sales"})],
40
  expected_output="source_shape",
41
- success_criteria="describe_source returns the orders table with the 3 columns.",
42
  depends_on=[],
43
  parallelizable_with=[],
44
  estimated_cost="low",
@@ -118,7 +118,7 @@ _EXAMPLE_B = TaskList(
118
  objective="Confirm the sales source exposes order date, revenue, and region.",
119
  tool_calls=[ToolCall(tool="describe_source", args={"source_id": "src_sales"})],
120
  expected_output="source_shape",
121
- success_criteria="describe_source returns the orders table with the needed columns.",
122
  depends_on=[],
123
  parallelizable_with=[],
124
  estimated_cost="low",
@@ -213,7 +213,7 @@ _EXAMPLE_C = TaskList(
213
  objective="Confirm the sales source exposes order date and revenue.",
214
  tool_calls=[ToolCall(tool="describe_source", args={"source_id": "src_sales"})],
215
  expected_output="source_shape",
216
- success_criteria="describe_source returns the orders table with date and revenue.",
217
  depends_on=[],
218
  parallelizable_with=["t4"],
219
  estimated_cost="low",
 
38
  objective="Confirm the sales source exposes category, revenue, and order date.",
39
  tool_calls=[ToolCall(tool="describe_source", args={"source_id": "src_sales"})],
40
  expected_output="source_shape",
41
+ success_criteria="Produced the orders table schema; the 3 needed columns are present.",
42
  depends_on=[],
43
  parallelizable_with=[],
44
  estimated_cost="low",
 
118
  objective="Confirm the sales source exposes order date, revenue, and region.",
119
  tool_calls=[ToolCall(tool="describe_source", args={"source_id": "src_sales"})],
120
  expected_output="source_shape",
121
+ success_criteria="Produced the orders table schema; the needed columns are present.",
122
  depends_on=[],
123
  parallelizable_with=[],
124
  estimated_cost="low",
 
213
  objective="Confirm the sales source exposes order date and revenue.",
214
  tool_calls=[ToolCall(tool="describe_source", args={"source_id": "src_sales"})],
215
  expected_output="source_shape",
216
+ success_criteria="Produced the orders table schema; date and revenue columns present.",
217
  depends_on=[],
218
  parallelizable_with=["t4"],
219
  estimated_cost="low",