Rifqi Hafizuddin Claude Opus 4.8 commited on
Commit ·
f87e5ec
1
Parent(s): 3b27b6b
[KM-626][AI] Planner few-shot: make describe_source success_criteria checkable
Browse filesLive end-to-end test against real user data exposed a Planner robustness bug:
the validator requires success_criteria to contain a measurable signal word, but
the three describe_source examples modeled "describe_source returns the orders
table..." (no signal word). The LLM imitated that phrasing and failed its own
validation (3 retries exhausted). Reworded to "Produced the orders table
schema..." so the examples are consistent with the validator. After the fix the
same question plans in one shot (no retries).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
src/agents/planner/examples.py
CHANGED
|
@@ -38,7 +38,7 @@ _EXAMPLE_A = TaskList(
|
|
| 38 |
objective="Confirm the sales source exposes category, revenue, and order date.",
|
| 39 |
tool_calls=[ToolCall(tool="describe_source", args={"source_id": "src_sales"})],
|
| 40 |
expected_output="source_shape",
|
| 41 |
-
success_criteria="
|
| 42 |
depends_on=[],
|
| 43 |
parallelizable_with=[],
|
| 44 |
estimated_cost="low",
|
|
@@ -118,7 +118,7 @@ _EXAMPLE_B = TaskList(
|
|
| 118 |
objective="Confirm the sales source exposes order date, revenue, and region.",
|
| 119 |
tool_calls=[ToolCall(tool="describe_source", args={"source_id": "src_sales"})],
|
| 120 |
expected_output="source_shape",
|
| 121 |
-
success_criteria="
|
| 122 |
depends_on=[],
|
| 123 |
parallelizable_with=[],
|
| 124 |
estimated_cost="low",
|
|
@@ -213,7 +213,7 @@ _EXAMPLE_C = TaskList(
|
|
| 213 |
objective="Confirm the sales source exposes order date and revenue.",
|
| 214 |
tool_calls=[ToolCall(tool="describe_source", args={"source_id": "src_sales"})],
|
| 215 |
expected_output="source_shape",
|
| 216 |
-
success_criteria="
|
| 217 |
depends_on=[],
|
| 218 |
parallelizable_with=["t4"],
|
| 219 |
estimated_cost="low",
|
|
|
|
| 38 |
objective="Confirm the sales source exposes category, revenue, and order date.",
|
| 39 |
tool_calls=[ToolCall(tool="describe_source", args={"source_id": "src_sales"})],
|
| 40 |
expected_output="source_shape",
|
| 41 |
+
success_criteria="Produced the orders table schema; the 3 needed columns are present.",
|
| 42 |
depends_on=[],
|
| 43 |
parallelizable_with=[],
|
| 44 |
estimated_cost="low",
|
|
|
|
| 118 |
objective="Confirm the sales source exposes order date, revenue, and region.",
|
| 119 |
tool_calls=[ToolCall(tool="describe_source", args={"source_id": "src_sales"})],
|
| 120 |
expected_output="source_shape",
|
| 121 |
+
success_criteria="Produced the orders table schema; the needed columns are present.",
|
| 122 |
depends_on=[],
|
| 123 |
parallelizable_with=[],
|
| 124 |
estimated_cost="low",
|
|
|
|
| 213 |
objective="Confirm the sales source exposes order date and revenue.",
|
| 214 |
tool_calls=[ToolCall(tool="describe_source", args={"source_id": "src_sales"})],
|
| 215 |
expected_output="source_shape",
|
| 216 |
+
success_criteria="Produced the orders table schema; date and revenue columns present.",
|
| 217 |
depends_on=[],
|
| 218 |
parallelizable_with=["t4"],
|
| 219 |
estimated_cost="low",
|