Commit Β·
11de970
1
Parent(s): 9070d67
/fix planner count question
Browse filesScalar "how many X" questions pulled raw rows and let the assembler LLM
tally them, which miscounted (188 vs actual 207). Add a count-aggregate
recipe (R2b) in planner.md plus a few-shot (Example L) so the planner
emits a COUNT(*) IR with the filter, returning the exact count in one
row. No tool/IR/compiler change; verified live (planner now plans a
single retrieve_data count task, answer = 207).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- src/agents/planner/examples.py +63 -0
- src/config/prompts/planner.md +10 -0
src/agents/planner/examples.py
CHANGED
|
@@ -843,6 +843,68 @@ _EXAMPLE_K = TaskList(
|
|
| 843 |
)
|
| 844 |
|
| 845 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 846 |
EXAMPLES: list[tuple[str, TaskList]] = [
|
| 847 |
("Which product categories drove last quarter's revenue?", _EXAMPLE_A),
|
| 848 |
("How has monthly revenue trended by region this year, and what's unusual?", _EXAMPLE_B),
|
|
@@ -859,6 +921,7 @@ EXAMPLES: list[tuple[str, TaskList]] = [
|
|
| 859 |
),
|
| 860 |
("Show me a bar chart of total revenue per product category.", _EXAMPLE_J),
|
| 861 |
("Plot the customer churn rate by month as a line chart.", _EXAMPLE_K),
|
|
|
|
| 862 |
]
|
| 863 |
|
| 864 |
|
|
|
|
| 843 |
)
|
| 844 |
|
| 845 |
|
| 846 |
+
# --------------------------------------------------------------------------- #
|
| 847 |
+
# Example L β scalar count with a filter (no grouping, no analyze step).
|
| 848 |
+
# "How many orders have zero revenue?"
|
| 849 |
+
# Shows: a "how many rows match X" question is answered by a SINGLE retrieve_data
|
| 850 |
+
# IR whose select is a COUNT(*) aggregate ({"kind": "agg", "fn": "count"}, with
|
| 851 |
+
# column_id omitted) plus the filter β it returns the exact number in one row.
|
| 852 |
+
# Do NOT select the raw column and count the returned rows: that caps at `limit`
|
| 853 |
+
# and leaves the tally to be eyeballed. count(*) omits column_id (the validator
|
| 854 |
+
# lets only 'count' do so); no group_by is needed for a scalar count.
|
| 855 |
+
# --------------------------------------------------------------------------- #
|
| 856 |
+
|
| 857 |
+
_EXAMPLE_L = TaskList(
|
| 858 |
+
plan_id="example_l",
|
| 859 |
+
goal_restated="Count how many orders have revenue equal to 0.",
|
| 860 |
+
assumptions=[],
|
| 861 |
+
open_questions=[],
|
| 862 |
+
tasks=[
|
| 863 |
+
Task(
|
| 864 |
+
id="t1",
|
| 865 |
+
stage="data_understanding",
|
| 866 |
+
objective="Confirm the sales source exposes order revenue.",
|
| 867 |
+
tool_calls=[ToolCall(tool="check_data", args={"source_id": "src_sales"})],
|
| 868 |
+
expected_output="source_shape",
|
| 869 |
+
success_criteria="Produced the orders table schema; revenue is present.",
|
| 870 |
+
depends_on=[],
|
| 871 |
+
estimated_cost="low",
|
| 872 |
+
),
|
| 873 |
+
Task(
|
| 874 |
+
id="t2",
|
| 875 |
+
stage="evaluation",
|
| 876 |
+
objective="Count the orders whose revenue equals 0.",
|
| 877 |
+
tool_calls=[
|
| 878 |
+
ToolCall(
|
| 879 |
+
tool="retrieve_data",
|
| 880 |
+
args={
|
| 881 |
+
"ir": {
|
| 882 |
+
"source_id": "src_sales",
|
| 883 |
+
"table_id": "t_orders",
|
| 884 |
+
"select": [
|
| 885 |
+
{"kind": "agg", "fn": "count", "alias": "order_count"}
|
| 886 |
+
],
|
| 887 |
+
"filters": [
|
| 888 |
+
{
|
| 889 |
+
"column_id": "c_revenue",
|
| 890 |
+
"op": "=",
|
| 891 |
+
"value": 0,
|
| 892 |
+
"value_type": "decimal",
|
| 893 |
+
}
|
| 894 |
+
],
|
| 895 |
+
}
|
| 896 |
+
},
|
| 897 |
+
)
|
| 898 |
+
],
|
| 899 |
+
expected_output="zero_revenue_count",
|
| 900 |
+
success_criteria="Produced one row holding the count of zero-revenue orders.",
|
| 901 |
+
depends_on=["t1"],
|
| 902 |
+
estimated_cost="low",
|
| 903 |
+
),
|
| 904 |
+
],
|
| 905 |
+
)
|
| 906 |
+
|
| 907 |
+
|
| 908 |
EXAMPLES: list[tuple[str, TaskList]] = [
|
| 909 |
("Which product categories drove last quarter's revenue?", _EXAMPLE_A),
|
| 910 |
("How has monthly revenue trended by region this year, and what's unusual?", _EXAMPLE_B),
|
|
|
|
| 921 |
),
|
| 922 |
("Show me a bar chart of total revenue per product category.", _EXAMPLE_J),
|
| 923 |
("Plot the customer churn rate by month as a line chart.", _EXAMPLE_K),
|
| 924 |
+
("How many orders have zero revenue?", _EXAMPLE_L),
|
| 925 |
]
|
| 926 |
|
| 927 |
|
src/config/prompts/planner.md
CHANGED
|
@@ -46,6 +46,7 @@ recipe verbatim; a genuinely multi-part question composes recipes.
|
|
| 46 |
|---|---|---|
|
| 47 |
| R1 descriptive | a summary/distribution of columns | `retrieve_data` β `analyze_descriptive` |
|
| 48 |
| R2 aggregate / top-N | totals or averages per group, "top N by β¦" | ONE grouped `retrieve_data` IR (Β± `analyze_aggregate`) |
|
|
|
|
| 49 |
| R3 trend | movement over time | `retrieve_data` β `analyze_trend` |
|
| 50 |
| R4 correlation | the relationship between numeric columns | `retrieve_data` β `analyze_correlation` |
|
| 51 |
| R5 two-metric merge | "which X has both A and B" | `retrieve_data` Γ2 β `analyze_merge` β β¦ |
|
|
@@ -75,6 +76,15 @@ recipe verbatim; a genuinely multi-part question composes recipes.
|
|
| 75 |
so an `analyze_*` fed from them finds no columns to analyze and fails.
|
| 76 |
`check_data` is only for inspecting *what exists*; always `retrieve_data` to
|
| 77 |
pull the rows before analyzing them.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 78 |
- **Measure by a dimension in another table (joins).** When the number you are
|
| 79 |
aggregating and the grouping dimension live in DIFFERENT tables of the same
|
| 80 |
database source, add a `joins` entry to the `retrieve_data` IR. **Join ONLY on a
|
|
|
|
| 46 |
|---|---|---|
|
| 47 |
| R1 descriptive | a summary/distribution of columns | `retrieve_data` β `analyze_descriptive` |
|
| 48 |
| R2 aggregate / top-N | totals or averages per group, "top N by β¦" | ONE grouped `retrieve_data` IR (Β± `analyze_aggregate`) |
|
| 49 |
+
| R2b scalar count/total | a single number with NO grouping β "how many rows match X", "berapa banyak β¦", "total β¦" | ONE `retrieve_data` IR with a `count`/`sum` aggregate + filter, NO `group_by`, NO `analyze_*` step |
|
| 50 |
| R3 trend | movement over time | `retrieve_data` β `analyze_trend` |
|
| 51 |
| R4 correlation | the relationship between numeric columns | `retrieve_data` β `analyze_correlation` |
|
| 52 |
| R5 two-metric merge | "which X has both A and B" | `retrieve_data` Γ2 β `analyze_merge` β β¦ |
|
|
|
|
| 76 |
so an `analyze_*` fed from them finds no columns to analyze and fails.
|
| 77 |
`check_data` is only for inspecting *what exists*; always `retrieve_data` to
|
| 78 |
pull the rows before analyzing them.
|
| 79 |
+
- **Counting rows is a `count` aggregate, not a manual tally.** For a "how many
|
| 80 |
+
rows match X" / "berapa banyak" question β a single scalar count with no
|
| 81 |
+
grouping β emit ONE `retrieve_data` IR whose `select` is
|
| 82 |
+
`[{"kind": "agg", "fn": "count"}]` (COUNT(*); `column_id` omitted) plus the
|
| 83 |
+
filter. It returns the exact number in one row. Do **NOT** `select` the raw
|
| 84 |
+
column and let a later step (or the reader) count the returned rows β that caps
|
| 85 |
+
at `limit` and leaves the tally to be eyeballed, so the count comes out wrong.
|
| 86 |
+
A scalar total/min/max/avg (no grouping) works the same way: aggregate it in the
|
| 87 |
+
IR, don't pull raw rows.
|
| 88 |
- **Measure by a dimension in another table (joins).** When the number you are
|
| 89 |
aggregating and the grouping dimension live in DIFFERENT tables of the same
|
| 90 |
database source, add a `joins` entry to the `retrieve_data` IR. **Join ONLY on a
|