sofhiaazzhr Claude Opus 4.8 commited on
Commit
11de970
Β·
1 Parent(s): 9070d67

/fix planner count question

Browse files

Scalar "how many X" questions pulled raw rows and let the assembler LLM
tally them, which miscounted (188 vs actual 207). Add a count-aggregate
recipe (R2b) in planner.md plus a few-shot (Example L) so the planner
emits a COUNT(*) IR with the filter, returning the exact count in one
row. No tool/IR/compiler change; verified live (planner now plans a
single retrieve_data count task, answer = 207).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

src/agents/planner/examples.py CHANGED
@@ -843,6 +843,68 @@ _EXAMPLE_K = TaskList(
843
  )
844
 
845
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
846
  EXAMPLES: list[tuple[str, TaskList]] = [
847
  ("Which product categories drove last quarter's revenue?", _EXAMPLE_A),
848
  ("How has monthly revenue trended by region this year, and what's unusual?", _EXAMPLE_B),
@@ -859,6 +921,7 @@ EXAMPLES: list[tuple[str, TaskList]] = [
859
  ),
860
  ("Show me a bar chart of total revenue per product category.", _EXAMPLE_J),
861
  ("Plot the customer churn rate by month as a line chart.", _EXAMPLE_K),
 
862
  ]
863
 
864
 
 
843
  )
844
 
845
 
846
+ # --------------------------------------------------------------------------- #
847
+ # Example L β€” scalar count with a filter (no grouping, no analyze step).
848
+ # "How many orders have zero revenue?"
849
+ # Shows: a "how many rows match X" question is answered by a SINGLE retrieve_data
850
+ # IR whose select is a COUNT(*) aggregate ({"kind": "agg", "fn": "count"}, with
851
+ # column_id omitted) plus the filter β€” it returns the exact number in one row.
852
+ # Do NOT select the raw column and count the returned rows: that caps at `limit`
853
+ # and leaves the tally to be eyeballed. count(*) omits column_id (the validator
854
+ # lets only 'count' do so); no group_by is needed for a scalar count.
855
+ # --------------------------------------------------------------------------- #
856
+
857
+ _EXAMPLE_L = TaskList(
858
+ plan_id="example_l",
859
+ goal_restated="Count how many orders have revenue equal to 0.",
860
+ assumptions=[],
861
+ open_questions=[],
862
+ tasks=[
863
+ Task(
864
+ id="t1",
865
+ stage="data_understanding",
866
+ objective="Confirm the sales source exposes order revenue.",
867
+ tool_calls=[ToolCall(tool="check_data", args={"source_id": "src_sales"})],
868
+ expected_output="source_shape",
869
+ success_criteria="Produced the orders table schema; revenue is present.",
870
+ depends_on=[],
871
+ estimated_cost="low",
872
+ ),
873
+ Task(
874
+ id="t2",
875
+ stage="evaluation",
876
+ objective="Count the orders whose revenue equals 0.",
877
+ tool_calls=[
878
+ ToolCall(
879
+ tool="retrieve_data",
880
+ args={
881
+ "ir": {
882
+ "source_id": "src_sales",
883
+ "table_id": "t_orders",
884
+ "select": [
885
+ {"kind": "agg", "fn": "count", "alias": "order_count"}
886
+ ],
887
+ "filters": [
888
+ {
889
+ "column_id": "c_revenue",
890
+ "op": "=",
891
+ "value": 0,
892
+ "value_type": "decimal",
893
+ }
894
+ ],
895
+ }
896
+ },
897
+ )
898
+ ],
899
+ expected_output="zero_revenue_count",
900
+ success_criteria="Produced one row holding the count of zero-revenue orders.",
901
+ depends_on=["t1"],
902
+ estimated_cost="low",
903
+ ),
904
+ ],
905
+ )
906
+
907
+
908
  EXAMPLES: list[tuple[str, TaskList]] = [
909
  ("Which product categories drove last quarter's revenue?", _EXAMPLE_A),
910
  ("How has monthly revenue trended by region this year, and what's unusual?", _EXAMPLE_B),
 
921
  ),
922
  ("Show me a bar chart of total revenue per product category.", _EXAMPLE_J),
923
  ("Plot the customer churn rate by month as a line chart.", _EXAMPLE_K),
924
+ ("How many orders have zero revenue?", _EXAMPLE_L),
925
  ]
926
 
927
 
src/config/prompts/planner.md CHANGED
@@ -46,6 +46,7 @@ recipe verbatim; a genuinely multi-part question composes recipes.
46
  |---|---|---|
47
  | R1 descriptive | a summary/distribution of columns | `retrieve_data` β†’ `analyze_descriptive` |
48
  | R2 aggregate / top-N | totals or averages per group, "top N by …" | ONE grouped `retrieve_data` IR (Β± `analyze_aggregate`) |
 
49
  | R3 trend | movement over time | `retrieve_data` β†’ `analyze_trend` |
50
  | R4 correlation | the relationship between numeric columns | `retrieve_data` β†’ `analyze_correlation` |
51
  | R5 two-metric merge | "which X has both A and B" | `retrieve_data` Γ—2 β†’ `analyze_merge` β†’ … |
@@ -75,6 +76,15 @@ recipe verbatim; a genuinely multi-part question composes recipes.
75
  so an `analyze_*` fed from them finds no columns to analyze and fails.
76
  `check_data` is only for inspecting *what exists*; always `retrieve_data` to
77
  pull the rows before analyzing them.
 
 
 
 
 
 
 
 
 
78
  - **Measure by a dimension in another table (joins).** When the number you are
79
  aggregating and the grouping dimension live in DIFFERENT tables of the same
80
  database source, add a `joins` entry to the `retrieve_data` IR. **Join ONLY on a
 
46
  |---|---|---|
47
  | R1 descriptive | a summary/distribution of columns | `retrieve_data` β†’ `analyze_descriptive` |
48
  | R2 aggregate / top-N | totals or averages per group, "top N by …" | ONE grouped `retrieve_data` IR (Β± `analyze_aggregate`) |
49
+ | R2b scalar count/total | a single number with NO grouping β€” "how many rows match X", "berapa banyak …", "total …" | ONE `retrieve_data` IR with a `count`/`sum` aggregate + filter, NO `group_by`, NO `analyze_*` step |
50
  | R3 trend | movement over time | `retrieve_data` β†’ `analyze_trend` |
51
  | R4 correlation | the relationship between numeric columns | `retrieve_data` β†’ `analyze_correlation` |
52
  | R5 two-metric merge | "which X has both A and B" | `retrieve_data` Γ—2 β†’ `analyze_merge` β†’ … |
 
76
  so an `analyze_*` fed from them finds no columns to analyze and fails.
77
  `check_data` is only for inspecting *what exists*; always `retrieve_data` to
78
  pull the rows before analyzing them.
79
+ - **Counting rows is a `count` aggregate, not a manual tally.** For a "how many
80
+ rows match X" / "berapa banyak" question β€” a single scalar count with no
81
+ grouping β€” emit ONE `retrieve_data` IR whose `select` is
82
+ `[{"kind": "agg", "fn": "count"}]` (COUNT(*); `column_id` omitted) plus the
83
+ filter. It returns the exact number in one row. Do **NOT** `select` the raw
84
+ column and let a later step (or the reader) count the returned rows β€” that caps
85
+ at `limit` and leaves the tally to be eyeballed, so the count comes out wrong.
86
+ A scalar total/min/max/avg (no grouping) works the same way: aggregate it in the
87
+ IR, don't pull raw rows.
88
  - **Measure by a dimension in another table (joins).** When the number you are
89
  aggregating and the grouping dimension live in DIFFERENT tables of the same
90
  database source, add a `joins` entry to the `retrieve_data` IR. **Join ONLY on a