[KM-652] fix(readiness,ps): substantive = successful analysis; PS needs explicit goal

T1 (is_report_ready): a record is "substantive" only when an analysis task
(analyze_*) SUCCEEDED — not when it merely has findings. A failed run still
writes findings (narrating the failure) and its data-access tasks succeed, so
the old "has findings" check marked failed analyses as report-ready. Fixes the
e2e case where analyze_aggregate failed yet the floor would have said ready.

T3a (problem_statement): require an explicitly-stated objective + metric before
validating. New `missing` field on the draft; is_valid = statement present AND
nothing missing. Prompt hardened ("explicitly stated", not "implied") with a
negative few-shot: a bare data question yields missing=[objective,metric] and
does NOT validate, so the gate stops being a formality. The robust gate (user
confirmation) is T3b, with the frontend.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

src/agents/handlers/problem_statement.py

@@ -1,13 +1,17 @@
 """Problem Statement skill — guide the user to a usable problem statement.
 
 Routed by the orchestrator (intent `problem_statement`) and callable as a skill.
-An LLM drafts/refines the statement from the analysis title + the user's message;
-a **deterministic** check decides whether it's complete enough to validate
-(`problem_statement` + `objective` + `metric` all present) — so the pass is a rule,
-not an LLM judgment. On a valid draft it persists `problem_statement` +
-`problem_validated=True` to the analysis state; otherwise it streams guidance and
+An LLM drafts/refines the statement from the analysis title + the user's message and
+declares what's still `missing`; a check validates only when nothing is missing. The
+model is instructed to fill `objective`/`metric` ONLY from what the user explicitly
+stated — a bare data question ("which X has the most Y?") leaves them in `missing`, so
+it does not auto-validate (the gate stays meaningful). On a valid draft it persists
+`problem_statement` + `problem_validated=True`; otherwise it streams guidance and
 leaves the analysis un-validated.
 
+NOTE: completeness is still a (hardened) LLM judgment — the truly deterministic gate
+is an explicit user confirmation, planned with the frontend (see T3b / #11).
+
 See `ORCHESTRATOR_REWORK_PLAN.md` §4 and the 2026-06-18 checkpoint.
 """
 
@@ -44,10 +48,17 @@ class ProblemStatementDraft(BaseModel):
         ..., description="The refined, standalone problem statement (never empty)."
     )
     objective: str = Field(
-        "", description="What success looks like; empty string if not yet clear."
+        "", description="What success looks like — fill ONLY when the user explicitly "
+        "stated it; never inferred from a data question. Empty otherwise."
     )
     metric: str = Field(
-        "", description="The KPI to move/investigate; empty string if not yet clear."
+        "", description="The KPI to move/investigate — fill ONLY when the user "
+        "explicitly stated it; never inferred from a data question. Empty otherwise."
+    )
+    missing: list[str] = Field(
+        default_factory=list,
+        description="Which of 'objective' / 'metric' the user has NOT explicitly stated "
+        "yet. A bare data question leaves both here. Empty list = complete.",
     )
     feedback: str = Field(
         ...,
@@ -56,12 +67,13 @@ class ProblemStatementDraft(BaseModel):
 
 
 def is_valid(draft: ProblemStatementDraft) -> bool:
-    """Deterministic completeness check — the pass is a rule, not LLM judgment."""
-    return bool(
-        draft.problem_statement.strip()
-        and draft.objective.strip()
-        and draft.metric.strip()
-    )
+    """Complete iff there's a statement and the model flagged nothing missing.
+
+    Keying on the model's explicit `missing` list (rather than 'are objective/metric
+    non-empty') is what stops a bare data question from auto-validating: the hardened
+    prompt puts the un-stated parts in `missing`, so this returns False for it.
+    """
+    return bool(draft.problem_statement.strip()) and not draft.missing
 
 
 def _load_prompt_text() -> str:

src/agents/report/readiness.py

@@ -9,9 +9,12 @@ The rule mirrors what makes a real report non-empty and worth generating, so Hel
 never suggest an action that would 409 or produce a duplicate:
   1. `problem_validated` — the gate's own precondition (no validated goal, no
      analysis worth reporting). Same rule `gate.gate` applies to `structured_flow`.
-  2. at least one **substantive** persisted `AnalysisRecord` — a record that actually
-     produced findings. A failed/empty run still persists a record (failure tasks, no
-     findings); it must not count. We count *results*, not chat turns.
+  2. at least one **substantive** persisted `AnalysisRecord` — a record whose
+     *analysis* task succeeded. A failed run still persists a record WITH findings
+     (they narrate the failure), and data-access tasks (check_/retrieve_) succeed even
+     when the analysis fails — so neither "has findings" nor "any task succeeded" is
+     enough. We require a genuine analysis tool (analyze_*) to have completed. We count
+     *results*, not chat turns.
   3. delta-since-report — if a report already exists (`state.report_id`), only ready
      when there's a substantive analysis newer than the latest report; otherwise the
      new report would be identical.
@@ -68,6 +71,20 @@ def _is_newer(a: datetime, b: datetime) -> bool:
     return a > b
 
 
+def _has_successful_analysis(record) -> bool:
+    """True if the record has at least one *analysis* task that succeeded.
+
+    A failed run still writes findings (narrating the failure) and its data-access
+    tasks (check_/retrieve_) succeed, so we can't key on findings or on "any task
+    succeeded". An analysis tool (analyze_*) completing is the real "we produced a
+    result" signal.
+    """
+    return any(
+        t.status == "success" and any(tool.startswith("analyze") for tool in t.tools_used)
+        for t in record.tasks_run
+    )
+
+
 async def is_report_ready(
     analysis_id: str | None,
     state: AnalysisState,
@@ -89,7 +106,7 @@ async def is_report_ready(
         try:
             store = record_store or _default_record_store()
             records = await store.list_for_analysis(analysis_id)
-            substantive = [r for r in records if r.findings]
+            substantive = [r for r in records if _has_successful_analysis(r)]
         except Exception as exc:  # noqa: BLE001 — never-throw; fail closed to not-ready
             logger.warning(
                 "is_report_ready: record store read failed — not ready",

src/config/prompts/problem_statement.md

@@ -8,18 +8,34 @@ You do not run analysis. You only shape the problem statement and decide whether
 2. **objective** — what success looks like (e.g. "reduce churn", "grow north-region revenue", "understand drivers of retention").
 3. **metric** — the concrete measure to move or investigate (e.g. "churn rate", "monthly revenue", "retention score").
 
-A statement is **complete** only when all three are present and concrete. If the user hasn't given enough to fill `objective` or `metric`, leave them empty and ask for what's missing in `feedback`.
+A statement is **complete** only when all three are present and concrete, and **the user explicitly stated the objective and the metric in their own words**. If they haven't, leave that field empty, list it in `missing`, and ask for it in `feedback`.
+
+**A bare data question is NOT a complete problem statement.** Questions like "which product category has the most revenue?", "what's our top region?", "how many orders last month?" only tell you *what to compute* — they do not state a business objective or a target metric to move. Do **not** infer `objective`/`metric` from such a question. Put both in `missing` and ask the user for the actual goal.
 
 ## Output (structured)
 
 - **`problem_statement`** — your best refined version so far (never empty; use the title if that's all you have).
-- **`objective`** — filled only when clear; otherwise empty string.
-- **`metric`** — filled only when clear; otherwise empty string.
-- **`feedback`** — a short, friendly message to the user. If incomplete: explain what's missing and ask one focused question. If complete: confirm the problem statement back to them and say they can now start analyzing.
+- **`objective`** — filled ONLY when the user explicitly stated it; otherwise empty string.
+- **`metric`** — filled ONLY when the user explicitly stated it; otherwise empty string.
+- **`missing`** — the list of which fields among `objective` / `metric` the user has not yet explicitly stated. Empty list means the statement is complete and will be validated. A bare data question must yield `missing: ["objective", "metric"]`.
+- **`feedback`** — a short, friendly message. If `missing` is non-empty: explain what's missing and ask one focused question. If complete: confirm the problem statement back and say they can start analyzing.
 
 ## Rules
 
 - Be concise and concrete. One focused follow-up question at a time — don't interrogate.
-- Don't invent an objective or metric the user hasn't implied. Empty is correct when unknown.
+- Only fill `objective`/`metric` from what the user **explicitly stated**, never from what a question merely implies. Empty + listed in `missing` is correct when the user hasn't said it.
 - Keep `problem_statement` decision-oriented, not a restatement of the data.
 - Match the user's language (English / Indonesian).
+
+## Examples
+
+**Incomplete — a bare data question (do NOT validate):**
+User: "Which product category generates the most total revenue?"
+→ `problem_statement`: "Identify which product category drives the most total revenue."
+→ `objective`: "" · `metric`: "" · `missing`: ["objective", "metric"]
+→ `feedback`: "Good starting question. To set this up as an analysis goal: what business outcome are you trying to drive (e.g. grow revenue, cut cost), and which metric should we track (e.g. total revenue per category)?"
+
+**Complete — the user stated the goal + metric:**
+User: "Goal: grow total revenue by focusing marketing on the top categories. Metric: total revenue per category."
+→ `objective`: "grow total revenue by focusing on the top categories" · `metric`: "total revenue per category" · `missing`: []
+→ `feedback`: "Your problem statement is complete — you can start the analysis."