[NOTICKET] docs(plans): report reframe and prompt cleanup plan

Question-anchored report with a scope section and an appendix ledger (Phase 1 is
generator-only), and the prompt P0/P1 items from the 2026-10-28 reviews. Open
decisions in section 6.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs/README.md

@@ -59,6 +59,7 @@ Roll-up status: DEV_PLAN §0.8 · built state: REPO_STATUS §14 · eval harness:
 | [plans/PR2_EVALUATION.md](plans/PR2_EVALUATION.md) | 🔄 | **Read before PR-3.** Four layers, three of them free. Establishes that **14 of 14** tools have zero output-correctness coverage |
 | [plans/PR3_REPAIR_EDGE.md](plans/PR3_REPAIR_EDGE.md) | 🔄 | The feedback edge (trigger B, budget 1) **gated on measured hit-rate**, plus the determinism fixes that ship regardless |
 | [plans/PR23_PART2_PLANNER_EVAL.md](plans/PR23_PART2_PLANNER_EVAL.md) | 🔄 | **pr/23 part 2 (2026-09-23).** Planner, router and eval work re-checked against the code: measurement first, then router, retrieve-then-plan, planner structure. **A–E done 2026-09-23** (D4 reverted, F1 optional) |
+| [plans/REPORT_AND_PROMPT_PLAN.md](plans/REPORT_AND_PROMPT_PLAN.md) | 💡 | **2026-10-28.** Report reframe (question-anchored, scope section, appendix ledger; Phase 1 generator-only) + prompt cleanup (one home per fix, P0 token cuts). Open decisions in §6 |
 | [plans/UNSTRUCTURED_V2_PLAN.md](plans/UNSTRUCTURED_V2_PLAN.md) | 🔄 | **The demo track (2026-09-23).** Parsing v2 → embeddings, private images via 1h signed URLs, analysis-scoped retrieval, the mixed structured+unstructured path **without a new intent**. Pre-empts the prompt-architecture work |
 
 ---

docs/plans/REPORT_AND_PROMPT_PLAN.md

@@ -0,0 +1,148 @@
+# Report reframe + prompt cleanup — plan
+
+<!-- doc-status -->
+> 💡 **PROPOSAL — agreed direction, not yet built (2026-10-28).** Report first (Phase 1),
+> then the prompt P0 items. Items marked **DECISION** need Rifqi/team sign-off before work starts.
+
+**Branch context:** `pr/25`. **Why now:** the SDJ / OB HAULER end-to-end test (2026-10-27,
+live Space) produced a report whose defects all came from *accumulation* — every chat turn
+copied in, including wrong and contradictory ones — and two independent reviews (2026-10-28)
+reached the same diagnosis for the report and for the prompts. Defects of that test are logged
+as #27–#32 in [AGENTIC_DRY_RUN_LOG.md](../testing/AGENTIC_DRY_RUN_LOG.md).
+
+---
+
+## 1. Diagnosis
+
+**Report.** It reads as a transcript, not a report. The design guards one risk (the report-time
+LLM inventing numbers) and that risk caused none of the SDJ defects. They entered upstream or in
+selection, and the report certified them:
+
+| Defect in the SDJ report | Enters at |
+|---|---|
+| Scope drift: no site filter, `week 1–6` as Jan–Jun, a follow-up dropped site and year | Planner IR (the report never sees filters) |
+| Two contradictory answers to the same question both included (HDKM46049 vs HDKM46087; DRIVE SYSTEMS vs ENGINE) | Accumulation — no supersession or conflict check |
+| "Most breakdowns" answered with the highest *average* over 7 rows | Metric/question fit — not checked |
+| "Answered" on 5 periods, on the wrong metric, on leaky drivers | Status decided by the LLM alone |
+| Target leakage as "drivers" (`mttr_usm` explains `mttr`) | `analyze_importance` guards train/test leakage only |
+| Site-id lookup (`2005`) shown 6×, raw model inputs as evidence | `_collect_evidence`: any table ≤ 8 columns qualifies |
+| Garbled text, "chart displayed successfully" as findings | Copied verbatim (INV-4) |
+| Data Sources lists all 16 tables / 315 columns | Lists the catalog, not what was queried |
+
+INV-4 ("findings copied verbatim") keeps the report faithful to the chat, not to the data:
+findings are Assembler prose. Real provenance is already persisted per tool output
+(`meta.query`, `meta.table_name`, `meta.row_count`, `tools/data_access.py`).
+
+**Prompts.** Sound design, but `planner.md` is growing one incident at a time again
+(6,023 → 7,142 tokens since 2026-09-22). Rules live in three places (prose + example +
+validator). Tool descriptions contradict `planner.md` (`analyze_aggregate` "total per region",
+`aggregation.py:83`). 11 of 27 examples open with `check_data`, against "smallest plan".
+Incident-specific text in general prompts. Document context placed before the catalog breaks
+prefix caching. A planner call is ~22–23.5k input tokens, ~15k of it instructions + examples.
+
+## 2. Target report shape
+
+Question-anchored, two layers: a short main body (≈ 2 pages) + an appendix ledger. Records stay
+the input (the locked decision holds through Phase 2).
+
+| § | Section | Built by |
+|---|---|---|
+| 1 | Thesis title (one line) + meta | LLM title (checked numbers only), code meta |
+| 2 | Executive summary (3–5 sentences) | LLM, verified numbers only |
+| 3 | **Scope & data**: period actually covered, filters, tables queried, rows, coverage gaps | **Code** (from stored queries) |
+| 4 | **Answers by business question**: answer, status (code may only lower it, with a reason), metric named, one evidence table | LLM drafts, code caps + picks evidence |
+| 5 | Conflicts (only when present) | Code detects, LLM one line |
+| 6 | Limitations + "what this report does not claim" | Code-detected flags + fixed text |
+| 7 | Next analyses to run (as chat prompts) | LLM |
+| 8 | Charts (deduped) | Code |
+| A | **Appendix: analysis ledger** — one row per turn: question, tables, filters, rows, tools, flags, superseded-by | **Code only** |
+
+Not copied from Julius: the generic 30-day playbook, the 11-page length, hypotheses presented as
+recommendations.
+
+## 3. Report work items
+
+Type: Code = deterministic · Prompt · Handoff = outside this repo. Verify: free = local tests /
+fixture; smoke = small paid subset.
+
+### Phase 1 — `src/agents/report/` only; no Go/FE/schema; no extra LLM calls
+
+| ID | Item | Type | Verify | Status |
+|---|---|---|---|---|
+| R1 | Evidence filter: drop 1-row lookups and raw model inputs, dedupe identical tables, title in the report language | Code | free | ⬜ |
+| R2 | Data Sources = tables actually queried + rows returned | Code | free | ⬜ |
+| R3 | Scope fingerprint: read-only sqlglot parse of each stored `meta.query` → tables / filters / group-by / limit | Code | free | ⬜ |
+| R4 | Appendix A: analysis ledger | Code | free | ⬜ |
+| R5 | Status caps (scope mismatch, small sample, conflict, leakage name-check), code-appended reason | Code | free | ⬜ |
+| R6 | Number tracing against stored results; untraced → "unverified" (snapshots keep 10 rows, so mark, don't delete). Reuse the `eval/faithfulness` matcher as a shared module | Code | free | ⬜ |
+| R7 | Text sanity filter: unexpected scripts, tool-success narration | Code | free | ⬜ |
+| R8 | Compact "Scope & limitations" + fixed non-claims footer. **DECISION: reverses the 2026-07-09 "compact report" drop** | Code | free | ⬜ |
+| R9 | `report_summary.md`: feed scope + flags, name the metric per question, allow "inconclusive", thesis title. **First** build a free fixture check from the SDJ v2 records (no eval covers this prompt today) | Prompt | fixture + 1 smoke | ⬜ |
+
+### Phase 2 — Python, touches the Assembler and the stored record shape
+
+| ID | Item | Type | Note |
+|---|---|---|---|
+| R10 | Tag each turn with the business question it answers, at turn time | Prompt + Code | |
+| R11 | Supersession (a later, tighter analysis replaces an earlier one) + Conflicts section | Code | |
+| R12 | Store all numbers a turn saw (fact sheet), not 10 rows | Code | Ask Harry whether it fits `report_inputs` jsonb |
+| R13 | Findings as templates with cell references; code renders numbers | Prompt + Code | `assembler.md` → chat_sim smoke + faithfulness eval |
+| R14 | Per-question layout | Code | |
+| H1 | FE: choose analyses before Generate (the `exclude_record_ids` API exists), show status reasons, collapsible appendix | Handoff | FE via Rifqi |
+
+### Phase 3 — opt-in "Deep report"
+
+| ID | Item | Type | Note |
+|---|---|---|---|
+| R15 | Report-planning agent re-queries through the existing planner/IR pipeline (fair per-exposure comparisons, stability tests, uncertainty); ≤ 8 tasks, token cap, degrades to Phase 2 | Code + Prompt | **DECISION: reverses the locked "records-based" decision.** ~120–250k tokens per report |
+| H2 | Background job + progress (the HF proxy times out long requests — cf. the ingest 502) | Handoff | Harry + FE |
+
+## 4. Prompt work items
+
+### P0 — cheap
+
+| ID | Item | Type | Saves | Verify | Status |
+|---|---|---|---|---|---|
+| P1 | **One home per fix**: validator if checkable, example if a plan shape, prose only for judgment | Process | prevents growth | — | ⬜ |
+| P2 | Strip HTML comments when prompts load (`help.md` sends a 719-token changelog every call) | Code | ~950 tok/call | free + 1 smoke per prompt | ⬜ |
+| P3 | Move `# Document context` to just before `# Question` (`planner/prompt.py:208-229`) | Code | caching of ~10–12k tok | free + planner smoke | ⬜ |
+| P4 | Remove the `check_data` opener from examples A–M | Prompt | ~700 tok/call | `--only` ~6 cases | ⬜ |
+| H3 | Tool descriptions contradicting `planner.md` (`analyze_aggregate` "total per region"; "plot revenue by region") | Handoff | — | tools owner | ⬜ |
+
+### P1 — consolidation (one approved run)
+
+| ID | Item | Verify |
+|---|---|---|
+| P5 | Drop prose for rules the validator already enforces (top-1 limit, typed ids, capped importance, args, aliases) | ~30-case run (~720k) — **DECISION: spend approval** |
+| P6 | Merge the "can't answer" rule (restated ~8×) | same run |
+| P7 | Remove incident-specific text ("kalo di XYZ", "site ABC", hard-coded 10,000-row budget, EXHC36007 in `assembler.md`) | same run + chat smoke |
+| P8 | Add ~6 "can't answer" eval cases first (2 of 68 today) | free to write |
+
+Target after P0 + P1: `planner.md` ~7.1k → ~5k tokens, 15–20% fewer tokens per planner call,
+no known contradictions.
+
+### P2 — later
+
+| ID | Item |
+|---|---|
+| P9 | Bug #30 (score direction) as catalog metadata ("lower is worse"), not another prose rule |
+| P10 | Select examples by relevance instead of sending the whole group |
+
+Keep: stable/dynamic split, group-scoped tools and examples, data fencing, retry-error placement,
+paired feasible/infeasible examples, guard/guardrails/report prompts.
+
+## 5. Order of work
+
+1. Report Phase 1: R1 + R2 → R3 + R4 → R5 + R7 → R6 → R8 (after DECISION) → R9.
+2. Prompt P2 + P3 (free/smoke), then P4.
+3. Handoffs H1–H3 sent by Rifqi.
+4. Phase 2 / P1 consolidation after Phase 1 is in use.
+
+## 6. Open decisions
+
+| Decision | Owner |
+|---|---|
+| Bring back a compact limitations section (R8) | Rifqi / team |
+| Deep report re-querying data (R15) | Rifqi / team / mentor |
+| Spend for the P1 consolidation eval (~720k tokens) | Rifqi |
+| Leakage guard inside `analyze_importance` | Tools owner, via Rifqi |