[NOTICKET] Add revision plan md file

DEV_PLAN.md

@@ -0,0 +1,119 @@
+# Data Eyond — Current Development Plan (post 2026-06-24 meeting)
+
+**Purpose:** context file for Claude Code sessions working on the current sprint.
+**Branch:** `pr/4` · **Snapshot:** 2026-06-25.
+**Companion:** [REPO_STATUS.md](REPO_STATUS.md) describes the repo's *current built state*; this file
+describes the *in-flight plan* that changes it.
+
+---
+
+## 1. The direction change (locked decisions from 2026-06-24)
+
+1. **"Problem statement" is replaced by two user-entered fields: `objective` + `business_questions`.**
+   User fills them at onboarding; **both mandatory to submit; NO agent validation.**
+2. The **gate (`problem_validated`) and the `problem_statement` skill/intent are removed** (comment out, don't delete).
+3. **Report is records-based** (reads persisted `AnalysisRecord`s) — **decided and pushed** (KM-674).
+   It is formal markdown: title, date, "generated by {user}", objective, business questions,
+   findings, insights. **NOT gated** on whether business questions were answered.
+4. **`owner_id` → `user_id`** everywhere (Harry mirrors in dedorch/Go).
+5. **State writes go through a request to Go**, not direct Python DB writes.
+6. **FE-callable surface = 4 endpoints:** `call_agent` (chat/stream), `list_skills` (`GET /tools`),
+   **skill: help**, **skill: report**. `problem_statement` removed; `check_data` not FE-facing from
+   Python (Go provides it); analysis CRUD not needed from FE (comment, don't delete).
+7. Deliverables for Harry: (a) API endpoint doc (MD); (b) full Python project doc (MD → PDF/Word BRD).
+8. Integration tested via Swagger `/docs` on the HF Python build (simulating FE manually). Target ~Wed.
+
+## 2. What is already done (KM-674, pushed on `pr/4`)
+
+Report layer adapted to the new goal shape:
+- `report/schemas.py::ProblemStatement` → `objective: str` + `business_questions: list[str]`
+  (old `target_value`/`scope`/`metric_direction`/`target_metric` dropped). Class name kept for now
+  (rename to `ReportGoal` once the upstream AnalysisState rename lands).
+- `report/generator.py` renders **Objective** + numbered **Business Questions** + a
+  **"generated by {user}"** line.
+- `api/v1/report.py::_problem_statement_from` is **tolerant**: prefers new `objective` /
+  `business_questions` from state, falls back to legacy `problem_statement` — works before AND after
+  Harry's migration.
+- `config/prompts/report_summary.md` updated to objective + business questions.
+- Report stays **records-based**; the floor gate (`problem_validated`) was deliberately left for task #2.
+
+**This tolerant-migration pattern (getattr fallback) is the model for tasks #2 and #4.**
+
+## 3. Assessment — gaps & contradictions to resolve before building
+
+These came out of reviewing the plan against the actual code. They are folded into the task table (§4) as tasks 15–19.
+
+- **G1 (→ task 15). Records-based reports need the slow path ON.** `AnalysisRecord`s persist only in
+  `chat_handler._run_slow_path`, which runs only when `ENABLE_SLOW_PATH=true`. Default is off → no
+  records → `POST /report` 409s. The Swagger demo can't show a non-empty report unless slow path is
+  flipped on and a `structured_flow` question is run first. `BusinessContext` is still a stub but the
+  slow path runs fine on it.
+- **G2 (→ task 16). `analysis_records` ownership is now required and collides with `SKIP_INIT_DB`.**
+  It's created today by Python `create_all` (`db/postgres/init_db.py`), is in no dedorch/Go migration,
+  and after the dedorch cutover (`SKIP_INIT_DB=true`) Python stops running `create_all` → the table
+  won't exist → reports break. Decide: dedorch migration (Harry) OR a Python carve-out that creates
+  just this one table even under `SKIP_INIT_DB`.
+- **G3 (→ task 17). `chat_history` in the report contract is vestigial.** Records-based generation
+  reads records by `analysis_id`; it never uses chat history. Drop `chat_history` from the report
+  skill contract, or mark it reserved/unused.
+- **G4 (→ task 4 note). Make #2/#4 tolerant of both state shapes.** If Harry drops
+  `problem_validated`/`owner_id` from dedorch before Python stops reading them, Python's gate +
+  state_store break. Use the same `getattr` tolerance KM-674 used. The `owner_id`→`user_id` rename
+  also touches `api/v1/analysis.py` (`_serialize_state`, `list_analyses`, `get_analysis`), not just
+  the model + state_store.
+- **G5 (→ task 18). "State writes via Go" is bigger than `report_id`.** Python still writes state in
+  `/analysis/create` (state + room + bindings, plus the data-first gate and soon the mandatory-field
+  check) and in `state_store.ensure` per turn. If creation moves to Go (consistent with commenting
+  analysis CRUD), then Go owns ALL state writes + both creation gates, and Python's `ensure` must
+  become a **read-only get** (Go must guarantee the row exists before any chat turn).
+- **G6 (smaller).**
+  - Removing `problem_statement` (task 1) means neutering it in 4 places: the `Intent` literal
+    (`agents/orchestration.py`), the router prompt (`config/prompts/intent_router.md`), the handler,
+    and the gate's redirect *target*. Do it with task 2.
+  - "generated by {user}" currently prints the raw `user_id`; a formal report wants a name — source
+    from `users.fullname` or have Go pass a display name (task 19).
+  - The meeting's outline (background / EDA / insights) isn't fully in the renderer; map those
+    sections onto the record fields deliberately (task 5 follow-up).
+  - The full project doc (task 11) should reuse [REPO_STATUS.md](REPO_STATUS.md), not restart.
+
+## 4. Task table
+
+Status legend: ⬜ not started · 🔄 in progress · ✅ done · ⛔ blocked · 🔎 verify.
+
+| # | Task | Owner | Status | Note |
+|---|---|---|---|---|
+| 1 | Comment out `problem_statement` skill **+ `Intent` literal + router prompt + gate redirect target**; remove `/problem-statement` from `list_tools` | Rifqi | ⬜ | Do with #2 — 4 places name it |
+| 2 | Drop `problem_validated`: gate → identity/presence-check (or remove); `is_report_ready` → presence(`objective` + ≥1 `business_question`) + ≥1 analysis, no-LLM | Rifqi | ⬜ | Merges Sofhia 2+3. Make reads tolerant (getattr) |
+| 3 | dedorch `analysis` migration: drop `problem_statement`/`problem_validated`, add `objective` + `business_questions`, rename `owner_id`→`user_id` | Harry | ⬜ | Blocks #4 |
+| 4 | Update Python `analysis` model + `state_store` + `analysis.py` to match #3; `owner_id`→`user_id` | Rifqi/Sofhia | ⛔ after #3 | getattr-tolerant so deploy order isn't a hard blocker |
+| 5 | Report generator → `objective`+`business_questions`, "generated by {user}", formal outline | Sofhia | 🔄 pushed (KM-674) | Goal-shape done. Open: author name source (#19); background/EDA/insights mapping |
+| 6 | Report skill input contract: `analysis_id` + `user_id` (+ `chat_history`?) | Sofhia/Rifqi | ⬜ | Records-based ⇒ chat_history unused — see #17 |
+| 7 | `report_id` state update via request to Go, not direct DB | Sofhia + Harry | ⬜ | Needs Go endpoint. See #18 for the rest of the writes |
+| 8 | Expose/confirm 4 FE endpoints; comment `check_data` + analysis CRUD | Sofhia | ⬜ | Confirm `help` via `call_agent` (recommended) vs own endpoint |
+| 9 | Verify `analysis_id` in `call_agent` contract | Sofhia | 🔎 | Present as `room_id` (== `analysis_id`); document the equivalence |
+| 10 | API endpoint doc (MD), 4 endpoints, for Go integration | Rifqi + Sofhia | ⬜ | — |
+| 11 | Full Python project doc (MD → PDF/Word BRD) | Rifqi | ⬜ | Reuse REPO_STATUS.md as the base |
+| 12 | Reconcile/open the `list_tools` PR cleanly (stacked commits) | Rifqi | ⬜ | — |
+| 13 | Merge HF Python build → test 4 endpoints via Swagger | Sofhia + Harry | ⬜ | Not E2E. Blocked by #15 |
+| 14 | `analysis_records` home | Rifqi + Sofhia + lead | ⬜ → required | Records-based decided ⇒ no longer conditional; see #16 |
+| 15 | Flip `ENABLE_SLOW_PATH=true` on HF + verify an `AnalysisRecord` persists from a `structured_flow` question | Rifqi | ⬜ new | Precondition for any report demo (#13) |
+| 16 | Decide `analysis_records` creation under `SKIP_INIT_DB`: dedorch migration (Harry) or Python carve-out | Rifqi + Harry | ⬜ new | Table won't exist post-cutover otherwise |
+| 17 | Reconcile report contract with records-based: remove/flag `chat_history` | Sofhia/Rifqi | ⬜ new | Resolves #5 ↔ #6 contradiction |
+| 18 | Confirm Go owns ALL analysis-state writes + both creation gates; make Python `state_store.ensure` read-only | Rifqi + Harry | ⬜ new | Extends #7 to the full decision #4 (G5) |
+| 19 | Decide report author display-name source (`users.fullname` vs Go-passed name) | Sofhia | ⬜ new | Raw `user_id` in a formal report looks unprofessional |
+
+## 5. Critical path & sequencing
+
+- **Critical path:** #3 → #4 (de-risked by getattr tolerance). #15 + #16 → #13.
+- **Parallelizable now (no blockers):** #1+#2 (together), #8, #10, #11, #12.
+- **Harry-blocked:** #3 (blocks #4), #7 (Go endpoint), #16/#18 (dedorch + Go ownership).
+- **Demo gate for Wednesday (#13):** slow path on (#15) + records table exists (#16) + report contract
+  settled (#17). Without #15/#16 the report endpoint 409s in Swagger.
+
+## 6. Decisions still open (need the team / Harry / lead)
+
+- `analysis_records`: dedorch-owned vs Python-owned-with-carve-out (#16/#14).
+- Whether `help` is its own endpoint or via `call_agent` (#8).
+- Author display-name source for the report (#19).
+- Confirm Go takes over analysis creation + both creation gates (data-first + mandatory fields) (#18).
+- Keep vs drop `chat_history` in the report contract (#17).