DEV_PLAN.md

# Data Eyond — Current Development Plan (post 2026-06-24 meeting + 2026-06-25 checkpoint)

**Purpose:** context file for Claude Code sessions working on the current sprint.
**Branch:** `pr/4` · **Snapshot:** 2026-06-25.
**Companion:** [REPO_STATUS.md](REPO_STATUS.md) describes the repo's *current built state*; this file
describes the *in-flight plan* that changes it. New decisions from the 2026-06-25 checkpoint are in
[§1.5](#15-2026-06-25-checkpoint-deltas).

---

## 1. The direction change (locked decisions from 2026-06-24)

1. **"Problem statement" is replaced by two user-entered fields: `objective` + `business_questions`.**
   User fills them at onboarding; **both mandatory to submit; NO agent validation.**
2. The **gate (`problem_validated`) and the `problem_statement` skill/intent are removed** (comment out, don't delete).
3. **Report is records-based** (reads persisted `AnalysisRecord`s) — **decided and pushed** (KM-674).
   It is formal markdown: title, date, "generated by {user}", objective, business questions,
   findings, insights. **NOT gated** on whether business questions were answered.
4. **`owner_id` → `user_id`** everywhere (Harry mirrors in dedorch/Go).
5. **State writes go through a request to Go**, not direct Python DB writes.
6. **FE-callable surface = 4 endpoints:** `call_agent` (chat/stream), `list_skills` (`GET /tools`),
   **skill: help**, **skill: report**. `problem_statement` removed; `check_data` not FE-facing from
   Python (Go provides it); analysis CRUD not needed from FE (comment, don't delete).
7. Deliverables for Harry: (a) API endpoint doc (MD); (b) full Python project doc (MD → PDF/Word BRD).
8. Integration tested via Swagger `/docs` on the HF Python build (simulating FE manually). Target ~Wed.

## 1.5. 2026-06-25 checkpoint deltas

Confirms the 2026-06-24 direction and adds these concrete changes (folded into §4 as tasks 21–28):

1. **Rename `analysis_records` → `report_inputs`** (DONE #21) — names the table by purpose (the rows
   report generation reads); avoids clashing with Go's `analyses_messages` and with Langfuse
   observability. **Stays Python-owned**; finalized schema handed to Harry so his dedorch migration
   creates it post-`SKIP_INIT_DB` (#22, resolves #16). Write scope = **one row per slow-path analysis
   run** (decided — not per-agent-call telemetry; that stays Langfuse).
2. **`analyses` table (Go) — `status`, `data_bind` + `data_bind_version`, `report_collection`** (id+version).
   **Verified 2026-06-25: these + `user_id` are ALREADY present in dedorch `analyses`.** Plus Harry drops
   the duplicate/wrong singular `analysis` table. (→ #3)
3. **`analyses_messages` (Go) = the analysis chat room** (user Q + agent A) — replaces the now-**deprecated**
   `chat_messages`/`rooms`; Python's chat read/write must migrate here before cutover. (→ #25)
4. **Reports: Go owns ALL writes.** Report stays a **skill** (no router intent): FE → Go → Python;
   Python only returns content. Input = the records table (now `agent_observability`); edit-mode may
   also need the last report. (→ #7/#18/#24)
5. **Markdown minimum now:** tables, **bold**, *italic*, horizontal separators — optimize that before
   anything fancier. (→ #23)
6. **Deferred:** charts (prefer **Plotly→JSON** in a future `chart` table over matplotlib PNGs) and
   images (image table keyed by analysis/message/report + originals in a bucket). (→ #26/#27)
7. **Near-term:** the remove-`problem_statement` work isn't on HF yet → **PR + deploy + test in the
   playground** (#13). Harry stabilizes Go ~Fri; FE manual testing ~Mon. **Keep it playground-able.**
8. **UI research** (no dedicated UI person): new-analysis form (title/objective/business_questions),
   knowledge menu (user-level vs analysis-level binding), report artifacts panel + version selector;
   interview + old analysis UI removed. (→ #28)

## 2. What is already done (KM-674, pushed on `pr/4`)

Report layer adapted to the new goal shape:
- `report/schemas.py::ProblemStatement` → `objective: str` + `business_questions: list[str]`
  (old `target_value`/`scope`/`metric_direction`/`target_metric` dropped). Class name kept for now
  (rename to `ReportGoal` once the upstream AnalysisState rename lands).
- `report/generator.py` renders **Objective** + numbered **Business Questions** + a
  **"generated by {user}"** line.
- `api/v1/report.py::_problem_statement_from` is **tolerant**: prefers new `objective` /
  `business_questions` from state, falls back to legacy `problem_statement` — works before AND after
  Harry's migration.
- `config/prompts/report_summary.md` updated to objective + business questions.
- Report stays **records-based**; the floor gate (`problem_validated`) was deliberately left for task #2.

**This tolerant-migration pattern (getattr fallback) is the model for tasks #2 and #4.**

## 3. Assessment — gaps & contradictions to resolve before building

These came out of reviewing the plan against the actual code. They are folded into the task table (§4) as tasks 15–19.

- **G1 (→ task 15). Records-based reports need the slow path ON.** `AnalysisRecord`s persist only in
  `chat_handler._run_slow_path`, which runs only when `ENABLE_SLOW_PATH=true`. Default is off → no
  records → `POST /report` 409s. The Swagger demo can't show a non-empty report unless slow path is
  flipped on and a `structured_flow` question is run first. `BusinessContext` is still a stub but the
  slow path runs fine on it.
- **G2 (→ task 16). `analysis_records` ownership is now required and collides with `SKIP_INIT_DB`.**
  It's created today by Python `create_all` (`db/postgres/init_db.py`), is in no dedorch/Go migration,
  and after the dedorch cutover (`SKIP_INIT_DB=true`) Python stops running `create_all` → the table
  won't exist → reports break. Decide: dedorch migration (Harry) OR a Python carve-out that creates
  just this one table even under `SKIP_INIT_DB`. *(Resolved 2026-06-25 — see §1.5.1 / #16 / #22.)*
- **G3 (→ task 17). `chat_history` in the report contract is vestigial.** Records-based generation
  reads records by `analysis_id`; it never uses chat history. Drop `chat_history` from the report
  skill contract, or mark it reserved/unused.
- **G4 (→ task 4 note). Make #2/#4 tolerant of both state shapes.** If Harry drops
  `problem_validated`/`owner_id` from dedorch before Python stops reading them, Python's gate +
  state_store break. Use the same `getattr` tolerance KM-674 used. The `owner_id`→`user_id` rename
  also touches `api/v1/analysis.py` (`_serialize_state`, `list_analyses`, `get_analysis`), not just
  the model + state_store.
- **G5 (→ task 18). "State writes via Go" is bigger than `report_id`.** Python still writes state in
  `/analysis/create` (state + room + bindings, plus the data-first gate and soon the mandatory-field
  check) and in `state_store.ensure` per turn. If creation moves to Go (consistent with commenting
  analysis CRUD), then Go owns ALL state writes + both creation gates, and Python's `ensure` must
  become a **read-only get** (Go must guarantee the row exists before any chat turn).
- **G6 (smaller).**
  - Removing `problem_statement` (task 1) means neutering it in 4 places: the `Intent` literal
    (`agents/orchestration.py`), the router prompt (`config/prompts/intent_router.md`), the handler,
    and the gate's redirect *target*. Do it with task 2.
  - "generated by {user}" currently prints the raw `user_id`; a formal report wants a name — source
    from `users.fullname` or have Go pass a display name (task 19).
  - The meeting's outline (background / EDA / insights) isn't fully in the renderer; map those
    sections onto the record fields deliberately (task 5 follow-up).
  - The full project doc (task 11) should reuse [REPO_STATUS.md](REPO_STATUS.md), not restart.

## 4. Task table

Status legend: ⬜ not started · 🔄 in progress · ✅ done · ⛔ blocked · 🔎 verify · ⏸️ deferred.

| # | Task | Owner | Status | Note |
|---|---|---|---|---|
| 1 | Comment out `problem_statement` skill **+ `Intent` literal + router prompt + gate redirect target**; remove `/problem-statement` from `list_tools` | Rifqi | ✅ | Done 2026-06-25 (one commit w/ #2). Unwired in `orchestration.py`, `intent_router.md`, `chat_handler.py`, `tools.py`; `problem_statement.py` kept intact |
| 2 | Drop `problem_validated`: gate neutered; `is_report_ready`/`report_floor` → **≥1 completed analysis** only, no-LLM | Rifqi | ✅ | Done 2026-06-25. `gate.py` no-op, gate call site commented in `chat_handler.py`, `report_floor` drops the goal check. Tests updated (`test_gate`/`test_chat_handler`/`test_readiness`). Suite: **284 passed, 7 skipped**; ruff clean |
| 3 | dedorch `analyses` migration: drop `problem_statement`/`problem_validated`, add `objective` + `business_questions` | Harry | 🔄 | **Verified dedorch 2026-06-25:** `analyses` (plural) ALREADY has `user_id` + `status` + `data_bind` + `data_bind_version` + `report_collection` → those parts done. **Remaining:** drop `problem_statement`/`problem_validated` + add `objective`/`business_questions`. Singular `analysis` = deprecated duplicate to drop |
| 4 | Update Python `analyses` model + `state_store` + `analysis.py` to match dedorch; `owner_id`→`user_id` | Rifqi/Sofhia | ✅ | Done 2026-06-26. `owner_id`→`user_id` + added `status`/`data_bind`/`data_bind_version`/`report_collection` (DB-only, not in the `AnalysisState` pydantic) across `models.py`/`gate.py`/`state_store.py`/`analysis.py` + 3 local tests; also `report_inputs` `id`/`analysis_id` → `uuid`. Kept `problem_statement`/`problem_validated`; `objective`/`business_questions` wait on Harry's #3. Suite **284 passed** |
| 5 | Report generator → `objective`+`business_questions`, "generated by {user}", formal outline | Sofhia | ✅ | Goal-shape (KM-674) + author name (#19) + outline (KM-680): Objective → Business Questions → Executive Summary → Key Findings → EDA → Notes & Limitations → How This Was Analyzed |
| 6 | Report skill input contract: `analysis_id` + `user_id` (no `chat_history`) | Sofhia/Rifqi | ✅ | No-op: `POST /report` already takes only analysis_id + user_id (records-based). Documented in API_ENDPOINTS.md §5. *(Edit-mode input revisited in #24.)* |
| 7 | `report_id` state update via request to Go, not direct DB | Sofhia + Harry | ⬜ | Needs Go endpoint. **Checkpoint:** Go owns ALL `reports` writes; Python stops any direct insert/update and only returns content; report stays a **skill** (no intent). See #18 |
| 8 | Expose/confirm 4 FE endpoints; comment `check_data` + analysis CRUD | Sofhia | ✅ | KM-678: `list_tools` trimmed to `/help` + `/report` (analytics/check/retrieve commented in the **menu**). `help` confirmed as a `call_agent` intent — no own endpoint. Analysis CRUD endpoint left **registered**: "comment the rest" was about the FE slash menu, not killing HTTP routes Go needs |
| 9 | Verify `analysis_id` in `call_agent` contract | Sofhia | ✅ | Verified: no separate field — carried as `room_id` (`analysis_id == room_id`), per REPO_STATUS §4/§11. Action for Go: send the id as `room_id` |
| 10 | API endpoint doc (MD), 4 endpoints, for Go integration | Rifqi + Sofhia | ✅ | Done 2026-06-25 — `API_ENDPOINTS.md` (repo root). 4 FE surfaces with request/response **examples** (chat SSE transcript, report 201/409 JSON, version list), schemas, §9 full 32-route inventory + task-8 reading |
| 11 | Full Python project doc (MD → PDF/Word BRD) | Rifqi | ✅ | Done 2026-06-26 — `PROJECT_BRD.md` (repo root): purpose/context, FR-1..9 capabilities, lifecycle, architecture, data model, API (→ API_ENDPOINTS), NFRs, integrations, open items. Reuses REPO_STATUS/API_ENDPOINTS; convert to PDF/Word for distribution |
| 12 | Reconcile/open the `list_tools` PR cleanly (stacked commits) | Rifqi | ✅ | N/A — we develop directly on the single active branch `pr/4` (KM-652 + KM-678 already stacked there); no separate PR to reconcile |
| 13 | Deploy HF Python build (remove-`problem_statement` work) → test 4 endpoints via Swagger / playground | Sofhia + Harry | 🔄 | **Unblocked (#15 ✅).** Remove-PS work is on `pr/4` but **not on HF `main` yet** → PR + deploy, then manual test. Harry stabilizes Go ~Fri; FE testing ~Mon |
| 14 | `analysis_records` home | Rifqi + Sofhia + lead | ✅ | **Resolved 2026-06-25:** stays Python-owned, **renamed** (→ #21); schema handed to Harry so the dedorch migration creates it post-cutover (→ #22). Not moved to Go |
| 15 | Flip `ENABLE_SLOW_PATH=true` + verify an `AnalysisRecord` persists from a `structured_flow` question | Rifqi | ✅ | Verified locally 2026-06-25 (in-process). structured_flow on Titanic.csv → 3-task plan `check_data→retrieve_data→analyze_aggregate` (all success) → AnalysisRecord persisted (substantive) → `report_floor` pass → report generates (201). HF env-flip + Swagger run folds into #13 |
| 16 | Decide `analysis_records` creation under `SKIP_INIT_DB` | Rifqi + Harry | ✅ | **Resolved 2026-06-25:** Python defines it; **Harry's dedorch migration creates it** on env-move (Python still creates locally meanwhile) → exists post-cutover. Execution = #22 |
| 17 | Reconcile report contract with records-based: remove/flag `chat_history` | Sofhia/Rifqi | ✅ | Nothing to remove — `chat_history` was never in the report contract/code (only in help.md). Confirmed via grep; API_ENDPOINTS.md §5 documents the clean contract |
| 18 | Confirm Go owns ALL analysis-state writes + both creation gates; make Python `state_store.ensure` read-only | Rifqi + Harry | ⬜ | **Confirmed by 2026-06-25 checkpoint** (Python read-only; Go owns writes + new tables). Execution pending Go endpoints |
| 19 | Decide report author display-name source (`users.fullname` vs Go-passed name) | Sofhia | ✅ | Done 2026-06-25. `AnalysisReport.user_name`; `generator` renders `user_name or user_id`; `api/v1/report.py::_resolve_user_name` reads `users.fullname` never-throw (fallback `user_id`). Decided: resolve in Python (unblocked); swap to Go-passed name later if preferred |
| 20 | **Help handoff:** update `handlers/help.py` + `help.md` — drop the `problem_validated` tier + `define_problem_statement` action (the skill it points at is gone as of #1) | Sofhia | ✅ | Done 2026-06-25. `help.py`: actions = `ask_analysis_question` (always) + `generate_report` (if ready); renders objective/business_questions (getattr-tolerant). `help.md` v1→v2: 3 tiers, no `/problem_statement`, `/generate report`→`/report`. Local test_help updated → 11 pass |
| 21 | Rename `analysis_records` → **`report_inputs`** (table, ORM `ReportInputRow`, store `*ReportInputStore`) | Rifqi | ✅ | Done 2026-06-26. `sed` rename across 9 files; Pydantic `AnalysisRecord` kept; columns stay String (pure rename — uuid+FK is the #22 Harry schema). Name `report_inputs` (purpose; avoids Langfuse/`analyses_messages` clash). Write scope = one row per slow-path run. Suite **284 passed** |
| 22 | Finalize `report_inputs` schema → hand to Harry for the dedorch migration | Rifqi → Harry | 🔄 | **DDL ready** (uuid `id`/`analysis_id` + FK→`analyses(id)`; `user_id`/`plan_id` text; `data` jsonb = serialized `AnalysisRecord`, shape documented). dedorch has empty `analysis_records` → rename. Resolves #16. **Action: send Harry the DDL + `data` shape** |
| 23 | Report markdown formatting: tables, **bold**, *italic*, horizontal separators | Sofhia | ✅ | Done 2026-06-25. Added `---` separators between header + each section in `_render_markdown`. Tables (EDA) / bold (method labels) / italic (meta + citations) already emitted. Relaxed `report_summary.md` to allow inline `**bold**`/`*italic*` for emphasis (kept no-headings/no-bullets so it doesn't duplicate the section structure / Key Findings). Compile + ruff clean |
| 24 | Clarify report input contract: records table (+ `last_report` for edit mode?) | Rifqi/Sofhia ↔ Harry | ⬜ new | Edit-mode input left open at the checkpoint |
| 25 | Migrate Python chat path to Go `analyses_messages` (+ `analyses`) | Rifqi ↔ Harry | ⬜ | **Bigger than "confirm" (verified 2026-06-25):** dedorch `rooms` + `chat_messages` are **deprecated** (`zdeprecated_*`). Python's `Room`/`ChatMessage` models + `chat.py` `load_history`/`save_messages` target them → **break post-cutover**. Move history read/write to `analyses_messages` before the conn-string cutover |
| 26 | **Charts (DEFERRED):** store Plotly JSON in a future `chart` table (not matplotlib PNG) | — | ⏸️ | After the markdown path is done end-to-end |
| 27 | **Images (DEFERRED):** image table (id, analysis_id, msg/report ref, order) + originals in a bucket | — | ⏸️ | Maintenance-heavy; parked |
| 28 | **UI research** (FE): new-analysis form, knowledge menu (user vs analysis level), report artifacts + version selector | Team | ⬜ new | No dedicated UI person; interview + old analysis UI removed |

## 5. Critical path & sequencing

- **Critical path:** #22 (send Harry the `report_inputs` schema). HF deploy (#13) for the playground. (#4 ✅, #21 ✅; Harry's #3 no longer blocks us — Python is getattr-tolerant.)
- **Parallelizable now:** #22 (handoff). (#4 ✅, #11 ✅ done.)
- **Harry-blocked / coordinated:** #3 (now 🔄, blocks #4), #7 (Go endpoint), #18 (Go state ownership), #24 (contract). **#25 = chat-path migration to `analyses_messages` — a cutover blocker.**
- **Demo gate (playground, #13):** deploy the remove-`problem_statement` work to HF — slow path (#15 ✅)
  and the report path are verified locally, and #16 is resolved (#22 hands Harry the schema). **Keep it
  playground-able.**

## 6. Decisions still open (need the team / Harry / lead)

- ~~`analysis_records`: dedorch-owned vs Python-owned (#16/#14).~~ RESOLVED: Python-owned + renamed **`report_inputs`** (#21 done); Harry's migration creates it (#22).
- ~~Whether `help` is its own endpoint or via `call_agent` (#8).~~ RESOLVED: `help` is a `call_agent` intent (no own endpoint).
- ~~Author display-name source for the report (#19).~~ RESOLVED: Python resolves `users.fullname` (fallback `user_id`); swap to a Go-passed name later if preferred.
- ~~Keep vs drop `chat_history` in the report contract (#17).~~ RESOLVED: never in the contract; report is records-based (analysis_id + user_id only).
- Confirm Go takes over analysis creation + both creation gates (data-first + mandatory fields) (#18).
- **Report input for edit mode** — does Python need the last report content? (#24)
- ~~`report_inputs` write scope — every agent call vs slow-path-only? (#21)~~ RESOLVED: one row per slow-path run (telemetry stays Langfuse).
- **Python history source** — confirm Go's `analysis_message` (#25).