Rifqi Hafizuddin Claude Opus 4.8 commited on
Commit
b126e18
·
1 Parent(s): 1759851

[NOTICKET] feat(db): reconcile analyses model to dedorch (#4) + project BRD (#11)

Browse files

- #4: AnalysisStateRow owner_id->user_id + add status/data_bind/data_bind_version/
report_collection (DB-only, not in the AnalysisState pydantic); readers updated
(gate.py, state_store.py, api/v1/analysis.py) + 3 local tests; report_inputs
id/analysis_id -> uuid. Kept problem_statement/problem_validated; objective/
business_questions await Harry's #3. Suite: 284 passed, 7 skipped.
- #11: PROJECT_BRD.md - project requirements + design (reuses REPO_STATUS / API_ENDPOINTS).
- DEV_PLAN: #4, #11 marked done.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

DEV_PLAN.md CHANGED
@@ -113,14 +113,14 @@ Status legend: ⬜ not started · 🔄 in progress · ✅ done · ⛔ blocked ·
113
  | 1 | Comment out `problem_statement` skill **+ `Intent` literal + router prompt + gate redirect target**; remove `/problem-statement` from `list_tools` | Rifqi | ✅ | Done 2026-06-25 (one commit w/ #2). Unwired in `orchestration.py`, `intent_router.md`, `chat_handler.py`, `tools.py`; `problem_statement.py` kept intact |
114
  | 2 | Drop `problem_validated`: gate neutered; `is_report_ready`/`report_floor` → **≥1 completed analysis** only, no-LLM | Rifqi | ✅ | Done 2026-06-25. `gate.py` no-op, gate call site commented in `chat_handler.py`, `report_floor` drops the goal check. Tests updated (`test_gate`/`test_chat_handler`/`test_readiness`). Suite: **284 passed, 7 skipped**; ruff clean |
115
  | 3 | dedorch `analyses` migration: drop `problem_statement`/`problem_validated`, add `objective` + `business_questions` | Harry | 🔄 | **Verified dedorch 2026-06-25:** `analyses` (plural) ALREADY has `user_id` + `status` + `data_bind` + `data_bind_version` + `report_collection` → those parts done. **Remaining:** drop `problem_statement`/`problem_validated` + add `objective`/`business_questions`. Singular `analysis` = deprecated duplicate to drop |
116
- | 4 | Update Python `analyses` model + `state_store` + `analysis.py` to match dedorch; `owner_id`→`user_id` | Rifqi/Sofhia | 🔄 | **Tablename done** (`models.py` `analysis`→`analyses`, uncommitted). **Remaining:** `owner_id`→`user_id` + add `status`/`data_bind`/`data_bind_version`/`report_collection` across `models.py`/`gate.py`/`state_store.py`/`analysis.py` (getattr-tolerant) + a test run before HF |
117
  | 5 | Report generator → `objective`+`business_questions`, "generated by {user}", formal outline | Sofhia | ✅ | Goal-shape (KM-674) + author name (#19) + outline (KM-680): Objective → Business Questions → Executive Summary → Key Findings → EDA → Notes & Limitations → How This Was Analyzed |
118
  | 6 | Report skill input contract: `analysis_id` + `user_id` (no `chat_history`) | Sofhia/Rifqi | ✅ | No-op: `POST /report` already takes only analysis_id + user_id (records-based). Documented in API_ENDPOINTS.md §5. *(Edit-mode input revisited in #24.)* |
119
  | 7 | `report_id` state update via request to Go, not direct DB | Sofhia + Harry | ⬜ | Needs Go endpoint. **Checkpoint:** Go owns ALL `reports` writes; Python stops any direct insert/update and only returns content; report stays a **skill** (no intent). See #18 |
120
  | 8 | Expose/confirm 4 FE endpoints; comment `check_data` + analysis CRUD | Sofhia | ✅ | KM-678: `list_tools` trimmed to `/help` + `/report` (analytics/check/retrieve commented in the **menu**). `help` confirmed as a `call_agent` intent — no own endpoint. Analysis CRUD endpoint left **registered**: "comment the rest" was about the FE slash menu, not killing HTTP routes Go needs |
121
  | 9 | Verify `analysis_id` in `call_agent` contract | Sofhia | ✅ | Verified: no separate field — carried as `room_id` (`analysis_id == room_id`), per REPO_STATUS §4/§11. Action for Go: send the id as `room_id` |
122
  | 10 | API endpoint doc (MD), 4 endpoints, for Go integration | Rifqi + Sofhia | ✅ | Done 2026-06-25 — `API_ENDPOINTS.md` (repo root). 4 FE surfaces with request/response **examples** (chat SSE transcript, report 201/409 JSON, version list), schemas, §9 full 32-route inventory + task-8 reading |
123
- | 11 | Full Python project doc (MD → PDF/Word BRD) | Rifqi | | Reuse REPO_STATUS.md as the base |
124
  | 12 | Reconcile/open the `list_tools` PR cleanly (stacked commits) | Rifqi | ✅ | N/A — we develop directly on the single active branch `pr/4` (KM-652 + KM-678 already stacked there); no separate PR to reconcile |
125
  | 13 | Deploy HF Python build (remove-`problem_statement` work) → test 4 endpoints via Swagger / playground | Sofhia + Harry | 🔄 | **Unblocked (#15 ✅).** Remove-PS work is on `pr/4` but **not on HF `main` yet** → PR + deploy, then manual test. Harry stabilizes Go ~Fri; FE testing ~Mon |
126
  | 14 | `analysis_records` home | Rifqi + Sofhia + lead | ✅ | **Resolved 2026-06-25:** stays Python-owned, **renamed** (→ #21); schema handed to Harry so the dedorch migration creates it post-cutover (→ #22). Not moved to Go |
@@ -141,8 +141,8 @@ Status legend: ⬜ not started · 🔄 in progress · ✅ done · ⛔ blocked ·
141
 
142
  ## 5. Critical path & sequencing
143
 
144
- - **Critical path:** #3 → #4 (de-risked by getattr tolerance). #22 (send Harry the `report_inputs` schema; #21 done). HF deploy (#13) for the playground.
145
- - **Parallelizable now:** #11, #22 (handoff). #4 in progress (tablename done; columns + readers next).
146
  - **Harry-blocked / coordinated:** #3 (now 🔄, blocks #4), #7 (Go endpoint), #18 (Go state ownership), #24 (contract). **#25 = chat-path migration to `analyses_messages` — a cutover blocker.**
147
  - **Demo gate (playground, #13):** deploy the remove-`problem_statement` work to HF — slow path (#15 ✅)
148
  and the report path are verified locally, and #16 is resolved (#22 hands Harry the schema). **Keep it
 
113
  | 1 | Comment out `problem_statement` skill **+ `Intent` literal + router prompt + gate redirect target**; remove `/problem-statement` from `list_tools` | Rifqi | ✅ | Done 2026-06-25 (one commit w/ #2). Unwired in `orchestration.py`, `intent_router.md`, `chat_handler.py`, `tools.py`; `problem_statement.py` kept intact |
114
  | 2 | Drop `problem_validated`: gate neutered; `is_report_ready`/`report_floor` → **≥1 completed analysis** only, no-LLM | Rifqi | ✅ | Done 2026-06-25. `gate.py` no-op, gate call site commented in `chat_handler.py`, `report_floor` drops the goal check. Tests updated (`test_gate`/`test_chat_handler`/`test_readiness`). Suite: **284 passed, 7 skipped**; ruff clean |
115
  | 3 | dedorch `analyses` migration: drop `problem_statement`/`problem_validated`, add `objective` + `business_questions` | Harry | 🔄 | **Verified dedorch 2026-06-25:** `analyses` (plural) ALREADY has `user_id` + `status` + `data_bind` + `data_bind_version` + `report_collection` → those parts done. **Remaining:** drop `problem_statement`/`problem_validated` + add `objective`/`business_questions`. Singular `analysis` = deprecated duplicate to drop |
116
+ | 4 | Update Python `analyses` model + `state_store` + `analysis.py` to match dedorch; `owner_id`→`user_id` | Rifqi/Sofhia | | Done 2026-06-26. `owner_id`→`user_id` + added `status`/`data_bind`/`data_bind_version`/`report_collection` (DB-only, not in the `AnalysisState` pydantic) across `models.py`/`gate.py`/`state_store.py`/`analysis.py` + 3 local tests; also `report_inputs` `id`/`analysis_id` → `uuid`. Kept `problem_statement`/`problem_validated`; `objective`/`business_questions` wait on Harry's #3. Suite **284 passed** |
117
  | 5 | Report generator → `objective`+`business_questions`, "generated by {user}", formal outline | Sofhia | ✅ | Goal-shape (KM-674) + author name (#19) + outline (KM-680): Objective → Business Questions → Executive Summary → Key Findings → EDA → Notes & Limitations → How This Was Analyzed |
118
  | 6 | Report skill input contract: `analysis_id` + `user_id` (no `chat_history`) | Sofhia/Rifqi | ✅ | No-op: `POST /report` already takes only analysis_id + user_id (records-based). Documented in API_ENDPOINTS.md §5. *(Edit-mode input revisited in #24.)* |
119
  | 7 | `report_id` state update via request to Go, not direct DB | Sofhia + Harry | ⬜ | Needs Go endpoint. **Checkpoint:** Go owns ALL `reports` writes; Python stops any direct insert/update and only returns content; report stays a **skill** (no intent). See #18 |
120
  | 8 | Expose/confirm 4 FE endpoints; comment `check_data` + analysis CRUD | Sofhia | ✅ | KM-678: `list_tools` trimmed to `/help` + `/report` (analytics/check/retrieve commented in the **menu**). `help` confirmed as a `call_agent` intent — no own endpoint. Analysis CRUD endpoint left **registered**: "comment the rest" was about the FE slash menu, not killing HTTP routes Go needs |
121
  | 9 | Verify `analysis_id` in `call_agent` contract | Sofhia | ✅ | Verified: no separate field — carried as `room_id` (`analysis_id == room_id`), per REPO_STATUS §4/§11. Action for Go: send the id as `room_id` |
122
  | 10 | API endpoint doc (MD), 4 endpoints, for Go integration | Rifqi + Sofhia | ✅ | Done 2026-06-25 — `API_ENDPOINTS.md` (repo root). 4 FE surfaces with request/response **examples** (chat SSE transcript, report 201/409 JSON, version list), schemas, §9 full 32-route inventory + task-8 reading |
123
+ | 11 | Full Python project doc (MD → PDF/Word BRD) | Rifqi | | Done 2026-06-26 — `PROJECT_BRD.md` (repo root): purpose/context, FR-1..9 capabilities, lifecycle, architecture, data model, API (→ API_ENDPOINTS), NFRs, integrations, open items. Reuses REPO_STATUS/API_ENDPOINTS; convert to PDF/Word for distribution |
124
  | 12 | Reconcile/open the `list_tools` PR cleanly (stacked commits) | Rifqi | ✅ | N/A — we develop directly on the single active branch `pr/4` (KM-652 + KM-678 already stacked there); no separate PR to reconcile |
125
  | 13 | Deploy HF Python build (remove-`problem_statement` work) → test 4 endpoints via Swagger / playground | Sofhia + Harry | 🔄 | **Unblocked (#15 ✅).** Remove-PS work is on `pr/4` but **not on HF `main` yet** → PR + deploy, then manual test. Harry stabilizes Go ~Fri; FE testing ~Mon |
126
  | 14 | `analysis_records` home | Rifqi + Sofhia + lead | ✅ | **Resolved 2026-06-25:** stays Python-owned, **renamed** (→ #21); schema handed to Harry so the dedorch migration creates it post-cutover (→ #22). Not moved to Go |
 
141
 
142
  ## 5. Critical path & sequencing
143
 
144
+ - **Critical path:** #22 (send Harry the `report_inputs` schema). HF deploy (#13) for the playground. (#4 ✅, #21 ✅; Harry's #3 no longer blocks us — Python is getattr-tolerant.)
145
+ - **Parallelizable now:** #22 (handoff). (#4 ✅, #11 done.)
146
  - **Harry-blocked / coordinated:** #3 (now 🔄, blocks #4), #7 (Go endpoint), #18 (Go state ownership), #24 (contract). **#25 = chat-path migration to `analyses_messages` — a cutover blocker.**
147
  - **Demo gate (playground, #13):** deploy the remove-`problem_statement` work to HF — slow path (#15 ✅)
148
  and the report path are verified locally, and #16 is resolved (#22 hands Harry the schema). **Keep it
PROJECT_BRD.md ADDED
@@ -0,0 +1,150 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Data Eyond — Python Agentic Service: Business Requirements & Design (BRD)
2
+
3
+ **Status:** draft for review · **Date:** 2026-06-26 · **Branch:** `pr/4`
4
+ **Audience:** Harry (Go gateway) + leads/stakeholders.
5
+ **Scope:** the Python **agentic LLM service** (`Agentic-Service-Data-Eyond-Catalog`) only — its
6
+ requirements, capabilities, architecture, data, and integration contract.
7
+ **Companions (source of truth, not duplicated here):** [REPO_STATUS.md](REPO_STATUS.md) (current
8
+ built state) · [API_ENDPOINTS.md](API_ENDPOINTS.md) (FE-callable API) · [DEV_PLAN.md](DEV_PLAN.md)
9
+ (in-flight plan). This BRD synthesizes those into a stakeholder-facing document; convert to PDF/Word
10
+ for distribution.
11
+
12
+ ---
13
+
14
+ ## 1. Purpose & scope
15
+ Data Eyond is an **"AI data scientist"** for business analytics, modelled on **CRISP-DM** (Business
16
+ Understanding → Data Understanding → Preparation → Modeling → Evaluation → Deployment). A user sets a
17
+ goal, connects data (databases or files), asks natural-language analytical questions, and receives
18
+ CRISP-DM-structured answers that can be exported as a versioned **report**. The aim is a *"junior data
19
+ scientist that hands back a decision-ready deliverable,"* not a *"chatbot over a database."*
20
+
21
+ This document covers the **Python service** — the agentic reasoning layer. It does **not** specify the
22
+ Go gateway or the React frontend except at their integration boundaries (§9, §11).
23
+
24
+ ## 2. Business context & objectives
25
+ - **Target users:** executives doing self-serve deep-dives; analysts offloading routine work.
26
+ - **Value:** turn a business question + connected data into auditable, CRISP-DM-structured findings
27
+ and a formal report, without the user writing SQL or code.
28
+ - **Objectives:** (a) accurate, grounded analysis over the user's own data; (b) a decision-ready,
29
+ versioned report artifact; (c) safe, read-only access to user data; (d) a clean service contract the
30
+ Go gateway can integrate against.
31
+
32
+ ## 3. Stakeholders & actors
33
+ | Actor | Role |
34
+ |---|---|
35
+ | End user (exec/analyst) | Defines the analysis goal, asks questions, generates reports (via the FE) |
36
+ | Frontend (React/Vite) | Talks to Go for everything; to Python only for chat streaming |
37
+ | Go gateway (`Orchestrator-Agent-Service`) | Auth/JWT, rooms, documents, DB-credential storage, catalog ingestion, **all DB migrations**, and now all analysis-state writes |
38
+ | Python agentic service (this repo) | Router, skills, slow analytical path, structured query engine, RAG, report generation |
39
+ | Harry | Owns the Go gateway + dedorch DB migrations |
40
+
41
+ ## 4. Solution overview
42
+ Request flow is **FE → Go → Python**; the FE calls Python directly only for chat streaming. The Python
43
+ service is a **FastAPI** app that classifies each user message and dispatches it to the right
44
+ capability, streaming results back over SSE. Heavy analysis runs through a deterministic **slow path**
45
+ (plan → execute → assemble) whose structured output is persisted and later rendered into reports.
46
+
47
+ ## 5. Functional requirements (capabilities)
48
+ | ID | Capability | Description |
49
+ |---|---|---|
50
+ | FR-1 | **Intent routing** | One GPT-4o call classifies each message into one of 5 intents — `chat`, `help`, `check`, `unstructured_flow`, `structured_flow` — with history-aware query rewriting (EN/ID). |
51
+ | FR-2 | **Help skill** | State-aware, next-step guidance (LLM, streamed); only offers actions the current state allows (e.g., a report only when one is generatable). |
52
+ | FR-3 | **Check skill** | No-LLM inventory of available structured data + uploaded documents. |
53
+ | FR-4 | **Structured analysis (slow path)** | Planner → TaskRunner → Assembler: a static DAG of tool-call chains, degrade-and-continue execution, narrative authored by one LLM call; produces a structured run record. |
54
+ | FR-5 | **Structured query engine** | Catalog-driven JSON IR → deterministic SQL/pandas compiler → read-only executor, with single-level FK joins (DB sources). |
55
+ | FR-6 | **Unstructured RAG** | Retrieval over PGVector document chunks, answered by the chatbot. |
56
+ | FR-7 | **Analytics tools** | Composite `analyze_*` (descriptive, aggregate, correlation, trend) over data-access tools (`check_*`, `retrieve_*`). |
57
+ | FR-8 | **Report generation** | Deterministic assembly of findings/EDA/limitations/method from persisted run records + one LLM call for the executive summary; **versioned**, formal markdown. |
58
+ | FR-9 | **Analysis sessions** | One session = one analysis = one chat room (`analysis_id == room_id`); per-analysis data-source binding. |
59
+
60
+ **Goal capture (post-2026-06-24 pivot):** the analysis goal is **two user-entered fields** —
61
+ `objective` + `business_questions` — captured at onboarding, **both mandatory, no agent validation**.
62
+ The former agent-validated "problem statement" + its gate are removed.
63
+
64
+ ## 6. Analysis & report lifecycle
65
+ 1. **Create analysis** (via Go) — session row + chat room + chosen data-source bindings; goal =
66
+ `objective` + `business_questions`.
67
+ 2. **Ask questions** — `POST /chat/stream`; the router dispatches; `structured_flow` questions run the
68
+ slow path and **persist one `report_inputs` row per run** (the report's source of truth).
69
+ 3. **Generate report** — the report skill reads the session's `report_inputs`, assembles the structured
70
+ sections + an executive summary, and persists an immutable **versioned** report (markdown).
71
+ 4. **Read reports** — list versions / fetch a version.
72
+
73
+ > Reports are **records-based** (never from chat history) and require the slow path to have run
74
+ > (`enable_slow_path=true`) so records exist.
75
+
76
+ ## 7. System architecture (subsystems)
77
+ FastAPI + async SQLAlchemy + LangChain (Azure GPT-4o) + Redis + Azure Blob + PGVector. Key subsystems
78
+ (detail in REPO_STATUS §9):
79
+ - **Router** (`agents/orchestration.py`) — 5-intent classifier.
80
+ - **Skills** (`agents/handlers/`) — `help` (LLM), `check` (no-LLM).
81
+ - **Slow path** (`agents/slow_path/` + `agents/planner/`) — Planner, TaskRunner, Assembler.
82
+ - **Structured query engine** (`query/`) — IR validate → compile → read-only execute (never raises).
83
+ - **Report** (`agents/report/`) — generator, store (advisory-locked versioning), readiness floor.
84
+ - **Observability** — Langfuse tracing (PII-masked); Redis caching; pooled DB engines.
85
+
86
+ ## 8. Data model
87
+ SQLAlchemy models in `src/db/postgres/models.py` (detail in REPO_STATUS §8). The service is moving to
88
+ the shared **dedorch** DB (Go owns migrations; Python is consumer-only — §11).
89
+
90
+ | Table | Purpose | Owner |
91
+ |---|---|---|
92
+ | `users` | accounts (incl. `fullname` for report authorship) | Go |
93
+ | `analyses` *(plural)* | per-analysis session state: `objective`/`business_questions` (pivot), `user_id`, `status`, `data_bind`(+version), `report_collection`, `report_id` | Go (dedorch) |
94
+ | `analyses_messages` | the analysis chat room (user Q + agent A) — replaces deprecated `chat_messages`/`rooms` | Go (dedorch) |
95
+ | `report_inputs` | one jsonb row per slow-path run — the report's source of truth (was `analysis_records`) | **Python** (schema handed to Go) |
96
+ | `reports` | versioned report artifacts (markdown) | Go (dedorch) |
97
+ | `data_sources` | per-analysis source bindings | Go (dedorch) |
98
+ | `documents`, `databases`, `data_catalog` | uploads, DB credentials (Fernet), per-user catalog | Go ingestion |
99
+ | `langchain_pg_embedding` | PGVector document chunks | Go ingestion |
100
+
101
+ ## 9. API surface (FE-callable)
102
+ Full contract + request/response examples in [API_ENDPOINTS.md](API_ENDPOINTS.md). The FE-callable
103
+ surface is **4 things**:
104
+ 1. **`call_agent`** — `POST /api/v1/chat/stream` (SSE).
105
+ 2. **`list_skills`** — `GET /api/v1/tools` (slash-command catalog; cacheable).
106
+ 3. **skill: `help`** — via `call_agent` (router intent; no dedicated endpoint).
107
+ 4. **skill: `report`** — `POST /api/v1/report` + `GET` list/version.
108
+
109
+ `analysis_id == room_id`. Auth is terminated at Go; Python trusts `user_id`/`room_id`.
110
+
111
+ ## 10. Non-functional requirements
112
+ | Area | Requirement / mechanism |
113
+ |---|---|
114
+ | **Security — data access** | All structured queries are read-only: IR validation + SQL compiler whitelist + sqlglot SELECT-only guard + read-only session + LIMIT/timeout. DB credentials are Fernet-encrypted with an owner check. |
115
+ | **Security — PII** | PII columns carry no sample values into prompts; Langfuse masks PII on assembler/chatbot spans. |
116
+ | **Reliability** | Never-throw seams across tools/query/executors/state/report — failures degrade to soft output rather than crashing a turn. |
117
+ | **Performance** | Redis response cache (stateless `chat` only) + retrieval cache; pooled DB engines + speculative prewarm; warm Azure clients per process. |
118
+ | **Observability** | Langfuse: one trace per request (router/planner/assembler/chatbot + tool spans), tokens + latency. |
119
+ | **Portability** | Runs on HuggingFace Spaces (Linux) and Windows (`run.py` sets the selector event-loop policy for psycopg3 async). |
120
+
121
+ ## 11. Integrations & dependencies
122
+ - **Two-repo boundary:** Python is edited independently; Go + FE are reference-only. Python reads/writes
123
+ shared Postgres, reads Azure Blob (Parquet for tabular sources), uses Redis.
124
+ - **dedorch migration:** Python is moving from the `dataeyond` DB to **dedorch**. **Go owns all
125
+ migrations; Python is consumer-only** — if Python needs a table, it hands Go the schema. Table names
126
+ are **plural** (`analyses`, `analyses_messages`); `rooms`/`chat_messages` are deprecated there.
127
+ - **State writes via Go:** all analysis-state writes move behind Go; Python's per-turn state access
128
+ becomes a read-only get (in progress).
129
+ - **External services:** Azure OpenAI (GPT-4o + embeddings), Azure Blob, Postgres (+ PGVector), Redis,
130
+ Langfuse.
131
+
132
+ ## 12. Constraints & assumptions
133
+ - The slow path must be enabled (`enable_slow_path=true`) for reports to have content.
134
+ - `report_inputs` is Python-owned but its schema is provided to Go so the dedorch migration creates it
135
+ (so it survives the `SKIP_INIT_DB` cutover).
136
+ - Charts and images are **out of scope for now** — reports are markdown (tables/bold/italic/separators);
137
+ charts (Plotly JSON) and images (table + bucket) are deferred.
138
+ - The frontend has no dedicated UI designer; UI is being researched in parallel.
139
+
140
+ ## 13. Open items & roadmap
141
+ Tracked in [DEV_PLAN.md](DEV_PLAN.md) §4. Headlines: finish Go-side state ownership (#7/#18), the
142
+ dedorch `analyses` migration (#3, mostly done), HF deploy + playground test (#13), chat-path migration
143
+ to `analyses_messages` (#25), and the deferred charts/images/UI work (#26/#27/#28).
144
+
145
+ ## 14. Glossary
146
+ - **Slow path** — the deterministic Planner→TaskRunner→Assembler analytical pipeline.
147
+ - **`report_inputs`** — the jsonb table of slow-path run records the report reads (formerly `analysis_records`).
148
+ - **dedorch** — the shared Postgres DB the service is migrating to; Go owns its migrations.
149
+ - **CRISP-DM** — the cross-industry standard data-mining process the analysis is structured around.
150
+ - **`analysis_id == room_id`** — one analysis session is one chat room, identified by the same id.
src/agents/gate.py CHANGED
@@ -40,7 +40,7 @@ class AnalysisState(BaseModel):
40
  analysis_title: str
41
  problem_statement: str
42
  problem_validated: bool = False
43
- owner_id: str
44
  report_id: str | None = None
45
  created_at: datetime
46
  updated_at: datetime
@@ -78,7 +78,7 @@ def stub_analysis_state(*, problem_validated: bool = False) -> AnalysisState:
78
  analysis_title="Stub analysis",
79
  problem_statement="Stub problem statement" if problem_validated else "",
80
  problem_validated=problem_validated,
81
- owner_id="stub-user",
82
  report_id=None,
83
  created_at=now,
84
  updated_at=now,
 
40
  analysis_title: str
41
  problem_statement: str
42
  problem_validated: bool = False
43
+ user_id: str
44
  report_id: str | None = None
45
  created_at: datetime
46
  updated_at: datetime
 
78
  analysis_title="Stub analysis",
79
  problem_statement="Stub problem statement" if problem_validated else "",
80
  problem_validated=problem_validated,
81
+ user_id="stub-user",
82
  report_id=None,
83
  created_at=now,
84
  updated_at=now,
src/agents/state_store.py CHANGED
@@ -27,7 +27,7 @@ def _row_to_state(row: AnalysisStateRow) -> AnalysisState:
27
  analysis_title=row.analysis_title,
28
  problem_statement=row.problem_statement,
29
  problem_validated=row.problem_validated,
30
- owner_id=row.owner_id,
31
  report_id=row.report_id,
32
  created_at=row.created_at,
33
  updated_at=row.updated_at,
@@ -45,7 +45,7 @@ class AnalysisStateStore:
45
  async def ensure(
46
  self,
47
  analysis_id: str,
48
- owner_id: str,
49
  analysis_title: str = "New analysis",
50
  ) -> AnalysisState:
51
  """Get-or-create the state row for a session (idempotent, race-safe).
@@ -62,7 +62,7 @@ class AnalysisStateStore:
62
  insert(AnalysisStateRow)
63
  .values(
64
  id=analysis_id,
65
- owner_id=owner_id,
66
  analysis_title=analysis_title,
67
  problem_statement="",
68
  problem_validated=False,
@@ -78,7 +78,7 @@ class AnalysisStateStore:
78
  self,
79
  *,
80
  analysis_id: str,
81
- owner_id: str,
82
  analysis_title: str = "New analysis",
83
  problem_statement: str = "",
84
  ) -> AnalysisState:
@@ -86,7 +86,7 @@ class AnalysisStateStore:
86
  async with AsyncSessionLocal() as session:
87
  row = AnalysisStateRow(
88
  id=analysis_id,
89
- owner_id=owner_id,
90
  analysis_title=analysis_title,
91
  problem_statement=problem_statement,
92
  problem_validated=False,
 
27
  analysis_title=row.analysis_title,
28
  problem_statement=row.problem_statement,
29
  problem_validated=row.problem_validated,
30
+ user_id=row.user_id,
31
  report_id=row.report_id,
32
  created_at=row.created_at,
33
  updated_at=row.updated_at,
 
45
  async def ensure(
46
  self,
47
  analysis_id: str,
48
+ user_id: str,
49
  analysis_title: str = "New analysis",
50
  ) -> AnalysisState:
51
  """Get-or-create the state row for a session (idempotent, race-safe).
 
62
  insert(AnalysisStateRow)
63
  .values(
64
  id=analysis_id,
65
+ user_id=user_id,
66
  analysis_title=analysis_title,
67
  problem_statement="",
68
  problem_validated=False,
 
78
  self,
79
  *,
80
  analysis_id: str,
81
+ user_id: str,
82
  analysis_title: str = "New analysis",
83
  problem_statement: str = "",
84
  ) -> AnalysisState:
 
86
  async with AsyncSessionLocal() as session:
87
  row = AnalysisStateRow(
88
  id=analysis_id,
89
+ user_id=user_id,
90
  analysis_title=analysis_title,
91
  problem_statement=problem_statement,
92
  problem_validated=False,
src/api/v1/analysis.py CHANGED
@@ -30,7 +30,7 @@ def _serialize_state(row: AnalysisStateRow, data_source_ids: list[str]) -> dict:
30
  "analysis_title": row.analysis_title,
31
  "problem_statement": row.problem_statement,
32
  "problem_validated": row.problem_validated,
33
- "owner_id": row.owner_id,
34
  "report_id": row.report_id,
35
  "data_source_ids": data_source_ids,
36
  "created_at": row.created_at.isoformat() if row.created_at else None,
@@ -94,7 +94,7 @@ async def create_analysis(
94
  # id, created atomically in one transaction.
95
  state_row = AnalysisStateRow(
96
  id=analysis_id,
97
- owner_id=request.user_id,
98
  analysis_title=request.analysis_title,
99
  problem_statement=request.problem_statement,
100
  problem_validated=False,
@@ -144,7 +144,7 @@ async def list_analyses(user_id: str, db: AsyncSession = Depends(get_db)):
144
  """
145
  result = await db.execute(
146
  select(AnalysisStateRow)
147
- .where(AnalysisStateRow.owner_id == user_id)
148
  .order_by(AnalysisStateRow.updated_at.desc())
149
  )
150
  rows = result.scalars().all()
 
30
  "analysis_title": row.analysis_title,
31
  "problem_statement": row.problem_statement,
32
  "problem_validated": row.problem_validated,
33
+ "user_id": row.user_id,
34
  "report_id": row.report_id,
35
  "data_source_ids": data_source_ids,
36
  "created_at": row.created_at.isoformat() if row.created_at else None,
 
94
  # id, created atomically in one transaction.
95
  state_row = AnalysisStateRow(
96
  id=analysis_id,
97
+ user_id=request.user_id,
98
  analysis_title=request.analysis_title,
99
  problem_statement=request.problem_statement,
100
  problem_validated=False,
 
144
  """
145
  result = await db.execute(
146
  select(AnalysisStateRow)
147
+ .where(AnalysisStateRow.user_id == user_id)
148
  .order_by(AnalysisStateRow.updated_at.desc())
149
  )
150
  rows = result.scalars().all()
src/db/postgres/models.py CHANGED
@@ -151,10 +151,10 @@ class ReportInputRow(Base):
151
  """
152
  __tablename__ = "report_inputs"
153
 
154
- # Python keeps id/analysis_id as String for now (pure rename); dedorch's `report_inputs`
155
- # will be uuid + FK (see #22). Aligning these to UUID is a small cutover-readiness follow-up.
156
- id = Column(String, primary_key=True) # AnalysisRecord.record_id
157
- analysis_id = Column(String, index=True) # the analysis session id (nullable for now)
158
  user_id = Column(String, nullable=False, index=True)
159
  plan_id = Column(String, nullable=False)
160
  data = Column(JSONB, nullable=False)
@@ -191,11 +191,12 @@ class AnalysisStateRow(Base):
191
  `problem_validated`, `status` (text 'active'|'inactive' — soft-delete),
192
  `data_bind` (jsonb), `data_bind_version` (int), `report_collection` (jsonb).
193
 
194
- ⚠️ This model is NOT yet column-reconciled to that shape (task #4): it still declares
195
- `owner_id` (dedorch uses `user_id`) and omits `status`/`data_bind`/`data_bind_version`/
196
- `report_collection`. dedorch still carries `problem_statement`/`problem_validated` and
197
- does NOT yet have `objective`/`business_questions` Harry's #3 drops the former + adds
198
- the latter. Readers are getattr-tolerant so the eventual swap stays non-breaking.
 
199
 
200
  `analysis` (singular) is the deprecated DUPLICATE table Harry will drop — never use it.
201
  Class name kept as `AnalysisStateRow`.
@@ -206,8 +207,13 @@ class AnalysisStateRow(Base):
206
  analysis_title = Column(String, nullable=False, default="New analysis")
207
  problem_statement = Column(Text, nullable=False, default="")
208
  problem_validated = Column(Boolean, nullable=False, default=False)
209
- owner_id = Column(String, nullable=False, index=True)
210
  report_id = Column(UUID(as_uuid=False), nullable=True)
 
 
 
 
 
211
  created_at = Column(DateTime(timezone=True), server_default=func.now())
212
  updated_at = Column(
213
  DateTime(timezone=True), server_default=func.now(), onupdate=func.now()
 
151
  """
152
  __tablename__ = "report_inputs"
153
 
154
+ # id/analysis_id are `uuid` to match dedorch's `report_inputs` + the analysis-family
155
+ # (analyses/reports/data_sources). No FK declared in Python (dedorch's migration owns it, #22).
156
+ id = Column(UUID(as_uuid=False), primary_key=True) # AnalysisRecord.record_id (uuid hex ok)
157
+ analysis_id = Column(UUID(as_uuid=False), index=True) # the analysis session id (nullable for now)
158
  user_id = Column(String, nullable=False, index=True)
159
  plan_id = Column(String, nullable=False)
160
  data = Column(JSONB, nullable=False)
 
191
  `problem_validated`, `status` (text 'active'|'inactive' — soft-delete),
192
  `data_bind` (jsonb), `data_bind_version` (int), `report_collection` (jsonb).
193
 
194
+ Reconciled to that shape (#4, 2026-06-26): `user_id` (was `owner_id`) + `status`/`data_bind`/
195
+ `data_bind_version`/`report_collection` added. dedorch still carries `problem_statement`/
196
+ `problem_validated` and does NOT yet have `objective`/`business_questions` — Harry's #3 drops
197
+ the former + adds the latter; the report layer reads the goal getattr-tolerantly so that swap
198
+ stays non-breaking. The new FE/Go columns are stored to match dedorch but NOT surfaced in the
199
+ `AnalysisState` pydantic contract (no Python reader needs them yet).
200
 
201
  `analysis` (singular) is the deprecated DUPLICATE table Harry will drop — never use it.
202
  Class name kept as `AnalysisStateRow`.
 
207
  analysis_title = Column(String, nullable=False, default="New analysis")
208
  problem_statement = Column(Text, nullable=False, default="")
209
  problem_validated = Column(Boolean, nullable=False, default=False)
210
+ user_id = Column(String, nullable=False, index=True) # was owner_id (dedorch uses user_id)
211
  report_id = Column(UUID(as_uuid=False), nullable=True)
212
+ # dedorch `analyses` columns (FE/Go concerns; carried so create_all matches dedorch).
213
+ status = Column(String, nullable=False, default="active") # active | inactive (soft-delete)
214
+ data_bind = Column(JSONB, nullable=False, default=list)
215
+ data_bind_version = Column(Integer, nullable=False, default=1)
216
+ report_collection = Column(JSONB, nullable=False, default=list)
217
  created_at = Column(DateTime(timezone=True), server_default=func.now())
218
  updated_at = Column(
219
  DateTime(timezone=True), server_default=func.now(), onupdate=func.now()