Rifqi Hafizuddin Claude Opus 4.8 commited on
Commit ·
b126e18
1
Parent(s): 1759851
[NOTICKET] feat(db): reconcile analyses model to dedorch (#4) + project BRD (#11)
Browse files- #4: AnalysisStateRow owner_id->user_id + add status/data_bind/data_bind_version/
report_collection (DB-only, not in the AnalysisState pydantic); readers updated
(gate.py, state_store.py, api/v1/analysis.py) + 3 local tests; report_inputs
id/analysis_id -> uuid. Kept problem_statement/problem_validated; objective/
business_questions await Harry's #3. Suite: 284 passed, 7 skipped.
- #11: PROJECT_BRD.md - project requirements + design (reuses REPO_STATUS / API_ENDPOINTS).
- DEV_PLAN: #4, #11 marked done.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- DEV_PLAN.md +4 -4
- PROJECT_BRD.md +150 -0
- src/agents/gate.py +2 -2
- src/agents/state_store.py +5 -5
- src/api/v1/analysis.py +3 -3
- src/db/postgres/models.py +16 -10
DEV_PLAN.md
CHANGED
|
@@ -113,14 +113,14 @@ Status legend: ⬜ not started · 🔄 in progress · ✅ done · ⛔ blocked ·
|
|
| 113 |
| 1 | Comment out `problem_statement` skill **+ `Intent` literal + router prompt + gate redirect target**; remove `/problem-statement` from `list_tools` | Rifqi | ✅ | Done 2026-06-25 (one commit w/ #2). Unwired in `orchestration.py`, `intent_router.md`, `chat_handler.py`, `tools.py`; `problem_statement.py` kept intact |
|
| 114 |
| 2 | Drop `problem_validated`: gate neutered; `is_report_ready`/`report_floor` → **≥1 completed analysis** only, no-LLM | Rifqi | ✅ | Done 2026-06-25. `gate.py` no-op, gate call site commented in `chat_handler.py`, `report_floor` drops the goal check. Tests updated (`test_gate`/`test_chat_handler`/`test_readiness`). Suite: **284 passed, 7 skipped**; ruff clean |
|
| 115 |
| 3 | dedorch `analyses` migration: drop `problem_statement`/`problem_validated`, add `objective` + `business_questions` | Harry | 🔄 | **Verified dedorch 2026-06-25:** `analyses` (plural) ALREADY has `user_id` + `status` + `data_bind` + `data_bind_version` + `report_collection` → those parts done. **Remaining:** drop `problem_statement`/`problem_validated` + add `objective`/`business_questions`. Singular `analysis` = deprecated duplicate to drop |
|
| 116 |
-
| 4 | Update Python `analyses` model + `state_store` + `analysis.py` to match dedorch; `owner_id`→`user_id` | Rifqi/Sofhia |
|
| 117 |
| 5 | Report generator → `objective`+`business_questions`, "generated by {user}", formal outline | Sofhia | ✅ | Goal-shape (KM-674) + author name (#19) + outline (KM-680): Objective → Business Questions → Executive Summary → Key Findings → EDA → Notes & Limitations → How This Was Analyzed |
|
| 118 |
| 6 | Report skill input contract: `analysis_id` + `user_id` (no `chat_history`) | Sofhia/Rifqi | ✅ | No-op: `POST /report` already takes only analysis_id + user_id (records-based). Documented in API_ENDPOINTS.md §5. *(Edit-mode input revisited in #24.)* |
|
| 119 |
| 7 | `report_id` state update via request to Go, not direct DB | Sofhia + Harry | ⬜ | Needs Go endpoint. **Checkpoint:** Go owns ALL `reports` writes; Python stops any direct insert/update and only returns content; report stays a **skill** (no intent). See #18 |
|
| 120 |
| 8 | Expose/confirm 4 FE endpoints; comment `check_data` + analysis CRUD | Sofhia | ✅ | KM-678: `list_tools` trimmed to `/help` + `/report` (analytics/check/retrieve commented in the **menu**). `help` confirmed as a `call_agent` intent — no own endpoint. Analysis CRUD endpoint left **registered**: "comment the rest" was about the FE slash menu, not killing HTTP routes Go needs |
|
| 121 |
| 9 | Verify `analysis_id` in `call_agent` contract | Sofhia | ✅ | Verified: no separate field — carried as `room_id` (`analysis_id == room_id`), per REPO_STATUS §4/§11. Action for Go: send the id as `room_id` |
|
| 122 |
| 10 | API endpoint doc (MD), 4 endpoints, for Go integration | Rifqi + Sofhia | ✅ | Done 2026-06-25 — `API_ENDPOINTS.md` (repo root). 4 FE surfaces with request/response **examples** (chat SSE transcript, report 201/409 JSON, version list), schemas, §9 full 32-route inventory + task-8 reading |
|
| 123 |
-
| 11 | Full Python project doc (MD → PDF/Word BRD) | Rifqi |
|
| 124 |
| 12 | Reconcile/open the `list_tools` PR cleanly (stacked commits) | Rifqi | ✅ | N/A — we develop directly on the single active branch `pr/4` (KM-652 + KM-678 already stacked there); no separate PR to reconcile |
|
| 125 |
| 13 | Deploy HF Python build (remove-`problem_statement` work) → test 4 endpoints via Swagger / playground | Sofhia + Harry | 🔄 | **Unblocked (#15 ✅).** Remove-PS work is on `pr/4` but **not on HF `main` yet** → PR + deploy, then manual test. Harry stabilizes Go ~Fri; FE testing ~Mon |
|
| 126 |
| 14 | `analysis_records` home | Rifqi + Sofhia + lead | ✅ | **Resolved 2026-06-25:** stays Python-owned, **renamed** (→ #21); schema handed to Harry so the dedorch migration creates it post-cutover (→ #22). Not moved to Go |
|
|
@@ -141,8 +141,8 @@ Status legend: ⬜ not started · 🔄 in progress · ✅ done · ⛔ blocked ·
|
|
| 141 |
|
| 142 |
## 5. Critical path & sequencing
|
| 143 |
|
| 144 |
-
- **Critical path:** #
|
| 145 |
-
- **Parallelizable now:** #
|
| 146 |
- **Harry-blocked / coordinated:** #3 (now 🔄, blocks #4), #7 (Go endpoint), #18 (Go state ownership), #24 (contract). **#25 = chat-path migration to `analyses_messages` — a cutover blocker.**
|
| 147 |
- **Demo gate (playground, #13):** deploy the remove-`problem_statement` work to HF — slow path (#15 ✅)
|
| 148 |
and the report path are verified locally, and #16 is resolved (#22 hands Harry the schema). **Keep it
|
|
|
|
| 113 |
| 1 | Comment out `problem_statement` skill **+ `Intent` literal + router prompt + gate redirect target**; remove `/problem-statement` from `list_tools` | Rifqi | ✅ | Done 2026-06-25 (one commit w/ #2). Unwired in `orchestration.py`, `intent_router.md`, `chat_handler.py`, `tools.py`; `problem_statement.py` kept intact |
|
| 114 |
| 2 | Drop `problem_validated`: gate neutered; `is_report_ready`/`report_floor` → **≥1 completed analysis** only, no-LLM | Rifqi | ✅ | Done 2026-06-25. `gate.py` no-op, gate call site commented in `chat_handler.py`, `report_floor` drops the goal check. Tests updated (`test_gate`/`test_chat_handler`/`test_readiness`). Suite: **284 passed, 7 skipped**; ruff clean |
|
| 115 |
| 3 | dedorch `analyses` migration: drop `problem_statement`/`problem_validated`, add `objective` + `business_questions` | Harry | 🔄 | **Verified dedorch 2026-06-25:** `analyses` (plural) ALREADY has `user_id` + `status` + `data_bind` + `data_bind_version` + `report_collection` → those parts done. **Remaining:** drop `problem_statement`/`problem_validated` + add `objective`/`business_questions`. Singular `analysis` = deprecated duplicate to drop |
|
| 116 |
+
| 4 | Update Python `analyses` model + `state_store` + `analysis.py` to match dedorch; `owner_id`→`user_id` | Rifqi/Sofhia | ✅ | Done 2026-06-26. `owner_id`→`user_id` + added `status`/`data_bind`/`data_bind_version`/`report_collection` (DB-only, not in the `AnalysisState` pydantic) across `models.py`/`gate.py`/`state_store.py`/`analysis.py` + 3 local tests; also `report_inputs` `id`/`analysis_id` → `uuid`. Kept `problem_statement`/`problem_validated`; `objective`/`business_questions` wait on Harry's #3. Suite **284 passed** |
|
| 117 |
| 5 | Report generator → `objective`+`business_questions`, "generated by {user}", formal outline | Sofhia | ✅ | Goal-shape (KM-674) + author name (#19) + outline (KM-680): Objective → Business Questions → Executive Summary → Key Findings → EDA → Notes & Limitations → How This Was Analyzed |
|
| 118 |
| 6 | Report skill input contract: `analysis_id` + `user_id` (no `chat_history`) | Sofhia/Rifqi | ✅ | No-op: `POST /report` already takes only analysis_id + user_id (records-based). Documented in API_ENDPOINTS.md §5. *(Edit-mode input revisited in #24.)* |
|
| 119 |
| 7 | `report_id` state update via request to Go, not direct DB | Sofhia + Harry | ⬜ | Needs Go endpoint. **Checkpoint:** Go owns ALL `reports` writes; Python stops any direct insert/update and only returns content; report stays a **skill** (no intent). See #18 |
|
| 120 |
| 8 | Expose/confirm 4 FE endpoints; comment `check_data` + analysis CRUD | Sofhia | ✅ | KM-678: `list_tools` trimmed to `/help` + `/report` (analytics/check/retrieve commented in the **menu**). `help` confirmed as a `call_agent` intent — no own endpoint. Analysis CRUD endpoint left **registered**: "comment the rest" was about the FE slash menu, not killing HTTP routes Go needs |
|
| 121 |
| 9 | Verify `analysis_id` in `call_agent` contract | Sofhia | ✅ | Verified: no separate field — carried as `room_id` (`analysis_id == room_id`), per REPO_STATUS §4/§11. Action for Go: send the id as `room_id` |
|
| 122 |
| 10 | API endpoint doc (MD), 4 endpoints, for Go integration | Rifqi + Sofhia | ✅ | Done 2026-06-25 — `API_ENDPOINTS.md` (repo root). 4 FE surfaces with request/response **examples** (chat SSE transcript, report 201/409 JSON, version list), schemas, §9 full 32-route inventory + task-8 reading |
|
| 123 |
+
| 11 | Full Python project doc (MD → PDF/Word BRD) | Rifqi | ✅ | Done 2026-06-26 — `PROJECT_BRD.md` (repo root): purpose/context, FR-1..9 capabilities, lifecycle, architecture, data model, API (→ API_ENDPOINTS), NFRs, integrations, open items. Reuses REPO_STATUS/API_ENDPOINTS; convert to PDF/Word for distribution |
|
| 124 |
| 12 | Reconcile/open the `list_tools` PR cleanly (stacked commits) | Rifqi | ✅ | N/A — we develop directly on the single active branch `pr/4` (KM-652 + KM-678 already stacked there); no separate PR to reconcile |
|
| 125 |
| 13 | Deploy HF Python build (remove-`problem_statement` work) → test 4 endpoints via Swagger / playground | Sofhia + Harry | 🔄 | **Unblocked (#15 ✅).** Remove-PS work is on `pr/4` but **not on HF `main` yet** → PR + deploy, then manual test. Harry stabilizes Go ~Fri; FE testing ~Mon |
|
| 126 |
| 14 | `analysis_records` home | Rifqi + Sofhia + lead | ✅ | **Resolved 2026-06-25:** stays Python-owned, **renamed** (→ #21); schema handed to Harry so the dedorch migration creates it post-cutover (→ #22). Not moved to Go |
|
|
|
|
| 141 |
|
| 142 |
## 5. Critical path & sequencing
|
| 143 |
|
| 144 |
+
- **Critical path:** #22 (send Harry the `report_inputs` schema). HF deploy (#13) for the playground. (#4 ✅, #21 ✅; Harry's #3 no longer blocks us — Python is getattr-tolerant.)
|
| 145 |
+
- **Parallelizable now:** #22 (handoff). (#4 ✅, #11 ✅ done.)
|
| 146 |
- **Harry-blocked / coordinated:** #3 (now 🔄, blocks #4), #7 (Go endpoint), #18 (Go state ownership), #24 (contract). **#25 = chat-path migration to `analyses_messages` — a cutover blocker.**
|
| 147 |
- **Demo gate (playground, #13):** deploy the remove-`problem_statement` work to HF — slow path (#15 ✅)
|
| 148 |
and the report path are verified locally, and #16 is resolved (#22 hands Harry the schema). **Keep it
|
PROJECT_BRD.md
ADDED
|
@@ -0,0 +1,150 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Data Eyond — Python Agentic Service: Business Requirements & Design (BRD)
|
| 2 |
+
|
| 3 |
+
**Status:** draft for review · **Date:** 2026-06-26 · **Branch:** `pr/4`
|
| 4 |
+
**Audience:** Harry (Go gateway) + leads/stakeholders.
|
| 5 |
+
**Scope:** the Python **agentic LLM service** (`Agentic-Service-Data-Eyond-Catalog`) only — its
|
| 6 |
+
requirements, capabilities, architecture, data, and integration contract.
|
| 7 |
+
**Companions (source of truth, not duplicated here):** [REPO_STATUS.md](REPO_STATUS.md) (current
|
| 8 |
+
built state) · [API_ENDPOINTS.md](API_ENDPOINTS.md) (FE-callable API) · [DEV_PLAN.md](DEV_PLAN.md)
|
| 9 |
+
(in-flight plan). This BRD synthesizes those into a stakeholder-facing document; convert to PDF/Word
|
| 10 |
+
for distribution.
|
| 11 |
+
|
| 12 |
+
---
|
| 13 |
+
|
| 14 |
+
## 1. Purpose & scope
|
| 15 |
+
Data Eyond is an **"AI data scientist"** for business analytics, modelled on **CRISP-DM** (Business
|
| 16 |
+
Understanding → Data Understanding → Preparation → Modeling → Evaluation → Deployment). A user sets a
|
| 17 |
+
goal, connects data (databases or files), asks natural-language analytical questions, and receives
|
| 18 |
+
CRISP-DM-structured answers that can be exported as a versioned **report**. The aim is a *"junior data
|
| 19 |
+
scientist that hands back a decision-ready deliverable,"* not a *"chatbot over a database."*
|
| 20 |
+
|
| 21 |
+
This document covers the **Python service** — the agentic reasoning layer. It does **not** specify the
|
| 22 |
+
Go gateway or the React frontend except at their integration boundaries (§9, §11).
|
| 23 |
+
|
| 24 |
+
## 2. Business context & objectives
|
| 25 |
+
- **Target users:** executives doing self-serve deep-dives; analysts offloading routine work.
|
| 26 |
+
- **Value:** turn a business question + connected data into auditable, CRISP-DM-structured findings
|
| 27 |
+
and a formal report, without the user writing SQL or code.
|
| 28 |
+
- **Objectives:** (a) accurate, grounded analysis over the user's own data; (b) a decision-ready,
|
| 29 |
+
versioned report artifact; (c) safe, read-only access to user data; (d) a clean service contract the
|
| 30 |
+
Go gateway can integrate against.
|
| 31 |
+
|
| 32 |
+
## 3. Stakeholders & actors
|
| 33 |
+
| Actor | Role |
|
| 34 |
+
|---|---|
|
| 35 |
+
| End user (exec/analyst) | Defines the analysis goal, asks questions, generates reports (via the FE) |
|
| 36 |
+
| Frontend (React/Vite) | Talks to Go for everything; to Python only for chat streaming |
|
| 37 |
+
| Go gateway (`Orchestrator-Agent-Service`) | Auth/JWT, rooms, documents, DB-credential storage, catalog ingestion, **all DB migrations**, and now all analysis-state writes |
|
| 38 |
+
| Python agentic service (this repo) | Router, skills, slow analytical path, structured query engine, RAG, report generation |
|
| 39 |
+
| Harry | Owns the Go gateway + dedorch DB migrations |
|
| 40 |
+
|
| 41 |
+
## 4. Solution overview
|
| 42 |
+
Request flow is **FE → Go → Python**; the FE calls Python directly only for chat streaming. The Python
|
| 43 |
+
service is a **FastAPI** app that classifies each user message and dispatches it to the right
|
| 44 |
+
capability, streaming results back over SSE. Heavy analysis runs through a deterministic **slow path**
|
| 45 |
+
(plan → execute → assemble) whose structured output is persisted and later rendered into reports.
|
| 46 |
+
|
| 47 |
+
## 5. Functional requirements (capabilities)
|
| 48 |
+
| ID | Capability | Description |
|
| 49 |
+
|---|---|---|
|
| 50 |
+
| FR-1 | **Intent routing** | One GPT-4o call classifies each message into one of 5 intents — `chat`, `help`, `check`, `unstructured_flow`, `structured_flow` — with history-aware query rewriting (EN/ID). |
|
| 51 |
+
| FR-2 | **Help skill** | State-aware, next-step guidance (LLM, streamed); only offers actions the current state allows (e.g., a report only when one is generatable). |
|
| 52 |
+
| FR-3 | **Check skill** | No-LLM inventory of available structured data + uploaded documents. |
|
| 53 |
+
| FR-4 | **Structured analysis (slow path)** | Planner → TaskRunner → Assembler: a static DAG of tool-call chains, degrade-and-continue execution, narrative authored by one LLM call; produces a structured run record. |
|
| 54 |
+
| FR-5 | **Structured query engine** | Catalog-driven JSON IR → deterministic SQL/pandas compiler → read-only executor, with single-level FK joins (DB sources). |
|
| 55 |
+
| FR-6 | **Unstructured RAG** | Retrieval over PGVector document chunks, answered by the chatbot. |
|
| 56 |
+
| FR-7 | **Analytics tools** | Composite `analyze_*` (descriptive, aggregate, correlation, trend) over data-access tools (`check_*`, `retrieve_*`). |
|
| 57 |
+
| FR-8 | **Report generation** | Deterministic assembly of findings/EDA/limitations/method from persisted run records + one LLM call for the executive summary; **versioned**, formal markdown. |
|
| 58 |
+
| FR-9 | **Analysis sessions** | One session = one analysis = one chat room (`analysis_id == room_id`); per-analysis data-source binding. |
|
| 59 |
+
|
| 60 |
+
**Goal capture (post-2026-06-24 pivot):** the analysis goal is **two user-entered fields** —
|
| 61 |
+
`objective` + `business_questions` — captured at onboarding, **both mandatory, no agent validation**.
|
| 62 |
+
The former agent-validated "problem statement" + its gate are removed.
|
| 63 |
+
|
| 64 |
+
## 6. Analysis & report lifecycle
|
| 65 |
+
1. **Create analysis** (via Go) — session row + chat room + chosen data-source bindings; goal =
|
| 66 |
+
`objective` + `business_questions`.
|
| 67 |
+
2. **Ask questions** — `POST /chat/stream`; the router dispatches; `structured_flow` questions run the
|
| 68 |
+
slow path and **persist one `report_inputs` row per run** (the report's source of truth).
|
| 69 |
+
3. **Generate report** — the report skill reads the session's `report_inputs`, assembles the structured
|
| 70 |
+
sections + an executive summary, and persists an immutable **versioned** report (markdown).
|
| 71 |
+
4. **Read reports** — list versions / fetch a version.
|
| 72 |
+
|
| 73 |
+
> Reports are **records-based** (never from chat history) and require the slow path to have run
|
| 74 |
+
> (`enable_slow_path=true`) so records exist.
|
| 75 |
+
|
| 76 |
+
## 7. System architecture (subsystems)
|
| 77 |
+
FastAPI + async SQLAlchemy + LangChain (Azure GPT-4o) + Redis + Azure Blob + PGVector. Key subsystems
|
| 78 |
+
(detail in REPO_STATUS §9):
|
| 79 |
+
- **Router** (`agents/orchestration.py`) — 5-intent classifier.
|
| 80 |
+
- **Skills** (`agents/handlers/`) — `help` (LLM), `check` (no-LLM).
|
| 81 |
+
- **Slow path** (`agents/slow_path/` + `agents/planner/`) — Planner, TaskRunner, Assembler.
|
| 82 |
+
- **Structured query engine** (`query/`) — IR validate → compile → read-only execute (never raises).
|
| 83 |
+
- **Report** (`agents/report/`) — generator, store (advisory-locked versioning), readiness floor.
|
| 84 |
+
- **Observability** — Langfuse tracing (PII-masked); Redis caching; pooled DB engines.
|
| 85 |
+
|
| 86 |
+
## 8. Data model
|
| 87 |
+
SQLAlchemy models in `src/db/postgres/models.py` (detail in REPO_STATUS §8). The service is moving to
|
| 88 |
+
the shared **dedorch** DB (Go owns migrations; Python is consumer-only — §11).
|
| 89 |
+
|
| 90 |
+
| Table | Purpose | Owner |
|
| 91 |
+
|---|---|---|
|
| 92 |
+
| `users` | accounts (incl. `fullname` for report authorship) | Go |
|
| 93 |
+
| `analyses` *(plural)* | per-analysis session state: `objective`/`business_questions` (pivot), `user_id`, `status`, `data_bind`(+version), `report_collection`, `report_id` | Go (dedorch) |
|
| 94 |
+
| `analyses_messages` | the analysis chat room (user Q + agent A) — replaces deprecated `chat_messages`/`rooms` | Go (dedorch) |
|
| 95 |
+
| `report_inputs` | one jsonb row per slow-path run — the report's source of truth (was `analysis_records`) | **Python** (schema handed to Go) |
|
| 96 |
+
| `reports` | versioned report artifacts (markdown) | Go (dedorch) |
|
| 97 |
+
| `data_sources` | per-analysis source bindings | Go (dedorch) |
|
| 98 |
+
| `documents`, `databases`, `data_catalog` | uploads, DB credentials (Fernet), per-user catalog | Go ingestion |
|
| 99 |
+
| `langchain_pg_embedding` | PGVector document chunks | Go ingestion |
|
| 100 |
+
|
| 101 |
+
## 9. API surface (FE-callable)
|
| 102 |
+
Full contract + request/response examples in [API_ENDPOINTS.md](API_ENDPOINTS.md). The FE-callable
|
| 103 |
+
surface is **4 things**:
|
| 104 |
+
1. **`call_agent`** — `POST /api/v1/chat/stream` (SSE).
|
| 105 |
+
2. **`list_skills`** — `GET /api/v1/tools` (slash-command catalog; cacheable).
|
| 106 |
+
3. **skill: `help`** — via `call_agent` (router intent; no dedicated endpoint).
|
| 107 |
+
4. **skill: `report`** — `POST /api/v1/report` + `GET` list/version.
|
| 108 |
+
|
| 109 |
+
`analysis_id == room_id`. Auth is terminated at Go; Python trusts `user_id`/`room_id`.
|
| 110 |
+
|
| 111 |
+
## 10. Non-functional requirements
|
| 112 |
+
| Area | Requirement / mechanism |
|
| 113 |
+
|---|---|
|
| 114 |
+
| **Security — data access** | All structured queries are read-only: IR validation + SQL compiler whitelist + sqlglot SELECT-only guard + read-only session + LIMIT/timeout. DB credentials are Fernet-encrypted with an owner check. |
|
| 115 |
+
| **Security — PII** | PII columns carry no sample values into prompts; Langfuse masks PII on assembler/chatbot spans. |
|
| 116 |
+
| **Reliability** | Never-throw seams across tools/query/executors/state/report — failures degrade to soft output rather than crashing a turn. |
|
| 117 |
+
| **Performance** | Redis response cache (stateless `chat` only) + retrieval cache; pooled DB engines + speculative prewarm; warm Azure clients per process. |
|
| 118 |
+
| **Observability** | Langfuse: one trace per request (router/planner/assembler/chatbot + tool spans), tokens + latency. |
|
| 119 |
+
| **Portability** | Runs on HuggingFace Spaces (Linux) and Windows (`run.py` sets the selector event-loop policy for psycopg3 async). |
|
| 120 |
+
|
| 121 |
+
## 11. Integrations & dependencies
|
| 122 |
+
- **Two-repo boundary:** Python is edited independently; Go + FE are reference-only. Python reads/writes
|
| 123 |
+
shared Postgres, reads Azure Blob (Parquet for tabular sources), uses Redis.
|
| 124 |
+
- **dedorch migration:** Python is moving from the `dataeyond` DB to **dedorch**. **Go owns all
|
| 125 |
+
migrations; Python is consumer-only** — if Python needs a table, it hands Go the schema. Table names
|
| 126 |
+
are **plural** (`analyses`, `analyses_messages`); `rooms`/`chat_messages` are deprecated there.
|
| 127 |
+
- **State writes via Go:** all analysis-state writes move behind Go; Python's per-turn state access
|
| 128 |
+
becomes a read-only get (in progress).
|
| 129 |
+
- **External services:** Azure OpenAI (GPT-4o + embeddings), Azure Blob, Postgres (+ PGVector), Redis,
|
| 130 |
+
Langfuse.
|
| 131 |
+
|
| 132 |
+
## 12. Constraints & assumptions
|
| 133 |
+
- The slow path must be enabled (`enable_slow_path=true`) for reports to have content.
|
| 134 |
+
- `report_inputs` is Python-owned but its schema is provided to Go so the dedorch migration creates it
|
| 135 |
+
(so it survives the `SKIP_INIT_DB` cutover).
|
| 136 |
+
- Charts and images are **out of scope for now** — reports are markdown (tables/bold/italic/separators);
|
| 137 |
+
charts (Plotly JSON) and images (table + bucket) are deferred.
|
| 138 |
+
- The frontend has no dedicated UI designer; UI is being researched in parallel.
|
| 139 |
+
|
| 140 |
+
## 13. Open items & roadmap
|
| 141 |
+
Tracked in [DEV_PLAN.md](DEV_PLAN.md) §4. Headlines: finish Go-side state ownership (#7/#18), the
|
| 142 |
+
dedorch `analyses` migration (#3, mostly done), HF deploy + playground test (#13), chat-path migration
|
| 143 |
+
to `analyses_messages` (#25), and the deferred charts/images/UI work (#26/#27/#28).
|
| 144 |
+
|
| 145 |
+
## 14. Glossary
|
| 146 |
+
- **Slow path** — the deterministic Planner→TaskRunner→Assembler analytical pipeline.
|
| 147 |
+
- **`report_inputs`** — the jsonb table of slow-path run records the report reads (formerly `analysis_records`).
|
| 148 |
+
- **dedorch** — the shared Postgres DB the service is migrating to; Go owns its migrations.
|
| 149 |
+
- **CRISP-DM** — the cross-industry standard data-mining process the analysis is structured around.
|
| 150 |
+
- **`analysis_id == room_id`** — one analysis session is one chat room, identified by the same id.
|
src/agents/gate.py
CHANGED
|
@@ -40,7 +40,7 @@ class AnalysisState(BaseModel):
|
|
| 40 |
analysis_title: str
|
| 41 |
problem_statement: str
|
| 42 |
problem_validated: bool = False
|
| 43 |
-
|
| 44 |
report_id: str | None = None
|
| 45 |
created_at: datetime
|
| 46 |
updated_at: datetime
|
|
@@ -78,7 +78,7 @@ def stub_analysis_state(*, problem_validated: bool = False) -> AnalysisState:
|
|
| 78 |
analysis_title="Stub analysis",
|
| 79 |
problem_statement="Stub problem statement" if problem_validated else "",
|
| 80 |
problem_validated=problem_validated,
|
| 81 |
-
|
| 82 |
report_id=None,
|
| 83 |
created_at=now,
|
| 84 |
updated_at=now,
|
|
|
|
| 40 |
analysis_title: str
|
| 41 |
problem_statement: str
|
| 42 |
problem_validated: bool = False
|
| 43 |
+
user_id: str
|
| 44 |
report_id: str | None = None
|
| 45 |
created_at: datetime
|
| 46 |
updated_at: datetime
|
|
|
|
| 78 |
analysis_title="Stub analysis",
|
| 79 |
problem_statement="Stub problem statement" if problem_validated else "",
|
| 80 |
problem_validated=problem_validated,
|
| 81 |
+
user_id="stub-user",
|
| 82 |
report_id=None,
|
| 83 |
created_at=now,
|
| 84 |
updated_at=now,
|
src/agents/state_store.py
CHANGED
|
@@ -27,7 +27,7 @@ def _row_to_state(row: AnalysisStateRow) -> AnalysisState:
|
|
| 27 |
analysis_title=row.analysis_title,
|
| 28 |
problem_statement=row.problem_statement,
|
| 29 |
problem_validated=row.problem_validated,
|
| 30 |
-
|
| 31 |
report_id=row.report_id,
|
| 32 |
created_at=row.created_at,
|
| 33 |
updated_at=row.updated_at,
|
|
@@ -45,7 +45,7 @@ class AnalysisStateStore:
|
|
| 45 |
async def ensure(
|
| 46 |
self,
|
| 47 |
analysis_id: str,
|
| 48 |
-
|
| 49 |
analysis_title: str = "New analysis",
|
| 50 |
) -> AnalysisState:
|
| 51 |
"""Get-or-create the state row for a session (idempotent, race-safe).
|
|
@@ -62,7 +62,7 @@ class AnalysisStateStore:
|
|
| 62 |
insert(AnalysisStateRow)
|
| 63 |
.values(
|
| 64 |
id=analysis_id,
|
| 65 |
-
|
| 66 |
analysis_title=analysis_title,
|
| 67 |
problem_statement="",
|
| 68 |
problem_validated=False,
|
|
@@ -78,7 +78,7 @@ class AnalysisStateStore:
|
|
| 78 |
self,
|
| 79 |
*,
|
| 80 |
analysis_id: str,
|
| 81 |
-
|
| 82 |
analysis_title: str = "New analysis",
|
| 83 |
problem_statement: str = "",
|
| 84 |
) -> AnalysisState:
|
|
@@ -86,7 +86,7 @@ class AnalysisStateStore:
|
|
| 86 |
async with AsyncSessionLocal() as session:
|
| 87 |
row = AnalysisStateRow(
|
| 88 |
id=analysis_id,
|
| 89 |
-
|
| 90 |
analysis_title=analysis_title,
|
| 91 |
problem_statement=problem_statement,
|
| 92 |
problem_validated=False,
|
|
|
|
| 27 |
analysis_title=row.analysis_title,
|
| 28 |
problem_statement=row.problem_statement,
|
| 29 |
problem_validated=row.problem_validated,
|
| 30 |
+
user_id=row.user_id,
|
| 31 |
report_id=row.report_id,
|
| 32 |
created_at=row.created_at,
|
| 33 |
updated_at=row.updated_at,
|
|
|
|
| 45 |
async def ensure(
|
| 46 |
self,
|
| 47 |
analysis_id: str,
|
| 48 |
+
user_id: str,
|
| 49 |
analysis_title: str = "New analysis",
|
| 50 |
) -> AnalysisState:
|
| 51 |
"""Get-or-create the state row for a session (idempotent, race-safe).
|
|
|
|
| 62 |
insert(AnalysisStateRow)
|
| 63 |
.values(
|
| 64 |
id=analysis_id,
|
| 65 |
+
user_id=user_id,
|
| 66 |
analysis_title=analysis_title,
|
| 67 |
problem_statement="",
|
| 68 |
problem_validated=False,
|
|
|
|
| 78 |
self,
|
| 79 |
*,
|
| 80 |
analysis_id: str,
|
| 81 |
+
user_id: str,
|
| 82 |
analysis_title: str = "New analysis",
|
| 83 |
problem_statement: str = "",
|
| 84 |
) -> AnalysisState:
|
|
|
|
| 86 |
async with AsyncSessionLocal() as session:
|
| 87 |
row = AnalysisStateRow(
|
| 88 |
id=analysis_id,
|
| 89 |
+
user_id=user_id,
|
| 90 |
analysis_title=analysis_title,
|
| 91 |
problem_statement=problem_statement,
|
| 92 |
problem_validated=False,
|
src/api/v1/analysis.py
CHANGED
|
@@ -30,7 +30,7 @@ def _serialize_state(row: AnalysisStateRow, data_source_ids: list[str]) -> dict:
|
|
| 30 |
"analysis_title": row.analysis_title,
|
| 31 |
"problem_statement": row.problem_statement,
|
| 32 |
"problem_validated": row.problem_validated,
|
| 33 |
-
"
|
| 34 |
"report_id": row.report_id,
|
| 35 |
"data_source_ids": data_source_ids,
|
| 36 |
"created_at": row.created_at.isoformat() if row.created_at else None,
|
|
@@ -94,7 +94,7 @@ async def create_analysis(
|
|
| 94 |
# id, created atomically in one transaction.
|
| 95 |
state_row = AnalysisStateRow(
|
| 96 |
id=analysis_id,
|
| 97 |
-
|
| 98 |
analysis_title=request.analysis_title,
|
| 99 |
problem_statement=request.problem_statement,
|
| 100 |
problem_validated=False,
|
|
@@ -144,7 +144,7 @@ async def list_analyses(user_id: str, db: AsyncSession = Depends(get_db)):
|
|
| 144 |
"""
|
| 145 |
result = await db.execute(
|
| 146 |
select(AnalysisStateRow)
|
| 147 |
-
.where(AnalysisStateRow.
|
| 148 |
.order_by(AnalysisStateRow.updated_at.desc())
|
| 149 |
)
|
| 150 |
rows = result.scalars().all()
|
|
|
|
| 30 |
"analysis_title": row.analysis_title,
|
| 31 |
"problem_statement": row.problem_statement,
|
| 32 |
"problem_validated": row.problem_validated,
|
| 33 |
+
"user_id": row.user_id,
|
| 34 |
"report_id": row.report_id,
|
| 35 |
"data_source_ids": data_source_ids,
|
| 36 |
"created_at": row.created_at.isoformat() if row.created_at else None,
|
|
|
|
| 94 |
# id, created atomically in one transaction.
|
| 95 |
state_row = AnalysisStateRow(
|
| 96 |
id=analysis_id,
|
| 97 |
+
user_id=request.user_id,
|
| 98 |
analysis_title=request.analysis_title,
|
| 99 |
problem_statement=request.problem_statement,
|
| 100 |
problem_validated=False,
|
|
|
|
| 144 |
"""
|
| 145 |
result = await db.execute(
|
| 146 |
select(AnalysisStateRow)
|
| 147 |
+
.where(AnalysisStateRow.user_id == user_id)
|
| 148 |
.order_by(AnalysisStateRow.updated_at.desc())
|
| 149 |
)
|
| 150 |
rows = result.scalars().all()
|
src/db/postgres/models.py
CHANGED
|
@@ -151,10 +151,10 @@ class ReportInputRow(Base):
|
|
| 151 |
"""
|
| 152 |
__tablename__ = "report_inputs"
|
| 153 |
|
| 154 |
-
#
|
| 155 |
-
#
|
| 156 |
-
id = Column(
|
| 157 |
-
analysis_id = Column(
|
| 158 |
user_id = Column(String, nullable=False, index=True)
|
| 159 |
plan_id = Column(String, nullable=False)
|
| 160 |
data = Column(JSONB, nullable=False)
|
|
@@ -191,11 +191,12 @@ class AnalysisStateRow(Base):
|
|
| 191 |
`problem_validated`, `status` (text 'active'|'inactive' — soft-delete),
|
| 192 |
`data_bind` (jsonb), `data_bind_version` (int), `report_collection` (jsonb).
|
| 193 |
|
| 194 |
-
|
| 195 |
-
`
|
| 196 |
-
`
|
| 197 |
-
|
| 198 |
-
|
|
|
|
| 199 |
|
| 200 |
`analysis` (singular) is the deprecated DUPLICATE table Harry will drop — never use it.
|
| 201 |
Class name kept as `AnalysisStateRow`.
|
|
@@ -206,8 +207,13 @@ class AnalysisStateRow(Base):
|
|
| 206 |
analysis_title = Column(String, nullable=False, default="New analysis")
|
| 207 |
problem_statement = Column(Text, nullable=False, default="")
|
| 208 |
problem_validated = Column(Boolean, nullable=False, default=False)
|
| 209 |
-
|
| 210 |
report_id = Column(UUID(as_uuid=False), nullable=True)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 211 |
created_at = Column(DateTime(timezone=True), server_default=func.now())
|
| 212 |
updated_at = Column(
|
| 213 |
DateTime(timezone=True), server_default=func.now(), onupdate=func.now()
|
|
|
|
| 151 |
"""
|
| 152 |
__tablename__ = "report_inputs"
|
| 153 |
|
| 154 |
+
# id/analysis_id are `uuid` to match dedorch's `report_inputs` + the analysis-family
|
| 155 |
+
# (analyses/reports/data_sources). No FK declared in Python (dedorch's migration owns it, #22).
|
| 156 |
+
id = Column(UUID(as_uuid=False), primary_key=True) # AnalysisRecord.record_id (uuid hex ok)
|
| 157 |
+
analysis_id = Column(UUID(as_uuid=False), index=True) # the analysis session id (nullable for now)
|
| 158 |
user_id = Column(String, nullable=False, index=True)
|
| 159 |
plan_id = Column(String, nullable=False)
|
| 160 |
data = Column(JSONB, nullable=False)
|
|
|
|
| 191 |
`problem_validated`, `status` (text 'active'|'inactive' — soft-delete),
|
| 192 |
`data_bind` (jsonb), `data_bind_version` (int), `report_collection` (jsonb).
|
| 193 |
|
| 194 |
+
Reconciled to that shape (#4, 2026-06-26): `user_id` (was `owner_id`) + `status`/`data_bind`/
|
| 195 |
+
`data_bind_version`/`report_collection` added. dedorch still carries `problem_statement`/
|
| 196 |
+
`problem_validated` and does NOT yet have `objective`/`business_questions` — Harry's #3 drops
|
| 197 |
+
the former + adds the latter; the report layer reads the goal getattr-tolerantly so that swap
|
| 198 |
+
stays non-breaking. The new FE/Go columns are stored to match dedorch but NOT surfaced in the
|
| 199 |
+
`AnalysisState` pydantic contract (no Python reader needs them yet).
|
| 200 |
|
| 201 |
`analysis` (singular) is the deprecated DUPLICATE table Harry will drop — never use it.
|
| 202 |
Class name kept as `AnalysisStateRow`.
|
|
|
|
| 207 |
analysis_title = Column(String, nullable=False, default="New analysis")
|
| 208 |
problem_statement = Column(Text, nullable=False, default="")
|
| 209 |
problem_validated = Column(Boolean, nullable=False, default=False)
|
| 210 |
+
user_id = Column(String, nullable=False, index=True) # was owner_id (dedorch uses user_id)
|
| 211 |
report_id = Column(UUID(as_uuid=False), nullable=True)
|
| 212 |
+
# dedorch `analyses` columns (FE/Go concerns; carried so create_all matches dedorch).
|
| 213 |
+
status = Column(String, nullable=False, default="active") # active | inactive (soft-delete)
|
| 214 |
+
data_bind = Column(JSONB, nullable=False, default=list)
|
| 215 |
+
data_bind_version = Column(Integer, nullable=False, default=1)
|
| 216 |
+
report_collection = Column(JSONB, nullable=False, default=list)
|
| 217 |
created_at = Column(DateTime(timezone=True), server_default=func.now())
|
| 218 |
updated_at = Column(
|
| 219 |
DateTime(timezone=True), server_default=func.now(), onupdate=func.now()
|