PROJECT_BRD.md

# Data Eyond — Python Agentic Service: Business Requirements & Design (BRD)

**Status:** draft for review · **Date:** 2026-06-26 · **Branch:** `pr/4`
**Audience:** Harry (Go gateway) + leads/stakeholders.
**Scope:** the Python **agentic LLM service** (`Agentic-Service-Data-Eyond-Catalog`) only — its
requirements, capabilities, architecture, data, and integration contract.
**Companions (source of truth, not duplicated here):** [REPO_STATUS.md](REPO_STATUS.md) (current
built state) · [API_ENDPOINTS.md](API_ENDPOINTS.md) (FE-callable API) · [DEV_PLAN.md](DEV_PLAN.md)
(in-flight plan). This BRD synthesizes those into a stakeholder-facing document; convert to PDF/Word
for distribution.

---

## 1. Purpose & scope
Data Eyond is an **"AI data scientist"** for business analytics, modelled on **CRISP-DM** (Business
Understanding → Data Understanding → Preparation → Modeling → Evaluation → Deployment). A user sets a
goal, connects data (databases or files), asks natural-language analytical questions, and receives
CRISP-DM-structured answers that can be exported as a versioned **report**. The aim is a *"junior data
scientist that hands back a decision-ready deliverable,"* not a *"chatbot over a database."*

This document covers the **Python service** — the agentic reasoning layer. It does **not** specify the
Go gateway or the React frontend except at their integration boundaries (§9, §11).

## 2. Business context & objectives
- **Target users:** executives doing self-serve deep-dives; analysts offloading routine work.
- **Value:** turn a business question + connected data into auditable, CRISP-DM-structured findings
  and a formal report, without the user writing SQL or code.
- **Objectives:** (a) accurate, grounded analysis over the user's own data; (b) a decision-ready,
  versioned report artifact; (c) safe, read-only access to user data; (d) a clean service contract the
  Go gateway can integrate against.

## 3. Stakeholders & actors
| Actor | Role |
|---|---|
| End user (exec/analyst) | Defines the analysis goal, asks questions, generates reports (via the FE) |
| Frontend (React/Vite) | Talks to Go for everything; to Python only for chat streaming |
| Go gateway (`Orchestrator-Agent-Service`) | Auth/JWT, rooms, documents, DB-credential storage, catalog ingestion, **all DB migrations**, and now all analysis-state writes |
| Python agentic service (this repo) | Router, skills, slow analytical path, structured query engine, RAG, report generation |
| Harry | Owns the Go gateway + dedorch DB migrations |

## 4. Solution overview
Request flow is **FE → Go → Python**; the FE calls Python directly only for chat streaming. The Python
service is a **FastAPI** app that classifies each user message and dispatches it to the right
capability, streaming results back over SSE. Heavy analysis runs through a deterministic **slow path**
(plan → execute → assemble) whose structured output is persisted and later rendered into reports.

## 5. Functional requirements (capabilities)
| ID | Capability | Description |
|---|---|---|
| FR-1 | **Intent routing** | One GPT-4o call classifies each message into one of 5 intents — `chat`, `help`, `check`, `unstructured_flow`, `structured_flow` — with history-aware query rewriting (EN/ID). |
| FR-2 | **Help skill** | State-aware, next-step guidance (LLM, streamed); only offers actions the current state allows (e.g., a report only when one is generatable). |
| FR-3 | **Check skill** | No-LLM inventory of available structured data + uploaded documents. |
| FR-4 | **Structured analysis (slow path)** | Planner → TaskRunner → Assembler: a static DAG of tool-call chains, degrade-and-continue execution, narrative authored by one LLM call; produces a structured run record. |
| FR-5 | **Structured query engine** | Catalog-driven JSON IR → deterministic SQL/pandas compiler → read-only executor, with single-level FK joins (DB sources). |
| FR-6 | **Unstructured RAG** | Retrieval over PGVector document chunks, answered by the chatbot. |
| FR-7 | **Analytics tools** | Composite `analyze_*` (descriptive, aggregate, correlation, trend) over data-access tools (`check_*`, `retrieve_*`). |
| FR-8 | **Report generation** | Deterministic assembly of findings/EDA/limitations/method from persisted run records + one LLM call for the executive summary; **versioned**, formal markdown. |
| FR-9 | **Analysis sessions** | One session = one analysis = one chat room (`analysis_id == room_id`); per-analysis data-source binding. |

**Goal capture (post-2026-06-24 pivot):** the analysis goal is **two user-entered fields** —
`objective` + `business_questions` — captured at onboarding, **both mandatory, no agent validation**.
The former agent-validated "problem statement" + its gate are removed.

## 6. Analysis & report lifecycle
1. **Create analysis** (via Go) — session row + chat room + chosen data-source bindings; goal =
   `objective` + `business_questions`.
2. **Ask questions** — `POST /chat/stream`; the router dispatches; `structured_flow` questions run the
   slow path and **persist one `report_inputs` row per run** (the report's source of truth).
3. **Generate report** — the report skill reads the session's `report_inputs`, assembles the structured
   sections + an executive summary, and persists an immutable **versioned** report (markdown).
4. **Read reports** — list versions / fetch a version.

> Reports are **records-based** (never from chat history) and require the slow path to have run
> (`enable_slow_path=true`) so records exist.

## 7. System architecture (subsystems)
FastAPI + async SQLAlchemy + LangChain (Azure GPT-4o) + Redis + Azure Blob + PGVector. Key subsystems
(detail in REPO_STATUS §9):
- **Router** (`agents/orchestration.py`) — 5-intent classifier.
- **Skills** (`agents/handlers/`) — `help` (LLM), `check` (no-LLM).
- **Slow path** (`agents/slow_path/` + `agents/planner/`) — Planner, TaskRunner, Assembler.
- **Structured query engine** (`query/`) — IR validate → compile → read-only execute (never raises).
- **Report** (`agents/report/`) — generator, store (advisory-locked versioning), readiness floor.
- **Observability** — Langfuse tracing (PII-masked); Redis caching; pooled DB engines.

## 8. Data model
SQLAlchemy models in `src/db/postgres/models.py` (detail in REPO_STATUS §8). The service is moving to
the shared **dedorch** DB (Go owns migrations; Python is consumer-only — §11).

| Table | Purpose | Owner |
|---|---|---|
| `users` | accounts (incl. `fullname` for report authorship) | Go |
| `analyses` *(plural)* | per-analysis session state: `objective`/`business_questions` (pivot), `user_id`, `status`, `data_bind`(+version), `report_collection`, `report_id` | Go (dedorch) |
| `analyses_messages` | the analysis chat room (user Q + agent A) — replaces deprecated `chat_messages`/`rooms` | Go (dedorch) |
| `report_inputs` | one jsonb row per slow-path run — the report's source of truth (was `analysis_records`) | **Python** (schema handed to Go) |
| `reports` | versioned report artifacts (markdown) | Go (dedorch) |
| `data_sources` | per-analysis source bindings | Go (dedorch) |
| `documents`, `databases`, `data_catalog` | uploads, DB credentials (Fernet), per-user catalog | Go ingestion |
| `langchain_pg_embedding` | PGVector document chunks | Go ingestion |

## 9. API surface (FE-callable)
Full contract + request/response examples in [API_ENDPOINTS.md](API_ENDPOINTS.md). The FE-callable
surface is **4 things**:
1. **`call_agent`** — `POST /api/v1/chat/stream` (SSE).
2. **`list_skills`** — `GET /api/v1/tools` (slash-command catalog; cacheable).
3. **skill: `help`** — via `call_agent` (router intent; no dedicated endpoint).
4. **skill: `report`** — `POST /api/v1/report` + `GET` list/version.

`analysis_id == room_id`. Auth is terminated at Go; Python trusts `user_id`/`room_id`.

## 10. Non-functional requirements
| Area | Requirement / mechanism |
|---|---|
| **Security — data access** | All structured queries are read-only: IR validation + SQL compiler whitelist + sqlglot SELECT-only guard + read-only session + LIMIT/timeout. DB credentials are Fernet-encrypted with an owner check. |
| **Security — PII** | PII columns carry no sample values into prompts; Langfuse masks PII on assembler/chatbot spans. |
| **Reliability** | Never-throw seams across tools/query/executors/state/report — failures degrade to soft output rather than crashing a turn. |
| **Performance** | Redis response cache (stateless `chat` only) + retrieval cache; pooled DB engines + speculative prewarm; warm Azure clients per process. |
| **Observability** | Langfuse: one trace per request (router/planner/assembler/chatbot + tool spans), tokens + latency. |
| **Portability** | Runs on HuggingFace Spaces (Linux) and Windows (`run.py` sets the selector event-loop policy for psycopg3 async). |

## 11. Integrations & dependencies
- **Two-repo boundary:** Python is edited independently; Go + FE are reference-only. Python reads/writes
  shared Postgres, reads Azure Blob (Parquet for tabular sources), uses Redis.
- **dedorch migration:** Python is moving from the `dataeyond` DB to **dedorch**. **Go owns all
  migrations; Python is consumer-only** — if Python needs a table, it hands Go the schema. Table names
  are **plural** (`analyses`, `analyses_messages`); `rooms`/`chat_messages` are deprecated there.
- **State writes via Go:** all analysis-state writes move behind Go; Python's per-turn state access
  becomes a read-only get (in progress).
- **External services:** Azure OpenAI (GPT-4o + embeddings), Azure Blob, Postgres (+ PGVector), Redis,
  Langfuse.

## 12. Constraints & assumptions
- The slow path must be enabled (`enable_slow_path=true`) for reports to have content.
- `report_inputs` is Python-owned but its schema is provided to Go so the dedorch migration creates it
  (so it survives the `SKIP_INIT_DB` cutover).
- Charts and images are **out of scope for now** — reports are markdown (tables/bold/italic/separators);
  charts (Plotly JSON) and images (table + bucket) are deferred.
- The frontend has no dedicated UI designer; UI is being researched in parallel.

## 13. Open items & roadmap
Tracked in [DEV_PLAN.md](DEV_PLAN.md) §4. Headlines: finish Go-side state ownership (#7/#18), the
dedorch `analyses` migration (#3, mostly done), HF deploy + playground test (#13), chat-path migration
to `analyses_messages` (#25), and the deferred charts/images/UI work (#26/#27/#28).

## 14. Glossary
- **Slow path** — the deterministic Planner→TaskRunner→Assembler analytical pipeline.
- **`report_inputs`** — the jsonb table of slow-path run records the report reads (formerly `analysis_records`).
- **dedorch** — the shared Postgres DB the service is migrating to; Go owns its migrations.
- **CRISP-DM** — the cross-industry standard data-mining process the analysis is structured around.
- **`analysis_id == room_id`** — one analysis session is one chat room, identified by the same id.