PROJECT_BRD.md

Data Eyond — Python Agentic Service: Business Requirements & Design (BRD)

Status: draft for review · Date: 2026-06-26 · Branch: pr/4 Audience: Harry (Go gateway) + leads/stakeholders. Scope: the Python agentic LLM service (Agentic-Service-Data-Eyond-Catalog) only — its requirements, capabilities, architecture, data, and integration contract. Companions (source of truth, not duplicated here): REPO_STATUS.md (current built state) · API_ENDPOINTS.md (FE-callable API) · DEV_PLAN.md (in-flight plan). This BRD synthesizes those into a stakeholder-facing document; convert to PDF/Word for distribution.

---

1. Purpose & scope

Data Eyond is an "AI data scientist" for business analytics, modelled on CRISP-DM (Business Understanding → Data Understanding → Preparation → Modeling → Evaluation → Deployment). A user sets a goal, connects data (databases or files), asks natural-language analytical questions, and receives CRISP-DM-structured answers that can be exported as a versioned report. The aim is a "junior data scientist that hands back a decision-ready deliverable," not a "chatbot over a database."

This document covers the Python service — the agentic reasoning layer. It does not specify the Go gateway or the React frontend except at their integration boundaries (§9, §11).

2. Business context & objectives

• Target users: executives doing self-serve deep-dives; analysts offloading routine work.
• Value: turn a business question + connected data into auditable, CRISP-DM-structured findings and a formal report, without the user writing SQL or code.
• Objectives: (a) accurate, grounded analysis over the user's own data; (b) a decision-ready, versioned report artifact; (c) safe, read-only access to user data; (d) a clean service contract the Go gateway can integrate against.

3. Stakeholders & actors

Actor	Role
End user (exec/analyst)	Defines the analysis goal, asks questions, generates reports (via the FE)
Frontend (React/Vite)	Talks to Go for everything; to Python only for chat streaming
Go gateway (Orchestrator-Agent-Service)	Auth/JWT, rooms, documents, DB-credential storage, catalog ingestion, all DB migrations, and now all analysis-state writes
Python agentic service (this repo)	Router, skills, slow analytical path, structured query engine, RAG, report generation
Harry	Owns the Go gateway + dedorch DB migrations

4. Solution overview

Request flow is FE → Go → Python; the FE calls Python directly only for chat streaming. The Python service is a FastAPI app that classifies each user message and dispatches it to the right capability, streaming results back over SSE. Heavy analysis runs through a deterministic slow path (plan → execute → assemble) whose structured output is persisted and later rendered into reports.

5. Functional requirements (capabilities)

ID	Capability	Description
FR-1	Intent routing	One GPT-4o call classifies each message into one of 5 intents — chat, help, check, unstructured_flow, structured_flow — with history-aware query rewriting (EN/ID).
FR-2	Help skill	State-aware, next-step guidance (LLM, streamed); only offers actions the current state allows (e.g., a report only when one is generatable).
FR-3	Check skill	No-LLM inventory of available structured data + uploaded documents.
FR-4	Structured analysis (slow path)	Planner → TaskRunner → Assembler: a static DAG of tool-call chains, degrade-and-continue execution, narrative authored by one LLM call; produces a structured run record.
FR-5	Structured query engine	Catalog-driven JSON IR → deterministic SQL/pandas compiler → read-only executor, with single-level FK joins (DB sources).
FR-6	Unstructured RAG	Retrieval over PGVector document chunks, answered by the chatbot.
FR-7	Analytics tools	Composite analyze_* (descriptive, aggregate, correlation, trend) over data-access tools (check_*, retrieve_*).
FR-8	Report generation	Deterministic assembly of findings/EDA/limitations/method from persisted run records + one LLM call for the executive summary; versioned, formal markdown.
FR-9	Analysis sessions	One session = one analysis = one chat room (analysis_id == room_id); per-analysis data-source binding.

Goal capture (post-2026-06-24 pivot): the analysis goal is two user-entered fields — objective + business_questions — captured at onboarding, both mandatory, no agent validation. The former agent-validated "problem statement" + its gate are removed.

6. Analysis & report lifecycle

1. Create analysis (via Go) — session row + chat room + chosen data-source bindings; goal = objective + business_questions.
2. Ask questions — POST /chat/stream; the router dispatches; structured_flow questions run the slow path and persist one report_inputs row per run (the report's source of truth).
3. Generate report — the report skill reads the session's report_inputs, assembles the structured sections + an executive summary, and persists an immutable versioned report (markdown).
4. Read reports — list versions / fetch a version.

Reports are records-based (never from chat history) and require the slow path to have run (enable_slow_path=true) so records exist.

7. System architecture (subsystems)

FastAPI + async SQLAlchemy + LangChain (Azure GPT-4o) + Redis + Azure Blob + PGVector. Key subsystems (detail in REPO_STATUS §9):

• Router (agents/orchestration.py) — 5-intent classifier.
• Skills (agents/handlers/) — help (LLM), check (no-LLM).
• Slow path (agents/slow_path/ + agents/planner/) — Planner, TaskRunner, Assembler.
• Structured query engine (query/) — IR validate → compile → read-only execute (never raises).
• Report (agents/report/) — generator, store (advisory-locked versioning), readiness floor.
• Observability — Langfuse tracing (PII-masked); Redis caching; pooled DB engines.

8. Data model

SQLAlchemy models in src/db/postgres/models.py (detail in REPO_STATUS §8). The service is moving to the shared dedorch DB (Go owns migrations; Python is consumer-only — §11).

Table	Purpose	Owner
users	accounts (incl. fullname for report authorship)	Go
analyses (plural)	per-analysis session state: objective/business_questions (pivot), user_id, status, data_bind(+version), report_collection, report_id	Go (dedorch)
analyses_messages	the analysis chat room (user Q + agent A) — replaces deprecated chat_messages/rooms	Go (dedorch)
report_inputs	one jsonb row per slow-path run — the report's source of truth (was analysis_records)	Python (schema handed to Go)
reports	versioned report artifacts (markdown)	Go (dedorch)
data_sources	per-analysis source bindings	Go (dedorch)
documents, databases, data_catalog	uploads, DB credentials (Fernet), per-user catalog	Go ingestion
langchain_pg_embedding	PGVector document chunks	Go ingestion

9. API surface (FE-callable)

Full contract + request/response examples in API_ENDPOINTS.md. The FE-callable surface is 4 things:

1. call_agent — POST /api/v1/chat/stream (SSE).
2. list_skills — GET /api/v1/tools (slash-command catalog; cacheable).
3. skill: help — via call_agent (router intent; no dedicated endpoint).
4. skill: report — POST /api/v1/report + GET list/version.

analysis_id == room_id. Auth is terminated at Go; Python trusts user_id/room_id.

10. Non-functional requirements

Area	Requirement / mechanism
Security — data access	All structured queries are read-only: IR validation + SQL compiler whitelist + sqlglot SELECT-only guard + read-only session + LIMIT/timeout. DB credentials are Fernet-encrypted with an owner check.
Security — PII	PII columns carry no sample values into prompts; Langfuse masks PII on assembler/chatbot spans.
Reliability	Never-throw seams across tools/query/executors/state/report — failures degrade to soft output rather than crashing a turn.
Performance	Redis response cache (stateless chat only) + retrieval cache; pooled DB engines + speculative prewarm; warm Azure clients per process.
Observability	Langfuse: one trace per request (router/planner/assembler/chatbot + tool spans), tokens + latency.
Portability	Runs on HuggingFace Spaces (Linux) and Windows (run.py sets the selector event-loop policy for psycopg3 async).

11. Integrations & dependencies

• Two-repo boundary: Python is edited independently; Go + FE are reference-only. Python reads/writes shared Postgres, reads Azure Blob (Parquet for tabular sources), uses Redis.
• dedorch migration: Python is moving from the dataeyond DB to dedorch. Go owns all migrations; Python is consumer-only — if Python needs a table, it hands Go the schema. Table names are plural (analyses, analyses_messages); rooms/chat_messages are deprecated there.
• State writes via Go: all analysis-state writes move behind Go; Python's per-turn state access becomes a read-only get (in progress).
• External services: Azure OpenAI (GPT-4o + embeddings), Azure Blob, Postgres (+ PGVector), Redis, Langfuse.

12. Constraints & assumptions

• The slow path must be enabled (enable_slow_path=true) for reports to have content.
• report_inputs is Python-owned but its schema is provided to Go so the dedorch migration creates it (so it survives the SKIP_INIT_DB cutover).
• Charts and images are out of scope for now — reports are markdown (tables/bold/italic/separators); charts (Plotly JSON) and images (table + bucket) are deferred.
• The frontend has no dedicated UI designer; UI is being researched in parallel.

13. Open items & roadmap

Tracked in DEV_PLAN.md §4. Headlines: finish Go-side state ownership (#7/#18), the dedorch analyses migration (#3, mostly done), HF deploy + playground test (#13), chat-path migration to analyses_messages (#25), and the deferred charts/images/UI work (#26/#27/#28).

14. Glossary

• Slow path — the deterministic Planner→TaskRunner→Assembler analytical pipeline.
• report_inputs — the jsonb table of slow-path run records the report reads (formerly analysis_records).
• dedorch — the shared Postgres DB the service is migrating to; Go owns its migrations.
• CRISP-DM — the cross-industry standard data-mining process the analysis is structured around.
• analysis_id == room_id — one analysis session is one chat room, identified by the same id.