[NOTICKET] Add project repo documentation

REPO_STATUS.md

@@ -0,0 +1,303 @@
+# Data Eyond — Python Agentic Service: Current Status
+
+**Audience:** teammates onboarding onto the Python repo (`Agentic-Service-Data-Eyond-Catalog`).
+**Scope:** what the code does **right now** (branch `pr/3`, ticket KM-652). Describes current state only — no roadmap or to-dos.
+**Snapshot date:** 2026-06-24.
+
+> This file is grounded in the source, not the older design docs. Where the two
+> disagree, the code wins — see [§11 Doc-vs-code](#11-where-the-older-docs-are-stale).
+> `REPO_CONTEXT.md` / `ARCHITECTURE.md` are the original Phase-2 design docs and are
+> stale on the router, joins, and the analysis/report stack.
+
+---
+
+## 1. The product in one paragraph
+
+Data Eyond is an **"AI data scientist"** for business analytics, modelled on **CRISP-DM**
+(Business Understanding → Data Understanding → Preparation → Modeling → Evaluation →
+Deployment). It targets executives doing self-serve deep-dives and analysts offloading
+routine work. A user defines a goal, connects data (DB or files), asks natural-language
+analytical questions, and gets CRISP-DM-structured answers that can be exported as a
+versioned **report**. The aim is "junior data scientist that hands back a decision-ready
+deliverable," not "chatbot over a database."
+
+---
+
+## 2. Three repos, one hard ownership rule
+
+Request flow is **FE → Go → Python**. The FE never calls Python directly except for chat
+streaming.
+
+| Repo | Role | We edit? |
+|---|---|---|
+| **Python** — `Agentic-Service-Data-Eyond-Catalog` (this repo) | The agentic LLM service: router, gate, skills, slow analytical path, structured query engine, unstructured RAG, report generation, analysis-session state. FastAPI + async SQLAlchemy + LangChain + Azure GPT-4o. | **Yes — the only repo we edit.** |
+| **Go** — `Orchestrator-Agent-Service` | Gateway / data plane: interview agent, auth/JWT, rooms, documents (Azure Blob + CSV/XLSX→Parquet + embeddings), database_clients (Fernet creds), catalog ingestion, **all DB migrations**. | Reference only. |
+| **FE** — `E2E-Frontend-Data-Eyond` | React/Vite SPA. Talks to Go for everything and to Python only for chat streaming. | Reference only. |
+
+Shared infra: **Postgres** (app tables + `data_catalog` jsonb + PGVector `langchain_pg_embedding`), **Azure Blob**, and (Python-only) **Redis**.
+
+---
+
+## 3. Tech stack & how to run
+
+- Python 3.12, FastAPI, uvicorn, sse-starlette
+- Async SQLAlchemy 2.0 + asyncpg (Postgres); psycopg3 for the PGVector engine
+- LangChain + langchain-openai (Azure OpenAI GPT-4o) + langchain-postgres (PGVector)
+- Redis (response + retrieval cache), Azure Blob (uploads + Parquet)
+- pandas / pyarrow, sqlglot, pydantic v2, structlog, slowapi, langfuse
+- DB connectors: psycopg2, pymysql, pymssql, sqlalchemy-bigquery, snowflake-sqlalchemy
+
+Run (Linux/Docker): `uv run --no-sync uvicorn main:app --host 0.0.0.0 --port 7860`
+Run (Windows): `uv run --no-sync python run.py` (sets `WindowsSelectorEventLoopPolicy` for psycopg3 async — don't call uvicorn directly on Windows).
+
+Tests live locally and are gitignored. Run with `./.venv/Scripts/python.exe -m pytest`.
+
+---
+
+## 4. Chat request lifecycle
+
+Entry: `POST /api/v1/chat/stream` (`src/api/v1/chat.py`) → `ChatHandler.handle(...)`
+(`src/agents/chat_handler.py`). One shared `ChatHandler` per process keeps the Azure clients warm.
+
+```
+POST /chat/stream { user_id, room_id, message }
+  │  (analysis_id == room_id — one session = one analysis = one chat room)
+  ├─ Redis response-cache check (1h TTL, key chat:{room}:{user}:{message})  ── hit → replay
+  ├─ greeting/farewell short-circuit (_fast_intent, EN+ID)                  ── hit → canned reply
+  ├─ load last-10 history
+  └─ ChatHandler.handle:
+       1. classify → RouterDecision               [1 GPT-4o call]
+       2. ensure analysis-state row (get-or-create, idempotent)
+       3. gate (if ENABLE_GATE): structured_flow + not validated → problem_statement
+       4. emit `intent` (internal; gates caching), then dispatch:
+            chat              → ChatbotAgent → SSE
+            help              → HelpAgent (state + history + readiness) → SSE
+            check             → check_data/check_knowledge tool → rendered table  [no LLM]
+            problem_statement → PS draft + validate → write state
+            unstructured_flow → DocumentRetriever (PGVector RAG) → ChatbotAgent → SSE
+            structured_flow   → CatalogReader → (slow path | QueryService) → SSE
+       5. SSE events: intent (internal), sources, chunk, status, done | error
+```
+
+Only the `chat` intent is cached (stateless). Messages persist on `done`.
+
+---
+
+## 5. Report lifecycle
+
+The report is a **dedicated API, not a chat route** (`src/api/v1/report.py`):
+
+```
+POST /report?analysis_id&user_id
+  ├─ load analysis state; enforce the report FLOOR
+  │     (validated goal + ≥1 substantive analyze_* success) → else 409
+  ├─ ReportGenerator.generate (src/agents/report/generator.py):
+  │     read persisted AnalysisRecords (list_for_analysis)
+  │     deterministically assemble findings / caveats / open-questions /
+  │       data-source appendix / CRISP-DM method appendix  (copied verbatim)
+  │     ONE LLM call → executive summary only (deterministic fallback on failure)
+  │     render markdown
+  ├─ ReportStore.save: advisory-locked version assignment → dedorch `reports`
+  └─ write report_id back onto analysis state
+
+GET /report/{analysis_id}        → list versions (oldest-first)
+GET /report/{analysis_id}/{ver}  → fetch one version
+```
+
+Two facts to internalise:
+- **Records only exist on the slow path.** With `ENABLE_SLOW_PATH=false` (the default) no
+  records accumulate, so generation 409s — by design, not a bug.
+- **dedorch `reports` stores markdown only.** Structured report fields are computed at
+  generation, rendered into `rendered_markdown`, and only the markdown is persisted; on
+  read-back the structured fields come back empty.
+
+---
+
+## 6. Feature list (what's built)
+
+- **6-intent handler router** with history-aware query rewriting (EN/ID).
+- **Deterministic gate** — `problem_validated` precondition for analytical queries.
+- **Skills:** `help` (LLM, state-aware next-step guidance), `check` (no-LLM data/document inventory), `problem_statement` (LLM drafter + completeness validator).
+- **Slow analytical path:** Planner → TaskRunner → Assembler (static plan, degrade-and-continue, 3 LLM calls fixed).
+- **Structured query engine:** catalog-driven JSON IR → deterministic SQL/pandas compiler → read-only executor, with **single-level FK joins** (DB sources only).
+- **Unstructured RAG** over PGVector.
+- **Analytics tools:** 4 registered composite `analyze_*` (descriptive, aggregate, correlation, trend) + 4 data-access tools (check_data, check_knowledge, retrieve_data, retrieve_knowledge). Four further composites (comparison, contribution, profile, segment) exist in code but are **not registered** with the Planner.
+- **Versioned report generation** from persisted records.
+- **Analysis sessions:** data-first creation gate (≥1 bound source), per-analysis data-source binding (#10).
+- **Langfuse tracing** (PII-masked), **Redis caching**, **pooled DB engines** + speculative prewarm.
+
+---
+
+## 7. API surface (this repo, all under `/api/v1`)
+
+| Endpoint | Purpose | Caller |
+|---|---|---|
+| `POST /chat/stream` | Main chat SSE (router → gate → dispatch) | FE → Go → Python (the only FE→Python call today) |
+| `DELETE /chat/cache` · `/chat/cache/room/{id}` · `/retrieval/cache/{user_id}` | Cache management | internal / ops |
+| `POST /analysis/create` · `GET /analysis` · `GET /analysis/{id}` | Analysis-session CRUD (state + room + bindings created atomically) | intended FE → Go |
+| `POST /report` · `GET /report/{id}` · `GET /report/{id}/{ver}` | Report generate / list / fetch | FE → Go (report button) |
+| `GET /tools` | Slash-command catalog (static, cacheable) | Go caches it for the FE "/" menu |
+| `users` · `room` · `document` · `db_client` · `data_catalog` routers | Phase-1 legacy; functionally migrated to Go | mostly dormant |
+
+---
+
+## 8. Data model
+
+SQLAlchemy models in `src/db/postgres/models.py`. Created on startup by `init_db()`
+unless `SKIP_INIT_DB=true`.
+
+| Table | Shape | Written by | Read by |
+|---|---|---|---|
+| `users`, `rooms`, `chat_messages`, `message_sources` | base app | chat endpoint, Go | chat history |
+| `documents`, `databases` | uploads + DB creds (Fernet-encrypted) | Go ingestion | executor cred resolution |
+| `data_catalog` | per-user jsonb `Catalog` (Source → Table → Column) | Go ingestion / Python pipeline | CatalogReader, planner, tools |
+| `langchain_pg_embedding` | PGVector document chunks | Go ingestion | DocumentRetriever |
+| `analysis_records` | jsonb `AnalysisRecord`, one per slow-path run | slow path | ReportGenerator, report readiness |
+| `analysis` *(dedorch)* | uuid id, `owner_id`, `problem_statement`, `problem_validated`, `report_id` | `/analysis/create`, state store | gate, Help, report |
+| `reports` *(dedorch)* | uuid, `title` + markdown `content` + `version` | ReportStore | report API |
+| `data_sources` *(dedorch)* | per-analysis binding; `reference_id` = catalog source_id | `/analysis/create` | structured-flow scoping, report appendix |
+
+**Catalog shape** (the jsonb in `data_catalog`):
+`Catalog → Source[ {source_id, source_type ∈ schema|tabular|unstructured, name, location_ref} → Table[ {table_id, name, row_count, foreign_keys[]} → Column[ {column_id, name, data_type, nullable, pii_flag, sample_values|null, stats} ] ] ]`. PII columns have `sample_values: null` so real values never enter prompts.
+
+**QueryIR shape** (`src/query/ir/models.py`):
+`{ source_id, table_id, joins[], select[], filters[], group_by[], order_by[], limit }`.
+Joins are single-level equi-joins to a related table **in the same source**, FK-backed,
+**DB sources only**.
+
+---
+
+## 9. Subsystems (where the code lives)
+
+### Router — `src/agents/orchestration.py`
+One GPT-4o structured-output call → `RouterDecision{intent, rewritten_query, confidence}`,
+`intent ∈ {chat, help, problem_statement, check, unstructured_flow, structured_flow}`. It's a
+*handler* classifier: `structured_flow` = slow path, `unstructured_flow` = fast RAG; the
+data-modality mix on the slow path is the Planner's job. Prompt: `src/config/prompts/intent_router.md`.
+
+### Gate — `src/agents/gate.py`
+Pure function, no I/O. One rule: `structured_flow` + `not problem_validated` → redirect to
+`problem_statement`; everything else passes. Fails **closed** (not-validated stub) on a missing
+row / read error. `AnalysisState` is the locked contract (id, analysis_title, problem_statement,
+problem_validated, owner_id, report_id, created_at, updated_at). Applied in `chat_handler.handle`
+only when `ENABLE_GATE` and `analysis_id` are present.
+
+### Skills — `src/agents/handlers/`
+- `help.py` — LLM (streamed). A consistency guard derives the *allowed* actions from state
+  (mirrors the gate) and feeds them to the prompt, so Help can't suggest a report when the goal
+  isn't validated or there's nothing to report. Consumes a deterministic readiness signal.
+- `check.py` — **no LLM.** Keyword cues route to `check_data`, `check_knowledge`, or both
+  (helicopter view, concurrent). Renders tool tables to markdown.
+- `problem_statement.py` — LLM drafts `{problem_statement, objective, metric, missing[], feedback}`;
+  valid iff a statement exists and `missing` is empty (a bare data question stays incomplete). On a
+  valid draft, writes `problem_statement` + `problem_validated=True`.
+
+### Slow path — `src/agents/slow_path/` + `src/agents/planner/`
+- **Planner** (`planner/service.py`) — 1 LLM call → `TaskList` (DAG of tool-call chains). 8-check
+  validator with re-prompt retry (max 3). `BusinessContext` is a **stub** (`planner/business_context.py`),
+  which is why the slow path stays opt-in.
+- **TaskRunner** (`slow_path/task_runner.py`) — deterministic, 0 LLM. Wave-based execution,
+  `${t<id>}` placeholder resolution (Pattern A), never-throw invocation, **degrade-and-continue**
+  (failed task → dependents skipped, independent branches run). No replanning.
+- **Assembler** (`slow_path/assembler.py`) — 1 LLM call authoring only the narrative; code copies
+  the structured `results_snapshot` / `tasks_run` from the run state into the `AnalysisRecord`
+  (the report's source of truth).
+
+Streaming + persistence: `chat_handler._run_slow_path` bridges per-stage progress to SSE `status`
+events, prewarms the DB engine in parallel with planning, emits the answer, then persists the
+record stamped with `user_id` + `analysis_id`.
+
+### Structured query engine — `src/query/`
+`QueryService.run` (`query/service.py`): plan → validate → retry(3) → dispatch → execute; **never
+raises** (errors land in `QueryResult.error`). `IRValidator` (`query/ir/validator.py`) checks
+source/table/column existence, op/agg whitelists, type compatibility, limit cap, and **FK-backed
+joins** (DB only). `DbExecutor` (`query/executor/db.py`): SqlCompiler → sqlglot SELECT-only guard →
+Fernet-decrypt creds (with owner check) → `asyncio.to_thread` (30s timeout) → pooled engine
+(read-only + statement_timeout) → 10k row cap. Defense-in-depth: IR validation + compiler whitelist
++ sqlglot guard + read-only session + LIMIT/timeout.
+
+### Data-source binding (#10) — `src/agents/binding_store.py`
+At `/analysis/create`, chosen `data_source_ids` become `data_sources` rows. On a `structured_flow`
+turn the catalog reader is wrapped so the Planner and the tools' re-reads see the same scoped
+catalog. **Fail-open**: empty/disjoint binding → whole catalog.
+
+### Tool layer — `src/tools/data_access.py`, `src/agents/planner/registry.py`
+`DataAccessToolInvoker` implements the never-throw tool seam for the 4 data-access tools.
+`retrieve_data` runs a pre-built IR (validate → dispatch → execute, skipping the planner) and
+coerces `Decimal`→`float` — the Pattern A handoff the `analyze_*` tools consume. The planner
+registry composes a local data-access spec stub (name-checked against `DATA_ACCESS_TOOLS`) with the
+real `analytics_registry()`.
+
+### Report — `src/agents/report/`
+`generator.py` reads records, deterministically assembles structured fields, 1 LLM call for the
+executive summary; `store.py` versions under an advisory lock and persists markdown to dedorch
+`reports`; `readiness.py` defines the **report floor** (validated goal + ≥1 successful `analyze_*`)
+shared by the report API and the Help readiness signal so the two can't disagree.
+
+### Observability — Langfuse
+The endpoint's `ChatHandler` runs with `enable_tracing=True`. One trace per request groups
+router/planner/assembler/chatbot + tool spans. PII policy: router/planner unmasked (PII-safe
+summaries); assembler/chatbot masked (see real rows); tool spans carry name + arg keys + row counts
+only.
+
+---
+
+## 10. Feature flags
+
+| Flag | Where | Default | Effect |
+|---|---|---|---|
+| `ENABLE_SLOW_PATH` | `settings.enable_slow_path` | **off** | Route `structured_flow` through Planner/TaskRunner/Assembler (vs single-query `QueryService`). Records persist only on the slow path → reports require this on. |
+| `ENABLE_GATE` | `settings.enable_gate` | **off** | Apply the `problem_validated` redirect. Off because legacy rooms have no state row. |
+| `SKIP_INIT_DB` | env, `main.py` | off | Skip `create_all` on startup — the dedorch cutover switch (Go owns dedorch migrations). |
+| `enable_tracing` | hardcoded `True` in `chat.py` | on (endpoint) | Langfuse tracing. |
+
+---
+
+## 11. Where the older docs are stale
+
+Trust the code. The original Phase-2 docs (`ARCHITECTURE.md`, `REPO_CONTEXT.md`) and the Go repo's
+copies disagree with the current code on:
+
+| Topic | Old docs | Current code |
+|---|---|---|
+| Router | 3-way `source_hint` (chat/unstructured/structured) | Flat **6-intent** `RouterDecision` |
+| Joins in IR | "single-table only; deferred" | **Single-level FK-backed joins** (DB sources only) |
+| Analysis / report / gate / slow path | "Phase 2 spine only" | All built and present |
+| `analysis_id` | open question | resolved: **`analysis_id == room_id`** |
+| Report source | (newer invariant) "from records, never chat history" | confirmed: generator reads `AnalysisRecord`s |
+
+---
+
+## 12. dedorch migration — current state
+
+The Python DB is moving from `dataeyond` → **dedorch** (Go owns dedorch migrations; Python is
+consumer-only). Current state:
+
+- Base tables already match dedorch.
+- The analysis-family models have been **renamed to dedorch** on `pr/3`: `analysis` (was
+  `analysis_states`, uuid ids), `data_sources` (was `analysis_data_sources`), `reports` (was
+  `analysis_reports`, flattened to title + markdown content + version).
+- `analysis_records` (the slow-path structured output) has **no dedorch home** — it remains a
+  Python-owned jsonb table.
+- The connection-string cutover (paired with `SKIP_INIT_DB`) is a coordinated step that has not
+  happened yet; Python still creates tables on startup until then.
+
+The dedorch migrations themselves live outside the three checked-out repos (Harry owns them), so the
+dedorch table shapes are asserted by the Python model docstrings, not visible in the Go repo here.
+
+---
+
+## 13. Conventions & gotchas
+
+- **Two Postgres engines:** app engine + a separate PGVector engine (`prepared_statement_cache_size=0`)
+  because PGVector emits multi-statement strings asyncpg rejects.
+- **Identifiers vs values:** identifiers come from the catalog and are inlined as quoted; filter
+  values are always parameterized.
+- **Settings aliases:** `.env` uses double-underscore names (`azureai__api_key__4o`); `Settings`
+  exposes them as `azureai_api_key_4o`.
+- **Never-throw seams** are pervasive (tool invoker, query service, executors, state/binding reads,
+  record persistence, report summary). Failures degrade into soft output rather than raising — good
+  for UX, but they can mask real breakage (e.g. a binding silently fail-opening to the full catalog).
+- **Prompts** live in `src/config/prompts/*.md`. `chatbot_system.md` has `guardrails.md` appended so
+  guardrails win on conflict.
+- **Tests** are gitignored (team decision) — run them locally.