Agentic-Service-Data-Eyond-Catalog

Running

App Files Files Community

Agentic-Service-Data-Eyond-Catalog / REPO_STATUS.md

ishaq101

/fix change model to gpt 5.4 mini (#17)

d65c41d about 3 hours ago

preview code

Raw

History Blame Contribute Delete

38.2 kB

Data Eyond — Python Agentic Service: Current Status

Audience: teammates onboarding onto the Python repo (Agentic-Service-Data-Eyond-Catalog). Scope: what the code does right now (branch pr/4, ticket KM-652). Describes current state only — no roadmap or to-dos. Snapshot date: 2026-06-25. Data-layer reconcile 2026-07-01: §8/§12 updated — dedorch cutover done, data_catalog model reconciled. Query-path fix 2026-07-02: §8/§13 — dedorch catalogs ship no FKs → Python infers them (fk_inference.py); shared-Fernet-key gotcha documented. Agent-quality fixes 2026-07-08 (pr/13): from the scoped live-test review — the planner gains an explicit infeasible outcome (TaskList.infeasible_reason → deterministic EN/ID data-gap reply via refusals.data_gap_message; no more force-mapping absent measures like pa AS "revenue"), the IR validator rejects bare selects under group_by (self-corrects via the planner retry), analyze_trend handles integer year/month columns (was collapsing every row into one 1970-01 bucket), planner few-shots add top-N (Example G) + infeasible (Example H), numeric catalog sample_values are base64-decoded at read (catalog/sample_decode.py — stopgap for Go's byte-marshaling; primary fix is Go-side), traceability no longer emits null source rows for failed retrievals, and check_data hides -1 row counts. Report v2 + analyze_merge planner support 2026-07-09 (pr/13): Sofia's analyze_merge tool (8abf635, KM-703) is now planner-supported (_validate_data_source guards data_right, two-retrieve→merge few-shot Example I, planner.md "Two measures per entity" rule); the report gains per-business-question answers (bq_answers — drafted by the SAME single LLM call, index-based record refs, deterministic fallback unchanged), "Attempted, Unresolved" + "Excluded Analyses" sections (failed runs are no longer silently dropped), evidence tables copied from results_snapshot (table-kind outputs, ≤3/record ≤10 rows ≤8 cols, check_* skipped), normalized caveat dedupe with caps (12/10), and single-language output via detect_reply_language; the report surface adds GET /tools/report/{analysis_id}/records (curation list), GET …/readiness (FE delta guard), and exclude_record_ids on POST — see API_CONTRACT_BE_PYTHON.md. Report compaction 2026-07-09 (pr/13): the rendered markdown drops the "Notes & Limitations", "Attempted, Unresolved", and "How This Was Analyzed" sections (team decision — compact report; render blocks commented out in report/generator.py, not deleted). The JSON body keeps caveats/open_questions/unresolved/method_steps and the curation/records endpoints are unchanged. Traceability data_used layer 2026-07-13 (pr/15): GET /api/v1/traceability gains a resolved, user-facing data_used[] block (one per retrieve_data call) — real source/table/column names, joins, plain-language filters, and result columns split into read-from-data vs computed (with formula, e.g. total_revenue = SUM(line_total), so an alias is never shown as a real column). Ids are kept but machine-only (FE must not render). Also adds tool_calls[].summary, and sources[] now carry source_name + every table touched. Deterministic catalog resolution (new src/traceability/resolve.py), no LLM, never-throw; catalog threaded to the scratchpad at the slow-path composition root. Additive/non-breaking; contract + TRACEABILITY_FE_HANDOFF.md updated. Spine v2 W2+W1 2026-07-13 (pr/16): §6/§7/§8/§9/§12 — render_chart tool lands (first of the render_* family: deterministic Plotly-JSON dataeyond.chart.v1 envelope, hand-built, no plotly dependency; planner-selected only on an explicit chart ask, EN/ID) + Python-owned message_charts store + GET /api/v1/charts (FE fetches on done, same pattern as traceability; empty list = valid 200); planner gains a named recipe table + viz few-shots (Example J tail, Example K viz-infeasible) + validator Check 10 (render_chart.data must reference a table-producing task); and the slow path gains the S1a quality checkpoint (slow_path/checkpoint.py, 0 LLM, never-throw, between runner and assembler: CK1 all-failed → deterministic honest-failure answer with no assembler call, CK2 empty retrieve + downstream, CK3 10k-cap truncation, CK4 single trend bucket, CK5 all-null column, CK6 chart-spec sanity; flags render as an "Execution assessment" block in the assembler input and every flag logs repair_candidate — the S1b evidence base). Design + handoff doc: SPINE_V2_PLAN.md. LLM model + env rename 2026-07-14 (pr/17): §2/§3/§9/§13 — the generation LLM is now Azure GPT-5.4-mini (deployment gpt-5.4-mini), not GPT-4o, and the settings quad is renamed azureai__*__4o → azureai__*__54m across all 9 LLM call sites. Hard rename, no __4o fallback — an environment that still sets only __4o resolves to empty strings and fails on the first LLM call, by design: the silent-wrong-model drift is exactly how HF stayed on GPT-4o while local ran 5.4-mini (identical question, identical catalog, divergent planner output — HF hallucinated catalog ids, local planned correctly). Deploying requires all four __54m vars set in the HF Space secrets. Cross-repo update 2026-06-29: §2/§8/§11/§12 re-verified against the Go source (Orchestrator-Agent-Service), not its docs. The Go service has moved well past its own (uncommitted, stale) design docs: it now hosts the dedorch SQL migrations in-repo and a full /api/v1/analyses + /api/v1/skills REST surface. Go does not call Python yet — those skills are placeholders (see §12).

This file is grounded in the source, not the older design docs. Where the two disagree, the code wins — see §11 Doc-vs-code. REPO_CONTEXT.md / ARCHITECTURE.md are the original Phase-2 design docs and are stale on the router, joins, and the analysis/report stack.

🚧 Direction update 2026-06-30 (pr/5 — DECIDED · IN PROGRESS). The 30 June checkpoint locked a restructure (contract: API_ENDPOINTS_RESTRUCTURE.md; live tracker: DEV_PLAN §0). Python is becoming a generation/AI-only service — Go owns the full analysis lifecycle and the data-plane endpoints. Scope:

Unwired from main + Swagger (router files kept, not deleted): analysis CRUD, room, db_client, document, data_catalog, users/login. ✅ DONE — KM-686, commit 0b2d678 (so the §7 rows for these are now commented out of main.py).

AI surface that stays live: chat → POST /api/v2/chat/stream (explicit analysis_id, not room_id); the skills regroup under /api/v1/tools/ (list · help · report); plus a new GET /api/v1/traceability (user-facing provenance per answer, backed by a Python-owned message_traceability store — renamed from observability, KM-691). ✅ built.

Only chat/stream moves to /api/v2; everything else stays /api/v1.

§2/§4/§7 below still describe the pre-restructure wiring except the unwire above, which has landed.

1. The product in one paragraph

Data Eyond is an "AI data scientist" for business analytics, modelled on CRISP-DM (Business Understanding → Data Understanding → Preparation → Modeling → Evaluation → Deployment). It targets executives doing self-serve deep-dives and analysts offloading routine work. A user defines a goal, connects data (DB or files), asks natural-language analytical questions, and gets CRISP-DM-structured answers that can be exported as a versioned report. The aim is "junior data scientist that hands back a decision-ready deliverable," not "chatbot over a database."

2. Three repos, one hard ownership rule

Request flow is FE → Go → Python. The FE never calls Python directly except for chat streaming.

Repo	Role	We edit?
Python — `Agentic-Service-Data-Eyond-Catalog` (this repo)	The agentic LLM service: router, gate, skills, slow analytical path, structured query engine, unstructured RAG, report generation, analysis-session state. FastAPI + async SQLAlchemy + LangChain + Azure GPT-5.4-mini (was GPT-4o until 2026-07-14).	Yes — the only repo we edit.
Go — `Orchestrator-Agent-Service`	Gateway / data plane: auth/JWT, documents (Azure Blob + CSV/XLSX→Parquet + embeddings), database_clients (Fernet creds), catalog ingestion (moved into Go, KM-578/590), all dedorch SQL migrations (now embedded in the Go repo: `internal/repository/postgres/migrations/0001–0004`), and the full analysis-lifecycle REST surface (`/api/v1/analyses` CRUD + messages + reports, `/api/v1/skills`). The interview agent and chat-rooms are deprecated → HTTP 410 (`internal/api/deprecation.go`).	Reference only.
FE — `E2E-Frontend-Data-Eyond`	React/Vite SPA. Talks to Go for everything and to Python only for chat streaming.	Reference only.

» pr/5 (decided, not yet in code): Python's non-AI endpoints (analysis CRUD, room, document, db_client, data_catalog, users/login) are being unwired — Python keeps only the generation/AI surface (chat, tools: help/report/list, traceability). See the Direction-update banner.

Shared infra: Postgres (app tables + data_catalog jsonb + PGVector langchain_pg_embedding), Azure Blob, and (Python-only) Redis.

3. Tech stack & how to run

Python 3.12, FastAPI, uvicorn, sse-starlette
Async SQLAlchemy 2.0 + asyncpg (Postgres); psycopg3 for the PGVector engine
LangChain + langchain-openai (Azure OpenAI GPT-5.4-mini, deployment gpt-5.4-mini; env quad azureai__*__54m — renamed from __4o 2026-07-14) + langchain-postgres (PGVector)
Redis (response + retrieval cache), Azure Blob (uploads + Parquet)
pandas / pyarrow, sqlglot, pydantic v2, structlog, slowapi, langfuse
DB connectors: psycopg2, pymysql, pymssql, sqlalchemy-bigquery, snowflake-sqlalchemy

Run (Linux/Docker): uv run --no-sync uvicorn main:app --host 0.0.0.0 --port 7860 Run (Windows): uv run --no-sync python run.py (sets WindowsSelectorEventLoopPolicy for psycopg3 async — don't call uvicorn directly on Windows).

Tests live locally and are gitignored. Run with ./.venv/Scripts/python.exe -m pytest.

4. Chat request lifecycle

Entry: POST /api/v1/chat/stream (src/api/v1/chat.py) → ChatHandler.handle(...) (src/agents/chat_handler.py). One shared ChatHandler per process keeps the Azure clients warm.

» pr/5: this endpoint moves to POST /api/v2/chat/stream with an explicit analysis_id field (replacing room_id), and the traceability detail (planning / tool I/O / sources) moves out of the stream to a separate GET /api/v1/traceability call. See the Direction-update banner.

POST /chat/stream { user_id, room_id, message }
  │  (analysis_id == room_id — one session = one analysis = one chat room)
  ├─ Redis response-cache check (1h TTL, key chat:{room}:{user}:{message})  ── hit → replay
  ├─ greeting/farewell short-circuit (_fast_intent, EN+ID)                  ── hit → canned reply
  ├─ load last-10 history
  └─ ChatHandler.handle:
       1. classify → RouterDecision               [1 LLM call]
       2. ensure analysis-state row (get-or-create, idempotent)
       3. emit `intent` (internal; gates caching), then dispatch:
            chat              → ChatbotAgent → SSE
            help              → HelpAgent (state + history + readiness) → SSE
            check             → check_data/check_knowledge tool → rendered table  [no LLM]
            unstructured_flow → DocumentRetriever (PGVector RAG) → ChatbotAgent → SSE
            structured_flow   → CatalogReader → (slow path | QueryService) → SSE
       4. SSE events: intent (internal), sources, chunk, status, done | error

Only the chat intent is cached (stateless). Messages persist on done.

The router emits 5 intents now. The problem_statement skill and the problem_validated gate were removed 2026-06-25 (KM-652) — the analysis goal is two user-entered fields (objective + business_questions) captured at onboarding, with no agent validation.

5. Report lifecycle

The report is a dedicated API, not a chat route (src/api/v1/report.py):

POST /report?analysis_id&user_id
  ├─ load analysis state; enforce the report FLOOR
  │     (≥1 substantive analyze_* success) → else 409
  ├─ ReportGenerator.generate (src/agents/report/generator.py):
  │     read persisted AnalysisRecords (list_for_analysis)
  │     deterministically assemble findings / caveats / open-questions /
  │       data-source appendix / CRISP-DM method appendix  (copied verbatim)
  │     ONE LLM call → executive summary only (deterministic fallback on failure)
  │     render markdown
  ├─ ReportStore.save: advisory-locked version assignment → dedorch `reports`
  └─ write report_id back onto analysis state

GET /report/{analysis_id}        → list versions (oldest-first)
GET /report/{analysis_id}/{ver}  → fetch one version

Two facts to internalise:

Records only exist on the slow path. The slow path is now always on for structured_flow (the ENABLE_SLOW_PATH flag was removed 2026-07-02), so every structured question persists a record. Reports still 409 until at least one analyze_* task has actually succeeded (chat/help/check/unstructured turns write no record).
dedorch reports stores markdown only. Structured report fields are computed at generation, rendered into rendered_markdown, and only the markdown is persisted; on read-back the structured fields come back empty.

6. Feature list (what's built)

6-intent handler router (chat/help/check/unstructured_flow/structured_flow/out_of_scope, the last added 2026-07-03) with history-aware query rewriting (EN/ID).
Skills: help (LLM, state-aware next-step guidance), check (no-LLM data/document inventory). (The problem_statement skill and the problem_validated gate were removed 2026-06-25 — KM-652; gate.py kept as a no-op seam, problem_statement.py kept but unwired.)
Slow analytical path: Planner → TaskRunner → S1a quality checkpoint (0 LLM, added 2026-07-13) → Assembler (static plan, degrade-and-continue, 3 LLM calls fixed; an all-failed run now short-circuits to a deterministic honest-failure answer — 2 calls).
Structured query engine: catalog-driven JSON IR → deterministic SQL/pandas compiler → read-only executor, with single-level FK joins (DB sources only).
Unstructured RAG over PGVector.
Analytics tools: 6 registered (5 composite analyze_* — descriptive, aggregate, correlation, trend, merge — plus render_chart, added 2026-07-13) + 4 data-access tools (check_data, check_knowledge, retrieve_data, retrieve_knowledge). Four further composites (comparison, contribution, profile, segment) exist in code but are not registered with the Planner (W3, deferred).
Charts (S2, 2026-07-13; updated 2026-07-14): planner-selected render_chart (only on an explicit chart ask, EN/ID) builds a dataeyond.chart.v1 Plotly-JSON envelope; persisted to Python-owned message_charts before done, fetched via GET /api/v1/charts?message_id= (tri-state status marker: success/empty/not_found). SSE stays text-only. Reports embed charts too (2026-07-14): the generator copies chart envelopes from results_snapshot into an ## EDA section as ```plotly fenced blocks (FE hook renders them); a successful render_chart now counts toward the report floor (has_successful_analysis).
Versioned report generation from persisted records.
Analysis sessions: data-first creation gate (≥1 bound source); each turn reads the analysis-scope catalog so it sees only that analysis's bound sources.
Langfuse tracing (PII-masked), Redis caching, pooled DB engines + speculative prewarm.

7. API surface (this repo)

✅ pr/5 restructure IN CODE (table refreshed 2026-07-13). The banner that stood here ("decided, not yet in code") is done: chat lives at /api/v2/chat/stream, the skills regrouped under /api/v1/tools/*, traceability and (2026-07-13) charts are mounted, and the analysis-CRUD / room / users / document / db_client / data_catalog routers are unwired from main + Swagger (files kept, commented mounts). Table below is the live surface (main.py mounts).

Endpoint	Purpose	Caller
`POST /api/v2/chat/stream`	Main chat SSE (`analysis_id`; router → dispatch)	FE → Go → Python
`GET /api/v1/tools/list`	Slash-command catalog (static, cacheable)	Go caches it for the FE "/" menu
`POST /api/v1/tools/help`	State-aware help skill	FE → Go → Python
`POST /api/v1/tools/report` (+ `GET …/records` · `…/readiness` · `…/{analysis_id}/{version}` GETs)	Report generate / curate / readiness / fetch	FE → Go (report button)
`GET /api/v1/traceability`	Per-turn provenance (fetched on `done`)	FE → Go → Python
`GET /api/v1/charts?message_id=`	Per-turn `render_chart` envelopes (fetched on `done`); always 200 with `status: success\|empty\|not_found` — added 2026-07-13, reshaped per lead review 2026-07-14	FE → Go → Python
`users` · `room` · `document` · `db_client` · `data_catalog` · v1 `chat` · analysis-CRUD routers	Unwired (files kept in tree, not mounted)	—

8. Data model

SQLAlchemy models in src/db/postgres/models.py. Created on startup by init_db() unless SKIP_INIT_DB=true.

Table	Shape	Written by	Read by
`users`, `rooms`, `chat_messages`, `message_sources`	base app	chat endpoint, Go	chat history
`documents`, `databases`	uploads + DB creds (Fernet-encrypted)	Go ingestion	executor cred resolution
`data_catalog` (dedorch, Go-owned)	`id` uuid, `scope_type` ('user'\|'analysis'), `user_id`, `analysis_id`, `catalog_payload` jsonb (the `Catalog`: Source → Table → Column), schema_version, generated_at, updated_at; partial-unique on `user_id WHERE scope_type='user'`	Go `catalog.Service` (all writes: DB/file ingestion)	CatalogReader → CatalogStore (read-only), planner, tools
`langchain_pg_embedding`	PGVector document chunks	Go ingestion	DocumentRetriever
`report_inputs` (was `analysis_records`)	jsonb `AnalysisRecord`, one per slow-path run; Python-owned	slow path	ReportGenerator, report readiness
`analyses` (dedorch, plural)	uuid `id`, `user_id`, `analysis_title`, `objective`, `business_questions` jsonb, `status` (active\|inactive), `data_bind`(+`data_bind_version`), `report_id`, `report_collection` — defined by Go migrations; `problem_statement`/`problem_validated`/`owner_id` already dropped there (`0003`/`0004`)	Go `/api/v1/analyses`; Python state store	gate (no-op), Help, report
`reports` (dedorch)	uuid, `analysis_id`, `user_id`, `title` + markdown `content` + `version` (UNIQUE per analysis)	Go + Python ReportStore	report API
`data_sources` (dedorch, Go-owned)	per-analysis binding table. Python no longer reads or writes it — bindings live in Go's `analyses.data_bind`, which Go materializes into the analysis-scope `data_catalog` row; Python scopes off that row. The table exists (Go migration) but Python is fully decoupled — do not drop it manually	Go migration	— (unused by Python)
`analyses_messages` (dedorch)	the analysis chat room (`role ∈ user\|ai`); replaces deprecated `rooms`/`chat_messages`	Go `/analyses/{id}/messages`	Python chat path not yet migrated here (§12)
`message_traceability` (Python-owned)	one jsonb `TraceabilityPayload` per assistant turn (PK `message_id`); flushed before `done`	chat pipeline (KM-691)	`GET /api/v1/traceability`
`message_charts` (Python-owned, added 2026-07-13)	one row per `render_chart` chart — `spec` jsonb holds the full `dataeyond.chart.v1` envelope; keyed (`analysis_id`, `message_id`), multiple rows per turn allowed; written before `done`, never-throw	slow-path chart persist (`chat_handler._run_slow_path`)	`GET /api/v1/charts`

✅ Python ORM ↔ dedorch drift — reconciled 2026-07-01. AnalysisStateRow (analyses) dropped problem_statement/problem_validated and added objective/business_questions (Harry's #3); data_catalog was the last stale model. Its Catalog ORM (old user_id-PK + data jsonb) is now the dedorch shape (id PK, scope_type, catalog_payload), and CatalogStore reads catalog_payload WHERE scope_type='user' (matching Go's catalog.Service). This closed a live bug: the check skill / CatalogReader still selected the dropped data_catalog.data column, so every catalog read 500'd after the cutover ("what data do I have" → "Sorry, I couldn't look that up: column data_catalog.data does not exist"). Python's catalog write methods (upsert/ remove_source/StructuredPipeline) were reconciled but are now legacy — Go owns ingestion.

Catalog shape (the jsonb in data_catalog): Catalog → Source[ {source_id, source_type ∈ schema|tabular|unstructured, name, location_ref} → Table[ {table_id, name, row_count, foreign_keys[]} → Column[ {column_id, name, data_type, nullable, pii_flag, sample_values|null, stats} ] ] ]. PII columns have sample_values: null so real values never enter prompts.

⚠️ dedorch catalogs ship empty foreign_keys (Go's introspection drops FK constraints), yet the IR validator only allows FK-backed joins — so every cross-table question failed validation until 2026-07-02. src/catalog/fk_inference.py (wired into CatalogStore.get) now infers the obvious <base>_id → <table>.id edges at read time: conservative (single unambiguous target, matching data_type, schema sources only) and self-disabling once any real FK is present. It's a stopgap — the durable fix is Go emitting real FKs during introspection.

QueryIR shape (src/query/ir/models.py): { source_id, table_id, joins[], select[], filters[], group_by[], order_by[], limit }. Joins are single-level equi-joins to a related table in the same source, FK-backed, DB sources only.

9. Subsystems (where the code lives)

Router — `src/agents/orchestration.py`

One structured-output LLM call (GPT-5.4-mini) → RouterDecision{intent, rewritten_query, confidence}, intent ∈ {chat, help, check, unstructured_flow, structured_flow} (problem_statement removed 2026-06-25). It's a handler classifier: structured_flow = slow path, unstructured_flow = fast RAG; the data-modality mix on the slow path is the Planner's job. Prompt: src/config/prompts/intent_router.md.

Gate — `src/agents/gate.py`

Neutered 2026-06-25 (KM-652): gate() now passes every intent through unchanged — the problem_validated redirect was removed (the goal is user-entered, no agent validation). The function + AnalysisState contract are kept as a no-op seam; the call site in chat_handler.handle is commented out. AnalysisState still carries (id, analysis_title, problem_statement, problem_validated, owner_id, report_id, created_at, updated_at) until the dedorch state migration (#3/#4) renames it.

Skills — `src/agents/handlers/`

help.py — LLM (streamed). A consistency guard derives the allowed actions from state (mirrors the gate) and feeds them to the prompt, so Help can't suggest a report when the goal isn't validated or there's nothing to report. Consumes a deterministic readiness signal.
check.py — no LLM. Keyword cues route to check_data, check_knowledge, or both (helicopter view, concurrent). Renders tool tables to markdown.
problem_statement.py — unwired 2026-06-25 (no longer routed to; file kept intact). Was an LLM drafter that validated a goal and wrote problem_validated.

Slow path — `src/agents/slow_path/` + `src/agents/planner/`

Planner (planner/service.py) — 1 LLM call → TaskList (DAG of tool-call chains). 8-check validator with re-prompt retry (max 3). BusinessContext is a stub (planner/business_context.py), which is why the slow path stays opt-in.
TaskRunner (slow_path/task_runner.py) — deterministic, 0 LLM. Wave-based execution, ${t<id>} placeholder resolution (Pattern A), never-throw invocation, degrade-and-continue (failed task → dependents skipped, independent branches run). No replanning.
Quality checkpoint (S1a) (slow_path/checkpoint.py, added 2026-07-13) — deterministic, 0 LLM, never-throw inspection between runner and assembler. CK1 all-failed → the coordinator returns a deterministic honest-failure answer (refusals.run_failure_message, EN/ID) with no assembler call and a non-substantive record; CK2 empty retrieve (+ transitive dependents), CK3 10k-cap truncation, CK4 single trend bucket, CK5 all-null column consumed, CK6 chart-spec sanity (§4.6 of SPINE_V2_PLAN). Degraded flags render as an "# Execution assessment" block in the assembler's human content; every flag logs repair_candidate via structlog (the gated-S1b evidence base). A clean run renders nothing — zero behavior change.
Assembler (slow_path/assembler.py) — 1 LLM call authoring only the narrative; code copies the structured results_snapshot / tasks_run from the run state into the AnalysisRecord (the report's source of truth).

Streaming + persistence: chat_handler._run_slow_path bridges per-stage progress to SSE status events, prewarms the DB engine in parallel with planning, emits the answer, then persists the record stamped with user_id + analysis_id, and (2026-07-13) any kind="chart" outputs to message_charts — both never-throw, both before done.

Structured query engine — `src/query/`

QueryService.run (query/service.py): plan → validate → retry(3) → dispatch → execute; never raises (errors land in QueryResult.error). IRValidator (query/ir/validator.py) checks source/table/column existence, op/agg whitelists, type compatibility, limit cap, and FK-backed joins (DB only). DbExecutor (query/executor/db.py): SqlCompiler → sqlglot SELECT-only guard → Fernet-decrypt creds (with owner check) → asyncio.to_thread (30s timeout) → pooled engine (read-only + statement_timeout) → 10k row cap. Defense-in-depth: IR validation + compiler whitelist

sqlglot guard + read-only session + LIMIT/timeout.

Analysis-scoped catalog reads — `src/catalog/reader.py::AnalysisScopedCatalogReader`

An analysis is scoped to the sources the user picked by reading the analysis-scope catalog (data_catalog scope_type='analysis', Go-materialized with the bound db + file sources under their real names). On a structured_flow turn the catalog reader is wrapped so the Planner and the tools' re-reads see the same analysis-scoped snapshot; check and the report's data-source appendix read it too. Fail-open: no analysis-scope row → user-scope catalog. The old data_sources binding table + AnalysisDataSourceStore/_ScopedCatalogReader (#10) were removed — the writer (/analysis/create) is Go-owned/unwired, so the table was always empty and its consumers fail-opened to the whole (mis-named) user catalog.

Tool layer — `src/tools/data_access.py`, `src/agents/planner/registry.py`

DataAccessToolInvoker implements the never-throw tool seam for the 4 data-access tools. retrieve_data runs a pre-built IR (validate → dispatch → execute, skipping the planner) and coerces Decimal→float — the Pattern A handoff the analyze_* tools consume. The planner registry composes a local data-access spec stub (name-checked against DATA_ACCESS_TOOLS) with the real analytics_registry(). 2026-07-13: analytics_registry() also exposes render_chart (src/tools/analytics/visualization.py, category analytics.visualization, output_kind="chart" — ToolOutput.kind gained "chart"): a pure spec builder mapping a table to a Plotly-JSON envelope (bar/line/pie/scatter, fixed house style preset, no plotly import); the planner validator's Check 10 forces its data to reference a table-producing task.

Report — `src/agents/report/`

generator.py reads records, deterministically assembles structured fields, 1 LLM call for the executive summary; store.py versions under an advisory lock and persists markdown to dedorch reports; readiness.py defines the report floor (≥1 successful analyze_* or, since 2026-07-14, render_chart — a chart-only session is substantive; the problem_validated precondition was dropped 2026-06-25) shared by the report API and the Help readiness signal so the two can't disagree. 2026-07-14: the report embeds charts — _collect_charts copies dataeyond.chart.v1 envelopes verbatim (INV-4) from results_snapshot into AnalysisReport.charts, rendered as ```plotly fenced blocks in the ## EDA section (fence content = the full v1 envelope, pretty-printed — the shape the FE's fence hook parses, verified 2026-07-14).

Observability — Langfuse

The endpoint's ChatHandler runs with enable_tracing=True. One trace per request groups router/planner/assembler/chatbot + tool spans. PII policy: router/planner unmasked (PII-safe summaries); assembler/chatbot masked (see real rows); tool spans carry name + arg keys + row counts only.

10. Feature flags

Flag	Where	Default	Effect
~~`ENABLE_SLOW_PATH`~~	—	removed 2026-07-02	Flag deleted. `structured_flow` now always runs Planner/TaskRunner/Assembler (the single-query `QueryService` fast path was retired from the chat handler), so records always persist. `extra="allow"` ignores a stale `ENABLE_SLOW_PATH` left in any `.env`.
`ENABLE_GATE`	`settings.enable_gate`	off	Deprecated 2026-06-25 — gate neutered; the flag has no effect. Kept to avoid `.env` churn.
`SKIP_INIT_DB`	`settings.skip_init_db` (.env/env)	on	Skip `init_db()` on startup — the dedorch cutover switch. Defaults TRUE (Go owns the dedorch schema); set `false` only for a local Python-owned DB.
`enable_tracing`	hardcoded `True` in `chat.py`	on (endpoint)	Langfuse tracing.

11. Where the older docs are stale

Trust the code. The original Phase-2 docs (ARCHITECTURE.md, REPO_CONTEXT.md) and the Go repo's copies disagree with the current code on:

Topic	Old docs	Current code
Router	3-way `source_hint` (chat/unstructured/structured)	Flat 5-intent `RouterDecision` (was 6; `problem_statement` removed 2026-06-25)
Joins in IR	"single-table only; deferred"	Single-level FK-backed joins (DB sources only)
Analysis / report / gate / slow path	"Phase 2 spine only"	All built and present
`analysis_id`	open question	resolved: `analysis_id == room_id`
Report source	(newer invariant) "from records, never chat history"	confirmed: generator reads `AnalysisRecord`s
Go service scope	"interview agent + ingestion; dedorch migrations live outside the repos"	Go now hosts the dedorch migrations in-repo + a full `/api/v1/analyses` + `/api/v1/skills` REST surface; interview/rooms deprecated (410). (Go's own `PROJECT_SUMMARY.md`/`REPO_CONTEXT.md` are uncommitted + stale.)

12. dedorch migration — current state

The Python DB has moved from dataeyond → dedorch (cutover 2026-07-01; Go owns dedorch migrations; Python is consumer-only). State re-verified against the Go source 2026-06-29:

The dedorch migrations now live IN the Go repo — embedded SQL at internal/repository/postgres/migrations/0001_create_core_schema.sql … 0004_replace_chat_with_analysis_scope.sql, run on startup by RunMigrations. (This corrects the earlier note that the migrations were invisible / asserted only by Python docstrings.) The full schema is now readable there.
Go owns the analysis family end-to-end. analyses / analyses_messages / reports / data_sources / message_sources / data_catalog are created by Go migrations and served by a full REST surface: internal/api/analysis.go (CRUD + data-bind w/ optimistic expected_version
- messages + reports) and internal/api/skills.go. analyses already has the pivot shape (objective + business_questions, status, data_bind/_version, report_collection) and has dropped problem_statement/problem_validated/owner_id. Migration 0004 renames the legacy rooms/chat_messages/interview_* tables to zdeprecated_*.
report_inputs (the slow-path structured output, formerly analysis_records) stays Python-owned; its finalized schema goes to Harry so the dedorch migration creates it post-cutover. Same pattern for message_traceability (created manually 2026-07-06) and message_charts (created manually 2026-07-13, DDL in SPINE_V2_PLAN.md §4.4; live e2e verified same day — Harry's migration handoff for both is still the open item).
Connection-string cutover DONE (2026-07-01). Python's postgres_connstring now points at dedorch and reads the Go-migrated tables directly. Every ORM model Python reads (analyses, analyses_messages, data_catalog) has been reconciled to its dedorch shape. init_db() is now skipped by default (settings.skip_init_db defaults True): its privileged DDL (ALTER TABLE rooms …, index creation) fails on Go-owned tables (InsufficientPrivilegeError: must be owner of table rooms). Skipping is safe — Go migration 0001 already provides the vector extension + the langchain FTS index. Set SKIP_INIT_DB=false (.env or env) only for a local Python-owned DB. report_inputs is not in any Go migration yet (#22) — create it in dedorch before enabling the slow path, else report/slow-path writes fail (chat path unaffected).

⚠️ Integration gap (verified — the big one). Go's /api/v1/analyses and /api/v1/skills (help / report) are placeholders that return dummy data — the SendMessage / GenerateReport handlers and the skills handler explicitly note "placeholder integrasi backend agentic … will be replaced by the external skills service." Go currently never calls Python's /chat/stream, /report, or any skill (no outbound HTTP to the agentic service exists in the Go source). So today there are two parallel, unconnected analysis stacks: Go's self-contained placeholder lifecycle (gate: ≥3 user messages; AI replies are canned) and Python's real agentic spine (router → slow path → records-based report; floor: ≥1 analyze_* success). Wiring Go → Python is the open integration work (DEV_PLAN #7/#18/#25), plus reconciling the two different report gates.

13. Conventions & gotchas

Two Postgres engines: app engine + a separate PGVector engine (prepared_statement_cache_size=0) because PGVector emits multi-statement strings asyncpg rejects.
Identifiers vs values: identifiers come from the catalog and are inlined as quoted; filter values are always parameterized.
Settings aliases: .env uses double-underscore names (azureai__api_key__54m); Settings exposes them as azureai_api_key_54m.
LLM env quad renamed __4o → __54m (2026-07-14). The generation LLM is GPT-5.4-mini; the four vars are azureai__api_key__54m, azureai__endpoint__url__54m, azureai__deployment__name__54m, azureai__api__version__54m. Hard rename — no __4o fallback, so an environment missing the new names fails loudly instead of silently serving a different model (which is exactly how HF drifted onto 4o while local ran 5.4-mini).
Shared Fernet key across repos (gotcha). User DB credentials in databases are written + encrypted by Go and decrypted by Python; both read the same env var dataeyond__db__credential__key (Go: configs/app.yaml → credentials.fernet_key). The two deployments MUST hold the identical value or Python's decrypt throws cryptography.fernet.InvalidToken — whose str() is empty, so it logged as error="" and masqueraded as a DB-connection failure (the executor now logs repr(e) to expose it). Tell-apart: a valid-but-wrong key → InvalidToken; a malformed key → a non-empty ValueError at cipher build.
Storage-provider parity with Go (gotcha, found 2026-07-13). Go's data plane uploads tabular parquet to Supabase S3 and writes location_ref: object_storage://…; Python's TabularExecutor picks its download backend from settings.storage_provider (azure_blob | supabase_s3, blank → Azure legacy). If the .env still says azure_blob, every tabular retrieve_data fails with an Azure BlobNotFound — and the never-throw path degrades it into an honest "data not available" answer, so it masquerades as a data problem. Tell-apart: BlobNotFound + location_ref starting object_storage:// ⇒ env gap; set storage_provider=supabase_s3 + the five supabase_s3_* values (match Go's data plane).
Never-throw seams are pervasive (tool invoker, query service, executors, state/catalog reads, record persistence, report summary). Failures degrade into soft output rather than raising — good for UX, but they can mask real breakage (e.g. a missing analysis-scope catalog silently falling back to the whole user catalog).
Prompts live in src/config/prompts/*.md. chatbot_system.md has guardrails.md appended so guardrails win on conflict.
Tests are gitignored (team decision) — run them locally.