[NOTICKET] docs: pr/5 sprint tracker + endpoint-restructure contract + REPO_STATUS banner

DEV_PLAN $0 = pr/5 sprint table (P0 contract done; Phase 1 unwire done via KM-686 0b2d678; v2/regroup/observability pending). Header bumped to pr/5 / 2026-06-30.

Add API_ENDPOINTS_RESTRUCTURE.md: contract sent to Harry — chat -> /api/v2/chat/stream (analysis_id), skills regroup under /api/v1/tools/ (list/help/report), new /api/v1/observability (tentative).

REPO_STATUS: pr/5 direction banner (Phase 1 marked done) + 2026-06-29 cross-repo Go verification.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

API_ENDPOINTS_RESTRUCTURE.md

@@ -0,0 +1,391 @@
+# Backend Agentic Service — API Endpoint Docs (endpoint restructure)
+
+**Status:** contract draft for FE/Go integration (2026-06-30). Covers the AI-only surface after the
+restructure. Sections marked **TENTATIVE** (observability) may still change — send feedback before we
+lock them.
+
+**What changed**
+- **Only the chat pilot moves to `/api/v2`.** Everything else stays on `/api/v1`, regrouped under `/tools`.
+- **Chat pilot (`/api/v2/chat/stream`) uses `analysis_id`, not `room_id`.**
+- **Skills are grouped under `/api/v1/tools`:** `list` / `help` / `report`.
+- **New:** `GET /api/v1/observability` — Responsible-AI provenance per chat answer.
+- Python is **generative-AI only.** It never creates/updates an analysis, room, document, DB
+  client, or catalog — Go owns those. Python just receives `analysis_id`. Those v1 routers are
+  unwired from `main` + Swagger (not deleted).
+
+**Open coordination questions (need a decision with Harry) — flagged inline as ⚠️:**
+1. **`message_id` origin** — who mints the assistant turn id used to correlate stream ↔ observability? (Recommend: Go mints it, passes it in the chat request, Python echoes on `done`.)
+2. **Deterministic `/help` dispatch** — dedicated endpoint (recommended below) vs router classification.
+3. **Observability storage** — single JSONB row per message (recommended) vs 3 normalized tables.
+
+---
+
+## 1. call_agent — `POST /api/v2/chat/stream`
+
+The only FE→Python call in normal operation. Same as v1 except **`room_id` → `analysis_id`**, and
+the `done` event now carries the assistant `message_id` for observability correlation.
+
+**Request body** (`application/json`) — `ChatRequest`:
+
+```json
+{
+  "user_id": "u_1a2b3c",
+  "analysis_id": "an_42",
+  "message_id": "msg_88f1",
+  "message": "What were total sales by region last quarter?"
+}
+```
+
+- `analysis_id` is the analysis-session id (replaces `room_id`). No auth header (handled by Go).
+- ⚠️ `message_id` (optional): the assistant turn id. **Recommended: Go mints it** alongside the
+  `analyses_messages` row and passes it here, so the FE can call `/api/v1/observability?message_id=...`
+  in parallel. If omitted, Python mints one and returns it on `done`.
+
+**Response:** `text/event-stream`. Events arrive in this order:
+
+| event | data | notes |
+|---|---|---|
+| `sources` | JSON array of `{document_id, filename, page_label}` | structured: one per executed table; unstructured: deduped doc/page; chat/help/error: `[]`. |
+| `status` | text | slow-path only — progress pings ("Planning…", "Running N steps…"). Safe to surface or ignore. |
+| `chunk` | text fragment | concatenate in order to form the answer. |
+| `done` | `{"message_id": "..."}` | **v2 change:** was empty; now returns the turn id for the observability lookup. |
+| `error` | text | terminal error; stream stops after this. |
+
+The internal `intent` event is consumed inside Python (gates caching) and **not** forwarded.
+
+**Stream carries the answer text ONLY.** Planning / tool calls / sources detail are **not** in the
+stream (it would slow it down) — fetch them from `/observability` (§7), called in parallel.
+
+**Example — `structured_flow` answer** (raw SSE; blank line separates events):
+
+```
+event: sources
+data: [{"document_id":"u_1a2b3c_orders","filename":"orders","page_label":null}]
+
+event: status
+data: Planning analysis…
+
+event: status
+data: Running 3 steps…
+
+event: chunk
+data: Total sales by region last quarter: 
+
+event: chunk
+data: Central led at $1.21M (38%), East $0.74M, West $0.55M (down 12% QoQ).
+
+event: done
+data: {"message_id":"msg_88f1"}
+```
+
+**Example — simple chat reply** (no status pings, empty sources):
+
+```
+event: sources
+data: []
+
+event: chunk
+data: I'm your AI data analyst — connect a source or ask a question to get started.
+
+event: done
+data: {"message_id":"msg_12"}
+```
+
+Behavior unchanged from v1: 1h Redis response-cache on the stateless `chat` intent only;
+greeting/farewell fast-path (canned, no LLM); LLM router classifies every message into one of 5
+intents (`chat · help · check · unstructured_flow · structured_flow`); messages persist on `done`.
+
+---
+
+## 2. list_skills — `GET /api/v1/tools/list`
+
+Static, deterministic, safe for Go to cache. (Was `GET /api/v1/tools`.)
+
+**Request:** none.
+
+**Response 200** (`ListToolsResponse`):
+
+```json
+{
+  "count": 2,
+  "tools": [
+    { "command": "/help",   "name": "help",   "type": "skill",
+      "description": "Show what the assistant can do and guide your next step." },
+    { "command": "/report", "name": "report", "type": "skill",
+      "description": "Generate a versioned analysis report (background, EDA, key findings, insights)." }
+  ]
+}
+```
+
+`CommandResponse = { command, name, type, description }`, `type ∈ {skill, analytics, data_access}`.
+Catalog is `/help` + `/report` only; the `analyze_*` / `check_*` / `retrieve_*` and retired
+`/problem-statement` entries are commented out (kept for restorability), not deleted.
+
+**FE behavior:** the `/` slash menu surfaces **`/help` only**. **Report is a right-side button, not
+a slash command** (it fires only when an analysis is finished — saves tokens).
+
+---
+
+## 3. skill: help — `POST /api/v1/tools/help`
+
+⚠️ **Proposed dedicated endpoint** (new in v2). In v1 there was no `/help` endpoint — help was reached
+only by letting the LLM router classify a chat message. A dedicated endpoint makes `/help` dispatch
+**deterministic** (no risk the router mis-classifies the slash command) and gives it a clean home in
+the tools group. State-aware: reads analysis state + history to guide the next step.
+
+> Alternative if we *don't* add this endpoint: FE keeps calling `POST /chat/stream` and trusts the
+> router to classify the help intent. We recommend the dedicated endpoint — decision pending (open
+> question #2).
+
+**Request body** (`application/json`):
+
+```json
+{
+  "user_id": "u_1a2b3c",
+  "analysis_id": "an_42"
+}
+```
+
+**Response:** `text/event-stream` — same SSE shape as chat, with `sources: []` and no `status`
+pings (help never references documents). Streams a next-step guidance reply.
+
+```
+event: sources
+data: []
+
+event: chunk
+data: Your goal is set — you can start exploring now. Try a question like "average order value by month", then I can generate a report.
+
+event: done
+data: {"message_id":"msg_h7"}
+```
+
+---
+
+## 4. skill: report — `POST /api/v1/tools/report`
+
+The "Generate Report" button. Same as v1, moved under `/tools`. Generate, persist, and return a new
+report version. Currently renders **Markdown** (FE preview); PPT/PDF/infographic export is future work
+(triggered on a download button, not here).
+
+**Query params:** `analysis_id` (required), `user_id` (required). No request body.
+
+```
+POST /api/v1/tools/report?analysis_id=an_42&user_id=u_1a2b3c
+```
+
+| status | meaning |
+|---|---|
+| 201 | new version generated → `AnalysisReport` body. |
+| 409 | floor not met — no recorded analyses yet for this session, nothing to report. |
+| 500 | generation or persistence failed. |
+
+**201 response** (`AnalysisReport`):
+
+```json
+{
+  "report_id": "8f3a2b1c9d4e4f6a8b0c1d2e3f4a5b6c",
+  "analysis_id": "an_42",
+  "user_id": "u_1a2b3c",
+  "version": 2,
+  "generated_at": "2026-06-30T09:14:33.512Z",
+  "problem_statement": {
+    "objective": "Understand which regions drive revenue and why Q1 dipped.",
+    "business_questions": [
+      "Which regions contribute most to total revenue?",
+      "Did any region decline quarter-over-quarter?"
+    ]
+  },
+  "record_ids": ["rec_a1", "rec_b2"],
+  "executive_summary": "Revenue is concentrated in the Central region (38% of total). The West was the only region to contract, down 12% QoQ — the main driver of the Q1 dip.",
+  "findings": [
+    { "text": "Central region contributed 38% of total revenue, the largest share.",
+      "record_ids": ["rec_a1"], "supporting_data": null },
+    { "text": "West region revenue fell 12% quarter-over-quarter.",
+      "record_ids": ["rec_b2"], "supporting_data": null }
+  ],
+  "caveats": [
+    { "text": "March data for the East region was partially missing (~6% of rows).",
+      "record_ids": ["rec_b2"] }
+  ],
+  "open_questions": [
+    { "text": "What drove the West region's QoQ decline?", "record_ids": ["rec_b2"] }
+  ],
+  "data_sources": [
+    { "source_id": "src_sales_db", "name": "orders", "source_type": "postgres",
+      "detail": { "tables": ["orders"], "row_count": 48213,
+                  "columns": ["region", "amount", "ordered_at"] } }
+  ],
+  "method_steps": [
+    { "task_id": "t1", "stage": "data_understanding", "objective": "Inventory the sales source",
+      "status": "success", "tools_used": ["check_data"] },
+    { "task_id": "t2", "stage": "modeling", "objective": "Aggregate revenue by region",
+      "status": "success", "tools_used": ["analyze_aggregate"] }
+  ],
+  "rendered_markdown": "# Analysis Report\n\n*Generated 2026-06-30 by u_1a2b3c · 2 analyses · 1 source(s)*\n\n## Objective\nUnderstand which regions drive revenue…\n\n## Key Findings\n1. Central region contributed 38%…"
+}
+```
+
+**409 response** (floor not met — the demo's most common error):
+
+```json
+{ "detail": "Not ready to generate a report — still needs at least one completed analysis." }
+```
+
+⚠️ **Precondition:** `AnalysisRecord`s persist only on the slow path, so reports require
+`ENABLE_SLOW_PATH=true` on the Python deployment and ≥1 prior `structured_flow` question in the
+session. With slow path off, `POST` 409s by design.
+
+---
+
+## 5. report versions — `GET /api/v1/tools/report/{analysis_id}` and `/{analysis_id}/{version}`
+
+List a session's report versions (oldest-first). Returns `[ReportVersionEntry]`; `[]` if none.
+
+```json
+[
+  { "report_id": "1b2c3d4e…", "version": 1, "generated_at": "2026-06-24T15:02:11Z", "record_count": 1 },
+  { "report_id": "8f3a2b1c…", "version": 2, "generated_at": "2026-06-25T09:14:33Z", "record_count": 2 }
+]
+```
+
+`GET /api/v1/tools/report/{analysis_id}/{version}` → one `AnalysisReport` (same shape as the POST
+201 body); 404 if that version doesn't exist:
+
+```json
+{ "detail": "No report v3 for analysis 'an_42'." }
+```
+
+---
+
+## 6. Unwired in v2 (mounted in v1, OFF in v2)
+
+Commented out of `main` + Swagger, **files kept**. Go owns these; Python is generative-only:
+`POST /analysis/create` + analysis CRUD · `room` · `db_client` · `document` · `data_catalog` ·
+`users`/login. Re-mounting is a one-line `include_router` if ever needed.
+
+---
+
+## 7. observability — `GET /api/v1/observability`  **(NEW · TENTATIVE)**
+
+Responsible-AI provenance for **one chat answer**. Separate endpoint, called **in parallel with the
+stream** — never embedded in it. The FE renders it as a collapsed dropdown the user can expand
+(planning / tool calls / sources), Claude/Codex-style.
+
+**Design (recommended):** one endpoint returns one merged object, backed by **one JSONB row per
+message** written by an accumulating "scratchpad" decorator inside the chat agent and flushed on
+`done`. The 3 facets (`planning` / `tool_calls` / `sources`) are **logical sections of the JSON**,
+not separate tables — so the shape can evolve without a dedorch migration each time. (Storage is open
+question #3.)
+
+**Query params:** `analysis_id` (required), `message_id` (required).
+
+```
+GET /api/v1/observability?analysis_id=an_42&message_id=msg_88f1
+```
+
+**Timing:** the row is written when the turn finishes, so call this **after** the stream's `done`
+event (or poll until 200). "Parallel" = a separate call the FE fires alongside the stream, not data
+embedded in the stream.
+
+**Field rules (by intent):**
+- `planning` — present **only when the planner ran** (slow path); `null` otherwise.
+- `tool_calls` — every tool invoked, with input + output. `[]` for pure chat / greeting / help.
+- `sources` — **required for retrieve flows** (`structured_flow`, `unstructured_flow`). **Empty for
+  greeting / `chat` / `help`** (they don't reference documents).
+- `thinking` — optional reasoning text; `null` if none.
+
+**200 response — full `structured_flow` turn** (planner ran → all sections present):
+
+```json
+{
+  "analysis_id": "an_42",
+  "message_id": "msg_88f1",
+  "intent": "structured_flow",
+  "generated_at": "2026-06-30T03:21:09.114Z",
+  "planning": {
+    "goal_restated": "Find which regions drive revenue and why Q1 dipped.",
+    "assumptions": ["'last quarter' = Q1 2026"],
+    "steps": [
+      { "step": 1, "stage": "data_understanding", "objective": "Inventory the sales source" },
+      { "step": 2, "stage": "modeling", "objective": "Aggregate revenue by region" }
+    ]
+  },
+  "thinking": "The question needs a per-region breakdown plus a cause, so I inventory the source, aggregate revenue by region, then compare quarters.",
+  "tool_calls": [
+    {
+      "order": 1,
+      "name": "check_data",
+      "input": { "source_hint": "structured" },
+      "output": { "kind": "table", "summary": "1 source · 1 table (orders) · 48,213 rows" },
+      "status": "success"
+    },
+    {
+      "order": 2,
+      "name": "retrieve_data",
+      "input": { "source_id": "src_sales_db", "table_id": "orders",
+                 "select": ["region", "amount"], "group_by": ["region"] },
+      "output": { "kind": "table", "columns": ["region", "total"], "row_count": 4,
+                  "preview": [["Central", 1210000], ["East", 740000]] },
+      "status": "success"
+    }
+  ],
+  "sources": [
+    {
+      "type": "database",
+      "source_id": "src_sales_db",
+      "name": "orders",
+      "query": "SELECT region, SUM(amount) AS total FROM orders GROUP BY region",
+      "detail": { "tables": ["orders"], "row_count": 48213 }
+    }
+  ]
+}
+```
+
+**200 response — `unstructured_flow` turn** (no planner; source = document, with the retrieval query):
+
+```json
+{
+  "analysis_id": "an_42",
+  "message_id": "msg_55",
+  "intent": "unstructured_flow",
+  "generated_at": "2026-06-30T03:40:02.001Z",
+  "planning": null,
+  "thinking": null,
+  "tool_calls": [
+    { "order": 1, "name": "retrieve_knowledge",
+      "input": { "query": "technology stack used in this project", "top_k": 4 },
+      "output": { "kind": "documents", "row_count": 4 }, "status": "success" }
+  ],
+  "sources": [
+    { "type": "document", "document_id": "doc_7", "filename": "tech_handbook.pdf",
+      "page_label": "12", "query": "technology stack used in this project",
+      "snippet": "The backend is built on FastAPI with async SQLAlchemy…", "score": 0.83 }
+  ]
+}
+```
+
+**200 response — simple `chat` / greeting turn** (nothing to trace):
+
+```json
+{
+  "analysis_id": "an_42",
+  "message_id": "msg_12",
+  "intent": "chat",
+  "generated_at": "2026-06-30T03:05:00.000Z",
+  "planning": null,
+  "thinking": null,
+  "tool_calls": [],
+  "sources": []
+}
+```
+
+**404** — no provenance for that message yet (turn still running or unknown id):
+
+```json
+{ "detail": "No observability for message 'msg_88f1' yet." }
+```
+
+> ⚠️ **Richness is path-dependent.** Full `planning` + tool I/O exist only when
+> `ENABLE_SLOW_PATH=true`. Fast chat / single-query / help still record `sources` + the single
+> tool call but have `planning: null`. This matches the rule "planning only when the planner runs."

DEV_PLAN.md

@@ -1,10 +1,47 @@
-# Data Eyond — Current Development Plan (post 2026-06-24 meeting + 2026-06-25 checkpoint)
+# Data Eyond — Current Development Plan (post 2026-06-24 → 2026-06-30 checkpoints)
 
 **Purpose:** context file for Claude Code sessions working on the current sprint.
-**Branch:** `pr/4` · **Snapshot:** 2026-06-25.
+**Branch:** `pr/5` · **Snapshot:** 2026-06-30.
 **Companion:** [REPO_STATUS.md](REPO_STATUS.md) describes the repo's *current built state*; this file
-describes the *in-flight plan* that changes it. New decisions from the 2026-06-25 checkpoint are in
-[§1.5](#15-2026-06-25-checkpoint-deltas).
+describes the *in-flight plan* that changes it. The **active sprint is pr/5** ([§0](#0-current-sprint--pr5-observability--endpoint-restructure)); sections §1–§6 are the prior
+2026-06-24/25 pivot (now largely ✅), kept for context.
+
+---
+
+## 0. Current sprint — pr/5: Observability + Endpoint Restructure
+
+From the **2026-06-30 checkpoint**. Direction: **Python → generation/AI-only**; Go owns the analysis
+lifecycle + data plane. Endpoint contract sent to Harry on 2026-06-30:
+[API_ENDPOINTS_RESTRUCTURE.md](API_ENDPOINTS_RESTRUCTURE.md) (chat→v2, tools regroup, observability —
+observability marked tentative). REPO_STATUS carries a matching `pr/5` direction banner.
+
+Mentor's task order: **unwire → regroup endpoints → add tools (retrieve-data + observability)**; share
+the endpoint contract *before* coding the tools. Status legend: ⬜ not started · 🔄 in progress · ✅ done ·
+⛔ blocked · 🔎 verify · ⏸️ deferred.
+
+| Phase | Task | Owner | Status | Notes |
+|---|---|---|---|---|
+| **P0 — contract** | Draft + send endpoint contract to Harry (chat v2 · tools group · observability) | Rifqi + Sofhia | ✅ | `API_ENDPOINTS_RESTRUCTURE.md` sent 2026-06-30 (before-noon deadline met). Observability section flagged tentative. |
+| **1 — unwire** | Unwire `users`(login)/`document`/`room`/`db_client`/`data_catalog`/`analysis` from `main` + Swagger | Sofhia | ✅ | **KM-686**, commit `0b2d678`. Commented, not deleted; `chat`/`report`/`tools` kept mounted. Resolves the analysis-CRUD scope Q — whole `analysis` router unwired (Go owns it). |
+| **2 — v2 + regroup** | Create `src/api/v2/` and move the chat pilot there | Rifqi/Sofhia | ⬜ | Only chat moves to v2; mirror v1 structure. |
+| **2 — v2 + regroup** | Chat: `room_id` → **`analysis_id`** (request field + handler + history) | Rifqi | ⬜ | `ChatRequest` [chat.py:51](src/api/v1/chat.py:51); `done` event returns `message_id`. Ties to #25 (`analyses_messages`). |
+| **2 — v2 + regroup** | Move report under tools → `/api/v1/tools/report` (+ version routes) | Rifqi | ⬜ | Today standalone `/api/v1/report` (still mounted). Same functionality, new path. |
+| **2 — v2 + regroup** | Move help under tools → `POST /api/v1/tools/help` (dedicated endpoint) | Sofhia | ⬜ | Today help is a chat **intent** only. Dedicated endpoint = deterministic dispatch (open Q for Harry). |
+| **2 — v2 + regroup** | Tools list → `/api/v1/tools/list` | Sofhia | ⬜ | Today `GET /api/v1/tools` ([tools.py:133](src/api/v1/tools.py:133)). |
+| **2 — v2 + regroup** | FE: slash menu = `/help` only; report = right-side button | Mentor (FE) | ⬜ | Coordination note, not Python work. |
+| **3 — tools + obs** | Finish `help` so it actually **calls** (not just lists) + test | Sofhia | ⬜ | Mentor: help currently only lists tools. Core #2 after chat. |
+| **3 — tools + obs** | Observability **scratchpad** (decorator) accumulating in the chat agent | Rifqi + Sofhia | ⬜ | Capture planning / tool I/O / sources during the run; flush one record on `done`. |
+| **3 — tools + obs** | Audit `report_inputs` — covers planning + tool I/O + source? add cols / new store | Rifqi | ⬜ | **Rec:** dedicated provenance store = 1 JSONB row per message (logical 3 sections); keep Langfuse for engineering. |
+| **3 — tools + obs** | Build `GET /api/v1/observability` (one merged response) | Rifqi | ⬜ | Intent-based source rules (greeting/help = none; retrieve = required). Richness path-dependent (full planning only on slow path). |
+| **3 — tools + obs** | Keep stream **text-only**; observability is a separate parallel call | Rifqi | ⬜ | Per mentor — don't slow the stream. |
+| **3 — tools + obs** | Resolve `message_id` correlation (stream ↔ observability) with Harry | Rifqi ↔ Harry | ⬜ | **Rec:** Go mints `message_id`, passes in chat request, Python echoes on `done`. |
+| **4 — biz questions** | Get Go folder; confirm `business_questions` in create-analysis (max 5); sync Python | Harry/Mentor → Rifqi | ⬜ | Go currently missing the field ("lagi difixing"). Python already models objective + business_questions. |
+| **deferred** | Report formats: PPT (preferred) / PDF / infographic on download | — | ⏸️ | MD is fine for the FE preview stage now. |
+| **deferred** | Charts (Plotly→JSON) + images tables | — | ⏸️ | Carried from §4 #26/#27. |
+
+**Next up:** Phase 2 — stand up `src/api/v2/` for the chat pilot (`analysis_id`) and regroup
+`help`/`report`/`list` under `/api/v1/tools/`. The observability scratchpad (Phase 3) can start in
+parallel since its shape is already speced in the contract.
 
 ---
 

REPO_STATUS.md

@@ -2,13 +2,27 @@
 
 **Audience:** teammates onboarding onto the Python repo (`Agentic-Service-Data-Eyond-Catalog`).
 **Scope:** what the code does **right now** (branch `pr/4`, ticket KM-652). Describes current state only — no roadmap or to-dos.
-**Snapshot date:** 2026-06-25.
+**Snapshot date:** 2026-06-25. **Cross-repo update 2026-06-29:** §2/§8/§11/§12 re-verified against
+the **Go source** (`Orchestrator-Agent-Service`), not its docs. The Go service has moved well past its
+own (uncommitted, stale) design docs: it now hosts the **dedorch SQL migrations** in-repo and a full
+**`/api/v1/analyses` + `/api/v1/skills`** REST surface. Go does **not** call Python yet — those skills
+are placeholders (see §12).
 
 > This file is grounded in the source, not the older design docs. Where the two
 > disagree, the code wins — see [§11 Doc-vs-code](#11-where-the-older-docs-are-stale).
 > `REPO_CONTEXT.md` / `ARCHITECTURE.md` are the original Phase-2 design docs and are
 > stale on the router, joins, and the analysis/report stack.
 
+> 🚧 **Direction update 2026-06-30 (pr/5 — DECIDED · IN PROGRESS).** The 30 June checkpoint locked a
+> restructure (contract: [API_ENDPOINTS_RESTRUCTURE.md](API_ENDPOINTS_RESTRUCTURE.md); live tracker:
+> [DEV_PLAN §0](DEV_PLAN.md)). **Python is becoming a generation/AI-only service** — Go owns the full
+> analysis lifecycle *and* the data-plane endpoints. Scope:
+> - **Unwired from `main` + Swagger** (router files kept, *not* deleted): `analysis` CRUD, `room`, `db_client`, `document`, `data_catalog`, `users`/login. **✅ DONE — KM-686, commit `0b2d678`** (so the §7 rows for these are now commented out of `main.py`).
+> - **AI surface that stays live:** `chat` → **`POST /api/v2/chat/stream`** (explicit **`analysis_id`**, not `room_id`); the skills regroup under **`/api/v1/tools/`** (`list` · `help` · `report`); plus a **new `GET /api/v1/observability`** (Responsible-AI provenance per answer, backed by a provenance store — shape TBD). **⬜ pending.**
+> - **Only `chat/stream` moves to `/api/v2`;** everything else stays `/api/v1`.
+>
+> §2/§4/§7 below still describe the **pre-restructure wiring** except the unwire above, which has landed.
+
 ---
 
 ## 1. The product in one paragraph
@@ -31,9 +45,13 @@ streaming.
 | Repo | Role | We edit? |
 |---|---|---|
 | **Python** — `Agentic-Service-Data-Eyond-Catalog` (this repo) | The agentic LLM service: router, gate, skills, slow analytical path, structured query engine, unstructured RAG, report generation, analysis-session state. FastAPI + async SQLAlchemy + LangChain + Azure GPT-4o. | **Yes — the only repo we edit.** |
-| **Go** — `Orchestrator-Agent-Service` | Gateway / data plane: interview agent, auth/JWT, rooms, documents (Azure Blob + CSV/XLSX→Parquet + embeddings), database_clients (Fernet creds), catalog ingestion, **all DB migrations**. | Reference only. |
+| **Go** — `Orchestrator-Agent-Service` | Gateway / data plane: auth/JWT, documents (Azure Blob + CSV/XLSX→Parquet + embeddings), database_clients (Fernet creds), **catalog ingestion** (moved into Go, KM-578/590), **all dedorch SQL migrations** (now embedded in the Go repo: `internal/repository/postgres/migrations/0001–0004`), and the **full analysis-lifecycle REST surface** (`/api/v1/analyses` CRUD + messages + reports, `/api/v1/skills`). The **interview agent and chat-rooms are deprecated → HTTP 410** (`internal/api/deprecation.go`). | Reference only. |
 | **FE** — `E2E-Frontend-Data-Eyond` | React/Vite SPA. Talks to Go for everything and to Python only for chat streaming. | Reference only. |
 
+> **» pr/5 (decided, not yet in code):** Python's non-AI endpoints (analysis CRUD, `room`, `document`,
+> `db_client`, `data_catalog`, `users`/login) are being **unwired** — Python keeps only the
+> generation/AI surface (chat, tools: `help`/`report`/`list`, observability). See the Direction-update banner.
+
 Shared infra: **Postgres** (app tables + `data_catalog` jsonb + PGVector `langchain_pg_embedding`), **Azure Blob**, and (Python-only) **Redis**.
 
 ---
@@ -59,6 +77,10 @@ Tests live locally and are gitignored. Run with `./.venv/Scripts/python.exe -m p
 Entry: `POST /api/v1/chat/stream` (`src/api/v1/chat.py`) → `ChatHandler.handle(...)`
 (`src/agents/chat_handler.py`). One shared `ChatHandler` per process keeps the Azure clients warm.
 
+> **» pr/5:** this endpoint moves to **`POST /api/v2/chat/stream`** with an explicit **`analysis_id`**
+> field (replacing `room_id`), and the observability detail (planning / tool I/O / sources) moves out of
+> the stream to a separate `GET /api/v1/observability` call. See the Direction-update banner.
+
 ```
 POST /chat/stream { user_id, room_id, message }
   │  (analysis_id == room_id — one session = one analysis = one chat room)
@@ -131,6 +153,11 @@ Two facts to internalise:
 
 ## 7. API surface (this repo, all under `/api/v1`)
 
+> **» pr/5 (decided, not yet in code):** chat → `/api/v2/chat/stream` (`analysis_id`); `/tools` splits
+> into `/tools/list` + `/tools/help` + `/tools/report`; new `/api/v1/observability`; and the
+> analysis-CRUD / `room` / `users` / `document` / `db_client` / `data_catalog` rows are unwired from
+> `main` + Swagger. See the Direction-update banner.
+
 | Endpoint | Purpose | Caller |
 |---|---|---|
 | `POST /chat/stream` | Main chat SSE (router → dispatch) | FE → Go → Python (the only FE→Python call today) |
@@ -153,10 +180,18 @@ unless `SKIP_INIT_DB=true`.
 | `documents`, `databases` | uploads + DB creds (Fernet-encrypted) | Go ingestion | executor cred resolution |
 | `data_catalog` | per-user jsonb `Catalog` (Source → Table → Column) | Go ingestion / Python pipeline | CatalogReader, planner, tools |
 | `langchain_pg_embedding` | PGVector document chunks | Go ingestion | DocumentRetriever |
-| `analysis_records` | jsonb `AnalysisRecord`, one per slow-path run | slow path | ReportGenerator, report readiness |
-| `analysis` *(dedorch)* | uuid id, `owner_id`, `problem_statement`, `problem_validated`, `report_id` | `/analysis/create`, state store | gate, Help, report |
-| `reports` *(dedorch)* | uuid, `title` + markdown `content` + `version` | ReportStore | report API |
-| `data_sources` *(dedorch)* | per-analysis binding; `reference_id` = catalog source_id | `/analysis/create` | structured-flow scoping, report appendix |
+| `report_inputs` *(was `analysis_records`)* | jsonb `AnalysisRecord`, one per slow-path run; **Python-owned** | slow path | ReportGenerator, report readiness |
+| `analyses` *(dedorch, plural)* | uuid `id`, `user_id`, `analysis_title`, `objective`, `business_questions` jsonb, `status` (active\|inactive), `data_bind`(+`data_bind_version`), `report_id`, `report_collection` — **defined by Go migrations**; `problem_statement`/`problem_validated`/`owner_id` already **dropped** there (`0003`/`0004`) | Go `/api/v1/analyses`; Python state store | gate (no-op), Help, report |
+| `reports` *(dedorch)* | uuid, `analysis_id`, `user_id`, `title` + markdown `content` + `version` (UNIQUE per analysis) | Go + Python ReportStore | report API |
+| `data_sources` *(dedorch)* | per-analysis binding; `reference_id` = catalog source_id; `type ∈ document\|database` | Go `/analyses/{id}/data-bind` (+ Python `/analysis/create`) | structured-flow scoping, report appendix |
+| `analyses_messages` *(dedorch)* | the analysis chat room (`role ∈ user\|ai`); replaces deprecated `rooms`/`chat_messages` | Go `/analyses/{id}/messages` | Python chat path **not yet migrated here** (§12) |
+
+> ⚠️ **Python ORM ↔ dedorch drift (verified 2026-06-29).** Python's `AnalysisStateRow` + `state_store.py`
+> still model **`problem_statement` / `problem_validated`** and do **not** carry `objective` /
+> `business_questions`, but the Go migrations have already dropped the former and added the latter.
+> Pre-cutover this is harmless (Python runs `create_all` on its own copy); **post-`SKIP_INIT_DB`**, when
+> Python reads dedorch directly, ORM column selection on the dropped columns will break. Reconcile the
+> Python model before the connection-string cutover.
 
 **Catalog shape** (the jsonb in `data_catalog`):
 `Catalog → Source[ {source_id, source_type ∈ schema|tabular|unstructured, name, location_ref} → Table[ {table_id, name, row_count, foreign_keys[]} → Column[ {column_id, name, data_type, nullable, pii_flag, sample_values|null, stats} ] ] ]`. PII columns have `sample_values: null` so real values never enter prompts.
@@ -268,25 +303,40 @@ copies disagree with the current code on:
 | Analysis / report / gate / slow path | "Phase 2 spine only" | All built and present |
 | `analysis_id` | open question | resolved: **`analysis_id == room_id`** |
 | Report source | (newer invariant) "from records, never chat history" | confirmed: generator reads `AnalysisRecord`s |
+| Go service scope | "interview agent + ingestion; dedorch migrations live outside the repos" | Go now hosts the **dedorch migrations in-repo** + a full **`/api/v1/analyses` + `/api/v1/skills`** REST surface; interview/rooms **deprecated (410)**. (Go's own `PROJECT_SUMMARY.md`/`REPO_CONTEXT.md` are uncommitted + stale.) |
 
 ---
 
 ## 12. dedorch migration — current state
 
 The Python DB is moving from `dataeyond` → **dedorch** (Go owns dedorch migrations; Python is
-consumer-only). Current state:
-
-- Base tables already match dedorch.
-- The analysis-family models have been **renamed to dedorch** on `pr/3`: `analysis` (was
-  `analysis_states`, uuid ids), `data_sources` (was `analysis_data_sources`), `reports` (was
-  `analysis_reports`, flattened to title + markdown content + version).
-- `analysis_records` (the slow-path structured output) has **no dedorch home** — it remains a
-  Python-owned jsonb table.
-- The connection-string cutover (paired with `SKIP_INIT_DB`) is a coordinated step that has not
-  happened yet; Python still creates tables on startup until then.
-
-The dedorch migrations themselves live outside the three checked-out repos (Harry owns them), so the
-dedorch table shapes are asserted by the Python model docstrings, not visible in the Go repo here.
+consumer-only). State **re-verified against the Go source 2026-06-29**:
+
+- **The dedorch migrations now live IN the Go repo** — embedded SQL at
+  `internal/repository/postgres/migrations/0001_create_core_schema.sql … 0004_replace_chat_with_analysis_scope.sql`,
+  run on startup by `RunMigrations`. (This corrects the earlier note that the migrations were
+  invisible / asserted only by Python docstrings.) The full schema is now readable there.
+- **Go owns the analysis family end-to-end.** `analyses` / `analyses_messages` / `reports` /
+  `data_sources` / `message_sources` / `data_catalog` are created by Go migrations and served by a
+  full REST surface: `internal/api/analysis.go` (CRUD + `data-bind` w/ optimistic `expected_version`
+  + messages + reports) and `internal/api/skills.go`. `analyses` already has the **pivot shape**
+  (`objective` + `business_questions`, `status`, `data_bind`/`_version`, `report_collection`) and has
+  **dropped** `problem_statement`/`problem_validated`/`owner_id`. Migration `0004` renames the legacy
+  `rooms`/`chat_messages`/`interview_*` tables to `zdeprecated_*`.
+- **`report_inputs`** (the slow-path structured output, formerly `analysis_records`) stays
+  **Python-owned**; its finalized schema goes to Harry so the dedorch migration creates it post-cutover.
+- The connection-string cutover (paired with `SKIP_INIT_DB`) **has not happened yet**; Python still
+  runs `create_all` on its own models until then.
+
+**⚠️ Integration gap (verified — the big one).** Go's `/api/v1/analyses` and `/api/v1/skills`
+(`help` / `report`) are **placeholders that return dummy data** — the `SendMessage` / `GenerateReport`
+handlers and the skills handler explicitly note *"placeholder integrasi backend agentic … will be
+replaced by the external skills service."* **Go currently never calls Python's `/chat/stream`,
+`/report`, or any skill** (no outbound HTTP to the agentic service exists in the Go source). So today
+there are **two parallel, unconnected analysis stacks**: Go's self-contained placeholder lifecycle
+(gate: ≥3 user messages; AI replies are canned) and Python's real agentic spine (router → slow path →
+records-based report; floor: ≥1 `analyze_*` success). Wiring Go → Python is the open integration work
+(DEV_PLAN #7/#18/#25), plus reconciling the two different report gates.
 
 ---