[NOTICKET] docs: add API_ENDPOINTS.md (FE surface for Go) + update DEV_PLAN tracker

- API_ENDPOINTS.md (repo root): the 4 FE-callable surfaces (call_agent,
list_skills, help, report) with request/response examples (chat SSE
transcript, report 201/409 JSON, version list), schemas, and a full
32-route inventory + task-8 reading.
- DEV_PLAN: #10 (API doc) and #12 (PR) marked done; tracker reflects KM-678.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

API_ENDPOINTS.md

@@ -0,0 +1,373 @@
+# Data Eyond — Python Agentic Service: FE-Callable API (for Go integration)
+
+**Audience:** Harry (Go gateway) wiring the FE → Go → Python surface.
+**Scope:** the **4 FE-callable surfaces** the Python service exposes after the 2026-06-24 pivot
+(DEV_PLAN decision #6). Everything else under `/api/v1` is internal / Phase-1 legacy / Go-owned —
+see [§7](#7-not-fe-facing) and the full inventory in [§9](#9-appendix--complete-endpoint-inventory-all-registered-routes).
+**Branch:** `pr/4` · **Snapshot:** 2026-06-25 · **Companion:** [REPO_STATUS.md](REPO_STATUS.md).
+
+> Request flow is **FE → Go → Python**. The FE never calls Python directly except for chat
+> streaming. Auth/JWT is terminated at the Go gateway; Python receives `user_id` / `room_id` as
+> **trusted inputs** and does no auth of its own.
+
+---
+
+## 1. The 4 FE-callable surfaces
+
+| # | Logical name | HTTP | How it's invoked |
+|---|---|---|---|
+| 1 | **`call_agent`** | `POST /api/v1/chat/stream` | The one streaming chat call. Router classifies + dispatches. |
+| 2 | **`list_skills`** | `GET /api/v1/tools` | Static slash-command catalog for the FE "/" menu. Cacheable. |
+| 3 | **skill: `help`** | *(via `call_agent`)* | **No dedicated endpoint** — the router resolves it to the `help` intent inside `/chat/stream`. |
+| 4 | **skill: `report`** | `POST /api/v1/report` (+ 2 `GET`s) | Dedicated REST API. **Not** through `/chat/stream`. |
+
+**Key consequence for Go:** the two catalog skills are invoked **differently**. `/help` goes through
+`/chat/stream`; `/report` is a direct REST call to the Report API. The catalog's `name` field is the
+internal route key (`help` = router intent; `report` = the Report API), not a uniform dispatch key.
+
+**Conventions:**
+- Base path: `/api/v1`.
+- **`room_id == analysis_id`** — one chat room == one analysis session (#9). Callers pass `room_id`
+  to chat; it *is* the `analysis_id` used by the report API.
+- Streaming uses **SSE** (`text/event-stream`, `sse-starlette`).
+
+---
+
+## 2. `call_agent` — `POST /api/v1/chat/stream`
+
+The only FE→Python call in normal operation. Source: [chat.py:169](src/api/v1/chat.py:169).
+
+**Request body** (`application/json`) — `ChatRequest`:
+
+```json
+{
+  "user_id": "u_1a2b3c",
+  "room_id": "room_42",
+  "message": "What were total sales by region last quarter?"
+}
+```
+
+`room_id` is the analysis session id. No auth header (handled by Go).
+
+**Response:** `text/event-stream`. Events arrive in this order:
+
+| `event:` | `data:` payload | Notes |
+|---|---|---|
+| `sources` | JSON array of source refs | `{document_id, filename, page_label}`. Structured: one per executed table (`document_id = "{user_id}_{table}"`, `page_label = null`). Unstructured: deduped doc/page. `chat`/`help`/`error`: `[]`. |
+| `status` | text | **Slow-path only** — progress pings ("Planning…", "Running N steps…"). Keeps the SSE alive; safe to surface or ignore. |
+| `chunk` | text fragment | Concatenate in order to form the answer. |
+| `done` | *(empty)* | End of stream. |
+| `error` | text | Terminal error; stream stops after this. |
+
+> The handler also emits an internal `intent` event — it is **consumed inside Python** (gates
+> caching) and **not forwarded** to the client. Go/FE will never see it.
+
+**Example — `structured_flow` answer** (raw SSE wire; blank line separates events). Source shape:
+[chat_handler.py:607](src/agents/chat_handler.py:607).
+
+```
+event: sources
+data: [{"document_id":"u_1a2b3c_orders","filename":"orders","page_label":null}]
+
+event: status
+data: Planning analysis…
+
+event: status
+data: Running 3 steps…
+
+event: chunk
+data: Total sales by region last quarter: 
+
+event: chunk
+data: Central led at $1.21M (38%), East $0.74M, West $0.55M (down 12% QoQ).
+
+event: done
+data: 
+```
+
+**Example — simple `chat` reply** (no status pings, empty sources):
+
+```
+event: sources
+data: []
+
+event: chunk
+data: I'm your AI data analyst — connect a source or ask a question to get started.
+
+event: done
+data: 
+```
+
+**Behavior worth knowing for integration:**
+- **Redis response cache** (1h TTL) is applied to the stateless `chat` intent only; cached replies
+  replay as `sources`/`chunk`/`done`.
+- **Greeting/farewell fast-path** returns a canned reply with no LLM call.
+- The LLM **router** classifies every message into one of **5 intents** —
+  `chat` · `help` · `check` · `unstructured_flow` · `structured_flow` — and dispatches. Messages
+  persist (user + assistant) on `done`.
+
+---
+
+## 3. `list_skills` — `GET /api/v1/tools`
+
+Static, deterministic, **safe for Go to cache**. Source: [tools.py:133](src/api/v1/tools.py:133).
+
+**Request:** none (no params, no body).
+
+**Response** `200` (`ListToolsResponse`):
+
+```json
+{
+  "count": 2,
+  "tools": [
+    { "command": "/help",   "name": "help",   "type": "skill",
+      "description": "Show what the assistant can do and guide your next step." },
+    { "command": "/report", "name": "report", "type": "skill",
+      "description": "Generate a versioned analysis report (background, EDA, key findings, insights)." }
+  ]
+}
+```
+
+`CommandResponse` = `{ command, name, type, description }`, `type ∈ {skill, analytics, data_access}`.
+Post-KM-678 the catalog is **`/help` + `/report` only**; the `analyze_*`, `check_*`, `retrieve_*`
+and retired `/problem-statement` entries are commented out (kept for restorability), not deleted.
+
+---
+
+## 4. skill: `help` — via `call_agent`
+
+**There is no `/help` endpoint.** The FE "/" menu surfaces `/help`; to invoke it, call
+`POST /api/v1/chat/stream` and let the router classify the message as the `help` intent
+([chat_handler.py:363](src/agents/chat_handler.py:363)). Help streams `chunk` events (same SSE
+shape as §2, with `sources: []` and no `status` pings) — a state-aware, next-step guidance reply.
+
+```
+event: sources
+data: []
+
+event: chunk
+data: Your goal is set — you can start exploring now. Try a question like "average order value by month", then I can generate a report.
+
+event: done
+data: 
+```
+
+> **Open integration question (for Harry):** the Python `/chat/stream` contract has **no
+> forced-intent / slash-bypass param** — `handle()` always routes via the LLM classifier. So
+> deterministic `/help` dispatch depends on either (a) Go forwarding the literal slash text and
+> trusting the router to classify it as `help`, or (b) adding a forced-intent input to the chat
+> contract. The `tools.py` docstring's "slash invocation bypasses the router to the tool directly"
+> is **not yet true on the Python side.** Needs a decision. (DEV_PLAN #8/#18.)
+
+---
+
+## 5. skill: `report` — Report API
+
+Dedicated REST surface (the "Generate Report" button), **not** a chat route.
+Source: [report.py](src/api/v1/report.py).
+
+### `POST /api/v1/report`
+Generate, persist, and return a new report **version**.
+
+**Query params:** `analysis_id` (required), `user_id` (required). No request body.
+
+```
+POST /api/v1/report?analysis_id=room_42&user_id=u_1a2b3c
+```
+
+| Status | Meaning |
+|---|---|
+| `201` | New version generated → `AnalysisReport` body. |
+| `409` | Floor not met — **no recorded analyses yet** for this session, nothing to report. |
+| `500` | Generation or persistence failed. |
+
+**`201` response** (`AnalysisReport`):
+
+```json
+{
+  "report_id": "8f3a2b1c9d4e4f6a8b0c1d2e3f4a5b6c",
+  "analysis_id": "room_42",
+  "user_id": "u_1a2b3c",
+  "version": 2,
+  "generated_at": "2026-06-25T09:14:33.512Z",
+  "problem_statement": {
+    "objective": "Understand which regions drive revenue and why Q1 dipped.",
+    "business_questions": [
+      "Which regions contribute most to total revenue?",
+      "Did any region decline quarter-over-quarter?"
+    ]
+  },
+  "record_ids": ["rec_a1", "rec_b2"],
+  "executive_summary": "Revenue is concentrated in the Central region (38% of total). The West was the only region to contract, down 12% QoQ — the main driver of the Q1 dip.",
+  "findings": [
+    { "text": "Central region contributed 38% of total revenue, the largest share.",
+      "record_ids": ["rec_a1"], "supporting_data": null },
+    { "text": "West region revenue fell 12% quarter-over-quarter.",
+      "record_ids": ["rec_b2"], "supporting_data": null }
+  ],
+  "caveats": [
+    { "text": "March data for the East region was partially missing (~6% of rows).",
+      "record_ids": ["rec_b2"] }
+  ],
+  "open_questions": [
+    { "text": "What drove the West region's QoQ decline?", "record_ids": ["rec_b2"] }
+  ],
+  "data_sources": [
+    { "source_id": "src_sales_db", "name": "orders", "source_type": "postgres",
+      "detail": { "tables": ["orders"], "row_count": 48213,
+                  "columns": ["region", "amount", "ordered_at"] } }
+  ],
+  "method_steps": [
+    { "task_id": "t1", "stage": "data_understanding", "objective": "Inventory the sales source",
+      "status": "success", "tools_used": ["check_data"] },
+    { "task_id": "t2", "stage": "modeling", "objective": "Aggregate revenue by region",
+      "status": "success", "tools_used": ["analyze_aggregate"] }
+  ],
+  "rendered_markdown": "# Analysis Report\n\n*Generated 2026-06-25 by u_1a2b3c · 2 analyses · 1 source(s)*\n\n## Objective\nUnderstand which regions drive revenue…\n\n## Key Findings\n1. Central region contributed 38%…"
+}
+```
+
+**`409` response** (floor not met — the demo's most common error):
+
+```json
+{ "detail": "Not ready to generate a report — still needs at least one completed analysis." }
+```
+
+> ⚠️ **Demo/integration precondition:** `AnalysisRecord`s persist **only on the slow path**, so
+> reports require **`enable_slow_path=true`** on the Python deployment *and* ≥1 prior
+> `structured_flow` question in the session. With slow path off, `POST /report` **409s by design**,
+> not a bug. (DEV_PLAN #15/#16.)
+
+### `GET /api/v1/report/{analysis_id}`
+List a session's report versions (oldest-first). Returns `[ReportVersionEntry]`; `[]` if none.
+
+```json
+[
+  { "report_id": "1b2c3d4e…", "version": 1, "generated_at": "2026-06-24T15:02:11Z", "record_count": 1 },
+  { "report_id": "8f3a2b1c…", "version": 2, "generated_at": "2026-06-25T09:14:33Z", "record_count": 2 }
+]
+```
+
+### `GET /api/v1/report/{analysis_id}/{version}`
+Fetch one version → `AnalysisReport` (same shape as the `POST` 201 body above); `404` if that
+version doesn't exist.
+
+```json
+{ "detail": "No report v3 for analysis 'room_42'." }
+```
+
+---
+
+## 6. Schemas
+
+**`AnalysisReport`** (POST + GET-version body):
+
+| Field | Type | Notes |
+|---|---|---|
+| `report_id` | str | |
+| `analysis_id` | str | == `room_id` |
+| `user_id` | str \| null | |
+| `version` | int | monotonic V1, V2, … |
+| `generated_at` | datetime | ISO 8601, UTC |
+| `problem_statement` | `{ objective: str, business_questions: string[] }` | the frozen goal snapshot (new pivot shape) |
+| `record_ids` | string[] | records the version was built from |
+| `executive_summary` | str | the **only** LLM-authored field |
+| `findings` | `ReportFinding[]` | `{ text, record_ids[], supporting_data? }` |
+| `caveats` | `AttributedNote[]` | `{ text, record_ids[] }` |
+| `open_questions` | `AttributedNote[]` | `{ text, record_ids[] }` |
+| `data_sources` | `DataSourceRef[]` | `{ source_id, name, source_type, detail }` |
+| `method_steps` | `TaskSummary[]` | `{ task_id, stage, objective, status, tools_used[] }`; `stage` ∈ CRISP-DM phases |
+| `rendered_markdown` | str | the full rendered report |
+
+> **Persistence caveat:** dedorch `reports` stores **markdown only**. On read-back via the `GET`
+> endpoints, the structured fields above come back **empty** and `rendered_markdown` is the source of
+> truth. (REPO_STATUS §5.)
+
+**`ReportVersionEntry`** (GET-list rows): `{ report_id, version, generated_at, record_count }`.
+
+---
+
+## 7. Not FE-facing
+
+Registered under `/api/v1` but **not** part of the FE→Python surface — do not wire these from the FE:
+
+- **Analysis CRUD** — `POST /analysis/create`, `GET /analysis`, `GET /analysis/{id}`. Intended to
+  move behind Go (state writes via Go, per decision #5/#18). Router still **mounted** (Go may use it);
+  the FE should not call it.
+- **`check_data` / `check_knowledge`** — served by **Go**, not surfaced as Python FE endpoints.
+- **Chat cache management** — `DELETE /chat/cache`, `/chat/cache/room/{id}`, `/retrieval/cache/{user_id}`
+  (ops/internal).
+- **Phase-1 legacy routers** — `users`, `room`, `document`, `db_client`, `data_catalog`
+  (functionally migrated to Go; mostly dormant).
+- **Health/root** — `GET /`, `GET /health` (liveness only).
+
+---
+
+## 8. Open items affecting this contract
+
+1. **`/help` dispatch mechanism** — router-classify vs. forced-intent param (§4). *(DEV_PLAN #8/#18)*
+2. **`/report` needs `enable_slow_path=true`** + a prior `structured_flow` question, else 409.
+   *(DEV_PLAN #15)*
+3. **`analysis_records` home** post-`SKIP_INIT_DB` cutover — the report API depends on this table
+   existing. *(DEV_PLAN #14/#16)*
+4. **Analysis-state writes** — once Go owns creation + state writes, Python's per-turn state
+   `ensure` becomes a read-only get (Go must guarantee the row exists before any chat turn).
+   *(DEV_PLAN #18)*
+
+---
+
+## 9. Appendix — complete endpoint inventory (all registered routes)
+
+Every route mounted in [main.py](main.py), so task #8 can be decided against the full picture.
+**32 routes** across 9 routers + 2 app-level. Status legend:
+**✅ FE-callable** (one of the 4 surfaces — keep) · **✂️ comment out** (task #8 target) ·
+**🟦 legacy → Go** (Phase-1, functionally migrated; not FE→Python; mostly dormant) ·
+**⚙️ internal/ops**.
+
+| Method | Path | Purpose | Router | Status |
+|---|---|---|---|---|
+| POST | `/api/v1/chat/stream` | Main chat SSE — **`call_agent`**; carries chat/help/check/structured/unstructured intents | Chat | ✅ FE-callable (#1, +help #3) |
+| GET | `/api/v1/tools` | Slash-command catalog — **`list_skills`** (Go caches) | Tools | ✅ FE-callable (#2) |
+| POST | `/api/v1/report` | Generate a report version | Report | ✅ FE-callable (#4) |
+| GET | `/api/v1/report/{analysis_id}` | List report versions | Report | ✅ FE-callable (#4) |
+| GET | `/api/v1/report/{analysis_id}/{version}` | Fetch one report version | Report | ✅ FE-callable (#4) |
+| POST | `/api/v1/analysis/create` | Create session (state + room + bindings) | Analysis | ✂️ comment (#8 → Go) |
+| GET | `/api/v1/analysis` | List a user's analyses | Analysis | ✂️ comment (#8) |
+| GET | `/api/v1/analysis/{analysis_id}` | Get one session's state + sources | Analysis | ✂️ comment (#8) |
+| DELETE | `/api/v1/chat/cache` | Clear one cached reply | Chat | ⚙️ internal/ops |
+| DELETE | `/api/v1/chat/cache/room/{room_id}` | Clear a room's cache | Chat | ⚙️ internal/ops |
+| DELETE | `/api/v1/retrieval/cache/{user_id}` | Clear a user's retrieval cache | Chat | ⚙️ internal/ops |
+| GET | `/` | Service status | (app) | ⚙️ internal/ops |
+| GET | `/health` | Liveness probe | (app) | ⚙️ internal/ops |
+| POST | `/api/login` | Login by email + password ⚠️ mounted at `/api`, **not** `/api/v1` | Users | 🟦 legacy → Go |
+| GET | `/api/v1/documents/doctypes` | Supported document types | Documents | 🟦 legacy → Go |
+| GET | `/api/v1/documents/{user_id}` | List a user's documents | Documents | 🟦 legacy → Go |
+| POST | `/api/v1/document/upload` | Upload a document (10/min) | Documents | 🟦 legacy → Go |
+| DELETE | `/api/v1/document/delete` | Delete a document | Documents | 🟦 legacy → Go |
+| POST | `/api/v1/document/process` | Process / ingest a document | Documents | 🟦 legacy → Go |
+| GET | `/api/v1/rooms/{user_id}` | List a user's rooms | Rooms | 🟦 legacy → Go |
+| GET | `/api/v1/room/{room_id}` | Get one room | Rooms | 🟦 legacy → Go |
+| DELETE | `/api/v1/room/{room_id}` | Delete a room | Rooms | 🟦 legacy → Go |
+| POST | `/api/v1/room/create` | Create a room | Rooms | 🟦 legacy → Go |
+| GET | `/api/v1/data-catalog/{user_id}` | List catalog index | Data Catalog | 🟦 legacy → Go |
+| POST | `/api/v1/data-catalog/rebuild` | Rebuild a user's catalog | Data Catalog | 🟦 legacy → Go |
+| GET | `/api/v1/database-clients/dbtypes` | Supported DB types | Database Clients | 🟦 legacy → Go |
+| POST | `/api/v1/database-clients` | Create a DB connection | Database Clients | 🟦 legacy → Go |
+| GET | `/api/v1/database-clients/{user_id}` | List a user's DB connections | Database Clients | 🟦 legacy → Go |
+| GET | `/api/v1/database-clients/{user_id}/{client_id}` | Get one DB connection | Database Clients | 🟦 legacy → Go |
+| PUT | `/api/v1/database-clients/{client_id}` | Update a DB connection | Database Clients | 🟦 legacy → Go |
+| DELETE | `/api/v1/database-clients/{client_id}` | Delete a DB connection | Database Clients | 🟦 legacy → Go |
+| POST | `/api/v1/database-clients/{client_id}/ingest` | Build the catalog for a DB connection | Database Clients | 🟦 legacy → Go |
+
+**Tally:** 5 ✅ FE-callable · 3 ✂️ to comment (#8) · 19 🟦 legacy→Go · 5 ⚙️ internal/ops.
+
+**Task #8 reading:**
+- **Keep exposed:** the 5 ✅ rows (`chat/stream`, `/tools`, the 3 `report` routes). `help` rides on
+  `chat/stream` — no route of its own.
+- **Comment out (the #8 to-do):** the 3 `analysis` routes — analysis CRUD moves behind Go (#5/#18).
+- **`check_data` is not an HTTP endpoint** — it's the `check` router intent (runs inside
+  `chat/stream`) plus its now-commented slash-catalog entry (KM-678); Go serves it to the FE. So
+  "comment check_data" = the catalog line (done) + don't expose a Python route (there isn't one).
+- The 19 🟦 routers (`users`, `document`, `room`, `data_catalog`, `db_client`) are Phase-1 legacy,
+  already functionally in Go (REPO_STATUS §7). They're out of the FE→Python path but **still
+  mounted** — a separate cleanup from #8's analysis-CRUD scope.

DEV_PLAN.md

@@ -91,9 +91,9 @@ Status legend: ⬜ not started · 🔄 in progress · ✅ done · ⛔ blocked ·
 | 7 | `report_id` state update via request to Go, not direct DB | Sofhia + Harry | ⬜ | Needs Go endpoint. See #18 for the rest of the writes |
 | 8 | Expose/confirm 4 FE endpoints; comment `check_data` + analysis CRUD | Sofhia | ✅ | KM-678: `list_tools` trimmed to `/help` + `/report` (analytics/check/retrieve commented in the **menu**). `help` confirmed as a `call_agent` intent — no own endpoint. Analysis CRUD endpoint left **registered**: "comment the rest" was about the FE slash menu, not killing HTTP routes Go needs |
 | 9 | Verify `analysis_id` in `call_agent` contract | Sofhia | ✅ | Verified: no separate field — carried as `room_id` (`analysis_id == room_id`), per REPO_STATUS §4/§11. Action for Go: send the id as `room_id` |
-| 10 | API endpoint doc (MD), 4 endpoints, for Go integration | Rifqi + Sofhia | ⬜ | — |
+| 10 | API endpoint doc (MD), 4 endpoints, for Go integration | Rifqi + Sofhia | ✅ | Done 2026-06-25 — `API_ENDPOINTS.md` (repo root). 4 FE surfaces with request/response **examples** (chat SSE transcript, report 201/409 JSON, version list), schemas, §9 full 32-route inventory + task-8 reading |
 | 11 | Full Python project doc (MD → PDF/Word BRD) | Rifqi | ⬜ | Reuse REPO_STATUS.md as the base |
-| 12 | Reconcile/open the `list_tools` PR cleanly (stacked commits) | Rifqi | ⬜ | — |
+| 12 | Reconcile/open the `list_tools` PR cleanly (stacked commits) | Rifqi | ✅ | N/A — we develop directly on the single active branch `pr/4` (KM-652 + KM-678 already stacked there); no separate PR to reconcile |
 | 13 | Merge HF Python build → test 4 endpoints via Swagger | Sofhia + Harry | ⬜ | Not E2E. Blocked by #15 |
 | 14 | `analysis_records` home | Rifqi + Sofhia + lead | ⬜ → required | Records-based decided ⇒ no longer conditional; see #16 |
 | 15 | Flip `ENABLE_SLOW_PATH=true` on HF + verify an `AnalysisRecord` persists from a `structured_flow` question | Rifqi | ⬜ new | Precondition for any report demo (#13) |