feat/ Endpoint Restructure

API_CONTRACT_BE_GOLANG.md

@@ -0,0 +1,947 @@
+# Frontend API Contract
+
+Dokumen ini merangkum endpoint Orchestration Agent Service yang dipakai oleh frontend. Fokus flow:
+
+1. User login dan menyimpan token.
+2. User menyiapkan knowledge source: upload/proses file, connect database, ingest schema, dan rebuild data catalog.
+3. Setelah knowledge siap, user membuat `new analysis` dengan judul, objective, business question, dan data source binding.
+4. Frontend mengirim pertanyaan ke AI Agent Service terpisah.
+5. Service ini hanya merekam riwayat tanya jawab ke `analyses_messages`.
+
+Base URL lokal contoh: `http://localhost:8080`
+
+## Konvensi
+
+Semua endpoint protected wajib memakai:
+
+```http
+Authorization: Bearer <access_token>
+```
+
+Endpoint public:
+
+- `GET /health`
+- `POST /api/login`
+- `POST /api/refresh`
+
+Sebagian besar response memakai envelope:
+
+```json
+{
+  "status": "success",
+  "message": "human-readable message",
+  "data": {}
+}
+```
+
+Error response:
+
+```json
+{
+  "status": "error",
+  "message": "error message",
+  "data": {
+    "code": "OPTIONAL_ERROR_CODE"
+  }
+}
+```
+
+Catatan ownership: beberapa endpoint masih menerima `user_id` di body, path, atau query untuk kompatibilitas. Nilainya wajib sama dengan user dari Bearer token.
+
+## Flow Frontend
+
+### 1. Login
+
+Frontend memanggil `POST /api/login`, lalu simpan:
+
+- `data.user.id` sebagai `user_id`
+- `data.access_token` untuk header Bearer
+- `data.refresh_token` untuk refresh token rotation
+
+`access_token` berlaku 1 jam. `refresh_token` berlaku 7 hari dan akan diganti setiap kali refresh sukses.
+
+### 2. Refresh Token
+
+Jika request protected menerima `401` karena token expired, panggil `POST /api/refresh` menggunakan refresh token terakhir. Setelah sukses, ganti access token dan refresh token lama dengan token baru dari response.
+
+Refresh token lama tidak boleh dipakai lagi setelah refresh sukses.
+
+### 3. Menyiapkan Knowledge Source
+
+Frontend dapat menyediakan knowledge source dari dokumen dan/atau database.
+
+Untuk dokumen:
+
+1. Ambil tipe file yang didukung: `GET /api/v1/documents/doctypes`
+2. Upload file: `POST /api/v1/document/upload`
+3. Proses dokumen: `POST /api/v1/document/process`
+4. Pantau status dokumen: `GET /api/v1/documents/{user_id}`
+
+Untuk database:
+
+1. Ambil tipe database dan schema form: `GET /api/v1/database-clients/dbtypes`
+2. Simpan koneksi database: `POST /api/v1/database-clients`
+3. Ingest schema database: `POST /api/v1/database-clients/{client_id}/ingest?user_id={user_id}`
+4. Pantau koneksi: `GET /api/v1/database-clients/{user_id}`
+
+Setelah dokumen/database siap, frontend dapat rebuild dan membaca user data catalog:
+
+1. `POST /api/v1/data-catalog/rebuild`
+2. `GET /api/v1/data-catalog/{user_id}`
+
+### 4. Membuat New Analysis
+
+Frontend menampilkan form:
+
+- `analysis_title`
+- `objective`
+- `business_questions`
+- `data_bind`
+
+`POST /api/v1/analyses` wajib menerima `analysis_title`, `objective`, `business_questions`, dan `data_bind`. `business_questions` berbentuk array string karena satu analysis dapat membawa lebih dari satu pertanyaan bisnis awal.
+
+Flow yang direkomendasikan:
+
+1. `POST /api/v1/analyses` dengan title, objective, business_questions, dan data_bind.
+2. Ambil `data.id` dari response sebagai `analysis_id`.
+3. Frontend memanggil AI Agent Service terpisah memakai context analysis, business_questions, dan catalog.
+4. Saat user mulai bertanya ke AI Agent Service, rekam pertanyaan dengan `role=user` ke endpoint messages.
+5. Setelah AI Agent Service menjawab, simpan jawaban dengan `role=ai` ke endpoint messages.
+
+### 5. Conversation Recording
+
+Endpoint message di service ini tidak memanggil AI agent, tidak melakukan reasoning, dan tidak membuat balasan otomatis.
+
+Frontend bertanggung jawab melakukan dua write terpisah:
+
+1. Rekam pertanyaan user:
+
+```json
+{
+  "role": "user",
+  "content": "Apa penyebab revenue turun di Q3?"
+}
+```
+
+2. Setelah AI Agent Service menjawab, rekam jawaban agent:
+
+```json
+{
+  "role": "ai",
+  "content": "Revenue Q3 turun terutama karena penurunan volume transaksi di segmen enterprise..."
+}
+```
+
+## Endpoint Ringkas
+
+| Method | Path | Kegunaan |
+| --- | --- | --- |
+| `GET` | `/health` | Health check service |
+| `POST` | `/api/login` | Login dan issue token pair |
+| `POST` | `/api/refresh` | Rotate refresh token dan issue token pair baru |
+| `GET` | `/api/v1/documents/doctypes` | List tipe dokumen yang didukung |
+| `POST` | `/api/v1/document/upload` | Upload dokumen ke Azure Blob Storage |
+| `POST` | `/api/v1/document/upload-local` | Upload dokumen ke local filesystem untuk benchmark |
+| `POST` | `/api/v1/document/process` | Proses dokumen async |
+| `GET` | `/api/v1/documents/{user_id}` | List dokumen milik user |
+| `DELETE` | `/api/v1/document/delete` | Hapus dokumen |
+| `GET` | `/api/v1/database-clients/dbtypes` | List tipe database dan schema credential form |
+| `POST` | `/api/v1/database-clients` | Buat koneksi database |
+| `GET` | `/api/v1/database-clients/{user_id}` | List koneksi database user |
+| `GET` | `/api/v1/database-clients/{user_id}/{client_id}` | Detail koneksi database |
+| `PUT` | `/api/v1/database-clients/{client_id}` | Update koneksi database |
+| `DELETE` | `/api/v1/database-clients/{client_id}` | Hapus koneksi database |
+| `POST` | `/api/v1/database-clients/{client_id}/ingest` | Introspect schema database ke catalog |
+| `POST` | `/api/v1/data-catalog/rebuild` | Rebuild user data catalog |
+| `GET` | `/api/v1/data-catalog/{user_id}` | Ambil user data catalog index |
+| `POST` | `/api/v1/analyses` | Buat analysis baru |
+| `GET` | `/api/v1/analyses` | List analysis user |
+| `GET` | `/api/v1/analyses/{id}` | Detail analysis |
+| `PATCH` | `/api/v1/analyses/{id}` | Update metadata/status analysis |
+| `DELETE` | `/api/v1/analyses/{id}` | Hapus analysis |
+| `PUT` | `/api/v1/analyses/{id}/data-bind` | Update data source binding analysis |
+| `GET` | `/api/v1/analyses/{id}/data-catalog` | Ambil catalog yang scoped ke analysis |
+| `POST` | `/api/v1/analyses/{id}/data-catalog/rebuild` | Rebuild catalog scoped ke analysis dari data_bind |
+| `GET` | `/api/v1/analyses/{id}/messages` | Ambil riwayat pesan analysis |
+| `POST` | `/api/v1/analyses/{id}/messages` | Rekam satu pesan conversation |
+
+## Auth
+
+### `POST /api/login`
+
+Login user dengan email dan password.
+
+Request:
+
+```json
+{
+  "email": "user@example.com",
+  "password": "password"
+}
+```
+
+Success `200`:
+
+```json
+{
+  "status": "success",
+  "message": "login successful",
+  "data": {
+    "user": {
+      "id": "user-id",
+      "email": "user@example.com",
+      "fullname": "User Name",
+      "role": "user",
+      "status": "active"
+    },
+    "access_token": "jwt-access-token",
+    "refresh_token": "opaque-refresh-token",
+    "token_type": "Bearer",
+    "expires_in": 3600,
+    "refresh_expires_in": 604800
+  }
+}
+```
+
+Errors: `400`, `401`, `403`, `404`, `500`.
+
+### `POST /api/refresh`
+
+Menukar refresh token aktif dengan token pair baru.
+
+Request:
+
+```json
+{
+  "refresh_token": "opaque-refresh-token"
+}
+```
+
+Success `200` mengembalikan bentuk `data` yang sama dengan login, berisi user, access token baru, refresh token baru, `token_type`, `expires_in`, dan `refresh_expires_in`.
+
+Errors: `400`, `401`, `403`, `500`.
+
+## Documents
+
+### Document Model
+
+```json
+{
+  "id": "document-id",
+  "user_id": "user-id",
+  "filename": "sales.csv",
+  "blob_name": "user-id/document-id/sales.csv",
+  "file_size": 2048,
+  "file_type": "csv",
+  "status": "uploaded",
+  "chunks_count": 0,
+  "processed_at": "2026-06-30T08:00:00Z",
+  "error_message": null,
+  "created_at": "2026-06-30T08:00:00Z"
+}
+```
+
+Status umum: `uploaded`, `processing`, `processed`, `failed`.
+
+### `GET /api/v1/documents/doctypes`
+
+Mengambil tipe dokumen yang didukung.
+
+Success `200`:
+
+```json
+{
+  "status": "success",
+  "message": "supported document types",
+  "data": [
+    {
+      "type": "pdf",
+      "max_size_mb": 10,
+      "status": "active",
+      "message": null
+    }
+  ]
+}
+```
+
+### `POST /api/v1/document/upload`
+
+Upload dokumen ke Azure Blob Storage. Maksimum 10 MB. Mendukung `pdf`, `docx`, `txt`, `csv`, dan `xlsx`.
+
+Content-Type: `multipart/form-data`
+
+Form fields:
+
+| Field | Required | Keterangan |
+| --- | --- | --- |
+| `user_id` | Yes | Harus sama dengan user dari token |
+| `file` | Yes | File dokumen |
+
+Success `201`: `data` berisi Document Model.
+
+Errors: `400`, `401`, `403`, `429`, `500`.
+
+### `POST /api/v1/document/upload-local`
+
+Upload file ke filesystem lokal untuk benchmarking. Kontrak form sama dengan upload Azure.
+
+Success `201`:
+
+```json
+{
+  "status": "success",
+  "message": "file saved locally",
+  "data": {
+    "path": "files/user-id/sales.csv"
+  }
+}
+```
+
+### `POST /api/v1/document/process`
+
+Memulai proses dokumen secara async. Untuk dokumen unstructured, service melakukan extract text dan embedding jika tersedia. Untuk dokumen tabular, service membuat parquet/catalog source.
+
+Request:
+
+```json
+{
+  "document_id": "document-id",
+  "user_id": "user-id"
+}
+```
+
+Success `202`:
+
+```json
+{
+  "status": "success",
+  "message": "document processing started",
+  "data": {
+    "document_id": "document-id",
+    "file_type": "csv",
+    "status": "processing"
+  }
+}
+```
+
+Pantau hasilnya lewat `GET /api/v1/documents/{user_id}`.
+
+Errors: `400`, `401`, `403`, `404`, `500`.
+
+### `GET /api/v1/documents/{user_id}`
+
+List dokumen milik user.
+
+Success `200`:
+
+```json
+{
+  "status": "success",
+  "message": "documents",
+  "data": []
+}
+```
+
+Errors: `401`, `403`, `500`.
+
+### `DELETE /api/v1/document/delete`
+
+Menghapus dokumen dari storage, embedding/parquet terkait, dan record database.
+
+Request:
+
+```json
+{
+  "document_id": "document-id",
+  "user_id": "user-id"
+}
+```
+
+Success `200`:
+
+```json
+{
+  "status": "success",
+  "message": "document deleted"
+}
+```
+
+Errors: `400`, `401`, `403`, `404`, `500`.
+
+## Database Clients
+
+### DB Client Model
+
+```json
+{
+  "id": "client-id",
+  "user_id": "user-id",
+  "name": "Analytics Warehouse",
+  "db_type": "postgres",
+  "status": "active",
+  "created_at": "2026-06-30T08:00:00Z",
+  "updated_at": "2026-06-30T08:00:00Z"
+}
+```
+
+### `GET /api/v1/database-clients/dbtypes`
+
+Mengambil tipe database dan daftar field credential untuk render form dinamis. Saat ini `postgres` aktif; tipe lain dapat muncul sebagai `inactive`.
+
+### `POST /api/v1/database-clients`
+
+Menyimpan koneksi database. Credentials disimpan terenkripsi. Jika koneksi dengan identity yang sama sudah ada, response `200` mengembalikan client existing.
+
+Request:
+
+```json
+{
+  "user_id": "user-id",
+  "name": "Analytics Warehouse",
+  "db_type": "postgres",
+  "credentials": {
+    "host": "db.example.com",
+    "port": 5432,
+    "database": "analytics",
+    "username": "db_user",
+    "password": "db_password",
+    "ssl_mode": "require"
+  }
+}
+```
+
+Success:
+
+- `201`: database client created
+- `200`: database client already exists
+
+Errors: `400`, `401`, `403`, `429`, `500`.
+
+### `GET /api/v1/database-clients/{user_id}`
+
+List koneksi database milik user.
+
+Success `200`:
+
+```json
+{
+  "status": "success",
+  "message": "database clients",
+  "data": []
+}
+```
+
+Errors: `401`, `403`, `500`.
+
+### `GET /api/v1/database-clients/{user_id}/{client_id}`
+
+Ambil detail satu koneksi database. Success `200` dengan `data` berisi DB Client Model.
+
+Errors: `401`, `403`, `404`, `500`.
+
+### `PUT /api/v1/database-clients/{client_id}?user_id={user_id}`
+
+Update nama, credentials, atau status koneksi. Semua field body optional, tetapi body harus JSON valid.
+
+Request:
+
+```json
+{
+  "name": "Updated Warehouse",
+  "credentials": {
+    "host": "db.example.com",
+    "port": 5432,
+    "database": "analytics",
+    "username": "db_user",
+    "password": "new_password",
+    "ssl_mode": "require"
+  },
+  "status": "active"
+}
+```
+
+Success `200`: `data` berisi DB Client Model.
+
+Errors: `400`, `401`, `403`, `404`, `500`.
+
+### `DELETE /api/v1/database-clients/{client_id}?user_id={user_id}`
+
+Menghapus koneksi database dan memicu pembersihan catalog.
+
+Success `200`:
+
+```json
+{
+  "status": "success",
+  "message": "database client deleted"
+}
+```
+
+Errors: `400`, `401`, `403`, `404`, `500`.
+
+### `POST /api/v1/database-clients/{client_id}/ingest?user_id={user_id}`
+
+Melakukan introspection schema database dan menyimpan hasilnya ke catalog. Tidak membutuhkan request body.
+
+Success `200`:
+
+```json
+{
+  "status": "success",
+  "message": "schema ingested",
+  "data": {
+    "tables": []
+  }
+}
+```
+
+Errors: `400`, `401`, `403`, `404`, `409`, `429`, `500`.
+
+## Data Catalog
+
+### `POST /api/v1/data-catalog/rebuild`
+
+Rebuild seluruh catalog user dari dokumen tabular dan database client aktif.
+
+Request:
+
+```json
+{
+  "user_id": "user-id"
+}
+```
+
+Success `200`:
+
+```json
+{
+  "status": "success",
+  "message": "catalog rebuilt",
+  "data": {
+    "user_id": "user-id",
+    "schema_version": "1.0",
+    "generated_at": "2026-06-30T08:00:00Z",
+    "sources": []
+  }
+}
+```
+
+Errors: `400`, `401`, `403`, `429`, `500`.
+
+### `GET /api/v1/data-catalog/{user_id}`
+
+Mengambil index catalog user. Response `sources` berisi ringkasan source, tanpa detail table penuh.
+
+Success `200`:
+
+```json
+{
+  "status": "success",
+  "message": "data catalog",
+  "data": {
+    "user_id": "user-id",
+    "schema_version": "1.0",
+    "generated_at": "2026-06-30T08:00:00Z",
+    "sources": [
+      {
+        "source_id": "document-or-client-id",
+        "source_type": "tabular",
+        "name": "sales.csv",
+        "location_ref": "blob/path/or/db-ref",
+        "table_count": 1,
+        "updated_at": "2026-06-30T08:00:00Z"
+      }
+    ]
+  }
+}
+```
+
+Errors: `401`, `403`, `500`.
+
+### `POST /api/v1/analyses/{id}/data-catalog/rebuild`
+
+Membangun ulang catalog khusus analysis berdasarkan `data_bind` terbaru milik analysis tersebut. Endpoint ini hanya memakai source yang ter-bind ke analysis, bukan semua knowledge source user.
+
+Gunakan endpoint ini setelah perubahan binding jika frontend ingin memicu rebuild secara eksplisit. `PUT /api/v1/analyses/{id}/data-bind` juga melakukan rebuild analysis catalog sebagai bagian dari update binding.
+
+Tidak membutuhkan request body.
+
+Success `200`:
+
+```json
+{
+  "status": "success",
+  "message": "analysis catalog rebuilt",
+  "data": {
+    "user_id": "user-id",
+    "schema_version": "1.0",
+    "generated_at": "2026-06-30T08:00:00Z",
+    "sources": []
+  }
+}
+```
+
+Errors: `400`, `401`, `404`, `500`.
+
+### `GET /api/v1/analyses/{id}/data-catalog`
+
+Mengambil catalog yang scoped ke analysis dan mengikuti `data_bind` analysis, bukan seluruh catalog user.
+
+Success `200`:
+
+```json
+{
+  "status": "success",
+  "message": "analysis data catalog",
+  "data": {
+    "user_id": "user-id",
+    "schema_version": "1.0",
+    "generated_at": "2026-06-30T08:00:00Z",
+    "sources": []
+  }
+}
+```
+
+Errors: `401`, `404`.
+
+## Analyses
+
+### Data Bind Item
+
+`data_bind` adalah daftar source yang dipilih user untuk analysis.
+
+```json
+{
+  "id": "source-id",
+  "name": "sales.csv",
+  "group_type": "document",
+  "type": "csv"
+}
+```
+
+Field:
+
+| Field | Required | Keterangan |
+| --- | --- | --- |
+| `id` | Yes | `document.id` atau `database_client.id` |
+| `name` | Yes | Nama yang ditampilkan di UI |
+| `group_type` | Yes | `document` atau `database` |
+| `type` | Yes | File type (`csv`, `pdf`, `xlsx`) atau database type (`postgres`) |
+
+Rules:
+
+- `data_bind` wajib berisi minimal satu source.
+- Semua source harus milik user yang sedang login.
+- Duplicate source dalam satu `data_bind` ditolak.
+
+### Analysis Model
+
+```json
+{
+  "id": "analysis-id",
+  "user_id": "user-id",
+  "analysis_title": "Q3 Revenue Analysis",
+  "objective": "Find revenue movement and root cause",
+  "business_questions": [
+    "Why did revenue drop in Q3?",
+    "Which customer segment contributed the most to the change?"
+  ],
+  "status": "active",
+  "data_bind": [
+    {
+      "id": "document-id",
+      "name": "sales.csv",
+      "group_type": "document",
+      "type": "csv"
+    }
+  ],
+  "data_bind_version": 1,
+  "report_collection": [],
+  "created_at": "2026-06-30T08:00:00Z",
+  "updated_at": "2026-06-30T08:00:00Z"
+}
+```
+
+### `POST /api/v1/analyses`
+
+Membuat analysis aktif dengan source binding awal.
+
+Request:
+
+```json
+{
+  "analysis_title": "Q3 Revenue Analysis",
+  "objective": "Find revenue movement and root cause",
+  "business_questions": [
+    "Why did revenue drop in Q3?",
+    "Which customer segment contributed the most to the change?"
+  ],
+  "data_bind": [
+    {
+      "id": "document-id",
+      "name": "sales.csv",
+      "group_type": "document",
+      "type": "csv"
+    },
+    {
+      "id": "database-client-id",
+      "name": "Analytics Warehouse",
+      "group_type": "database",
+      "type": "postgres"
+    }
+  ]
+}
+```
+
+Success `201`: `data` berisi Analysis Model.
+
+Validation:
+
+- `business_questions` wajib berisi minimal satu string non-empty.
+- `data_bind` wajib berisi minimal satu source.
+
+Errors: `400`, `401`, `409`.
+
+### `GET /api/v1/analyses`
+
+List analysis milik user.
+
+Query:
+
+| Query | Default | Keterangan |
+| --- | --- | --- |
+| `status` | `active` | `active` atau `inactive` |
+| `page` | `1` | Nomor halaman |
+| `limit` | `20` | Maksimum `100` |
+
+Success `200`:
+
+```json
+{
+  "status": "success",
+  "message": "Analyses retrieved",
+  "data": {
+    "analyses": [],
+    "pagination": {
+      "page": 1,
+      "limit": 20
+    }
+  }
+}
+```
+
+Errors: `401`, `500`.
+
+### `GET /api/v1/analyses/{id}`
+
+Ambil detail analysis milik user. Success `200` dengan `data` berisi Analysis Model.
+
+Errors: `400`, `401`, `404`.
+
+### `PATCH /api/v1/analyses/{id}`
+
+Update metadata analysis. Field optional.
+
+Request:
+
+```json
+{
+  "analysis_title": "Updated title",
+  "objective": "Updated objective",
+  "status": "inactive"
+}
+```
+
+`status` hanya `active` atau `inactive`.
+
+Success `200`: `data` berisi Analysis Model.
+
+Errors: `400`, `401`, `404`.
+
+### `DELETE /api/v1/analyses/{id}`
+
+Hapus analysis milik user.
+
+Success `204` tanpa response body.
+
+Errors: `400`, `401`, `404`.
+
+### `PUT /api/v1/analyses/{id}/data-bind`
+
+Mengganti daftar source yang ter-bind ke analysis secara atomic dengan optimistic version check. Jika update berhasil, service juga rebuild catalog scope analysis dari `data_bind` terbaru. Jika rebuild catalog gagal, perubahan binding ditolak/rollback.
+
+Request:
+
+```json
+{
+  "expected_version": 1,
+  "data_bind": [
+    {
+      "id": "new-document-id",
+      "name": "updated-sales.csv",
+      "group_type": "document",
+      "type": "csv"
+    }
+  ]
+}
+```
+
+Success `200`: `data` berisi Analysis Model dengan `data_bind_version` yang sudah naik.
+
+Errors:
+
+- `400`: payload invalid, empty binding, source invalid
+- `401`: token invalid/missing
+- `409`: stale `expected_version`, inactive analysis, atau limit violation
+
+## Analysis Messages
+
+### Message Model
+
+```json
+{
+  "id": "message-id",
+  "analysis_id": "analysis-id",
+  "user_id": "user-id",
+  "role": "user",
+  "content": "Apa penyebab revenue turun di Q3?",
+  "created_at": "2026-06-30T08:00:00Z"
+}
+```
+
+`role` hanya:
+
+- `user`: pertanyaan atau instruksi dari user
+- `ai`: jawaban dari AI Agent Service
+
+### `POST /api/v1/analyses/{id}/messages`
+
+Merekam tepat satu pesan conversation ke `analyses_messages`.
+
+Request untuk pertanyaan user:
+
+```json
+{
+  "role": "user",
+  "content": "Apa penyebab revenue turun di Q3?"
+}
+```
+
+Request untuk jawaban AI:
+
+```json
+{
+  "role": "ai",
+  "content": "Revenue turun karena penurunan transaksi enterprise dan kenaikan churn di wilayah barat."
+}
+```
+
+Success `201`:
+
+```json
+{
+  "status": "success",
+  "message": "Message created",
+  "data": {
+    "message": {
+      "id": "message-id",
+      "analysis_id": "analysis-id",
+      "user_id": "user-id",
+      "role": "user",
+      "content": "Apa penyebab revenue turun di Q3?",
+      "created_at": "2026-06-30T08:00:00Z"
+    }
+  }
+}
+```
+
+Errors:
+
+- `400`: invalid role/content, invalid analysis ID
+- `401`: token invalid/missing
+- `409`: inactive analysis atau message limit tercapai
+
+### `GET /api/v1/analyses/{id}/messages`
+
+Mengambil riwayat conversation analysis.
+
+Query:
+
+| Query | Default | Keterangan |
+| --- | --- | --- |
+| `limit` | `100` | Maksimum `100` |
+
+Success `200`:
+
+```json
+{
+  "status": "success",
+  "message": "Messages retrieved",
+  "data": {
+    "messages": []
+  }
+}
+```
+
+Errors: `400`, `401`, `404`.
+
+## Suggested Frontend Integration Sequence
+
+### Login
+
+```text
+POST /api/login
+store access_token, refresh_token, user.id
+```
+
+### Upload and Process File
+
+```text
+GET  /api/v1/documents/doctypes
+POST /api/v1/document/upload
+POST /api/v1/document/process
+GET  /api/v1/documents/{user_id}
+```
+
+### Connect Database
+
+```text
+GET  /api/v1/database-clients/dbtypes
+POST /api/v1/database-clients
+POST /api/v1/database-clients/{client_id}/ingest?user_id={user_id}
+GET  /api/v1/database-clients/{user_id}
+```
+
+### Generate Knowledge Catalog
+
+```text
+POST /api/v1/data-catalog/rebuild
+GET  /api/v1/data-catalog/{user_id}
+```
+
+### Create Analysis and Start Conversation
+
+```text
+POST /api/v1/analyses with business_questions
+call AI Agent Service outside this service
+POST /api/v1/analyses/{analysis_id}/messages with role=user and content=user_question
+POST /api/v1/analyses/{analysis_id}/messages with role=ai and content=agent_answer
+GET  /api/v1/analyses/{analysis_id}/messages
+```
+
+## Important Frontend Notes
+
+- Jangan mengirim pesan user ke `POST /api/v1/analyses/{id}/messages` dengan ekspektasi service ini akan menjawab. Endpoint ini hanya persistence.
+- AI Agent Service adalah service terpisah. Service ini menyimpan metadata analysis, knowledge catalog, data binding, dan history conversation.
+- User-level catalog berisi seluruh knowledge source user; analysis-level catalog hanya berisi source yang ada di `data_bind` analysis.
+- Perubahan `data_bind` akan rebuild analysis-level catalog. Frontend juga dapat memanggil endpoint rebuild analysis catalog secara eksplisit jika diperlukan.
+- Setelah refresh token sukses, selalu replace refresh token lama dengan refresh token baru.
+- Untuk endpoint yang membutuhkan `user_id`, gunakan `data.user.id` dari login dan pastikan sama dengan token aktif.
+- Untuk binding analysis, pakai source yang sudah berhasil diupload/diproses atau database client yang sudah diingest.
+
+
+
+
+
+
+

API_CONTRACT_BE_PYTHON.md

@@ -0,0 +1,521 @@
+# Backend Agentic Service API Contract
+
+This document describes the Python agentic backend used by the frontend for AI chat, help/report tools, and observability data shown alongside chat answers.
+
+Base path examples use relative URLs. Configure the frontend with the deployed Python service base URL.
+
+## Overview
+
+The Python backend owns the generative AI interaction surface:
+
+1. Stream chat answers from the AI agent.
+2. Execute tool-style actions for help and report generation.
+3. Return report versions and report details.
+4. Return observability/provenance for a completed assistant answer.
+
+The frontend uses this service during the analysis conversation flow:
+
+1. User sends a chat message.
+2. Frontend calls `POST /api/v2/chat/stream` and renders the streamed answer.
+3. When the stream emits `done`, frontend stores or reads the returned `message_id`.
+4. Frontend calls `GET /api/v1/observability` for planning, tool calls, and source provenance.
+5. Frontend calls `/api/v1/tools/help` for guided help and `/api/v1/tools/report` for report generation.
+
+## Endpoint Summary
+
+| Method | Path | Purpose |
+| --- | --- | --- |
+| `POST` | `/api/v2/chat/stream` | Stream an AI chat answer for one analysis conversation. |
+| `GET` | `/api/v1/tools/list` | List available frontend tools. |
+| `POST` | `/api/v1/tools/help` | Stream contextual help for the current analysis conversation. |
+| `POST` | `/api/v1/tools/report` | Generate and persist a new report version. |
+| `GET` | `/api/v1/tools/report/{analysis_id}` | List report versions for an analysis. |
+| `GET` | `/api/v1/tools/report/{analysis_id}/{version}` | Retrieve one report version. |
+| `GET` | `/api/v1/observability` | Retrieve provenance for one assistant answer. |
+
+## Common Concepts
+
+### Identifiers
+
+- `user_id`: user identifier passed by the frontend.
+- `analysis_id`: analysis conversation identifier.
+- `message_id`: assistant answer identifier used to correlate chat streaming with observability.
+
+### Server-Sent Events
+
+Chat and help endpoints return `text/event-stream`.
+
+Frontend should parse events by `event` name and `data` payload. Blank lines separate SSE events.
+
+Common event types:
+
+| Event | Data | Meaning |
+| --- | --- | --- |
+| `sources` | JSON array | Sources available early in the stream. May be empty. |
+| `status` | text | Optional progress update for slower paths. |
+| `chunk` | text | Answer text fragment. Concatenate chunks in order. |
+| `done` | JSON object | Terminal success event. Includes `message_id`. |
+| `error` | text | Terminal error event. Stream stops after this. |
+
+The stream carries answer text only. Planning, tool call details, and full provenance are fetched from `GET /api/v1/observability` after the stream is done.
+
+## Chat
+
+### `POST /api/v2/chat/stream`
+
+Streams an AI answer for one user message in an analysis conversation.
+
+Request body:
+
+```json
+{
+  "user_id": "u_1a2b3c",
+  "analysis_id": "an_42",
+  "message_id": "msg_88f1",
+  "message": "What were total sales by region last quarter?"
+}
+```
+
+Fields:
+
+| Field | Required | Description |
+| --- | --- | --- |
+| `user_id` | Yes | User identifier. |
+| `analysis_id` | Yes | Analysis conversation identifier. |
+| `message_id` | No | Assistant answer id for observability correlation. If omitted, Python returns one in `done`. |
+| `message` | Yes | User message text. |
+
+Response: `text/event-stream`.
+
+Example structured answer:
+
+```text
+event: sources
+data: [{"document_id":"u_1a2b3c_orders","filename":"orders","page_label":null}]
+
+event: status
+data: Planning analysis...
+
+event: status
+data: Running 3 steps...
+
+event: chunk
+data: Total sales by region last quarter: 
+
+event: chunk
+data: Central led at $1.21M (38%), East $0.74M, West $0.55M (down 12% QoQ).
+
+event: done
+data: {"message_id":"msg_88f1"}
+```
+
+Example simple chat answer:
+
+```text
+event: sources
+data: []
+
+event: chunk
+data: I'm your AI data analyst. Connect a source or ask a question to get started.
+
+event: done
+data: {"message_id":"msg_12"}
+```
+
+Behavior notes:
+
+- Greeting and farewell messages may use a fast canned path.
+- Stateless `chat` intent may use a 1-hour Redis response cache.
+- The router may classify messages into intents such as `chat`, `help`, `check`, `unstructured_flow`, or `structured_flow`.
+- `sources` can be empty for chat/help/error paths.
+- `status` events are optional and should be safe for the frontend to ignore.
+
+## Tools
+
+### `GET /api/v1/tools/list`
+
+Returns the deterministic list of tools available to the frontend.
+
+Request: none.
+
+Response `200`:
+
+```json
+{
+  "count": 2,
+  "tools": [
+    {
+      "command": "/help",
+      "name": "help",
+      "type": "skill",
+      "description": "Show what the assistant can do and guide your next step."
+    },
+    {
+      "command": "/report",
+      "name": "report",
+      "type": "skill",
+      "description": "Generate a versioned analysis report with background, EDA, key findings, and insights."
+    }
+  ]
+}
+```
+
+Tool item shape:
+
+```json
+{
+  "command": "/help",
+  "name": "help",
+  "type": "skill",
+  "description": "Show what the assistant can do and guide your next step."
+}
+```
+
+Frontend behavior:
+
+- Surface `/help` in the slash menu.
+- Surface report generation as a button or explicit UI action.
+
+### `POST /api/v1/tools/help`
+
+Streams contextual guidance for the current analysis conversation.
+
+Request body:
+
+```json
+{
+  "user_id": "u_1a2b3c",
+  "analysis_id": "an_42"
+}
+```
+
+Response: `text/event-stream` using the same event shape as chat.
+
+Help responses usually emit `sources: []` and no `status` pings.
+
+Example:
+
+```text
+event: sources
+data: []
+
+event: chunk
+data: Your goal is set. You can start exploring now. Try a question like "average order value by month", then I can generate a report.
+
+event: done
+data: {"message_id":"msg_h7"}
+```
+
+## Reports
+
+### `POST /api/v1/tools/report`
+
+Generates, persists, and returns a new report version for an analysis.
+
+Query params:
+
+| Query | Required | Description |
+| --- | --- | --- |
+| `analysis_id` | Yes | Analysis identifier. |
+| `user_id` | Yes | User identifier. |
+
+Example:
+
+```text
+POST /api/v1/tools/report?analysis_id=an_42&user_id=u_1a2b3c
+```
+
+Status codes:
+
+| Status | Meaning |
+| --- | --- |
+| `201` | New report version generated. |
+| `409` | Report floor/precondition not met. |
+| `500` | Generation or persistence failed. |
+
+Response `201`:
+
+```json
+{
+  "report_id": "8f3a2b1c9d4e4f6a8b0c1d2e3f4a5b6c",
+  "analysis_id": "an_42",
+  "user_id": "u_1a2b3c",
+  "version": 2,
+  "generated_at": "2026-06-30T09:14:33.512Z",
+  "problem_statement": {
+    "objective": "Understand which regions drive revenue and why Q1 dipped.",
+    "business_questions": [
+      "Which regions contribute most to total revenue?",
+      "Did any region decline quarter-over-quarter?"
+    ]
+  },
+  "record_ids": ["rec_a1", "rec_b2"],
+  "executive_summary": "Revenue is concentrated in the Central region (38% of total). The West was the only region to contract, down 12% QoQ, the main driver of the Q1 dip.",
+  "findings": [
+    {
+      "text": "Central region contributed 38% of total revenue, the largest share.",
+      "record_ids": ["rec_a1"],
+      "supporting_data": null
+    },
+    {
+      "text": "West region revenue fell 12% quarter-over-quarter.",
+      "record_ids": ["rec_b2"],
+      "supporting_data": null
+    }
+  ],
+  "caveats": [
+    {
+      "text": "March data for the East region was partially missing, around 6% of rows.",
+      "record_ids": ["rec_b2"]
+    }
+  ],
+  "open_questions": [
+    {
+      "text": "What drove the West region's QoQ decline?",
+      "record_ids": ["rec_b2"]
+    }
+  ],
+  "data_sources": [
+    {
+      "source_id": "src_sales_db",
+      "name": "orders",
+      "source_type": "postgres",
+      "detail": {
+        "tables": ["orders"],
+        "row_count": 48213,
+        "columns": ["region", "amount", "ordered_at"]
+      }
+    }
+  ],
+  "method_steps": [
+    {
+      "task_id": "t1",
+      "stage": "data_understanding",
+      "objective": "Inventory the sales source",
+      "status": "success",
+      "tools_used": ["check_data"]
+    },
+    {
+      "task_id": "t2",
+      "stage": "modeling",
+      "objective": "Aggregate revenue by region",
+      "status": "success",
+      "tools_used": ["analyze_aggregate"]
+    }
+  ],
+  "rendered_markdown": "# Analysis Report\n\n*Generated 2026-06-30 by u_1a2b3c*\n\n## Objective\nUnderstand which regions drive revenue..."
+}
+```
+
+Response `409`:
+
+```json
+{
+  "detail": "Not ready to generate a report - still needs at least one completed analysis."
+}
+```
+
+Precondition:
+
+- Reports require at least one completed analysis record for the session.
+- If slow-path analysis recording is disabled, report generation can return `409` by design.
+
+### `GET /api/v1/tools/report/{analysis_id}`
+
+Lists report versions for one analysis, oldest first.
+
+Response `200`:
+
+```json
+[
+  {
+    "report_id": "1b2c3d4e",
+    "version": 1,
+    "generated_at": "2026-06-24T15:02:11Z",
+    "record_count": 1
+  },
+  {
+    "report_id": "8f3a2b1c",
+    "version": 2,
+    "generated_at": "2026-06-25T09:14:33Z",
+    "record_count": 2
+  }
+]
+```
+
+If no reports exist, returns `[]`.
+
+### `GET /api/v1/tools/report/{analysis_id}/{version}`
+
+Returns one report version. Shape is the same as the `201` response from `POST /api/v1/tools/report`.
+
+Response `404`:
+
+```json
+{
+  "detail": "No report v3 for analysis 'an_42'."
+}
+```
+
+## Observability
+
+### `GET /api/v1/observability`
+
+Returns Responsible AI provenance for one assistant answer.
+
+The frontend should call this after the chat/help stream emits `done`, using the `message_id` from the `done` event. If the row is not ready yet, the frontend may poll until `200` or stop on `404` according to product behavior.
+
+Query params:
+
+| Query | Required | Description |
+| --- | --- | --- |
+| `analysis_id` | Yes | Analysis identifier. |
+| `message_id` | Yes | Assistant answer identifier returned by the stream. |
+
+Example:
+
+```text
+GET /api/v1/observability?analysis_id=an_42&message_id=msg_88f1
+```
+
+Field rules:
+
+- `planning`: present only when the planner ran; otherwise `null`.
+- `thinking`: optional reasoning summary; `null` if unavailable.
+- `tool_calls`: every invoked tool with input, output, and status; empty for pure chat or greeting paths.
+- `sources`: required for retrieval flows; empty for chat/help paths that do not reference data.
+
+Response `200` for `structured_flow`:
+
+```json
+{
+  "analysis_id": "an_42",
+  "message_id": "msg_88f1",
+  "intent": "structured_flow",
+  "generated_at": "2026-06-30T03:21:09.114Z",
+  "planning": {
+    "goal_restated": "Find which regions drive revenue and why Q1 dipped.",
+    "assumptions": ["'last quarter' = Q1 2026"],
+    "steps": [
+      {
+        "step": 1,
+        "stage": "data_understanding",
+        "objective": "Inventory the sales source"
+      },
+      {
+        "step": 2,
+        "stage": "modeling",
+        "objective": "Aggregate revenue by region"
+      }
+    ]
+  },
+  "thinking": "The question needs a per-region breakdown plus a cause, so I inventory the source, aggregate revenue by region, then compare quarters.",
+  "tool_calls": [
+    {
+      "order": 1,
+      "name": "check_data",
+      "input": { "source_hint": "structured" },
+      "output": { "kind": "table", "summary": "1 source, 1 table, 48,213 rows" },
+      "status": "success"
+    },
+    {
+      "order": 2,
+      "name": "retrieve_data",
+      "input": {
+        "source_id": "src_sales_db",
+        "table_id": "orders",
+        "select": ["region", "amount"],
+        "group_by": ["region"]
+      },
+      "output": {
+        "kind": "table",
+        "columns": ["region", "total"],
+        "row_count": 4,
+        "preview": [["Central", 1210000], ["East", 740000]]
+      },
+      "status": "success"
+    }
+  ],
+  "sources": [
+    {
+      "type": "database",
+      "source_id": "src_sales_db",
+      "name": "orders",
+      "query": "SELECT region, SUM(amount) AS total FROM orders GROUP BY region",
+      "detail": {
+        "tables": ["orders"],
+        "row_count": 48213
+      }
+    }
+  ]
+}
+```
+
+Response `200` for `unstructured_flow`:
+
+```json
+{
+  "analysis_id": "an_42",
+  "message_id": "msg_55",
+  "intent": "unstructured_flow",
+  "generated_at": "2026-06-30T03:40:02.001Z",
+  "planning": null,
+  "thinking": null,
+  "tool_calls": [
+    {
+      "order": 1,
+      "name": "retrieve_knowledge",
+      "input": {
+        "query": "technology stack used in this project",
+        "top_k": 4
+      },
+      "output": {
+        "kind": "documents",
+        "row_count": 4
+      },
+      "status": "success"
+    }
+  ],
+  "sources": [
+    {
+      "type": "document",
+      "document_id": "doc_7",
+      "filename": "tech_handbook.pdf",
+      "page_label": "12",
+      "query": "technology stack used in this project",
+      "snippet": "The backend is built on FastAPI with async SQLAlchemy...",
+      "score": 0.83
+    }
+  ]
+}
+```
+
+Response `200` for simple chat or greeting:
+
+```json
+{
+  "analysis_id": "an_42",
+  "message_id": "msg_12",
+  "intent": "chat",
+  "generated_at": "2026-06-30T03:05:00.000Z",
+  "planning": null,
+  "thinking": null,
+  "tool_calls": [],
+  "sources": []
+}
+```
+
+Response `404`:
+
+```json
+{
+  "detail": "No observability for message 'msg_88f1' yet."
+}
+```
+
+Frontend rendering guidance:
+
+- Render observability separately from the streamed answer.
+- Default state can be collapsed.
+- Show planning, tool calls, and sources as separate sections.
+- Treat `planning: null`, `tool_calls: []`, and `sources: []` as valid states.

API_ENDPOINTS.md

@@ -1,373 +0,0 @@
-# Data Eyond — Python Agentic Service: FE-Callable API (for Go integration)
-
-**Audience:** Harry (Go gateway) wiring the FE → Go → Python surface.
-**Scope:** the **4 FE-callable surfaces** the Python service exposes after the 2026-06-24 pivot
-(DEV_PLAN decision #6). Everything else under `/api/v1` is internal / Phase-1 legacy / Go-owned —
-see [§7](#7-not-fe-facing) and the full inventory in [§9](#9-appendix--complete-endpoint-inventory-all-registered-routes).
-**Branch:** `pr/4` · **Snapshot:** 2026-06-25 · **Companion:** [REPO_STATUS.md](REPO_STATUS.md).
-
-> Request flow is **FE → Go → Python**. The FE never calls Python directly except for chat
-> streaming. Auth/JWT is terminated at the Go gateway; Python receives `user_id` / `room_id` as
-> **trusted inputs** and does no auth of its own.
-
----
-
-## 1. The 4 FE-callable surfaces
-
-| # | Logical name | HTTP | How it's invoked |
-|---|---|---|---|
-| 1 | **`call_agent`** | `POST /api/v1/chat/stream` | The one streaming chat call. Router classifies + dispatches. |
-| 2 | **`list_skills`** | `GET /api/v1/tools` | Static slash-command catalog for the FE "/" menu. Cacheable. |
-| 3 | **skill: `help`** | *(via `call_agent`)* | **No dedicated endpoint** — the router resolves it to the `help` intent inside `/chat/stream`. |
-| 4 | **skill: `report`** | `POST /api/v1/report` (+ 2 `GET`s) | Dedicated REST API. **Not** through `/chat/stream`. |
-
-**Key consequence for Go:** the two catalog skills are invoked **differently**. `/help` goes through
-`/chat/stream`; `/report` is a direct REST call to the Report API. The catalog's `name` field is the
-internal route key (`help` = router intent; `report` = the Report API), not a uniform dispatch key.
-
-**Conventions:**
-- Base path: `/api/v1`.
-- **`room_id == analysis_id`** — one chat room == one analysis session (#9). Callers pass `room_id`
-  to chat; it *is* the `analysis_id` used by the report API.
-- Streaming uses **SSE** (`text/event-stream`, `sse-starlette`).
-
----
-
-## 2. `call_agent` — `POST /api/v1/chat/stream`
-
-The only FE→Python call in normal operation. Source: [chat.py:169](src/api/v1/chat.py:169).
-
-**Request body** (`application/json`) — `ChatRequest`:
-
-```json
-{
-  "user_id": "u_1a2b3c",
-  "room_id": "room_42",
-  "message": "What were total sales by region last quarter?"
-}
-```
-
-`room_id` is the analysis session id. No auth header (handled by Go).
-
-**Response:** `text/event-stream`. Events arrive in this order:
-
-| `event:` | `data:` payload | Notes |
-|---|---|---|
-| `sources` | JSON array of source refs | `{document_id, filename, page_label}`. Structured: one per executed table (`document_id = "{user_id}_{table}"`, `page_label = null`). Unstructured: deduped doc/page. `chat`/`help`/`error`: `[]`. |
-| `status` | text | **Slow-path only** — progress pings ("Planning…", "Running N steps…"). Keeps the SSE alive; safe to surface or ignore. |
-| `chunk` | text fragment | Concatenate in order to form the answer. |
-| `done` | *(empty)* | End of stream. |
-| `error` | text | Terminal error; stream stops after this. |
-
-> The handler also emits an internal `intent` event — it is **consumed inside Python** (gates
-> caching) and **not forwarded** to the client. Go/FE will never see it.
-
-**Example — `structured_flow` answer** (raw SSE wire; blank line separates events). Source shape:
-[chat_handler.py:607](src/agents/chat_handler.py:607).
-
-```
-event: sources
-data: [{"document_id":"u_1a2b3c_orders","filename":"orders","page_label":null}]
-
-event: status
-data: Planning analysis…
-
-event: status
-data: Running 3 steps…
-
-event: chunk
-data: Total sales by region last quarter: 
-
-event: chunk
-data: Central led at $1.21M (38%), East $0.74M, West $0.55M (down 12% QoQ).
-
-event: done
-data: 
-```
-
-**Example — simple `chat` reply** (no status pings, empty sources):
-
-```
-event: sources
-data: []
-
-event: chunk
-data: I'm your AI data analyst — connect a source or ask a question to get started.
-
-event: done
-data: 
-```
-
-**Behavior worth knowing for integration:**
-- **Redis response cache** (1h TTL) is applied to the stateless `chat` intent only; cached replies
-  replay as `sources`/`chunk`/`done`.
-- **Greeting/farewell fast-path** returns a canned reply with no LLM call.
-- The LLM **router** classifies every message into one of **5 intents** —
-  `chat` · `help` · `check` · `unstructured_flow` · `structured_flow` — and dispatches. Messages
-  persist (user + assistant) on `done`.
-
----
-
-## 3. `list_skills` — `GET /api/v1/tools`
-
-Static, deterministic, **safe for Go to cache**. Source: [tools.py:133](src/api/v1/tools.py:133).
-
-**Request:** none (no params, no body).
-
-**Response** `200` (`ListToolsResponse`):
-
-```json
-{
-  "count": 2,
-  "tools": [
-    { "command": "/help",   "name": "help",   "type": "skill",
-      "description": "Show what the assistant can do and guide your next step." },
-    { "command": "/report", "name": "report", "type": "skill",
-      "description": "Generate a versioned analysis report (background, EDA, key findings, insights)." }
-  ]
-}
-```
-
-`CommandResponse` = `{ command, name, type, description }`, `type ∈ {skill, analytics, data_access}`.
-Post-KM-678 the catalog is **`/help` + `/report` only**; the `analyze_*`, `check_*`, `retrieve_*`
-and retired `/problem-statement` entries are commented out (kept for restorability), not deleted.
-
----
-
-## 4. skill: `help` — via `call_agent`
-
-**There is no `/help` endpoint.** The FE "/" menu surfaces `/help`; to invoke it, call
-`POST /api/v1/chat/stream` and let the router classify the message as the `help` intent
-([chat_handler.py:363](src/agents/chat_handler.py:363)). Help streams `chunk` events (same SSE
-shape as §2, with `sources: []` and no `status` pings) — a state-aware, next-step guidance reply.
-
-```
-event: sources
-data: []
-
-event: chunk
-data: Your goal is set — you can start exploring now. Try a question like "average order value by month", then I can generate a report.
-
-event: done
-data: 
-```
-
-> **Open integration question (for Harry):** the Python `/chat/stream` contract has **no
-> forced-intent / slash-bypass param** — `handle()` always routes via the LLM classifier. So
-> deterministic `/help` dispatch depends on either (a) Go forwarding the literal slash text and
-> trusting the router to classify it as `help`, or (b) adding a forced-intent input to the chat
-> contract. The `tools.py` docstring's "slash invocation bypasses the router to the tool directly"
-> is **not yet true on the Python side.** Needs a decision. (DEV_PLAN #8/#18.)
-
----
-
-## 5. skill: `report` — Report API
-
-Dedicated REST surface (the "Generate Report" button), **not** a chat route.
-Source: [report.py](src/api/v1/report.py).
-
-### `POST /api/v1/report`
-Generate, persist, and return a new report **version**.
-
-**Query params:** `analysis_id` (required), `user_id` (required). No request body.
-
-```
-POST /api/v1/report?analysis_id=room_42&user_id=u_1a2b3c
-```
-
-| Status | Meaning |
-|---|---|
-| `201` | New version generated → `AnalysisReport` body. |
-| `409` | Floor not met — **no recorded analyses yet** for this session, nothing to report. |
-| `500` | Generation or persistence failed. |
-
-**`201` response** (`AnalysisReport`):
-
-```json
-{
-  "report_id": "8f3a2b1c9d4e4f6a8b0c1d2e3f4a5b6c",
-  "analysis_id": "room_42",
-  "user_id": "u_1a2b3c",
-  "version": 2,
-  "generated_at": "2026-06-25T09:14:33.512Z",
-  "problem_statement": {
-    "objective": "Understand which regions drive revenue and why Q1 dipped.",
-    "business_questions": [
-      "Which regions contribute most to total revenue?",
-      "Did any region decline quarter-over-quarter?"
-    ]
-  },
-  "record_ids": ["rec_a1", "rec_b2"],
-  "executive_summary": "Revenue is concentrated in the Central region (38% of total). The West was the only region to contract, down 12% QoQ — the main driver of the Q1 dip.",
-  "findings": [
-    { "text": "Central region contributed 38% of total revenue, the largest share.",
-      "record_ids": ["rec_a1"], "supporting_data": null },
-    { "text": "West region revenue fell 12% quarter-over-quarter.",
-      "record_ids": ["rec_b2"], "supporting_data": null }
-  ],
-  "caveats": [
-    { "text": "March data for the East region was partially missing (~6% of rows).",
-      "record_ids": ["rec_b2"] }
-  ],
-  "open_questions": [
-    { "text": "What drove the West region's QoQ decline?", "record_ids": ["rec_b2"] }
-  ],
-  "data_sources": [
-    { "source_id": "src_sales_db", "name": "orders", "source_type": "postgres",
-      "detail": { "tables": ["orders"], "row_count": 48213,
-                  "columns": ["region", "amount", "ordered_at"] } }
-  ],
-  "method_steps": [
-    { "task_id": "t1", "stage": "data_understanding", "objective": "Inventory the sales source",
-      "status": "success", "tools_used": ["check_data"] },
-    { "task_id": "t2", "stage": "modeling", "objective": "Aggregate revenue by region",
-      "status": "success", "tools_used": ["analyze_aggregate"] }
-  ],
-  "rendered_markdown": "# Analysis Report\n\n*Generated 2026-06-25 by u_1a2b3c · 2 analyses · 1 source(s)*\n\n## Objective\nUnderstand which regions drive revenue…\n\n## Key Findings\n1. Central region contributed 38%…"
-}
-```
-
-**`409` response** (floor not met — the demo's most common error):
-
-```json
-{ "detail": "Not ready to generate a report — still needs at least one completed analysis." }
-```
-
-> ⚠️ **Demo/integration precondition:** `AnalysisRecord`s persist **only on the slow path**, so
-> reports require **`enable_slow_path=true`** on the Python deployment *and* ≥1 prior
-> `structured_flow` question in the session. With slow path off, `POST /report` **409s by design**,
-> not a bug. (DEV_PLAN #15/#16.)
-
-### `GET /api/v1/report/{analysis_id}`
-List a session's report versions (oldest-first). Returns `[ReportVersionEntry]`; `[]` if none.
-
-```json
-[
-  { "report_id": "1b2c3d4e…", "version": 1, "generated_at": "2026-06-24T15:02:11Z", "record_count": 1 },
-  { "report_id": "8f3a2b1c…", "version": 2, "generated_at": "2026-06-25T09:14:33Z", "record_count": 2 }
-]
-```
-
-### `GET /api/v1/report/{analysis_id}/{version}`
-Fetch one version → `AnalysisReport` (same shape as the `POST` 201 body above); `404` if that
-version doesn't exist.
-
-```json
-{ "detail": "No report v3 for analysis 'room_42'." }
-```
-
----
-
-## 6. Schemas
-
-**`AnalysisReport`** (POST + GET-version body):
-
-| Field | Type | Notes |
-|---|---|---|
-| `report_id` | str | |
-| `analysis_id` | str | == `room_id` |
-| `user_id` | str \| null | |
-| `version` | int | monotonic V1, V2, … |
-| `generated_at` | datetime | ISO 8601, UTC |
-| `problem_statement` | `{ objective: str, business_questions: string[] }` | the frozen goal snapshot (new pivot shape) |
-| `record_ids` | string[] | records the version was built from |
-| `executive_summary` | str | the **only** LLM-authored field |
-| `findings` | `ReportFinding[]` | `{ text, record_ids[], supporting_data? }` |
-| `caveats` | `AttributedNote[]` | `{ text, record_ids[] }` |
-| `open_questions` | `AttributedNote[]` | `{ text, record_ids[] }` |
-| `data_sources` | `DataSourceRef[]` | `{ source_id, name, source_type, detail }` |
-| `method_steps` | `TaskSummary[]` | `{ task_id, stage, objective, status, tools_used[] }`; `stage` ∈ CRISP-DM phases |
-| `rendered_markdown` | str | the full rendered report |
-
-> **Persistence caveat:** dedorch `reports` stores **markdown only**. On read-back via the `GET`
-> endpoints, the structured fields above come back **empty** and `rendered_markdown` is the source of
-> truth. (REPO_STATUS §5.)
-
-**`ReportVersionEntry`** (GET-list rows): `{ report_id, version, generated_at, record_count }`.
-
----
-
-## 7. Not FE-facing
-
-Registered under `/api/v1` but **not** part of the FE→Python surface — do not wire these from the FE:
-
-- **Analysis CRUD** — `POST /analysis/create`, `GET /analysis`, `GET /analysis/{id}`. Intended to
-  move behind Go (state writes via Go, per decision #5/#18). Router still **mounted** (Go may use it);
-  the FE should not call it.
-- **`check_data` / `check_knowledge`** — served by **Go**, not surfaced as Python FE endpoints.
-- **Chat cache management** — `DELETE /chat/cache`, `/chat/cache/room/{id}`, `/retrieval/cache/{user_id}`
-  (ops/internal).
-- **Phase-1 legacy routers** — `users`, `room`, `document`, `db_client`, `data_catalog`
-  (functionally migrated to Go; mostly dormant).
-- **Health/root** — `GET /`, `GET /health` (liveness only).
-
----
-
-## 8. Open items affecting this contract
-
-1. **`/help` dispatch mechanism** — router-classify vs. forced-intent param (§4). *(DEV_PLAN #8/#18)*
-2. **`/report` needs `enable_slow_path=true`** + a prior `structured_flow` question, else 409.
-   *(DEV_PLAN #15)*
-3. **`analysis_records` home** post-`SKIP_INIT_DB` cutover — the report API depends on this table
-   existing. *(DEV_PLAN #14/#16)*
-4. **Analysis-state writes** — once Go owns creation + state writes, Python's per-turn state
-   `ensure` becomes a read-only get (Go must guarantee the row exists before any chat turn).
-   *(DEV_PLAN #18)*
-
----
-
-## 9. Appendix — complete endpoint inventory (all registered routes)
-
-Every route mounted in [main.py](main.py), so task #8 can be decided against the full picture.
-**32 routes** across 9 routers + 2 app-level. Status legend:
-**✅ FE-callable** (one of the 4 surfaces — keep) · **✂️ comment out** (task #8 target) ·
-**🟦 legacy → Go** (Phase-1, functionally migrated; not FE→Python; mostly dormant) ·
-**⚙️ internal/ops**.
-
-| Method | Path | Purpose | Router | Status |
-|---|---|---|---|---|
-| POST | `/api/v1/chat/stream` | Main chat SSE — **`call_agent`**; carries chat/help/check/structured/unstructured intents | Chat | ✅ FE-callable (#1, +help #3) |
-| GET | `/api/v1/tools` | Slash-command catalog — **`list_skills`** (Go caches) | Tools | ✅ FE-callable (#2) |
-| POST | `/api/v1/report` | Generate a report version | Report | ✅ FE-callable (#4) |
-| GET | `/api/v1/report/{analysis_id}` | List report versions | Report | ✅ FE-callable (#4) |
-| GET | `/api/v1/report/{analysis_id}/{version}` | Fetch one report version | Report | ✅ FE-callable (#4) |
-| POST | `/api/v1/analysis/create` | Create session (state + room + bindings) | Analysis | ✂️ comment (#8 → Go) |
-| GET | `/api/v1/analysis` | List a user's analyses | Analysis | ✂️ comment (#8) |
-| GET | `/api/v1/analysis/{analysis_id}` | Get one session's state + sources | Analysis | ✂️ comment (#8) |
-| DELETE | `/api/v1/chat/cache` | Clear one cached reply | Chat | ⚙️ internal/ops |
-| DELETE | `/api/v1/chat/cache/room/{room_id}` | Clear a room's cache | Chat | ⚙️ internal/ops |
-| DELETE | `/api/v1/retrieval/cache/{user_id}` | Clear a user's retrieval cache | Chat | ⚙️ internal/ops |
-| GET | `/` | Service status | (app) | ⚙️ internal/ops |
-| GET | `/health` | Liveness probe | (app) | ⚙️ internal/ops |
-| POST | `/api/login` | Login by email + password ⚠️ mounted at `/api`, **not** `/api/v1` | Users | 🟦 legacy → Go |
-| GET | `/api/v1/documents/doctypes` | Supported document types | Documents | 🟦 legacy → Go |
-| GET | `/api/v1/documents/{user_id}` | List a user's documents | Documents | 🟦 legacy → Go |
-| POST | `/api/v1/document/upload` | Upload a document (10/min) | Documents | 🟦 legacy → Go |
-| DELETE | `/api/v1/document/delete` | Delete a document | Documents | 🟦 legacy → Go |
-| POST | `/api/v1/document/process` | Process / ingest a document | Documents | 🟦 legacy → Go |
-| GET | `/api/v1/rooms/{user_id}` | List a user's rooms | Rooms | 🟦 legacy → Go |
-| GET | `/api/v1/room/{room_id}` | Get one room | Rooms | 🟦 legacy → Go |
-| DELETE | `/api/v1/room/{room_id}` | Delete a room | Rooms | 🟦 legacy → Go |
-| POST | `/api/v1/room/create` | Create a room | Rooms | 🟦 legacy → Go |
-| GET | `/api/v1/data-catalog/{user_id}` | List catalog index | Data Catalog | 🟦 legacy → Go |
-| POST | `/api/v1/data-catalog/rebuild` | Rebuild a user's catalog | Data Catalog | 🟦 legacy → Go |
-| GET | `/api/v1/database-clients/dbtypes` | Supported DB types | Database Clients | 🟦 legacy → Go |
-| POST | `/api/v1/database-clients` | Create a DB connection | Database Clients | 🟦 legacy → Go |
-| GET | `/api/v1/database-clients/{user_id}` | List a user's DB connections | Database Clients | 🟦 legacy → Go |
-| GET | `/api/v1/database-clients/{user_id}/{client_id}` | Get one DB connection | Database Clients | 🟦 legacy → Go |
-| PUT | `/api/v1/database-clients/{client_id}` | Update a DB connection | Database Clients | 🟦 legacy → Go |
-| DELETE | `/api/v1/database-clients/{client_id}` | Delete a DB connection | Database Clients | 🟦 legacy → Go |
-| POST | `/api/v1/database-clients/{client_id}/ingest` | Build the catalog for a DB connection | Database Clients | 🟦 legacy → Go |
-
-**Tally:** 5 ✅ FE-callable · 3 ✂️ to comment (#8) · 19 🟦 legacy→Go · 5 ⚙️ internal/ops.
-
-**Task #8 reading:**
-- **Keep exposed:** the 5 ✅ rows (`chat/stream`, `/tools`, the 3 `report` routes). `help` rides on
-  `chat/stream` — no route of its own.
-- **Comment out (the #8 to-do):** the 3 `analysis` routes — analysis CRUD moves behind Go (#5/#18).
-- **`check_data` is not an HTTP endpoint** — it's the `check` router intent (runs inside
-  `chat/stream`) plus its now-commented slash-catalog entry (KM-678); Go serves it to the FE. So
-  "comment check_data" = the catalog line (done) + don't expose a Python route (there isn't one).
-- The 19 🟦 routers (`users`, `document`, `room`, `data_catalog`, `db_client`) are Phase-1 legacy,
-  already functionally in Go (REPO_STATUS §7). They're out of the FE→Python path but **still
-  mounted** — a separate cleanup from #8's analysis-CRUD scope.

API_ENDPOINTS_RESTRUCTURE.md

@@ -0,0 +1,391 @@
+# Backend Agentic Service — API Endpoint Docs (endpoint restructure)
+
+**Status:** contract draft for FE/Go integration (2026-06-30). Covers the AI-only surface after the
+restructure. Sections marked **TENTATIVE** (observability) may still change — send feedback before we
+lock them.
+
+**What changed**
+- **Only the chat pilot moves to `/api/v2`.** Everything else stays on `/api/v1`, regrouped under `/tools`.
+- **Chat pilot (`/api/v2/chat/stream`) uses `analysis_id`, not `room_id`.**
+- **Skills are grouped under `/api/v1/tools`:** `list` / `help` / `report`.
+- **New:** `GET /api/v1/observability` — Responsible-AI provenance per chat answer.
+- Python is **generative-AI only.** It never creates/updates an analysis, room, document, DB
+  client, or catalog — Go owns those. Python just receives `analysis_id`. Those v1 routers are
+  unwired from `main` + Swagger (not deleted).
+
+**Open coordination questions (need a decision with Harry) — flagged inline as ⚠️:**
+1. **`message_id` origin** — who mints the assistant turn id used to correlate stream ↔ observability? (Recommend: Go mints it, passes it in the chat request, Python echoes on `done`.)
+2. **Deterministic `/help` dispatch** — dedicated endpoint (recommended below) vs router classification.
+3. **Observability storage** — single JSONB row per message (recommended) vs 3 normalized tables.
+
+---
+
+## 1. call_agent — `POST /api/v2/chat/stream`
+
+The only FE→Python call in normal operation. Same as v1 except **`room_id` → `analysis_id`**, and
+the `done` event now carries the assistant `message_id` for observability correlation.
+
+**Request body** (`application/json`) — `ChatRequest`:
+
+```json
+{
+  "user_id": "u_1a2b3c",
+  "analysis_id": "an_42",
+  "message_id": "msg_88f1",
+  "message": "What were total sales by region last quarter?"
+}
+```
+
+- `analysis_id` is the analysis-session id (replaces `room_id`). No auth header (handled by Go).
+- ⚠️ `message_id` (optional): the assistant turn id. **Recommended: Go mints it** alongside the
+  `analyses_messages` row and passes it here, so the FE can call `/api/v1/observability?message_id=...`
+  in parallel. If omitted, Python mints one and returns it on `done`.
+
+**Response:** `text/event-stream`. Events arrive in this order:
+
+| event | data | notes |
+|---|---|---|
+| `sources` | JSON array of `{document_id, filename, page_label}` | structured: one per executed table; unstructured: deduped doc/page; chat/help/error: `[]`. |
+| `status` | text | slow-path only — progress pings ("Planning…", "Running N steps…"). Safe to surface or ignore. |
+| `chunk` | text fragment | concatenate in order to form the answer. |
+| `done` | `{"message_id": "..."}` | **v2 change:** was empty; now returns the turn id for the observability lookup. |
+| `error` | text | terminal error; stream stops after this. |
+
+The internal `intent` event is consumed inside Python (gates caching) and **not** forwarded.
+
+**Stream carries the answer text ONLY.** Planning / tool calls / sources detail are **not** in the
+stream (it would slow it down) — fetch them from `/observability` (§7), called in parallel.
+
+**Example — `structured_flow` answer** (raw SSE; blank line separates events):
+
+```
+event: sources
+data: [{"document_id":"u_1a2b3c_orders","filename":"orders","page_label":null}]
+
+event: status
+data: Planning analysis…
+
+event: status
+data: Running 3 steps…
+
+event: chunk
+data: Total sales by region last quarter: 
+
+event: chunk
+data: Central led at $1.21M (38%), East $0.74M, West $0.55M (down 12% QoQ).
+
+event: done
+data: {"message_id":"msg_88f1"}
+```
+
+**Example — simple chat reply** (no status pings, empty sources):
+
+```
+event: sources
+data: []
+
+event: chunk
+data: I'm your AI data analyst — connect a source or ask a question to get started.
+
+event: done
+data: {"message_id":"msg_12"}
+```
+
+Behavior unchanged from v1: 1h Redis response-cache on the stateless `chat` intent only;
+greeting/farewell fast-path (canned, no LLM); LLM router classifies every message into one of 5
+intents (`chat · help · check · unstructured_flow · structured_flow`); messages persist on `done`.
+
+---
+
+## 2. list_skills — `GET /api/v1/tools/list`
+
+Static, deterministic, safe for Go to cache. (Was `GET /api/v1/tools`.)
+
+**Request:** none.
+
+**Response 200** (`ListToolsResponse`):
+
+```json
+{
+  "count": 2,
+  "tools": [
+    { "command": "/help",   "name": "help",   "type": "skill",
+      "description": "Show what the assistant can do and guide your next step." },
+    { "command": "/report", "name": "report", "type": "skill",
+      "description": "Generate a versioned analysis report (background, EDA, key findings, insights)." }
+  ]
+}
+```
+
+`CommandResponse = { command, name, type, description }`, `type ∈ {skill, analytics, data_access}`.
+Catalog is `/help` + `/report` only; the `analyze_*` / `check_*` / `retrieve_*` and retired
+`/problem-statement` entries are commented out (kept for restorability), not deleted.
+
+**FE behavior:** the `/` slash menu surfaces **`/help` only**. **Report is a right-side button, not
+a slash command** (it fires only when an analysis is finished — saves tokens).
+
+---
+
+## 3. skill: help — `POST /api/v1/tools/help`
+
+⚠️ **Proposed dedicated endpoint** (new in v2). In v1 there was no `/help` endpoint — help was reached
+only by letting the LLM router classify a chat message. A dedicated endpoint makes `/help` dispatch
+**deterministic** (no risk the router mis-classifies the slash command) and gives it a clean home in
+the tools group. State-aware: reads analysis state + history to guide the next step.
+
+> Alternative if we *don't* add this endpoint: FE keeps calling `POST /chat/stream` and trusts the
+> router to classify the help intent. We recommend the dedicated endpoint — decision pending (open
+> question #2).
+
+**Request body** (`application/json`):
+
+```json
+{
+  "user_id": "u_1a2b3c",
+  "analysis_id": "an_42"
+}
+```
+
+**Response:** `text/event-stream` — same SSE shape as chat, with `sources: []` and no `status`
+pings (help never references documents). Streams a next-step guidance reply.
+
+```
+event: sources
+data: []
+
+event: chunk
+data: Your goal is set — you can start exploring now. Try a question like "average order value by month", then I can generate a report.
+
+event: done
+data: {"message_id":"msg_h7"}
+```
+
+---
+
+## 4. skill: report — `POST /api/v1/tools/report`
+
+The "Generate Report" button. Same as v1, moved under `/tools`. Generate, persist, and return a new
+report version. Currently renders **Markdown** (FE preview); PPT/PDF/infographic export is future work
+(triggered on a download button, not here).
+
+**Query params:** `analysis_id` (required), `user_id` (required). No request body.
+
+```
+POST /api/v1/tools/report?analysis_id=an_42&user_id=u_1a2b3c
+```
+
+| status | meaning |
+|---|---|
+| 201 | new version generated → `AnalysisReport` body. |
+| 409 | floor not met — no recorded analyses yet for this session, nothing to report. |
+| 500 | generation or persistence failed. |
+
+**201 response** (`AnalysisReport`):
+
+```json
+{
+  "report_id": "8f3a2b1c9d4e4f6a8b0c1d2e3f4a5b6c",
+  "analysis_id": "an_42",
+  "user_id": "u_1a2b3c",
+  "version": 2,
+  "generated_at": "2026-06-30T09:14:33.512Z",
+  "problem_statement": {
+    "objective": "Understand which regions drive revenue and why Q1 dipped.",
+    "business_questions": [
+      "Which regions contribute most to total revenue?",
+      "Did any region decline quarter-over-quarter?"
+    ]
+  },
+  "record_ids": ["rec_a1", "rec_b2"],
+  "executive_summary": "Revenue is concentrated in the Central region (38% of total). The West was the only region to contract, down 12% QoQ — the main driver of the Q1 dip.",
+  "findings": [
+    { "text": "Central region contributed 38% of total revenue, the largest share.",
+      "record_ids": ["rec_a1"], "supporting_data": null },
+    { "text": "West region revenue fell 12% quarter-over-quarter.",
+      "record_ids": ["rec_b2"], "supporting_data": null }
+  ],
+  "caveats": [
+    { "text": "March data for the East region was partially missing (~6% of rows).",
+      "record_ids": ["rec_b2"] }
+  ],
+  "open_questions": [
+    { "text": "What drove the West region's QoQ decline?", "record_ids": ["rec_b2"] }
+  ],
+  "data_sources": [
+    { "source_id": "src_sales_db", "name": "orders", "source_type": "postgres",
+      "detail": { "tables": ["orders"], "row_count": 48213,
+                  "columns": ["region", "amount", "ordered_at"] } }
+  ],
+  "method_steps": [
+    { "task_id": "t1", "stage": "data_understanding", "objective": "Inventory the sales source",
+      "status": "success", "tools_used": ["check_data"] },
+    { "task_id": "t2", "stage": "modeling", "objective": "Aggregate revenue by region",
+      "status": "success", "tools_used": ["analyze_aggregate"] }
+  ],
+  "rendered_markdown": "# Analysis Report\n\n*Generated 2026-06-30 by u_1a2b3c · 2 analyses · 1 source(s)*\n\n## Objective\nUnderstand which regions drive revenue…\n\n## Key Findings\n1. Central region contributed 38%…"
+}
+```
+
+**409 response** (floor not met — the demo's most common error):
+
+```json
+{ "detail": "Not ready to generate a report — still needs at least one completed analysis." }
+```
+
+⚠️ **Precondition:** `AnalysisRecord`s persist only on the slow path, so reports require
+`ENABLE_SLOW_PATH=true` on the Python deployment and ≥1 prior `structured_flow` question in the
+session. With slow path off, `POST` 409s by design.
+
+---
+
+## 5. report versions — `GET /api/v1/tools/report/{analysis_id}` and `/{analysis_id}/{version}`
+
+List a session's report versions (oldest-first). Returns `[ReportVersionEntry]`; `[]` if none.
+
+```json
+[
+  { "report_id": "1b2c3d4e…", "version": 1, "generated_at": "2026-06-24T15:02:11Z", "record_count": 1 },
+  { "report_id": "8f3a2b1c…", "version": 2, "generated_at": "2026-06-25T09:14:33Z", "record_count": 2 }
+]
+```
+
+`GET /api/v1/tools/report/{analysis_id}/{version}` → one `AnalysisReport` (same shape as the POST
+201 body); 404 if that version doesn't exist:
+
+```json
+{ "detail": "No report v3 for analysis 'an_42'." }
+```
+
+---
+
+## 6. Unwired in v2 (mounted in v1, OFF in v2)
+
+Commented out of `main` + Swagger, **files kept**. Go owns these; Python is generative-only:
+`POST /analysis/create` + analysis CRUD · `room` · `db_client` · `document` · `data_catalog` ·
+`users`/login. Re-mounting is a one-line `include_router` if ever needed.
+
+---
+
+## 7. observability — `GET /api/v1/observability`  **(NEW · TENTATIVE)**
+
+Responsible-AI provenance for **one chat answer**. Separate endpoint, called **in parallel with the
+stream** — never embedded in it. The FE renders it as a collapsed dropdown the user can expand
+(planning / tool calls / sources), Claude/Codex-style.
+
+**Design (recommended):** one endpoint returns one merged object, backed by **one JSONB row per
+message** written by an accumulating "scratchpad" decorator inside the chat agent and flushed on
+`done`. The 3 facets (`planning` / `tool_calls` / `sources`) are **logical sections of the JSON**,
+not separate tables — so the shape can evolve without a dedorch migration each time. (Storage is open
+question #3.)
+
+**Query params:** `analysis_id` (required), `message_id` (required).
+
+```
+GET /api/v1/observability?analysis_id=an_42&message_id=msg_88f1
+```
+
+**Timing:** the row is written when the turn finishes, so call this **after** the stream's `done`
+event (or poll until 200). "Parallel" = a separate call the FE fires alongside the stream, not data
+embedded in the stream.
+
+**Field rules (by intent):**
+- `planning` — present **only when the planner ran** (slow path); `null` otherwise.
+- `tool_calls` — every tool invoked, with input + output. `[]` for pure chat / greeting / help.
+- `sources` — **required for retrieve flows** (`structured_flow`, `unstructured_flow`). **Empty for
+  greeting / `chat` / `help`** (they don't reference documents).
+- `thinking` — optional reasoning text; `null` if none.
+
+**200 response — full `structured_flow` turn** (planner ran → all sections present):
+
+```json
+{
+  "analysis_id": "an_42",
+  "message_id": "msg_88f1",
+  "intent": "structured_flow",
+  "generated_at": "2026-06-30T03:21:09.114Z",
+  "planning": {
+    "goal_restated": "Find which regions drive revenue and why Q1 dipped.",
+    "assumptions": ["'last quarter' = Q1 2026"],
+    "steps": [
+      { "step": 1, "stage": "data_understanding", "objective": "Inventory the sales source" },
+      { "step": 2, "stage": "modeling", "objective": "Aggregate revenue by region" }
+    ]
+  },
+  "thinking": "The question needs a per-region breakdown plus a cause, so I inventory the source, aggregate revenue by region, then compare quarters.",
+  "tool_calls": [
+    {
+      "order": 1,
+      "name": "check_data",
+      "input": { "source_hint": "structured" },
+      "output": { "kind": "table", "summary": "1 source · 1 table (orders) · 48,213 rows" },
+      "status": "success"
+    },
+    {
+      "order": 2,
+      "name": "retrieve_data",
+      "input": { "source_id": "src_sales_db", "table_id": "orders",
+                 "select": ["region", "amount"], "group_by": ["region"] },
+      "output": { "kind": "table", "columns": ["region", "total"], "row_count": 4,
+                  "preview": [["Central", 1210000], ["East", 740000]] },
+      "status": "success"
+    }
+  ],
+  "sources": [
+    {
+      "type": "database",
+      "source_id": "src_sales_db",
+      "name": "orders",
+      "query": "SELECT region, SUM(amount) AS total FROM orders GROUP BY region",
+      "detail": { "tables": ["orders"], "row_count": 48213 }
+    }
+  ]
+}
+```
+
+**200 response — `unstructured_flow` turn** (no planner; source = document, with the retrieval query):
+
+```json
+{
+  "analysis_id": "an_42",
+  "message_id": "msg_55",
+  "intent": "unstructured_flow",
+  "generated_at": "2026-06-30T03:40:02.001Z",
+  "planning": null,
+  "thinking": null,
+  "tool_calls": [
+    { "order": 1, "name": "retrieve_knowledge",
+      "input": { "query": "technology stack used in this project", "top_k": 4 },
+      "output": { "kind": "documents", "row_count": 4 }, "status": "success" }
+  ],
+  "sources": [
+    { "type": "document", "document_id": "doc_7", "filename": "tech_handbook.pdf",
+      "page_label": "12", "query": "technology stack used in this project",
+      "snippet": "The backend is built on FastAPI with async SQLAlchemy…", "score": 0.83 }
+  ]
+}
+```
+
+**200 response — simple `chat` / greeting turn** (nothing to trace):
+
+```json
+{
+  "analysis_id": "an_42",
+  "message_id": "msg_12",
+  "intent": "chat",
+  "generated_at": "2026-06-30T03:05:00.000Z",
+  "planning": null,
+  "thinking": null,
+  "tool_calls": [],
+  "sources": []
+}
+```
+
+**404** — no provenance for that message yet (turn still running or unknown id):
+
+```json
+{ "detail": "No observability for message 'msg_88f1' yet." }
+```
+
+> ⚠️ **Richness is path-dependent.** Full `planning` + tool I/O exist only when
+> `ENABLE_SLOW_PATH=true`. Fast chat / single-query / help still record `sources` + the single
+> tool call but have `planning: null`. This matches the rule "planning only when the planner runs."

DEV_PLAN.md

@@ -1,10 +1,47 @@
-# Data Eyond — Current Development Plan (post 2026-06-24 meeting + 2026-06-25 checkpoint)
+# Data Eyond — Current Development Plan (post 2026-06-24 → 2026-06-30 checkpoints)
 
 **Purpose:** context file for Claude Code sessions working on the current sprint.
-**Branch:** `pr/4` · **Snapshot:** 2026-06-25.
+**Branch:** `pr/5` · **Snapshot:** 2026-06-30.
 **Companion:** [REPO_STATUS.md](REPO_STATUS.md) describes the repo's *current built state*; this file
-describes the *in-flight plan* that changes it. New decisions from the 2026-06-25 checkpoint are in
-[§1.5](#15-2026-06-25-checkpoint-deltas).
+describes the *in-flight plan* that changes it. The **active sprint is pr/5** ([§0](#0-current-sprint--pr5-observability--endpoint-restructure)); sections §1–§6 are the prior
+2026-06-24/25 pivot (now largely ✅), kept for context.
+
+---
+
+## 0. Current sprint — pr/5: Observability + Endpoint Restructure
+
+From the **2026-06-30 checkpoint**. Direction: **Python → generation/AI-only**; Go owns the analysis
+lifecycle + data plane. Endpoint contract sent to Harry on 2026-06-30:
+[API_ENDPOINTS_RESTRUCTURE.md](API_ENDPOINTS_RESTRUCTURE.md) (chat→v2, tools regroup, observability —
+observability marked tentative). REPO_STATUS carries a matching `pr/5` direction banner.
+
+Mentor's task order: **unwire → regroup endpoints → add tools (retrieve-data + observability)**; share
+the endpoint contract *before* coding the tools. Status legend: ⬜ not started · 🔄 in progress · ✅ done ·
+⛔ blocked · 🔎 verify · ⏸️ deferred.
+
+| Phase | Task | Owner | Status | Notes |
+|---|---|---|---|---|
+| **P0 — contract** | Draft + send endpoint contract to Harry (chat v2 · tools group · observability) | Rifqi + Sofhia | ✅ | `API_ENDPOINTS_RESTRUCTURE.md` sent 2026-06-30 (before-noon deadline met). Observability section flagged tentative. |
+| **1 — unwire** | Unwire `users`(login)/`document`/`room`/`db_client`/`data_catalog`/`analysis` from `main` + Swagger | Sofhia | ✅ | **KM-686**, commit `0b2d678`. Commented, not deleted; `chat`/`report`/`tools` kept mounted. Resolves the analysis-CRUD scope Q — whole `analysis` router unwired (Go owns it). |
+| **2 — v2 + regroup** | Create `src/api/v2/` and move the chat pilot there | Rifqi | ✅ | New `src/api/v2/__init__.py` + `src/api/v2/chat.py` (`POST /api/v2/chat/stream`), mounted in `main.py`. Only chat in v2; v1 `/chat/stream` kept mounted until FE moves over. Routes import-verified. |
+| **2 — v2 + regroup** | Chat: `room_id` → **`analysis_id`** (request field + handler + history) | Rifqi | ✅ | v2 `ChatRequest{user_id, analysis_id, message, message_id?}`; reuses warm `ChatHandler` + v1 cache/history helpers; `done` returns `{message_id}` (minted Python-side if Go omits, open-Q #1). Persistence kept transitionally → still ties to #25 (`analyses_messages`); ruff-clean. |
+| **2 — v2 + regroup** | Move report under tools → `/api/v1/tools/report` (+ version routes) | Rifqi | ✅ | report router re-prefixed `/api/v1` → `/api/v1/tools` (all 3 routes move together), tag → `Tools`; old `/api/v1/report` gone. Same functionality, new home. Import-verified. |
+| **2 — v2 + regroup** | Move help under tools → `POST /api/v1/tools/help` (dedicated endpoint) | Sofhia | ✅ | New `src/api/v1/help.py` (SSE: `sources:[]`→`chunk`→`done{message_id}`) + additive `ChatHandler.stream_help()` (reuses HelpAgent+state+readiness, no router). Generative-only (no persist). **Router `help` intent KEPT** — both paths live by design. message_id minted Python-side if Go omits (open-Q #1). Import-verified. |
+| **2 — v2 + regroup** | Tools list → `/api/v1/tools/list` | Sofhia | ✅ | Renamed route `GET /api/v1/tools` → `GET /api/v1/tools/list` ([tools.py:133](src/api/v1/tools.py:133)). |
+| **2 — v2 + regroup** | FE: slash menu = `/help` only; report = right-side button | Mentor (FE) | ⬜ | Coordination note, not Python work. |
+| **3 — tools + obs** | Finish `help` so it actually **calls** (not just lists) + test | Sofhia | ⬜ | Mentor: help currently only lists tools. Core #2 after chat. |
+| **3 — tools + obs** | Observability **scratchpad** (decorator) accumulating in the chat agent | Rifqi + Sofhia | ⬜ | Capture planning / tool I/O / sources during the run; flush one record on `done`. |
+| **3 — tools + obs** | Audit `report_inputs` — covers planning + tool I/O + source? add cols / new store | Rifqi | ⬜ | **Rec:** dedicated provenance store = 1 JSONB row per message (logical 3 sections); keep Langfuse for engineering. |
+| **3 — tools + obs** | Build `GET /api/v1/observability` (one merged response) | Rifqi | ⬜ | Intent-based source rules (greeting/help = none; retrieve = required). Richness path-dependent (full planning only on slow path). |
+| **3 — tools + obs** | Keep stream **text-only**; observability is a separate parallel call | Rifqi | ⬜ | Per mentor — don't slow the stream. |
+| **3 — tools + obs** | Resolve `message_id` correlation (stream ↔ observability) with Harry | Rifqi ↔ Harry | ⬜ | **Rec:** Go mints `message_id`, passes in chat request, Python echoes on `done`. |
+| **4 — biz questions** | Get Go folder; confirm `business_questions` in create-analysis (max 5); sync Python | Harry/Mentor → Rifqi | ⬜ | Go currently missing the field ("lagi difixing"). Python already models objective + business_questions. |
+| **deferred** | Report formats: PPT (preferred) / PDF / infographic on download | — | ⏸️ | MD is fine for the FE preview stage now. |
+| **deferred** | Charts (Plotly→JSON) + images tables | — | ⏸️ | Carried from §4 #26/#27. |
+
+**Next up:** Phase 2 Python work is **done** (chat→v2 `analysis_id`; `help`/`report`/`list` regrouped
+under `/api/v1/tools/`). Remaining: **Phase 3** — the observability scratchpad + `GET /api/v1/observability`
+(shape already speced in the contract), then **Phase 4** (business questions, Go-blocked).
 
 ---
 

REPO_STATUS.md

@@ -2,13 +2,27 @@
 
 **Audience:** teammates onboarding onto the Python repo (`Agentic-Service-Data-Eyond-Catalog`).
 **Scope:** what the code does **right now** (branch `pr/4`, ticket KM-652). Describes current state only — no roadmap or to-dos.
-**Snapshot date:** 2026-06-25.
+**Snapshot date:** 2026-06-25. **Cross-repo update 2026-06-29:** §2/§8/§11/§12 re-verified against
+the **Go source** (`Orchestrator-Agent-Service`), not its docs. The Go service has moved well past its
+own (uncommitted, stale) design docs: it now hosts the **dedorch SQL migrations** in-repo and a full
+**`/api/v1/analyses` + `/api/v1/skills`** REST surface. Go does **not** call Python yet — those skills
+are placeholders (see §12).
 
 > This file is grounded in the source, not the older design docs. Where the two
 > disagree, the code wins — see [§11 Doc-vs-code](#11-where-the-older-docs-are-stale).
 > `REPO_CONTEXT.md` / `ARCHITECTURE.md` are the original Phase-2 design docs and are
 > stale on the router, joins, and the analysis/report stack.
 
+> 🚧 **Direction update 2026-06-30 (pr/5 — DECIDED · IN PROGRESS).** The 30 June checkpoint locked a
+> restructure (contract: [API_ENDPOINTS_RESTRUCTURE.md](API_ENDPOINTS_RESTRUCTURE.md); live tracker:
+> [DEV_PLAN §0](DEV_PLAN.md)). **Python is becoming a generation/AI-only service** — Go owns the full
+> analysis lifecycle *and* the data-plane endpoints. Scope:
+> - **Unwired from `main` + Swagger** (router files kept, *not* deleted): `analysis` CRUD, `room`, `db_client`, `document`, `data_catalog`, `users`/login. **✅ DONE — KM-686, commit `0b2d678`** (so the §7 rows for these are now commented out of `main.py`).
+> - **AI surface that stays live:** `chat` → **`POST /api/v2/chat/stream`** (explicit **`analysis_id`**, not `room_id`); the skills regroup under **`/api/v1/tools/`** (`list` · `help` · `report`); plus a **new `GET /api/v1/observability`** (Responsible-AI provenance per answer, backed by a provenance store — shape TBD). **⬜ pending.**
+> - **Only `chat/stream` moves to `/api/v2`;** everything else stays `/api/v1`.
+>
+> §2/§4/§7 below still describe the **pre-restructure wiring** except the unwire above, which has landed.
+
 ---
 
 ## 1. The product in one paragraph
@@ -31,9 +45,13 @@ streaming.
 | Repo | Role | We edit? |
 |---|---|---|
 | **Python** — `Agentic-Service-Data-Eyond-Catalog` (this repo) | The agentic LLM service: router, gate, skills, slow analytical path, structured query engine, unstructured RAG, report generation, analysis-session state. FastAPI + async SQLAlchemy + LangChain + Azure GPT-4o. | **Yes — the only repo we edit.** |
-| **Go** — `Orchestrator-Agent-Service` | Gateway / data plane: interview agent, auth/JWT, rooms, documents (Azure Blob + CSV/XLSX→Parquet + embeddings), database_clients (Fernet creds), catalog ingestion, **all DB migrations**. | Reference only. |
+| **Go** — `Orchestrator-Agent-Service` | Gateway / data plane: auth/JWT, documents (Azure Blob + CSV/XLSX→Parquet + embeddings), database_clients (Fernet creds), **catalog ingestion** (moved into Go, KM-578/590), **all dedorch SQL migrations** (now embedded in the Go repo: `internal/repository/postgres/migrations/0001–0004`), and the **full analysis-lifecycle REST surface** (`/api/v1/analyses` CRUD + messages + reports, `/api/v1/skills`). The **interview agent and chat-rooms are deprecated → HTTP 410** (`internal/api/deprecation.go`). | Reference only. |
 | **FE** — `E2E-Frontend-Data-Eyond` | React/Vite SPA. Talks to Go for everything and to Python only for chat streaming. | Reference only. |
 
+> **» pr/5 (decided, not yet in code):** Python's non-AI endpoints (analysis CRUD, `room`, `document`,
+> `db_client`, `data_catalog`, `users`/login) are being **unwired** — Python keeps only the
+> generation/AI surface (chat, tools: `help`/`report`/`list`, observability). See the Direction-update banner.
+
 Shared infra: **Postgres** (app tables + `data_catalog` jsonb + PGVector `langchain_pg_embedding`), **Azure Blob**, and (Python-only) **Redis**.
 
 ---
@@ -59,6 +77,10 @@ Tests live locally and are gitignored. Run with `./.venv/Scripts/python.exe -m p
 Entry: `POST /api/v1/chat/stream` (`src/api/v1/chat.py`) → `ChatHandler.handle(...)`
 (`src/agents/chat_handler.py`). One shared `ChatHandler` per process keeps the Azure clients warm.
 
+> **» pr/5:** this endpoint moves to **`POST /api/v2/chat/stream`** with an explicit **`analysis_id`**
+> field (replacing `room_id`), and the observability detail (planning / tool I/O / sources) moves out of
+> the stream to a separate `GET /api/v1/observability` call. See the Direction-update banner.
+
 ```
 POST /chat/stream { user_id, room_id, message }
   │  (analysis_id == room_id — one session = one analysis = one chat room)
@@ -131,6 +153,11 @@ Two facts to internalise:
 
 ## 7. API surface (this repo, all under `/api/v1`)
 
+> **» pr/5 (decided, not yet in code):** chat → `/api/v2/chat/stream` (`analysis_id`); `/tools` splits
+> into `/tools/list` + `/tools/help` + `/tools/report`; new `/api/v1/observability`; and the
+> analysis-CRUD / `room` / `users` / `document` / `db_client` / `data_catalog` rows are unwired from
+> `main` + Swagger. See the Direction-update banner.
+
 | Endpoint | Purpose | Caller |
 |---|---|---|
 | `POST /chat/stream` | Main chat SSE (router → dispatch) | FE → Go → Python (the only FE→Python call today) |
@@ -153,10 +180,18 @@ unless `SKIP_INIT_DB=true`.
 | `documents`, `databases` | uploads + DB creds (Fernet-encrypted) | Go ingestion | executor cred resolution |
 | `data_catalog` | per-user jsonb `Catalog` (Source → Table → Column) | Go ingestion / Python pipeline | CatalogReader, planner, tools |
 | `langchain_pg_embedding` | PGVector document chunks | Go ingestion | DocumentRetriever |
-| `analysis_records` | jsonb `AnalysisRecord`, one per slow-path run | slow path | ReportGenerator, report readiness |
-| `analysis` *(dedorch)* | uuid id, `owner_id`, `problem_statement`, `problem_validated`, `report_id` | `/analysis/create`, state store | gate, Help, report |
-| `reports` *(dedorch)* | uuid, `title` + markdown `content` + `version` | ReportStore | report API |
-| `data_sources` *(dedorch)* | per-analysis binding; `reference_id` = catalog source_id | `/analysis/create` | structured-flow scoping, report appendix |
+| `report_inputs` *(was `analysis_records`)* | jsonb `AnalysisRecord`, one per slow-path run; **Python-owned** | slow path | ReportGenerator, report readiness |
+| `analyses` *(dedorch, plural)* | uuid `id`, `user_id`, `analysis_title`, `objective`, `business_questions` jsonb, `status` (active\|inactive), `data_bind`(+`data_bind_version`), `report_id`, `report_collection` — **defined by Go migrations**; `problem_statement`/`problem_validated`/`owner_id` already **dropped** there (`0003`/`0004`) | Go `/api/v1/analyses`; Python state store | gate (no-op), Help, report |
+| `reports` *(dedorch)* | uuid, `analysis_id`, `user_id`, `title` + markdown `content` + `version` (UNIQUE per analysis) | Go + Python ReportStore | report API |
+| `data_sources` *(dedorch)* | per-analysis binding; `reference_id` = catalog source_id; `type ∈ document\|database` | Go `/analyses/{id}/data-bind` (+ Python `/analysis/create`) | structured-flow scoping, report appendix |
+| `analyses_messages` *(dedorch)* | the analysis chat room (`role ∈ user\|ai`); replaces deprecated `rooms`/`chat_messages` | Go `/analyses/{id}/messages` | Python chat path **not yet migrated here** (§12) |
+
+> ⚠️ **Python ORM ↔ dedorch drift (verified 2026-06-29).** Python's `AnalysisStateRow` + `state_store.py`
+> still model **`problem_statement` / `problem_validated`** and do **not** carry `objective` /
+> `business_questions`, but the Go migrations have already dropped the former and added the latter.
+> Pre-cutover this is harmless (Python runs `create_all` on its own copy); **post-`SKIP_INIT_DB`**, when
+> Python reads dedorch directly, ORM column selection on the dropped columns will break. Reconcile the
+> Python model before the connection-string cutover.
 
 **Catalog shape** (the jsonb in `data_catalog`):
 `Catalog → Source[ {source_id, source_type ∈ schema|tabular|unstructured, name, location_ref} → Table[ {table_id, name, row_count, foreign_keys[]} → Column[ {column_id, name, data_type, nullable, pii_flag, sample_values|null, stats} ] ] ]`. PII columns have `sample_values: null` so real values never enter prompts.
@@ -268,25 +303,40 @@ copies disagree with the current code on:
 | Analysis / report / gate / slow path | "Phase 2 spine only" | All built and present |
 | `analysis_id` | open question | resolved: **`analysis_id == room_id`** |
 | Report source | (newer invariant) "from records, never chat history" | confirmed: generator reads `AnalysisRecord`s |
+| Go service scope | "interview agent + ingestion; dedorch migrations live outside the repos" | Go now hosts the **dedorch migrations in-repo** + a full **`/api/v1/analyses` + `/api/v1/skills`** REST surface; interview/rooms **deprecated (410)**. (Go's own `PROJECT_SUMMARY.md`/`REPO_CONTEXT.md` are uncommitted + stale.) |
 
 ---
 
 ## 12. dedorch migration — current state
 
 The Python DB is moving from `dataeyond` → **dedorch** (Go owns dedorch migrations; Python is
-consumer-only). Current state:
-
-- Base tables already match dedorch.
-- The analysis-family models have been **renamed to dedorch** on `pr/3`: `analysis` (was
-  `analysis_states`, uuid ids), `data_sources` (was `analysis_data_sources`), `reports` (was
-  `analysis_reports`, flattened to title + markdown content + version).
-- `analysis_records` (the slow-path structured output) has **no dedorch home** — it remains a
-  Python-owned jsonb table.
-- The connection-string cutover (paired with `SKIP_INIT_DB`) is a coordinated step that has not
-  happened yet; Python still creates tables on startup until then.
-
-The dedorch migrations themselves live outside the three checked-out repos (Harry owns them), so the
-dedorch table shapes are asserted by the Python model docstrings, not visible in the Go repo here.
+consumer-only). State **re-verified against the Go source 2026-06-29**:
+
+- **The dedorch migrations now live IN the Go repo** — embedded SQL at
+  `internal/repository/postgres/migrations/0001_create_core_schema.sql … 0004_replace_chat_with_analysis_scope.sql`,
+  run on startup by `RunMigrations`. (This corrects the earlier note that the migrations were
+  invisible / asserted only by Python docstrings.) The full schema is now readable there.
+- **Go owns the analysis family end-to-end.** `analyses` / `analyses_messages` / `reports` /
+  `data_sources` / `message_sources` / `data_catalog` are created by Go migrations and served by a
+  full REST surface: `internal/api/analysis.go` (CRUD + `data-bind` w/ optimistic `expected_version`
+  + messages + reports) and `internal/api/skills.go`. `analyses` already has the **pivot shape**
+  (`objective` + `business_questions`, `status`, `data_bind`/`_version`, `report_collection`) and has
+  **dropped** `problem_statement`/`problem_validated`/`owner_id`. Migration `0004` renames the legacy
+  `rooms`/`chat_messages`/`interview_*` tables to `zdeprecated_*`.
+- **`report_inputs`** (the slow-path structured output, formerly `analysis_records`) stays
+  **Python-owned**; its finalized schema goes to Harry so the dedorch migration creates it post-cutover.
+- The connection-string cutover (paired with `SKIP_INIT_DB`) **has not happened yet**; Python still
+  runs `create_all` on its own models until then.
+
+**⚠️ Integration gap (verified — the big one).** Go's `/api/v1/analyses` and `/api/v1/skills`
+(`help` / `report`) are **placeholders that return dummy data** — the `SendMessage` / `GenerateReport`
+handlers and the skills handler explicitly note *"placeholder integrasi backend agentic … will be
+replaced by the external skills service."* **Go currently never calls Python's `/chat/stream`,
+`/report`, or any skill** (no outbound HTTP to the agentic service exists in the Go source). So today
+there are **two parallel, unconnected analysis stacks**: Go's self-contained placeholder lifecycle
+(gate: ≥3 user messages; AI replies are canned) and Python's real agentic spine (router → slow path →
+records-based report; floor: ≥1 `analyze_*` success). Wiring Go → Python is the open integration work
+(DEV_PLAN #7/#18/#25), plus reconciling the two different report gates.
 
 ---
 

main.py

@@ -7,15 +7,20 @@ from src.middlewares.logging import configure_logging, get_logger
 from src.middlewares.cors import add_cors_middleware
 from src.middlewares.rate_limit import limiter, _rate_limit_exceeded_handler
 from slowapi.errors import RateLimitExceeded
-from src.api.v1.document import router as document_router
+# --- pr/5 Phase 1: unwire non-AI routers (Go owns these now). ---
+# Routers below are commented out, NOT deleted. The router files stay alive;
+# they're just not mounted, so they also disappear from Swagger.
+# from src.api.v1.document import router as document_router          # unwired: Go handles documents
+# from src.api.v1.room import router as room_router                  # unwired: replaced by analysis_id
+# from src.api.v1.users import router as users_router                # unwired: login moved off Python
+# from src.api.v1.db_client import router as db_client_router        # unwired: Go registers DB client
+# from src.api.v1.data_catalog import router as data_catalog_router  # unwired: Go handles the catalog
+# from src.api.v1.analysis import router as analysis_router          # unwired: Go owns create/update analysis
 from src.api.v1.chat import router as chat_router
-from src.api.v1.room import router as room_router
-from src.api.v1.users import router as users_router
-from src.api.v1.db_client import router as db_client_router
-from src.api.v1.data_catalog import router as data_catalog_router
 from src.api.v1.report import router as report_router
-from src.api.v1.analysis import router as analysis_router
 from src.api.v1.tools import router as tools_router
+from src.api.v1.help import router as help_router  # pr/5 Phase 2: dedicated /tools/help
+from src.api.v2.chat import router as chat_v2_router  # pr/5 Phase 2: v2 chat pilot (analysis_id)
 from src.db.postgres.init_db import init_db
 import os
 import uvicorn
@@ -50,15 +55,18 @@ app.state.limiter = limiter
 app.add_exception_handler(RateLimitExceeded, _rate_limit_exceeded_handler)
 
 # Include routers
-app.include_router(users_router)
-app.include_router(document_router)
-app.include_router(room_router)
-app.include_router(chat_router)
-app.include_router(db_client_router)
-app.include_router(data_catalog_router)
+# --- pr/5 Phase 1: AI-only surface. Non-AI routers unwired (Go owns them). ---
+# app.include_router(users_router)         # unwired: login moved off Python
+# app.include_router(document_router)      # unwired: Go handles documents
+# app.include_router(room_router)          # unwired: replaced by analysis_id
+# app.include_router(db_client_router)     # unwired: Go registers DB client
+# app.include_router(data_catalog_router)  # unwired: Go handles the catalog
+# app.include_router(analysis_router)      # unwired: Go owns create/update analysis
+app.include_router(chat_router)        # v1 chat/stream (room_id) — kept until FE moves to v2
 app.include_router(report_router)
-app.include_router(analysis_router)
 app.include_router(tools_router)
+app.include_router(help_router)
+app.include_router(chat_v2_router)     # pr/5 Phase 2: POST /api/v2/chat/stream (analysis_id)
 
 
 @app.get("/")

src/agents/chat_handler.py

@@ -227,6 +227,55 @@ class ChatHandler:
     # Public entry
     # ------------------------------------------------------------------
 
+    async def stream_help(
+        self,
+        user_id: str,
+        analysis_id: str | None,
+        history: list[BaseMessage] | None = None,
+        message: str | None = None,
+    ) -> AsyncIterator[dict[str, Any]]:
+        """Deterministic `help` dispatch for the dedicated `/api/v1/tools/help` endpoint.
+
+        Bypasses the intent router — the slash command IS the intent, so there is no
+        classify round-trip and no misclassification risk. Streams the same guidance as
+        the `help` branch of `handle()`, reusing the warm HelpAgent + state store.
+
+        Emits SSE-style events: `sources` (always `[]` — help never references
+        documents), `chunk`*, then `done` (data left empty; the endpoint stamps the
+        `message_id`). On failure, yields a terminal `error` event.
+        """
+        # Load (or lazily create) the analysis state; fail closed to a not-validated
+        # stub so help degrades gracefully on a missing row / read error / legacy id.
+        state: AnalysisState | None = None
+        if analysis_id:
+            try:
+                state = await self._get_state_store().ensure(analysis_id, user_id)
+            except Exception as e:  # noqa: BLE001 — never block help on a state read
+                logger.warning("help state ensure failed", analysis_id=analysis_id, error=str(e))
+        if state is None:
+            state = await self._load_analysis_state(analysis_id)
+
+        # report_ready (seam #5): deterministic, never-throws (fails closed to
+        # not-ready) — the HelpAgent guard only offers generate_report when ready.
+        from .report.readiness import is_report_ready
+
+        report_ready = await is_report_ready(analysis_id, state)
+
+        yield {"event": "sources", "data": json.dumps([])}
+        try:
+            async for token in self._get_help_agent().astream(
+                state,
+                history=history,
+                message=message,
+                report_ready=report_ready,
+            ):
+                yield {"event": "chunk", "data": token}
+        except Exception as e:  # noqa: BLE001
+            logger.error("help streaming failed", user_id=user_id, error=str(e))
+            yield {"event": "error", "data": f"Help generation failed: {e}"}
+            return
+        yield {"event": "done", "data": ""}
+
     async def handle(
         self,
         message: str,

src/api/v1/help.py

@@ -0,0 +1,82 @@
+"""`help` skill endpoint — dedicated, deterministic dispatch (pr/5 Phase 2).
+
+`POST /api/v1/tools/help` streams state-aware next-step guidance over SSE. Unlike v1
+— where `/help` was reachable only by letting the intent router classify a chat
+message — this endpoint dispatches Help directly: the slash command IS the intent, so
+there is no router round-trip and no misclassification risk (contract open-Q #2,
+resolved in favour of a dedicated endpoint).
+
+Contract: `API_ENDPOINTS_RESTRUCTURE.md` §3. The SSE shape mirrors `/chat/stream`, but
+help never references documents, so `sources` is always `[]` and there are no `status`
+pings. The `done` event carries the assistant `message_id` for observability
+correlation (§7).
+
+Python is generative-only (06-25 direction): this endpoint does NOT persist the turn —
+Go owns writes to `analyses_messages`. It only generates + streams.
+"""
+
+import json
+import uuid
+from typing import Optional
+
+from fastapi import APIRouter, Depends, HTTPException
+from pydantic import BaseModel
+from sqlalchemy.ext.asyncio import AsyncSession
+from sse_starlette.sse import EventSourceResponse
+
+# Reuse the warm, process-shared ChatHandler (keeps HelpAgent + Azure clients warm)
+# and the same history loader the chat endpoint uses. `load_history` reads by
+# `analysis_id` (== room_id today); it moves to `analyses_messages` with DEV_PLAN #25.
+from src.api.v1.chat import _chat_handler, load_history
+from src.db.postgres.connection import get_db
+from src.middlewares.logging import get_logger, log_execution
+
+logger = get_logger("help_api")
+
+router = APIRouter(prefix="/api/v1/tools", tags=["Tools"])
+
+
+class HelpRequest(BaseModel):
+    user_id: str
+    analysis_id: str
+    # ⚠️ open-Q #1: Go may mint the assistant turn id and pass it; if absent, Python
+    # mints one and returns it on `done` so the FE can call /observability in parallel.
+    message_id: Optional[str] = None
+
+
+@router.post("/help")
+@log_execution(logger)
+async def help_stream(request: HelpRequest, db: AsyncSession = Depends(get_db)):
+    """Stream state-aware next-step guidance (deterministic `/help` dispatch).
+
+    SSE event sequence:
+      1. sources  — always `[]` (help never references documents)
+      2. chunk    — text fragments of the guidance
+      3. done     — `{"message_id": "..."}` for the observability lookup
+    """
+    message_id = request.message_id or f"msg_{uuid.uuid4().hex[:12]}"
+    try:
+        history = await load_history(db, request.analysis_id, limit=10)
+
+        async def stream_response():
+            async for event in _chat_handler.stream_help(
+                request.user_id,
+                request.analysis_id,
+                history=history,
+                message=None,
+            ):
+                if event["event"] == "done":
+                    # Stamp the turn id so the FE can fetch /observability for it.
+                    yield {"event": "done", "data": json.dumps({"message_id": message_id})}
+                elif event["event"] == "error":
+                    yield event
+                    return
+                else:
+                    # `sources` ([]) and `chunk` pass through unchanged.
+                    yield event
+
+        return EventSourceResponse(stream_response())
+
+    except Exception as e:
+        logger.error("Help failed", error=str(e))
+        raise HTTPException(status_code=500, detail=f"Help failed: {str(e)}")

src/api/v1/report.py

@@ -1,9 +1,10 @@
 """Report API (KM-644) — the dedicated "Generate Report" surface.
 
-NOT a chat route. The frontend button calls these endpoints directly:
-  POST /report                       generate a new version for a session
-  GET  /report/{analysis_id}         list a session's report versions
-  GET  /report/{analysis_id}/{ver}   fetch one version
+NOT a chat route. The frontend button calls these endpoints directly (pr/5: regrouped
+under /tools — Go owns the analysis lifecycle, Python only generates):
+  POST /api/v1/tools/report                       generate a new version for a session
+  GET  /api/v1/tools/report/{analysis_id}         list a session's report versions
+  GET  /api/v1/tools/report/{analysis_id}/{ver}   fetch one version
 
 Generation reads persisted AnalysisRecords + Problem Statement, makes one LLM call
 (the executive summary), and persists an immutable versioned artifact. The
@@ -27,7 +28,10 @@ from src.models.api.report import ReportVersionEntry
 
 logger = get_logger("report_api")
 
-router = APIRouter(prefix="/api/v1", tags=["Report"])
+# pr/5 Phase 2: report regrouped under the tools surface (path → /api/v1/tools/report).
+# Prefix change moves all three routes at once; same functionality, new home. The
+# "Tools" tag groups it with /tools/list + /tools/help in Swagger.
+router = APIRouter(prefix="/api/v1/tools", tags=["Tools"])
 
 _generator = ReportGenerator()
 _store = ReportStore()

src/api/v1/tools.py

@@ -130,11 +130,14 @@ _COMMAND_CATALOG: list[CommandResponse] = [
 ]
 
 
-@router.get("/tools", response_model=ListToolsResponse)
+@router.get("/tools/list", response_model=ListToolsResponse)
 @log_execution(logger)
 async def list_tools() -> ListToolsResponse:
     """List the user-invocable slash-command catalog (skills + tools).
 
     Static per deployment — safe for the Golang backend to cache.
+
+    pr/5 Phase 2: moved from `GET /api/v1/tools` to `GET /api/v1/tools/list` so the
+    skills group is `/tools/list` · `/tools/help` · `/tools/report`.
     """
     return ListToolsResponse(count=len(_COMMAND_CATALOG), tools=_COMMAND_CATALOG)

src/api/v2/__init__.py

@@ -0,0 +1,4 @@
+"""API v2 (pr/5). Only the chat pilot lives here — keyed on `analysis_id` instead of
+`room_id`. The tools group (`/tools/list|help|report`) and observability stay on v1.
+See API_ENDPOINTS_RESTRUCTURE.md §1.
+"""

src/api/v2/chat.py

@@ -0,0 +1,165 @@
+"""Chat endpoint — v2 pilot (pr/5 Phase 2).
+
+`POST /api/v2/chat/stream` is the v2 of the only FE→Python call. It is identical to
+`POST /api/v1/chat/stream` except:
+  - the request carries an explicit **`analysis_id`** (replacing v1's `room_id`). The
+    two are the same session id today (`analysis_id == room_id`), so the warm,
+    process-shared `ChatHandler` and the v1 cache/history helpers are reused unchanged.
+  - the `done` event carries the assistant **`message_id`** (minted Python-side if Go
+    omits it — contract open-Q #1), so the FE can fetch `/api/v1/observability` for the
+    turn in parallel with the stream.
+
+Only chat moves to v2; the tools group + observability stay on `/api/v1` (contract:
+API_ENDPOINTS_RESTRUCTURE.md §1).
+
+⚠️ Persistence (transitional). This mirrors v1: it still load/saves turn history via the
+analysis-keyed message tables so multi-turn context works in the playground. Moving the
+read/write to Go-owned `analyses_messages` (and making Python read-only) is DEV_PLAN #25.
+Note Sofhia's `/tools/help` is already generative-only — align chat with that under #25.
+"""
+
+import json
+import uuid
+from typing import Any
+
+from fastapi import APIRouter, Depends, HTTPException
+from pydantic import BaseModel
+from sqlalchemy.ext.asyncio import AsyncSession
+from sse_starlette.sse import EventSourceResponse
+
+# Reuse the v1 chat machinery verbatim (warm ChatHandler + cache/history helpers) so
+# v2 stays a thin field-rename over the same logic. Importing the module-private helpers
+# is the established pattern here (handlers/help.py imports `_chat_handler` the same way).
+from src.api.v1.chat import (
+    _CACHEABLE_INTENTS,
+    _chat_cache_key,
+    _chat_handler,
+    _fast_intent,
+    cache_response,
+    get_cached_response,
+    load_history,
+    save_messages,
+)
+from src.db.postgres.connection import get_db
+from src.db.redis.connection import get_redis
+from src.middlewares.logging import get_logger, log_execution
+
+logger = get_logger("chat_api_v2")
+
+router = APIRouter(prefix="/api/v2", tags=["Chat"])
+
+
+def _mint_message_id(provided: str | None) -> str:
+    """Use Go's assistant turn id when provided; else mint one (contract open-Q #1)."""
+    return provided or f"msg_{uuid.uuid4().hex[:12]}"
+
+
+class ChatRequest(BaseModel):
+    user_id: str
+    analysis_id: str
+    message: str
+    # ⚠️ open-Q #1: Go may mint + pass the assistant turn id; if absent we mint one and
+    # echo it on `done` so the FE can correlate /observability with this answer.
+    message_id: str | None = None
+
+
+@router.post("/chat/stream")
+@log_execution(logger)
+async def chat_stream(request: ChatRequest, db: AsyncSession = Depends(get_db)):
+    """Chat endpoint with streaming response (v2 — keyed on `analysis_id`).
+
+    SSE event sequence:
+      1. sources  — JSON array of source refs (table for structured; deduped
+                    document_id/page_label for unstructured; [] for chat/help/error)
+      2. status   — slow-path progress pings (optional)
+      3. chunk    — text fragments of the answer
+      4. done     — {"message_id": "..."} for the observability lookup
+    """
+    analysis_id = request.analysis_id
+    message_id = _mint_message_id(request.message_id)
+    redis = await get_redis()
+    cache_key = _chat_cache_key(analysis_id, request.user_id, request.message)
+
+    # v2 `done` always carries the turn id (v1 sent an empty `done`).
+    done_event = {"event": "done", "data": json.dumps({"message_id": message_id})}
+
+    # Redis cache hit (stateless `chat` intent only).
+    cached = await get_cached_response(redis, cache_key)
+    logger.info("cache check", cache_key=cache_key, cache_hit=cached is not None)
+    if cached:
+        logger.info("Returning cached response")
+        cached_text = cached["response"]
+        cached_sources = cached["sources"]
+        await save_messages(db, analysis_id, request.message, cached_text, sources=cached_sources)
+
+        async def stream_cached():
+            yield {"event": "sources", "data": json.dumps(cached_sources)}
+            for i in range(0, len(cached_text), 50):
+                yield {"event": "chunk", "data": cached_text[i:i + 50]}
+            yield done_event
+
+        return EventSourceResponse(stream_cached())
+
+    try:
+        # Fast intent: greetings/farewells bypass the LLM entirely.
+        direct = _fast_intent(request.message)
+        if direct:
+            await cache_response(redis, cache_key, direct, sources=[])
+            await save_messages(db, analysis_id, request.message, direct, sources=[])
+
+            async def stream_direct():
+                yield {"event": "sources", "data": json.dumps([])}
+                yield {"event": "chunk", "data": direct}
+                yield done_event
+
+            return EventSourceResponse(stream_direct())
+
+        history = await load_history(db, analysis_id, limit=10)
+        handler = _chat_handler
+
+        async def stream_response():
+            logger.info("stream_response started", analysis_id=analysis_id, user_id=request.user_id)
+            full_response = ""
+            sources: list[dict[str, Any]] = []
+            effective_intent: str | None = None
+            async for event in handler.handle(
+                request.message, request.user_id, history, analysis_id=analysis_id
+            ):
+                if event["event"] == "intent":
+                    # consumed internally (not forwarded); gates caching below.
+                    try:
+                        effective_intent = json.loads(event["data"]).get("intent")
+                    except (TypeError, ValueError, AttributeError):
+                        effective_intent = None
+                elif event["event"] == "sources":
+                    try:
+                        sources = json.loads(event["data"]) or []
+                    except (TypeError, ValueError):
+                        sources = []
+                    yield event
+                elif event["event"] == "chunk":
+                    full_response += event["data"]
+                    yield event
+                elif event["event"] == "done":
+                    # Only cache stateless `chat` replies (see _CACHEABLE_INTENTS).
+                    if effective_intent in _CACHEABLE_INTENTS:
+                        await cache_response(redis, cache_key, full_response, sources=sources)
+                    try:
+                        await save_messages(
+                            db, analysis_id, request.message, full_response, sources=sources
+                        )
+                    except Exception as e:
+                        logger.error("save_messages failed", analysis_id=analysis_id, error=str(e))
+                    yield done_event
+                elif event["event"] == "status":
+                    # slow-path progress: forward so the client shows activity.
+                    yield event
+                elif event["event"] == "error":
+                    yield event
+                    return
+
+        return EventSourceResponse(stream_response())
+
+    except Exception as e:
+        logger.error("Chat failed", error=str(e))
+        raise HTTPException(status_code=500, detail=f"Chat failed: {str(e)}") from e