/fix message_id for chat/stream Endpoint

API_ENDPOINTS_RESTRUCTURE.md

@@ -14,7 +14,7 @@ lock them.
   unwired from `main` + Swagger (not deleted).
 
 **Open coordination questions (need a decision with Harry) — flagged inline as ⚠️:**
-1. **`message_id` origin** — who mints the assistant turn id used to correlate stream ↔ observability? (Recommend: Go mints it, passes it in the chat request, Python echoes on `done`.)
+1. ~~**`message_id` origin** — who mints the assistant turn id used to correlate stream ↔ observability?~~ **RESOLVED (pr/6):** **Python is the sole minter.** It is not a request field; Python always mints it (server-authoritative — keeps the `/observability` correlation key out of client control, for FE-security) and returns it on the `done` event. Go/FE read it off the stream. *(The `/observability` consumer itself is future work — a later PR.)*
 2. **Deterministic `/help` dispatch** — dedicated endpoint (recommended below) vs router classification.
 3. **Observability storage** — single JSONB row per message (recommended) vs 3 normalized tables.
 
@@ -31,15 +31,15 @@ the `done` event now carries the assistant `message_id` for observability correl
 {
   "user_id": "u_1a2b3c",
   "analysis_id": "an_42",
-  "message_id": "msg_88f1",
   "message": "What were total sales by region last quarter?"
 }
 ```
 
 - `analysis_id` is the analysis-session id (replaces `room_id`). No auth header (handled by Go).
-- ⚠️ `message_id` (optional): the assistant turn id. **Recommended: Go mints it** alongside the
-  `analyses_messages` row and passes it here, so the FE can call `/api/v1/observability?message_id=...`
-  in parallel. If omitted, Python mints one and returns it on `done`.
+- `message_id` is **not** a request field. Python always mints the assistant turn id
+  (server-authoritative, for FE-security) and returns it on `done`; the FE reads it there and passes
+  it to `/api/v1/observability?message_id=...`. Any `message_id` a caller sends is ignored. *(The
+  `/observability` endpoint is future work — a later PR; §7 is a forward-looking sketch.)*
 
 **Response:** `text/event-stream`. Events arrive in this order:
 

DEV_PLAN.md

@@ -24,9 +24,9 @@ the endpoint contract *before* coding the tools. Status legend: ⬜ not started
 | **P0 — contract** | Draft + send endpoint contract to Harry (chat v2 · tools group · observability) | Rifqi + Sofhia | ✅ | `API_ENDPOINTS_RESTRUCTURE.md` sent 2026-06-30 (before-noon deadline met). Observability section flagged tentative. |
 | **1 — unwire** | Unwire `users`(login)/`document`/`room`/`db_client`/`data_catalog`/`analysis` from `main` + Swagger | Sofhia | ✅ | **KM-686**, commit `0b2d678`. Commented, not deleted; `chat`/`report`/`tools` kept mounted. Resolves the analysis-CRUD scope Q — whole `analysis` router unwired (Go owns it). |
 | **2 — v2 + regroup** | Create `src/api/v2/` and move the chat pilot there | Rifqi | ✅ | New `src/api/v2/__init__.py` + `src/api/v2/chat.py` (`POST /api/v2/chat/stream`), mounted in `main.py`. Only chat in v2; v1 `/chat/stream` kept mounted until FE moves over. Routes import-verified. |
-| **2 — v2 + regroup** | Chat: `room_id` → **`analysis_id`** (request field + handler + history) | Rifqi | ✅ | v2 `ChatRequest{user_id, analysis_id, message, message_id?}`; reuses warm `ChatHandler` + v1 cache/history helpers; `done` returns `{message_id}` (minted Python-side if Go omits, open-Q #1). Persistence kept transitionally → still ties to #25 (`analyses_messages`); ruff-clean. |
+| **2 — v2 + regroup** | Chat: `room_id` → **`analysis_id`** (request field + handler + history) | Rifqi | ✅ | v2 `ChatRequest{user_id, analysis_id, message}`; reuses warm `ChatHandler` + v1 cache/history helpers; `done` returns `{message_id}` (always minted Python-side, server-authoritative — open-Q #1 resolved in pr/6). Persistence kept transitionally → still ties to #25 (`analyses_messages`); ruff-clean. |
 | **2 — v2 + regroup** | Move report under tools → `/api/v1/tools/report` (+ version routes) | Rifqi | ✅ | report router re-prefixed `/api/v1` → `/api/v1/tools` (all 3 routes move together), tag → `Tools`; old `/api/v1/report` gone. Same functionality, new home. Import-verified. |
-| **2 — v2 + regroup** | Move help under tools → `POST /api/v1/tools/help` (dedicated endpoint) | Sofhia | ✅ | New `src/api/v1/help.py` (SSE: `sources:[]`→`chunk`→`done{message_id}`) + additive `ChatHandler.stream_help()` (reuses HelpAgent+state+readiness, no router). Generative-only (no persist). **Router `help` intent KEPT** — both paths live by design. message_id minted Python-side if Go omits (open-Q #1). Import-verified. |
+| **2 — v2 + regroup** | Move help under tools → `POST /api/v1/tools/help` (dedicated endpoint) | Sofhia | ✅ | New `src/api/v1/help.py` (SSE: `sources:[]`→`chunk`→`done{message_id}`) + additive `ChatHandler.stream_help()` (reuses HelpAgent+state+readiness, no router). Generative-only (no persist). **Router `help` intent KEPT** — both paths live by design. message_id always minted Python-side, server-authoritative (open-Q #1 resolved, pr/6). Import-verified. |
 | **2 — v2 + regroup** | Tools list → `/api/v1/tools/list` | Sofhia | ✅ | Renamed route `GET /api/v1/tools` → `GET /api/v1/tools/list` ([tools.py:133](src/api/v1/tools.py:133)). |
 | **2 — v2 + regroup** | FE: slash menu = `/help` only; report = right-side button | Mentor (FE) | ⬜ | Coordination note, not Python work. |
 | **3 — tools + obs** | Finish `help` so it actually **calls** (not just lists) + test | Sofhia | ⬜ | Mentor: help currently only lists tools. Core #2 after chat. |
@@ -34,14 +34,16 @@ the endpoint contract *before* coding the tools. Status legend: ⬜ not started
 | **3 — tools + obs** | Audit `report_inputs` — covers planning + tool I/O + source? add cols / new store | Rifqi | ⬜ | **Rec:** dedicated provenance store = 1 JSONB row per message (logical 3 sections); keep Langfuse for engineering. |
 | **3 — tools + obs** | Build `GET /api/v1/observability` (one merged response) | Rifqi | ⬜ | Intent-based source rules (greeting/help = none; retrieve = required). Richness path-dependent (full planning only on slow path). |
 | **3 — tools + obs** | Keep stream **text-only**; observability is a separate parallel call | Rifqi | ⬜ | Per mentor — don't slow the stream. |
-| **3 — tools + obs** | Resolve `message_id` correlation (stream ↔ observability) with Harry | Rifqi ↔ Harry | ⬜ | **Rec:** Go mints `message_id`, passes in chat request, Python echoes on `done`. |
+| **3 — tools + obs** | Resolve `message_id` correlation (stream ↔ observability) with Harry | Rifqi ↔ Harry | ✅ | **RESOLVED (pr/6):** Python is the **sole minter** — `message_id` dropped from the `/api/v2/chat/stream` + `/api/v1/tools/help` request bodies; always minted server-side (server-authoritative, FE-security) and returned on `done`. Any caller-sent `message_id` is ignored. Contract open-Q #1 closed. |
 | **4 — biz questions** | Get Go folder; confirm `business_questions` in create-analysis (max 5); sync Python | Harry/Mentor → Rifqi | ⬜ | Go currently missing the field ("lagi difixing"). Python already models objective + business_questions. |
 | **deferred** | Report formats: PPT (preferred) / PDF / infographic on download | — | ⏸️ | MD is fine for the FE preview stage now. |
 | **deferred** | Charts (Plotly→JSON) + images tables | — | ⏸️ | Carried from §4 #26/#27. |
 
 **Next up:** Phase 2 Python work is **done** (chat→v2 `analysis_id`; `help`/`report`/`list` regrouped
-under `/api/v1/tools/`). Remaining: **Phase 3** — the observability scratchpad + `GET /api/v1/observability`
-(shape already speced in the contract), then **Phase 4** (business questions, Go-blocked).
+under `/api/v1/tools/`). The `message_id` correlation contract is now settled in **pr/6** (Python sole
+minter, stream-only). The full **Phase 3 observability build** — scratchpad + `GET /api/v1/observability`
+(shape sketched in the contract §7) — is slated for a **later PR**, then **Phase 4** (business
+questions, Go-blocked).
 
 ---
 

main.py

@@ -16,7 +16,8 @@ from slowapi.errors import RateLimitExceeded
 # from src.api.v1.db_client import router as db_client_router        # unwired: Go registers DB client
 # from src.api.v1.data_catalog import router as data_catalog_router  # unwired: Go handles the catalog
 # from src.api.v1.analysis import router as analysis_router          # unwired: Go owns create/update analysis
-from src.api.v1.chat import router as chat_router
+# from src.api.v1.chat import router as chat_router  # unwired: replaced by /api/v2/chat/stream
+# NOTE: src.api.v1.chat module still imported by v2 chat + /tools/help — keep the file.
 from src.api.v1.report import router as report_router
 from src.api.v1.tools import router as tools_router
 from src.api.v1.help import router as help_router  # pr/5 Phase 2: dedicated /tools/help
@@ -62,7 +63,7 @@ app.add_exception_handler(RateLimitExceeded, _rate_limit_exceeded_handler)
 # app.include_router(db_client_router)     # unwired: Go registers DB client
 # app.include_router(data_catalog_router)  # unwired: Go handles the catalog
 # app.include_router(analysis_router)      # unwired: Go owns create/update analysis
-app.include_router(chat_router)        # v1 chat/stream (room_id) — kept until FE moves to v2
+# app.include_router(chat_router)  # unwired: v2 chat replaces it (drops v1 cache ops routes)
 app.include_router(report_router)
 app.include_router(tools_router)
 app.include_router(help_router)

src/api/v1/help.py

@@ -8,8 +8,9 @@ resolved in favour of a dedicated endpoint).
 
 Contract: `API_ENDPOINTS_RESTRUCTURE.md` §3. The SSE shape mirrors `/chat/stream`, but
 help never references documents, so `sources` is always `[]` and there are no `status`
-pings. The `done` event carries the assistant `message_id` for observability
-correlation (§7).
+pings. The `done` event carries the assistant `message_id` — always minted Python-side,
+never accepted from the caller (server-authoritative; keys the future /observability
+lookup, §7).
 
 Python is generative-only (06-25 direction): this endpoint does NOT persist the turn —
 Go owns writes to `analyses_messages`. It only generates + streams.
@@ -17,7 +18,6 @@ Go owns writes to `analyses_messages`. It only generates + streams.
 
 import json
 import uuid
-from typing import Optional
 
 from fastapi import APIRouter, Depends, HTTPException
 from pydantic import BaseModel
@@ -39,9 +39,6 @@ router = APIRouter(prefix="/api/v1/tools", tags=["Tools"])
 class HelpRequest(BaseModel):
     user_id: str
     analysis_id: str
-    # ⚠️ open-Q #1: Go may mint the assistant turn id and pass it; if absent, Python
-    # mints one and returns it on `done` so the FE can call /observability in parallel.
-    message_id: Optional[str] = None
 
 
 @router.post("/help")
@@ -54,7 +51,8 @@ async def help_stream(request: HelpRequest, db: AsyncSession = Depends(get_db)):
       2. chunk    — text fragments of the guidance
       3. done     — `{"message_id": "..."}` for the observability lookup
     """
-    message_id = request.message_id or f"msg_{uuid.uuid4().hex[:12]}"
+    # Server-authoritative turn id — never accepted from the caller (keys /observability).
+    message_id = f"msg_{uuid.uuid4().hex[:12]}"
     try:
         history = await load_history(db, request.analysis_id, limit=10)
 
@@ -79,4 +77,4 @@ async def help_stream(request: HelpRequest, db: AsyncSession = Depends(get_db)):
 
     except Exception as e:
         logger.error("Help failed", error=str(e))
-        raise HTTPException(status_code=500, detail=f"Help failed: {str(e)}")
+        raise HTTPException(status_code=500, detail=f"Help failed: {str(e)}") from e

src/api/v2/chat.py

@@ -5,9 +5,9 @@
   - the request carries an explicit **`analysis_id`** (replacing v1's `room_id`). The
     two are the same session id today (`analysis_id == room_id`), so the warm,
     process-shared `ChatHandler` and the v1 cache/history helpers are reused unchanged.
-  - the `done` event carries the assistant **`message_id`** (minted Python-side if Go
-    omits it — contract open-Q #1), so the FE can fetch `/api/v1/observability` for the
-    turn in parallel with the stream.
+  - the `done` event carries the assistant **`message_id`**. It is always minted
+    Python-side and is **never accepted from the caller** (server-authoritative — it keys
+    the future `/observability` lookup; open-Q #1 resolved). The FE reads it off `done`.
 
 Only chat moves to v2; the tools group + observability stay on `/api/v1` (contract:
 API_ENDPOINTS_RESTRUCTURE.md §1).
@@ -49,18 +49,16 @@ logger = get_logger("chat_api_v2")
 router = APIRouter(prefix="/api/v2", tags=["Chat"])
 
 
-def _mint_message_id(provided: str | None) -> str:
-    """Use Go's assistant turn id when provided; else mint one (contract open-Q #1)."""
-    return provided or f"msg_{uuid.uuid4().hex[:12]}"
+def _mint_message_id() -> str:
+    """Mint the assistant turn id. Server-authoritative — never accepted from the caller
+    (it keys the future /observability lookup). Returned on `done`; open-Q #1 resolved."""
+    return f"msg_{uuid.uuid4().hex[:12]}"
 
 
 class ChatRequest(BaseModel):
     user_id: str
     analysis_id: str
     message: str
-    # ⚠️ open-Q #1: Go may mint + pass the assistant turn id; if absent we mint one and
-    # echo it on `done` so the FE can correlate /observability with this answer.
-    message_id: str | None = None
 
 
 @router.post("/chat/stream")
@@ -76,7 +74,7 @@ async def chat_stream(request: ChatRequest, db: AsyncSession = Depends(get_db)):
       4. done     — {"message_id": "..."} for the observability lookup
     """
     analysis_id = request.analysis_id
-    message_id = _mint_message_id(request.message_id)
+    message_id = _mint_message_id()
     redis = await get_redis()
     cache_key = _chat_cache_key(analysis_id, request.user_id, request.message)