API_ENDPOINTS_RESTRUCTURE.md

Backend Agentic Service — API Endpoint Docs (endpoint restructure)

Status: contract draft for FE/Go integration (2026-06-30). Covers the AI-only surface after the restructure. Sections marked TENTATIVE (observability) may still change — send feedback before we lock them.

What changed

• Only the chat pilot moves to /api/v2. Everything else stays on /api/v1, regrouped under /tools.
• Chat pilot (/api/v2/chat/stream) uses analysis_id, not room_id.
• Skills are grouped under /api/v1/tools: list / help / report.
• New: GET /api/v1/observability — Responsible-AI provenance per chat answer.
• Python is generative-AI only. It never creates/updates an analysis, room, document, DB client, or catalog — Go owns those. Python just receives analysis_id. Those v1 routers are unwired from main + Swagger (not deleted).

Open coordination questions (need a decision with Harry) — flagged inline as ⚠️:

1. message_id origin — who mints the assistant turn id used to correlate stream ↔ observability? RESOLVED (pr/6): Python is the sole minter. It is not a request field; Python always mints it (server-authoritative — keeps the /observability correlation key out of client control, for FE-security) and returns it on the done event. Go/FE read it off the stream. (The /observability consumer itself is future work — a later PR.)
2. Deterministic /help dispatch — dedicated endpoint (recommended below) vs router classification.
3. Observability storage — single JSONB row per message (recommended) vs 3 normalized tables.

---

1. call_agent — POST /api/v2/chat/stream

The only FE→Python call in normal operation. Same as v1 except room_id → analysis_id, and the done event now carries the assistant message_id for observability correlation.

Request body (application/json) — ChatRequest:

{
  "user_id": "u_1a2b3c",
  "analysis_id": "an_42",
  "message": "What were total sales by region last quarter?"
}

• analysis_id is the analysis-session id (replaces room_id). No auth header (handled by Go).
• message_id is not a request field. Python always mints the assistant turn id (server-authoritative, for FE-security) and returns it on done; the FE reads it there and passes it to /api/v1/observability?message_id=.... Any message_id a caller sends is ignored. (The /observability endpoint is future work — a later PR; §7 is a forward-looking sketch.)

Response: text/event-stream. Events arrive in this order:

event	data	notes
sources	JSON array of {document_id, filename, page_label}	structured: one per executed table; unstructured: deduped doc/page; chat/help/error: [].
status	text	slow-path only — progress pings ("Planning…", "Running N steps…"). Safe to surface or ignore.
chunk	text fragment	concatenate in order to form the answer.
done	{"message_id": "..."}	v2 change: was empty; now returns the turn id for the observability lookup.
error	text	terminal error; stream stops after this.

The internal intent event is consumed inside Python (gates caching) and not forwarded.

Stream carries the answer text ONLY. Planning / tool calls / sources detail are not in the stream (it would slow it down) — fetch them from /observability (§7), called in parallel.

Example — structured_flow answer (raw SSE; blank line separates events):

event: sources
data: [{"document_id":"u_1a2b3c_orders","filename":"orders","page_label":null}]

event: status
data: Planning analysis…

event: status
data: Running 3 steps…

event: chunk
data: Total sales by region last quarter: 

event: chunk
data: Central led at $1.21M (38%), East $0.74M, West $0.55M (down 12% QoQ).

event: done
data: {"message_id":"msg_88f1"}

Example — simple chat reply (no status pings, empty sources):

event: sources
data: []

event: chunk
data: I'm your AI data analyst — connect a source or ask a question to get started.

event: done
data: {"message_id":"msg_12"}

Behavior unchanged from v1: 1h Redis response-cache on the stateless chat intent only; greeting/farewell fast-path (canned, no LLM); LLM router classifies every message into one of 5 intents (chat · help · check · unstructured_flow · structured_flow); messages persist on done.

---

2. list_skills — GET /api/v1/tools/list

Static, deterministic, safe for Go to cache. (Was GET /api/v1/tools.)

Request: none.

Response 200 (ListToolsResponse):

{
  "count": 2,
  "tools": [
    { "command": "/help",   "name": "help",   "type": "skill",
      "description": "Show what the assistant can do and guide your next step." },
    { "command": "/report", "name": "report", "type": "skill",
      "description": "Generate a versioned analysis report (background, EDA, key findings, insights)." }
  ]
}

CommandResponse = { command, name, type, description }, type ∈ {skill, analytics, data_access}. Catalog is /help + /report only; the analyze_* / check_* / retrieve_* and retired /problem-statement entries are commented out (kept for restorability), not deleted.

FE behavior: the / slash menu surfaces /help only. Report is a right-side button, not a slash command (it fires only when an analysis is finished — saves tokens).

---

3. skill: help — POST /api/v1/tools/help

⚠️ Proposed dedicated endpoint (new in v2). In v1 there was no /help endpoint — help was reached only by letting the LLM router classify a chat message. A dedicated endpoint makes /help dispatch deterministic (no risk the router mis-classifies the slash command) and gives it a clean home in the tools group. State-aware: reads analysis state + history to guide the next step.

Alternative if we don't add this endpoint: FE keeps calling POST /chat/stream and trusts the router to classify the help intent. We recommend the dedicated endpoint — decision pending (open question #2).

Request body (application/json):

{
  "user_id": "u_1a2b3c",
  "analysis_id": "an_42"
}

Response: text/event-stream — same SSE shape as chat, with sources: [] and no status pings (help never references documents). Streams a next-step guidance reply.

event: sources
data: []

event: chunk
data: Your goal is set — you can start exploring now. Try a question like "average order value by month", then I can generate a report.

event: done
data: {"message_id":"msg_h7"}

---

4. skill: report — POST /api/v1/tools/report

The "Generate Report" button. Same as v1, moved under /tools. Generate, persist, and return a new report version. Currently renders Markdown (FE preview); PPT/PDF/infographic export is future work (triggered on a download button, not here).

Query params: analysis_id (required), user_id (required). No request body.

POST /api/v1/tools/report?analysis_id=an_42&user_id=u_1a2b3c

status	meaning
201	new version generated → AnalysisReport body.
409	floor not met — no recorded analyses yet for this session, nothing to report.
500	generation or persistence failed.

201 response (AnalysisReport):

{
  "report_id": "8f3a2b1c9d4e4f6a8b0c1d2e3f4a5b6c",
  "analysis_id": "an_42",
  "user_id": "u_1a2b3c",
  "version": 2,
  "generated_at": "2026-06-30T09:14:33.512Z",
  "problem_statement": {
    "objective": "Understand which regions drive revenue and why Q1 dipped.",
    "business_questions": [
      "Which regions contribute most to total revenue?",
      "Did any region decline quarter-over-quarter?"
    ]
  },
  "record_ids": ["rec_a1", "rec_b2"],
  "executive_summary": "Revenue is concentrated in the Central region (38% of total). The West was the only region to contract, down 12% QoQ — the main driver of the Q1 dip.",
  "findings": [
    { "text": "Central region contributed 38% of total revenue, the largest share.",
      "record_ids": ["rec_a1"], "supporting_data": null },
    { "text": "West region revenue fell 12% quarter-over-quarter.",
      "record_ids": ["rec_b2"], "supporting_data": null }
  ],
  "caveats": [
    { "text": "March data for the East region was partially missing (~6% of rows).",
      "record_ids": ["rec_b2"] }
  ],
  "open_questions": [
    { "text": "What drove the West region's QoQ decline?", "record_ids": ["rec_b2"] }
  ],
  "data_sources": [
    { "source_id": "src_sales_db", "name": "orders", "source_type": "postgres",
      "detail": { "tables": ["orders"], "row_count": 48213,
                  "columns": ["region", "amount", "ordered_at"] } }
  ],
  "method_steps": [
    { "task_id": "t1", "stage": "data_understanding", "objective": "Inventory the sales source",
      "status": "success", "tools_used": ["check_data"] },
    { "task_id": "t2", "stage": "modeling", "objective": "Aggregate revenue by region",
      "status": "success", "tools_used": ["analyze_aggregate"] }
  ],
  "rendered_markdown": "# Analysis Report\n\n*Generated 2026-06-30 by u_1a2b3c · 2 analyses · 1 source(s)*\n\n## Objective\nUnderstand which regions drive revenue…\n\n## Key Findings\n1. Central region contributed 38%…"
}

409 response (floor not met — the demo's most common error):

{ "detail": "Not ready to generate a report — still needs at least one completed analysis." }

⚠️ Precondition: AnalysisRecords persist only on the slow path, so reports require ENABLE_SLOW_PATH=true on the Python deployment and ≥1 prior structured_flow question in the session. With slow path off, POST 409s by design.

---

5. report versions — GET /api/v1/tools/report/{analysis_id} and /{analysis_id}/{version}

List a session's report versions (oldest-first). Returns [ReportVersionEntry]; [] if none.

[
  { "report_id": "1b2c3d4e…", "version": 1, "generated_at": "2026-06-24T15:02:11Z", "record_count": 1 },
  { "report_id": "8f3a2b1c…", "version": 2, "generated_at": "2026-06-25T09:14:33Z", "record_count": 2 }
]

GET /api/v1/tools/report/{analysis_id}/{version} → one AnalysisReport (same shape as the POST 201 body); 404 if that version doesn't exist:

{ "detail": "No report v3 for analysis 'an_42'." }

---

6. Unwired in v2 (mounted in v1, OFF in v2)

Commented out of main + Swagger, files kept. Go owns these; Python is generative-only: POST /analysis/create + analysis CRUD · room · db_client · document · data_catalog · users/login. Re-mounting is a one-line include_router if ever needed.

---

7. observability — GET /api/v1/observability (NEW · TENTATIVE)

Responsible-AI provenance for one chat answer. Separate endpoint, called in parallel with the stream — never embedded in it. The FE renders it as a collapsed dropdown the user can expand (planning / tool calls / sources), Claude/Codex-style.

Design (recommended): one endpoint returns one merged object, backed by one JSONB row per message written by an accumulating "scratchpad" decorator inside the chat agent and flushed on done. The 3 facets (planning / tool_calls / sources) are logical sections of the JSON, not separate tables — so the shape can evolve without a dedorch migration each time. (Storage is open question #3.)

Query params: analysis_id (required), message_id (required).

GET /api/v1/observability?analysis_id=an_42&message_id=msg_88f1

Timing: the row is written when the turn finishes, so call this after the stream's done event (or poll until 200). "Parallel" = a separate call the FE fires alongside the stream, not data embedded in the stream.

Field rules (by intent):

• planning — present only when the planner ran (slow path); null otherwise.
• tool_calls — every tool invoked, with input + output. [] for pure chat / greeting / help.
• sources — required for retrieve flows (structured_flow, unstructured_flow). Empty for greeting / chat / help (they don't reference documents).
• thinking — optional reasoning text; null if none.

200 response — full structured_flow turn (planner ran → all sections present):

{
  "analysis_id": "an_42",
  "message_id": "msg_88f1",
  "intent": "structured_flow",
  "generated_at": "2026-06-30T03:21:09.114Z",
  "planning": {
    "goal_restated": "Find which regions drive revenue and why Q1 dipped.",
    "assumptions": ["'last quarter' = Q1 2026"],
    "steps": [
      { "step": 1, "stage": "data_understanding", "objective": "Inventory the sales source" },
      { "step": 2, "stage": "modeling", "objective": "Aggregate revenue by region" }
    ]
  },
  "thinking": "The question needs a per-region breakdown plus a cause, so I inventory the source, aggregate revenue by region, then compare quarters.",
  "tool_calls": [
    {
      "order": 1,
      "name": "check_data",
      "input": { "source_hint": "structured" },
      "output": { "kind": "table", "summary": "1 source · 1 table (orders) · 48,213 rows" },
      "status": "success"
    },
    {
      "order": 2,
      "name": "retrieve_data",
      "input": { "source_id": "src_sales_db", "table_id": "orders",
                 "select": ["region", "amount"], "group_by": ["region"] },
      "output": { "kind": "table", "columns": ["region", "total"], "row_count": 4,
                  "preview": [["Central", 1210000], ["East", 740000]] },
      "status": "success"
    }
  ],
  "sources": [
    {
      "type": "database",
      "source_id": "src_sales_db",
      "name": "orders",
      "query": "SELECT region, SUM(amount) AS total FROM orders GROUP BY region",
      "detail": { "tables": ["orders"], "row_count": 48213 }
    }
  ]
}

200 response — unstructured_flow turn (no planner; source = document, with the retrieval query):

{
  "analysis_id": "an_42",
  "message_id": "msg_55",
  "intent": "unstructured_flow",
  "generated_at": "2026-06-30T03:40:02.001Z",
  "planning": null,
  "thinking": null,
  "tool_calls": [
    { "order": 1, "name": "retrieve_knowledge",
      "input": { "query": "technology stack used in this project", "top_k": 4 },
      "output": { "kind": "documents", "row_count": 4 }, "status": "success" }
  ],
  "sources": [
    { "type": "document", "document_id": "doc_7", "filename": "tech_handbook.pdf",
      "page_label": "12", "query": "technology stack used in this project",
      "snippet": "The backend is built on FastAPI with async SQLAlchemy…", "score": 0.83 }
  ]
}

200 response — simple chat / greeting turn (nothing to trace):

{
  "analysis_id": "an_42",
  "message_id": "msg_12",
  "intent": "chat",
  "generated_at": "2026-06-30T03:05:00.000Z",
  "planning": null,
  "thinking": null,
  "tool_calls": [],
  "sources": []
}

404 — no provenance for that message yet (turn still running or unknown id):

{ "detail": "No observability for message 'msg_88f1' yet." }

⚠️ Richness is path-dependent. Full planning + tool I/O exist only when ENABLE_SLOW_PATH=true. Fast chat / single-query / help still record sources + the single tool call but have planning: null. This matches the rule "planning only when the planner runs."