API_ENDPOINTS_RESTRUCTURE.md

# Backend Agentic Service — API Endpoint Docs (endpoint restructure)

**Status:** contract draft for FE/Go integration (2026-06-30). Covers the AI-only surface after the
restructure. Sections marked **TENTATIVE** (observability) may still change — send feedback before we
lock them.

**What changed**
- **Only the chat pilot moves to `/api/v2`.** Everything else stays on `/api/v1`, regrouped under `/tools`.
- **Chat pilot (`/api/v2/chat/stream`) uses `analysis_id`, not `room_id`.**
- **Skills are grouped under `/api/v1/tools`:** `list` / `help` / `report`.
- **New:** `GET /api/v1/observability` — Responsible-AI provenance per chat answer.
- Python is **generative-AI only.** It never creates/updates an analysis, room, document, DB
  client, or catalog — Go owns those. Python just receives `analysis_id`. Those v1 routers are
  unwired from `main` + Swagger (not deleted).

**Open coordination questions (need a decision with Harry) — flagged inline as ⚠️:**
1. ~~**`message_id` origin** — who mints the assistant turn id used to correlate stream ↔ observability?~~ **RESOLVED (pr/6):** **Python is the sole minter.** It is not a request field; Python always mints it (server-authoritative — keeps the `/observability` correlation key out of client control, for FE-security) and returns it on the `done` event. Go/FE read it off the stream. *(The `/observability` consumer itself is future work — a later PR.)*
2. **Deterministic `/help` dispatch** — dedicated endpoint (recommended below) vs router classification.
3. **Observability storage** — single JSONB row per message (recommended) vs 3 normalized tables.

---

## 1. call_agent — `POST /api/v2/chat/stream`

The only FE→Python call in normal operation. Same as v1 except **`room_id` → `analysis_id`**, and
the `done` event now carries the assistant `message_id` for observability correlation.

**Request body** (`application/json`) — `ChatRequest`:

```json
{
  "user_id": "u_1a2b3c",
  "analysis_id": "an_42",
  "message": "What were total sales by region last quarter?"
}
```

- `analysis_id` is the analysis-session id (replaces `room_id`). No auth header (handled by Go).
- `message_id` is **not** a request field. Python always mints the assistant turn id
  (server-authoritative, for FE-security) and returns it on `done`; the FE reads it there and passes
  it to `/api/v1/observability?message_id=...`. Any `message_id` a caller sends is ignored. *(The
  `/observability` endpoint is future work — a later PR; §7 is a forward-looking sketch.)*

**Response:** `text/event-stream`. Events arrive in this order:

| event | data | notes |
|---|---|---|
| `sources` | JSON array of `{document_id, filename, page_label}` | structured: one per executed table; unstructured: deduped doc/page; chat/help/error: `[]`. |
| `status` | text | slow-path only — progress pings ("Planning…", "Running N steps…"). Safe to surface or ignore. |
| `chunk` | text fragment | concatenate in order to form the answer. |
| `done` | `{"message_id": "..."}` | **v2 change:** was empty; now returns the turn id for the observability lookup. |
| `error` | text | terminal error; stream stops after this. |

The internal `intent` event is consumed inside Python (gates caching) and **not** forwarded.

**Stream carries the answer text ONLY.** Planning / tool calls / sources detail are **not** in the
stream (it would slow it down) — fetch them from `/observability` (§7), called in parallel.

**Example — `structured_flow` answer** (raw SSE; blank line separates events):

```
event: sources
data: [{"document_id":"u_1a2b3c_orders","filename":"orders","page_label":null}]

event: status
data: Planning analysis…

event: status
data: Running 3 steps…

event: chunk
data: Total sales by region last quarter: 

event: chunk
data: Central led at $1.21M (38%), East $0.74M, West $0.55M (down 12% QoQ).

event: done
data: {"message_id":"msg_88f1"}
```

**Example — simple chat reply** (no status pings, empty sources):

```
event: sources
data: []

event: chunk
data: I'm your AI data analyst — connect a source or ask a question to get started.

event: done
data: {"message_id":"msg_12"}
```

Behavior unchanged from v1: 1h Redis response-cache on the stateless `chat` intent only;
greeting/farewell fast-path (canned, no LLM); LLM router classifies every message into one of 5
intents (`chat · help · check · unstructured_flow · structured_flow`); messages persist on `done`.

---

## 2. list_skills — `GET /api/v1/tools/list`

Static, deterministic, safe for Go to cache. (Was `GET /api/v1/tools`.)

**Request:** none.

**Response 200** (`ListToolsResponse`):

```json
{
  "count": 2,
  "tools": [
    { "command": "/help",   "name": "help",   "type": "skill",
      "description": "Show what the assistant can do and guide your next step." },
    { "command": "/report", "name": "report", "type": "skill",
      "description": "Generate a versioned analysis report (background, EDA, key findings, insights)." }
  ]
}
```

`CommandResponse = { command, name, type, description }`, `type ∈ {skill, analytics, data_access}`.
Catalog is `/help` + `/report` only; the `analyze_*` / `check_*` / `retrieve_*` and retired
`/problem-statement` entries are commented out (kept for restorability), not deleted.

**FE behavior:** the `/` slash menu surfaces **`/help` only**. **Report is a right-side button, not
a slash command** (it fires only when an analysis is finished — saves tokens).

---

## 3. skill: help — `POST /api/v1/tools/help`

⚠️ **Proposed dedicated endpoint** (new in v2). In v1 there was no `/help` endpoint — help was reached
only by letting the LLM router classify a chat message. A dedicated endpoint makes `/help` dispatch
**deterministic** (no risk the router mis-classifies the slash command) and gives it a clean home in
the tools group. State-aware: reads analysis state + history to guide the next step.

> Alternative if we *don't* add this endpoint: FE keeps calling `POST /chat/stream` and trusts the
> router to classify the help intent. We recommend the dedicated endpoint — decision pending (open
> question #2).

**Request body** (`application/json`):

```json
{
  "user_id": "u_1a2b3c",
  "analysis_id": "an_42"
}
```

**Response:** `text/event-stream` — same SSE shape as chat, with `sources: []` and no `status`
pings (help never references documents). Streams a next-step guidance reply.

```
event: sources
data: []

event: chunk
data: Your goal is set — you can start exploring now. Try a question like "average order value by month", then I can generate a report.

event: done
data: {"message_id":"msg_h7"}
```

---

## 4. skill: report — `POST /api/v1/tools/report`

The "Generate Report" button. Same as v1, moved under `/tools`. Generate, persist, and return a new
report version. Currently renders **Markdown** (FE preview); PPT/PDF/infographic export is future work
(triggered on a download button, not here).

**Query params:** `analysis_id` (required), `user_id` (required). No request body.

```
POST /api/v1/tools/report?analysis_id=an_42&user_id=u_1a2b3c
```

| status | meaning |
|---|---|
| 201 | new version generated → `AnalysisReport` body. |
| 409 | floor not met — no recorded analyses yet for this session, nothing to report. |
| 500 | generation or persistence failed. |

**201 response** (`AnalysisReport`):

```json
{
  "report_id": "8f3a2b1c9d4e4f6a8b0c1d2e3f4a5b6c",
  "analysis_id": "an_42",
  "user_id": "u_1a2b3c",
  "version": 2,
  "generated_at": "2026-06-30T09:14:33.512Z",
  "problem_statement": {
    "objective": "Understand which regions drive revenue and why Q1 dipped.",
    "business_questions": [
      "Which regions contribute most to total revenue?",
      "Did any region decline quarter-over-quarter?"
    ]
  },
  "record_ids": ["rec_a1", "rec_b2"],
  "executive_summary": "Revenue is concentrated in the Central region (38% of total). The West was the only region to contract, down 12% QoQ — the main driver of the Q1 dip.",
  "findings": [
    { "text": "Central region contributed 38% of total revenue, the largest share.",
      "record_ids": ["rec_a1"], "supporting_data": null },
    { "text": "West region revenue fell 12% quarter-over-quarter.",
      "record_ids": ["rec_b2"], "supporting_data": null }
  ],
  "caveats": [
    { "text": "March data for the East region was partially missing (~6% of rows).",
      "record_ids": ["rec_b2"] }
  ],
  "open_questions": [
    { "text": "What drove the West region's QoQ decline?", "record_ids": ["rec_b2"] }
  ],
  "data_sources": [
    { "source_id": "src_sales_db", "name": "orders", "source_type": "postgres",
      "detail": { "tables": ["orders"], "row_count": 48213,
                  "columns": ["region", "amount", "ordered_at"] } }
  ],
  "method_steps": [
    { "task_id": "t1", "stage": "data_understanding", "objective": "Inventory the sales source",
      "status": "success", "tools_used": ["check_data"] },
    { "task_id": "t2", "stage": "modeling", "objective": "Aggregate revenue by region",
      "status": "success", "tools_used": ["analyze_aggregate"] }
  ],
  "rendered_markdown": "# Analysis Report\n\n*Generated 2026-06-30 by u_1a2b3c · 2 analyses · 1 source(s)*\n\n## Objective\nUnderstand which regions drive revenue…\n\n## Key Findings\n1. Central region contributed 38%…"
}
```

**409 response** (floor not met — the demo's most common error):

```json
{ "detail": "Not ready to generate a report — still needs at least one completed analysis." }
```

⚠️ **Precondition:** `AnalysisRecord`s persist only on the slow path, so reports require
`ENABLE_SLOW_PATH=true` on the Python deployment and ≥1 prior `structured_flow` question in the
session. With slow path off, `POST` 409s by design.

---

## 5. report versions — `GET /api/v1/tools/report/{analysis_id}` and `/{analysis_id}/{version}`

List a session's report versions (oldest-first). Returns `[ReportVersionEntry]`; `[]` if none.

```json
[
  { "report_id": "1b2c3d4e…", "version": 1, "generated_at": "2026-06-24T15:02:11Z", "record_count": 1 },
  { "report_id": "8f3a2b1c…", "version": 2, "generated_at": "2026-06-25T09:14:33Z", "record_count": 2 }
]
```

`GET /api/v1/tools/report/{analysis_id}/{version}` → one `AnalysisReport` (same shape as the POST
201 body); 404 if that version doesn't exist:

```json
{ "detail": "No report v3 for analysis 'an_42'." }
```

---

## 6. Unwired in v2 (mounted in v1, OFF in v2)

Commented out of `main` + Swagger, **files kept**. Go owns these; Python is generative-only:
`POST /analysis/create` + analysis CRUD · `room` · `db_client` · `document` · `data_catalog` ·
`users`/login. Re-mounting is a one-line `include_router` if ever needed.

---

## 7. observability — `GET /api/v1/observability`  **(NEW · TENTATIVE)**

Responsible-AI provenance for **one chat answer**. Separate endpoint, called **in parallel with the
stream** — never embedded in it. The FE renders it as a collapsed dropdown the user can expand
(planning / tool calls / sources), Claude/Codex-style.

**Design (recommended):** one endpoint returns one merged object, backed by **one JSONB row per
message** written by an accumulating "scratchpad" decorator inside the chat agent and flushed on
`done`. The 3 facets (`planning` / `tool_calls` / `sources`) are **logical sections of the JSON**,
not separate tables — so the shape can evolve without a dedorch migration each time. (Storage is open
question #3.)

**Query params:** `analysis_id` (required), `message_id` (required).

```
GET /api/v1/observability?analysis_id=an_42&message_id=msg_88f1
```

**Timing:** the row is written when the turn finishes, so call this **after** the stream's `done`
event (or poll until 200). "Parallel" = a separate call the FE fires alongside the stream, not data
embedded in the stream.

**Field rules (by intent):**
- `planning` — present **only when the planner ran** (slow path); `null` otherwise.
- `tool_calls` — every tool invoked, with input + output. `[]` for pure chat / greeting / help.
- `sources` — **required for retrieve flows** (`structured_flow`, `unstructured_flow`). **Empty for
  greeting / `chat` / `help`** (they don't reference documents).
- `thinking` — optional reasoning text; `null` if none.

**200 response — full `structured_flow` turn** (planner ran → all sections present):

```json
{
  "analysis_id": "an_42",
  "message_id": "msg_88f1",
  "intent": "structured_flow",
  "generated_at": "2026-06-30T03:21:09.114Z",
  "planning": {
    "goal_restated": "Find which regions drive revenue and why Q1 dipped.",
    "assumptions": ["'last quarter' = Q1 2026"],
    "steps": [
      { "step": 1, "stage": "data_understanding", "objective": "Inventory the sales source" },
      { "step": 2, "stage": "modeling", "objective": "Aggregate revenue by region" }
    ]
  },
  "thinking": "The question needs a per-region breakdown plus a cause, so I inventory the source, aggregate revenue by region, then compare quarters.",
  "tool_calls": [
    {
      "order": 1,
      "name": "check_data",
      "input": { "source_hint": "structured" },
      "output": { "kind": "table", "summary": "1 source · 1 table (orders) · 48,213 rows" },
      "status": "success"
    },
    {
      "order": 2,
      "name": "retrieve_data",
      "input": { "source_id": "src_sales_db", "table_id": "orders",
                 "select": ["region", "amount"], "group_by": ["region"] },
      "output": { "kind": "table", "columns": ["region", "total"], "row_count": 4,
                  "preview": [["Central", 1210000], ["East", 740000]] },
      "status": "success"
    }
  ],
  "sources": [
    {
      "type": "database",
      "source_id": "src_sales_db",
      "name": "orders",
      "query": "SELECT region, SUM(amount) AS total FROM orders GROUP BY region",
      "detail": { "tables": ["orders"], "row_count": 48213 }
    }
  ]
}
```

**200 response — `unstructured_flow` turn** (no planner; source = document, with the retrieval query):

```json
{
  "analysis_id": "an_42",
  "message_id": "msg_55",
  "intent": "unstructured_flow",
  "generated_at": "2026-06-30T03:40:02.001Z",
  "planning": null,
  "thinking": null,
  "tool_calls": [
    { "order": 1, "name": "retrieve_knowledge",
      "input": { "query": "technology stack used in this project", "top_k": 4 },
      "output": { "kind": "documents", "row_count": 4 }, "status": "success" }
  ],
  "sources": [
    { "type": "document", "document_id": "doc_7", "filename": "tech_handbook.pdf",
      "page_label": "12", "query": "technology stack used in this project",
      "snippet": "The backend is built on FastAPI with async SQLAlchemy…", "score": 0.83 }
  ]
}
```

**200 response — simple `chat` / greeting turn** (nothing to trace):

```json
{
  "analysis_id": "an_42",
  "message_id": "msg_12",
  "intent": "chat",
  "generated_at": "2026-06-30T03:05:00.000Z",
  "planning": null,
  "thinking": null,
  "tool_calls": [],
  "sources": []
}
```

**404** — no provenance for that message yet (turn still running or unknown id):

```json
{ "detail": "No observability for message 'msg_88f1' yet." }
```

> ⚠️ **Richness is path-dependent.** Full `planning` + tool I/O exist only when
> `ENABLE_SLOW_PATH=true`. Fast chat / single-query / help still record `sources` + the single
> tool call but have `planning: null`. This matches the rule "planning only when the planner runs."