API_ENDPOINTS.md

# Data Eyond — Python Agentic Service: FE-Callable API (for Go integration)

**Audience:** Harry (Go gateway) wiring the FE → Go → Python surface.
**Scope:** the **4 FE-callable surfaces** the Python service exposes after the 2026-06-24 pivot
(DEV_PLAN decision #6). Everything else under `/api/v1` is internal / Phase-1 legacy / Go-owned —
see [§7](#7-not-fe-facing) and the full inventory in [§9](#9-appendix--complete-endpoint-inventory-all-registered-routes).
**Branch:** `pr/4` · **Snapshot:** 2026-06-25 · **Companion:** [REPO_STATUS.md](REPO_STATUS.md).

> Request flow is **FE → Go → Python**. The FE never calls Python directly except for chat
> streaming. Auth/JWT is terminated at the Go gateway; Python receives `user_id` / `room_id` as
> **trusted inputs** and does no auth of its own.

---

## 1. The 4 FE-callable surfaces

| # | Logical name | HTTP | How it's invoked |
|---|---|---|---|
| 1 | **`call_agent`** | `POST /api/v1/chat/stream` | The one streaming chat call. Router classifies + dispatches. |
| 2 | **`list_skills`** | `GET /api/v1/tools` | Static slash-command catalog for the FE "/" menu. Cacheable. |
| 3 | **skill: `help`** | *(via `call_agent`)* | **No dedicated endpoint** — the router resolves it to the `help` intent inside `/chat/stream`. |
| 4 | **skill: `report`** | `POST /api/v1/report` (+ 2 `GET`s) | Dedicated REST API. **Not** through `/chat/stream`. |

**Key consequence for Go:** the two catalog skills are invoked **differently**. `/help` goes through
`/chat/stream`; `/report` is a direct REST call to the Report API. The catalog's `name` field is the
internal route key (`help` = router intent; `report` = the Report API), not a uniform dispatch key.

**Conventions:**
- Base path: `/api/v1`.
- **`room_id == analysis_id`** — one chat room == one analysis session (#9). Callers pass `room_id`
  to chat; it *is* the `analysis_id` used by the report API.
- Streaming uses **SSE** (`text/event-stream`, `sse-starlette`).

---

## 2. `call_agent` — `POST /api/v1/chat/stream`

The only FE→Python call in normal operation. Source: [chat.py:169](src/api/v1/chat.py:169).

**Request body** (`application/json`) — `ChatRequest`:

```json
{
  "user_id": "u_1a2b3c",
  "room_id": "room_42",
  "message": "What were total sales by region last quarter?"
}
```

`room_id` is the analysis session id. No auth header (handled by Go).

**Response:** `text/event-stream`. Events arrive in this order:

| `event:` | `data:` payload | Notes |
|---|---|---|
| `sources` | JSON array of source refs | `{document_id, filename, page_label}`. Structured: one per executed table (`document_id = "{user_id}_{table}"`, `page_label = null`). Unstructured: deduped doc/page. `chat`/`help`/`error`: `[]`. |
| `status` | text | **Slow-path only** — progress pings ("Planning…", "Running N steps…"). Keeps the SSE alive; safe to surface or ignore. |
| `chunk` | text fragment | Concatenate in order to form the answer. |
| `done` | *(empty)* | End of stream. |
| `error` | text | Terminal error; stream stops after this. |

> The handler also emits an internal `intent` event — it is **consumed inside Python** (gates
> caching) and **not forwarded** to the client. Go/FE will never see it.

**Example — `structured_flow` answer** (raw SSE wire; blank line separates events). Source shape:
[chat_handler.py:607](src/agents/chat_handler.py:607).

```
event: sources
data: [{"document_id":"u_1a2b3c_orders","filename":"orders","page_label":null}]

event: status
data: Planning analysis…

event: status
data: Running 3 steps…

event: chunk
data: Total sales by region last quarter: 

event: chunk
data: Central led at $1.21M (38%), East $0.74M, West $0.55M (down 12% QoQ).

event: done
data: 
```

**Example — simple `chat` reply** (no status pings, empty sources):

```
event: sources
data: []

event: chunk
data: I'm your AI data analyst — connect a source or ask a question to get started.

event: done
data: 
```

**Behavior worth knowing for integration:**
- **Redis response cache** (1h TTL) is applied to the stateless `chat` intent only; cached replies
  replay as `sources`/`chunk`/`done`.
- **Greeting/farewell fast-path** returns a canned reply with no LLM call.
- The LLM **router** classifies every message into one of **5 intents** —
  `chat` · `help` · `check` · `unstructured_flow` · `structured_flow` — and dispatches. Messages
  persist (user + assistant) on `done`.

---

## 3. `list_skills` — `GET /api/v1/tools`

Static, deterministic, **safe for Go to cache**. Source: [tools.py:133](src/api/v1/tools.py:133).

**Request:** none (no params, no body).

**Response** `200` (`ListToolsResponse`):

```json
{
  "count": 2,
  "tools": [
    { "command": "/help",   "name": "help",   "type": "skill",
      "description": "Show what the assistant can do and guide your next step." },
    { "command": "/report", "name": "report", "type": "skill",
      "description": "Generate a versioned analysis report (background, EDA, key findings, insights)." }
  ]
}
```

`CommandResponse` = `{ command, name, type, description }`, `type ∈ {skill, analytics, data_access}`.
Post-KM-678 the catalog is **`/help` + `/report` only**; the `analyze_*`, `check_*`, `retrieve_*`
and retired `/problem-statement` entries are commented out (kept for restorability), not deleted.

---

## 4. skill: `help` — via `call_agent`

**There is no `/help` endpoint.** The FE "/" menu surfaces `/help`; to invoke it, call
`POST /api/v1/chat/stream` and let the router classify the message as the `help` intent
([chat_handler.py:363](src/agents/chat_handler.py:363)). Help streams `chunk` events (same SSE
shape as §2, with `sources: []` and no `status` pings) — a state-aware, next-step guidance reply.

```
event: sources
data: []

event: chunk
data: Your goal is set — you can start exploring now. Try a question like "average order value by month", then I can generate a report.

event: done
data: 
```

> **Open integration question (for Harry):** the Python `/chat/stream` contract has **no
> forced-intent / slash-bypass param** — `handle()` always routes via the LLM classifier. So
> deterministic `/help` dispatch depends on either (a) Go forwarding the literal slash text and
> trusting the router to classify it as `help`, or (b) adding a forced-intent input to the chat
> contract. The `tools.py` docstring's "slash invocation bypasses the router to the tool directly"
> is **not yet true on the Python side.** Needs a decision. (DEV_PLAN #8/#18.)

---

## 5. skill: `report` — Report API

Dedicated REST surface (the "Generate Report" button), **not** a chat route.
Source: [report.py](src/api/v1/report.py).

### `POST /api/v1/report`
Generate, persist, and return a new report **version**.

**Query params:** `analysis_id` (required), `user_id` (required). No request body.

```
POST /api/v1/report?analysis_id=room_42&user_id=u_1a2b3c
```

| Status | Meaning |
|---|---|
| `201` | New version generated → `AnalysisReport` body. |
| `409` | Floor not met — **no recorded analyses yet** for this session, nothing to report. |
| `500` | Generation or persistence failed. |

**`201` response** (`AnalysisReport`):

```json
{
  "report_id": "8f3a2b1c9d4e4f6a8b0c1d2e3f4a5b6c",
  "analysis_id": "room_42",
  "user_id": "u_1a2b3c",
  "version": 2,
  "generated_at": "2026-06-25T09:14:33.512Z",
  "problem_statement": {
    "objective": "Understand which regions drive revenue and why Q1 dipped.",
    "business_questions": [
      "Which regions contribute most to total revenue?",
      "Did any region decline quarter-over-quarter?"
    ]
  },
  "record_ids": ["rec_a1", "rec_b2"],
  "executive_summary": "Revenue is concentrated in the Central region (38% of total). The West was the only region to contract, down 12% QoQ — the main driver of the Q1 dip.",
  "findings": [
    { "text": "Central region contributed 38% of total revenue, the largest share.",
      "record_ids": ["rec_a1"], "supporting_data": null },
    { "text": "West region revenue fell 12% quarter-over-quarter.",
      "record_ids": ["rec_b2"], "supporting_data": null }
  ],
  "caveats": [
    { "text": "March data for the East region was partially missing (~6% of rows).",
      "record_ids": ["rec_b2"] }
  ],
  "open_questions": [
    { "text": "What drove the West region's QoQ decline?", "record_ids": ["rec_b2"] }
  ],
  "data_sources": [
    { "source_id": "src_sales_db", "name": "orders", "source_type": "postgres",
      "detail": { "tables": ["orders"], "row_count": 48213,
                  "columns": ["region", "amount", "ordered_at"] } }
  ],
  "method_steps": [
    { "task_id": "t1", "stage": "data_understanding", "objective": "Inventory the sales source",
      "status": "success", "tools_used": ["check_data"] },
    { "task_id": "t2", "stage": "modeling", "objective": "Aggregate revenue by region",
      "status": "success", "tools_used": ["analyze_aggregate"] }
  ],
  "rendered_markdown": "# Analysis Report\n\n*Generated 2026-06-25 by u_1a2b3c · 2 analyses · 1 source(s)*\n\n## Objective\nUnderstand which regions drive revenue…\n\n## Key Findings\n1. Central region contributed 38%…"
}
```

**`409` response** (floor not met — the demo's most common error):

```json
{ "detail": "Not ready to generate a report — still needs at least one completed analysis." }
```

> ⚠️ **Demo/integration precondition:** `AnalysisRecord`s persist **only on the slow path**, so
> reports require **`enable_slow_path=true`** on the Python deployment *and* ≥1 prior
> `structured_flow` question in the session. With slow path off, `POST /report` **409s by design**,
> not a bug. (DEV_PLAN #15/#16.)

### `GET /api/v1/report/{analysis_id}`
List a session's report versions (oldest-first). Returns `[ReportVersionEntry]`; `[]` if none.

```json
[
  { "report_id": "1b2c3d4e…", "version": 1, "generated_at": "2026-06-24T15:02:11Z", "record_count": 1 },
  { "report_id": "8f3a2b1c…", "version": 2, "generated_at": "2026-06-25T09:14:33Z", "record_count": 2 }
]
```

### `GET /api/v1/report/{analysis_id}/{version}`
Fetch one version → `AnalysisReport` (same shape as the `POST` 201 body above); `404` if that
version doesn't exist.

```json
{ "detail": "No report v3 for analysis 'room_42'." }
```

---

## 6. Schemas

**`AnalysisReport`** (POST + GET-version body):

| Field | Type | Notes |
|---|---|---|
| `report_id` | str | |
| `analysis_id` | str | == `room_id` |
| `user_id` | str \| null | |
| `version` | int | monotonic V1, V2, … |
| `generated_at` | datetime | ISO 8601, UTC |
| `problem_statement` | `{ objective: str, business_questions: string[] }` | the frozen goal snapshot (new pivot shape) |
| `record_ids` | string[] | records the version was built from |
| `executive_summary` | str | the **only** LLM-authored field |
| `findings` | `ReportFinding[]` | `{ text, record_ids[], supporting_data? }` |
| `caveats` | `AttributedNote[]` | `{ text, record_ids[] }` |
| `open_questions` | `AttributedNote[]` | `{ text, record_ids[] }` |
| `data_sources` | `DataSourceRef[]` | `{ source_id, name, source_type, detail }` |
| `method_steps` | `TaskSummary[]` | `{ task_id, stage, objective, status, tools_used[] }`; `stage` ∈ CRISP-DM phases |
| `rendered_markdown` | str | the full rendered report |

> **Persistence caveat:** dedorch `reports` stores **markdown only**. On read-back via the `GET`
> endpoints, the structured fields above come back **empty** and `rendered_markdown` is the source of
> truth. (REPO_STATUS §5.)

**`ReportVersionEntry`** (GET-list rows): `{ report_id, version, generated_at, record_count }`.

---

## 7. Not FE-facing

Registered under `/api/v1` but **not** part of the FE→Python surface — do not wire these from the FE:

- **Analysis CRUD** — `POST /analysis/create`, `GET /analysis`, `GET /analysis/{id}`. Intended to
  move behind Go (state writes via Go, per decision #5/#18). Router still **mounted** (Go may use it);
  the FE should not call it.
- **`check_data` / `check_knowledge`** — served by **Go**, not surfaced as Python FE endpoints.
- **Chat cache management** — `DELETE /chat/cache`, `/chat/cache/room/{id}`, `/retrieval/cache/{user_id}`
  (ops/internal).
- **Phase-1 legacy routers** — `users`, `room`, `document`, `db_client`, `data_catalog`
  (functionally migrated to Go; mostly dormant).
- **Health/root** — `GET /`, `GET /health` (liveness only).

---

## 8. Open items affecting this contract

1. **`/help` dispatch mechanism** — router-classify vs. forced-intent param (§4). *(DEV_PLAN #8/#18)*
2. **`/report` needs `enable_slow_path=true`** + a prior `structured_flow` question, else 409.
   *(DEV_PLAN #15)*
3. **`analysis_records` home** post-`SKIP_INIT_DB` cutover — the report API depends on this table
   existing. *(DEV_PLAN #14/#16)*
4. **Analysis-state writes** — once Go owns creation + state writes, Python's per-turn state
   `ensure` becomes a read-only get (Go must guarantee the row exists before any chat turn).
   *(DEV_PLAN #18)*

---

## 9. Appendix — complete endpoint inventory (all registered routes)

Every route mounted in [main.py](main.py), so task #8 can be decided against the full picture.
**32 routes** across 9 routers + 2 app-level. Status legend:
**✅ FE-callable** (one of the 4 surfaces — keep) · **✂️ comment out** (task #8 target) ·
**🟦 legacy → Go** (Phase-1, functionally migrated; not FE→Python; mostly dormant) ·
**⚙️ internal/ops**.

| Method | Path | Purpose | Router | Status |
|---|---|---|---|---|
| POST | `/api/v1/chat/stream` | Main chat SSE — **`call_agent`**; carries chat/help/check/structured/unstructured intents | Chat | ✅ FE-callable (#1, +help #3) |
| GET | `/api/v1/tools` | Slash-command catalog — **`list_skills`** (Go caches) | Tools | ✅ FE-callable (#2) |
| POST | `/api/v1/report` | Generate a report version | Report | ✅ FE-callable (#4) |
| GET | `/api/v1/report/{analysis_id}` | List report versions | Report | ✅ FE-callable (#4) |
| GET | `/api/v1/report/{analysis_id}/{version}` | Fetch one report version | Report | ✅ FE-callable (#4) |
| POST | `/api/v1/analysis/create` | Create session (state + room + bindings) | Analysis | ✂️ comment (#8 → Go) |
| GET | `/api/v1/analysis` | List a user's analyses | Analysis | ✂️ comment (#8) |
| GET | `/api/v1/analysis/{analysis_id}` | Get one session's state + sources | Analysis | ✂️ comment (#8) |
| DELETE | `/api/v1/chat/cache` | Clear one cached reply | Chat | ⚙️ internal/ops |
| DELETE | `/api/v1/chat/cache/room/{room_id}` | Clear a room's cache | Chat | ⚙️ internal/ops |
| DELETE | `/api/v1/retrieval/cache/{user_id}` | Clear a user's retrieval cache | Chat | ⚙️ internal/ops |
| GET | `/` | Service status | (app) | ⚙️ internal/ops |
| GET | `/health` | Liveness probe | (app) | ⚙️ internal/ops |
| POST | `/api/login` | Login by email + password ⚠️ mounted at `/api`, **not** `/api/v1` | Users | 🟦 legacy → Go |
| GET | `/api/v1/documents/doctypes` | Supported document types | Documents | 🟦 legacy → Go |
| GET | `/api/v1/documents/{user_id}` | List a user's documents | Documents | 🟦 legacy → Go |
| POST | `/api/v1/document/upload` | Upload a document (10/min) | Documents | 🟦 legacy → Go |
| DELETE | `/api/v1/document/delete` | Delete a document | Documents | 🟦 legacy → Go |
| POST | `/api/v1/document/process` | Process / ingest a document | Documents | 🟦 legacy → Go |
| GET | `/api/v1/rooms/{user_id}` | List a user's rooms | Rooms | 🟦 legacy → Go |
| GET | `/api/v1/room/{room_id}` | Get one room | Rooms | 🟦 legacy → Go |
| DELETE | `/api/v1/room/{room_id}` | Delete a room | Rooms | 🟦 legacy → Go |
| POST | `/api/v1/room/create` | Create a room | Rooms | 🟦 legacy → Go |
| GET | `/api/v1/data-catalog/{user_id}` | List catalog index | Data Catalog | 🟦 legacy → Go |
| POST | `/api/v1/data-catalog/rebuild` | Rebuild a user's catalog | Data Catalog | 🟦 legacy → Go |
| GET | `/api/v1/database-clients/dbtypes` | Supported DB types | Database Clients | 🟦 legacy → Go |
| POST | `/api/v1/database-clients` | Create a DB connection | Database Clients | 🟦 legacy → Go |
| GET | `/api/v1/database-clients/{user_id}` | List a user's DB connections | Database Clients | 🟦 legacy → Go |
| GET | `/api/v1/database-clients/{user_id}/{client_id}` | Get one DB connection | Database Clients | 🟦 legacy → Go |
| PUT | `/api/v1/database-clients/{client_id}` | Update a DB connection | Database Clients | 🟦 legacy → Go |
| DELETE | `/api/v1/database-clients/{client_id}` | Delete a DB connection | Database Clients | 🟦 legacy → Go |
| POST | `/api/v1/database-clients/{client_id}/ingest` | Build the catalog for a DB connection | Database Clients | 🟦 legacy → Go |

**Tally:** 5 ✅ FE-callable · 3 ✂️ to comment (#8) · 19 🟦 legacy→Go · 5 ⚙️ internal/ops.

**Task #8 reading:**
- **Keep exposed:** the 5 ✅ rows (`chat/stream`, `/tools`, the 3 `report` routes). `help` rides on
  `chat/stream` — no route of its own.
- **Comment out (the #8 to-do):** the 3 `analysis` routes — analysis CRUD moves behind Go (#5/#18).
- **`check_data` is not an HTTP endpoint** — it's the `check` router intent (runs inside
  `chat/stream`) plus its now-commented slash-catalog entry (KM-678); Go serves it to the FE. So
  "comment check_data" = the catalog line (done) + don't expose a Python route (there isn't one).
- The 19 🟦 routers (`users`, `document`, `room`, `data_catalog`, `db_client`) are Phase-1 legacy,
  already functionally in Go (REPO_STATUS §7). They're out of the FE→Python path but **still
  mounted** — a separate cleanup from #8's analysis-CRUD scope.