API_CONTRACT_BE_PYTHON.md

# Backend Agentic Service API Contract

This document describes the Python agentic backend used by the frontend for AI chat, help/report tools, and traceability data shown alongside chat answers.

Base path examples use relative URLs. Configure the frontend with the deployed Python service base URL.

## Overview

The Python backend owns the generative AI interaction surface:

1. Stream chat answers from the AI agent.
2. Execute tool-style actions for help and report generation.
3. Return report versions and report details.
4. Return traceability/provenance for a completed assistant answer.

The frontend uses this service during the analysis conversation flow:

1. User sends a chat message.
2. Frontend calls `POST /api/v2/chat/stream` and renders the streamed answer.
3. When the stream emits `done`, frontend stores or reads the returned `message_id`.
4. Frontend calls `GET /api/v1/traceability` for planning, tool calls, and source provenance.
5. Frontend calls `/api/v1/tools/help` for guided help and `/api/v1/tools/report` for report generation.

## Endpoint Summary

| Method | Path | Purpose |
| --- | --- | --- |
| `POST` | `/api/v2/chat/stream` | Stream an AI chat answer for one analysis conversation. |
| `GET` | `/api/v1/tools/list` | List available frontend tools. |
| `POST` | `/api/v1/tools/help` | Stream contextual help for the current analysis conversation. |
| `POST` | `/api/v1/tools/report` | Generate and persist a new report version. |
| `GET` | `/api/v1/tools/report/{analysis_id}` | List report versions for an analysis. |
| `GET` | `/api/v1/tools/report/{analysis_id}/{version}` | Retrieve one report version. |
| `GET` | `/api/v1/traceability` | Retrieve provenance for one assistant answer. |

## Common Concepts

### Identifiers

- `user_id`: user identifier passed by the frontend.
- `analysis_id`: analysis conversation identifier.
- `message_id`: assistant answer identifier used to correlate chat streaming with traceability.

### Server-Sent Events

Chat and help endpoints return `text/event-stream`.

Frontend should parse events by `event` name and `data` payload. Blank lines separate SSE events.

Common event types:

| Event | Data | Meaning |
| --- | --- | --- |
| `sources` | JSON array | Always `[]` — sources moved to `GET /api/v1/traceability` (KM-691). Event kept for backward-compat; read `sources[]` from the traceability call. |
| `status` | text | Optional progress update for slower paths. |
| `chunk` | text | Answer text fragment. Concatenate chunks in order. |
| `done` | JSON object | Terminal success event. Includes `message_id`. |
| `error` | text | Terminal error event. Stream stops after this. |

The stream carries answer text only. Planning, tool call details, and full provenance are fetched from `GET /api/v1/traceability` after the stream is done.

## Chat

### `POST /api/v2/chat/stream`

Streams an AI answer for one user message in an analysis conversation.

Request body:

```json
{
  "user_id": "u_1a2b3c",
  "analysis_id": "an_42",
  "message_id": "msg_88f1",
  "message": "What were total sales by region last quarter?"
}
```

Fields:

| Field | Required | Description |
| --- | --- | --- |
| `user_id` | Yes | User identifier. |
| `analysis_id` | Yes | Analysis conversation identifier. |
| `message_id` | No | Assistant answer id for traceability correlation. If omitted, Python returns one in `done`. |
| `message` | Yes | User message text. |

Response: `text/event-stream`.

Example structured answer:

```text
event: sources
data: []

event: status
data: Planning analysis...

event: status
data: Running 3 steps...

event: chunk
data: Total sales by region last quarter: 

event: chunk
data: Central led at $1.21M (38%), East $0.74M, West $0.55M (down 12% QoQ).

event: done
data: {"message_id":"msg_88f1"}
```

Example simple chat answer:

```text
event: sources
data: []

event: chunk
data: I'm your AI data analyst. Connect a source or ask a question to get started.

event: done
data: {"message_id":"msg_12"}
```

Behavior notes:

- Greeting and farewell messages may use a fast canned path.
- Stateless `chat` intent may use a 1-hour Redis response cache.
- The router may classify messages into intents such as `chat`, `help`, `check`, `unstructured_flow`, or `structured_flow`.
- `sources` in the stream is **always `[]`** (KM-691) — read the real `sources[]` from `GET /api/v1/traceability` after `done`.
- `status` events are optional and should be safe for the frontend to ignore.

## Tools

### `GET /api/v1/tools/list`

Returns the deterministic list of tools available to the frontend.

Request: none.

Response `200`:

```json
{
  "count": 2,
  "tools": [
    {
      "command": "/help",
      "name": "help",
      "type": "skill",
      "description": "Show what the assistant can do and guide your next step."
    },
    {
      "command": "/report",
      "name": "report",
      "type": "skill",
      "description": "Generate a versioned analysis report with background, EDA, key findings, and insights."
    }
  ]
}
```

Tool item shape:

```json
{
  "command": "/help",
  "name": "help",
  "type": "skill",
  "description": "Show what the assistant can do and guide your next step."
}
```

Frontend behavior:

- Surface `/help` in the slash menu.
- Surface report generation as a button or explicit UI action.

### `POST /api/v1/tools/help`

Streams contextual guidance for the current analysis conversation.

Request body:

```json
{
  "user_id": "u_1a2b3c",
  "analysis_id": "an_42"
}
```

Response: `text/event-stream` using the same event shape as chat.

Help responses usually emit `sources: []` and no `status` pings.

Example:

```text
event: sources
data: []

event: chunk
data: Your goal is set. You can start exploring now. Try a question like "average order value by month", then I can generate a report.

event: done
data: {"message_id":"msg_h7"}
```

## Reports

### `POST /api/v1/tools/report`

Generates, persists, and returns a new report version for an analysis.

Query params:

| Query | Required | Description |
| --- | --- | --- |
| `analysis_id` | Yes | Analysis identifier. |
| `user_id` | Yes | User identifier. |

Example:

```text
POST /api/v1/tools/report?analysis_id=an_42&user_id=u_1a2b3c
```

Status codes:

| Status | Meaning |
| --- | --- |
| `201` | New report version generated. |
| `409` | Report floor/precondition not met. |
| `500` | Generation or persistence failed. |

Response `201`:

```json
{
  "report_id": "8f3a2b1c9d4e4f6a8b0c1d2e3f4a5b6c",
  "analysis_id": "an_42",
  "user_id": "u_1a2b3c",
  "version": 2,
  "generated_at": "2026-06-30T09:14:33.512Z",
  "problem_statement": {
    "objective": "Understand which regions drive revenue and why Q1 dipped.",
    "business_questions": [
      "Which regions contribute most to total revenue?",
      "Did any region decline quarter-over-quarter?"
    ]
  },
  "record_ids": ["rec_a1", "rec_b2"],
  "executive_summary": "Revenue is concentrated in the Central region (38% of total). The West was the only region to contract, down 12% QoQ, the main driver of the Q1 dip.",
  "findings": [
    {
      "text": "Central region contributed 38% of total revenue, the largest share.",
      "record_ids": ["rec_a1"],
      "supporting_data": null
    },
    {
      "text": "West region revenue fell 12% quarter-over-quarter.",
      "record_ids": ["rec_b2"],
      "supporting_data": null
    }
  ],
  "caveats": [
    {
      "text": "March data for the East region was partially missing, around 6% of rows.",
      "record_ids": ["rec_b2"]
    }
  ],
  "open_questions": [
    {
      "text": "What drove the West region's QoQ decline?",
      "record_ids": ["rec_b2"]
    }
  ],
  "data_sources": [
    {
      "source_id": "src_sales_db",
      "name": "orders",
      "source_type": "postgres",
      "detail": {
        "tables": ["orders"],
        "row_count": 48213,
        "columns": ["region", "amount", "ordered_at"]
      }
    }
  ],
  "method_steps": [
    {
      "task_id": "t1",
      "stage": "data_understanding",
      "objective": "Inventory the sales source",
      "status": "success",
      "tools_used": ["check_data"]
    },
    {
      "task_id": "t2",
      "stage": "modeling",
      "objective": "Aggregate revenue by region",
      "status": "success",
      "tools_used": ["analyze_aggregate"]
    }
  ],
  "rendered_markdown": "# Analysis Report\n\n*Generated 2026-06-30 by u_1a2b3c*\n\n## Objective\nUnderstand which regions drive revenue..."
}
```

Response `409`:

```json
{
  "detail": "Not ready to generate a report - still needs at least one completed analysis."
}
```

Precondition:

- Reports require at least one completed analysis record for the session.
- If slow-path analysis recording is disabled, report generation can return `409` by design.

### `GET /api/v1/tools/report/{analysis_id}`

Lists report versions for one analysis, oldest first.

Response `200`:

```json
[
  {
    "report_id": "1b2c3d4e",
    "version": 1,
    "generated_at": "2026-06-24T15:02:11Z",
    "record_count": 1
  },
  {
    "report_id": "8f3a2b1c",
    "version": 2,
    "generated_at": "2026-06-25T09:14:33Z",
    "record_count": 2
  }
]
```

If no reports exist, returns `[]`.

### `GET /api/v1/tools/report/{analysis_id}/{version}`

Returns one report version. Shape is the same as the `201` response from `POST /api/v1/tools/report`.

Response `404`:

```json
{
  "detail": "No report v3 for analysis 'an_42'."
}
```

## Traceability

> Renamed from `observability` (team decision 2026-07-06) so it is never confused with the internal Langfuse *observability* stack (engineering telemetry, PII-masked). Traceability is **user-facing** provenance — real tool args, output previews, and the executed query — shown alongside the answer.

### `GET /api/v1/traceability`

Returns user-facing provenance for one assistant answer.

The frontend should call this after the chat/help stream emits `done`, using the `message_id` from the `done` event. The row is written **before** `done`, so an immediate GET returns `200` (no polling race). A `404` means the id is unknown or the turn errored before completing (error turns never produce a row).

Query params:

| Query | Required | Description |
| --- | --- | --- |
| `analysis_id` | Yes | Analysis identifier. |
| `message_id` | Yes | Assistant answer identifier returned by the stream. |

Example:

```text
GET /api/v1/traceability?analysis_id=an_42&message_id=msg_88f1
```

`intent` values the frontend may see: `chat` · `help` · `check` · `unstructured_flow` · `structured_flow` · `out_of_scope` · `blocked` (`blocked` = input-guard or Azure content-filter refusal; `chat` also covers the greeting fast-path and cache replays).

Field rules:

- `planning`: present only when the planner ran (`structured_flow`); otherwise `null`.
- `thinking`: **always `null` in v1** — our agents are plain chat completions with no native reasoning output, and synthesizing it post-hoc would be unfaithful. The field stays in the payload so it can be populated later without a contract change.
- `tool_calls`: every invoked tool with `input`, `output`, `status`, `task_id` (nullable), and `error` (nullable); empty for chat / help / greeting / refusal paths.
- `sources`: required for retrieval flows; empty for chat / help / refusal paths and for `check`.
- The payload also carries an internal `user_id` (ownership); the frontend may ignore it.
- Truncation: `preview` ≤ 5 rows; any string inside `input`/`output`/`preview`/`snippet` ≤ 300 chars; rows beyond the preview are dropped (`row_count` is preserved).

Response `200` for `structured_flow`:

```json
{
  "analysis_id": "an_42",
  "message_id": "msg_88f1",
  "user_id": "user_7",
  "intent": "structured_flow",
  "generated_at": "2026-07-06T03:21:09.114Z",
  "planning": {
    "goal_restated": "Find which regions drive revenue and why Q1 dipped.",
    "assumptions": [],
    "steps": [
      {
        "step": 1,
        "stage": "data_understanding",
        "objective": "Inventory the sales source",
        "status": "success",
        "tools_used": ["check_data"]
      },
      {
        "step": 2,
        "stage": "modeling",
        "objective": "Aggregate revenue by region",
        "status": "success",
        "tools_used": ["retrieve_data", "analyze_aggregate"]
      }
    ]
  },
  "thinking": null,
  "tool_calls": [
    {
      "order": 1,
      "task_id": null,
      "name": "check_data",
      "input": { "source_hint": "structured" },
      "output": {
        "kind": "table",
        "columns": ["source_id", "name", "source_type", "table_count"],
        "row_count": 1,
        "preview": [["src_sales_db", "orders", "schema", 1]]
      },
      "status": "success",
      "error": null
    },
    {
      "order": 2,
      "task_id": null,
      "name": "retrieve_data",
      "input": { "ir": { "source_id": "src_sales_db", "table_id": "orders", "select": ["region", "amount"], "group_by": ["region"] } },
      "output": {
        "kind": "table",
        "columns": ["region", "total"],
        "row_count": 4,
        "preview": [["Central", 1210000], ["East", 740000]]
      },
      "status": "success",
      "error": null
    }
  ],
  "sources": [
    {
      "type": "database",
      "source_id": "src_sales_db",
      "name": "orders",
      "query": "SELECT region, SUM(amount) AS total FROM orders GROUP BY region",
      "detail": {
        "table": "orders",
        "row_count": 4
      }
    }
  ]
}
```

> Note: `retrieve_data`'s real `input` is the compiled query IR under an `ir` key (the planner builds an IR, never raw SQL). The executed SQL/rendered query appears on the corresponding `sources[].query`.

Response `200` for `unstructured_flow`:

```json
{
  "analysis_id": "an_42",
  "message_id": "msg_55",
  "user_id": "user_7",
  "intent": "unstructured_flow",
  "generated_at": "2026-07-06T03:40:02.001Z",
  "planning": null,
  "thinking": null,
  "tool_calls": [
    {
      "order": 1,
      "task_id": null,
      "name": "retrieve_knowledge",
      "input": { "query": "technology stack used in this project" },
      "output": { "kind": "documents", "row_count": 4 },
      "status": "success",
      "error": null
    }
  ],
  "sources": [
    {
      "type": "document",
      "document_id": "doc_7",
      "filename": "tech_handbook.pdf",
      "page_label": "12",
      "query": "technology stack used in this project",
      "snippet": "The backend is built on FastAPI with async SQLAlchemy...",
      "score": 0.83
    }
  ]
}
```

Response `200` for chat / greeting / help / refusals (`out_of_scope`, `blocked`):

```json
{
  "analysis_id": "an_42",
  "message_id": "msg_12",
  "user_id": "user_7",
  "intent": "chat",
  "generated_at": "2026-07-06T03:05:00.000Z",
  "planning": null,
  "thinking": null,
  "tool_calls": [],
  "sources": []
}
```

Response `404`:

```json
{
  "detail": "No traceability for message 'msg_88f1' yet."
}
```

Frontend rendering guidance:

- Render traceability separately from the streamed answer.
- Default state can be collapsed.
- Show planning, tool calls, and sources as separate sections.
- Treat `planning: null`, `tool_calls: []`, and `sources: []` as valid states.