API_CONTRACT_BE_PYTHON.md

# Backend Agentic Service API Contract

This document describes the Python agentic backend used by the frontend for AI chat, help/report tools, and traceability data shown alongside chat answers.

Base path examples use relative URLs. Configure the frontend with the deployed Python service base URL.

Overview

The Python backend owns the generative AI interaction surface:

1. Stream chat answers from the AI agent.
2. Execute tool-style actions for help and report generation.
3. Return report versions and report details.
4. Return traceability/provenance for a completed assistant answer.

The frontend uses this service during the analysis conversation flow:

1. User sends a chat message.
2. Frontend calls POST /api/v2/chat/stream and renders the streamed answer.
3. When the stream emits done, frontend stores or reads the returned message_id.
4. Frontend calls GET /api/v1/traceability for planning, tool calls, and source provenance.
5. Frontend calls /api/v1/tools/help for guided help and /api/v1/tools/report for report generation.

Endpoint Summary

Method	Path	Purpose
POST	/api/v2/chat/stream	Stream an AI chat answer for one analysis conversation.
GET	/api/v1/tools/list	List available frontend tools.
POST	/api/v1/tools/help	Stream contextual help for the current analysis conversation.
POST	/api/v1/tools/report	Generate and persist a new report version.
GET	/api/v1/tools/report/{analysis_id}	List report versions for an analysis.
GET	/api/v1/tools/report/{analysis_id}/records	List analysis records for report curation (added 2026-07-09).
GET	/api/v1/tools/report/{analysis_id}/readiness	Report-readiness signal for the Generate-Report button (added 2026-07-09).
GET	/api/v1/tools/report/{analysis_id}/{version}	Retrieve one report version.
GET	/api/v1/traceability	Retrieve provenance for one assistant answer.

Common Concepts

Identifiers

• user_id: user identifier passed by the frontend.
• analysis_id: analysis conversation identifier.
• message_id: assistant answer identifier used to correlate chat streaming with traceability. Server-minted, never accepted from the client. Updated 2026-07-09: it is a UUID string (e.g. 77f06761-0fdf-4cc5-84f8-5f81bcbb6f84), matching the shape of Go's analyses_messages.id. The msg_… values in the examples below are illustrative placeholders only.

Server-Sent Events

Chat and help endpoints return text/event-stream.

Frontend should parse events by event name and data payload. Blank lines separate SSE events.

Common event types:

Event	Data	Meaning
sources	JSON array	Always [] — sources moved to GET /api/v1/traceability (KM-691). Event kept for backward-compat; read sources[] from the traceability call.
status	text	Optional progress update for slower paths.
chunk	text	Answer text fragment. Concatenate chunks in order.
done	JSON object	Terminal success event. Includes message_id.
error	text	Terminal error event. Stream stops after this.

The stream carries answer text only. Planning, tool call details, and full provenance are fetched from GET /api/v1/traceability after the stream is done.

Chat

POST /api/v2/chat/stream

Streams an AI answer for one user message in an analysis conversation.

Request body:

{
  "user_id": "u_1a2b3c",
  "analysis_id": "an_42",
  "message": "What were total sales by region last quarter?"
}

Fields:

Field	Required	Description
user_id	Yes	User identifier.
analysis_id	Yes	Analysis conversation identifier.
message_id	—	Updated 2026-07-09: not a request field. Python always mints the id server-side and returns it on done; any caller-sent value is ignored (server-authoritative — open-Q #1).
message	Yes	User message text.

Response: text/event-stream.

Example structured answer:

event: sources
data: []

event: status
data: Planning analysis...

event: status
data: Running 3 steps...

event: chunk
data: Total sales by region last quarter: 

event: chunk
data: Central led at $1.21M (38%), East $0.74M, West $0.55M (down 12% QoQ).

event: done
data: {"message_id":"msg_88f1"}

Example simple chat answer:

event: sources
data: []

event: chunk
data: I'm your AI data analyst. Connect a source or ask a question to get started.

event: done
data: {"message_id":"msg_12"}

Behavior notes:

• Greeting and farewell messages may use a fast canned path.
• Stateless chat intent may use a 1-hour Redis response cache.
• The router may classify messages into intents such as chat, help, check, unstructured_flow, or structured_flow.
• sources in the stream is always [] (KM-691) — read the real sources[] from GET /api/v1/traceability after done.
• status events are optional and should be safe for the frontend to ignore.

Tools

GET /api/v1/tools/list

Returns the deterministic list of tools available to the frontend.

Request: none.

Response 200:

{
  "count": 1,
  "tools": [
    {
      "command": "/help",
      "name": "help",
      "type": "skill",
      "description": "Show what the assistant can do and guide your next step."
    }
  ]
}

The catalog is /help only (KM-711). /report was removed as a slash command — report generation is a right-side Generate button, not a / command. The report HTTP endpoint (POST /api/v1/tools/report) still exists; the button calls it.

Tool item shape:

{
  "command": "/help",
  "name": "help",
  "type": "skill",
  "description": "Show what the assistant can do and guide your next step."
}

Frontend behavior:

• Surface /help in the slash menu.
• Surface report generation as a button or explicit UI action.

POST /api/v1/tools/help

Streams contextual guidance for the current analysis conversation.

Request body:

{
  "user_id": "u_1a2b3c",
  "analysis_id": "an_42"
}

Response: text/event-stream using the same event shape as chat.

Help responses usually emit sources: [] and no status pings.

Example:

event: sources
data: []

event: chunk
data: Your goal is set. You can start exploring now. Try a question like "average order value by month", then I can generate a report.

event: done
data: {"message_id":"msg_h7"}

Reports

POST /api/v1/tools/report

Generates, persists, and returns a new report version for an analysis.

Query params:

Query	Required	Description
analysis_id	Yes	Analysis identifier.
user_id	Yes	User identifier.
exclude_record_ids	No	Record ids to leave out of this version (repeat the param per id). Added 2026-07-09; get ids from GET /tools/report/{analysis_id}/records. Excluded runs are listed in the report's "Excluded Analyses" section. Excluding every substantive record returns 409.

Example:

POST /api/v1/tools/report?analysis_id=an_42&user_id=u_1a2b3c
POST /api/v1/tools/report?analysis_id=an_42&user_id=u_1a2b3c&exclude_record_ids=rec_a1&exclude_record_ids=rec_c3

Status codes:

Status	Meaning
201	New report version generated.
409	Report floor/precondition not met.
500	Generation or persistence failed.

Response 201:

{
  "report_id": "8f3a2b1c9d4e4f6a8b0c1d2e3f4a5b6c",
  "analysis_id": "an_42",
  "user_id": "u_1a2b3c",
  "version": 2,
  "generated_at": "2026-06-30T09:14:33.512Z",
  "problem_statement": {
    "objective": "Understand which regions drive revenue and why Q1 dipped.",
    "business_questions": [
      "Which regions contribute most to total revenue?",
      "Did any region decline quarter-over-quarter?"
    ]
  },
  "record_ids": ["rec_a1", "rec_b2"],
  "executive_summary": "Revenue is concentrated in the Central region (38% of total). The West was the only region to contract, down 12% QoQ, the main driver of the Q1 dip.",
  "bq_answers": [
    {
      "question": "Which regions contribute most to total revenue?",
      "answer": "The Central region leads with 38% of total revenue.",
      "status": "answered",
      "record_ids": ["rec_a1"]
    },
    {
      "question": "Did any region decline quarter-over-quarter?",
      "answer": "Yes — the West region fell 12% QoQ.",
      "status": "answered",
      "record_ids": ["rec_b2"]
    }
  ],
  "findings": [
    {
      "text": "Central region contributed 38% of total revenue, the largest share.",
      "record_ids": ["rec_a1"],
      "supporting_data": null
    },
    {
      "text": "West region revenue fell 12% quarter-over-quarter.",
      "record_ids": ["rec_b2"],
      "supporting_data": null
    }
  ],
  "caveats": [
    {
      "text": "March data for the East region was partially missing, around 6% of rows.",
      "record_ids": ["rec_b2"]
    }
  ],
  "open_questions": [
    {
      "text": "What drove the West region's QoQ decline?",
      "record_ids": ["rec_b2"]
    }
  ],
  "unresolved": [
    {
      "text": "Correlate churn with tenure — churn column not found in the source.",
      "record_ids": ["rec_d4"]
    }
  ],
  "excluded": [],
  "evidence_tables": {
    "rec_a1": [
      {
        "title": "Aggregate revenue by region",
        "columns": ["region", "total_revenue"],
        "rows": [["Central", "18321"], ["West", "9954"]],
        "truncated": false
      }
    ]
  },
  "data_sources": [
    {
      "source_id": "src_sales_db",
      "name": "orders",
      "source_type": "postgres",
      "detail": {
        "tables": ["orders"],
        "row_count": 48213,
        "columns": ["region", "amount", "ordered_at"]
      }
    }
  ],
  "method_steps": [
    {
      "task_id": "t1",
      "stage": "data_understanding",
      "objective": "Inventory the sales source",
      "status": "success",
      "tools_used": ["check_data"]
    },
    {
      "task_id": "t2",
      "stage": "modeling",
      "objective": "Aggregate revenue by region",
      "status": "success",
      "tools_used": ["analyze_aggregate"]
    }
  ],
  "rendered_markdown": "# Analysis Report\n\n*Generated 2026-06-30 by u_1a2b3c*\n\n## Objective\nUnderstand which regions drive revenue..."
}

Response 409:

{
  "detail": "Not ready to generate a report - still needs at least one completed analysis."
}

Report v2 fields (added 2026-07-09; all default-empty, so older stored reports read back unchanged):

• bq_answers — one entry per business question. status is answered | partial | unanswered; record_ids cite the backing analyses. Written in the analysis's language (Indonesian objective → Indonesian answers).
• unresolved — runs that were attempted but produced no usable evidence (every analyze_* step failed). Not part of the findings body.
• excluded — runs the caller excluded via exclude_record_ids.
• evidence_tables — record_id → small result tables copied from the run's stored outputs (max 3 tables per record, max 10 rows each; truncated: true when rows were capped). Rendered as markdown tables under the matching Key Findings group in rendered_markdown.

Precondition:

• Reports require at least one completed analysis record for the session.
• If slow-path analysis recording is disabled, report generation can return 409 by design.

GET /api/v1/tools/report/{analysis_id}

Lists report versions for one analysis, oldest first.

Response 200:

[
  {
    "report_id": "1b2c3d4e",
    "version": 1,
    "generated_at": "2026-06-24T15:02:11Z",
    "record_count": 1
  },
  {
    "report_id": "8f3a2b1c",
    "version": 2,
    "generated_at": "2026-06-25T09:14:33Z",
    "record_count": 2
  }
]

If no reports exist, returns [].

GET /api/v1/tools/report/{analysis_id}/records (added 2026-07-09)

Lists the persisted analysis runs a report would be built from, oldest first. The frontend shows this before generating so the user can deselect runs; the chosen ids go to POST /tools/report as exclude_record_ids.

Response 200:

[
  {
    "record_id": "rec_a1",
    "goal_restated": "Rank regions by total revenue",
    "created_at": "2026-06-30T08:55:02Z",
    "substantive": true,
    "findings_count": 2
  },
  {
    "record_id": "rec_d4",
    "goal_restated": "Correlate churn with tenure",
    "created_at": "2026-06-30T09:01:47Z",
    "substantive": false,
    "findings_count": 1
  }
]

substantive: false means no analyze_* step succeeded — that run is listed in the report's unresolved JSON field rather than the findings body. (Since 2026-07-09 the rendered markdown is compact and no longer includes "Attempted, Unresolved" / "Notes & Limitations" / "How This Was Analyzed" sections; the JSON fields unresolved / caveats / open_questions / method_steps are unchanged.) If no runs exist, returns [].

GET /api/v1/tools/report/{analysis_id}/readiness (added 2026-07-09)

Deterministic report-readiness signal for the Generate-Report button — the same producer as Help's readiness signal, including the advisory delta-since-report check, so the button, Help, and this endpoint never disagree.

Response 200:

{
  "ready": false,
  "missing": ["a new analysis since the last report"]
}

Note: POST /tools/report itself only enforces the floor (at least one completed analysis) — a new version is always allowed. The delta gap in missing is a soft warning the frontend can surface ("nothing new since the last report") without blocking the button.

GET /api/v1/tools/report/{analysis_id}/{version}

Returns one report version. Shape is the same as the 201 response from POST /api/v1/tools/report.

Response 404:

{
  "detail": "No report v3 for analysis 'an_42'."
}

Traceability

Renamed from observability (team decision 2026-07-06) so it is never confused with the internal Langfuse observability stack (engineering telemetry, PII-masked). Traceability is user-facing provenance — real tool args, output previews, and the executed query — shown alongside the answer.

GET /api/v1/traceability

Returns user-facing provenance for one assistant answer.

The frontend should call this after the chat/help stream emits done, using the message_id from the done event. The row is written before done, so an immediate GET returns 200 (no polling race). A 404 means the id is unknown or the turn errored before completing (error turns never produce a row).

Query params:

Query	Required	Description
analysis_id	Yes	Analysis identifier.
message_id	Yes	Assistant answer identifier returned by the stream.

Example:

GET /api/v1/traceability?analysis_id=an_42&message_id=msg_88f1

intent values the frontend may see: chat · help · check · unstructured_flow · structured_flow · out_of_scope · blocked (blocked = input-guard or Azure content-filter refusal; chat also covers the greeting fast-path and cache replays).

Field rules:

• planning: present only when the planner ran (structured_flow); otherwise null.
• thinking: always null in v1 — our agents are plain chat completions with no native reasoning output, and synthesizing it post-hoc would be unfaithful. The field stays in the payload so it can be populated later without a contract change.
• tool_calls: every invoked tool with input, output, status, task_id (nullable), and error (nullable); empty for chat / help / greeting / refusal paths.
• sources: required for retrieval flows; empty for chat / help / refusal paths and for check.
• The payload also carries an internal user_id (ownership); the frontend may ignore it.
• Truncation: preview ≤ 5 rows; any string inside input/output/preview/snippet ≤ 300 chars; rows beyond the preview are dropped (row_count is preserved).

Response 200 for structured_flow:

{
  "analysis_id": "an_42",
  "message_id": "msg_88f1",
  "user_id": "user_7",
  "intent": "structured_flow",
  "generated_at": "2026-07-06T03:21:09.114Z",
  "planning": {
    "goal_restated": "Find which regions drive revenue and why Q1 dipped.",
    "assumptions": [],
    "steps": [
      {
        "step": 1,
        "stage": "data_understanding",
        "objective": "Inventory the sales source",
        "status": "success",
        "tools_used": ["check_data"]
      },
      {
        "step": 2,
        "stage": "modeling",
        "objective": "Aggregate revenue by region",
        "status": "success",
        "tools_used": ["retrieve_data", "analyze_aggregate"]
      }
    ]
  },
  "thinking": null,
  "tool_calls": [
    {
      "order": 1,
      "task_id": null,
      "name": "check_data",
      "input": { "source_hint": "structured" },
      "output": {
        "kind": "table",
        "columns": ["source_id", "name", "source_type", "table_count"],
        "row_count": 1,
        "preview": [["src_sales_db", "orders", "schema", 1]]
      },
      "status": "success",
      "error": null
    },
    {
      "order": 2,
      "task_id": null,
      "name": "retrieve_data",
      "input": { "ir": { "source_id": "src_sales_db", "table_id": "orders", "select": ["region", "amount"], "group_by": ["region"] } },
      "output": {
        "kind": "table",
        "columns": ["region", "total"],
        "row_count": 4,
        "preview": [["Central", 1210000], ["East", 740000]]
      },
      "status": "success",
      "error": null
    }
  ],
  "sources": [
    {
      "type": "database",
      "source_id": "src_sales_db",
      "name": "orders",
      "query": "SELECT region, SUM(amount) AS total FROM orders GROUP BY region",
      "detail": {
        "table": "orders",
        "row_count": 4
      }
    }
  ]
}

Note: retrieve_data's real input is the compiled query IR under an ir key (the planner builds an IR, never raw SQL). The executed SQL/rendered query appears on the corresponding sources[].query.

Response 200 for unstructured_flow:

{
  "analysis_id": "an_42",
  "message_id": "msg_55",
  "user_id": "user_7",
  "intent": "unstructured_flow",
  "generated_at": "2026-07-06T03:40:02.001Z",
  "planning": null,
  "thinking": null,
  "tool_calls": [
    {
      "order": 1,
      "task_id": null,
      "name": "retrieve_knowledge",
      "input": { "query": "technology stack used in this project" },
      "output": { "kind": "documents", "row_count": 4 },
      "status": "success",
      "error": null
    }
  ],
  "sources": [
    {
      "type": "document",
      "document_id": "doc_7",
      "filename": "tech_handbook.pdf",
      "page_label": "12",
      "query": "technology stack used in this project",
      "snippet": "The backend is built on FastAPI with async SQLAlchemy...",
      "score": 0.83
    }
  ]
}

Response 200 for chat / greeting / help / refusals (out_of_scope, blocked):

{
  "analysis_id": "an_42",
  "message_id": "msg_12",
  "user_id": "user_7",
  "intent": "chat",
  "generated_at": "2026-07-06T03:05:00.000Z",
  "planning": null,
  "thinking": null,
  "tool_calls": [],
  "sources": []
}

Response 404:

{
  "detail": "No traceability for message 'msg_88f1' yet."
}

Frontend rendering guidance:

• Render traceability separately from the streamed answer.
• Default state can be collapsed.
• Show planning, tool calls, and sources as separate sections.
• Treat planning: null, tool_calls: [], and sources: [] as valid states.