File size: 28,844 Bytes
3bacc1d
 
f873f92
3bacc1d
 
 
 
 
 
 
 
 
 
f873f92
3bacc1d
 
 
 
 
 
f873f92
3bacc1d
 
 
 
 
 
 
 
 
 
 
49b0848
 
3bacc1d
f873f92
5a60e93
3bacc1d
 
 
 
 
 
 
49b0848
3bacc1d
 
 
 
 
 
 
 
 
 
 
f873f92
3bacc1d
 
 
 
 
f873f92
3bacc1d
5a60e93
 
3bacc1d
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
49b0848
3bacc1d
 
 
 
 
 
 
 
f873f92
3bacc1d
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
f873f92
3bacc1d
 
 
 
 
 
 
 
 
 
 
 
 
 
17e31c6
3bacc1d
 
 
 
 
 
 
 
 
 
 
17e31c6
 
 
 
3bacc1d
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
49b0848
3bacc1d
 
 
 
 
49b0848
3bacc1d
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
49b0848
 
 
 
 
 
 
 
 
 
 
 
 
 
3bacc1d
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
49b0848
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
3bacc1d
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
49b0848
 
 
 
 
 
5a60e93
 
 
 
 
 
 
 
 
 
 
 
 
 
49b0848
3bacc1d
 
5a60e93
3bacc1d
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
49b0848
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
3bacc1d
 
 
 
 
 
 
 
 
 
 
 
f873f92
 
 
3bacc1d
f873f92
3bacc1d
f873f92
3bacc1d
f873f92
3bacc1d
 
 
 
 
 
 
 
 
 
 
f873f92
3bacc1d
 
f873f92
 
3bacc1d
 
f873f92
 
32abc41
 
 
 
 
f873f92
32abc41
3bacc1d
 
 
 
 
 
 
f873f92
3bacc1d
f873f92
3bacc1d
 
f873f92
3bacc1d
 
 
 
f873f92
 
 
3bacc1d
 
 
 
f873f92
 
 
3bacc1d
 
 
f873f92
3bacc1d
 
 
f873f92
3bacc1d
32abc41
3bacc1d
f873f92
 
 
 
 
 
 
 
3bacc1d
 
 
f873f92
3bacc1d
32abc41
f873f92
3bacc1d
 
 
 
 
 
f873f92
 
3bacc1d
 
32abc41
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
3bacc1d
 
 
 
32abc41
3bacc1d
32abc41
3bacc1d
 
f873f92
 
3bacc1d
 
 
 
 
 
32abc41
f873f92
3bacc1d
 
 
 
 
 
f873f92
3bacc1d
f873f92
3bacc1d
 
 
 
 
f873f92
3bacc1d
f873f92
 
 
 
3bacc1d
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
f873f92
3bacc1d
 
 
 
 
f873f92
3bacc1d
f873f92
3bacc1d
 
 
 
 
 
 
 
 
 
 
f873f92
3bacc1d
 
 
 
 
f873f92
3bacc1d
 
 
5a60e93
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
ο»Ώ# Backend Agentic Service API Contract

This document describes the Python agentic backend used by the frontend for AI chat, help/report tools, and traceability data shown alongside chat answers.

Base path examples use relative URLs. Configure the frontend with the deployed Python service base URL.

## Overview

The Python backend owns the generative AI interaction surface:

1. Stream chat answers from the AI agent.
2. Execute tool-style actions for help and report generation.
3. Return report versions and report details.
4. Return traceability/provenance for a completed assistant answer.

The frontend uses this service during the analysis conversation flow:

1. User sends a chat message.
2. Frontend calls `POST /api/v2/chat/stream` and renders the streamed answer.
3. When the stream emits `done`, frontend stores or reads the returned `message_id`.
4. Frontend calls `GET /api/v1/traceability` for planning, tool calls, and source provenance.
5. Frontend calls `/api/v1/tools/help` for guided help and `/api/v1/tools/report` for report generation.

## Endpoint Summary

| Method | Path | Purpose |
| --- | --- | --- |
| `POST` | `/api/v2/chat/stream` | Stream an AI chat answer for one analysis conversation. |
| `GET` | `/api/v1/tools/list` | List available frontend tools. |
| `POST` | `/api/v1/tools/help` | Stream contextual help for the current analysis conversation. |
| `POST` | `/api/v1/tools/report` | Generate and persist a new report version. |
| `GET` | `/api/v1/tools/report/{analysis_id}` | List report versions for an analysis. |
| `GET` | `/api/v1/tools/report/{analysis_id}/records` | List analysis records for report curation (added 2026-07-09). |
| `GET` | `/api/v1/tools/report/{analysis_id}/readiness` | Report-readiness signal for the Generate-Report button (added 2026-07-09). |
| `GET` | `/api/v1/tools/report/{analysis_id}/{version}` | Retrieve one report version. |
| `GET` | `/api/v1/traceability` | Retrieve provenance for one assistant answer. |
| `GET` | `/api/v1/charts` | Retrieve chart(s) produced by `render_chart` for one assistant answer (added 2026-07-13). |

## Common Concepts

### Identifiers

- `user_id`: user identifier passed by the frontend.
- `analysis_id`: analysis conversation identifier.
- `message_id`: assistant answer identifier used to correlate chat streaming with traceability. **Server-minted, never accepted from the client.** **Updated 2026-07-09:** it is a UUID string (e.g. `77f06761-0fdf-4cc5-84f8-5f81bcbb6f84`), matching the shape of Go's `analyses_messages.id`. The `msg_…` values in the examples below are illustrative placeholders only.

### Server-Sent Events

Chat and help endpoints return `text/event-stream`.

Frontend should parse events by `event` name and `data` payload. Blank lines separate SSE events.

Common event types:

| Event | Data | Meaning |
| --- | --- | --- |
| `sources` | JSON array | Always `[]` β€” sources moved to `GET /api/v1/traceability` (KM-691). Event kept for backward-compat; read `sources[]` from the traceability call. |
| `status` | text | Optional progress update for slower paths. |
| `chunk` | text | Answer text fragment. Concatenate chunks in order. |
| `done` | JSON object | Terminal success event. Includes `message_id`. |
| `error` | text | Terminal error event. Stream stops after this. |

The stream carries answer text only. Planning, tool call details, and full provenance are fetched from `GET /api/v1/traceability` after the stream is done.

**Charts (added 2026-07-13, SPINE_V2_PLAN Β§4.5):** the `done` event is unchanged by charts β€” no additive `chart_count`/`chart_ids` field yet (open question owned by Harry). Until that's resolved, the frontend fetches `GET /api/v1/charts` unconditionally on every `done`, the same fetch-on-`done` pattern already used for traceability. SSE order and every existing field stay exactly as documented above; `sources` stays `[]`.

## Chat

### `POST /api/v2/chat/stream`

Streams an AI answer for one user message in an analysis conversation.

Request body:

```json
{
  "user_id": "u_1a2b3c",
  "analysis_id": "an_42",
  "message": "What were total sales by region last quarter?"
}
```

Fields:

| Field | Required | Description |
| --- | --- | --- |
| `user_id` | Yes | User identifier. |
| `analysis_id` | Yes | Analysis conversation identifier. |
| ~~`message_id`~~ | β€” | **Updated 2026-07-09:** not a request field. Python always mints the id server-side and returns it on `done`; any caller-sent value is ignored (server-authoritative β€” open-Q #1). |
| `message` | Yes | User message text. |

Response: `text/event-stream`.

Example structured answer:

```text
event: sources
data: []

event: status
data: Planning analysis...

event: status
data: Running 3 steps...

event: chunk
data: Total sales by region last quarter: 

event: chunk
data: Central led at $1.21M (38%), East $0.74M, West $0.55M (down 12% QoQ).

event: done
data: {"message_id":"msg_88f1"}
```

Example simple chat answer:

```text
event: sources
data: []

event: chunk
data: I'm your AI data analyst. Connect a source or ask a question to get started.

event: done
data: {"message_id":"msg_12"}
```

Behavior notes:

- Greeting and farewell messages may use a fast canned path.
- Stateless `chat` intent may use a 1-hour Redis response cache.
- The router may classify messages into intents such as `chat`, `help`, `check`, `unstructured_flow`, or `structured_flow`.
- `sources` in the stream is **always `[]`** (KM-691) β€” read the real `sources[]` from `GET /api/v1/traceability` after `done`.
- `status` events are optional and should be safe for the frontend to ignore.

## Tools

### `GET /api/v1/tools/list`

Returns the deterministic list of tools available to the frontend.

Request: none.

Response `200`:

```json
{
  "count": 1,
  "tools": [
    {
      "command": "/help",
      "name": "help",
      "type": "skill",
      "description": "Show what the assistant can do and guide your next step."
    }
  ]
}
```

> The catalog is `/help` only (KM-711). `/report` was removed as a slash command β€”
> report generation is a right-side **Generate** button, not a `/` command. The report
> HTTP endpoint (`POST /api/v1/tools/report`) still exists; the button calls it.

Tool item shape:

```json
{
  "command": "/help",
  "name": "help",
  "type": "skill",
  "description": "Show what the assistant can do and guide your next step."
}
```

Frontend behavior:

- Surface `/help` in the slash menu.
- Surface report generation as a button or explicit UI action.

### `POST /api/v1/tools/help`

Streams contextual guidance for the current analysis conversation.

Request body:

```json
{
  "user_id": "u_1a2b3c",
  "analysis_id": "an_42"
}
```

Response: `text/event-stream` using the same event shape as chat.

Help responses usually emit `sources: []` and no `status` pings.

Example:

```text
event: sources
data: []

event: chunk
data: Your goal is set. You can start exploring now. Try a question like "average order value by month", then I can generate a report.

event: done
data: {"message_id":"msg_h7"}
```

## Reports

### `POST /api/v1/tools/report`

Generates, persists, and returns a new report version for an analysis.

Query params:

| Query | Required | Description |
| --- | --- | --- |
| `analysis_id` | Yes | Analysis identifier. |
| `user_id` | Yes | User identifier. |
| `exclude_record_ids` | No | Record ids to leave out of this version (repeat the param per id). Added 2026-07-09; get ids from `GET /tools/report/{analysis_id}/records`. Excluded runs are listed in the report's "Excluded Analyses" section. Excluding every substantive record returns `409`. |

Example:

```text
POST /api/v1/tools/report?analysis_id=an_42&user_id=u_1a2b3c
POST /api/v1/tools/report?analysis_id=an_42&user_id=u_1a2b3c&exclude_record_ids=rec_a1&exclude_record_ids=rec_c3
```

Status codes:

| Status | Meaning |
| --- | --- |
| `201` | New report version generated. |
| `409` | Report floor/precondition not met. |
| `500` | Generation or persistence failed. |

Response `201`:

```json
{
  "report_id": "8f3a2b1c9d4e4f6a8b0c1d2e3f4a5b6c",
  "analysis_id": "an_42",
  "user_id": "u_1a2b3c",
  "version": 2,
  "generated_at": "2026-06-30T09:14:33.512Z",
  "problem_statement": {
    "objective": "Understand which regions drive revenue and why Q1 dipped.",
    "business_questions": [
      "Which regions contribute most to total revenue?",
      "Did any region decline quarter-over-quarter?"
    ]
  },
  "record_ids": ["rec_a1", "rec_b2"],
  "executive_summary": "Revenue is concentrated in the Central region (38% of total). The West was the only region to contract, down 12% QoQ, the main driver of the Q1 dip.",
  "bq_answers": [
    {
      "question": "Which regions contribute most to total revenue?",
      "answer": "The Central region leads with 38% of total revenue.",
      "status": "answered",
      "record_ids": ["rec_a1"]
    },
    {
      "question": "Did any region decline quarter-over-quarter?",
      "answer": "Yes β€” the West region fell 12% QoQ.",
      "status": "answered",
      "record_ids": ["rec_b2"]
    }
  ],
  "findings": [
    {
      "text": "Central region contributed 38% of total revenue, the largest share.",
      "record_ids": ["rec_a1"],
      "supporting_data": null
    },
    {
      "text": "West region revenue fell 12% quarter-over-quarter.",
      "record_ids": ["rec_b2"],
      "supporting_data": null
    }
  ],
  "caveats": [
    {
      "text": "March data for the East region was partially missing, around 6% of rows.",
      "record_ids": ["rec_b2"]
    }
  ],
  "open_questions": [
    {
      "text": "What drove the West region's QoQ decline?",
      "record_ids": ["rec_b2"]
    }
  ],
  "unresolved": [
    {
      "text": "Correlate churn with tenure β€” churn column not found in the source.",
      "record_ids": ["rec_d4"]
    }
  ],
  "excluded": [],
  "evidence_tables": {
    "rec_a1": [
      {
        "title": "Aggregate revenue by region",
        "columns": ["region", "total_revenue"],
        "rows": [["Central", "18321"], ["West", "9954"]],
        "truncated": false
      }
    ]
  },
  "data_sources": [
    {
      "source_id": "src_sales_db",
      "name": "orders",
      "source_type": "postgres",
      "detail": {
        "tables": ["orders"],
        "row_count": 48213,
        "columns": ["region", "amount", "ordered_at"]
      }
    }
  ],
  "method_steps": [
    {
      "task_id": "t1",
      "stage": "data_understanding",
      "objective": "Inventory the sales source",
      "status": "success",
      "tools_used": ["check_data"]
    },
    {
      "task_id": "t2",
      "stage": "modeling",
      "objective": "Aggregate revenue by region",
      "status": "success",
      "tools_used": ["analyze_aggregate"]
    }
  ],
  "rendered_markdown": "# Analysis Report\n\n*Generated 2026-06-30 by u_1a2b3c*\n\n## Objective\nUnderstand which regions drive revenue..."
}
```

Response `409`:

```json
{
  "detail": "Not ready to generate a report - still needs at least one completed analysis."
}
```

Report v2 fields (added 2026-07-09; all default-empty, so older stored reports read back unchanged):

- `bq_answers` β€” one entry per business question. `status` is `answered` | `partial` | `unanswered`; `record_ids` cite the backing analyses. Written in the analysis's language (Indonesian objective β†’ Indonesian answers).
- `unresolved` β€” runs that were attempted but produced no usable evidence (every `analyze_*` step failed). Not part of the findings body.
- `excluded` β€” runs the caller excluded via `exclude_record_ids`.
- `evidence_tables` β€” `record_id` β†’ small result tables copied from the run's stored outputs (max 3 tables per record, max 10 rows each; `truncated: true` when rows were capped). Rendered as markdown tables under the matching Key Findings group in `rendered_markdown`.
- `charts` *(added 2026-07-14)* β€” `record_id` β†’ `dataeyond.chart.v1` envelopes (see Β§Charts) copied verbatim from the run's stored outputs (max 3 per record). `rendered_markdown` gains an `## EDA` section where each chart appears as a fenced block the frontend renders with plotly.js:

  ````text
  ```plotly
  {
    "schema": "dataeyond.chart.v1",
    "chart_type": "bar",
    "title": "…",
    "plotly": { "data": [ … ], "layout": { … } }
  }
  ```
  ````

  The fence content is the **full envelope** (same shape as `charts[].spec` on `GET /api/v1/charts`) β€” the FE hook parses it and renders `Plotly.newPlot(el, parsed.plotly.data, parsed.plotly.layout)`. A bold caption line (chart title) precedes each fence.

Precondition:

- Reports require at least one completed analysis record for the session (*updated 2026-07-14:* a run whose `render_chart` succeeded counts β€” a chart-only session can generate a report).
- If slow-path analysis recording is disabled, report generation can return `409` by design.

### `GET /api/v1/tools/report/{analysis_id}`

Lists report versions for one analysis, oldest first.

Response `200`:

```json
[
  {
    "report_id": "1b2c3d4e",
    "version": 1,
    "generated_at": "2026-06-24T15:02:11Z",
    "record_count": 1
  },
  {
    "report_id": "8f3a2b1c",
    "version": 2,
    "generated_at": "2026-06-25T09:14:33Z",
    "record_count": 2
  }
]
```

If no reports exist, returns `[]`.

### `GET /api/v1/tools/report/{analysis_id}/records` (added 2026-07-09)

Lists the persisted analysis runs a report would be built from, oldest first. The frontend shows this before generating so the user can deselect runs; the chosen ids go to `POST /tools/report` as `exclude_record_ids`.

Response `200`:

```json
[
  {
    "record_id": "rec_a1",
    "goal_restated": "Rank regions by total revenue",
    "created_at": "2026-06-30T08:55:02Z",
    "substantive": true,
    "findings_count": 2
  },
  {
    "record_id": "rec_d4",
    "goal_restated": "Correlate churn with tenure",
    "created_at": "2026-06-30T09:01:47Z",
    "substantive": false,
    "findings_count": 1
  }
]
```

`substantive: false` means no `analyze_*` step succeeded β€” that run is listed in the report's `unresolved` JSON field rather than the findings body. (Since 2026-07-09 the rendered markdown is compact and no longer includes "Attempted, Unresolved" / "Notes & Limitations" / "How This Was Analyzed" sections; the JSON fields `unresolved` / `caveats` / `open_questions` / `method_steps` are unchanged.) If no runs exist, returns `[]`.

### `GET /api/v1/tools/report/{analysis_id}/readiness` (added 2026-07-09)

Deterministic report-readiness signal for the Generate-Report button β€” the same producer as Help's readiness signal, including the advisory delta-since-report check, so the button, Help, and this endpoint never disagree.

Response `200`:

```json
{
  "ready": false,
  "missing": ["a new analysis since the last report"]
}
```

Note: `POST /tools/report` itself only enforces the floor (`at least one completed analysis`) β€” a new version is always allowed. The delta gap in `missing` is a soft warning the frontend can surface ("nothing new since the last report") without blocking the button.

### `GET /api/v1/tools/report/{analysis_id}/{version}`

Returns one report version. Shape is the same as the `201` response from `POST /api/v1/tools/report`.

Response `404`:

```json
{
  "detail": "No report v3 for analysis 'an_42'."
}
```

## Traceability

> Renamed from `observability` (team decision 2026-07-06) so it is never confused with the internal Langfuse *observability* stack (engineering telemetry, PII-masked). Traceability is **user-facing** provenance β€” real tool args, output previews, and the executed query β€” shown alongside the answer.

### `GET /api/v1/traceability`

Returns user-facing provenance for one assistant answer.

The frontend should call this after the chat/help stream emits `done`, using the `message_id` from the `done` event. The row is written **before** `done`, so an immediate GET returns `200` (no polling race). A `404` means the id is unknown or the turn errored before completing (error turns never produce a row).

Query params:

| Query | Required | Description |
| --- | --- | --- |
| `analysis_id` | Yes | Analysis identifier. |
| `message_id` | Yes | Assistant answer identifier returned by the stream. |

Example:

```text
GET /api/v1/traceability?analysis_id=an_42&message_id=msg_88f1
```

`intent` values the frontend may see: `chat` Β· `help` Β· `check` Β· `unstructured_flow` Β· `structured_flow` Β· `out_of_scope` Β· `blocked` (`blocked` = input-guard or Azure content-filter refusal; `chat` also covers the greeting fast-path and cache replays).

Field rules:

- `planning`: present only when the planner ran (`structured_flow`); otherwise `null`.
- `thinking`: **always `null` in v1** β€” our agents are plain chat completions with no native reasoning output, and synthesizing it post-hoc would be unfaithful. The field stays in the payload so it can be populated later without a contract change.
- `tool_calls`: every invoked tool with `summary` (plain-English one-liner), `input`, `output`, `status`, `task_id` (nullable), and `error` (nullable); empty for chat / help / greeting / refusal paths. `input`/`output` are the raw tool I/O (opaque ids) β€” render them in a collapsible "technical details" section, not the headline; use `summary` for the headline.
- `data_used`: one entry per `retrieve_data` call, resolved to **real names** for display (present only when a structured pull ran; empty otherwise). Split into `columns_read` (columns read straight from the user's data, each tagged with its `roles`) and `output_columns` (`kind: "column"` = read from data, `kind: "computed"` = calculated, carrying a `formula` and **no id**). Also carries `tables` (all touched, incl. join targets), `joins`, `filters` (with a plain-language `description`), `group_by`, `order_by`, `limit`, `rows_returned`, and the executed `query`. Built by deterministic catalog lookup β€” no LLM.
- **`id` fields are machine-only.** Every `id` in `data_used` (`source.id`, `tables[].id`, `columns_read[].id`) is for linking/reconciliation/audit β€” **the frontend must never render it.** Show `name` (qualified as `table.name`). A `computed` output column has no id by design.
- `sources`: required for retrieval flows; empty for chat / help / refusal paths and for `check`. Database sources also carry `source_name` (the DB's real name) and `tables` (every table touched).
- `thinking`, `filters[].description`, `tool_calls[].summary` are built from fixed templates, never an LLM β€” traceability adds no latency or token cost and cannot hallucinate.
- The payload also carries an internal `user_id` (ownership); the frontend may ignore it.
- Truncation: `preview` ≀ 5 rows; any string inside `input`/`output`/`preview`/`snippet` ≀ 300 chars (executed `query` ≀ 2000); rows beyond the preview are dropped (`row_count` is preserved).

Response `200` for `structured_flow`:

```json
{
  "analysis_id": "an_42",
  "message_id": "msg_88f1",
  "user_id": "user_7",
  "intent": "structured_flow",
  "generated_at": "2026-07-06T03:21:09.114Z",
  "planning": {
    "goal_restated": "Find which regions drive revenue and why Q1 dipped.",
    "assumptions": [],
    "steps": [
      {
        "step": 1,
        "stage": "data_understanding",
        "objective": "Inventory the sales source",
        "status": "success",
        "tools_used": ["check_data"]
      },
      {
        "step": 2,
        "stage": "modeling",
        "objective": "Aggregate revenue by region",
        "status": "success",
        "tools_used": ["retrieve_data", "analyze_aggregate"]
      }
    ]
  },
  "thinking": null,
  "tool_calls": [
    {
      "order": 1,
      "task_id": null,
      "name": "check_data",
      "summary": "Inspected your data source structure",
      "input": { "source_hint": "structured" },
      "output": {
        "kind": "table",
        "columns": ["source_id", "name", "source_type", "table_count"],
        "row_count": 1,
        "preview": [["src_sales_db", "orders", "schema", 1]]
      },
      "status": "success",
      "error": null
    },
    {
      "order": 2,
      "task_id": null,
      "name": "retrieve_data",
      "summary": "Retrieved 4 rows across 2 columns from orders",
      "input": { "ir": { "source_id": "src_sales_db", "table_id": "orders", "select": ["region", "amount"], "group_by": ["region"] } },
      "output": {
        "kind": "table",
        "columns": ["region", "total"],
        "row_count": 4,
        "preview": [["Central", 1210000], ["East", 740000]]
      },
      "status": "success",
      "error": null
    }
  ],
  "data_used": [
    {
      "source": { "id": "src_sales_db", "name": "sales db", "type": "schema" },
      "tables": [ { "id": "orders", "name": "orders", "role": "base" } ],
      "joins": [],
      "columns_read": [
        { "id": "c_region", "name": "region", "table": "orders", "data_type": "string",  "pii": false, "roles": ["selected", "grouped"] },
        { "id": "c_amount", "name": "amount", "table": "orders", "data_type": "decimal", "pii": false, "roles": ["aggregated"] }
      ],
      "output_columns": [
        { "name": "region", "kind": "column",   "from": "orders.region" },
        { "name": "total",  "kind": "computed", "from": "orders.amount", "formula": "SUM(orders.amount)" }
      ],
      "filters": [],
      "group_by": ["orders.region"],
      "order_by": [],
      "limit": null,
      "rows_returned": 4,
      "query": "SELECT region, SUM(amount) AS total FROM orders GROUP BY region"
    }
  ],
  "sources": [
    {
      "type": "database",
      "source_id": "src_sales_db",
      "source_name": "sales db",
      "name": "orders",
      "tables": ["orders"],
      "query": "SELECT region, SUM(amount) AS total FROM orders GROUP BY region",
      "detail": {
        "table": "orders",
        "row_count": 4
      }
    }
  ]
}
```

> Note: `tool_calls[].input` is the raw compiled query IR (opaque `column_id`/`table_id`) β€” the technical layer. **`data_used` is the user-facing layer**: the same pull resolved to real names, with computed columns (e.g. `total`) flagged `kind: "computed"` so they are never shown as if they were columns in the user's database. Every `id` there is machine-only.

Response `200` for `unstructured_flow`:

```json
{
  "analysis_id": "an_42",
  "message_id": "msg_55",
  "user_id": "user_7",
  "intent": "unstructured_flow",
  "generated_at": "2026-07-06T03:40:02.001Z",
  "planning": null,
  "thinking": null,
  "tool_calls": [
    {
      "order": 1,
      "task_id": null,
      "name": "retrieve_knowledge",
      "input": { "query": "technology stack used in this project" },
      "output": { "kind": "documents", "row_count": 4 },
      "status": "success",
      "error": null
    }
  ],
  "sources": [
    {
      "type": "document",
      "document_id": "doc_7",
      "filename": "tech_handbook.pdf",
      "page_label": "12",
      "query": "technology stack used in this project",
      "snippet": "The backend is built on FastAPI with async SQLAlchemy...",
      "score": 0.83
    }
  ]
}
```

Response `200` for chat / greeting / help / refusals (`out_of_scope`, `blocked`):

```json
{
  "analysis_id": "an_42",
  "message_id": "msg_12",
  "user_id": "user_7",
  "intent": "chat",
  "generated_at": "2026-07-06T03:05:00.000Z",
  "planning": null,
  "thinking": null,
  "tool_calls": [],
  "sources": []
}
```

Response `404`:

```json
{
  "detail": "No traceability for message 'msg_88f1' yet."
}
```

Frontend rendering guidance:

- Render traceability separately from the streamed answer.
- Default state can be collapsed.
- Show planning, tool calls, and sources as separate sections.
- Treat `planning: null`, `tool_calls: []`, and `sources: []` as valid states.

## Charts

> Added 2026-07-13 (SPINE_V2_PLAN Β§4.4/Β§4.5, S2 visualization). A chart is a `render_chart` tool output, planner-selected only when the user explicitly asks to plot/visualize. Delivery mirrors traceability: a Python-owned store plus a dedicated `GET` endpoint; the streamed answer (SSE) stays text-only β€” charts are never embedded in `chunk`.

### `GET /api/v1/charts`

Returns every chart produced by `render_chart` during one assistant answer.

The frontend should call this after the chat stream emits `done`, using the `message_id` from the `done` event β€” the same fetch-on-`done` pattern as traceability; the row(s) are written **before** `done`, so there is no polling race.

> **Updated 2026-07-13 (lead review):** lookup is now by **`message_id` alone** (it is a server-minted UUID β€” globally unique; `analysis_id` was removed from the query), and every response is **HTTP 200 with an explicit `status` marker** instead of a bare list: `success` (β‰₯1 chart), `empty` (the turn completed but produced no charts β€” the common case), `not_found` (no completed turn is known for this `message_id`: mistyped/stale id, or an error turn). *(Note the asymmetry: `GET /api/v1/traceability` still takes `analysis_id` + `message_id` β€” aligning it is a separate change if wanted.)*

Query params:

| Query | Required | Description |
| --- | --- | --- |
| `message_id` | Yes | Assistant answer identifier returned by the stream's `done` event. |

Example:

```text
GET /api/v1/charts?message_id=88f10c3a-6f03-4204-bf98-41ffc20388b2
```

The `dataeyond.chart.v1` envelope (the shape of `charts[].spec`, verbatim from `render_chart`):

```json
{
  "schema": "dataeyond.chart.v1",
  "chart_type": "bar",
  "title": "Revenue by region",
  "plotly": {
    "data":   [{ "type": "bar", "x": ["A", "B"], "y": [1, 2], "name": "revenue" }],
    "layout": { "title": {"text": "Revenue by region"}, "xaxis": {"title": {"text": "region"}},
                "yaxis": {"title": {"text": "revenue"}} }
  }
}
```

Frontend renders it with `Plotly.newPlot(el, spec.plotly.data, spec.plotly.layout)`. v1 chart types: `bar`, `line`, `pie`, `scatter`.

Response `200` (`status: success` β€” β‰₯1 chart):

```json
{
  "status": "success",
  "message": "1 chart(s) for this message.",
  "count": 1,
  "charts": [
    {
      "chart_id": "3fbd8e2e-8e21-4d4b-9b21-9e6b6a0a6a6e",
      "chart_type": "bar",
      "title": "Revenue by region",
      "spec": {
        "schema": "dataeyond.chart.v1",
        "chart_type": "bar",
        "title": "Revenue by region",
        "plotly": {
          "data":   [{ "type": "bar", "x": ["A", "B"], "y": [1, 2], "name": "revenue" }],
          "layout": { "title": {"text": "Revenue by region"}, "xaxis": {"title": {"text": "region"}},
                      "yaxis": {"title": {"text": "revenue"}} }
        }
      },
      "created_at": "2026-07-13T03:21:09.114Z"
    }
  ]
}
```

Response `200` (`status: empty` β€” chartless turn, the common case; not an error):

```json
{
  "status": "empty",
  "message": "This message completed without producing charts.",
  "count": 0,
  "charts": []
}
```

Response `200` (`status: not_found` β€” no completed turn known for this id):

```json
{
  "status": "not_found",
  "message": "No completed turn is known for this message_id.",
  "count": 0,
  "charts": []
}
```

Field rules:

- `status` is the outcome marker: `success` | `empty` | `not_found` (always HTTP 200 β€” the FE fetches unconditionally, so a missing turn is a data state, not a transport failure). `empty` vs `not_found` is decided against the turn's traceability row (written before `done`), so `not_found` reliably means "this id never completed a turn".
- `message` is a human-readable line for logs/debugging β€” do not parse it; branch on `status`.
- `spec` is the full `dataeyond.chart.v1` envelope, unmodified β€” it is the source of truth, not a projection; render straight from it.
- `chart_type` / `title` are copied out of `spec` for convenience (list rendering without parsing `spec`); `title` may be `null`.
- A turn can produce more than one chart (multiple `render_chart` calls in the same plan); `charts` is ordered by creation time.
- The payload carries no `user_id` / `analysis_id` β€” charts are keyed by `message_id` alone.

> **DDL note (Harry / dedorch migration):** the original manual index is `(analysis_id, message_id)`, which does not serve a `message_id`-only lookup. Additive index for the migration (also safe to run manually now):
> ```sql
> CREATE INDEX IF NOT EXISTS idx_message_charts_message ON message_charts (message_id);
> ```

Frontend rendering guidance:

- Fetch unconditionally on every `done` β€” see the SSE note above (no `chart_count` hint yet); an empty `charts[]` means render nothing extra.
- Render each chart under the assistant message it belongs to, via `Plotly.newPlot`.
- "Chart iteration" v1 = a follow-up chat turn (e.g. "make it a line chart") β€” the planner re-emits `render_chart` with patched args. There is no separate edit endpoint.