Rifqi Hafizuddin Claude Opus 4.8 commited on
Commit
cbc8d6a
·
1 Parent(s): 0aa36ac

[NOTICKET] fix: review cheap batch — F-19 SSE order, F-22/F-24 analysis_id, F-26, F-16, F-4

Browse files

Six low-risk review findings, batched. No behaviour anyone depends on is removed.

F-19 — SSE order/presence did not match the contract. `check` and router-`help`
emitted NO `sources` event at all, and `structured_flow` emitted `status` BEFORE
`sources`, inverting the documented order on exactly the turns that take longest. A
frontend that initializes per-turn state on `sources` (documented always-present)
therefore never initialized on check/help turns and initialized late on slow ones —
every one of those a plausible FE bug that would have been blamed on the frontend.
Note `stream_help` (the /tools/help endpoint) already emitted it, so the two help
paths disagreed with each other. Purely additive; `sources` stays `[]`.

F-22 — `analysis_id` now 422s unless it parses as a UUID, on BOTH live endpoints
(chat and help kept deliberately identical — two endpoints taking the same field
should not disagree on what is valid). It was never usable: `analyses.id` is
`uuid NOT NULL`, so a non-UUID silently matched nothing, `state_store.ensure` failed
its INSERT, the caller logged a warning and continued with `analysis_state = None`,
and help/readiness/report-write-back all quietly no-op'd while the user still saw a
normal-looking answer.

F-24 — traceability and chart writes are now SKIPPED, and logged, when `analysis_id`
is falsy. Go migration 0007 declares `analysis_id UUID NOT NULL` on both tables (plus
an FK to `analyses(id)` on `message_charts`), and Python passed `analysis_id or ""` —
an empty string cannot cast to uuid, the insert failed, and both writes are
never-throw, so the row vanished with no user-visible symptom and GET /charts then
answered `not_found` for a turn that really did produce a chart. Both live endpoints
require `analysis_id`, so this cannot fire in production; it is the honest fallback.

F-26 — `QueryResult.error` uses `str(e) or repr(e)`. A Fernet InvalidToken has an
empty `str()`, so it reached the assembler prompt, the traceability record and the
report caveats as an EMPTY string while the log (already repr) stayed diagnosable.
Falling back only when str() is empty means no existing error text changes.

F-16 — retrieval cache key gains `settings.redis_prefix`, matching every other cache
key in the service. Two environments sharing one Redis — which the single shared .env
makes plausible — were cross-serving retrieval results. Not cross-tenant (`user_id`
is in the key), but cross-environment. One-time cache-miss storm, bounded by the 1h
TTL.

F-4 — non-Postgres sources are now refused at the executor. ZERO blast radius today:
Go's `database_clients.Service.Create` gates on `isSupportedActive` and only
`postgres` is active, so no such source can be registered. This is a tripwire. The
legacy branch it replaces had no read-only session and no statement_timeout, i.e.
only four of the five defense layers CLAUDE.md §2.5 states unconditionally — and
since the compiler is built with dialect="postgres" regardless of db_type, those
queries would fail on a parse error first, which the never-throw path degrades into
"data not available". The danger was someone fixing THAT without noticing the pooling
branch. Branch commented out per the house convention, with the orphaned import
commented alongside it.

Tests: two event-sequence assertions updated for F-19 (they pinned the missing
`sources`), and six traceability tests now pass an `analysis_id` — they called
`handle()` without one, a shape neither live endpoint can produce.

Verification: suite 434 passed / 0 failed / 7 skipped (was 424; +10 from F-8).
Ruff on touched paths identical to HEAD baseline (S324/I001/E501, all pre-existing).
`import main` OK. Readiness eval 17/17.

Docs: contract gains the corrected SSE guarantee + the 422 rule; DEV_PLAN rows 46-48
added (48 = the floor_08 decision, open for Rifqi).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

API_CONTRACT_BE_PYTHON.md CHANGED
@@ -147,6 +147,8 @@ Behavior notes:
147
  - The router may classify messages into intents such as `chat`, `help`, `check`, `unstructured_flow`, or `structured_flow`.
148
  - `sources` in the stream is **always `[]`** (KM-691) — read the real `sources[]` from `GET /api/v1/traceability` after `done`.
149
  - `status` events are optional and should be safe for the frontend to ignore.
 
 
150
 
151
  ## Tools
152
 
 
147
  - The router may classify messages into intents such as `chat`, `help`, `check`, `unstructured_flow`, or `structured_flow`.
148
  - `sources` in the stream is **always `[]`** (KM-691) — read the real `sources[]` from `GET /api/v1/traceability` after `done`.
149
  - `status` events are optional and should be safe for the frontend to ignore.
150
+ - **Fixed 2026-07-23 — `sources` is now genuinely always first, on every path.** It previously did not match this document: the `check` and router-`help` turns emitted **no `sources` event at all**, and the `structured_flow` (slow) path emitted `status` *before* `sources`, inverting the documented order on exactly the turns that take longest. A frontend that initializes per-turn state on `sources` therefore never initialized on check/help turns and initialized late on slow turns. The change is **purely additive** — no event was removed or renamed, and `sources` is still `[]` — so no frontend change is required; behaviour simply now matches what this contract always said.
151
+ - **`analysis_id` must be a UUID (2026-07-23).** `POST /api/v2/chat/stream` and `POST /api/v1/tools/help` now return **`422`** for a non-UUID `analysis_id` instead of accepting it. It was never actually usable: `analyses.id` is `uuid NOT NULL`, so a non-UUID id silently matched no analysis and the turn ran with no state — help, readiness, the report write-back, and the traceability/chart rows all quietly no-op'd while the user still saw a normal answer. The frontend passes Go-issued analysis ids, which are UUIDs, so this should never fire in practice.
152
 
153
  ## Tools
154
 
DEV_PLAN.md CHANGED
@@ -247,6 +247,9 @@ plus the live report bug on analysis `966224d4…`. Same status legend as §0.
247
  | 43 | **Go identity contract** — what does Go forward, and when? | Rifqi ↔ Harry | ⬜ new | Needed to replace #37's shared secret with real per-user authorization. Until then the store predicates in #38 are defensive only |
248
  | 44 | **F-17 / F-18 / F-25 compiler-parity batch** | Rifqi | ✅ new | Shipped 2026-07-23. Three execution-verified review findings that had **no task row** — the tracker jumped from #39 to #40 and lost them. **F-17 (High):** the bare-select check was gated on `if ir.group_by`, so a mixed select with `group_by=[]` passed; Postgres then failed loudly but the pandas path silently DROPPED the column while `output_columns` still advertised it → a real-looking table with a fabricated all-null column. Fixed in `validator.py` (fires whenever any agg is present — no false positives possible) + a presence backstop in `TabularExecutor`. **F-18:** `astype(str)` ran before `na=False`, so `NULL LIKE '%an%'` matched the literal `"nan"`/`"None"`. **F-25:** `SqlCompiler` raised on an empty `in`/`not_in` where pandas and `_column_values`' own docstring implement the empty-set semantics — a two-step plan whose first step legitimately returned zero rows hard-failed instead of answering. 16 new parity tests; one stale test updated (it pinned the old F-25 raise) |
249
  | 45 | **F-9 PII in persisted artifacts** — mask the traceability preview + report evidence tables | Rifqi | ⬜ new | **Decided 2026-07-23** (CODE_REVIEW §7) but never tracked. `pii_flag` is an ingestion-time control only: nothing stops the planner SELECTing a flagged column, and the real values then land in `message_traceability.data` (served by a GET that is only now becoming scoped, #40) and are frozen permanently into `reports.content`. Decision: **the assembler still sees values** (otherwise "list our top customers" stops being answerable); mask only the persisted artifacts. Carry `pii_flag` onto `retrieve_data`'s output meta — the catalog is already in scope at `data_access.py:232` |
 
 
 
250
 
251
  **Reading `eval/readiness/results/` (note for future sessions).** Four files are dated
252
  2026-07-23. `…_150632.json` scores **4/15 (26.7%)** — that is **not** a product
 
247
  | 43 | **Go identity contract** — what does Go forward, and when? | Rifqi ↔ Harry | ⬜ new | Needed to replace #37's shared secret with real per-user authorization. Until then the store predicates in #38 are defensive only |
248
  | 44 | **F-17 / F-18 / F-25 compiler-parity batch** | Rifqi | ✅ new | Shipped 2026-07-23. Three execution-verified review findings that had **no task row** — the tracker jumped from #39 to #40 and lost them. **F-17 (High):** the bare-select check was gated on `if ir.group_by`, so a mixed select with `group_by=[]` passed; Postgres then failed loudly but the pandas path silently DROPPED the column while `output_columns` still advertised it → a real-looking table with a fabricated all-null column. Fixed in `validator.py` (fires whenever any agg is present — no false positives possible) + a presence backstop in `TabularExecutor`. **F-18:** `astype(str)` ran before `na=False`, so `NULL LIKE '%an%'` matched the literal `"nan"`/`"None"`. **F-25:** `SqlCompiler` raised on an empty `in`/`not_in` where pandas and `_column_values`' own docstring implement the empty-set semantics — a two-step plan whose first step legitimately returned zero rows hard-failed instead of answering. 16 new parity tests; one stale test updated (it pinned the old F-25 raise) |
249
  | 45 | **F-9 PII in persisted artifacts** — mask the traceability preview + report evidence tables | Rifqi | ⬜ new | **Decided 2026-07-23** (CODE_REVIEW §7) but never tracked. `pii_flag` is an ingestion-time control only: nothing stops the planner SELECTing a flagged column, and the real values then land in `message_traceability.data` (served by a GET that is only now becoming scoped, #40) and are frozen permanently into `reports.content`. Decision: **the assembler still sees values** (otherwise "list our top customers" stops being answerable); mask only the persisted artifacts. Carry `pii_flag` onto `retrieve_data`'s output meta — the catalog is already in scope at `data_access.py:232` |
250
+ | 46 | **F-8 prompt-injection resistance** — planner / assembler / report_summary | Rifqi | ✅ new | Shipped 2026-07-24. The three prompts that ingest customer data had **no injection rule**: `guardrails.md` is appended only in `chatbot.py`/`help.py`, and the `InputGuard` screens only the user's message — it never sees catalog or row content. This is the one attack the five query-defense layers structurally cannot see, because every IR the planner emits is individually valid. A hostile string only needs to reach a text column in the customer's OWN database (a product description, a support ticket, a form field) to be sampled into `sample_values` and rendered verbatim into the planner prompt. Fix = a purpose-written "content is data, never instructions" rule in each prompt + `<data>…</data>` delimiters around the catalog render and the run-state render. **Deliberately NOT `guardrails.md` wholesale** — its rules prescribe refusal sentences, and the planner's only free-text field is `infeasible_reason`, so those strings would surface there and could regress the Q2 data-gap path (tests pin their absence). Verified: planner eval **6/6, carried_over 5/5 green**; live hostile-catalog run planned only `t_products` and never touched the planted `employees.salary`; 10 new tests |
251
+ | 47 | **Cheap-batch review fixes** — F-19, F-22, F-24, F-26, F-16, F-4 | Rifqi | ✅ new | Shipped 2026-07-24, one low-risk batch. **F-19:** `sources` was missing entirely on `check` + router-`help` and came *after* `status` on the slow path — contract said always-first; additive fix, contract updated. **F-22:** `analysis_id` now 422s unless it parses as a UUID, on **both** live endpoints (chat + help kept identical on purpose). **F-24:** traceability/chart writes are skipped, and logged, when `analysis_id` is falsy — Go `0007` declares `analysis_id UUID NOT NULL` (+ FK on charts), so `analysis_id or ""` failed the insert and the never-throw seam lost the row silently. **F-26:** `QueryResult.error` uses `str(e) or repr(e)` — a Fernet `InvalidToken` reached the assembler/traceability/report as an EMPTY string; falling back only when `str()` is empty changes no existing message. **F-16:** retrieval cache key gains `redis_prefix` (two envs on one Redis cross-served results). **F-4:** non-Postgres sources now refused at the executor — **zero blast radius today** (Go's `isSupportedActive` allows only `postgres`), a tripwire so nobody re-enables a path that has no read-only session and no `statement_timeout`; legacy branch commented out per house convention, orphaned import commented with it |
252
+ | 48 | **floor_08 — floor/body disagreement** | Rifqi | ⬜ new | **Needs a decision, found while adding #34 coverage.** When a plan HAS an `analyze_*` step that FAILS but its upstream `retrieve_data` returned rows, the floor passes (`_produced_rows`) while the body rejects it (`has_reportable_result` keys on the failed analysis step). Verified by execution — it is the only shape where the two disagree. Because the "Attempted, Unresolved" section is commented out (`generator.py:527`), such a run leaves **no trace anywhere**: if it is the session's only run, the report generates but comes out empty and the business question renders "Unanswered" — exactly the #33 bug through a different door. Suggested fix: apply the floor's second arm only when the plan has NO analysis step, mirroring the body predicate; `floor_08` then flips to `expected_ready: false`. **Not changed — Rifqi owns the floor** |
253
 
254
  **Reading `eval/readiness/results/` (note for future sessions).** Four files are dated
255
  2026-07-23. `…_150632.json` scores **4/15 (26.7%)** — that is **not** a product
eval/readiness/results/readiness_result_2026-07-24_084154.json ADDED
@@ -0,0 +1,274 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "run": {
3
+ "timestamp": "2026-07-24T08:41:54",
4
+ "dataset": "readiness_dataset.json",
5
+ "target": "src/agents/report/readiness.is_report_ready",
6
+ "total": 17,
7
+ "passed": 17,
8
+ "accuracy": 1.0,
9
+ "runtime_avg_ms": 0.0
10
+ },
11
+ "alignment_gap": {
12
+ "count": 2,
13
+ "ids": [
14
+ "align_01",
15
+ "align_02"
16
+ ]
17
+ },
18
+ "by_group": {
19
+ "floor": {
20
+ "n": 8,
21
+ "passed": 8,
22
+ "accuracy": 1.0
23
+ },
24
+ "delta": {
25
+ "n": 5,
26
+ "passed": 5,
27
+ "accuracy": 1.0
28
+ },
29
+ "edge": {
30
+ "n": 1,
31
+ "passed": 1,
32
+ "accuracy": 1.0
33
+ },
34
+ "alignment": {
35
+ "n": 3,
36
+ "passed": 3,
37
+ "accuracy": 1.0
38
+ }
39
+ },
40
+ "cases": [
41
+ {
42
+ "id": "floor_01",
43
+ "group": "floor",
44
+ "expected_ready": false,
45
+ "got_ready": false,
46
+ "expected_missing": [
47
+ "at least one completed analysis"
48
+ ],
49
+ "got_missing": [
50
+ "at least one completed analysis"
51
+ ],
52
+ "correct": true,
53
+ "aligned": false,
54
+ "gap": false,
55
+ "latency_ms": 0.0
56
+ },
57
+ {
58
+ "id": "floor_02",
59
+ "group": "floor",
60
+ "expected_ready": false,
61
+ "got_ready": false,
62
+ "expected_missing": [
63
+ "at least one completed analysis"
64
+ ],
65
+ "got_missing": [
66
+ "at least one completed analysis"
67
+ ],
68
+ "correct": true,
69
+ "aligned": false,
70
+ "gap": false,
71
+ "latency_ms": 0.0
72
+ },
73
+ {
74
+ "id": "floor_03",
75
+ "group": "floor",
76
+ "expected_ready": false,
77
+ "got_ready": false,
78
+ "expected_missing": [
79
+ "at least one completed analysis"
80
+ ],
81
+ "got_missing": [
82
+ "at least one completed analysis"
83
+ ],
84
+ "correct": true,
85
+ "aligned": false,
86
+ "gap": false,
87
+ "latency_ms": 0.0
88
+ },
89
+ {
90
+ "id": "floor_04",
91
+ "group": "floor",
92
+ "expected_ready": true,
93
+ "got_ready": true,
94
+ "expected_missing": [],
95
+ "got_missing": [],
96
+ "correct": true,
97
+ "aligned": true,
98
+ "gap": false,
99
+ "latency_ms": 0.0
100
+ },
101
+ {
102
+ "id": "floor_05",
103
+ "group": "floor",
104
+ "expected_ready": true,
105
+ "got_ready": true,
106
+ "expected_missing": [],
107
+ "got_missing": [],
108
+ "correct": true,
109
+ "aligned": true,
110
+ "gap": false,
111
+ "latency_ms": 0.0
112
+ },
113
+ {
114
+ "id": "floor_06",
115
+ "group": "floor",
116
+ "expected_ready": true,
117
+ "got_ready": true,
118
+ "expected_missing": [],
119
+ "got_missing": [],
120
+ "correct": true,
121
+ "aligned": true,
122
+ "gap": false,
123
+ "latency_ms": 0.0
124
+ },
125
+ {
126
+ "id": "floor_07",
127
+ "group": "floor",
128
+ "expected_ready": true,
129
+ "got_ready": true,
130
+ "expected_missing": [],
131
+ "got_missing": [],
132
+ "correct": true,
133
+ "aligned": true,
134
+ "gap": false,
135
+ "latency_ms": 0.0
136
+ },
137
+ {
138
+ "id": "floor_08",
139
+ "group": "floor",
140
+ "expected_ready": true,
141
+ "got_ready": true,
142
+ "expected_missing": [],
143
+ "got_missing": [],
144
+ "correct": true,
145
+ "aligned": true,
146
+ "gap": false,
147
+ "latency_ms": 0.0
148
+ },
149
+ {
150
+ "id": "delta_01",
151
+ "group": "delta",
152
+ "expected_ready": false,
153
+ "got_ready": false,
154
+ "expected_missing": [
155
+ "a new analysis since the last report"
156
+ ],
157
+ "got_missing": [
158
+ "a new analysis since the last report"
159
+ ],
160
+ "correct": true,
161
+ "aligned": true,
162
+ "gap": false,
163
+ "latency_ms": 0.0
164
+ },
165
+ {
166
+ "id": "delta_02",
167
+ "group": "delta",
168
+ "expected_ready": true,
169
+ "got_ready": true,
170
+ "expected_missing": [],
171
+ "got_missing": [],
172
+ "correct": true,
173
+ "aligned": true,
174
+ "gap": false,
175
+ "latency_ms": 0.0
176
+ },
177
+ {
178
+ "id": "delta_03",
179
+ "group": "delta",
180
+ "expected_ready": true,
181
+ "got_ready": true,
182
+ "expected_missing": [],
183
+ "got_missing": [],
184
+ "correct": true,
185
+ "aligned": true,
186
+ "gap": false,
187
+ "latency_ms": 0.0
188
+ },
189
+ {
190
+ "id": "delta_04",
191
+ "group": "delta",
192
+ "expected_ready": false,
193
+ "got_ready": false,
194
+ "expected_missing": [
195
+ "a new analysis since the last report"
196
+ ],
197
+ "got_missing": [
198
+ "a new analysis since the last report"
199
+ ],
200
+ "correct": true,
201
+ "aligned": true,
202
+ "gap": false,
203
+ "latency_ms": 0.0
204
+ },
205
+ {
206
+ "id": "delta_05",
207
+ "group": "delta",
208
+ "expected_ready": false,
209
+ "got_ready": false,
210
+ "expected_missing": [
211
+ "a new analysis since the last report"
212
+ ],
213
+ "got_missing": [
214
+ "a new analysis since the last report"
215
+ ],
216
+ "correct": true,
217
+ "aligned": true,
218
+ "gap": false,
219
+ "latency_ms": 0.0
220
+ },
221
+ {
222
+ "id": "edge_01",
223
+ "group": "edge",
224
+ "expected_ready": false,
225
+ "got_ready": false,
226
+ "expected_missing": [
227
+ "at least one completed analysis"
228
+ ],
229
+ "got_missing": [
230
+ "at least one completed analysis"
231
+ ],
232
+ "correct": true,
233
+ "aligned": false,
234
+ "gap": false,
235
+ "latency_ms": 0.0
236
+ },
237
+ {
238
+ "id": "align_01",
239
+ "group": "alignment",
240
+ "expected_ready": true,
241
+ "got_ready": true,
242
+ "expected_missing": [],
243
+ "got_missing": [],
244
+ "correct": true,
245
+ "aligned": false,
246
+ "gap": true,
247
+ "latency_ms": 0.0
248
+ },
249
+ {
250
+ "id": "align_02",
251
+ "group": "alignment",
252
+ "expected_ready": true,
253
+ "got_ready": true,
254
+ "expected_missing": [],
255
+ "got_missing": [],
256
+ "correct": true,
257
+ "aligned": false,
258
+ "gap": true,
259
+ "latency_ms": 0.0
260
+ },
261
+ {
262
+ "id": "align_03",
263
+ "group": "alignment",
264
+ "expected_ready": true,
265
+ "got_ready": true,
266
+ "expected_missing": [],
267
+ "got_missing": [],
268
+ "correct": true,
269
+ "aligned": true,
270
+ "gap": false,
271
+ "latency_ms": 0.0
272
+ }
273
+ ]
274
+ }
src/agents/chat_handler.py CHANGED
@@ -470,6 +470,11 @@ class ChatHandler:
470
  yield {"event": "error", "data": f"Document retrieval failed: {e}"}
471
  return
472
  elif intent == "check":
 
 
 
 
 
473
  try:
474
  # Scope check to the analysis catalog: it holds only this room's
475
  # bound sources and their real names (a DB shows as "xl test", not
@@ -516,6 +521,10 @@ class ChatHandler:
516
  # yield {"event": "done", "data": ""}
517
  # return
518
  elif intent == "help":
 
 
 
 
519
  try:
520
  state = analysis_state or await self._load_analysis_state(analysis_id)
521
  except Exception as e:
@@ -670,8 +679,21 @@ class ChatHandler:
670
  """
671
  if pad.message_id is None:
672
  return
 
 
 
 
 
 
 
 
 
 
 
 
 
673
  try:
674
- payload = pad.build(analysis_id or "", user_id, pad.message_id)
675
  await self._get_traceability_store().save(payload)
676
  except Exception as e: # noqa: BLE001 — never break the answer on a trace slip
677
  logger.warning(
@@ -726,6 +748,13 @@ class ChatHandler:
726
  if ac:
727
  run_kw["assembler_callbacks"] = ac
728
 
 
 
 
 
 
 
 
729
  # R4: bridge the coordinator's per-stage progress callback to SSE `status`
730
  # events so the stream isn't silent for ~12s (and proxies don't drop the
731
  # idle connection). Status events only appear if the coordinator calls back.
@@ -763,9 +792,6 @@ class ChatHandler:
763
  yield {"event": "error", "data": f"Analysis failed: {e}"}
764
  return
765
 
766
- # Sources live in traceability now (KM-691), derived from the run's
767
- # retrieve_data calls; the stream stays text-only.
768
- yield {"event": "sources", "data": json.dumps([])}
769
  yield {"event": "chunk", "data": result.chat_answer}
770
  try:
771
  # Stamp identity from the request scope: owner + the shared session id
@@ -790,13 +816,18 @@ class ChatHandler:
790
  # SPINE_V2_PLAN §4.4: chart rows are written before `done`; the FE fetches
791
  # GET /api/v1/charts unconditionally on every `done` (no polling race).
792
  try:
793
- if pad is not None and pad.message_id:
 
 
 
 
 
794
  for task_result in result.analysis_record.results_snapshot.values():
795
  for output in task_result.outputs:
796
  if output.kind == "chart" and isinstance(output.value, dict):
797
  await self._get_chart_store().save(
798
  message_id=pad.message_id,
799
- analysis_id=analysis_id or "",
800
  user_id=user_id,
801
  record_id=result.analysis_record.record_id,
802
  envelope=output.value,
 
470
  yield {"event": "error", "data": f"Document retrieval failed: {e}"}
471
  return
472
  elif intent == "check":
473
+ # The contract documents `sources` as always present and always first
474
+ # (it stays `[]` — real sources live in GET /traceability). This branch
475
+ # used to skip it, so an FE that initializes per-turn state on `sources`
476
+ # silently never initialized on a check turn. Purely additive. (F-19)
477
+ yield {"event": "sources", "data": json.dumps([])}
478
  try:
479
  # Scope check to the analysis catalog: it holds only this room's
480
  # bound sources and their real names (a DB shows as "xl test", not
 
521
  # yield {"event": "done", "data": ""}
522
  # return
523
  elif intent == "help":
524
+ # Same as the check branch: `sources` is contractually always-first.
525
+ # Note `stream_help` (the dedicated /tools/help endpoint) already emits
526
+ # it, so the two help paths disagreed with each other until now. (F-19)
527
+ yield {"event": "sources", "data": json.dumps([])}
528
  try:
529
  state = analysis_state or await self._load_analysis_state(analysis_id)
530
  except Exception as e:
 
679
  """
680
  if pad.message_id is None:
681
  return
682
+ if not analysis_id:
683
+ # `message_traceability.analysis_id` is `UUID NOT NULL` (Go migration
684
+ # 0007). Passing `analysis_id or ""` sent an empty string, which cannot
685
+ # cast to uuid — the INSERT failed, the never-throw seam swallowed it, and
686
+ # the row vanished with no user-visible symptom. Skipping the write for a
687
+ # turn that genuinely has no analysis is the honest outcome, and it says
688
+ # so in the log instead of failing invisibly. (F-24)
689
+ logger.info(
690
+ "traceability flush skipped — no analysis_id",
691
+ degraded_seam="traceability_flush_no_analysis",
692
+ message_id=pad.message_id,
693
+ )
694
+ return
695
  try:
696
+ payload = pad.build(analysis_id, user_id, pad.message_id)
697
  await self._get_traceability_store().save(payload)
698
  except Exception as e: # noqa: BLE001 — never break the answer on a trace slip
699
  logger.warning(
 
748
  if ac:
749
  run_kw["assembler_callbacks"] = ac
750
 
751
+ # Contract order is `sources` -> `status`* -> `chunk`* -> `done`. This used to
752
+ # be emitted after the status loop, so the slow path (the ONLY path that emits
753
+ # `status`) inverted the documented order and the FE saw `status` first on
754
+ # exactly the turns that take longest. Sources live in traceability now
755
+ # (KM-691); the event stays `[]` and the stream stays text-only. (F-19)
756
+ yield {"event": "sources", "data": json.dumps([])}
757
+
758
  # R4: bridge the coordinator's per-stage progress callback to SSE `status`
759
  # events so the stream isn't silent for ~12s (and proxies don't drop the
760
  # idle connection). Status events only appear if the coordinator calls back.
 
792
  yield {"event": "error", "data": f"Analysis failed: {e}"}
793
  return
794
 
 
 
 
795
  yield {"event": "chunk", "data": result.chat_answer}
796
  try:
797
  # Stamp identity from the request scope: owner + the shared session id
 
816
  # SPINE_V2_PLAN §4.4: chart rows are written before `done`; the FE fetches
817
  # GET /api/v1/charts unconditionally on every `done` (no polling race).
818
  try:
819
+ # `message_charts.analysis_id` is `uuid NOT NULL REFERENCES analyses(id)`
820
+ # (Go migration 0007), so an empty string both fails the cast and violates
821
+ # the FK. The write is never-throw, so the chart row disappeared silently
822
+ # and GET /api/v1/charts then answered `not_found` for a turn that really
823
+ # did produce a chart. Skip rather than lose it invisibly. (F-24)
824
+ if pad is not None and pad.message_id and analysis_id:
825
  for task_result in result.analysis_record.results_snapshot.values():
826
  for output in task_result.outputs:
827
  if output.kind == "chart" and isinstance(output.value, dict):
828
  await self._get_chart_store().save(
829
  message_id=pad.message_id,
830
+ analysis_id=analysis_id,
831
  user_id=user_id,
832
  record_id=result.analysis_record.record_id,
833
  envelope=output.value,
src/api/v1/help.py CHANGED
@@ -20,7 +20,7 @@ import json
20
  import uuid
21
 
22
  from fastapi import APIRouter, Depends, HTTPException
23
- from pydantic import BaseModel
24
  from sqlalchemy.ext.asyncio import AsyncSession
25
  from sse_starlette.sse import EventSourceResponse
26
 
@@ -40,6 +40,24 @@ class HelpRequest(BaseModel):
40
  user_id: str
41
  analysis_id: str
42
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
43
 
44
  @router.post("/help")
45
  @log_execution(logger)
 
20
  import uuid
21
 
22
  from fastapi import APIRouter, Depends, HTTPException
23
+ from pydantic import BaseModel, field_validator
24
  from sqlalchemy.ext.asyncio import AsyncSession
25
  from sse_starlette.sse import EventSourceResponse
26
 
 
40
  user_id: str
41
  analysis_id: str
42
 
43
+ @field_validator("analysis_id")
44
+ @classmethod
45
+ def _analysis_id_is_uuid(cls, v: str) -> str:
46
+ """Same boundary rule as `POST /api/v2/chat/stream`. (F-22)
47
+
48
+ Kept identical on purpose: a non-UUID id can never match a row in `analyses`
49
+ (`id uuid NOT NULL`), so the state read silently returns nothing and Help
50
+ answers from an empty state instead of saying the id was wrong. Two live
51
+ endpoints taking the same field should not disagree on what is valid.
52
+ """
53
+ try:
54
+ uuid.UUID(v)
55
+ except (ValueError, AttributeError, TypeError):
56
+ raise ValueError(
57
+ "analysis_id must be a UUID (the id of an existing analysis)"
58
+ ) from None
59
+ return v
60
+
61
 
62
  @router.post("/help")
63
  @log_execution(logger)
src/api/v2/chat.py CHANGED
@@ -24,7 +24,7 @@ import uuid
24
  from typing import Any
25
 
26
  from fastapi import APIRouter, Depends, HTTPException, Request
27
- from pydantic import BaseModel
28
  from sqlalchemy.ext.asyncio import AsyncSession
29
  from sse_starlette.sse import EventSourceResponse
30
 
@@ -82,6 +82,31 @@ class ChatRequest(BaseModel):
82
  analysis_id: str
83
  message: str
84
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
85
 
86
  @router.post("/chat/stream")
87
  # Rate limit per client IP. `slowapi` needs a Starlette `Request` param named
 
24
  from typing import Any
25
 
26
  from fastapi import APIRouter, Depends, HTTPException, Request
27
+ from pydantic import BaseModel, field_validator
28
  from sqlalchemy.ext.asyncio import AsyncSession
29
  from sse_starlette.sse import EventSourceResponse
30
 
 
82
  analysis_id: str
83
  message: str
84
 
85
+ @field_validator("analysis_id")
86
+ @classmethod
87
+ def _analysis_id_is_uuid(cls, v: str) -> str:
88
+ """422 on a non-UUID `analysis_id` instead of degrading silently. (F-22)
89
+
90
+ `analyses.id` is `uuid NOT NULL`, so a non-UUID value can never match a real
91
+ analysis. Without this the turn still ran: `state_store.ensure` failed its
92
+ INSERT, the caller logged a warning and continued with `analysis_state = None`,
93
+ and help, readiness and the report write-back all silently no-op'd for that
94
+ turn. The user saw a normal-looking answer whose analysis state was never
95
+ persisted. A clear client error is the honest outcome.
96
+
97
+ Also closes the F-24 half of the same problem: `message_traceability` and
98
+ `message_charts` declare `analysis_id UUID NOT NULL` (Go migration 0007, with
99
+ an FK to `analyses(id)` on the charts table), and both writes are never-throw
100
+ — so a non-UUID id silently lost the provenance and chart rows too.
101
+ """
102
+ try:
103
+ uuid.UUID(v)
104
+ except (ValueError, AttributeError, TypeError):
105
+ raise ValueError(
106
+ "analysis_id must be a UUID (the id of an existing analysis)"
107
+ ) from None
108
+ return v
109
+
110
 
111
  @router.post("/chat/stream")
112
  # Rate limit per client IP. `slowapi` needs a Starlette `Request` param named
src/query/executor/db.py CHANGED
@@ -32,12 +32,16 @@ from ...database_client.database_client_service import database_client_service
32
  from ...database_client.engine import user_engine_cache
33
  from ...db.postgres.connection import AsyncSessionLocal
34
  from ...middlewares.logging import get_logger
35
- from ...pipeline.db_pipeline import db_pipeline_service
36
  from ...utils.db_credential_encryption import decrypt_credentials_dict
37
  from ..compiler.sql import CompiledSql, SqlCompiler
38
  from ..ir.models import QueryIR
39
  from .base import BaseExecutor, QueryResult
40
 
 
 
 
 
 
41
  logger = get_logger("db_executor")
42
 
43
  _QUERY_TIMEOUT_SECONDS = 30
@@ -131,7 +135,13 @@ class DbExecutor(BaseExecutor):
131
  source_id=ir.source_id,
132
  backend="sql",
133
  elapsed_ms=elapsed_ms,
134
- error=str(e),
 
 
 
 
 
 
135
  table_id=ir.table_id,
136
  table_name=table_name,
137
  source_name=source_name,
@@ -207,12 +217,36 @@ class DbExecutor(BaseExecutor):
207
  result = conn.execute(text(compiled.sql), compiled.params)
208
  return list(result.keys()), [dict(row) for row in result.mappings()]
209
 
210
- # Legacy per-call path for non-postgres db_types (connect once, dispose).
211
- # These never set read-only/timeout before, so behavior is unchanged.
212
- with db_pipeline_service.engine_scope(db_type, creds) as eng:
213
- with eng.connect() as conn:
214
- result = conn.execute(text(compiled.sql), compiled.params)
215
- return list(result.keys()), [dict(row) for row in result.mappings()]
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
216
 
217
  # ------------------------------------------------------------------
218
  # Speculative pre-connect (DB3)
 
32
  from ...database_client.engine import user_engine_cache
33
  from ...db.postgres.connection import AsyncSessionLocal
34
  from ...middlewares.logging import get_logger
 
35
  from ...utils.db_credential_encryption import decrypt_credentials_dict
36
  from ..compiler.sql import CompiledSql, SqlCompiler
37
  from ..ir.models import QueryIR
38
  from .base import BaseExecutor, QueryResult
39
 
40
+ # Orphaned by the F-4 tripwire (2026-07-23): the only consumer was the legacy
41
+ # non-postgres branch in `_run_sync`, now commented out there. Restore this import
42
+ # alongside that branch (kept out of the block above so import sorting stays clean).
43
+ # from ...pipeline.db_pipeline import db_pipeline_service
44
+
45
  logger = get_logger("db_executor")
46
 
47
  _QUERY_TIMEOUT_SECONDS = 30
 
135
  source_id=ir.source_id,
136
  backend="sql",
137
  elapsed_ms=elapsed_ms,
138
+ # `str(e) or repr(e)`, not bare repr: the log above already uses repr,
139
+ # but this payload reaches the assembler prompt, the traceability
140
+ # record and the report caveats — where a Fernet InvalidToken arrived
141
+ # as an EMPTY string, so the user-facing artifact said nothing while
142
+ # the log was diagnosable. Falling back only when str() is empty means
143
+ # no existing error text changes. (F-26)
144
+ error=str(e) or repr(e),
145
  table_id=ir.table_id,
146
  table_name=table_name,
147
  source_name=source_name,
 
217
  result = conn.execute(text(compiled.sql), compiled.params)
218
  return list(result.keys()), [dict(row) for row in result.mappings()]
219
 
220
+ # TRIPWIRE (F-4, 2026-07-23). Below this line was the legacy per-call path for
221
+ # non-postgres db_types, whose own comment conceded "these never set
222
+ # read-only/timeout before, so behavior is unchanged". That means such a source
223
+ # would get only four of the five documented defense layers: IR validation, the
224
+ # compiler whitelist, the sqlglot guard and LIMIT — but NO read-only session and
225
+ # NO statement_timeout. `CLAUDE.md` §2.5 states all five unconditionally; in
226
+ # truth they were conditional on db_type, and nothing said so where a reader
227
+ # would look.
228
+ #
229
+ # Zero blast radius today: Go's `database_clients.Service.Create` gates on
230
+ # `isSupportedActive`, and only `postgres` is `active` — mysql/sqlserver/
231
+ # bigquery/snowflake are all "Coming soon", so no such source can be registered.
232
+ # This refuses loudly the day someone flips that flag, instead of silently
233
+ # executing against a customer's database with two guardrails missing.
234
+ #
235
+ # Note the compiler is built with dialect="postgres" regardless of db_type and
236
+ # the sqlglot guard parses with read="postgres", so these queries would fail on
237
+ # a parse error anyway — which the never-throw path would degrade into "data not
238
+ # available", masquerading as a data problem. The danger is someone fixing THAT
239
+ # without noticing the pooling branch. Re-enabling this path requires
240
+ # dialect-correct compilation AND session hardening, not just a dialect string.
241
+ raise ValueError(
242
+ f"source type {db_type!r} is not supported for analysis yet — only "
243
+ "PostgreSQL sources can be queried safely (read-only session and query "
244
+ "timeout are not yet implemented for other database types)"
245
+ )
246
+ # with db_pipeline_service.engine_scope(db_type, creds) as eng:
247
+ # with eng.connect() as conn:
248
+ # result = conn.execute(text(compiled.sql), compiled.params)
249
+ # return list(result.keys()), [dict(row) for row in result.mappings()]
250
 
251
  # ------------------------------------------------------------------
252
  # Speculative pre-connect (DB3)
src/query/executor/tabular.py CHANGED
@@ -204,15 +204,18 @@ class TabularExecutor(BaseExecutor):
204
  elapsed_ms = int((time.perf_counter() - started) * 1000)
205
  logger.error(
206
  "tabular executor failed",
 
207
  source_id=ir.source_id,
208
- error=str(e),
209
  elapsed_ms=elapsed_ms,
210
  )
211
  return QueryResult(
212
  source_id=ir.source_id,
213
  backend="tabular",
214
  elapsed_ms=elapsed_ms,
215
- error=str(e),
 
 
216
  table_id=ir.table_id,
217
  table_name=table_name,
218
  source_name=source_name,
 
204
  elapsed_ms = int((time.perf_counter() - started) * 1000)
205
  logger.error(
206
  "tabular executor failed",
207
+ degraded_seam="tabular_execute",
208
  source_id=ir.source_id,
209
+ error=repr(e),
210
  elapsed_ms=elapsed_ms,
211
  )
212
  return QueryResult(
213
  source_id=ir.source_id,
214
  backend="tabular",
215
  elapsed_ms=elapsed_ms,
216
+ # See db.py: fall back to repr only when str() is empty, so no
217
+ # existing user-facing error text changes. (F-26)
218
+ error=str(e) or repr(e),
219
  table_id=ir.table_id,
220
  table_name=table_name,
221
  source_name=source_name,
src/retrieval/router.py CHANGED
@@ -13,6 +13,7 @@ import hashlib
13
  import json
14
  from dataclasses import asdict
15
 
 
16
  from src.db.redis.connection import get_redis
17
  from src.middlewares.logging import get_logger
18
  from src.retrieval.base import RetrievalResult
@@ -21,7 +22,13 @@ from src.retrieval.document import DocumentRetriever
21
  logger = get_logger("retrieval_router")
22
 
23
  _CACHE_TTL = 3600
24
- _CACHE_KEY_PREFIX = "retrieval"
 
 
 
 
 
 
25
 
26
 
27
  class RetrievalRouter:
 
13
  import json
14
  from dataclasses import asdict
15
 
16
+ from src.config.settings import settings
17
  from src.db.redis.connection import get_redis
18
  from src.middlewares.logging import get_logger
19
  from src.retrieval.base import RetrievalResult
 
22
  logger = get_logger("retrieval_router")
23
 
24
  _CACHE_TTL = 3600
25
+ # Namespaced with `settings.redis_prefix`, like every other cache key in the service
26
+ # (cf. `api/v1/chat.py::build_cache_key`). Without it, two environments pointed at the
27
+ # same Redis — which the single shared `.env` makes entirely plausible — cross-serve
28
+ # each other's retrieval results. Not cross-TENANT (`user_id` is in the key), but
29
+ # cross-environment, which is confusing in exactly the way stale-cache bugs are. The
30
+ # one-time cost is a cache-miss storm bounded by the 1h TTL. (F-16)
31
+ _CACHE_KEY_PREFIX = f"{settings.redis_prefix}retrieval"
32
 
33
 
34
  class RetrievalRouter: