[NOTICKET] fix: write reports.user_id (Neon NOT NULL column broke report persistence)

The new Neon dedorch instance carries `reports.user_id text NOT NULL`, a column
`AnalysisReportRow` did not have, so `ReportStore.save` emitted an INSERT without
it and every `POST /api/v1/tools/report` 500'd with NotNullViolationError.

- models.py: add `user_id = Column(Text)` — nullable on purpose, so Python matches
the loosest deployment shape rather than pinning to one DB's constraint.
- store.py: write `report.user_id`, already required by the endpoint
(`user_id: str = Query(...)`) and threaded through `generator.generate`.

Verified: `information_schema` confirms the live column; Rifqi live-tested a 201.
Suite 381 passed, 2 failed (both pre-existing), 7 skipped — baseline unchanged.

Docs: REPO_STATUS §8/§9/§13 (+ header delta) and DEV_PLAN #30/#31. The §13 gotcha
generalizes it — Go-owned column additions are invisible on Python's read paths and
only surface on its five write paths.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

DEV_PLAN.md

@@ -223,6 +223,8 @@ Status legend: ⬜ not started · 🔄 in progress · ✅ done · ⛔ blocked ·
 | 27 | **Images (DEFERRED):** image table (id, analysis_id, msg/report ref, order) + originals in a bucket | — | ⏸️ | Maintenance-heavy; parked |
 | 28 | **UI research** (FE): new-analysis form, knowledge menu (user vs analysis level), report artifacts + version selector | Team | ⬜ new | No dedicated UI person; interview + old analysis UI removed |
 | 29 | **LLM env quad rename `__4o` → `__54m`** + set the four `azureai__*__54m` secrets on the HF Space | Rifqi | 🔄 | Code done 2026-07-14 (pr/17): settings quad + 9 call sites + 2 eval runners renamed; deployment is `gpt-5.4-mini`. Suite **381 passed, 2 failed (both pre-existing), 7 skipped** — same as baseline; ruff clean on touched files; `import main` 0. **Hard rename — no `__4o` fallback**, so HF fails loudly on the first LLM call until the four `__54m` secrets are set. Root cause it fixes: HF silently ran GPT-4o while local ran 5.4-mini → same question, same catalog, planner hallucinated catalog ids on HF only. **Action: set HF secrets, redeploy, re-run the failing question.** Local `.env` still carries the now-dead `__4o` keys — safe to delete |
+| 30 | **Neon `reports.user_id` NOT NULL** — reconcile the ORM + `ReportStore` write | Rifqi | ✅ | Done 2026-07-22 (pr/18). The new Neon dedorch instance has `reports.user_id text NOT NULL` (verified via `information_schema`; **not** present in Go migrations `0001–0004`), a column `AnalysisReportRow` lacked → `ReportStore.save` INSERTed without it → **every `POST /api/v1/tools/report` 500'd** with `NotNullViolationError`. Fix: `models.py` adds `user_id = Column(Text)` (**nullable** on purpose — Python matches the loosest deployment shape, §7D getattr-tolerance) + `store.py` writes `report.user_id`, which the endpoint already required (`user_id: str = Query(...)`) and threaded via `generator.generate`. Live-tested by Rifqi → **201**. `import main` 0; ruff on touched files unchanged (3 pre-existing E501 in `models.py`, same count on HEAD). **Follow-up → #31** |
+| 31 | **Schema-change notification from Go/Harry** — Python must learn about new columns on tables it *writes* | Rifqi → Harry | ⬜ new | Raised by #30: the `reports.user_id` addition reached the Neon DB with no Python-side heads-up, and read-path getattr-tolerance hides such drift until an INSERT fails. Python writes only `reports`, `report_inputs`, `message_traceability`, `message_charts`, `analyses` (via `state_store.ensure`) — a column added to any of those breaks Python silently until runtime. **Ask:** Harry announces DDL touching those five, or the migration set becomes the single verifiable source (`0001–0004` did not contain this column). **Action: Rifqi raises with Harry** |
 
 ## 5. Critical path & sequencing
 

REPO_STATUS.md

@@ -2,7 +2,7 @@
 
 **Audience:** teammates onboarding onto the Python repo (`Agentic-Service-Data-Eyond-Catalog`).
 **Scope:** what the code does **right now** (branch `pr/4`, ticket KM-652). Describes current state only — no roadmap or to-dos.
-**Snapshot date:** 2026-06-25. **Data-layer reconcile 2026-07-01:** §8/§12 updated — dedorch cutover done, `data_catalog` model reconciled. **Query-path fix 2026-07-02:** §8/§13 — dedorch catalogs ship no FKs → Python infers them (`fk_inference.py`); shared-Fernet-key gotcha documented. **Agent-quality fixes 2026-07-08 (pr/13):** from the scoped live-test review — the planner gains an explicit **infeasible** outcome (`TaskList.infeasible_reason` → deterministic EN/ID data-gap reply via `refusals.data_gap_message`; no more force-mapping absent measures like `pa` AS "revenue"), the IR validator rejects bare selects under `group_by` (self-corrects via the planner retry), `analyze_trend` handles integer year/month columns (was collapsing every row into one 1970-01 bucket), planner few-shots add top-N (Example G) + infeasible (Example H), numeric catalog `sample_values` are base64-decoded at read (`catalog/sample_decode.py` — stopgap for Go's byte-marshaling; primary fix is Go-side), traceability no longer emits null source rows for failed retrievals, and `check_data` hides `-1` row counts. **Report v2 + analyze_merge planner support 2026-07-09 (pr/13):** Sofia's `analyze_merge` tool (8abf635, KM-703) is now planner-supported (`_validate_data_source` guards `data_right`, two-retrieve→merge few-shot Example I, planner.md "Two measures per entity" rule); the report gains per-business-question answers (`bq_answers` — drafted by the SAME single LLM call, index-based record refs, deterministic fallback unchanged), "Attempted, Unresolved" + "Excluded Analyses" sections (failed runs are no longer silently dropped), evidence tables copied from `results_snapshot` (table-kind outputs, ≤3/record ≤10 rows ≤8 cols, `check_*` skipped), normalized caveat dedupe with caps (12/10), and single-language output via `detect_reply_language`; the report surface adds `GET /tools/report/{analysis_id}/records` (curation list), `GET …/readiness` (FE delta guard), and `exclude_record_ids` on POST — see API_CONTRACT_BE_PYTHON.md. **Report compaction 2026-07-09 (pr/13):** the rendered markdown drops the "Notes & Limitations", "Attempted, Unresolved", and "How This Was Analyzed" sections (team decision — compact report; render blocks commented out in `report/generator.py`, not deleted). The JSON body keeps `caveats`/`open_questions`/`unresolved`/`method_steps` and the curation/records endpoints are unchanged. **Traceability `data_used` layer 2026-07-13 (pr/15):** `GET /api/v1/traceability` gains a resolved, user-facing `data_used[]` block (one per `retrieve_data` call) — real source/table/column names, joins, plain-language filters, and result columns split into read-from-data vs `computed` (with `formula`, e.g. `total_revenue` = `SUM(line_total)`, so an alias is never shown as a real column). Ids are kept but **machine-only (FE must not render)**. Also adds `tool_calls[].summary`, and `sources[]` now carry `source_name` + every table touched. Deterministic catalog resolution (new `src/traceability/resolve.py`), no LLM, never-throw; catalog threaded to the scratchpad at the slow-path composition root. Additive/non-breaking; contract + `TRACEABILITY_FE_HANDOFF.md` updated. **Spine v2 W2+W1 2026-07-13 (pr/16):** §6/§7/§8/§9/§12 — `render_chart` tool lands (first of the `render_*` family: deterministic Plotly-JSON `dataeyond.chart.v1` envelope, hand-built, **no plotly dependency**; planner-selected only on an explicit chart ask, EN/ID) + Python-owned `message_charts` store + `GET /api/v1/charts` (FE fetches on `done`, same pattern as traceability; empty list = valid 200); planner gains a named **recipe table** + viz few-shots (Example J tail, Example K viz-infeasible) + validator Check 10 (`render_chart.data` must reference a table-producing task); and the slow path gains the **S1a quality checkpoint** (`slow_path/checkpoint.py`, 0 LLM, never-throw, between runner and assembler: CK1 all-failed → deterministic honest-failure answer with **no** assembler call, CK2 empty retrieve + downstream, CK3 10k-cap truncation, CK4 single trend bucket, CK5 all-null column, CK6 chart-spec sanity; flags render as an "Execution assessment" block in the assembler input and every flag logs `repair_candidate` — the S1b evidence base). Design + handoff doc: `SPINE_V2_PLAN.md`. **LLM model + env rename 2026-07-14 (pr/17):** §2/§3/§9/§13 — the generation LLM is now **Azure GPT-5.4-mini** (deployment `gpt-5.4-mini`), not GPT-4o, and the settings quad is renamed `azureai__*__4o` → **`azureai__*__54m`** across all 9 LLM call sites. **Hard rename, no `__4o` fallback** — an environment that still sets only `__4o` resolves to empty strings and fails on the first LLM call, by design: the silent-wrong-model drift is exactly how HF stayed on GPT-4o while local ran 5.4-mini (identical question, identical catalog, divergent planner output — HF hallucinated catalog ids, local planned correctly). Deploying requires all four `__54m` vars set in the HF Space secrets. **Cross-repo update 2026-06-29:** §2/§8/§11/§12 re-verified against
+**Snapshot date:** 2026-06-25. **Data-layer reconcile 2026-07-01:** §8/§12 updated — dedorch cutover done, `data_catalog` model reconciled. **Query-path fix 2026-07-02:** §8/§13 — dedorch catalogs ship no FKs → Python infers them (`fk_inference.py`); shared-Fernet-key gotcha documented. **Agent-quality fixes 2026-07-08 (pr/13):** from the scoped live-test review — the planner gains an explicit **infeasible** outcome (`TaskList.infeasible_reason` → deterministic EN/ID data-gap reply via `refusals.data_gap_message`; no more force-mapping absent measures like `pa` AS "revenue"), the IR validator rejects bare selects under `group_by` (self-corrects via the planner retry), `analyze_trend` handles integer year/month columns (was collapsing every row into one 1970-01 bucket), planner few-shots add top-N (Example G) + infeasible (Example H), numeric catalog `sample_values` are base64-decoded at read (`catalog/sample_decode.py` — stopgap for Go's byte-marshaling; primary fix is Go-side), traceability no longer emits null source rows for failed retrievals, and `check_data` hides `-1` row counts. **Report v2 + analyze_merge planner support 2026-07-09 (pr/13):** Sofia's `analyze_merge` tool (8abf635, KM-703) is now planner-supported (`_validate_data_source` guards `data_right`, two-retrieve→merge few-shot Example I, planner.md "Two measures per entity" rule); the report gains per-business-question answers (`bq_answers` — drafted by the SAME single LLM call, index-based record refs, deterministic fallback unchanged), "Attempted, Unresolved" + "Excluded Analyses" sections (failed runs are no longer silently dropped), evidence tables copied from `results_snapshot` (table-kind outputs, ≤3/record ≤10 rows ≤8 cols, `check_*` skipped), normalized caveat dedupe with caps (12/10), and single-language output via `detect_reply_language`; the report surface adds `GET /tools/report/{analysis_id}/records` (curation list), `GET …/readiness` (FE delta guard), and `exclude_record_ids` on POST — see API_CONTRACT_BE_PYTHON.md. **Report compaction 2026-07-09 (pr/13):** the rendered markdown drops the "Notes & Limitations", "Attempted, Unresolved", and "How This Was Analyzed" sections (team decision — compact report; render blocks commented out in `report/generator.py`, not deleted). The JSON body keeps `caveats`/`open_questions`/`unresolved`/`method_steps` and the curation/records endpoints are unchanged. **Traceability `data_used` layer 2026-07-13 (pr/15):** `GET /api/v1/traceability` gains a resolved, user-facing `data_used[]` block (one per `retrieve_data` call) — real source/table/column names, joins, plain-language filters, and result columns split into read-from-data vs `computed` (with `formula`, e.g. `total_revenue` = `SUM(line_total)`, so an alias is never shown as a real column). Ids are kept but **machine-only (FE must not render)**. Also adds `tool_calls[].summary`, and `sources[]` now carry `source_name` + every table touched. Deterministic catalog resolution (new `src/traceability/resolve.py`), no LLM, never-throw; catalog threaded to the scratchpad at the slow-path composition root. Additive/non-breaking; contract + `TRACEABILITY_FE_HANDOFF.md` updated. **Spine v2 W2+W1 2026-07-13 (pr/16):** §6/§7/§8/§9/§12 — `render_chart` tool lands (first of the `render_*` family: deterministic Plotly-JSON `dataeyond.chart.v1` envelope, hand-built, **no plotly dependency**; planner-selected only on an explicit chart ask, EN/ID) + Python-owned `message_charts` store + `GET /api/v1/charts` (FE fetches on `done`, same pattern as traceability; empty list = valid 200); planner gains a named **recipe table** + viz few-shots (Example J tail, Example K viz-infeasible) + validator Check 10 (`render_chart.data` must reference a table-producing task); and the slow path gains the **S1a quality checkpoint** (`slow_path/checkpoint.py`, 0 LLM, never-throw, between runner and assembler: CK1 all-failed → deterministic honest-failure answer with **no** assembler call, CK2 empty retrieve + downstream, CK3 10k-cap truncation, CK4 single trend bucket, CK5 all-null column, CK6 chart-spec sanity; flags render as an "Execution assessment" block in the assembler input and every flag logs `repair_candidate` — the S1b evidence base). Design + handoff doc: `SPINE_V2_PLAN.md`. **LLM model + env rename 2026-07-14 (pr/17):** §2/§3/§9/§13 — the generation LLM is now **Azure GPT-5.4-mini** (deployment `gpt-5.4-mini`), not GPT-4o, and the settings quad is renamed `azureai__*__4o` → **`azureai__*__54m`** across all 9 LLM call sites. **Hard rename, no `__4o` fallback** — an environment that still sets only `__4o` resolves to empty strings and fails on the first LLM call, by design: the silent-wrong-model drift is exactly how HF stayed on GPT-4o while local ran 5.4-mini (identical question, identical catalog, divergent planner output — HF hallucinated catalog ids, local planned correctly). Deploying requires all four `__54m` vars set in the HF Space secrets. **`reports.user_id` fix 2026-07-22 (pr/18):** §8/§9/§13 — the new **Neon** dedorch instance carries `reports.user_id text NOT NULL`, a column Python's `AnalysisReportRow` did not have, so `ReportStore.save` emitted an INSERT without it and **every `POST /api/v1/tools/report` 500'd** (`NotNullViolationError`; report persistence deliberately re-raises rather than degrading). The ORM gains `user_id` (declared **nullable** so Python tolerates both the pre- and post-migration shape) and the store writes `report.user_id`, which the endpoint already required as a query param and threaded through the generator. Live-verified against `information_schema` + a green 201. Generalized as a §13 gotcha: Go-owned column additions are invisible on read paths and only surface on Python's few write paths. **Cross-repo update 2026-06-29:** §2/§8/§11/§12 re-verified against
 the **Go source** (`Orchestrator-Agent-Service`), not its docs. The Go service has moved well past its
 own (uncommitted, stale) design docs: it now hosts the **dedorch SQL migrations** in-repo and a full
 **`/api/v1/analyses` + `/api/v1/skills`** REST surface. Go does **not** call Python yet — those skills
@@ -187,7 +187,7 @@ unless `SKIP_INIT_DB=true`.
 | `langchain_pg_embedding` | PGVector document chunks | Go ingestion | DocumentRetriever |
 | `report_inputs` *(was `analysis_records`)* | jsonb `AnalysisRecord`, one per slow-path run; **Python-owned** | slow path | ReportGenerator, report readiness |
 | `analyses` *(dedorch, plural)* | uuid `id`, `user_id`, `analysis_title`, `objective`, `business_questions` jsonb, `status` (active\|inactive), `data_bind`(+`data_bind_version`), `report_id`, `report_collection` — **defined by Go migrations**; `problem_statement`/`problem_validated`/`owner_id` already **dropped** there (`0003`/`0004`) | Go `/api/v1/analyses`; Python state store | gate (no-op), Help, report |
-| `reports` *(dedorch)* | uuid, `analysis_id`, `user_id`, `title` + markdown `content` + `version` (UNIQUE per analysis) | Go + Python ReportStore | report API |
+| `reports` *(dedorch)* | uuid, `analysis_id`, **`user_id` text NOT NULL**, `title` + markdown `content` + `version` (UNIQUE per analysis). **2026-07-22:** `AnalysisReportRow` gained `user_id` and `ReportStore.save` now writes it — before that the INSERT omitted the column and every `POST /tools/report` 500'd on the new Neon instance (§13) | Go + Python ReportStore | report API |
 | `data_sources` *(dedorch, Go-owned)* | per-analysis binding table. **Python no longer reads or writes it** — bindings live in Go's `analyses.data_bind`, which Go materializes into the analysis-scope `data_catalog` row; Python scopes off that row. The table exists (Go migration) but Python is fully decoupled — do **not** drop it manually | Go migration | — (unused by Python) |
 | `analyses_messages` *(dedorch)* | the analysis chat room (`role ∈ user\|ai`); replaces deprecated `rooms`/`chat_messages` | Go `/analyses/{id}/messages` | Python chat path **not yet migrated here** (§12) |
 | `message_traceability` *(Python-owned)* | one jsonb `TraceabilityPayload` per assistant turn (PK `message_id`); flushed before `done` | chat pipeline (KM-691) | `GET /api/v1/traceability` |
@@ -298,7 +298,9 @@ validator's Check 10 forces its `data` to reference a table-producing task.
 ### Report — `src/agents/report/`
 `generator.py` reads records, deterministically assembles structured fields, 1 LLM call for the
 executive summary; `store.py` versions under an advisory lock and persists markdown to dedorch
-`reports`; `readiness.py` defines the **report floor** (≥1 successful `analyze_*` **or**, since
+`reports` (**2026-07-22:** the row now carries `user_id`, threaded endpoint → generator → store —
+`POST /tools/report` takes it as a required query param, so it is never null on the live path);
+`readiness.py` defines the **report floor** (≥1 successful `analyze_*` **or**, since
 2026-07-14, `render_chart` — a chart-only session is substantive; the `problem_validated`
 precondition was dropped 2026-06-25) shared by the report API and the Help readiness signal so the
 two can't disagree. **2026-07-14:** the report embeds charts — `_collect_charts` copies
@@ -413,6 +415,18 @@ records-based report; floor: ≥1 `analyze_*` success). Wiring Go → Python is
   degrades it into an honest "data not available" answer, so it masquerades as a data problem.
   Tell-apart: `BlobNotFound` + `location_ref` starting `object_storage://` ⇒ env gap; set
   `storage_provider=supabase_s3` + the five `supabase_s3_*` values (match Go's data plane).
+- **Go-owned columns drift silently until an INSERT hits them (gotcha, found 2026-07-22).** Python
+  reads Go-owned tables getattr-tolerantly, so a *new* column on a table Python only reads is
+  invisible. But `reports` is one of the few tables Python **writes** — when the new Neon instance
+  brought a `reports.user_id text NOT NULL` the ORM did not have, the INSERT omitted the column and
+  every `POST /api/v1/tools/report` 500'd with `Report persistence failed: … NotNullViolationError`
+  (report persistence is **not** a never-throw seam — `api/v1/report.py` re-raises as a 500, by
+  design, so a lost report is loud). Tell-apart: `NotNullViolationError` naming a column absent from
+  `src/db/postgres/models.py`. Fix pattern: add the column to the ORM as **nullable** (Python
+  matches the loosest deployment shape) and always write it — never make Python's model NOT NULL to
+  match one DB. Verify with
+  `select column_name, is_nullable from information_schema.columns where table_name='<t>'` against
+  the live DB, not against the Go migrations (this column was not in `0001–0004`).
 - **Never-throw seams** are pervasive (tool invoker, query service, executors, state/catalog reads,
   record persistence, report summary). Failures degrade into soft output rather than raising — good
   for UX, but they can mask real breakage (e.g. a missing analysis-scope catalog silently falling

src/agents/report/store.py

@@ -80,6 +80,7 @@ class ReportStore:
                     AnalysisReportRow(
                         id=report.report_id,
                         analysis_id=report.analysis_id,
+                        user_id=report.user_id,
                         title=_report_title(report),
                         content=report.rendered_markdown or "",
                         generated_at=report.generated_at,

src/db/postgres/models.py

@@ -196,6 +196,10 @@ class AnalysisReportRow(Base):
 
     id = Column(UUID(as_uuid=False), primary_key=True)  # AnalysisReport.report_id (uuid)
     analysis_id = Column(UUID(as_uuid=False), nullable=False, index=True)
+    # Added by Go 2026-07 (text, like `analyses.user_id`); NOT NULL in some deployments,
+    # nullable in others — declared nullable here so Python matches the loosest shape,
+    # but ReportStore always writes it (a missing value 500s on the strict DBs).
+    user_id = Column(Text)
     title = Column(String, nullable=False)
     content = Column(Text, nullable=False)  # rendered markdown
     generated_at = Column(DateTime(timezone=True), nullable=False, server_default=func.now())