DEV_PLAN.md

# Data Eyond — Current Development Plan (post 2026-06-24 → 2026-06-30 checkpoints)

**Purpose:** context file for Claude Code sessions working on the current sprint.
**Branch:** `pr/5` · **Snapshot:** 2026-06-30.
**Companion:** [REPO_STATUS.md](REPO_STATUS.md) describes the repo's *current built state*; this file
describes the *in-flight plan* that changes it. The **active sprint is pr/5** ([§0](#0-current-sprint--pr5-observability--endpoint-restructure)); sections §1–§6 are the prior
2026-06-24/25 pivot (now largely ✅), kept for context.

---

## 0. Current sprint — pr/5: Observability + Endpoint Restructure

From the **2026-06-30 checkpoint**. Direction: **Python → generation/AI-only**; Go owns the analysis
lifecycle + data plane. Endpoint contract sent to Harry on 2026-06-30:
[API_ENDPOINTS_RESTRUCTURE.md](API_ENDPOINTS_RESTRUCTURE.md) (chat→v2, tools regroup, observability —
observability marked tentative). REPO_STATUS carries a matching `pr/5` direction banner.

Mentor's task order: **unwire → regroup endpoints → add tools (retrieve-data + observability)**; share
the endpoint contract *before* coding the tools. Status legend: ⬜ not started · 🔄 in progress · ✅ done ·
⛔ blocked · 🔎 verify · ⏸️ deferred.

| Phase | Task | Owner | Status | Notes |
|---|---|---|---|---|
| **P0 — contract** | Draft + send endpoint contract to Harry (chat v2 · tools group · observability) | Rifqi + Sofhia | ✅ | `API_ENDPOINTS_RESTRUCTURE.md` sent 2026-06-30 (before-noon deadline met). Observability section flagged tentative. |
| **1 — unwire** | Unwire `users`(login)/`document`/`room`/`db_client`/`data_catalog`/`analysis` from `main` + Swagger | Sofhia | ✅ | **KM-686**, commit `0b2d678`. Commented, not deleted; `chat`/`report`/`tools` kept mounted. Resolves the analysis-CRUD scope Q — whole `analysis` router unwired (Go owns it). |
| **2 — v2 + regroup** | Create `src/api/v2/` and move the chat pilot there | Rifqi | ✅ | New `src/api/v2/__init__.py` + `src/api/v2/chat.py` (`POST /api/v2/chat/stream`), mounted in `main.py`. Only chat in v2; v1 `/chat/stream` kept mounted until FE moves over. Routes import-verified. |
| **2 — v2 + regroup** | Chat: `room_id` → **`analysis_id`** (request field + handler + history) | Rifqi | ✅ | v2 `ChatRequest{user_id, analysis_id, message}`; reuses warm `ChatHandler` + v1 cache/history helpers; `done` returns `{message_id}` (always minted Python-side, server-authoritative — open-Q #1 resolved in pr/6). Persistence kept transitionally → still ties to #25 (`analyses_messages`); ruff-clean. |
| **2 — v2 + regroup** | Move report under tools → `/api/v1/tools/report` (+ version routes) | Rifqi | ✅ | report router re-prefixed `/api/v1` → `/api/v1/tools` (all 3 routes move together), tag → `Tools`; old `/api/v1/report` gone. Same functionality, new home. Import-verified. |
| **2 — v2 + regroup** | Move help under tools → `POST /api/v1/tools/help` (dedicated endpoint) | Sofhia | ✅ | New `src/api/v1/help.py` (SSE: `sources:[]`→`chunk`→`done{message_id}`) + additive `ChatHandler.stream_help()` (reuses HelpAgent+state+readiness, no router). Generative-only (no persist). **Router `help` intent KEPT** — both paths live by design. message_id always minted Python-side, server-authoritative (open-Q #1 resolved, pr/6). Import-verified. |
| **2 — v2 + regroup** | Tools list → `/api/v1/tools/list` | Sofhia | ✅ | Renamed route `GET /api/v1/tools` → `GET /api/v1/tools/list` ([tools.py:133](src/api/v1/tools.py:133)). |
| **2 — v2 + regroup** | FE: slash menu = `/help` only; report = right-side button | Mentor (FE) | ⬜ | Coordination note, not Python work. |
| **3 — tools + obs** | Finish `help` so it actually **calls** (not just lists) + test | Sofhia | ⬜ | Mentor: help currently only lists tools. Core #2 after chat. |
| **3 — tools + obs** | Traceability **scratchpad** accumulating in the chat agent | Rifqi + Sofhia | ✅ | **KM-691.** `TraceabilityScratchpad` + `TraceabilityToolInvoker` (`src/traceability/`) capture planning / tool I/O / sources during the run; flushed one row before every `done` (all 8 sites; error turns = no row). Renamed observability→traceability (vs. Langfuse). |
| **3 — tools + obs** | Audit `report_inputs` — covers planning + tool I/O + source? add cols / new store | Rifqi | ✅ | **KM-691.** Chose a dedicated store: `message_traceability` = 1 JSONB row per message (Python-owned, like `report_inputs`; DDL run manually against dedorch, handed to Harry). Langfuse kept for engineering. |
| **3 — tools + obs** | Build `GET /api/v1/traceability` (one merged response) | Rifqi | ✅ | **KM-691.** `src/api/v1/traceability.py` → store.get → payload/404. Intent-based source rules (greeting/help/refusals = none; retrieve = required); full planning only on slow path. Contract §7 updated. |
| **3 — tools + obs** | Keep stream **text-only**; traceability is a separate parallel call | Rifqi | ✅ | **KM-691.** No trace data in the SSE stream; the FE fetches `/traceability` on `done`. |
| **3 — tools + obs** | Resolve `message_id` correlation (stream ↔ traceability) with Harry | Rifqi ↔ Harry | ✅ | **RESOLVED (pr/6):** Python is the **sole minter** — `message_id` dropped from the `/api/v2/chat/stream` + `/api/v1/tools/help` request bodies; always minted server-side (server-authoritative, FE-security) and returned on `done`. Any caller-sent `message_id` is ignored. Contract open-Q #1 closed. **Updated 2026-07-09 (pr/13):** id format changed from `msg_<hex>` to a canonical UUID string (`str(uuid.uuid4())`, both mint sites) to mirror Go's `analyses_messages.id` shape — the value is still independently Python-minted (not the real row id), only format-compatible for a future swap. |
| **4 — biz questions** | Get Go folder; confirm `business_questions` in create-analysis (max 5); sync Python | Harry/Mentor → Rifqi | ⬜ | Go currently missing the field ("lagi difixing"). Python already models objective + business_questions. |
| **deferred** | Report formats: PPT (preferred) / PDF / infographic on download | — | ⏸️ | MD is fine for the FE preview stage now. |
| **deferred** | Charts (Plotly→JSON) + images tables | — | ⏸️ | Carried from §4 #26/#27. |

**Next up:** Phase 2 Python work is **done** (chat→v2 `analysis_id`; `help`/`report`/`list` regrouped
under `/api/v1/tools/`). The `message_id` correlation contract is now settled in **pr/6** (Python sole
minter, stream-only). The **Phase 3 traceability build** — scratchpad + `GET /api/v1/traceability`
(contract §7) — is **done (KM-691)**. Next is **Phase 4** (business questions, Go-blocked).

---

## 0.5. pr/13 sprint — agent-quality fixes (2026-07-08 live-test review)

Findings from the scoped live sessions (mining analysis, 2026-07-07/08 traces): the planner
force-mapped absent measures (`pa` aliased as "revenue"), top-N ranked raw rows (duplicate models),
`analyze_trend` collapsed integer months into a single 1970-01 bucket, an invalid grouped IR reached
Postgres, failed retrievals wrote all-null traceability sources, and numeric catalog samples arrive
base64-mangled from Go. Fix tasks (same status legend as §0):

| # | Task | Owner | Status | Note |
|---|---|---|---|---|
| Q1 | IR validator: reject bare selects under `group_by` (planner retry self-corrects) | Rifqi | ✅ | `query/ir/validator.py` |
| Q2 | Planner **infeasible** path: `TaskList.infeasible_reason` + deterministic EN/ID data-gap reply | Rifqi | ✅ | schemas/validator/coordinator/refusals + planner.md "When the catalog cannot answer"; refusal wording → Rifqi to review |
| Q3 | `analyze_trend`: integer year/month handling (epoch-parse bug) | Rifqi | ✅ | `temporal.py` + 5 local tests |
| Q4 | Planner few-shots: top-N (Example G) + infeasible (Example H) + entity-vs-row ranking rule | Rifqi | ✅ | live-tested 2026-07-08: backlog top-3 correct via single-IR group+sum; "best PA performance" correct in-process (avg-per-model, assumption recorded). Stale-server trace was a false alarm |
| Q5 | Catalog numeric `sample_values` base64-decode stopgap (`catalog/sample_decode.py`) | Rifqi | ✅ | self-disabling; **primary fix = Go marshaling — DDL-free handoff to Harry** |
| Q6 | Traceability null-source suppression + `check_data` `-1` row-count hiding | Rifqi | ✅ | `scratchpad.py` / `data_access.py` |
| Q7 | `analyze_merge` two-table combine tool (unblocks "worst A + biggest B" questions) | tool owner | ✅ | tool shipped by Sofia (8abf635, KM-703); planner slice done 2026-07-09: `_validate_data_source` guards `data_right`, two-retrieve→merge few-shot (Example I), planner.md "Two measures per entity" bullet |
| Q8 | Report v2: business-question answer section, unresolved/excluded sections, evidence tables from `results_snapshot`, caveat dedupe, single language | Rifqi/Sofhia | ✅ | done 2026-07-09: still exactly ONE LLM call (extended to also draft `bq_answers`, index-based record refs, deterministic fallback = v1 behavior); evidence tables from table-kind outputs (≤3/record, ≤10 rows, ≤8 cols, `check_*` skipped); reply language via `detect_reply_language` on objective+BQs; verified in-process against live analysis 935a091e |
| Q9 | Record-curation endpoint (`GET …/records` + `exclude_record_ids`) + readiness GET for the FE delta guard | Rifqi ↔ FE | ✅ | done 2026-07-09: `GET /tools/report/{analysis_id}/records` + `/readiness` (registered before `/{version}` — int-coercion route-order trap), `exclude_record_ids` on POST; contract updated same change; FE wiring pending (Rifqi → FE) |
| Q10 | Traceability `data_used` layer — resolve IR ids → real names for the FE (users couldn't map `c_…`/`t_…` ids back to their data; aggregate aliases like `total_revenue` looked like real columns) | Rifqi ↔ FE | ✅ | done 2026-07-13 (pr/15): new `src/traceability/resolve.py` builds `data_used[]` (real source/table/column names; joins; plain-language filters; `columns_read` vs `output_columns` with `computed`+`formula`), `tool_calls[].summary`, `sources[]` gains `source_name`+all tables; **ids kept but machine-only (FE must not render)**; deterministic no-LLM, never-throw; catalog threaded to the scratchpad at the composition root. Contract + `TRACEABILITY_FE_HANDOFF.md` updated; FE wiring pending (Rifqi → FE). Additive/non-breaking |
| Q11 | IR wart: `OrderByClause.column_id` may hold a SELECT **alias** (a computed output), not a catalog column_id | Rifqi | 🔎 | surfaced by Q10 — the resolver tolerates it (`kind: "computed"` fallback), but the IR field name is misleading. Consider an explicit `by_alias` field or renaming. Low priority; no functional bug |

## 0.6. pr/16 sprint — Spine v2: W2 charts + W1 checkpoint (SPINE_V2_PLAN, approved 2026-07-13)

Scope approved 2026-07-13 (Rifqi + Sofia): build W2 (`render_chart` + chart store + `GET /charts`
+ planner viz slice) and W1 (S1a quality checkpoint). **W3 (activate deferred `analyze_*`) deferred
at approval — do not start until Rifqi re-opens.** W4 (S1b repair) stays gated on INV-6 sign-off +
S1a telemetry. Design + handoff source: `SPINE_V2_PLAN.md`.

| # | Task | Owner | Status | Note |
|---|---|---|---|---|
| V1 | `render_chart` tool slice: `visualization.py` (Plotly-JSON `dataeyond.chart.v1` envelope, no plotly dep) + `ToolOutput.kind` `"chart"` + registry + invoker | Rifqi (Sofia signed off on the tool-layer edit) | ✅ | done 2026-07-13; deterministic spec builder (bar/line/pie/scatter, fixed style preset), traceability scratchpad summarizes chart outputs compactly (point_count, not the raw arrays) |
| V2 | Chart store + API: `MessageChartRow` (`message_charts`) + `src/charts/store.py` (never-throw save) + write site in `_run_slow_path` + `GET /api/v1/charts` + contract §charts | Rifqi | ✅ | done 2026-07-13; empty list = valid 200; `done` event unchanged (no `chart_count` — open, Harry); FE fetches unconditionally on `done` |
| V3 | Planner viz slice: recipe table + "Charts only on explicit ask" rule (planner.md), Example J (viz tail) + Example K (viz-infeasible), validator Check 10 (`render_chart.data` must be table-kind), assembler chart one-liner guard | Rifqi | ✅ | done 2026-07-13; behavioral matrix verified in-process (real LLM): explicit ask EN/ID → chart tail; plain question → no chart; absent dimension → infeasible (Example K + prompt guard added after the first smoke force-mapped `status` AS "region") |
| V4 | W1 S1a quality checkpoint: `slow_path/checkpoint.py` (CK1–CK6) + `RunAssessment` schemas + coordinator call site + "Execution assessment" block in the assembler input + `refusals.run_failure_message` | Rifqi | ✅ | done 2026-07-13; 13 local tests (one per CK rule + never-throw + prompt + coordinator); CK1 all-failed → deterministic honest failure, **no assembler call**; clean run renders nothing (zero behavior change); every flag logs `repair_candidate` (S1b evidence) |
| V5 | `message_charts` DDL: run manually against dedorch (block for the live e2e chart test), then hand the schema to Harry for the dedorch migration | Rifqi → Harry | ✅ | Rifqi ran the DDL 2026-07-13; **live e2e ALL PASS** same day (real v2 endpoint: viz turn → chart row keyed by `done` message_id → `GET /charts` valid v1 envelope; chartless → 200 empty; injected `render_chart` failure → answer streams as a table, no row). **Remaining: send Harry the migration handoff** (schema + contract §charts) |
| V6 | Restore the `eval.chat_sim` harness — `eval/chat_sim/*.py` is missing from disk AND git (only `__pycache__` remains); §7B prompt gate can't run | Rifqi | ✅ | restored by Rifqi 2026-07-13 (accidental delete). ⚠️ its hard-coded `DEFAULT_USER_ID`/`TITANIC_SOURCE_ID` are stale (that user has no catalog; the Titanic blobs are gone) — update the constants before the next full run |
| V7 | Local `.env` lagged Go's Supabase-S3 data plane: `storage_provider=azure_blob` + empty `supabase_s3_*` made EVERY local tabular retrieve fail `BlobNotFound` | Rifqi | ✅ | found during the e2e (masked as "data not available" by the honest-degrade path); Rifqi set the six values 2026-07-13. Gotcha documented in REPO_STATUS §13 |
| V8 | Lead review of `GET /charts`: lookup by `message_id` alone + tri-state response marker (`status: success \| empty \| not_found` + `message`) instead of a bare list | Rifqi (lead ask) | ✅ | done 2026-07-14; `not_found` vs `empty` decided against the turn's traceability row (PK lookup); always HTTP 200; contract §charts updated. ⚠️ additive DDL for the new lookup: `CREATE INDEX IF NOT EXISTS idx_message_charts_message ON message_charts (message_id);` (run manually + include in Harry's migration). Traceability GET still takes both params — aligning it is open |
| V9 | Report chart embedding: `AnalysisReport.charts` (verbatim envelopes per record) + `## EDA` section with ` ```plotly ` fences (content = the **full v1 envelope**, pretty-printed — the FE hook's verified shape); `has_successful_analysis` extended so a successful `render_chart` counts (chart-only sessions satisfy the report floor) | Rifqi | ✅ | done 2026-07-14; first cut emitted bare `{data, layout}` — FE test showed the hook parses the full envelope, fixed same day; live reports v3 (wrong fence) → **v4 (correct)** for analysis `7be50846…` (3 charts embedded); suite **381 passed, 7 skipped** (+5 chart-embed tests; same 2 pre-existing failures) |

Full-suite evidence for this sprint: **376 passed, 7 skipped** (+13 new checkpoint tests; the 2
failures — `test_chat_handler::test_structured_flow_runs_slow_path`,
`test_reader::test_structured_read_falls_back_to_user_scope_when_no_analysis_row` — reproduce at
HEAD before this diff, i.e. pre-existing). Ruff clean on all touched paths; `import main` OK.

## 1. The direction change (locked decisions from 2026-06-24)

1. **"Problem statement" is replaced by two user-entered fields: `objective` + `business_questions`.**
   User fills them at onboarding; **both mandatory to submit; NO agent validation.**
2. The **gate (`problem_validated`) and the `problem_statement` skill/intent are removed** (comment out, don't delete).
3. **Report is records-based** (reads persisted `AnalysisRecord`s) — **decided and pushed** (KM-674).
   It is formal markdown: title, date, "generated by {user}", objective, business questions,
   findings, insights. **NOT gated** on whether business questions were answered.
4. **`owner_id` → `user_id`** everywhere (Harry mirrors in dedorch/Go).
5. **State writes go through a request to Go**, not direct Python DB writes.
6. **FE-callable surface = 4 endpoints:** `call_agent` (chat/stream), `list_skills` (`GET /tools`),
   **skill: help**, **skill: report**. `problem_statement` removed; `check_data` not FE-facing from
   Python (Go provides it); analysis CRUD not needed from FE (comment, don't delete).
7. Deliverables for Harry: (a) API endpoint doc (MD); (b) full Python project doc (MD → PDF/Word BRD).
8. Integration tested via Swagger `/docs` on the HF Python build (simulating FE manually). Target ~Wed.

## 1.5. 2026-06-25 checkpoint deltas

Confirms the 2026-06-24 direction and adds these concrete changes (folded into §4 as tasks 21–28):

1. **Rename `analysis_records` → `report_inputs`** (DONE #21) — names the table by purpose (the rows
   report generation reads); avoids clashing with Go's `analyses_messages` and with Langfuse
   observability. **Stays Python-owned**; finalized schema handed to Harry so his dedorch migration
   creates it post-`SKIP_INIT_DB` (#22, resolves #16). Write scope = **one row per slow-path analysis
   run** (decided — not per-agent-call telemetry; that stays Langfuse).
2. **`analyses` table (Go) — `status`, `data_bind` + `data_bind_version`, `report_collection`** (id+version).
   **Verified 2026-06-25: these + `user_id` are ALREADY present in dedorch `analyses`.** Plus Harry drops
   the duplicate/wrong singular `analysis` table. (→ #3)
3. **`analyses_messages` (Go) = the analysis chat room** (user Q + agent A) — replaces the now-**deprecated**
   `chat_messages`/`rooms`; Python's chat read/write must migrate here before cutover. (→ #25)
4. **Reports: Go owns ALL writes.** Report stays a **skill** (no router intent): FE → Go → Python;
   Python only returns content. Input = the records table (now `agent_observability`); edit-mode may
   also need the last report. (→ #7/#18/#24)
5. **Markdown minimum now:** tables, **bold**, *italic*, horizontal separators — optimize that before
   anything fancier. (→ #23)
6. **Deferred:** charts (prefer **Plotly→JSON** in a future `chart` table over matplotlib PNGs) and
   images (image table keyed by analysis/message/report + originals in a bucket). (→ #26/#27)
7. **Near-term:** the remove-`problem_statement` work isn't on HF yet → **PR + deploy + test in the
   playground** (#13). Harry stabilizes Go ~Fri; FE manual testing ~Mon. **Keep it playground-able.**
8. **UI research** (no dedicated UI person): new-analysis form (title/objective/business_questions),
   knowledge menu (user-level vs analysis-level binding), report artifacts panel + version selector;
   interview + old analysis UI removed. (→ #28)

## 2. What is already done (KM-674, pushed on `pr/4`)

Report layer adapted to the new goal shape:
- `report/schemas.py::ProblemStatement` → `objective: str` + `business_questions: list[str]`
  (old `target_value`/`scope`/`metric_direction`/`target_metric` dropped). Class name kept for now
  (rename to `ReportGoal` once the upstream AnalysisState rename lands).
- `report/generator.py` renders **Objective** + numbered **Business Questions** + a
  **"generated by {user}"** line.
- `api/v1/report.py::_problem_statement_from` is **tolerant**: prefers new `objective` /
  `business_questions` from state, falls back to legacy `problem_statement` — works before AND after
  Harry's migration.
- `config/prompts/report_summary.md` updated to objective + business questions.
- Report stays **records-based**; the floor gate (`problem_validated`) was deliberately left for task #2.

**This tolerant-migration pattern (getattr fallback) is the model for tasks #2 and #4.**

## 3. Assessment — gaps & contradictions to resolve before building

These came out of reviewing the plan against the actual code. They are folded into the task table (§4) as tasks 15–19.

- **G1 (→ task 15). Records-based reports need the slow path ON.** `AnalysisRecord`s persist only in
  `chat_handler._run_slow_path`, which runs only when `ENABLE_SLOW_PATH=true`. Default is off → no
  records → `POST /report` 409s. The Swagger demo can't show a non-empty report unless slow path is
  flipped on and a `structured_flow` question is run first. `BusinessContext` is still a stub but the
  slow path runs fine on it.
- **G2 (→ task 16). `analysis_records` ownership is now required and collides with `SKIP_INIT_DB`.**
  It's created today by Python `create_all` (`db/postgres/init_db.py`), is in no dedorch/Go migration,
  and after the dedorch cutover (`SKIP_INIT_DB=true`) Python stops running `create_all` → the table
  won't exist → reports break. Decide: dedorch migration (Harry) OR a Python carve-out that creates
  just this one table even under `SKIP_INIT_DB`. *(Resolved 2026-06-25 — see §1.5.1 / #16 / #22.)*
- **G3 (→ task 17). `chat_history` in the report contract is vestigial.** Records-based generation
  reads records by `analysis_id`; it never uses chat history. Drop `chat_history` from the report
  skill contract, or mark it reserved/unused.
- **G4 (→ task 4 note). Make #2/#4 tolerant of both state shapes.** If Harry drops
  `problem_validated`/`owner_id` from dedorch before Python stops reading them, Python's gate +
  state_store break. Use the same `getattr` tolerance KM-674 used. The `owner_id`→`user_id` rename
  also touches `api/v1/analysis.py` (`_serialize_state`, `list_analyses`, `get_analysis`), not just
  the model + state_store.
- **G5 (→ task 18). "State writes via Go" is bigger than `report_id`.** Python still writes state in
  `/analysis/create` (state + room + bindings, plus the data-first gate and soon the mandatory-field
  check) and in `state_store.ensure` per turn. If creation moves to Go (consistent with commenting
  analysis CRUD), then Go owns ALL state writes + both creation gates, and Python's `ensure` must
  become a **read-only get** (Go must guarantee the row exists before any chat turn).
- **G6 (smaller).**
  - Removing `problem_statement` (task 1) means neutering it in 4 places: the `Intent` literal
    (`agents/orchestration.py`), the router prompt (`config/prompts/intent_router.md`), the handler,
    and the gate's redirect *target*. Do it with task 2.
  - "generated by {user}" currently prints the raw `user_id`; a formal report wants a name — source
    from `users.fullname` or have Go pass a display name (task 19).
  - The meeting's outline (background / EDA / insights) isn't fully in the renderer; map those
    sections onto the record fields deliberately (task 5 follow-up).
  - The full project doc (task 11) should reuse [REPO_STATUS.md](REPO_STATUS.md), not restart.

## 4. Task table

Status legend: ⬜ not started · 🔄 in progress · ✅ done · ⛔ blocked · 🔎 verify · ⏸️ deferred.

| # | Task | Owner | Status | Note |
|---|---|---|---|---|
| 1 | Comment out `problem_statement` skill **+ `Intent` literal + router prompt + gate redirect target**; remove `/problem-statement` from `list_tools` | Rifqi | ✅ | Done 2026-06-25 (one commit w/ #2). Unwired in `orchestration.py`, `intent_router.md`, `chat_handler.py`, `tools.py`; `problem_statement.py` kept intact |
| 2 | Drop `problem_validated`: gate neutered; `is_report_ready`/`report_floor` → **≥1 completed analysis** only, no-LLM | Rifqi | ✅ | Done 2026-06-25. `gate.py` no-op, gate call site commented in `chat_handler.py`, `report_floor` drops the goal check. Tests updated (`test_gate`/`test_chat_handler`/`test_readiness`). Suite: **284 passed, 7 skipped**; ruff clean |
| 3 | dedorch `analyses` migration: drop `problem_statement`/`problem_validated`, add `objective` + `business_questions` | Harry | 🔄 | **Verified dedorch 2026-06-25:** `analyses` (plural) ALREADY has `user_id` + `status` + `data_bind` + `data_bind_version` + `report_collection` → those parts done. **Remaining:** drop `problem_statement`/`problem_validated` + add `objective`/`business_questions`. Singular `analysis` = deprecated duplicate to drop |
| 4 | Update Python `analyses` model + `state_store` + `analysis.py` to match dedorch; `owner_id`→`user_id` | Rifqi/Sofhia | ✅ | Done 2026-06-26. `owner_id`→`user_id` + added `status`/`data_bind`/`data_bind_version`/`report_collection` (DB-only, not in the `AnalysisState` pydantic) across `models.py`/`gate.py`/`state_store.py`/`analysis.py` + 3 local tests; also `report_inputs` `id`/`analysis_id` → `uuid`. Kept `problem_statement`/`problem_validated`; `objective`/`business_questions` wait on Harry's #3. Suite **284 passed** |
| 5 | Report generator → `objective`+`business_questions`, "generated by {user}", formal outline | Sofhia | ✅ | Goal-shape (KM-674) + author name (#19) + outline (KM-680): Objective → Business Questions → Executive Summary → Key Findings → EDA → Notes & Limitations → How This Was Analyzed |
| 6 | Report skill input contract: `analysis_id` + `user_id` (no `chat_history`) | Sofhia/Rifqi | ✅ | No-op: `POST /report` already takes only analysis_id + user_id (records-based). Documented in API_ENDPOINTS.md §5. *(Edit-mode input revisited in #24.)* |
| 7 | `report_id` state update via request to Go, not direct DB | Sofhia + Harry | ⬜ | Needs Go endpoint. **Checkpoint:** Go owns ALL `reports` writes; Python stops any direct insert/update and only returns content; report stays a **skill** (no intent). See #18 |
| 8 | Expose/confirm 4 FE endpoints; comment `check_data` + analysis CRUD | Sofhia | ✅ | KM-678: `list_tools` trimmed to `/help` + `/report` (analytics/check/retrieve commented in the **menu**). `help` confirmed as a `call_agent` intent — no own endpoint. Analysis CRUD endpoint left **registered**: "comment the rest" was about the FE slash menu, not killing HTTP routes Go needs |
| 9 | Verify `analysis_id` in `call_agent` contract | Sofhia | ✅ | Verified: no separate field — carried as `room_id` (`analysis_id == room_id`), per REPO_STATUS §4/§11. Action for Go: send the id as `room_id` |
| 10 | API endpoint doc (MD), 4 endpoints, for Go integration | Rifqi + Sofhia | ✅ | Done 2026-06-25 — `API_ENDPOINTS.md` (repo root). 4 FE surfaces with request/response **examples** (chat SSE transcript, report 201/409 JSON, version list), schemas, §9 full 32-route inventory + task-8 reading |
| 11 | Full Python project doc (MD → PDF/Word BRD) | Rifqi | ✅ | Done 2026-06-26 — `PROJECT_BRD.md` (repo root): purpose/context, FR-1..9 capabilities, lifecycle, architecture, data model, API (→ API_ENDPOINTS), NFRs, integrations, open items. Reuses REPO_STATUS/API_ENDPOINTS; convert to PDF/Word for distribution |
| 12 | Reconcile/open the `list_tools` PR cleanly (stacked commits) | Rifqi | ✅ | N/A — we develop directly on the single active branch `pr/4` (KM-652 + KM-678 already stacked there); no separate PR to reconcile |
| 13 | Deploy HF Python build (remove-`problem_statement` work) → test 4 endpoints via Swagger / playground | Sofhia + Harry | 🔄 | **Unblocked (#15 ✅).** Remove-PS work is on `pr/4` but **not on HF `main` yet** → PR + deploy, then manual test. Harry stabilizes Go ~Fri; FE testing ~Mon |
| 14 | `analysis_records` home | Rifqi + Sofhia + lead | ✅ | **Resolved 2026-06-25:** stays Python-owned, **renamed** (→ #21); schema handed to Harry so the dedorch migration creates it post-cutover (→ #22). Not moved to Go |
| 15 | Flip `ENABLE_SLOW_PATH=true` + verify an `AnalysisRecord` persists from a `structured_flow` question | Rifqi | ✅ | Verified locally 2026-06-25 (in-process). structured_flow on Titanic.csv → 3-task plan `check_data→retrieve_data→analyze_aggregate` (all success) → AnalysisRecord persisted (substantive) → `report_floor` pass → report generates (201). HF env-flip + Swagger run folds into #13 |
| 16 | Decide `analysis_records` creation under `SKIP_INIT_DB` | Rifqi + Harry | ✅ | **Resolved 2026-06-25:** Python defines it; **Harry's dedorch migration creates it** on env-move (Python still creates locally meanwhile) → exists post-cutover. Execution = #22 |
| 17 | Reconcile report contract with records-based: remove/flag `chat_history` | Sofhia/Rifqi | ✅ | Nothing to remove — `chat_history` was never in the report contract/code (only in help.md). Confirmed via grep; API_ENDPOINTS.md §5 documents the clean contract |
| 18 | Confirm Go owns ALL analysis-state writes + both creation gates; make Python `state_store.ensure` read-only | Rifqi + Harry | ⬜ | **Confirmed by 2026-06-25 checkpoint** (Python read-only; Go owns writes + new tables). Execution pending Go endpoints |
| 19 | Decide report author display-name source (`users.fullname` vs Go-passed name) | Sofhia | ✅ | Done 2026-06-25. `AnalysisReport.user_name`; `generator` renders `user_name or user_id`; `api/v1/report.py::_resolve_user_name` reads `users.fullname` never-throw (fallback `user_id`). Decided: resolve in Python (unblocked); swap to Go-passed name later if preferred |
| 20 | **Help handoff:** update `handlers/help.py` + `help.md` — drop the `problem_validated` tier + `define_problem_statement` action (the skill it points at is gone as of #1) | Sofhia | ✅ | Done 2026-06-25. `help.py`: actions = `ask_analysis_question` (always) + `generate_report` (if ready); renders objective/business_questions (getattr-tolerant). `help.md` v1→v2: 3 tiers, no `/problem_statement`, `/generate report`→`/report`. Local test_help updated → 11 pass |
| 21 | Rename `analysis_records` → **`report_inputs`** (table, ORM `ReportInputRow`, store `*ReportInputStore`) | Rifqi | ✅ | Done 2026-06-26. `sed` rename across 9 files; Pydantic `AnalysisRecord` kept; columns stay String (pure rename — uuid+FK is the #22 Harry schema). Name `report_inputs` (purpose; avoids Langfuse/`analyses_messages` clash). Write scope = one row per slow-path run. Suite **284 passed** |
| 22 | Finalize `report_inputs` schema → hand to Harry for the dedorch migration | Rifqi → Harry | ✅ | **DDL ready** (uuid `id`/`analysis_id` + FK→`analyses(id)`; `user_id`/`plan_id` text; `data` jsonb = serialized `AnalysisRecord`, shape documented). dedorch has empty `analysis_records` → rename. Resolves #16. ~~**Action: send Harry the DDL + `data` shape**~~ **RESOLVED 2026-07-22:** Harry's `0001_create_core_schema.sql` now carries `report_inputs` with exactly this shape — FK to `analyses(id)` included. Python's `ReportInputRow` diffed against the live Neon table: **column-for-column identical, no drift.** The two remaining un-migrated Python tables (`message_traceability`, `message_charts`) split out as → #32 |
| 23 | Report markdown formatting: tables, **bold**, *italic*, horizontal separators | Sofhia | ✅ | Done 2026-06-25. Added `---` separators between header + each section in `_render_markdown`. Tables (EDA) / bold (method labels) / italic (meta + citations) already emitted. Relaxed `report_summary.md` to allow inline `**bold**`/`*italic*` for emphasis (kept no-headings/no-bullets so it doesn't duplicate the section structure / Key Findings). Compile + ruff clean |
| 24 | Clarify report input contract: records table (+ `last_report` for edit mode?) | Rifqi/Sofhia ↔ Harry | ⬜ new | Edit-mode input left open at the checkpoint |
| 25 | Migrate Python chat path to Go `analyses_messages` (+ `analyses`) | Rifqi ↔ Harry | ✅ | Done 2026-07-02. Read path already on `analyses_messages` (commit `0066161`). This change makes Python **read-only**: removed the `save_messages` calls from `/api/v2/chat/stream` so **Go is the sole writer** — fixes the double-write both Go+Python were producing. `load_history` still reads `analyses_messages`. v1 `/chat/stream` is unwired so left untouched |
| 26 | **Charts (DEFERRED):** store Plotly JSON in a future `chart` table (not matplotlib PNG) | — | 🔎 | **Landed 2026-07-13 (pr/16, §0.6 V1–V3):** `render_chart` + `message_charts` + `GET /api/v1/charts`, Plotly JSON as decided. 🔎 pending the dedorch DDL run (V5) + live e2e |
| 27 | **Images (DEFERRED):** image table (id, analysis_id, msg/report ref, order) + originals in a bucket | — | ⏸️ | Maintenance-heavy; parked |
| 28 | **UI research** (FE): new-analysis form, knowledge menu (user vs analysis level), report artifacts + version selector | Team | ⬜ new | No dedicated UI person; interview + old analysis UI removed |
| 29 | **LLM env quad rename `__4o` → `__54m`** + set the four `azureai__*__54m` secrets on the HF Space | Rifqi | 🔄 | Code done 2026-07-14 (pr/17): settings quad + 9 call sites + 2 eval runners renamed; deployment is `gpt-5.4-mini`. Suite **381 passed, 2 failed (both pre-existing), 7 skipped** — same as baseline; ruff clean on touched files; `import main` 0. **Hard rename — no `__4o` fallback**, so HF fails loudly on the first LLM call until the four `__54m` secrets are set. Root cause it fixes: HF silently ran GPT-4o while local ran 5.4-mini → same question, same catalog, planner hallucinated catalog ids on HF only. **Action: set HF secrets, redeploy, re-run the failing question.** Local `.env` still carries the now-dead `__4o` keys — safe to delete |
| 30 | **Neon `reports.user_id` NOT NULL** — reconcile the ORM + `ReportStore` write | Rifqi | ✅ | Done 2026-07-22 (pr/18, commit `13f74e1`). `ReportStore.save` never wrote `user_id`; on Neon that column is `text NOT NULL` → **every `POST /api/v1/tools/report` 500'd** with `NotNullViolationError`. Fix: `models.py` adds `user_id = Column(Text)` (**nullable** on purpose — Python matches the loosest deployment shape, §7D getattr-tolerance) + `store.py` writes `report.user_id`, which the endpoint already required (`user_id: str = Query(...)`) and threaded via `generator.generate`. Live-tested by Rifqi → **201**. Suite 381 passed / 2 pre-existing failures / 7 skipped; `import main` 0; ruff unchanged (3 pre-existing E501 in `models.py`, same count on HEAD). **Correction 2026-07-22 (after pulling Go `dd37b38`):** the commit message and the first version of this row said the column was *not* in Go migrations `0001–0004`. **That was wrong** — `0001_create_core_schema.sql` has declared `reports.user_id TEXT NOT NULL` since the schema was written. Real root cause → #31 |
| 31 | **Go migration set is not convergent** — fresh vs migrated dedorch DBs get different NOT NULL constraints | Rifqi → Harry | ⬜ new | Root cause of #30, verified in the Go source 2026-07-22. `0001_create_core_schema.sql` creates `reports.user_id TEXT NOT NULL` and `analyses_messages.user_id TEXT NOT NULL`; `0002_cleanup_legacy_schema.sql` (L95, L92) and `0004_replace_chat_with_analysis_scope.sql` (L61, L57) retrofit the same two columns onto pre-existing DBs as **nullable** `ALTER TABLE … ADD COLUMN IF NOT EXISTS user_id TEXT`. Because `CREATE TABLE IF NOT EXISTS` no-ops on an existing table, **the same migration set yields two different schemas** — the old dedorch DB got nullable (hiding Python's missing write for months), Neon got NOT NULL. **Ask Harry:** make the retrofit converge (backfill + `ALTER COLUMN … SET NOT NULL`, or add the constraint in a new migration) so every instance matches `0001`. Until then, `information_schema` on the target instance — not the migration files — is the only reliable schema source (REPO_STATUS §13). **Action: Rifqi raises with Harry** |
| 32 | **`message_traceability` + `message_charts` are in no Go migration** — hand the DDL to Harry | Rifqi → Harry | ⬜ new | Found by the 2026-07-22 drift scan. Both tables exist on Neon **only because they were created by hand** (2026-07-06 / 2026-07-13); neither appears in `0001–0006`. Any newly provisioned dedorch instance will be missing them → traceability flush and chart persist fail (both never-throw, so they degrade **silently** — no 500 like #30 to make it visible). DDL for `message_charts` is in `SPINE_V2_PLAN.md` §4.4. This is the surviving half of #22. **Action: Rifqi sends Harry both DDL blocks** |

## 0.7. pr/19 — code review remediation (2026-07-23)

From the end-to-end review in `CODE_REVIEW_2026-07-23.md` (findings are cited as **F-n** there)
plus the live report bug on analysis `966224d4…`. Same status legend as §0.

| # | Task | Owner | Status | Note |
|---|---|---|---|---|
| 33 | **Report body vs floor split** — `has_reportable_result` for the body, `has_successful_analysis` stays the floor | Rifqi | ✅ | Shipped 2026-07-23. Root cause: planner R2/R2b make `analyze_*` optional, so a correct analyze-free run was classed non-substantive, dropped from the body, and its business question rendered **"Unanswered"**. Live-verified by Rifqi on `966224d4…` |
| 34 | **Report floor extension** — a successful `retrieve_data` **that returned rows** clears the floor | Rifqi | ✅ | Same root cause; fixes a hard **409** for a session where every question is R2/R2b-shaped. Guardrail-adjacent, authorised 2026-07-23. NOT the "Floor Fixer" failure mode: the floor still asks "did we produce a real result" — empty retrievals, `check_*`-only and fully-failed runs all still fail it |
| 35 | **`GET …/records` `substantive` flag** repointed to the body predicate + contract §records updated | Rifqi | ✅ | The curation list was contradicting the artifact it curates. Behavioral, non-breaking: no field added/removed/retyped |
| 36 | **CK5b** — quality checkpoint covers analyze-free plans | Rifqi | ✅ | CK5 only inspected `analyze_*` tasks, so an all-null aggregate column on an R2/R2b plan reached the answer unflagged. `check_*` excluded (uncounted tables legitimately carry nulls) |
| 37 | **F-2 service-secret gate** — `X-Dataeyond-Service-Secret`, router-level dependency | Rifqi | ⏸️ | Shipped inert 2026-07-23. **UNWIRED 2026-07-27 (lead decision).** Verified in `E2E-Frontend-Data-Eyond/src/services/agenticApi.ts` that the **browser SPA is the only caller** of Python and we don't own it — it sends only `Content-Type`, and cannot be changed by us to send the header. So the gate could never be armed without a 401 outage: a wired-but-unarmable gate is a footgun (any operator setting `dataeyond__service__secret` breaks prod). Guard dependency removed from all six router mounts in `main.py`; `service_auth.py` kept in-tree but parked (comment-out-don't-delete); the `dataeyond_service_secret` setting commented. A prepared FE proxy (Node `server.js` holding the secret and injecting the header server-side) is the way to arm it later without exposing the secret to the browser — but that needs FE-repo access we don't have. **Net: the live surface is unauthenticated by design; real auth = #43. F-1 tenant predicates (#38) stay** as defensive-in-depth. |
| 38 | **F-1 tenant scoping** — `user_id` predicate on the six analysis-keyed reads | Rifqi | ✅ | `CatalogStore.get_by_analysis` filtered on `analysis_id` alone where Go filters on both; the catalog payload carries the owner's `user_id`, so `DbExecutor`'s ownership check compared the victim's id against itself and passed → cross-tenant **query execution against a customer DB**. Defence-in-depth only until #37 is armed (`user_id` is caller-supplied, and `GET /traceability` leaks it) |
| 39 | **Stale tests resolved** — the two long-standing suite failures | Rifqi | ✅ | Both encoded the pre-2026-07-13 user-scope fallback that `reader.py` deliberately removed. Not product bugs. Suite is now **394 passed / 0 failed / 7 skipped** — fully green for the first time |
| 40 | ~~**F-3** — scope `GET /charts` + `GET /traceability` by `user_id`~~ | Rifqi | ⛔ | **RESOLVED 2026-07-23 — DECLINED by Rifqi. Do not re-open without his sign-off.** Both endpoints keep their existing lookup keys: `/traceability` by `(analysis_id, message_id)`, `/charts` by `message_id` alone (the 2026-07-13 lead decision). F-3 proposed adding a `user_id` parameter to both; an optional-param version was implemented on 2026-07-23 and then **reverted in full** (endpoints, charts-store predicates, and the contract notes) once the decision was restated — no FE change is required and none should be requested. **Accepted consequence:** both endpoints remain unauthenticated capability URLs over real customer data (`charts[].spec.plotly.data` is actual table values; the traceability payload carries 5-row previews, the executed SQL, and the owner's `user_id`). **The service-secret gate (#37) is therefore the only control protecting them** — which raises #37 from important to load-bearing. The stores' optional `user_id` parameters are kept as dead capability for a future Go-forwarded identity (#43); `PostgresChartStore` was returned to its message_id-only form. |
| 41 | **F-12 / F-13** — bound the planner catalog render and the tabular blob read | Rifqi | ✅ | Shipped 2026-07-23. **F-13:** neither storage backend could report an object size, so `object_size()` was added to both (S3 `head_object`→`ContentLength`, Azure `get_blob_properties().size`, each returning None rather than raising) and `TabularExecutor` now refuses >500 MB **before** downloading, with a post-download byte check as the fallback when the probe is unavailable. **F-12:** `render()` gained TWO ceilings — `_MAX_TABLES=150` and `_MAX_CATALOG_CHARS=250_000` — because a table-count cap alone leaves wide tables unbounded (20 tables × 300 cols is as fatal as 400 × 30). Measured after: 400×30 went from ~241k tokens to ~63k; 100×30 (~45k tokens) still renders **in full**, so no realistic catalog is touched. Truncation emits an explicit "N more tables not shown" line so the planner knows it saw a subset, and logs — that log is the signal to retune. 12 new tests |
| 42 | **F-20 observability** — `degraded_seam=<name>` on every never-throw / silent-drop path | Rifqi | 🔄 | The 2026-07-23 report bug was invisible by construction: the record was dropped with zero logging. **Partially shipped 2026-07-23** — the 10 seams where silent degradation is user-visible now emit a stable `degraded_seam` field (+ `repr(e)` instead of `str(e)`, so an empty-`str()` Fernet error is no longer a blank log): `input_guard_fail_open`, `analysis_catalog_read`, `report_floor_record_read`, `traceability_persist`, `traceability_flush`, `chart_persist` (×2), `report_input_persist` (×2), `analysis_state_ensure`. **Remaining:** the other ~76 `except Exception` sites, most of which are in unwired routers (`db_client`, `data_catalog`, `users`) or non-live paths — deliberately not swept, since a blanket edit across unwired code is exactly the drive-by §7A forbids. Control flow unchanged throughout (§5.4) |
| 43 | **Go identity contract** — what does Go forward, and when? | Rifqi ↔ Harry | ⬜ | **Now the sole path to caller auth** after #37 was unwired 2026-07-27. Python cannot authenticate the caller alone; either Go forwards a verified per-user token, or the FE (once we can change it) forwards the Go bearer token it already holds (`orchestrationApi.ts` shows the FE has one). Until then the live surface is unauthenticated and the #38 predicates are defensive only. Options for arming interim protection when FE access returns: the `server.js` BFF proxy (secret server-side) or route agentic calls through Go |
| 44 | **F-17 / F-18 / F-25 compiler-parity batch** | Rifqi | ✅ new | Shipped 2026-07-23. Three execution-verified review findings that had **no task row** — the tracker jumped from #39 to #40 and lost them. **F-17 (High):** the bare-select check was gated on `if ir.group_by`, so a mixed select with `group_by=[]` passed; Postgres then failed loudly but the pandas path silently DROPPED the column while `output_columns` still advertised it → a real-looking table with a fabricated all-null column. Fixed in `validator.py` (fires whenever any agg is present — no false positives possible) + a presence backstop in `TabularExecutor`. **F-18:** `astype(str)` ran before `na=False`, so `NULL LIKE '%an%'` matched the literal `"nan"`/`"None"`. **F-25:** `SqlCompiler` raised on an empty `in`/`not_in` where pandas and `_column_values`' own docstring implement the empty-set semantics — a two-step plan whose first step legitimately returned zero rows hard-failed instead of answering. 16 new parity tests; one stale test updated (it pinned the old F-25 raise) |
| 45 | **F-9 PII in persisted artifacts** — mask the traceability preview + report evidence tables | Rifqi | ✅ | Shipped 2026-07-24. `pii_flag` was an **ingestion-time control only**: it nulls `sample_values` into the planner prompt, but nothing stops the planner SELECTing a flagged column — and "list our top 20 customers" legitimately selects `customer_name`/`email`. Real values then reached two PERSISTED sinks: `message_traceability.data` (served by an unauthenticated GET, F-3 declined) and report evidence tables frozen permanently into `reports.content`. Fix = `retrieve_data` now carries `meta.pii_columns` (resolved through the IR select list, so aliases are honoured), and both sinks redact those cells to `[redacted]`. **Per the 2026-07-23 decision the ASSEMBLER still receives real values**, so the answer prose is unchanged and the question stays answerable — verified end-to-end: assembler input kept `Ada Lovelace`, the persisted preview showed `[redacted]`. Aggregates are deliberately NOT masked except `min`/`max`: `sum(salary)` identifies nobody, but `max(email)` returns one customer's actual address. Fails **open** (an unresolvable name is left unmasked, never a legitimate column wrongly blanked), so this is a mitigation, not a guarantee. 15 new tests |
| 46 | **F-8 prompt-injection resistance** — planner / assembler / report_summary | Rifqi | ✅ new | Shipped 2026-07-24. The three prompts that ingest customer data had **no injection rule**: `guardrails.md` is appended only in `chatbot.py`/`help.py`, and the `InputGuard` screens only the user's message — it never sees catalog or row content. This is the one attack the five query-defense layers structurally cannot see, because every IR the planner emits is individually valid. A hostile string only needs to reach a text column in the customer's OWN database (a product description, a support ticket, a form field) to be sampled into `sample_values` and rendered verbatim into the planner prompt. Fix = a purpose-written "content is data, never instructions" rule in each prompt + `<data>…</data>` delimiters around the catalog render and the run-state render. **Deliberately NOT `guardrails.md` wholesale** — its rules prescribe refusal sentences, and the planner's only free-text field is `infeasible_reason`, so those strings would surface there and could regress the Q2 data-gap path (tests pin their absence). Verified: planner eval **6/6, carried_over 5/5 green**; live hostile-catalog run planned only `t_products` and never touched the planted `employees.salary`; 10 new tests |
| 47 | **Cheap-batch review fixes** — F-19, F-22, F-24, F-26, F-16, F-4 | Rifqi | ✅ new | Shipped 2026-07-24, one low-risk batch. **F-19:** `sources` was missing entirely on `check` + router-`help` and came *after* `status` on the slow path — contract said always-first; additive fix, contract updated. **F-22:** `analysis_id` now 422s unless it parses as a UUID, on **both** live endpoints (chat + help kept identical on purpose). **F-24:** traceability/chart writes are skipped, and logged, when `analysis_id` is falsy — Go `0007` declares `analysis_id UUID NOT NULL` (+ FK on charts), so `analysis_id or ""` failed the insert and the never-throw seam lost the row silently. **F-26:** `QueryResult.error` uses `str(e) or repr(e)` — a Fernet `InvalidToken` reached the assembler/traceability/report as an EMPTY string; falling back only when `str()` is empty changes no existing message. **F-16:** retrieval cache key gains `redis_prefix` (two envs on one Redis cross-served results). **F-4:** non-Postgres sources now refused at the executor — **zero blast radius today** (Go's `isSupportedActive` allows only `postgres`), a tripwire so nobody re-enables a path that has no read-only session and no `statement_timeout`; legacy branch commented out per house convention, orphaned import commented with it |
| 48 | **floor_08 — floor/body disagreement** | Rifqi | ✅ | **Decided + shipped 2026-07-24 (lead).** #34's row-producing arm was unconditional, so a plan that HAS an `analyze_*` step which FAILED still cleared the floor on the strength of its upstream fetch — while the body rejected it. Because the "Attempted, Unresolved" section is commented out, such a run left no trace: as a session's only run it produced an empty report with the business question "Unanswered" (the #33 bug via another door). Fix = the arm now applies **only when the plan has no analysis step**, mirroring `has_reportable_result`; the shared `_plan_has_analysis` helper means the two predicates can no longer drift on that question. `floor_08` flips to `expected_ready: false`. The INTENDED asymmetry is preserved and re-verified: a zero-row retrieval still fails the floor but passes the body (floor_03). Readiness eval **17/17** |
| 49 | **F-5 timeout does not stop the customer's query** | Rifqi | ✅ new | Shipped 2026-07-24. `asyncio.wait_for` cancels the awaiting **coroutine**; the `to_thread` worker is not cancellable and runs to completion, holding a thread and a connection on the **customer's** database after we already answered "timed out". Two fixes. **(a) Dedicated bounded pool** — DB work moves to its own `ThreadPoolExecutor(50, 'dbexec')` via `run_in_executor`; abandoned workers previously accumulated in the shared default pool (`min(32, cpu_count+4)`) alongside every other `to_thread` caller, notably the tabular Parquet loader, so a few slow customer queries could stall unrelated work process-wide. **(b) Session hardening is no longer best-effort** — and a **latent bug** was found while reading it: both SETs shared one `try`, so a `statement_timeout` failure **skipped `default_transaction_read_only` entirely** and the connection served queries in a WRITABLE session behind a `logger.warning`. Now independent: read-only **fails the connection** if it cannot be set (a writable session against a customer DB is not something to degrade into); `statement_timeout` logs at **error** with `degraded_seam` but does not refuse service, since it is their I/O at risk rather than our correctness. ⚠️ **Blast radius to watch:** if any deployment currently fails that SET silently, its sources now fail loudly instead. Believed impossible (Neon accepts it as a SET — the existing comment documents this), but it is the one judgement call here. 7 new tests |

**Reading `eval/readiness/results/` (note for future sessions).** Four files are dated
2026-07-23. `…_150632.json` scores **4/15 (26.7%)** — that is **not** a product
regression. It is the run that exposed the eval harness itself being broken by
`fd4865b` (`_FakeRecord` had no `results_snapshot` for #34, `_FakeStore` took no
`user_id` for #38; both errors were swallowed by `report_floor`'s fail-closed seam, so
every case reported "not ready"). `…_150859` (15/15) is post-harness-fix, `…_150948`
and `…_152615` (17/17) add the two cases that exercise #34. **The current baseline is
`…_152615.json` — 17/17.** The intermediate files are kept as the audit trail for the
drift; per §7F no result file is ever deleted or overwritten.

**Not re-raised:** F-4 (non-Postgres read-only/timeout gap) was **downgraded to latent** —
Go's `database_clients.Service.Create` enforces `isSupportedActive`, and only `postgres` is
`active`, so no such source can be registered today.

**Decided 2026-07-27 (lead) — F-2 gate unwired, not armed.** The service-secret gate (#37) was
removed from the router mounts because the sole caller is a browser SPA we don't own and can't
change to send the header; arming it would 401 the whole app. The live surface is **unauthenticated
by design** until #43 (Go-forwarded identity). Not a gap to re-raise — it is a recorded posture.
CORS was left at `["*"]` on purpose (tightening it needs the FE origin as config, which we chose
not to set for now).

## 0.8. Knowledge pipeline — test integration (planning, 2026-08-19)

**Nothing has landed in this repo.** This section exists so the in-flight work is tracked in the
usual place; no code, dependency, table or endpoint has been added here yet. Do not read any of it
into REPO_STATUS.

Context and intent: [knowledge_pipeline_context.md](knowledge_pipeline_context.md). Task breakdown,
owners and verified statuses: [KNOWLEDGE_PIPELINE_TODO.md](KNOWLEDGE_PIPELINE_TODO.md).

**Shape.** A six-stage pipeline (parse → free CPU/regex filter → one LLM call per *term cluster* →
deterministic span validation → diff vs. the active glossary → frequency-sorted expert review) that
turns client documents into reviewable knowledge entries, ultimately served over MCP. It splits into
two independently runnable halves — **parsing (Sofhia, MinerU)** and **knowledge extraction
(Rifqi)** — meeting only at a **persisted, versioned parsed-document artifact**. Extraction never
takes a file path; that constraint is the whole point of the split.

**Where it stands (audited 2026-08-19).** A working **prototype** (`kex`, ~2,763 LOC, 21 passing
tests, 8-command CLI, four experiments against real Azure calls) exists outside version control at
`Documents/Dataeyond_catalog/knowledge-pipeline/`. It is **not the product and will not be ported** —
it answered the design questions; extraction v2 gets rebuilt in this repo. What carries forward is
the *evidence* (gold sets, experiment results) and the *tuned constants*, not the code.
Measured on the BUMA standard (9 pages): GLiNER Indonesian recall **0.854** (kill line 0.70, PASS —
this closes the context doc's headline open risk), clustering compression **2.56×** (kill 2.0, PASS),
nano schema-fill precision **0.75** (kill 0.80, **FAIL**), span-validation rejection **1.00** with
0.00 false rejections (kill 0.90, PASS). Cost **$0.0069** for the document.

| # | Task | Owner | Status | Note |
|---|---|---|---|---|
| P1 | Freeze the `kex` prototype (read-only snapshot) | Rifqi | ⬜ | No `.git`. ~2,763 LOC + both gold sets + all four experiment results on one machine. It is an archive, **not** a codebase to develop — v2 is a rebuild here, not a port |
| P2 | Harvest the prototype's tuned constants | Rifqi | ⬜ | Every number that took runs to find, with its reason, **before** v2 code: rapidfuzz 92, fuzzy off below 5 chars (`PA`/`UA` collide), K=3, the evidence weight table, the 1024-token prefix pad (below it, caching does not engage — worth the measured 54% hit rate), label + cue sets, the 4 prompts |
| P3 | Migrate the evidence into `eval/knowledge/` | Rifqi | ⬜ | Gold sets (41 terms + 15 rules) + scorer + experiment runners, following the existing `eval/{intent,help,readiness,planner}/` convention. **Before any v2 pipeline code** — without a standing baseline, "improved" is unfalsifiable |
| S1 | Agree the parsed-artifact schema | Sofhia + Rifqi | 🔄 | Sofhia's draft `contracts.py` (2026-08-19) adopted as the base. Open: document-level envelope (no home for `version`), the section-structure claim, `source_wording` |
| S5 | Keep the seam off the file path | Sofhia + Rifqi | ⬜ | The prototype's boundary was `Parser.parse(path) -> list[Page]` — the shape the plan forbids. Settle **before** the MinerU parser is built against it |
| X* | Build extraction v2 in this repo | Rifqi | ⬜ | 17 stages, `src/query/`-shaped. Design is validated by the prototype; **none of it exists here yet** — see the two-column table in the checklist |
| D2 | Artifact + candidate-entry persistence | Rifqi → Harry | ⬜ | Parsed artifacts, candidate entries, glossary versions + approval audit trail. Go owns dedorch DDL → one consolidated Harry-ready handoff, never DDL from Python (§2.2) |
| R1 | Fix the literal-source-wording violation | Rifqi | ⬜ | A locked decision is violated: the BUMA heading reads "Physical **of** Availability (PA)"; the extracted entry carries "Physical Availability". The discrepancy is meant to reach the expert |
| D3 | Provision a mini deployment | Rifqi | ⛔ | Only nano is provisioned, so all four branches route there — including `rule` and `summary`, whose failure mode is least detectable. Blocks the nano-vs-mini A/B |
| D6 | Expert review of the gold set | Rifqi → Mas Beta | ⛔ | The 41-term / 15-rule gold set is self-bootstrapped and provisional. Now the baseline v2 is measured against |
| D5 | v2 module name | Rifqi | ⬜ | `src/knowledge/` is taken by the existing OCR→pgvector service. Pick a non-colliding name before the first commit |

**Not decided, do not assume:** the v2 module name (D5); whether the admin ingest surface is HTTP
at all (the offline script path is the honest first milestone); and GLiNER + LangExtract ensembling
stays ⏸️ flagged as scope creep. **`kex` is not being ported** — it is a prototype that answered the
design questions; v2 is built here against this repo's patterns.

## 5. Critical path & sequencing

- **Critical path:** ~~#22 (send Harry the `report_inputs` schema)~~ **✅ resolved 2026-07-22** — now **#32** (`message_traceability` + `message_charts` DDL to Harry) and **#31** (non-convergent migration set). HF deploy (#13) for the playground. (#4 ✅, #21 ✅; Harry's #3 no longer blocks us — Python is getattr-tolerant.)
- **Parallelizable now:** #31 + #32 (both are Harry handoffs). (#4 ✅, #11 ✅, #22 ✅ done.)
- **Harry-blocked / coordinated:** #3 (now 🔄, blocks #4), #7 (Go endpoint), #18 (Go state ownership), #24 (contract). **#25 = chat-path migration to `analyses_messages` — a cutover blocker.**
- **Demo gate (playground, #13):** deploy the remove-`problem_statement` work to HF — slow path (#15 ✅)
  and the report path are verified locally, and #16 is resolved (#22 hands Harry the schema). **Keep it
  playground-able.**

## 6. Decisions still open (need the team / Harry / lead)

- ~~`analysis_records`: dedorch-owned vs Python-owned (#16/#14).~~ RESOLVED: Python-owned + renamed **`report_inputs`** (#21 done); Harry's migration creates it (#22).
- ~~Whether `help` is its own endpoint or via `call_agent` (#8).~~ RESOLVED: `help` is a `call_agent` intent (no own endpoint).
- ~~Author display-name source for the report (#19).~~ RESOLVED: Python resolves `users.fullname` (fallback `user_id`); swap to a Go-passed name later if preferred.
- ~~Keep vs drop `chat_history` in the report contract (#17).~~ RESOLVED: never in the contract; report is records-based (analysis_id + user_id only).
- Confirm Go takes over analysis creation + both creation gates (data-first + mandatory fields) (#18).
- **Report input for edit mode** — does Python need the last report content? (#24)
- ~~`report_inputs` write scope — every agent call vs slow-path-only? (#21)~~ RESOLVED: one row per slow-path run (telemetry stays Langfuse).
- **Python history source** — confirm Go's `analysis_message` (#25).
- **`done.chart_count`** — additive field on the SSE `done` event so the FE can skip `GET /charts`
  on chartless turns? (Harry; SPINE_V2_PLAN §4.5.) Until decided the FE fetches unconditionally.
- **W3 re-open timing** (deferred `analyze_*` activation) — Rifqi (deferred at the 2026-07-13 approval).
- **INV-6 relaxation for S1b targeted repair** — team, only after S1a `repair_candidate` telemetry
  shows a meaningful hit-rate (SPINE_V2_PLAN §6).
- **Knowledge pipeline — v2 module name** (§0.8 D5): `src/knowledge/` is already the OCR→pgvector
  service, so extraction v2 needs its own module name before the first commit.
- **Knowledge pipeline — admin ingest surface** (§0.8 D4): whether it is HTTP at all, and if so its
  auth posture. A write surface triggered by an admin is a different risk class from the current
  unauthenticated read-only chat surface (§0.7) — raise it, do not inherit the posture by default.