DEV_PLAN.md

# Data Eyond — Current Development Plan (post 2026-06-24 → 2026-06-30 checkpoints)

**Purpose:** context file for Claude Code sessions working on the current sprint.
**Branch:** `pr/5` · **Snapshot:** 2026-06-30.
**Companion:** [REPO_STATUS.md](REPO_STATUS.md) describes the repo's *current built state*; this file
describes the *in-flight plan* that changes it. The **active sprint is pr/5** ([§0](#0-current-sprint--pr5-observability--endpoint-restructure)); sections §1–§6 are the prior
2026-06-24/25 pivot (now largely ✅), kept for context.

---

## 0. Current sprint — pr/5: Observability + Endpoint Restructure

From the **2026-06-30 checkpoint**. Direction: **Python → generation/AI-only**; Go owns the analysis
lifecycle + data plane. Endpoint contract sent to Harry on 2026-06-30:
[API_ENDPOINTS_RESTRUCTURE.md](API_ENDPOINTS_RESTRUCTURE.md) (chat→v2, tools regroup, observability —
observability marked tentative). REPO_STATUS carries a matching `pr/5` direction banner.

Mentor's task order: **unwire → regroup endpoints → add tools (retrieve-data + observability)**; share
the endpoint contract *before* coding the tools. Status legend: ⬜ not started · 🔄 in progress · ✅ done ·
⛔ blocked · 🔎 verify · ⏸️ deferred.

| Phase | Task | Owner | Status | Notes |
|---|---|---|---|---|
| **P0 — contract** | Draft + send endpoint contract to Harry (chat v2 · tools group · observability) | Rifqi + Sofhia | ✅ | `API_ENDPOINTS_RESTRUCTURE.md` sent 2026-06-30 (before-noon deadline met). Observability section flagged tentative. |
| **1 — unwire** | Unwire `users`(login)/`document`/`room`/`db_client`/`data_catalog`/`analysis` from `main` + Swagger | Sofhia | ✅ | **KM-686**, commit `0b2d678`. Commented, not deleted; `chat`/`report`/`tools` kept mounted. Resolves the analysis-CRUD scope Q — whole `analysis` router unwired (Go owns it). |
| **2 — v2 + regroup** | Create `src/api/v2/` and move the chat pilot there | Rifqi | ✅ | New `src/api/v2/__init__.py` + `src/api/v2/chat.py` (`POST /api/v2/chat/stream`), mounted in `main.py`. Only chat in v2; v1 `/chat/stream` kept mounted until FE moves over. Routes import-verified. |
| **2 — v2 + regroup** | Chat: `room_id` → **`analysis_id`** (request field + handler + history) | Rifqi | ✅ | v2 `ChatRequest{user_id, analysis_id, message}`; reuses warm `ChatHandler` + v1 cache/history helpers; `done` returns `{message_id}` (always minted Python-side, server-authoritative — open-Q #1 resolved in pr/6). Persistence kept transitionally → still ties to #25 (`analyses_messages`); ruff-clean. |
| **2 — v2 + regroup** | Move report under tools → `/api/v1/tools/report` (+ version routes) | Rifqi | ✅ | report router re-prefixed `/api/v1` → `/api/v1/tools` (all 3 routes move together), tag → `Tools`; old `/api/v1/report` gone. Same functionality, new home. Import-verified. |
| **2 — v2 + regroup** | Move help under tools → `POST /api/v1/tools/help` (dedicated endpoint) | Sofhia | ✅ | New `src/api/v1/help.py` (SSE: `sources:[]`→`chunk`→`done{message_id}`) + additive `ChatHandler.stream_help()` (reuses HelpAgent+state+readiness, no router). Generative-only (no persist). **Router `help` intent KEPT** — both paths live by design. message_id always minted Python-side, server-authoritative (open-Q #1 resolved, pr/6). Import-verified. |
| **2 — v2 + regroup** | Tools list → `/api/v1/tools/list` | Sofhia | ✅ | Renamed route `GET /api/v1/tools` → `GET /api/v1/tools/list` ([tools.py:133](src/api/v1/tools.py:133)). |
| **2 — v2 + regroup** | FE: slash menu = `/help` only; report = right-side button | Mentor (FE) | ⬜ | Coordination note, not Python work. |
| **3 — tools + obs** | Finish `help` so it actually **calls** (not just lists) + test | Sofhia | ⬜ | Mentor: help currently only lists tools. Core #2 after chat. |
| **3 — tools + obs** | Traceability **scratchpad** accumulating in the chat agent | Rifqi + Sofhia | ✅ | **KM-691.** `TraceabilityScratchpad` + `TraceabilityToolInvoker` (`src/traceability/`) capture planning / tool I/O / sources during the run; flushed one row before every `done` (all 8 sites; error turns = no row). Renamed observability→traceability (vs. Langfuse). |
| **3 — tools + obs** | Audit `report_inputs` — covers planning + tool I/O + source? add cols / new store | Rifqi | ✅ | **KM-691.** Chose a dedicated store: `message_traceability` = 1 JSONB row per message (Python-owned, like `report_inputs`; DDL run manually against dedorch, handed to Harry). Langfuse kept for engineering. |
| **3 — tools + obs** | Build `GET /api/v1/traceability` (one merged response) | Rifqi | ✅ | **KM-691.** `src/api/v1/traceability.py` → store.get → payload/404. Intent-based source rules (greeting/help/refusals = none; retrieve = required); full planning only on slow path. Contract §7 updated. |
| **3 — tools + obs** | Keep stream **text-only**; traceability is a separate parallel call | Rifqi | ✅ | **KM-691.** No trace data in the SSE stream; the FE fetches `/traceability` on `done`. |
| **3 — tools + obs** | Resolve `message_id` correlation (stream ↔ traceability) with Harry | Rifqi ↔ Harry | ✅ | **RESOLVED (pr/6):** Python is the **sole minter** — `message_id` dropped from the `/api/v2/chat/stream` + `/api/v1/tools/help` request bodies; always minted server-side (server-authoritative, FE-security) and returned on `done`. Any caller-sent `message_id` is ignored. Contract open-Q #1 closed. **Updated 2026-07-09 (pr/13):** id format changed from `msg_<hex>` to a canonical UUID string (`str(uuid.uuid4())`, both mint sites) to mirror Go's `analyses_messages.id` shape — the value is still independently Python-minted (not the real row id), only format-compatible for a future swap. |
| **4 — biz questions** | Get Go folder; confirm `business_questions` in create-analysis (max 5); sync Python | Harry/Mentor → Rifqi | ⬜ | Go currently missing the field ("lagi difixing"). Python already models objective + business_questions. |
| **deferred** | Report formats: PPT (preferred) / PDF / infographic on download | — | ⏸️ | MD is fine for the FE preview stage now. |
| **deferred** | Charts (Plotly→JSON) + images tables | — | ⏸️ | Carried from §4 #26/#27. |

**Next up:** Phase 2 Python work is **done** (chat→v2 `analysis_id`; `help`/`report`/`list` regrouped
under `/api/v1/tools/`). The `message_id` correlation contract is now settled in **pr/6** (Python sole
minter, stream-only). The **Phase 3 traceability build** — scratchpad + `GET /api/v1/traceability`
(contract §7) — is **done (KM-691)**. Next is **Phase 4** (business questions, Go-blocked).

---

## 0.5. pr/13 sprint — agent-quality fixes (2026-07-08 live-test review)

Findings from the scoped live sessions (mining analysis, 2026-07-07/08 traces): the planner
force-mapped absent measures (`pa` aliased as "revenue"), top-N ranked raw rows (duplicate models),
`analyze_trend` collapsed integer months into a single 1970-01 bucket, an invalid grouped IR reached
Postgres, failed retrievals wrote all-null traceability sources, and numeric catalog samples arrive
base64-mangled from Go. Fix tasks (same status legend as §0):

| # | Task | Owner | Status | Note |
|---|---|---|---|---|
| Q1 | IR validator: reject bare selects under `group_by` (planner retry self-corrects) | Rifqi | ✅ | `query/ir/validator.py` |
| Q2 | Planner **infeasible** path: `TaskList.infeasible_reason` + deterministic EN/ID data-gap reply | Rifqi | ✅ | schemas/validator/coordinator/refusals + planner.md "When the catalog cannot answer"; refusal wording → Rifqi to review |
| Q3 | `analyze_trend`: integer year/month handling (epoch-parse bug) | Rifqi | ✅ | `temporal.py` + 5 local tests |
| Q4 | Planner few-shots: top-N (Example G) + infeasible (Example H) + entity-vs-row ranking rule | Rifqi | ✅ | live-tested 2026-07-08: backlog top-3 correct via single-IR group+sum; "best PA performance" correct in-process (avg-per-model, assumption recorded). Stale-server trace was a false alarm |
| Q5 | Catalog numeric `sample_values` base64-decode stopgap (`catalog/sample_decode.py`) | Rifqi | ✅ | self-disabling; **primary fix = Go marshaling — DDL-free handoff to Harry** |
| Q6 | Traceability null-source suppression + `check_data` `-1` row-count hiding | Rifqi | ✅ | `scratchpad.py` / `data_access.py` |
| Q7 | `analyze_merge` two-table combine tool (unblocks "worst A + biggest B" questions) | tool owner | ✅ | tool shipped by Sofia (8abf635, KM-703); planner slice done 2026-07-09: `_validate_data_source` guards `data_right`, two-retrieve→merge few-shot (Example I), planner.md "Two measures per entity" bullet |
| Q8 | Report v2: business-question answer section, unresolved/excluded sections, evidence tables from `results_snapshot`, caveat dedupe, single language | Rifqi/Sofhia | ✅ | done 2026-07-09: still exactly ONE LLM call (extended to also draft `bq_answers`, index-based record refs, deterministic fallback = v1 behavior); evidence tables from table-kind outputs (≤3/record, ≤10 rows, ≤8 cols, `check_*` skipped); reply language via `detect_reply_language` on objective+BQs; verified in-process against live analysis 935a091e |
| Q9 | Record-curation endpoint (`GET …/records` + `exclude_record_ids`) + readiness GET for the FE delta guard | Rifqi ↔ FE | ✅ | done 2026-07-09: `GET /tools/report/{analysis_id}/records` + `/readiness` (registered before `/{version}` — int-coercion route-order trap), `exclude_record_ids` on POST; contract updated same change; FE wiring pending (Rifqi → FE) |
| Q10 | Traceability `data_used` layer — resolve IR ids → real names for the FE (users couldn't map `c_…`/`t_…` ids back to their data; aggregate aliases like `total_revenue` looked like real columns) | Rifqi ↔ FE | ✅ | done 2026-07-13 (pr/15): new `src/traceability/resolve.py` builds `data_used[]` (real source/table/column names; joins; plain-language filters; `columns_read` vs `output_columns` with `computed`+`formula`), `tool_calls[].summary`, `sources[]` gains `source_name`+all tables; **ids kept but machine-only (FE must not render)**; deterministic no-LLM, never-throw; catalog threaded to the scratchpad at the composition root. Contract + `TRACEABILITY_FE_HANDOFF.md` updated; FE wiring pending (Rifqi → FE). Additive/non-breaking |
| Q11 | IR wart: `OrderByClause.column_id` may hold a SELECT **alias** (a computed output), not a catalog column_id | Rifqi | 🔎 | surfaced by Q10 — the resolver tolerates it (`kind: "computed"` fallback), but the IR field name is misleading. Consider an explicit `by_alias` field or renaming. Low priority; no functional bug |

## 0.6. pr/16 sprint — Spine v2: W2 charts + W1 checkpoint (SPINE_V2_PLAN, approved 2026-07-13)

Scope approved 2026-07-13 (Rifqi + Sofia): build W2 (`render_chart` + chart store + `GET /charts`
+ planner viz slice) and W1 (S1a quality checkpoint). **W3 (activate deferred `analyze_*`) deferred
at approval — do not start until Rifqi re-opens.** W4 (S1b repair) stays gated on INV-6 sign-off +
S1a telemetry. Design + handoff source: `SPINE_V2_PLAN.md`.

| # | Task | Owner | Status | Note |
|---|---|---|---|---|
| V1 | `render_chart` tool slice: `visualization.py` (Plotly-JSON `dataeyond.chart.v1` envelope, no plotly dep) + `ToolOutput.kind` `"chart"` + registry + invoker | Rifqi (Sofia signed off on the tool-layer edit) | ✅ | done 2026-07-13; deterministic spec builder (bar/line/pie/scatter, fixed style preset), traceability scratchpad summarizes chart outputs compactly (point_count, not the raw arrays) |
| V2 | Chart store + API: `MessageChartRow` (`message_charts`) + `src/charts/store.py` (never-throw save) + write site in `_run_slow_path` + `GET /api/v1/charts` + contract §charts | Rifqi | ✅ | done 2026-07-13; empty list = valid 200; `done` event unchanged (no `chart_count` — open, Harry); FE fetches unconditionally on `done` |
| V3 | Planner viz slice: recipe table + "Charts only on explicit ask" rule (planner.md), Example J (viz tail) + Example K (viz-infeasible), validator Check 10 (`render_chart.data` must be table-kind), assembler chart one-liner guard | Rifqi | ✅ | done 2026-07-13; behavioral matrix verified in-process (real LLM): explicit ask EN/ID → chart tail; plain question → no chart; absent dimension → infeasible (Example K + prompt guard added after the first smoke force-mapped `status` AS "region") |
| V4 | W1 S1a quality checkpoint: `slow_path/checkpoint.py` (CK1–CK6) + `RunAssessment` schemas + coordinator call site + "Execution assessment" block in the assembler input + `refusals.run_failure_message` | Rifqi | ✅ | done 2026-07-13; 13 local tests (one per CK rule + never-throw + prompt + coordinator); CK1 all-failed → deterministic honest failure, **no assembler call**; clean run renders nothing (zero behavior change); every flag logs `repair_candidate` (S1b evidence) |
| V5 | `message_charts` DDL: run manually against dedorch (block for the live e2e chart test), then hand the schema to Harry for the dedorch migration | Rifqi → Harry | ✅ | Rifqi ran the DDL 2026-07-13; **live e2e ALL PASS** same day (real v2 endpoint: viz turn → chart row keyed by `done` message_id → `GET /charts` valid v1 envelope; chartless → 200 empty; injected `render_chart` failure → answer streams as a table, no row). **Remaining: send Harry the migration handoff** (schema + contract §charts) |
| V6 | Restore the `eval.chat_sim` harness — `eval/chat_sim/*.py` is missing from disk AND git (only `__pycache__` remains); §7B prompt gate can't run | Rifqi | ✅ | restored by Rifqi 2026-07-13 (accidental delete). ⚠️ its hard-coded `DEFAULT_USER_ID`/`TITANIC_SOURCE_ID` are stale (that user has no catalog; the Titanic blobs are gone) — update the constants before the next full run |
| V7 | Local `.env` lagged Go's Supabase-S3 data plane: `storage_provider=azure_blob` + empty `supabase_s3_*` made EVERY local tabular retrieve fail `BlobNotFound` | Rifqi | ✅ | found during the e2e (masked as "data not available" by the honest-degrade path); Rifqi set the six values 2026-07-13. Gotcha documented in REPO_STATUS §13 |
| V8 | Lead review of `GET /charts`: lookup by `message_id` alone + tri-state response marker (`status: success \| empty \| not_found` + `message`) instead of a bare list | Rifqi (lead ask) | ✅ | done 2026-07-14; `not_found` vs `empty` decided against the turn's traceability row (PK lookup); always HTTP 200; contract §charts updated. ⚠️ additive DDL for the new lookup: `CREATE INDEX IF NOT EXISTS idx_message_charts_message ON message_charts (message_id);` (run manually + include in Harry's migration). Traceability GET still takes both params — aligning it is open |
| V9 | Report chart embedding: `AnalysisReport.charts` (verbatim envelopes per record) + `## EDA` section with ` ```plotly ` fences (content = the **full v1 envelope**, pretty-printed — the FE hook's verified shape); `has_successful_analysis` extended so a successful `render_chart` counts (chart-only sessions satisfy the report floor) | Rifqi | ✅ | done 2026-07-14; first cut emitted bare `{data, layout}` — FE test showed the hook parses the full envelope, fixed same day; live reports v3 (wrong fence) → **v4 (correct)** for analysis `7be50846…` (3 charts embedded); suite **381 passed, 7 skipped** (+5 chart-embed tests; same 2 pre-existing failures) |

Full-suite evidence for this sprint: **376 passed, 7 skipped** (+13 new checkpoint tests; the 2
failures — `test_chat_handler::test_structured_flow_runs_slow_path`,
`test_reader::test_structured_read_falls_back_to_user_scope_when_no_analysis_row` — reproduce at
HEAD before this diff, i.e. pre-existing). Ruff clean on all touched paths; `import main` OK.

## 1. The direction change (locked decisions from 2026-06-24)

1. **"Problem statement" is replaced by two user-entered fields: `objective` + `business_questions`.**
   User fills them at onboarding; **both mandatory to submit; NO agent validation.**
2. The **gate (`problem_validated`) and the `problem_statement` skill/intent are removed** (comment out, don't delete).
3. **Report is records-based** (reads persisted `AnalysisRecord`s) — **decided and pushed** (KM-674).
   It is formal markdown: title, date, "generated by {user}", objective, business questions,
   findings, insights. **NOT gated** on whether business questions were answered.
4. **`owner_id` → `user_id`** everywhere (Harry mirrors in dedorch/Go).
5. **State writes go through a request to Go**, not direct Python DB writes.
6. **FE-callable surface = 4 endpoints:** `call_agent` (chat/stream), `list_skills` (`GET /tools`),
   **skill: help**, **skill: report**. `problem_statement` removed; `check_data` not FE-facing from
   Python (Go provides it); analysis CRUD not needed from FE (comment, don't delete).
7. Deliverables for Harry: (a) API endpoint doc (MD); (b) full Python project doc (MD → PDF/Word BRD).
8. Integration tested via Swagger `/docs` on the HF Python build (simulating FE manually). Target ~Wed.

## 1.5. 2026-06-25 checkpoint deltas

Confirms the 2026-06-24 direction and adds these concrete changes (folded into §4 as tasks 21–28):

1. **Rename `analysis_records` → `report_inputs`** (DONE #21) — names the table by purpose (the rows
   report generation reads); avoids clashing with Go's `analyses_messages` and with Langfuse
   observability. **Stays Python-owned**; finalized schema handed to Harry so his dedorch migration
   creates it post-`SKIP_INIT_DB` (#22, resolves #16). Write scope = **one row per slow-path analysis
   run** (decided — not per-agent-call telemetry; that stays Langfuse).
2. **`analyses` table (Go) — `status`, `data_bind` + `data_bind_version`, `report_collection`** (id+version).
   **Verified 2026-06-25: these + `user_id` are ALREADY present in dedorch `analyses`.** Plus Harry drops
   the duplicate/wrong singular `analysis` table. (→ #3)
3. **`analyses_messages` (Go) = the analysis chat room** (user Q + agent A) — replaces the now-**deprecated**
   `chat_messages`/`rooms`; Python's chat read/write must migrate here before cutover. (→ #25)
4. **Reports: Go owns ALL writes.** Report stays a **skill** (no router intent): FE → Go → Python;
   Python only returns content. Input = the records table (now `agent_observability`); edit-mode may
   also need the last report. (→ #7/#18/#24)
5. **Markdown minimum now:** tables, **bold**, *italic*, horizontal separators — optimize that before
   anything fancier. (→ #23)
6. **Deferred:** charts (prefer **Plotly→JSON** in a future `chart` table over matplotlib PNGs) and
   images (image table keyed by analysis/message/report + originals in a bucket). (→ #26/#27)
7. **Near-term:** the remove-`problem_statement` work isn't on HF yet → **PR + deploy + test in the
   playground** (#13). Harry stabilizes Go ~Fri; FE manual testing ~Mon. **Keep it playground-able.**
8. **UI research** (no dedicated UI person): new-analysis form (title/objective/business_questions),
   knowledge menu (user-level vs analysis-level binding), report artifacts panel + version selector;
   interview + old analysis UI removed. (→ #28)

## 2. What is already done (KM-674, pushed on `pr/4`)

Report layer adapted to the new goal shape:
- `report/schemas.py::ProblemStatement` → `objective: str` + `business_questions: list[str]`
  (old `target_value`/`scope`/`metric_direction`/`target_metric` dropped). Class name kept for now
  (rename to `ReportGoal` once the upstream AnalysisState rename lands).
- `report/generator.py` renders **Objective** + numbered **Business Questions** + a
  **"generated by {user}"** line.
- `api/v1/report.py::_problem_statement_from` is **tolerant**: prefers new `objective` /
  `business_questions` from state, falls back to legacy `problem_statement` — works before AND after
  Harry's migration.
- `config/prompts/report_summary.md` updated to objective + business questions.
- Report stays **records-based**; the floor gate (`problem_validated`) was deliberately left for task #2.

**This tolerant-migration pattern (getattr fallback) is the model for tasks #2 and #4.**

## 3. Assessment — gaps & contradictions to resolve before building

These came out of reviewing the plan against the actual code. They are folded into the task table (§4) as tasks 15–19.

- **G1 (→ task 15). Records-based reports need the slow path ON.** `AnalysisRecord`s persist only in
  `chat_handler._run_slow_path`, which runs only when `ENABLE_SLOW_PATH=true`. Default is off → no
  records → `POST /report` 409s. The Swagger demo can't show a non-empty report unless slow path is
  flipped on and a `structured_flow` question is run first. `BusinessContext` is still a stub but the
  slow path runs fine on it.
- **G2 (→ task 16). `analysis_records` ownership is now required and collides with `SKIP_INIT_DB`.**
  It's created today by Python `create_all` (`db/postgres/init_db.py`), is in no dedorch/Go migration,
  and after the dedorch cutover (`SKIP_INIT_DB=true`) Python stops running `create_all` → the table
  won't exist → reports break. Decide: dedorch migration (Harry) OR a Python carve-out that creates
  just this one table even under `SKIP_INIT_DB`. *(Resolved 2026-06-25 — see §1.5.1 / #16 / #22.)*
- **G3 (→ task 17). `chat_history` in the report contract is vestigial.** Records-based generation
  reads records by `analysis_id`; it never uses chat history. Drop `chat_history` from the report
  skill contract, or mark it reserved/unused.
- **G4 (→ task 4 note). Make #2/#4 tolerant of both state shapes.** If Harry drops
  `problem_validated`/`owner_id` from dedorch before Python stops reading them, Python's gate +
  state_store break. Use the same `getattr` tolerance KM-674 used. The `owner_id`→`user_id` rename
  also touches `api/v1/analysis.py` (`_serialize_state`, `list_analyses`, `get_analysis`), not just
  the model + state_store.
- **G5 (→ task 18). "State writes via Go" is bigger than `report_id`.** Python still writes state in
  `/analysis/create` (state + room + bindings, plus the data-first gate and soon the mandatory-field
  check) and in `state_store.ensure` per turn. If creation moves to Go (consistent with commenting
  analysis CRUD), then Go owns ALL state writes + both creation gates, and Python's `ensure` must
  become a **read-only get** (Go must guarantee the row exists before any chat turn).
- **G6 (smaller).**
  - Removing `problem_statement` (task 1) means neutering it in 4 places: the `Intent` literal
    (`agents/orchestration.py`), the router prompt (`config/prompts/intent_router.md`), the handler,
    and the gate's redirect *target*. Do it with task 2.
  - "generated by {user}" currently prints the raw `user_id`; a formal report wants a name — source
    from `users.fullname` or have Go pass a display name (task 19).
  - The meeting's outline (background / EDA / insights) isn't fully in the renderer; map those
    sections onto the record fields deliberately (task 5 follow-up).
  - The full project doc (task 11) should reuse [REPO_STATUS.md](REPO_STATUS.md), not restart.

## 4. Task table

Status legend: ⬜ not started · 🔄 in progress · ✅ done · ⛔ blocked · 🔎 verify · ⏸️ deferred.

| # | Task | Owner | Status | Note |
|---|---|---|---|---|
| 1 | Comment out `problem_statement` skill **+ `Intent` literal + router prompt + gate redirect target**; remove `/problem-statement` from `list_tools` | Rifqi | ✅ | Done 2026-06-25 (one commit w/ #2). Unwired in `orchestration.py`, `intent_router.md`, `chat_handler.py`, `tools.py`; `problem_statement.py` kept intact |
| 2 | Drop `problem_validated`: gate neutered; `is_report_ready`/`report_floor` → **≥1 completed analysis** only, no-LLM | Rifqi | ✅ | Done 2026-06-25. `gate.py` no-op, gate call site commented in `chat_handler.py`, `report_floor` drops the goal check. Tests updated (`test_gate`/`test_chat_handler`/`test_readiness`). Suite: **284 passed, 7 skipped**; ruff clean |
| 3 | dedorch `analyses` migration: drop `problem_statement`/`problem_validated`, add `objective` + `business_questions` | Harry | 🔄 | **Verified dedorch 2026-06-25:** `analyses` (plural) ALREADY has `user_id` + `status` + `data_bind` + `data_bind_version` + `report_collection` → those parts done. **Remaining:** drop `problem_statement`/`problem_validated` + add `objective`/`business_questions`. Singular `analysis` = deprecated duplicate to drop |
| 4 | Update Python `analyses` model + `state_store` + `analysis.py` to match dedorch; `owner_id`→`user_id` | Rifqi/Sofhia | ✅ | Done 2026-06-26. `owner_id`→`user_id` + added `status`/`data_bind`/`data_bind_version`/`report_collection` (DB-only, not in the `AnalysisState` pydantic) across `models.py`/`gate.py`/`state_store.py`/`analysis.py` + 3 local tests; also `report_inputs` `id`/`analysis_id` → `uuid`. Kept `problem_statement`/`problem_validated`; `objective`/`business_questions` wait on Harry's #3. Suite **284 passed** |
| 5 | Report generator → `objective`+`business_questions`, "generated by {user}", formal outline | Sofhia | ✅ | Goal-shape (KM-674) + author name (#19) + outline (KM-680): Objective → Business Questions → Executive Summary → Key Findings → EDA → Notes & Limitations → How This Was Analyzed |
| 6 | Report skill input contract: `analysis_id` + `user_id` (no `chat_history`) | Sofhia/Rifqi | ✅ | No-op: `POST /report` already takes only analysis_id + user_id (records-based). Documented in API_ENDPOINTS.md §5. *(Edit-mode input revisited in #24.)* |
| 7 | `report_id` state update via request to Go, not direct DB | Sofhia + Harry | ⬜ | Needs Go endpoint. **Checkpoint:** Go owns ALL `reports` writes; Python stops any direct insert/update and only returns content; report stays a **skill** (no intent). See #18 |
| 8 | Expose/confirm 4 FE endpoints; comment `check_data` + analysis CRUD | Sofhia | ✅ | KM-678: `list_tools` trimmed to `/help` + `/report` (analytics/check/retrieve commented in the **menu**). `help` confirmed as a `call_agent` intent — no own endpoint. Analysis CRUD endpoint left **registered**: "comment the rest" was about the FE slash menu, not killing HTTP routes Go needs |
| 9 | Verify `analysis_id` in `call_agent` contract | Sofhia | ✅ | Verified: no separate field — carried as `room_id` (`analysis_id == room_id`), per REPO_STATUS §4/§11. Action for Go: send the id as `room_id` |
| 10 | API endpoint doc (MD), 4 endpoints, for Go integration | Rifqi + Sofhia | ✅ | Done 2026-06-25 — `API_ENDPOINTS.md` (repo root). 4 FE surfaces with request/response **examples** (chat SSE transcript, report 201/409 JSON, version list), schemas, §9 full 32-route inventory + task-8 reading |
| 11 | Full Python project doc (MD → PDF/Word BRD) | Rifqi | ✅ | Done 2026-06-26 — `PROJECT_BRD.md` (repo root): purpose/context, FR-1..9 capabilities, lifecycle, architecture, data model, API (→ API_ENDPOINTS), NFRs, integrations, open items. Reuses REPO_STATUS/API_ENDPOINTS; convert to PDF/Word for distribution |
| 12 | Reconcile/open the `list_tools` PR cleanly (stacked commits) | Rifqi | ✅ | N/A — we develop directly on the single active branch `pr/4` (KM-652 + KM-678 already stacked there); no separate PR to reconcile |
| 13 | Deploy HF Python build (remove-`problem_statement` work) → test 4 endpoints via Swagger / playground | Sofhia + Harry | 🔄 | **Unblocked (#15 ✅).** Remove-PS work is on `pr/4` but **not on HF `main` yet** → PR + deploy, then manual test. Harry stabilizes Go ~Fri; FE testing ~Mon |
| 14 | `analysis_records` home | Rifqi + Sofhia + lead | ✅ | **Resolved 2026-06-25:** stays Python-owned, **renamed** (→ #21); schema handed to Harry so the dedorch migration creates it post-cutover (→ #22). Not moved to Go |
| 15 | Flip `ENABLE_SLOW_PATH=true` + verify an `AnalysisRecord` persists from a `structured_flow` question | Rifqi | ✅ | Verified locally 2026-06-25 (in-process). structured_flow on Titanic.csv → 3-task plan `check_data→retrieve_data→analyze_aggregate` (all success) → AnalysisRecord persisted (substantive) → `report_floor` pass → report generates (201). HF env-flip + Swagger run folds into #13 |
| 16 | Decide `analysis_records` creation under `SKIP_INIT_DB` | Rifqi + Harry | ✅ | **Resolved 2026-06-25:** Python defines it; **Harry's dedorch migration creates it** on env-move (Python still creates locally meanwhile) → exists post-cutover. Execution = #22 |
| 17 | Reconcile report contract with records-based: remove/flag `chat_history` | Sofhia/Rifqi | ✅ | Nothing to remove — `chat_history` was never in the report contract/code (only in help.md). Confirmed via grep; API_ENDPOINTS.md §5 documents the clean contract |
| 18 | Confirm Go owns ALL analysis-state writes + both creation gates; make Python `state_store.ensure` read-only | Rifqi + Harry | ⬜ | **Confirmed by 2026-06-25 checkpoint** (Python read-only; Go owns writes + new tables). Execution pending Go endpoints |
| 19 | Decide report author display-name source (`users.fullname` vs Go-passed name) | Sofhia | ✅ | Done 2026-06-25. `AnalysisReport.user_name`; `generator` renders `user_name or user_id`; `api/v1/report.py::_resolve_user_name` reads `users.fullname` never-throw (fallback `user_id`). Decided: resolve in Python (unblocked); swap to Go-passed name later if preferred |
| 20 | **Help handoff:** update `handlers/help.py` + `help.md` — drop the `problem_validated` tier + `define_problem_statement` action (the skill it points at is gone as of #1) | Sofhia | ✅ | Done 2026-06-25. `help.py`: actions = `ask_analysis_question` (always) + `generate_report` (if ready); renders objective/business_questions (getattr-tolerant). `help.md` v1→v2: 3 tiers, no `/problem_statement`, `/generate report`→`/report`. Local test_help updated → 11 pass |
| 21 | Rename `analysis_records` → **`report_inputs`** (table, ORM `ReportInputRow`, store `*ReportInputStore`) | Rifqi | ✅ | Done 2026-06-26. `sed` rename across 9 files; Pydantic `AnalysisRecord` kept; columns stay String (pure rename — uuid+FK is the #22 Harry schema). Name `report_inputs` (purpose; avoids Langfuse/`analyses_messages` clash). Write scope = one row per slow-path run. Suite **284 passed** |
| 22 | Finalize `report_inputs` schema → hand to Harry for the dedorch migration | Rifqi → Harry | 🔄 | **DDL ready** (uuid `id`/`analysis_id` + FK→`analyses(id)`; `user_id`/`plan_id` text; `data` jsonb = serialized `AnalysisRecord`, shape documented). dedorch has empty `analysis_records` → rename. Resolves #16. **Action: send Harry the DDL + `data` shape** |
| 23 | Report markdown formatting: tables, **bold**, *italic*, horizontal separators | Sofhia | ✅ | Done 2026-06-25. Added `---` separators between header + each section in `_render_markdown`. Tables (EDA) / bold (method labels) / italic (meta + citations) already emitted. Relaxed `report_summary.md` to allow inline `**bold**`/`*italic*` for emphasis (kept no-headings/no-bullets so it doesn't duplicate the section structure / Key Findings). Compile + ruff clean |
| 24 | Clarify report input contract: records table (+ `last_report` for edit mode?) | Rifqi/Sofhia ↔ Harry | ⬜ new | Edit-mode input left open at the checkpoint |
| 25 | Migrate Python chat path to Go `analyses_messages` (+ `analyses`) | Rifqi ↔ Harry | ✅ | Done 2026-07-02. Read path already on `analyses_messages` (commit `0066161`). This change makes Python **read-only**: removed the `save_messages` calls from `/api/v2/chat/stream` so **Go is the sole writer** — fixes the double-write both Go+Python were producing. `load_history` still reads `analyses_messages`. v1 `/chat/stream` is unwired so left untouched |
| 26 | **Charts (DEFERRED):** store Plotly JSON in a future `chart` table (not matplotlib PNG) | — | 🔎 | **Landed 2026-07-13 (pr/16, §0.6 V1–V3):** `render_chart` + `message_charts` + `GET /api/v1/charts`, Plotly JSON as decided. 🔎 pending the dedorch DDL run (V5) + live e2e |
| 27 | **Images (DEFERRED):** image table (id, analysis_id, msg/report ref, order) + originals in a bucket | — | ⏸️ | Maintenance-heavy; parked |
| 28 | **UI research** (FE): new-analysis form, knowledge menu (user vs analysis level), report artifacts + version selector | Team | ⬜ new | No dedicated UI person; interview + old analysis UI removed |
| 29 | **LLM env quad rename `__4o` → `__54m`** + set the four `azureai__*__54m` secrets on the HF Space | Rifqi | 🔄 | Code done 2026-07-14 (pr/17): settings quad + 9 call sites + 2 eval runners renamed; deployment is `gpt-5.4-mini`. Suite **381 passed, 2 failed (both pre-existing), 7 skipped** — same as baseline; ruff clean on touched files; `import main` 0. **Hard rename — no `__4o` fallback**, so HF fails loudly on the first LLM call until the four `__54m` secrets are set. Root cause it fixes: HF silently ran GPT-4o while local ran 5.4-mini → same question, same catalog, planner hallucinated catalog ids on HF only. **Action: set HF secrets, redeploy, re-run the failing question.** Local `.env` still carries the now-dead `__4o` keys — safe to delete |

## 5. Critical path & sequencing

- **Critical path:** #22 (send Harry the `report_inputs` schema). HF deploy (#13) for the playground. (#4 ✅, #21 ✅; Harry's #3 no longer blocks us — Python is getattr-tolerant.)
- **Parallelizable now:** #22 (handoff). (#4 ✅, #11 ✅ done.)
- **Harry-blocked / coordinated:** #3 (now 🔄, blocks #4), #7 (Go endpoint), #18 (Go state ownership), #24 (contract). **#25 = chat-path migration to `analyses_messages` — a cutover blocker.**
- **Demo gate (playground, #13):** deploy the remove-`problem_statement` work to HF — slow path (#15 ✅)
  and the report path are verified locally, and #16 is resolved (#22 hands Harry the schema). **Keep it
  playground-able.**

## 6. Decisions still open (need the team / Harry / lead)

- ~~`analysis_records`: dedorch-owned vs Python-owned (#16/#14).~~ RESOLVED: Python-owned + renamed **`report_inputs`** (#21 done); Harry's migration creates it (#22).
- ~~Whether `help` is its own endpoint or via `call_agent` (#8).~~ RESOLVED: `help` is a `call_agent` intent (no own endpoint).
- ~~Author display-name source for the report (#19).~~ RESOLVED: Python resolves `users.fullname` (fallback `user_id`); swap to a Go-passed name later if preferred.
- ~~Keep vs drop `chat_history` in the report contract (#17).~~ RESOLVED: never in the contract; report is records-based (analysis_id + user_id only).
- Confirm Go takes over analysis creation + both creation gates (data-first + mandatory fields) (#18).
- **Report input for edit mode** — does Python need the last report content? (#24)
- ~~`report_inputs` write scope — every agent call vs slow-path-only? (#21)~~ RESOLVED: one row per slow-path run (telemetry stays Langfuse).
- **Python history source** — confirm Go's `analysis_message` (#25).
- **`done.chart_count`** — additive field on the SSE `done` event so the FE can skip `GET /charts`
  on chartless turns? (Harry; SPINE_V2_PLAN §4.5.) Until decided the FE fetches unconditionally.
- **W3 re-open timing** (deferred `analyze_*` activation) — Rifqi (deferred at the 2026-07-13 approval).
- **INV-6 relaxation for S1b targeted repair** — team, only after S1a `repair_candidate` telemetry
  shows a meaningful hit-rate (SPINE_V2_PLAN §6).