DEV_PLAN.md

Data Eyond — Current Development Plan (post 2026-06-24 → 2026-06-30 checkpoints)

Purpose: context file for Claude Code sessions working on the current sprint. Branch: pr/5 · Snapshot: 2026-06-30. Companion: REPO_STATUS.md describes the repo's current built state; this file describes the in-flight plan that changes it. The active sprint is pr/5 (§0); sections §1–§6 are the prior 2026-06-24/25 pivot (now largely ✅), kept for context.

---

0. Current sprint — pr/5: Observability + Endpoint Restructure

From the 2026-06-30 checkpoint. Direction: Python → generation/AI-only; Go owns the analysis lifecycle + data plane. Endpoint contract sent to Harry on 2026-06-30: API_ENDPOINTS_RESTRUCTURE.md (chat→v2, tools regroup, observability — observability marked tentative). REPO_STATUS carries a matching pr/5 direction banner.

Mentor's task order: unwire → regroup endpoints → add tools (retrieve-data + observability); share the endpoint contract before coding the tools. Status legend: ⬜ not started · 🔄 in progress · ✅ done · ⛔ blocked · 🔎 verify · ⏸️ deferred.

Phase	Task	Owner	Status	Notes
P0 — contract	Draft + send endpoint contract to Harry (chat v2 · tools group · observability)	Rifqi + Sofhia	✅	API_ENDPOINTS_RESTRUCTURE.md sent 2026-06-30 (before-noon deadline met). Observability section flagged tentative.
1 — unwire	Unwire users(login)/document/room/db_client/data_catalog/analysis from main + Swagger	Sofhia	✅	KM-686, commit 0b2d678. Commented, not deleted; chat/report/tools kept mounted. Resolves the analysis-CRUD scope Q — whole analysis router unwired (Go owns it).
2 — v2 + regroup	Create src/api/v2/ and move the chat pilot there	Rifqi	✅	New src/api/v2/__init__.py + src/api/v2/chat.py (POST /api/v2/chat/stream), mounted in main.py. Only chat in v2; v1 /chat/stream kept mounted until FE moves over. Routes import-verified.
2 — v2 + regroup	Chat: room_id → analysis_id (request field + handler + history)	Rifqi	✅	v2 ChatRequest{user_id, analysis_id, message}; reuses warm ChatHandler + v1 cache/history helpers; done returns {message_id} (always minted Python-side, server-authoritative — open-Q #1 resolved in pr/6). Persistence kept transitionally → still ties to #25 (analyses_messages); ruff-clean.
2 — v2 + regroup	Move report under tools → /api/v1/tools/report (+ version routes)	Rifqi	✅	report router re-prefixed /api/v1 → /api/v1/tools (all 3 routes move together), tag → Tools; old /api/v1/report gone. Same functionality, new home. Import-verified.
2 — v2 + regroup	Move help under tools → POST /api/v1/tools/help (dedicated endpoint)	Sofhia	✅	New src/api/v1/help.py (SSE: sources:[]→chunk→done{message_id}) + additive ChatHandler.stream_help() (reuses HelpAgent+state+readiness, no router). Generative-only (no persist). Router help intent KEPT — both paths live by design. message_id always minted Python-side, server-authoritative (open-Q #1 resolved, pr/6). Import-verified.
2 — v2 + regroup	Tools list → /api/v1/tools/list	Sofhia	✅	Renamed route GET /api/v1/tools → GET /api/v1/tools/list (tools.py:133).
2 — v2 + regroup	FE: slash menu = /help only; report = right-side button	Mentor (FE)	⬜	Coordination note, not Python work.
3 — tools + obs	Finish help so it actually calls (not just lists) + test	Sofhia	⬜	Mentor: help currently only lists tools. Core #2 after chat.
3 — tools + obs	Traceability scratchpad accumulating in the chat agent	Rifqi + Sofhia	✅	KM-691. TraceabilityScratchpad + TraceabilityToolInvoker (src/traceability/) capture planning / tool I/O / sources during the run; flushed one row before every done (all 8 sites; error turns = no row). Renamed observability→traceability (vs. Langfuse).
3 — tools + obs	Audit report_inputs — covers planning + tool I/O + source? add cols / new store	Rifqi	✅	KM-691. Chose a dedicated store: message_traceability = 1 JSONB row per message (Python-owned, like report_inputs; DDL run manually against dedorch, handed to Harry). Langfuse kept for engineering.
3 — tools + obs	Build GET /api/v1/traceability (one merged response)	Rifqi	✅	KM-691. src/api/v1/traceability.py → store.get → payload/404. Intent-based source rules (greeting/help/refusals = none; retrieve = required); full planning only on slow path. Contract §7 updated.
3 — tools + obs	Keep stream text-only; traceability is a separate parallel call	Rifqi	✅	KM-691. No trace data in the SSE stream; the FE fetches /traceability on done.
3 — tools + obs	Resolve message_id correlation (stream ↔ traceability) with Harry	Rifqi ↔ Harry	✅	RESOLVED (pr/6): Python is the sole minter — message_id dropped from the /api/v2/chat/stream + /api/v1/tools/help request bodies; always minted server-side (server-authoritative, FE-security) and returned on done. Any caller-sent message_id is ignored. Contract open-Q #1 closed.
4 — biz questions	Get Go folder; confirm business_questions in create-analysis (max 5); sync Python	Harry/Mentor → Rifqi	⬜	Go currently missing the field ("lagi difixing"). Python already models objective + business_questions.
deferred	Report formats: PPT (preferred) / PDF / infographic on download	—	⏸️	MD is fine for the FE preview stage now.
deferred	Charts (Plotly→JSON) + images tables	—	⏸️	Carried from §4 #26/#27.

Next up: Phase 2 Python work is done (chat→v2 analysis_id; help/report/list regrouped under /api/v1/tools/). The message_id correlation contract is now settled in pr/6 (Python sole minter, stream-only). The Phase 3 traceability build — scratchpad + GET /api/v1/traceability (contract §7) — is done (KM-691). Next is Phase 4 (business questions, Go-blocked).

---

0.5. pr/13 sprint — agent-quality fixes (2026-07-08 live-test review)

Findings from the scoped live sessions (mining analysis, 2026-07-07/08 traces): the planner force-mapped absent measures (pa aliased as "revenue"), top-N ranked raw rows (duplicate models), analyze_trend collapsed integer months into a single 1970-01 bucket, an invalid grouped IR reached Postgres, failed retrievals wrote all-null traceability sources, and numeric catalog samples arrive base64-mangled from Go. Fix tasks (same status legend as §0):

#	Task	Owner	Status	Note
Q1	IR validator: reject bare selects under group_by (planner retry self-corrects)	Rifqi	✅	query/ir/validator.py
Q2	Planner infeasible path: TaskList.infeasible_reason + deterministic EN/ID data-gap reply	Rifqi	✅	schemas/validator/coordinator/refusals + planner.md "When the catalog cannot answer"; refusal wording → Rifqi to review
Q3	analyze_trend: integer year/month handling (epoch-parse bug)	Rifqi	✅	temporal.py + 5 local tests
Q4	Planner few-shots: top-N (Example G) + infeasible (Example H) + entity-vs-row ranking rule	Rifqi	✅	live-tested 2026-07-08: backlog top-3 correct via single-IR group+sum; "best PA performance" correct in-process (avg-per-model, assumption recorded). Stale-server trace was a false alarm
Q5	Catalog numeric sample_values base64-decode stopgap (catalog/sample_decode.py)	Rifqi	✅	self-disabling; primary fix = Go marshaling — DDL-free handoff to Harry
Q6	Traceability null-source suppression + check_data -1 row-count hiding	Rifqi	✅	scratchpad.py / data_access.py
Q7	analyze_merge two-table combine tool (unblocks "worst A + biggest B" questions)	tool owner	⬜	flagged out of this sprint; request brief sent by Rifqi
Q8	Report v2: business-question answer section, unresolved/excluded sections, evidence tables from results_snapshot, caveat dedupe, single language	Rifqi/Sofhia	⬜	next up; adds one LLM call (button-triggered); new prompt → eval per §7B
Q9	Record-curation endpoint (GET …/records + exclude_record_ids) + readiness GET for the FE delta guard	Rifqi ↔ FE	⬜	contract addition → API_CONTRACT_BE_PYTHON.md

1. The direction change (locked decisions from 2026-06-24)

1. "Problem statement" is replaced by two user-entered fields: objective + business_questions. User fills them at onboarding; both mandatory to submit; NO agent validation.
2. The gate (problem_validated) and the problem_statement skill/intent are removed (comment out, don't delete).
3. Report is records-based (reads persisted AnalysisRecords) — decided and pushed (KM-674). It is formal markdown: title, date, "generated by {user}", objective, business questions, findings, insights. NOT gated on whether business questions were answered.
4. owner_id → user_id everywhere (Harry mirrors in dedorch/Go).
5. State writes go through a request to Go, not direct Python DB writes.
6. FE-callable surface = 4 endpoints: call_agent (chat/stream), list_skills (GET /tools), skill: help, skill: report. problem_statement removed; check_data not FE-facing from Python (Go provides it); analysis CRUD not needed from FE (comment, don't delete).
7. Deliverables for Harry: (a) API endpoint doc (MD); (b) full Python project doc (MD → PDF/Word BRD).
8. Integration tested via Swagger /docs on the HF Python build (simulating FE manually). Target ~Wed.

1.5. 2026-06-25 checkpoint deltas

Confirms the 2026-06-24 direction and adds these concrete changes (folded into §4 as tasks 21–28):

1. Rename analysis_records → report_inputs (DONE #21) — names the table by purpose (the rows report generation reads); avoids clashing with Go's analyses_messages and with Langfuse observability. Stays Python-owned; finalized schema handed to Harry so his dedorch migration creates it post-SKIP_INIT_DB (#22, resolves #16). Write scope = one row per slow-path analysis run (decided — not per-agent-call telemetry; that stays Langfuse).
2. analyses table (Go) — status, data_bind + data_bind_version, report_collection (id+version). Verified 2026-06-25: these + user_id are ALREADY present in dedorch analyses. Plus Harry drops the duplicate/wrong singular analysis table. (→ #3)
3. analyses_messages (Go) = the analysis chat room (user Q + agent A) — replaces the now-deprecated chat_messages/rooms; Python's chat read/write must migrate here before cutover. (→ #25)
4. Reports: Go owns ALL writes. Report stays a skill (no router intent): FE → Go → Python; Python only returns content. Input = the records table (now agent_observability); edit-mode may also need the last report. (→ #7/#18/#24)
5. Markdown minimum now: tables, bold, italic, horizontal separators — optimize that before anything fancier. (→ #23)
6. Deferred: charts (prefer Plotly→JSON in a future chart table over matplotlib PNGs) and images (image table keyed by analysis/message/report + originals in a bucket). (→ #26/#27)
7. Near-term: the remove-problem_statement work isn't on HF yet → PR + deploy + test in the playground (#13). Harry stabilizes Go ~Fri; FE manual testing ~Mon. Keep it playground-able.
8. UI research (no dedicated UI person): new-analysis form (title/objective/business_questions), knowledge menu (user-level vs analysis-level binding), report artifacts panel + version selector; interview + old analysis UI removed. (→ #28)

2. What is already done (KM-674, pushed on pr/4)

Report layer adapted to the new goal shape:

• report/schemas.py::ProblemStatement → objective: str + business_questions: list[str] (old target_value/scope/metric_direction/target_metric dropped). Class name kept for now (rename to ReportGoal once the upstream AnalysisState rename lands).
• report/generator.py renders Objective + numbered Business Questions + a "generated by {user}" line.
• api/v1/report.py::_problem_statement_from is tolerant: prefers new objective / business_questions from state, falls back to legacy problem_statement — works before AND after Harry's migration.
• config/prompts/report_summary.md updated to objective + business questions.
• Report stays records-based; the floor gate (problem_validated) was deliberately left for task #2.

This tolerant-migration pattern (getattr fallback) is the model for tasks #2 and #4.

3. Assessment — gaps & contradictions to resolve before building

These came out of reviewing the plan against the actual code. They are folded into the task table (§4) as tasks 15–19.

• G1 (→ task 15). Records-based reports need the slow path ON. AnalysisRecords persist only in chat_handler._run_slow_path, which runs only when ENABLE_SLOW_PATH=true. Default is off → no records → POST /report 409s. The Swagger demo can't show a non-empty report unless slow path is flipped on and a structured_flow question is run first. BusinessContext is still a stub but the slow path runs fine on it.
• G2 (→ task 16). analysis_records ownership is now required and collides with SKIP_INIT_DB. It's created today by Python create_all (db/postgres/init_db.py), is in no dedorch/Go migration, and after the dedorch cutover (SKIP_INIT_DB=true) Python stops running create_all → the table won't exist → reports break. Decide: dedorch migration (Harry) OR a Python carve-out that creates just this one table even under SKIP_INIT_DB. (Resolved 2026-06-25 — see §1.5.1 / #16 / #22.)
• G3 (→ task 17). chat_history in the report contract is vestigial. Records-based generation reads records by analysis_id; it never uses chat history. Drop chat_history from the report skill contract, or mark it reserved/unused.
• G4 (→ task 4 note). Make #2/#4 tolerant of both state shapes. If Harry drops problem_validated/owner_id from dedorch before Python stops reading them, Python's gate + state_store break. Use the same getattr tolerance KM-674 used. The owner_id→user_id rename also touches api/v1/analysis.py (_serialize_state, list_analyses, get_analysis), not just the model + state_store.
• G5 (→ task 18). "State writes via Go" is bigger than report_id. Python still writes state in /analysis/create (state + room + bindings, plus the data-first gate and soon the mandatory-field check) and in state_store.ensure per turn. If creation moves to Go (consistent with commenting analysis CRUD), then Go owns ALL state writes + both creation gates, and Python's ensure must become a read-only get (Go must guarantee the row exists before any chat turn).
• G6 (smaller).
  • Removing problem_statement (task 1) means neutering it in 4 places: the Intent literal (agents/orchestration.py), the router prompt (config/prompts/intent_router.md), the handler, and the gate's redirect target. Do it with task 2.
  • "generated by {user}" currently prints the raw user_id; a formal report wants a name — source from users.fullname or have Go pass a display name (task 19).
  • The meeting's outline (background / EDA / insights) isn't fully in the renderer; map those sections onto the record fields deliberately (task 5 follow-up).
  • The full project doc (task 11) should reuse REPO_STATUS.md, not restart.

4. Task table

Status legend: ⬜ not started · 🔄 in progress · ✅ done · ⛔ blocked · 🔎 verify · ⏸️ deferred.

#	Task	Owner	Status	Note
1	Comment out problem_statement skill + Intent literal + router prompt + gate redirect target; remove /problem-statement from list_tools	Rifqi	✅	Done 2026-06-25 (one commit w/ #2). Unwired in orchestration.py, intent_router.md, chat_handler.py, tools.py; problem_statement.py kept intact
2	Drop problem_validated: gate neutered; is_report_ready/report_floor → ≥1 completed analysis only, no-LLM	Rifqi	✅	Done 2026-06-25. gate.py no-op, gate call site commented in chat_handler.py, report_floor drops the goal check. Tests updated (test_gate/test_chat_handler/test_readiness). Suite: 284 passed, 7 skipped; ruff clean
3	dedorch analyses migration: drop problem_statement/problem_validated, add objective + business_questions	Harry	🔄	Verified dedorch 2026-06-25: analyses (plural) ALREADY has user_id + status + data_bind + data_bind_version + report_collection → those parts done. Remaining: drop problem_statement/problem_validated + add objective/business_questions. Singular analysis = deprecated duplicate to drop
4	Update Python analyses model + state_store + analysis.py to match dedorch; owner_id→user_id	Rifqi/Sofhia	✅	Done 2026-06-26. owner_id→user_id + added status/data_bind/data_bind_version/report_collection (DB-only, not in the AnalysisState pydantic) across models.py/gate.py/state_store.py/analysis.py + 3 local tests; also report_inputs id/analysis_id → uuid. Kept problem_statement/problem_validated; objective/business_questions wait on Harry's #3. Suite 284 passed
5	Report generator → objective+business_questions, "generated by {user}", formal outline	Sofhia	✅	Goal-shape (KM-674) + author name (#19) + outline (KM-680): Objective → Business Questions → Executive Summary → Key Findings → EDA → Notes & Limitations → How This Was Analyzed
6	Report skill input contract: analysis_id + user_id (no chat_history)	Sofhia/Rifqi	✅	No-op: POST /report already takes only analysis_id + user_id (records-based). Documented in API_ENDPOINTS.md §5. (Edit-mode input revisited in #24.)
7	report_id state update via request to Go, not direct DB	Sofhia + Harry	⬜	Needs Go endpoint. Checkpoint: Go owns ALL reports writes; Python stops any direct insert/update and only returns content; report stays a skill (no intent). See #18
8	Expose/confirm 4 FE endpoints; comment check_data + analysis CRUD	Sofhia	✅	KM-678: list_tools trimmed to /help + /report (analytics/check/retrieve commented in the menu). help confirmed as a call_agent intent — no own endpoint. Analysis CRUD endpoint left registered: "comment the rest" was about the FE slash menu, not killing HTTP routes Go needs
9	Verify analysis_id in call_agent contract	Sofhia	✅	Verified: no separate field — carried as room_id (analysis_id == room_id), per REPO_STATUS §4/§11. Action for Go: send the id as room_id
10	API endpoint doc (MD), 4 endpoints, for Go integration	Rifqi + Sofhia	✅	Done 2026-06-25 — API_ENDPOINTS.md (repo root). 4 FE surfaces with request/response examples (chat SSE transcript, report 201/409 JSON, version list), schemas, §9 full 32-route inventory + task-8 reading
11	Full Python project doc (MD → PDF/Word BRD)	Rifqi	✅	Done 2026-06-26 — PROJECT_BRD.md (repo root): purpose/context, FR-1..9 capabilities, lifecycle, architecture, data model, API (→ API_ENDPOINTS), NFRs, integrations, open items. Reuses REPO_STATUS/API_ENDPOINTS; convert to PDF/Word for distribution
12	Reconcile/open the list_tools PR cleanly (stacked commits)	Rifqi	✅	N/A — we develop directly on the single active branch pr/4 (KM-652 + KM-678 already stacked there); no separate PR to reconcile
13	Deploy HF Python build (remove-problem_statement work) → test 4 endpoints via Swagger / playground	Sofhia + Harry	🔄	Unblocked (#15 ✅). Remove-PS work is on pr/4 but not on HF main yet → PR + deploy, then manual test. Harry stabilizes Go ~Fri; FE testing ~Mon
14	analysis_records home	Rifqi + Sofhia + lead	✅	Resolved 2026-06-25: stays Python-owned, renamed (→ #21); schema handed to Harry so the dedorch migration creates it post-cutover (→ #22). Not moved to Go
15	Flip ENABLE_SLOW_PATH=true + verify an AnalysisRecord persists from a structured_flow question	Rifqi	✅	Verified locally 2026-06-25 (in-process). structured_flow on Titanic.csv → 3-task plan check_data→retrieve_data→analyze_aggregate (all success) → AnalysisRecord persisted (substantive) → report_floor pass → report generates (201). HF env-flip + Swagger run folds into #13
16	Decide analysis_records creation under SKIP_INIT_DB	Rifqi + Harry	✅	Resolved 2026-06-25: Python defines it; Harry's dedorch migration creates it on env-move (Python still creates locally meanwhile) → exists post-cutover. Execution = #22
17	Reconcile report contract with records-based: remove/flag chat_history	Sofhia/Rifqi	✅	Nothing to remove — chat_history was never in the report contract/code (only in help.md). Confirmed via grep; API_ENDPOINTS.md §5 documents the clean contract
18	Confirm Go owns ALL analysis-state writes + both creation gates; make Python state_store.ensure read-only	Rifqi + Harry	⬜	Confirmed by 2026-06-25 checkpoint (Python read-only; Go owns writes + new tables). Execution pending Go endpoints
19	Decide report author display-name source (users.fullname vs Go-passed name)	Sofhia	✅	Done 2026-06-25. AnalysisReport.user_name; generator renders user_name or user_id; api/v1/report.py::_resolve_user_name reads users.fullname never-throw (fallback user_id). Decided: resolve in Python (unblocked); swap to Go-passed name later if preferred
20	Help handoff: update handlers/help.py + help.md — drop the problem_validated tier + define_problem_statement action (the skill it points at is gone as of #1)	Sofhia	✅	Done 2026-06-25. help.py: actions = ask_analysis_question (always) + generate_report (if ready); renders objective/business_questions (getattr-tolerant). help.md v1→v2: 3 tiers, no /problem_statement, /generate report→/report. Local test_help updated → 11 pass
21	Rename analysis_records → report_inputs (table, ORM ReportInputRow, store *ReportInputStore)	Rifqi	✅	Done 2026-06-26. sed rename across 9 files; Pydantic AnalysisRecord kept; columns stay String (pure rename — uuid+FK is the #22 Harry schema). Name report_inputs (purpose; avoids Langfuse/analyses_messages clash). Write scope = one row per slow-path run. Suite 284 passed
22	Finalize report_inputs schema → hand to Harry for the dedorch migration	Rifqi → Harry	🔄	DDL ready (uuid id/analysis_id + FK→analyses(id); user_id/plan_id text; data jsonb = serialized AnalysisRecord, shape documented). dedorch has empty analysis_records → rename. Resolves #16. Action: send Harry the DDL + data shape
23	Report markdown formatting: tables, bold, italic, horizontal separators	Sofhia	✅	Done 2026-06-25. Added --- separators between header + each section in _render_markdown. Tables (EDA) / bold (method labels) / italic (meta + citations) already emitted. Relaxed report_summary.md to allow inline **bold**/*italic* for emphasis (kept no-headings/no-bullets so it doesn't duplicate the section structure / Key Findings). Compile + ruff clean
24	Clarify report input contract: records table (+ last_report for edit mode?)	Rifqi/Sofhia ↔ Harry	⬜ new	Edit-mode input left open at the checkpoint
25	Migrate Python chat path to Go analyses_messages (+ analyses)	Rifqi ↔ Harry	✅	Done 2026-07-02. Read path already on analyses_messages (commit 0066161). This change makes Python read-only: removed the save_messages calls from /api/v2/chat/stream so Go is the sole writer — fixes the double-write both Go+Python were producing. load_history still reads analyses_messages. v1 /chat/stream is unwired so left untouched
26	Charts (DEFERRED): store Plotly JSON in a future chart table (not matplotlib PNG)	—	⏸️	After the markdown path is done end-to-end
27	Images (DEFERRED): image table (id, analysis_id, msg/report ref, order) + originals in a bucket	—	⏸️	Maintenance-heavy; parked
28	UI research (FE): new-analysis form, knowledge menu (user vs analysis level), report artifacts + version selector	Team	⬜ new	No dedicated UI person; interview + old analysis UI removed

5. Critical path & sequencing

• Critical path: #22 (send Harry the report_inputs schema). HF deploy (#13) for the playground. (#4 ✅, #21 ✅; Harry's #3 no longer blocks us — Python is getattr-tolerant.)
• Parallelizable now: #22 (handoff). (#4 ✅, #11 ✅ done.)
• Harry-blocked / coordinated: #3 (now 🔄, blocks #4), #7 (Go endpoint), #18 (Go state ownership), #24 (contract). #25 = chat-path migration to analyses_messages — a cutover blocker.
• Demo gate (playground, #13): deploy the remove-problem_statement work to HF — slow path (#15 ✅) and the report path are verified locally, and #16 is resolved (#22 hands Harry the schema). Keep it playground-able.

6. Decisions still open (need the team / Harry / lead)

• analysis_records: dedorch-owned vs Python-owned (#16/#14). RESOLVED: Python-owned + renamed report_inputs (#21 done); Harry's migration creates it (#22).
• Whether help is its own endpoint or via call_agent (#8). RESOLVED: help is a call_agent intent (no own endpoint).
• Author display-name source for the report (#19). RESOLVED: Python resolves users.fullname (fallback user_id); swap to a Go-passed name later if preferred.
• Keep vs drop chat_history in the report contract (#17). RESOLVED: never in the contract; report is records-based (analysis_id + user_id only).
• Confirm Go takes over analysis creation + both creation gates (data-first + mandatory fields) (#18).
• Report input for edit mode — does Python need the last report content? (#24)
• report_inputs write scope — every agent call vs slow-path-only? (#21) RESOLVED: one row per slow-path run (telemetry stays Langfuse).
• Python history source — confirm Go's analysis_message (#25).