chore(docs): tidy root markdown — archive stale docs, delete superseded ones

Archive (moved to local gitignored docs/_archive/): ARCHITECTURE, AGENT_ARCHITECTURE_CONTEXT_new, PROGRESS, CHECKPOINT_PLAN_2026-06-17, ORCHESTRATOR_REWORK_PLAN. Delete (stale/superseded by REPO_STATUS.md + DEV_PLAN.md): REPO_CONTEXT, PROJECT_SUMMARY, PHASE1_TO_PHASE2_REPORT. Root now keeps README + REPO_STATUS + DEV_PLAN.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

.gitignore

@@ -53,6 +53,5 @@ migratego/
 docs/specs/tabular_parquet_contract.md
 docs/specs/tabular_parquet.md
 
-# Personal / local working docs (not for the shared repo)
-AGENT_ARCHITECTURE_CONTEXT_new.md
-PROJECT_SUMMARY.md
+# Personal / local working docs (not for the shared repo) — archived out of root
+docs/_archive/

ARCHITECTURE.md

@@ -1,353 +0,0 @@
-# Architecture — Data Eyond Agentic Service
-
-**Last updated**: 2026-05-20
-**Status**: Phase 2 catalog path shipped; document ingestion has moved to a separate Go service. The long-term split is **Python = agent/ML layer, Go = data plane**; this document covers the Python side only.
-
----
-
-## Product vision (north star)
-
-Data Eyond is an *AI data scientist* for business analytics, structured around **CRISP-DM** (Business Understanding → Data Understanding → Data Preparation → Modeling → Evaluation → Deployment). Targets executives doing self-serve deep-dives and data analysts/scientists offloading routine work.
-
-Envisioned user flow: **interview agent** captures goal → user connects data sources → asks natural-language question → CRISP-DM-structured analytical response, exportable as a **presentation** or **notebook-style report**.
-
-The catalog-driven, IR-based architecture documented below is the *foundation*. The next architectural evolution is an agentic layer (analytical planner, per-stage CRISP-DM agents, evaluator, reporter) that consumes the existing IntentRouter → QueryPlanner → Executor → ChatbotAgent spine as its tool layer. See `REPO_CONTEXT.md` → *Roadmap — agentic evolution* for the target agent topology.
-
----
-
-## TL;DR
-
-A catalog-driven AI service for data analysis. Users upload documents and register databases or tabular files; they ask natural-language questions and get answers grounded in their data.
-
-The architecture has two paths:
-
-- **Unstructured** (PDF, DOCX, TXT) — dense similarity over prose chunks (the right primitive for free-form text). **Ingestion is handled by a separate Go service**; this Python service reads embeddings from PGVector at query time.
-- **Structured** (databases, XLSX, CSV, Parquet) — a per-user **data catalog** describes what tables/columns exist; an LLM produces a structured **JSON intermediate representation (IR)** of the user's intent; a deterministic compiler turns the IR into SQL or pandas operations.
-
-The LLM produces *intent*, not query syntax. Deterministic code does the rest.
-
----
-
-## 1. Why catalog-driven design
-
-For a database or spreadsheet, a user's question maps to *known tables and columns* — not to *similar text fragments*. Treating structured data with the same retrieval primitive as prose (chunk + embed + rank top-K) makes the right column survive a probabilistic ranking lottery. Catalog-based **lookup** is the right primitive instead.
-
-A central per-user catalog also means:
-
-- One place to keep table/column descriptions (AI-generated, refreshed when the source changes).
-- The query planner sees the user's full data landscape in a single prompt.
-- Schema stays stable across user sessions without hitting the source DB on every query.
-- New sources auto-update the catalog without re-embedding chunks.
-
----
-
-## 2. Source taxonomy
-
-```
-Sources
-├── Unstructured (pdf, docx, txt)        →  Cu  (prose chunks via DocumentRetriever)
-└── Structured
-    ├── Schema (DB)                       →  Cs  (DB tables + columns)
-    └── Tabular (xlsx, csv, parquet)      →  Ct  (sheets + columns)
-                                           Cs ∪ Ct = Data Catalog Context
-```
-
-- **Cu** = unstructured prose context. Retrieval primitive: dense similarity over chunks.
-- **Cs** = DB schema context (tables, columns, descriptions, sample values).
-- **Ct** = tabular file context (sheets, columns, descriptions, sample values).
-- **Data Catalog Context** = `Cs ∪ Ct`. Passed to the query planner as a single unified view.
-
-DB vs tabular is **not** a routing concern — it's a per-source attribute (`source_type`) on each catalog entry. The split only matters at execution time (SQL vs pandas).
-
----
-
-## 3. Routing model
-
-> **Superseded 2026-06-18** — the 3-way `source_hint` below was reworked into a flat **6-intent** handler router (`chat`, `help`, `problem_statement`, `check`, `unstructured_flow`, `structured_flow`). Modality (structured vs unstructured *data*) is now the Planner's job, not the router's. See `ORCHESTRATOR_REWORK_PLAN.md`.
-
-```
-source_hint ∈ { chat, unstructured, structured }
-```
-
-- `chat` — no search, conversational reply only
-- `unstructured` — DocumentRetriever path (Cu)
-- `structured` — catalog-driven path (Cs ∪ Ct → planner → compiler → executor)
-
-The router commits to one path. Cross-source questions ("compare DB sales vs uploaded customer file") are handled inside the structured path because the planner sees both Cs and Ct in one prompt.
-
----
-
-## 4. Core architectural decisions
-
-### 4.1 Catalog as primary context, not retrieval
-
-For most users (≤50 tables), the entire catalog fits in ~3-5k tokens and is passed verbatim to the planner. No vector search, no BM25, no chunk retrieval. The LLM reads the whole catalog and picks the right table.
-
-When a user has hundreds of tables, **catalog-level retrieval** (BM25 + table-level vectors with RRF) can be added as a slicer between `CatalogReader` and `Planner`. Deferred until measurably needed.
-
-### 4.2 JSON IR over raw SQL
-
-The planner LLM emits a structured JSON IR describing query intent — not a SQL string. A deterministic compiler turns the IR into SQL (per dialect) or pandas/polars operations.
-
-Benefits:
-
-- Validatable with Pydantic before execution
-- Compiler whitelists allowed operations (no DROP, DELETE, etc.)
-- Portable: same IR → SQL (any dialect) / pandas / polars
-- Cheaper tokens, easier to debug, trivially testable without an LLM
-- LLM cannot emit valid-but-wrong SQL syntax
-
-### 4.3 Deterministic compiler, not LLM SQL writer
-
-The LLM produces *intent* (the IR). All actual query construction is deterministic Python. Compiler bugs are reproducible and fixable. Same IR always produces the same query.
-
-### 4.4 Pipeline stage isolation
-
-Each stage is its own module with typed input and typed output. No god classes. Stages: `IntentRouter`, `CatalogReader`, `QueryPlanner`, `IRValidator`, `QueryCompiler`, `QueryExecutor`, `ChatbotAgent`. Each is testable in isolation.
-
-### 4.5 Minimal LLM surface
-
-LLM calls happen in exactly three places (KM-557 removed `CatalogEnricher`; ingestion is now LLM-free — the planner reads column names, stats, and sample rows directly):
-
-1. **`IntentRouter`** — once per user message
-2. **`QueryPlanner`** — once per structured query (produces the IR)
-3. **`ChatbotAgent`** — once per answer (formats the response)
-
-Compiler and executors are pure code. No LLM in the hot path of query construction.
-
----
-
-## 5. End-to-end flow
-
-### Ingestion (when user uploads a file or connects a DB)
-
-```
-Structured sources (DB connect / XLSX / CSV / Parquet upload) — Python:
-source upload / DB connect
-    ↓
-introspect schema (DB: information_schema; tabular: file headers + sample rows)
-    ↓
-validate (Pydantic)
-    ↓
-write to catalog store (Postgres jsonb in `data_catalog`, keyed by user_id)
-```
-
-**Unstructured ingestion (PDF / DOCX / TXT) is handled by a separate Go service**, which writes chunks + embeddings into the `documents` collection in PGVector. The Python service does not own this path — it reads only.
-
-### Query (per user message)
-
-```
-User message
-    ↓
-Chat cache check (Redis, 24h TTL)
-    ↓ miss
-Load chat history
-    ↓
-IntentRouter LLM   →  needs_search?  source_hint?
-    ↓
-    ├── chat        → ChatbotAgent → SSE stream
-    ├── unstructured → DocumentRetriever (raw SQL: pgvector `<=>` cosine or `<+>` manhattan) → answerer
-    └── structured  →
-            CatalogReader (load full Cs ∪ Ct for user)
-                ↓
-            QueryPlanner LLM  →  JSON IR
-                ↓
-            IRValidator  (Pydantic + columns-exist + ops whitelist)
-                ↓
-            QueryCompiler  →  SQL (schema source) or pandas (tabular source)
-                ↓
-            QueryExecutor  (DbExecutor or TabularExecutor)
-                ↓
-            QueryResult
-                ↓
-            ChatbotAgent → SSE stream
-```
-
----
-
-## 6. Data catalog
-
-### Storage
-
-Per-user JSON document, stored as a `jsonb` row in Postgres keyed by `user_id`.
-
-### Schema (initial scope)
-
-```
-Catalog
-├── user_id, schema_version, generated_at
-└── sources[]
-    └── Source
-        ├── source_id, source_type, name, description, location_ref, updated_at
-        └── tables[]
-            └── Table
-                ├── table_id, name, description, row_count
-                └── columns[]
-                    └── Column
-                        ├── column_id, name, data_type, description
-                        ├── nullable
-                        ├── pii_flag
-                        ├── sample_values[]
-                        └── stats: { min, max, distinct_count } | null
-```
-
-### Best-practice fields deferred
-
-`description_human`, `synonyms[]`, `tags[]`, `primary_key`, `foreign_keys`, `unit`, `semantic_type`, `example_questions[]`, `schema_hash`, `enrichment_status`. Add when justified by user need.
-
-### Stable IDs
-
-`source_id`, `table_id`, `column_id` are stable internal references. `name` fields can change (e.g. column rename in source DB) without invalidating cached IRs.
-
-### PII handling
-
-Columns with `pii_flag: true` have `sample_values: null` — real values never enter LLM prompts. Auto-detected at ingestion via name patterns + value regex.
-
----
-
-## 7. JSON IR
-
-### Schema (initial scope)
-
-```
-QueryIR
-├── ir_version          : "1.0"
-├── source_id           : str   (references catalog)
-├── table_id            : str   (references catalog)
-├── select[]            : SelectItem
-│   ├── { kind: "column", column_id, alias? }
-│   └── { kind: "agg",    fn, column_id?, alias? }
-├── filters[]           : { column_id, op, value, value_type }
-├── group_by[]          : column_id
-├── order_by[]          : { column_id | alias, dir }
-└── limit               : int | null
-```
-
-### Whitelisted operators
-
-```
-Filter ops:  = != < <= > >= in not_in is_null is_not_null like between
-Agg fns:     count count_distinct sum avg min max
-```
-
-### Validation rules (enforced before execution)
-
-- `source_id` exists in catalog for this user
-- `table_id` belongs to that source
-- Every `column_id` exists in that table
-- Every `agg.fn` and `filter.op` is whitelisted
-- `value_type` consistent with column's `data_type`
-- `limit` positive int, ≤ hard cap (e.g. 10000)
-
-If any rule fails → reject IR → re-prompt planner with error context (max 3 retries).
-
-### Deferred features
-
-`having`, `offset`, boolean tree filters (OR/NOT), `distinct`, joins, window functions. Add as user demand proves the limitation.
-
----
-
-## 8. Executors
-
-Same input (validated IR), same output (`QueryResult`), different backends.
-
-### DbExecutor (schema sources)
-
-```
-IR → SqlCompiler → SQL string + params
-     ↓
-sqlglot validation (SELECT-only, whitelist tables/columns, LIMIT enforced)
-     ↓
-asyncpg / pymysql in read-only transaction with timeout (30s)
-     ↓
-QueryResult
-```
-
-Identifiers come from catalog (verified at validation time, safe to inline as quoted identifiers). Values are always parameterized — never inlined as strings.
-
-### TabularExecutor (tabular sources)
-
-```
-IR → PandasCompiler → operation chain
-     ↓
-choose strategy by file size:
-  ≤ 100 MB    → eager pandas
-  100 MB-1 GB → pyarrow with predicate pushdown
-  > 1 GB      → polars lazy scan
-     ↓
-execute in asyncio.to_thread (CPU work off the event loop)
-     ↓
-QueryResult
-```
-
-Initially eager pandas is sufficient. Add the others when a real file is too big.
-
-### Shared safety guarantees
-
-1. IR validated before reaching compiler
-2. Compiler is deterministic (no LLM)
-3. Identifiers from catalog (trusted)
-4. Values parameterized
-5. sqlglot second-line defence for SQL
-6. Read-only at every layer
-7. Timeouts and row caps
-
----
-
-## 9. Implementation scope
-
-### Initial PR — what ships first
-
-| Item | Folder |
-|---|---|
-| Data catalog Pydantic models | `src/catalog/models.py` |
-| Catalog ingestion (introspect → enrich → validate → store) | `src/catalog/`, `src/pipeline/` |
-| `IntentRouter` with 3-way source_hint | `src/agents/` |
-| `CatalogReader` (loads full catalog) | `src/catalog/reader.py` |
-| `QueryPlanner` LLM call | `src/query/planner/` |
-| JSON IR Pydantic models | `src/query/ir/models.py` |
-| IR validator | `src/query/ir/validator.py` |
-
-**Output**: a validated JSON IR object. Execution lands in a follow-up PR.
-
-### Follow-up PRs
-
-| PR | Scope |
-|---|---|
-| 2 | `QueryCompiler` (IR → SQL / pandas) |
-| 3 | `QueryExecutor` split: `DbExecutor` + `TabularExecutor` |
-| 4 | Retry / self-correction loop on execution failure |
-| 5 | Eval harness (golden question→IR→result examples) |
-| 6 | Auto PII tagging in catalog |
-| Later | Joins in IR, schema drift detection, hybrid catalog search |
-
----
-
-## 10. Open questions
-
-| # | Question | Why it matters |
-|---|---|---|
-| 1 | Catalog storage: JSON file per user vs Postgres `jsonb` row? | Affects ingestion + read performance |
-| 2 | Should the catalog also list unstructured files (with descriptions only)? | Gives router unified view of all user sources |
-| 3 | Catalog refresh trigger: explicit "rebuild" button, on every upload, or background TTL? | Staleness vs latency tradeoff |
-| 4 | Confirm joins are out of initial IR scope? | Limits what user questions can be answered |
-| 5 | PII handling for sample_values: mask, synthesize, or skip? | Affects what gets sent to LLM prompts |
-
----
-
-## 11. References
-
-- `docs/flowchart.html` — interactive end-to-end diagram (open in browser)
-- `docs/flowchart.mmd` — mermaid source for the diagram
-
----
-
-## Glossary
-
-- **Cu** — unstructured context (prose chunks)
-- **Cs** — schema context (DB tables/columns from catalog)
-- **Ct** — tabular context (file sheets/columns from catalog)
-- **IR** — intermediate representation (the JSON query shape)
-- **PR** — pull request (a unit of code change)
-- **PII** — personally identifiable information (names, emails, etc.)
-- **ABC** — abstract base class (Python contract for subclasses)

CHECKPOINT_PLAN_2026-06-17.md

@@ -1,147 +0,0 @@
-# Checkpoint Plan — Wednesday, 17 June 2026
-
-Working plan for Sofhia & Rifqi based on the checkpoint with mas Harry on **Thursday, 11 June 2026**.
-Goal: everything below is **merged and demo-able before the next sync on Wednesday, 17 June (afternoon)**.
-
-**Updated at: Friday, 12 June 2026** (Sofhia + Rifqi)
-
-> Source of truth for decisions is the meeting itself. Note: the NotebookLM summary is **stale on two points** — Data Availability Check was *eliminated* as a tool, and Success Metrics was *folded into* the Problem Statement template. Do not build either as a standalone skill.
-
----
-
-## 0. Progress (per Fri 12 Jun — Sofhia)
-
-Dated snapshot of what landed this session. Live task status (incl. what's left) lives in §2 Ownership — this section only records the deltas + traceability.
-
-- ✅ **Tool matrix** built (xlsx, all ~10 tools + status colours) — presentation material ready.
-- ✅ **Registry trimmed to 4 active analytics** (`KM-641`, commit `66e2e4d`): `ACTIVE_ANALYTICS_TOOLS` (descriptive, aggregate, correlation, trend) vs `DEFERRED_ANALYTICS_TOOLS` (comparison, contribution, profile, segment) — specs + compute fns kept, only registry exposure withheld. Tests 206 pass, ruff/mypy clean.
-- ✅ **Planner few-shot synced**: Example A `analyze_contribution` → `analyze_aggregate` (so few-shots don't reference a deferred tool).
-- ✅ **Data-access tools renamed** (`KM-642`, commit `c38c0c2`): `query_structured` → `data_retrieve`, `retrieve_documents` → `knowledge_retrieve` across the tool layer + planner stub/prompt/validator/few-shots. Mechanical, no behavior change.
-- ✅ **`data_check` merge + `knowledge_check`** (`KM-643`, commit `4bd5f1e`): `list_sources` + `describe_source` → one parameterized `data_check` (no arg = list structured sources; `source_id` = schema) + new `knowledge_check` (unstructured). Tests 206 pass.
-- ✅ **Redis Cloud live** (free tier, TTL = 1 h), env vars shared in the group (Rifqi).
-- ✅ **Planner tool list verified** against the trimmed registry — no references to old tool names or deferred analytics anywhere in `src/` (Rifqi).
-- 📌 **Decision:** `tests/` stays gitignored — team decided not to push tests to origin (closes PROGRESS.md R3 as won't-do).
-- 📌 **Ownership:** Rifqi owns `generate_report` development + the `analysis_records` table / real `AnalysisStore` (contract still co-designed with Sofhia).
-- ✅ **R5 cache fix** (Rifqi, `b701e95`): chat cache scoped by `user_id`, TTL 24h→1h.
-- ✅ **AnalysisRecord persistence landed** (Rifqi): `stage` now flows to the record (CRISP-DM grouping for the report) + identity fields (`record_id`/`analysis_id`/`user_id`); `PostgresAnalysisStore` + `analysis_records` table replace `NullAnalysisStore`, wired into `ChatHandler`. Unblocks the `generate_report` renderer and the DoD "record persisted" step. Open: `analysis_id` handoff from Harry's Analysis State.
-- ✅ **Verb-first tool naming** (Sofhia, commit `2d6406d`): the 4 data/knowledge tools renamed to lead with a verb — `data_check`→`check_data`, `knowledge_check`→`check_knowledge`, `data_retrieve`→`retrieve_data`, `knowledge_retrieve`→`retrieve_knowledge` (the `analyze_*` tools already lead with a verb). These verb-first names are now canonical; the tool-set table + §3 below use them. Dated log entries above keep the old names as historical record.
-
----
-
-## 1. Locked decisions (from the 2026-06-11 checkpoint)
-
-1. **Single chat page.** The separate interview/survey page is killed. Sidebar = Knowledge menu (connect/manage data) + Analysis menu (sessions).
-2. **Data-first hard gate.** Creating a new analysis requires **≥ 1 bound data source** (server-side rejection, no empty sessions). User provides title + optional short description.
-3. **Analysis State lives in the DB.** Per-analysis row: `user_id`, `data_source_ids[]`, `interview_status` (default `not_pass`), `report_status` (default `no_report` → `V1`, `V2`, …). Explicitly **NOT cached, NOT in Redis** — the Orchestrator reads it from Postgres every turn.
-4. **Skills, not agents.** No separate interview agent. The Orchestrator routes per user turn using the Analysis State; an analytical request still executes through the existing Planner → TaskRunner → Assembler spine (static plan, no mid-run LLM).
-5. **Interview = one skill: Problem Statement.** Success metrics become fields inside the PS template (what to increase/decrease + target). Data availability check is handled by the data-first creation gate + PS validation cross-checking fields against the bound catalog — not a separate tool.
-6. **Analytics focus = 4 tools:** descriptive, aggregate, correlation, trend. The other four composites (comparison, contribution, profile, segment) are **deprioritized, not deleted** — keep the code, just don't register them. If "comparison" returns later it should be a proper statistical **test**, not a generic compare.
-7. **`describe_source` merges into the listing tool** — one call returns sources *with* their schema/metadata, fewer tools for the planner.
-8. **Report = on-demand, button-triggered (not a chat skill).** A dedicated "Generate Report" button in the Analysis menu calls a **report API** (not the chat route): trigger generation for a session, list its versions, fetch a version. Renders from accumulated **AnalysisRecords + the Problem Statement** — never from chat history. Each report is a **persisted, versioned artifact**: generation snapshots the record IDs it used and bumps `report_status` to `V<n>`. (Owner: Rifqi, KM-644.)
-9. **Help = deterministic guide.** No LLM: read Analysis State → tell the user the next required step. Callable in any state.
-10. **Redis Cloud free tier, TTL = 1 hour**, env shared in the team group — for retrieval/query caching only, never for state.
-
-### Final tool set (~10)
-
-| Tool (canonical, verb-first) | Maps to (lineage) | Status |
-|---|---|---|
-| `check_knowledge` | new — list user's documents + metadata | done |
-| `check_data` | `list_sources` + `describe_source` merged (catalog-backed) | done |
-| `retrieve_knowledge` | `retrieve_documents` → `knowledge_retrieve` | done |
-| `retrieve_data` | `query_structured` → `data_retrieve` (tabular: file + DB, both working) | done |
-| `analyze_descriptive` | `src/tools/analytics/descriptive.py` | done |
-| `analyze_aggregate` | `src/tools/analytics/aggregation.py` | done |
-| `analyze_correlation` | `src/tools/analytics/relationship.py` | done |
-| `analyze_trend` | `src/tools/analytics/temporal.py` | done |
-| `problem_statement` | new — interview skill (**Harry**) | Harry |
-| `generate_report` | new — on-demand, versioned | to design |
-| `help` | new — deterministic state guide | to build |
-
-(`problem_statement` + `help` live at the orchestrator level; `generate_report` is **button-triggered via a dedicated report API**, not chat-routed (decision #8). The TaskRunner registry holds the 4 analytics + 4 data/knowledge tools. Unregister `analyze_comparison`, `analyze_contribution`, `analyze_profile`, `analyze_segment` from the planner-visible registry — keep the modules.)
-
----
-
-## 2. Ownership
-
-### Sofhia
-- [x] 4 analytics tools: trim registry to 4 active, tests still pass after deprioritizing the other four. (`KM-641`, commit `66e2e4d`)
-- [x] Data/knowledge tools: merge `describe_source` into `data_check`, rename `retrieve_documents` → `knowledge_retrieve`, `query_structured` → `data_retrieve`, build `knowledge_check`. (`KM-642` `c38c0c2`, `KM-643` `4bd5f1e`)
-- [ ] Co-design `generate_report` contract with Rifqi (Rifqi owns development, see §3).
-- [x] Tool matrix (see §4).
-
-### Rifqi
-- [x] **Redis Cloud free tier** (~30–50 MB): create instance, set TTL = 1 h, share env vars in the group. (done 12 Jun)
-- [x] **R5 cache fix**: chat cache key scoped by `user_id`, TTL 24h→1h (urgent on shared Redis). (12 Jun, commit `b701e95`)
-- [x] **AnalysisRecord contract gaps closed**: `stage` (CRISP-DM) now flows Task→TaskResult→TaskSummary so the report can group the method appendix; `AnalysisRecord` gained `record_id`/`analysis_id`/`user_id` identity fields. (12 Jun)
-- [x] **`analysis_records` table + real `AnalysisStore`**: `PostgresAnalysisStore` (save + `list_for_analysis`, never-throw) replaces `NullAnalysisStore`; wired into `ChatHandler`, `user_id` stamped at save. Satisfies the DoD "record persisted" step. (12 Jun)
-- [ ] **Own `generate_report` development — KM-644 "Report Generator"** (contract co-designed with Sofhia, see §3). Button-triggered via a dedicated **report API** (trigger / list versions / fetch); reads `analysis_records` + Problem Statement; persists a versioned report artifact, bumps `report_status`. *(record persistence done above; report API + persistence + renderer + contract doc next)*
-- [x] Verify planner tool list matches the trimmed registry (4 analytics + 4 data/knowledge) and few-shots don't reference removed tools. (verified 12 Jun — no stale tool names in `src/`)
-- ⚠️ **Blocked-on-Harry**: `analysis_id` is `NULL` on persisted records until the Analysis State reaches the slow path — need the session-ID handoff so `generate_report` can group records per analysis.
-
-### Shared (Sofhia + Rifqi)
-- [ ] `generate_report` design + skeleton: input = AnalysisRecords for the session + Problem Statement from Analysis State; output = versioned artifact; bumps `report_status`. Agree on the contract even if rendering is stubbed for Wednesday. (Development: Rifqi.)
-- [ ] `help` skill: deterministic — read Analysis State, return the next required step. Small, do it together or whoever finishes first.
-- [ ] Tool behavior smoke test end-to-end on an easy case (descriptive/aggregate path), per Harry's ask: "robust tools before agents."
-
-### Harry (dependencies — not ours, but we block on them)
-- `problem_statement` skill + PS template (incl. increase/decrease target fields).
-- Analysis State class + DB table, frontend analysis-builder step.
-- Merging our PRs (he auto-merges; he clones from latest after).
-
----
-
-## 3. Per-tool behavior contract (how to build each one)
-
-Harry's framing: for every tool, define **goal / trigger / input / process / output**, and behave like a Claude-style skill — if a required argument is missing, respond with a polite feedback message asking for it (e.g. table/column name), never guess silently.
-
-- **`check_knowledge`** — "what documents do I have?" → list documents with name, type, uploaded-at.
-- **`check_data`** — "what data do I have?" → sources (file + DB) with schema/metadata from the data catalog, created/uploaded timestamps.
-- **`retrieve_knowledge`** — RAG over uploaded documents; returns passages with source attribution.
-- **`retrieve_data`** — query tabular data (file + DB) via QueryIR; output consumable by the `analyze_*` tools.
-- **`analyze_*` (4)** — require valid table/column references; if missing or wrong, return actionable feedback instead of guessing.
-- **`generate_report`** — button-triggered via a dedicated report API (not chat-routed); on-demand only (never auto); post-pass gated; renders from AnalysisRecords + PS; persists a versioned artifact, snapshots record IDs, bumps version. (KM-644, Rifqi.)
-- **`help`** — no LLM; state → next step. Repeating it is fine, that's its job.
-
----
-
-## 4. Tool matrix (deliverable for the sync)
-
-Harry explicitly asked for a matrix covering every tool. Produce one sheet/markdown table with columns:
-
-`tool | goal | trigger (when the orchestrator calls it) | input | process | output | gated by interview_status? | status (done / in progress / planned)`
-
-Use the tool set table in §1 as the row list. This doubles as the presentation material on Wednesday.
-
----
-
-## 5. Day-by-day
-
-| Day | Target |
-|---|---|
-| **Thu 11** | Checkpoint meeting + task split with Harry. |
-| **Fri 12 (today)** | ✅ Registry trimmed to 4 analytics + few-shot synced (Sofhia, KM-641). ✅ Tool matrix built. ⏳ Redis Cloud + env share (Rifqi). |
-| **Mon 15** | Data/knowledge tools done (`data_check` merge, renames, `knowledge_check`). `generate_report` contract agreed. |
-| **Tue 16** | `help` skill done. `generate_report` skeleton wired to AnalysisRecord. Tool matrix drafted. End-to-end smoke test on the easy path. |
-| **Wed 17 (AM)** | Buffer: fix fallout, finalize matrix, rehearse the demo flow. |
-| **Wed 17 (PM)** | **Sync with Harry.** |
-
----
-
-## 6. Open questions to confirm with Harry on Wednesday
-
-1. **Gate scope.** Proposal: keep the fast path + exploration tools (`check_knowledge`, `check_data`, retrieves, `help`, arguably `descriptive`) available **pre-pass**; gate only the insight tools (correlation, trend, report). Hard-gating everything risks frustrating users who just want to look at their data.
-2. **Who flips `interview_status` to `pass`?** Proposal: a deterministic validator (PS template slots complete + fields cross-checked against the bound catalog) makes the call — the LLM conducts the conversation but never decides the pass. ("Conversational skin, deterministic skeleton.")
-3. **Skills vs spine — one sentence to lock in writing:** *"Skills are registry tools executed by the existing Planner → TaskRunner → Assembler spine; the Analysis State gate is a pre-check in the Orchestrator."* This keeps the new flow and the locked architecture fully compatible.
-4. `generate_report` invocation goes through the same gate (post-pass only) — confirm.
-
----
-
-## 7. Definition of done for Wednesday
-
-- [ ] All team PRs merged; Harry unblocked on the Analysis State class.
-- [ ] Registry exposes exactly 4 analytics + 4 data/knowledge tools, all passing local tests.
-- [ ] Redis Cloud shared and working locally for all three of us (TTL 1 h).
-- [ ] `help` works against a (possibly stubbed) Analysis State.
-- [ ] `generate_report` contract written; skeleton callable.
-- [ ] Tool matrix ready to present.
-- [ ] One end-to-end happy path runs: create analysis (with data) → blocked pre-pass → interview stub passes → descriptive/aggregate answer → record persisted.

PHASE1_TO_PHASE2_REPORT.md

@@ -1,273 +0,0 @@
-# Phase 1 → Phase 2 Migration Report
-
-A walkthrough of what changed between the original retrieval-style backend (Phase 1) and the current catalog-driven backend (Phase 2). Intended as a hand-off for the lead.
-
----
-
-## 1. The conceptual change
-
-**Phase 1** was a single retrieval-style RAG pipeline. Every question — whether it pointed at a database, a spreadsheet, or a PDF — went through the same primitive: **chunk + embed + top-K** over PGVector. Schema and tabular columns were embedded as chunks and ranked alongside prose. When the question needed SQL, the LLM **wrote the SQL string directly** (via `query_executor`).
-
-**Phase 2** splits the system into two paths governed by an LLM router:
-
-| Path | Primitive | Why |
-|---|---|---|
-| Unstructured (PDF / DOCX / TXT) | Dense similarity over prose chunks (PGVector) | Right primitive for free text |
-| Structured (DB / CSV / XLSX / Parquet) | **Per-user data catalog** → LLM emits a **JSON IR** of intent → deterministic **compiler** → **executor** (SQL or pandas) | A column lookup shouldn't go through a similarity ranking lottery; the LLM emits intent, never SQL syntax |
-
-Three explicit LLM call sites only:
-
-1. **Intent router** (classifies the user message into `chat` / `unstructured` / `structured`)
-2. **Query planner** (turns the question + catalog into a Pydantic-validated `QueryIR`)
-3. **Chatbot agent** (formats the final answer, streamed over SSE)
-
-Everything else — IR validation, SQL/pandas compilation, execution — is deterministic Python.
-
----
-
-## 2. File-by-file changes
-
-### 2.1 Deleted (Phase 1 only)
-
-| Phase 1 path | Reason it was removed |
-|---|---|
-| `src/rag/base.py`, `src/rag/retriever.py`, `src/rag/router.py` | Replaced by `src/retrieval/` |
-| `src/rag/retrievers/baseline.py`, `schema.py`, `document.py` | Schema retrieval gone (catalog replaces it); document retriever rewritten in `src/retrieval/document.py` |
-| `src/tools/search.py` (whole `tools/` folder) | Only consumer was `rag/router.py` |
-| `src/query/base.py` | Duplicate of `query/executor/base.py` |
-| `src/query/query_executor.py` | Replaced by `src/query/service.py` |
-| `src/query/executors/db_executor.py` | Replaced by `src/query/executor/db.py` |
-| `src/query/executors/tabular.py` | Replaced by `src/query/executor/tabular.py` |
-| `src/agents/chatbot.py` (Phase 1 LangChain chatbot) | Phase 2 `ChatbotAgent` lives at the same path now — see §2.2 |
-| `src/api/v1/knowledge.py` | Fake `/knowledge/rebuild` endpoint, never wired |
-| `src/config/agents/system_prompt.md`, `guardrails_prompt.md` | Replaced by `src/config/prompts/{chatbot_system,guardrails}.md` |
-| `src/models/structured_output.py` (`IntentClassification`) | Replaced by `IntentRouterDecision` Pydantic model inside `agents/orchestration.py` |
-| `src/models/sql_query.py` | LLM no longer emits SQL; IR replaces it |
-| `src/pipeline/orchestrator.py` (empty stub) | Redundant — `StructuredPipeline` takes the introspector at `run()` time |
-
-### 2.2 Renamed / moved (same role, new home)
-
-| Phase 1 location | Phase 2 location | Notes |
-|---|---|---|
-| `src/agents/chatbot.py` (Phase 1) → deleted, then `src/agents/answer_agent.py` (`AnswerAgent`) → renamed | `src/agents/chatbot.py::ChatbotAgent` | Final answer formation; streams via `astream` |
-| `src/knowledge/parquet_service.py` | `src/storage/parquet.py` | Parquet upload/download helper |
-| `src/pipeline/document_pipeline/document_pipeline.py` (folder) | `src/pipeline/document_pipeline.py` (flat) | Single module |
-| `src/rag/retrievers/document.py` | `src/retrieval/document.py` | `DocumentRetriever` migrated; tabular file types filtered out of results. **Post-report update (mentor commit 61c746f, 2026-05-20):** rewritten to raw SQL (pgvector `<=>` cosine, `<+>` manhattan only) to dodge asyncpg type-mapping issues with the Go-ingested schema. MMR / euclidean / inner_product dropped. |
-| `src/rag/router.py` | `src/retrieval/router.py` | `RetrievalRouter`, Redis-cached, unstructured-only; dead `db: AsyncSession` + `source_hint` params removed |
-| `src/rag/base.py` (`RetrievalResult`, `BaseRetriever`) | `src/retrieval/base.py` | Same dataclass + ABC |
-
-> **Heads-up on the intent router**: the Phase 1 file `src/agents/orchestration.py` and its class `OrchestratorAgent` were **kept in place** for Phase 2 — but the body was fully rewritten. The class now emits `IntentRouterDecision(needs_search, source_hint ∈ {chat, unstructured, structured}, rewritten_query)`. The prompt file and test file use the `intent_router` name (`config/prompts/intent_router.md`, `tests/agents/test_intent_router.py`), but **the source module is still `orchestration.py` and the class is still `OrchestratorAgent`**. Existing imports continue to work; only the behavior changed.
-
-### 2.3 Added (Phase 2 new)
-
-**Catalog subsystem (whole new concept)**
-
-| Path | Role |
-|---|---|
-| `src/catalog/models.py` | Pydantic: `Catalog → Source[] → Table[] → Column[]`, `ForeignKey`, `ColumnStats.top_values` |
-| `src/catalog/introspect/base.py` | `BaseIntrospector` ABC |
-| `src/catalog/introspect/database.py` | DB introspector — wraps Phase 1 `db_pipeline/extractor.py` (`get_schema`, `profile_column`, `get_row_count`) |
-| `src/catalog/introspect/tabular.py` | CSV / XLSX / Parquet introspector — one `Table` per XLSX sheet |
-| `src/catalog/render.py` | Renders a `Source` for the planner prompt |
-| `src/catalog/validator.py` | Unique-ID + foreign-key-ref invariants |
-| `src/catalog/store.py` | Postgres `jsonb` upsert keyed by `user_id` (table `data_catalog`) |
-| `src/catalog/reader.py` | Loads + filters catalog by `source_hint` |
-| `src/catalog/pii_detector.py` | Flags PII columns at ingestion → suppresses `sample_values` |
-| `src/security/pii_patterns.py` | Name patterns + value regex used by the detector |
-
-**JSON IR + query subsystem**
-
-| Path | Role |
-|---|---|
-| `src/query/ir/models.py` | `QueryIR` Pydantic schema |
-| `src/query/ir/operators.py` | `ALLOWED_FILTER_OPS`, `ALLOWED_AGG_FNS`, `LIMIT_HARD_CAP`, `TYPE_COMPATIBILITY` |
-| `src/query/ir/validator.py` | Catalog-aware IR validation (rejects unknown column ids, bad ops, type mismatches, oversize limits) |
-| `src/query/planner/service.py` | `QueryPlannerService.plan(question, catalog, previous_error)` — Azure OpenAI structured output → `QueryIR` |
-| `src/query/planner/prompt.py` | Builds the planner prompt from catalog text |
-| `src/query/compiler/base.py` | Compiler ABC |
-| `src/query/compiler/sql.py` | `SqlCompiler` (Postgres) — all 12 filter ops, params as a dict |
-| `src/query/compiler/pandas.py` | `PandasCompiler` — returns `CompiledPandas(apply, output_columns)` |
-| `src/query/executor/base.py` | `BaseExecutor` + `QueryResult` |
-| `src/query/executor/db.py` | `DbExecutor` — sqlglot SELECT-only guard, RO txn, 30 s `statement_timeout`, 10 k row cap |
-| `src/query/executor/tabular.py` | `TabularExecutor` — Parquet via blob, `asyncio.to_thread`, 10 k cap |
-| `src/query/executor/dispatcher.py` | `ExecutorDispatcher.pick(ir)` — picks by `source.source_type` |
-| `src/query/service.py` | `QueryService.run(user_id, question, catalog)` — plan → validate → retry (max 3) → dispatch → execute |
-
-**Agents**
-
-| Path | Role |
-|---|---|
-| `src/agents/orchestration.py` | `OrchestratorAgent` — Phase 1 file/class name preserved; Phase 2 body. Emits `IntentRouterDecision` |
-| `src/agents/chatbot.py` | `ChatbotAgent` — formerly `AnswerAgent` in `agents/answer_agent.py`; renamed in Cleanup PR |
-| `src/agents/chat_handler.py` | `ChatHandler.handle(...)` — top-level orchestrator; yields `intent` / `chunk` / `done` / `error` SSE events |
-
-**Pipelines & API**
-
-| Path | Role |
-|---|---|
-| `src/pipeline/structured_pipeline.py` | DB / tabular ingestion: introspect → merge → validate → upsert |
-| `src/pipeline/triggers.py` | `on_db_registered`, `on_tabular_uploaded`, `on_document_uploaded`, `on_catalog_rebuild_requested` |
-| `src/api/v1/data_catalog.py` | `GET /api/v1/data-catalog/{user_id}` + `POST /api/v1/data-catalog/rebuild` |
-| `src/models/api/catalog.py` | Catalog request/response models |
-| `src/config/prompts/intent_router.md`, `query_planner.md`, `chatbot_system.md`, `guardrails.md` | New prompts. `guardrails.md` is appended to `chatbot_system.md` at load time |
-| `src/db/postgres/models.py` (added `Catalog` SQLAlchemy class) | Stores the per-user jsonb document in `data_catalog` |
-
-### 2.4 Rewired API endpoints
-
-| Endpoint | Phase 1 wiring | Phase 2 wiring |
-|---|---|---|
-| `POST /api/v1/chat/stream` | Inline in `chat.py`: `OrchestratorAgent` → `retriever` → `query_executor` → `chatbot` | Delegates to `ChatHandler.handle()`. Redis cache, fast intent, history load, and message persistence stay in the endpoint |
-| `POST /api/v1/database-clients/{id}/ingest` | Called `db_pipeline_service.run()` and dual-wrote vectors | Calls **only** `on_db_registered` (catalog build). Failure → HTTP 500 |
-| `POST /api/v1/document/process` | Always pushed to vector store | PDF/DOCX/TXT → `knowledge_processor` (vectors); CSV/XLSX → `on_tabular_uploaded` (catalog only, **no vector embedding**) |
-| `POST /api/v1/document/upload` | Storage + DB row | Same, plus `on_document_uploaded` trigger |
-| `POST /api/v1/data-catalog/rebuild` | — | New: iterates all sources, re-runs per-source trigger |
-| `GET /api/v1/data-catalog/{user_id}` | — | New: returns `list[CatalogIndexEntry]` |
-
-### 2.5 Phase 1 files still in production use
-
-These were **not rewritten** — Phase 2 imports them directly:
-
-- `src/database_client/database_client_service.py`
-- `src/utils/db_credential_encryption.py` (`decrypt_credentials_dict`) — `src/security/credentials.py` is still a stub
-- `src/pipeline/db_pipeline/db_pipeline_service.py` (`engine_scope` context manager — used by both the introspector and `DbExecutor`)
-- `src/pipeline/db_pipeline/extractor.py` (`get_schema`, `profile_column`, `get_row_count`)
-- `src/knowledge/processing_service.py` (PDF / DOCX / TXT extraction + embedding)
-- `src/db/postgres/{connection,init_db,vector_store}.py`, `src/storage/az_blob/`, `src/middlewares/`, `src/security/auth.py`
-
----
-
-## 3. End-to-end flow (current state)
-
-### 3.1 Ingestion
-
-```
-User action                Pipeline                                Storage
-──────────────             ────────────────────────────             ─────────────────
-upload PDF/DOCX/TXT  →   DocumentPipeline                       →  Azure Blob + PGVector
-                          (extract → chunk → embed)                 (table: langchain_pg_embedding)
-                          + on_document_uploaded                    + retrieval cache invalidate
-
-upload CSV/XLSX     →    TabularIntrospector                   →  Azure Blob (Parquet)
-                          (sheets / columns + sample + stats)       + data_catalog jsonb row
-                          → CatalogValidator → CatalogStore         (NO vector store — catalog only)
-                          via on_tabular_uploaded
-
-register DB         →    DatabaseIntrospector                  →  data_catalog jsonb row
-                          (information_schema + sample + FKs)
-                          → validate → store
-                          via on_db_registered
-```
-
-### 3.2 Query (per user message → SSE stream)
-
-```
-POST /api/v1/chat/stream
-        │
-        ├── Redis cache check (24h TTL) — hit returns cached stream
-        ├── _fast_intent (greetings / goodbyes) — bypass LLM
-        ├── load history from chat_messages
-        │
-        └── ChatHandler.handle(message, user_id, history)         [src/agents/chat_handler.py]
-                │
-                ├─ OrchestratorAgent.classify()                    [agents/orchestration.py]
-                │     → needs_search, source_hint, rewritten_query
-                │
-                ├── source_hint == "chat"
-                │     → ChatbotAgent.astream() → yield chunk events
-                │
-                ├── source_hint == "unstructured"
-                │     → RetrievalRouter.retrieve()                 [retrieval/router.py, Redis-cached]
-                │         → DocumentRetriever (raw SQL: pgvector `<=>` cosine or `<+>` manhattan)
-                │     → ChatbotAgent.astream(chunks=...)
-                │
-                └── source_hint == "structured"
-                      → CatalogReader.read(user_id, "structured")  [catalog/reader.py]
-                      → QueryService.run(user_id, question, catalog)   [query/service.py]
-                           │
-                           ├─ QueryPlannerService.plan(...)        [query/planner/service.py]
-                           │    LLM(catalog, question, prev_error?) → QueryIR
-                           │
-                           ├─ IRValidator.validate(ir, catalog)    [query/ir/validator.py]
-                           │    fail → loop back to planner with error context (max 3)
-                           │
-                           ├─ ExecutorDispatcher.pick(ir)          [query/executor/dispatcher.py]
-                           │    schema source  → DbExecutor
-                           │    tabular source → TabularExecutor
-                           │
-                           ├─ DbExecutor.run(ir):                  [query/executor/db.py]
-                           │    SqlCompiler → (sql, params)
-                           │      → sqlglot SELECT-only guard
-                           │      → engine_scope (Phase 1 utility) in asyncio.to_thread
-                           │      → RO txn + statement_timeout=30s + 10k cap
-                           │
-                           ├─ TabularExecutor.run(ir):             [query/executor/tabular.py]
-                           │    resolve Parquet blob path
-                           │      → download → PandasCompiler.apply(df)
-                           │      → asyncio.to_thread → 10k cap
-                           │
-                           └─ QueryResult { rows, columns, row_count,
-                                            truncated, source_id, error?, elapsed_ms }
-                      →
-                      ChatbotAgent.astream(query_result=...)
-                            → yield chunk events
-                │
-                └── final events: done / error
-        │
-        └── persist user + assistant messages to chat_messages
-        └── populate Redis cache
-```
-
-**Safety invariants for the structured path** (read-only at every layer):
-
-1. IR validated against the catalog before reaching the compiler
-2. Identifiers come from the catalog (trusted; inlined as quoted identifiers)
-3. Values from `IR.filters` are always parameterized
-4. Compiler is deterministic — no LLM in the hot path
-5. sqlglot rejects anything that isn't a pure SELECT
-6. DB connection is read-only with a 30 s `statement_timeout`
-7. Hard 10 000 row cap on both executors; neither raises — errors go in `QueryResult.error`
-
----
-
-## 4. Summary table for review
-
-| Concern | Phase 1 — where it lived | Phase 2 — where it lives | Change type |
-|---|---|---|---|
-| Intent classification | `agents/orchestration.py::OrchestratorAgent` (free-text intent) | **Same path + same class name** — body rewritten to emit `IntentRouterDecision` | Body rewrite only |
-| Top-level chat orchestration | Inline in `api/v1/chat.py` | `agents/chat_handler.py::ChatHandler` | Extracted to a reusable module |
-| Final answer formation | `agents/chatbot.py` (Phase 1 LangChain) | `agents/chatbot.py::ChatbotAgent` (was `AnswerAgent` in `answer_agent.py` mid-cycle) | Rewritten + renamed |
-| Schema retrieval (DB / tabular) | `rag/retrievers/schema.py` + PGVector chunks | **Removed**. Replaced by catalog (`catalog/store.py` jsonb) loaded verbatim into planner prompt | Whole concept replaced |
-| Doc retrieval (PDF / DOCX / TXT) | `rag/retrievers/document.py`, `rag/router.py` | `retrieval/document.py`, `retrieval/router.py` | Moved; Redis cache restored; tabular files filtered. **Post-report update:** rewritten to raw SQL (cosine / manhattan only); collection renamed `document_embeddings` → `documents` to match the Go ingestion service. |
-| Query writing | `query/query_executor.py` + `models/sql_query.py` (LLM writes SQL) | `query/planner/service.py` (LLM writes IR) + `query/compiler/sql.py` (deterministic) | LLM emits intent, not SQL |
-| DB execution | `query/executors/db_executor.py` | `query/executor/db.py::DbExecutor` | Folder renamed (`executors` → `executor`); sqlglot guard + RO txn + 30 s timeout kept |
-| Tabular execution | `query/executors/tabular.py` | `query/executor/tabular.py::TabularExecutor` | Parquet-only; pandas compiler split out |
-| Executor selection | Hard-coded in `query_executor.py` | `query/executor/dispatcher.py::ExecutorDispatcher` | New; routes by `source.source_type` |
-| Catalog (NEW) | — | `catalog/` (models, introspect/, validator, store, reader, pii_detector, render) | New subsystem |
-| Catalog persistence | (data was embedded in PGVector) | Postgres jsonb table `data_catalog`, keyed by `user_id` | New table |
-| Ingestion triggers | Inline in API endpoints | `pipeline/triggers.py` (`on_db_registered`, `on_tabular_uploaded`, `on_document_uploaded`, `on_catalog_rebuild_requested`) | Centralized event entry points |
-| Structured pipeline | `pipeline/db_pipeline/db_pipeline_service.py` (still present for `engine_scope` + extractor reuse) | `pipeline/structured_pipeline.py` (orchestrator) — reuses Phase 1 extractor | New orchestrator wraps Phase 1 introspection helpers |
-| Document pipeline | `pipeline/document_pipeline/document_pipeline.py` (folder) | `pipeline/document_pipeline.py` (file) | Flattened; CSV / XLSX now skip the vector store |
-| Parquet helper | `knowledge/parquet_service.py` | `storage/parquet.py` | Moved into `storage/` |
-| Prompts | `config/agents/system_prompt.md`, `guardrails_prompt.md` | `config/prompts/{intent_router,query_planner,chatbot_system,guardrails}.md` | Folder renamed; split into four files; guardrails appended to `chatbot_system` at load |
-| PII detection | — | `catalog/pii_detector.py` + `security/pii_patterns.py` | New. Columns flagged `pii_flag=true` get `sample_values: null` so PII never enters prompts |
-| Chat endpoint | `api/v1/chat.py` (does everything inline) | `api/v1/chat.py` (cache + history + persistence) → delegates to `ChatHandler` | Slimmed; SSE event shape is `intent` / `chunk` / `done` / `error` |
-| DB ingest endpoint | `api/v1/db_client.py::ingest` (Phase 1 `db_pipeline_service.run()`) | `api/v1/db_client.py::ingest` (calls `on_db_registered` only) | Phase 1 dual-write removed |
-| Document process endpoint | `api/v1/document.py::process` (always vectorize) | `api/v1/document.py::process` (PDF/DOCX/TXT → vectors; CSV/XLSX → catalog via `on_tabular_uploaded`) | Routing by file type |
-| Catalog management API | — | `api/v1/data_catalog.py` (GET index + POST rebuild) | New |
-
-**Bottom line.** Every Phase 1 file under `src/rag/`, `src/tools/`, `src/query/executors/`, `src/query/query_executor.py`, `src/query/base.py`, `src/api/v1/knowledge.py`, and `src/config/agents/` is gone. Phase 1 introspection helpers under `src/pipeline/db_pipeline/` and `src/database_client/` are still imported by Phase 2 — they were not rewritten, just wrapped. The three LLM call sites are now explicit and the SQL-writing one no longer exists; the planner emits a Pydantic-validated `QueryIR` instead.
-
-The one filename gotcha to remember: the **intent router** still lives at `src/agents/orchestration.py` as class `OrchestratorAgent` (Phase 1 name kept for import-site compatibility, Phase 2 body). The matching prompt and tests use the `intent_router` name, but the source module does not.
-
----
-
-## 5. Addendum — post-report changes (2026-05-20, mentor commit `61c746f`)
-
-This report was originally written as a snapshot at Phase 2 completion. The Phase 2 architecture itself hasn't changed, but a few implementation details have shifted as the Go migration progresses. Captured here so the report stays trustworthy:
-
-- **Doc ingestion is now a Go service.** PDF/DOCX/TXT chunking + embedding + writes into PGVector are no longer done by Python. The Python service reads only.
-- **PGVector collection renamed:** `document_embeddings` → `documents` (to match the Go service's writes). Touched files: `db/postgres/vector_store.py`, `retrieval/document.py`.
-- **`DocumentRetriever` rewritten to raw SQL.** Uses pgvector operators directly (`<=>` cosine, `<+>` manhattan). The LangChain ORM path couldn't cope with the schema written by the Go service (asyncpg type-mapping issues — id String vs UUID, jsonb_path_match binding quirks). MMR / euclidean / inner_product were dropped as part of the rewrite.
-- **Intent router defaults flipped.** Ambiguous topical/knowledge questions now prefer `unstructured` (was `structured`). Indonesian few-shot examples added to the prompt.
-- **Cache management endpoints added:** `DELETE /api/v1/chat/cache`, `DELETE /api/v1/chat/cache/room/{id}`, `DELETE /api/v1/retrieval/cache/{user_id}`. Redis chat cache now stores `{response, sources}` (was just `response`) so cached replies repopulate `message_sources`.
-- **Direction.** The long-term split is **Python = agent/ML layer, Go = data plane**. More pieces are expected to follow doc ingestion out of Python.

PROGRESS.md

@@ -1,692 +0,0 @@
-# Progress — Phase 2 catalog-driven build
-
-Persistent tracker mirroring the 42-item ownership table in `REPO_CONTEXT.md` "Team — division of work". Update as PRs land. Future Claude Code sessions read this to know what's already done.
-
-**Last updated**: 2026-06-12 (Redis Cloud live; R3 closed as won't-do; R5 cache fix; AnalysisRecord persistence landed — `PostgresAnalysisStore` + `analysis_records` table)
-**Current open PR**: `pr/3` — active.
-
----
-
-## What just shipped (2026-06-12 — AnalysisRecord persistence, Rifqi)
-
-Groundwork for `generate_report`. The slow path now persists a real, citable
-record; the report (next) renders from it.
-
-- **Contract gaps closed** (`agents/slow_path/schemas.py`): `stage: CrispStage`
-  added to `TaskResult` + `TaskSummary` and populated at all 3 `TaskResult` build
-  sites in `task_runner.py` + copied in `assembler._build_record` — so the report
-  can group its method appendix by CRISP-DM phase. `AnalysisRecord` gained identity:
-  `record_id` (auto uuid), `analysis_id`/`user_id` (optional; stamped at persist).
-- **Real store** (`agents/slow_path/store.py`): `PostgresAnalysisStore` —
-  `save()` (never-throw, idempotent upsert) + `list_for_analysis()` (oldest-first,
-  the report's render order). `NullAnalysisStore` kept (tests / disabled persistence).
-  `AnalysisStore` Protocol gained `list_for_analysis`.
-- **Table** (`db/postgres/models.py`): `analysis_records` jsonb table (one row per
-  run, indexed by `analysis_id` + `user_id`); registered in `init_db.py`, created by
-  `create_all` on startup (no migration — `data_catalog` precedent).
-- **Wired** (`agents/chat_handler.py`): default store flipped to `PostgresAnalysisStore`;
-  `user_id` stamped onto the record at the save site (in scope there).
-- **Open**: `analysis_id` is `NULL` until Harry's Analysis State reaches the slow
-  path (session-ID handoff needed to group records per analysis).
-
----
-
-## Principal architecture review (2026-06-10) — findings + fix tracker
-
-A full external review (read the context docs + the slow path, tool layer, query
-spine, catalog plumbing, chat endpoint, config/connection layers) landed. It confirmed
-the DB-latency diagnosis and surfaced several gaps **not previously tracked here**.
-Verified against code before logging. Severity: **critical** / important / nice-to-have.
-
-**Runtime / latency (the original problem):**
-- DB connection handling is the anomaly, NOT cold start. `DbExecutor._run_sync`
-  (`db.py:192`) → `engine_scope` does `create_engine → connect (TCP+TLS+SCRAM) → 2×SET
-  → dispose` on EVERY query. Measured ~6–8s for 60 rows; a 2nd warm-session query was
-  still ~6.6s → per-call handshake, never amortized. `engine_scope`'s connect-once-dispose
-  semantics were designed for the ingestion pipeline and wrongly inherited by the query path.
-- `describe_source` ~3.5s is **planner-induced waste**: every few-shot (`examples.py`)
-  opens with a `describe_source` task, so the LLM always plans a tool that re-reads from
-  the catalog DB the same catalog already rendered into its prompt. Its impl does 2
-  sequential full-catalog reads (`data_access.py:127-128`). Total catalog reads/request ~5×.
-- Azure LLM clients rebuilt per request: `ChatHandler(enable_tracing=True)` is constructed
-  per request (`chat.py:172`) → fresh Orchestrator/Chatbot → fresh AzureChatOpenAI → fresh
-  TLS to Azure each call. Planner/Assembler correctly use module singletons; the other two don't.
-- Tokens (~13k/request) are NORMAL for this design — do not optimize for $.
-- **Reject the scheduled DB-warmer idea**: targets cold start (~1.8s slice) not the per-call
-  handshake, keeps serverless user DBs awake 24/7 (their compute bill), and decrypts every
-  tenant's creds on a cron (attack surface). Strictly dominated by an engine cache +
-  request-scoped pre-connect.
-
-**Fix tracker (new):**
-
-| # | Fix | Severity | Owner | Status |
-|---|---|---|---|---|
-| R1 | **AuthN/AuthZ** on data endpoints — reject body-supplied `user_id`/`room_id`, derive identity from a verified token. `/chat/stream` has none (`chat.py:40,128`); tenant isolation is client honesty. **CORRECTION to the review:** `security/auth.py` is a STUB (all `NotImplementedError`); the real JWT impl lives in `src/users/users.py` (`encode_jwt`/`decode_jwt`, HS, env-keyed) **but is unused** — `/login` (`api/v1/users.py`) returns the user profile as plain JSON and mints NO token. So R1 is cross-team: (1) `/login` must issue a JWT, (2) frontend must send it as `Bearer`, (3) data endpoints validate it. **Gates the engine-cache work (DB2).** | **critical** | DB/B + frontend | `[ ]` |
-| R2 | **Always compile a LIMIT** — `sql.py` now emits a bound for every query: explicit limit honored (clamped to `MAX_RESULT_ROWS=10000`), unbounded queries get `LIMIT cap+1` so an unbounded SELECT can't stream a whole table into memory. `CompiledSql.row_cap` carries the cap; `DbExecutor` caps + flags truncation from it (dropped its own `_ROW_HARD_CAP`). Tests updated (`test_sql.py`, +3 cases); `S608` restored to `tests/**` ruff ignore (was dropped). | **critical** | DB | `[x]` |
-| R3 | **Commit `tests/` + minimal CI** — `tests/` is gitignored; the 200+ tests cited as done exist only on laptops (already caused rename rot). ~~GitHub origin carries tests; HF Space gets the Docker build.~~ **2026-06-12: team decided tests stay gitignored/local — closed as won't-do.** | **critical (process)** | shared | `[won't do]` |
-| DB1 | **In-memory `describe_source`** (request-scoped `MemoizingCatalogReader`, `reader.py`) + **LLM-client hoist** (shared module-level `ChatHandler` in `chat.py`). Measured live: `describe_source` 3.5s→~2.0s (structured read now served from the planner's cached snapshot; only the unstructured read remains a round-trip), catalog reads/request ~5→~2. External `query_structured` handshake unchanged (DB2's job) so total slow path is ~flat until DB2. Tests: `tests/catalog/test_reader.py`. | important | agent | `[x]` |
-| DB2 | **Keyed engine cache** — `src/database_client/engine.py::UserEngineCache` (process singleton): pooled engines keyed by `client_id + creds-hash` (rotation auto-invalidates), bounded LRU (50) + 600s idle TTL, `pool_pre_ping` + `pool_recycle=300`. `DbExecutor._run_sync` reuses the warm connection instead of `create_engine→connect→dispose` per query (postgres/supabase only; other db_types keep the legacy path — no regression). **Live-measured: warm `query_structured` 6.6–9.4s → ~2.5s** (the residual is the per-call catalog-DB client fetch + pre-ping, not the external handshake). **Finding:** Neon's transaction pooler REJECTS `default_transaction_read_only` as a libpq startup `option` — caught live; moved read-only + statement_timeout to a per-connection `connect` event (best-effort; authoritative read-only is the SELECT-only compiler + sqlglot guard, see R10). Per-request ownership/active check kept. Proceeded ahead of R1 per owner decision (marginal security delta over the existing no-auth state; auth tracked separately). Tests: `tests/database_client/test_engine.py`. First query/process still cold → DB3. | important | DB | `[x]` |
-| DB3 | **Speculative pre-connect** — `DbExecutor.prewarm(catalog, user_id)` warms the pooled engine for schema sources (fire-and-forget at slow-path entry) so the cold first-query handshake overlaps the ~4s Planner call. Best-effort, never raises; gated to the default path (skipped when a coordinator factory is injected). Verified live through `ChatHandler.handle`. | nice-to-have | DB | `[x]` |
-| R4 | **Per-stage progress events** — `SlowPathCoordinator.run` gained an optional `progress` callback; `ChatHandler` bridges it to SSE `status` events (`chat.py` forwards them). Live: stream now shows `Planning…`→`Running N steps…`→`Composing…` (max wire gap ~4.6s, was ~13s of silence) → fixes proxy idle-timeout + UX. **Deferred:** token-streaming the Assembler answer needs splitting it into a streamed prose call + a structured-record call — that doubles the Assembler LLM calls (cost/latency), so it's a separate decision; the answer is still emitted as one chunk after the (fast ~2.5s) Assembler. Test: `test_chat_handler_wiring.py`. | important | agent | `[~]` |
-| R5 | **Response cache**: key on `user_id` + catalog version; invalidate on ingest. Was `chat:{room_id}:{message}`, 24h TTL, no user → cross-user replay + stale answers. **2026-06-12 (Rifqi):** key now `chat:{room_id}:{user_id}:{message}` via `_chat_cache_key()`, TTL 24h→1h (checkpoint decision) — urgent now that Redis is a shared Cloud instance. `DELETE /chat/cache` gained a required `user_id` param (frontend heads-up); room-wide clear pattern unchanged. **Still open:** catalog-version in key / invalidate-on-ingest. | important | B | `[~]` |
-| R6 | **Hard time budget** — wrap `coordinator.run()` in `asyncio.wait_for` (60–90s). `Constraints.time_budget_seconds` is rendered but not enforced. | important | agent | `[ ]` |
-| R7 | **Root-task-failure short-circuit** before the Assembler (templated/fast-path fallback, NOT replanning) — stops paying ~2k tok to narrate an empty RunState. | important | agent | `[ ]` |
-| R8 | **Catalog upsert race** — per-user advisory lock around read-merge-upsert (`store.py`); concurrent uploads can drop a source. | important | DB | `[ ]` |
-| R9 | **`extra="ignore"`** in `settings.py:15` (currently `allow` → typo'd env vars silently swallowed); require Azure keys in prod. | nice-to-have | B | `[ ]` |
-| R10 | **Read-only enforcement is session-state, not a server role.** `REPO_CONTEXT.md` counts "read-only DB credentials" as a defense layer but nothing requests/verifies a read-only role. Either request read-only creds at registration (verify via `SELECT current_setting(...)`) or drop the claim. | important | DB | `[ ]` |
-| R11 | **De-duplicate** `_PLACEHOLDER_RE` (`task_runner.py:31` vs validator) and `_DATA_ACCESS_TOOLS` (invoker vs planner registry) — import one from the other; comments aren't a sync mechanism. **TAB slice done (90e80f9):** canonical `DATA_ACCESS_TOOLS` now lives once in `tools/data_access.py`; `invoker.py` imports it (was a duplicated frozenset synced by comment). **Agent slice done (2026-06-10):** `PLACEHOLDER_RE` single-sourced in `planner/schemas.py` (part of the ToolCall placeholder convention); validator + task_runner import it. `planner/registry.py` keeps local spec *bodies* (stub pending KM-465 #4) but name-checks them against `DATA_ACCESS_TOOLS` in `_data_access_slice()` — upstream rename/add now raises at `default_registry()` instead of drifting silently. Registry output unchanged (same 12 tools, same order). | nice-to-have | agent/tool | `[x]` |
-| R12 | **Doc/process hygiene** — some code docstrings cite internal design specs that are not committed to the repo (design docs are kept out of version control), so the references dangle for anyone but the author; `CLAUDE.md` lists deleted modules (enricher, `pipeline/orchestrator.py`); `main` is 38 commits behind on a dead architecture. | nice-to-have | agent | `[ ]` |
-| R13 | **Pre-existing test failure** (found during R2, NOT caused by it): `tests/query/planner/test_prompt.py::test_render_catalog_with_sources` fails — `query/planner/prompt.py::render_catalog` now renders stable IDs (`src_test_db`) the test asserts are absent. Old query-planner path; confirmed failing on a clean tree. | nice-to-have | DB | `[ ]` |
-| T1 | **`input_schema` is presence-only, not type-checked** — `ToolSpec.input_schema` comment said "validates ToolCall.args", but `TaskRunner._validate_args` only enforces `required` presence; the `properties` types are documentation, never validated at runtime. Clarified the contract in `tools/contracts.py` so nobody assumes type-safety (a wrong-typed arg passes validation, surfaces only inside the compute fn). Doc-only, no behavior change (90e80f9). | nice-to-have | TAB | `[x]` |
-| T2 | **Dead Python embed path?** — `document_pipeline.process()` → `knowledge_processor` → `vector_store.aadd_documents()` still writes PDF/DOCX/TXT embeddings to `langchain_pg_embedding`, contradicting CLAUDE.md's "Go is sole writer, Python reads only". Verified the Go service (`Orchestrator-Agent-Service/internal/documents`) IS a complete ingestion writer to the same tables for all 5 file types (OCR + chunk + embed) → the Python embed branch is very likely redundant. **Blocked on one operational fact:** does the frontend still upload to `/document/process` (Python) or to Go? Park until confirmed — deleting a live ingestion path would break unstructured RAG. The csv/xlsx parquet branch stays regardless (feeds the catalog/tabular path). | nice-to-have | TAB | `[blocked]` |
-
-**Slow-path endpoint wiring (2026-06-10):** the Orchestrator→slow-path is now wired
-into the live endpoint behind an **env flag**. `settings.enable_slow_path` (env
-`ENABLE_SLOW_PATH`, default **off**) is passed to the shared `ChatHandler` in
-`api/v1/chat.py`. Flip `ENABLE_SLOW_PATH=true` to route `structured` intents through
-Planner→TaskRunner→Assembler and test end-to-end from `/chat/stream` (status progress
-events + answer stream). Stays opt-in because `BusinessContext` is still the stub;
-fast/unstructured paths unchanged. Verified live via `ChatHandler.handle`.
-
-**Architecture verdict:** fundamentally sound (catalog-driven IR + deterministic compiler
-+ static plan is the right call). Debt is transitional duplication (two planners/registries/
-contract modules — documented, owned) and `ChatHandler` drifting toward a god object
-(extract the slow-path composition root + the SSE `_build_sources`/`_normalize_chunks`
-mappers when convenient).
-
----
-
-## What just shipped (2026-06-09/10 — tool layer, tracing, slow-path wiring)
-
-Big stretch since the slow-path workers landed. The tool layer (teammate-owned) is
-now **complete and real**, the slow path is **wired into `ChatHandler` behind a gate**,
-and the whole chat pipeline is **traced**. Fast path still untouched; live behavior
-unchanged (flags default off).
-
-**Tool layer — COMPLETE (teammate, KM-624→630).** `src/tools/` was re-created (the
-2026-05-11 note about deleting it is superseded). Now teammate-owned:
-- `src/tools/analytics/` — the 8 **composite** `analyze_*` computes (descriptive,
-  aggregate, comparison, contribution, profile, correlation, segment, trend) +
-  prompt-style DESCRIPTIONs (KM-624/625).
-- `src/tools/contracts.py` — canonical `ToolSpec`/`ToolRegistry`/`ToolOutput` (KM-627).
-  `agents/planner/contracts.py` now just re-exports them + keeps the `BusinessContext`
-  stub (lead's).
-- `src/tools/registry.py::analytics_registry()` (KM-628); `src/tools/invoker.py` +
-  `src/tools/data_access.py` — `AnalyticsToolInvoker` (KM-629), `DataAccessToolInvoker`
-  + `CompositeToolInvoker` (KM-630). All never-throw. **Pattern A confirmed** (`analyze_*`
-  take a `data` `${t<id>}` placeholder from an upstream `query_structured`).
-- **Verified live E2E (2026-06-09):** real `query_structured` against a user's Neon
-  Postgres → `analyze_trend` → Assembler. `analyze_contribution` surfaced a real tool
-  bug (Decimal vs float in `decomposition.py`) — degrade-and-continue held; **now fixed
-  by the tool owner** (`_coerce_decimals` in `invoker._materialize`, KM-630 / commit
-  1195870), so the whole `analyze_*` family is covered in one place. **Directive:** agent
-  side does NOT modify `src/tools/` without confirmation.
-
-**Planner — realigned to the real tools (KM-626).** `registry.py::default_registry()`
-composes the real `analytics_registry()` + a local stub for the 4 data-access tools.
-Few-shots grown to **A–D**: A `analyze_contribution`, B `analyze_trend`, C mixed
-structured+unstructured (`retrieve_documents`, independent branch), D `analyze_aggregate`.
-`parallelizable_with` **removed** from `Task` (schema/validator/examples/prompt) —
-TaskRunner derives parallelism from `depends_on` alone.
-
-**Slow-path wiring — built, GATED OFF (KM-626).** `agents/chat_handler.py` gains a
-`structured→slow` branch behind `ChatHandler(enable_slow_path=False)`: when on it builds
-a per-request `CompositeToolInvoker` (composition root) + `SlowPathCoordinator`, streams
-`chat_answer`, persists the `analysis_record`. Two seams isolate the remaining blockers:
-- `agents/planner/business_context.py::get_business_context(user_id)` — async stub
-  `BusinessContext`; TODO(lead) swap for the real read.
-- `agents/slow_path/store.py` — `AnalysisStore` Protocol + `NullAnalysisStore` (logs
-  only). Real store = `analysis_records` table in the catalog DB (Neon `dataeyond`) —
-  **table not created yet**. `chat_answer` still emitted as one chunk (not token-streamed).
-
-**Observability — Langfuse tracing wired (KM-631).** `src/observability/langfuse/
-tracing.py` — `RequestTracer`/`NullTracer`/`TracingToolInvoker` + `_redact`. One trace
-per request groups Orchestrator.classify, Planner.plan (each retry = its own generation),
-Assembler.assemble, Chatbot.astream + tool spans (latency/metadata only). Gated:
-`ChatHandler(enable_tracing=False)`; `api/v1/chat.py` opts in (`=True`). PII policy:
-Orchestrator+Planner unmasked (question + PII-safe summary); Assembler+Chatbot masked
-(see real rows/chunks); tool spans carry name + arg keys + row count only. Zero added
-LLM tokens; verified live to US Cloud.
-
-**Live evals green (2026-06-09, real Azure 4o):** `RUN_PLANNER_EVAL=1` and
-`RUN_SLOW_PATH_EVAL=1` both pass — Planner emits valid catalog-consistent `QueryIR` and
-wires Pattern A correctly; self-corrects via retry.
-
-**Open follow-ups:** real `BusinessContext` (lead); create `analysis_records` table +
-real `AnalysisStore` (**Rifqi owns, 2026-06-12** — folded into `generate_report` work,
-see `CHECKPOINT_PLAN_2026-06-17.md`); register data-access `ToolSpec`s upstream (`data_access_registry()`)
-or keep the planner stub; 4o → GPT-mini deployment swap; flip `enable_slow_path` on once
-`BusinessContext` is real. NOTE: 3 test files pre-existing broken from rename rot
-(`test_chat_handler.py`, `test_intent_router.py`, `test_answer_agent.py` import the old
-`answer_agent`/`intent_router` module names).
-
----
-
-## What just shipped (2026-06-10 — TAB: tool-layer hardening + DRY)
-
-Owner-side companion to the agent block above. After the live E2E surfaced real-data
-edge cases, the tool layer got a round of correctness hardening. All in TAB-owned paths
-(`src/tools/`, `src/catalog/`); no agent-side or API change.
-
-**JSON-safety across the `analyze_*` family.** Real DB rows carry scalar types that
-don't survive the jsonb / SSE round-trip:
-- `[KM-630] coerce DB Decimal → float` (commit 1195870) — `_coerce_decimals` in
-  `invoker._materialize` converts object-columns holding `decimal.Decimal` (asyncpg
-  returns NUMERIC as `Decimal`) to `float64` before any compute runs. Fixes the
-  `float + Decimal` TypeError in `decomposition.analyze_contribution` **and** the whole
-  family in one seam — only touches columns that actually contain a `Decimal`.
-- `[KM-624] non-JSON-safe scalars in mode & top_value` (commit 6981ed3) — normalize
-  numpy / non-native scalars so descriptive + top-value outputs serialize cleanly.
-
-**Planner↔Tools registry alignment + Timestamp keys** (commit 4bb7623, `fix(tools)`):
-- `registry.py` — `analyze_descriptive.required` corrected `["data"]` → `["data",
-  "column_ids"]` to match the compute signature (`column_ids` has no default). Prevents
-  the Planner from emitting a call that's missing a required arg. `analyze_profile` stays
-  `["data"]` (its `column_ids` defaults to `None`).
-- `aggregation._clean` — group-by over a datetime column produced `pd.Timestamp` group
-  keys that aren't JSON-safe; now normalized to `.isoformat()` alongside the existing
-  numpy `.item()` branch.
-
-**DRY: single `SAMPLE_LIMIT` constant** (commit 6d46ba5, `[NOTICKET] refactor(catalog)`):
-- One source of truth in `catalog/introspect/base.py` (`SAMPLE_LIMIT = 3`, down from 5 —
-  token cost: sample values feed the planner prompt). Both introspection paths import it:
-  `catalog/introspect/tabular.py` and `pipeline/db_pipeline/extractor.py` (which dropped
-  its own local `= 3`). Dependency direction is pipeline→catalog (no circular import).
-  Stale test `test_sample_values_capped_at_five` updated to assert the real cap (3).
-
-**Audit result:** Planner↔Tools arg alignment swept end-to-end — 7/8 `analyze_*` tools
-already matched; the 1 mismatch (`analyze_descriptive`) is the fix above. Pattern A holds
-across all of them.
-
----
-
-## What just shipped (2026-06-08 — KM-626: slow-path agent layer)
-
-The rest of the slow path after the Planner (KM-567) — TaskRunner, Assembler, and
-the coordinator. Built and tested against
-mocks; **not yet wired into the live `ChatHandler`** (waits on the tool team's real
-`ToolInvoker` + a real `BusinessContext`). Fast path untouched.
-
-**Naming:** "Orchestrator" = the entry dispatcher only (`agents/orchestration.py`).
-The slow-path **workers** live in **`agents/slow_path/`** — deliberately NOT named
-"orchestrator".
-
-**Files added** (`src/agents/slow_path/`):
-- `schemas.py` — `TaskResult`, `RunState`; `TaskSummary`, `AnalysisRecord`,
-  `AssembledOutput`, `AssemblerNarrative`. Reuses `ToolOutput`.
-- `invoker.py` — `ToolInvoker` Protocol only; the tool team owns the impl (KM-418).
-- `errors.py` — `SlowPathError`, `AssemblerError`.
-- `task_runner.py` — deterministic, 0 LLM: wave-based execution, `${t<id>}` placeholder
-  resolution, internal `validate_args`, never-throw invoke, status labeling,
-  degrade-and-continue → `RunState`.
-- `assembler.py` + `prompt.py` + `config/prompts/assembler.md` — single LLM call →
-  `AssemblerNarrative`; code merges with `RunState` to build the `AnalysisRecord`
-  (structured fields copied, never re-authored).
-- `coordinator.py` — `SlowPathCoordinator`: Planner → TaskRunner → Assembler.
-
-**Tests added** (`tests/agents/slow_path/`, 12 passing; gitignored): schema round-trips
-+ chat_answer-first; runner happy/placeholder/parallel/degrade/arg-miss; assembler
-narrative-vs-snapshot + question threading; coordinator end-to-end. `ruff` clean;
-tool-agnostic (no `src/tools/*` import).
-
-**Open follow-ups (not blockers):** wire `SlowPathCoordinator` into the expanded
-Orchestrator/`ChatHandler` once the real invoker + `BusinessContext` exist; swap the
-test `MockToolInvoker` for the tool team's real one (zero agent change, INV-7); 4o →
-GPT-mini deployment swap.
-
----
-
-## What just shipped (2026-06-08 — tool taxonomy + ownership revision)
-
-Team decisions after the teammate pushed KM-624 (`src/tools/analytics/`):
-
-- **Composite tools, not atomic.** v1 uses **composite "family" tools** (`analyze_*`),
-  not the atomic `compute_*` set the earlier draft assumed. One `analyze_*` call does a
-  whole analytical job (e.g. `analyze_descriptive` subsumes median/mode/stddev/percentile;
-  `analyze_trend` subsumes `date_trunc`). Tool-taxonomy decision recorded.
-- **Tool team owns ALL tools** — compute, data-access (`query_structured`,
-  `retrieve_documents`, `list_sources`, `describe_source`), the wrapper/invoker layer
-  (KM-418), and **all tool tests**. The agent team owns nothing below the registry contract.
-- **Planner stub realigned to the real tools.** `registry.py` rewritten from the 9 atomic
-  entries to **12 composite entries** (4 data-access + 8 `analyze_*`); `examples.py`
-  rewritten (Example A → `analyze_contribution`, Example B → `analyze_trend`); `planner.md`
-  bullet updated; planner tests updated. 32 passing + 1 gated, `ruff` clean.
-- **Open (tool team's call):** Pattern A (analyze_* take a `${t<id>}` `data` placeholder
-  from an upstream `query_structured`) vs Pattern B (self-fetch by `source_id`). Stub
-  assumes A; reshaped to match once decided (agent code unaffected, INV-7).
-- **New coupling:** the tool team's `query_structured`/`retrieve_documents` are expected
-  to call our existing `QueryService`/`RetrievalRouter`; `query_structured` stays
-  inline-`QueryIR` so `IRValidator` still applies. Interface to coordinate.
-
-**Next (our scope, all mock-able now):** TaskRunner + Assembler against a `MockToolInvoker`,
-then Orchestrator slow-path wiring. Stubs still to retire on integration: `contracts.py`
-(BusinessContext from lead; ToolSpec/ToolRegistry/ToolOutput from tool team) and `registry.py`
-(real registry from tool team). Infra: swap the 4o stand-in for a GPT-mini deployment.
-
----
-
-## What just shipped (2026-06-05 — Phase 3: Planner agent)
-
-First slow-path agent (the Planner). A single LLM
-call turns BusinessContext + Catalog + ToolRegistry + question + Constraints into a
-validated, **static** `TaskList` (DAG of fully-specified tool-call chains). No
-replanning (INV-6); tool-agnostic against a registry contract (INV-7). Fast path
-(`agents/orchestration.py`, `agents/chatbot.py`, `query/`) untouched.
-
-**Files added** (`src/agents/planner/`):
-- `contracts.py` — **STUB** Pydantic contracts pending reconciliation: `BusinessContext`
-  (+KeyTerm/DataTableNote/DataColumnNote, lead's), `ToolSpec`/`ToolRegistry` (tool
-  team KM-608), `ToolOutput` envelope.
-- `schemas.py` — `CrispStage`, `ToolCall`, `Task`, `TaskList`. No replan schemas.
-- `inputs.py` — `CatalogSummary` (condensed, PII `sample_values` nulled, `from_catalog`
-  builder + `render`) and `Constraints` (max_tasks=5, modeling_allowed=False).
-- `registry.py` — **STUB** v1 P0 registry: query_structured, retrieve_documents,
-  list_sources, describe_source, compute_median/stddev/percentile/mode, date_trunc.
-- `errors.py` — `PlannerError`, `PlannerValidationError`.
-- `prompt.py` + `config/prompts/planner.md` — system prompt (INV-1/6/7 + principles) +
-  per-call human content (context + catalog + tools + constraints + few-shots + question).
-- `examples.py` — two few-shots (A exploratory revenue-by-category; B descriptive
-  monthly-trend-by-region with date_trunc), built from the real `TaskList` schema.
-- `validator.py` — `PlannerValidator` running the 8 checks; reuses the existing
-  `IRValidator` for inline `query_structured` IRs.
-- `service.py` — `PlannerService` + `plan_analysis(...)`: chain (mirrors
-  `query/planner/service.py`) + validate-and-retry loop (max 3, mirrors `QueryService`).
-
-**Tests added** (`tests/agents/planner/`, 30 passing + 1 gated): `test_schemas.py`,
-`test_inputs.py`, `test_validator.py` (one failure per check + happy paths),
-`test_service.py` (`_FakeChain` + retry), `test_golden_questions.py` (live eval gated on
-`RUN_PLANNER_EVAL=1`). `ruff check` clean on planner paths.
-
-**Open follow-ups (not blockers):** reconcile `BusinessContext` with the lead and
-`ToolRegistry`/`ToolSpec` + real tools with teammate (KM-608); "GPT mini" currently uses
-the configured 4o deployment (swap `azure_deployment` when a mini deployment exists). Next:
-Orchestrator slow-path expansion + TaskRunner + Assembler.
-
----
-
-## Legend
-
-- `[x]` done and merged
-- `[~]` in progress (open PR or active branch)
-- `[ ]` not started
-- **DB** / **TAB** / **B** — ownership (from REPO_CONTEXT.md)
-
----
-
-## PR sequence
-
-| PR | Status | Owner(s) | Scope |
-|---|---|---|---|
-| PR1 | `[x]` merged | DB | Contract locks + catalog plumbing + DB introspector + IR validator + tests |
-| PR1-tab | `[x]` shipped | TAB | Tabular introspector + on_tabular_uploaded trigger + 31 unit tests |
-| PR2a | `[x]` merged | DB | CatalogEnricher + StructuredPipeline + on_db_registered trigger + FK extension on Table (enricher later removed in KM-557) |
-| KM-557 | `[x]` shipped | DB | Drop CatalogEnricher entirely (cost cut — planner uses stats + sample rows directly); rename jsonb table `catalogs` → `data_catalog`; add `GET /api/v1/data-catalog/{user_id}` index endpoint for catalog refresher |
-| PR2b | `[x]` shipped | DB-solo (B-review) | IntentRouter + planner prompt + planner LLM service |
-| PR3-DB | `[x]` shipped | DB | SqlCompiler (Postgres) + DbExecutor (sqlglot guard, RO + statement_timeout, asyncio.to_thread) + 36 golden IR→SQL tests |
-| PR3-TAB | `[x]` shipped | TAB | PandasCompiler + TabularExecutor + 43+12 golden IR→DataFrame tests |
-| PR4 | `[x]` | DB-solo (B-review) | ExecutorDispatcher + QueryService + ChatHandler module. **API rewired in Cleanup PR.** |
-| PR5 | `[x]` shipped | DB-solo (B-review) | Retry/self-correction loop on validation failure (lives in QueryService, max 3 attempts, planner re-prompted with prior error) |
-| PR6 | `[~]` scaffold | DB-solo (B-review) | Eval harness scaffold + 3 DB-targeting golden cases. Skipped without `RUN_PLANNER_EVAL=1` env. TAB extends with tabular cases. |
-| PR7 | `[x]` | DB-solo (B-review) | `ChatbotAgent` (renamed from `AnswerAgent`) + chatbot_system + guardrails prompts. `answer_agent.py` → `chatbot.py`, `AnswerAgent` → `ChatbotAgent`. API rewired in Cleanup PR. |
-| Cleanup | `[x]` | B | ChatHandler wired to chat.py; Phase 1 dual-write dropped from /ingest; on_catalog_rebuild_requested + POST /data-catalog/rebuild; dead modules deleted (chatbot Phase 1, orchestrator, query/base, knowledge.py, config/agents/); retrieval cache restored via RetrievalRouter; top_values added to ColumnStats; lifespan migration; knowledge_router removed. |
-
----
-
-## All items
-
-### Contracts (B — shared)
-
-| # | Item | Status | Notes |
-|---|---|---|---|
-| 1 | Catalog Pydantic models (`catalog/models.py`) | `[x]` | PR1 added `location_ref` URI-scheme docstring; PR2a added `ForeignKey` model + `Table.foreign_keys` field |
-| 2 | IR Pydantic models (`query/ir/models.py`) | `[x]` | Pre-existing scaffold |
-| 3 | IR operator whitelists (`query/ir/operators.py`) | `[x]` | PR1 filled `TYPE_COMPATIBILITY` matrix |
-| 4 | PII patterns / regex (`security/pii_patterns.py`) | `[x]` | Pre-existing |
-| — | `data_catalog` Postgres jsonb table (`db/postgres/models.py`) | `[x]` | PR1 added `Catalog` SQLAlchemy class + `init_db.py` import. KM-557 renamed `__tablename__` from `catalogs` → `data_catalog`; created fresh (no migration) |
-| — | `QueryResult` shape (`query/executor/base.py`) | `[x]` | Pre-existing scaffold; `columns: list[str]` added (TAB owner, PR1-tab) — DbExecutor updated to populate it. |
-| — | `Source.location_ref` URI scheme | `[x]` | PR1 documented in `catalog/models.py` docstring |
-
-### Ingestion — introspection
-
-| # | Item | Owner | Status | Notes |
-|---|---|---|---|---|
-| 5 | DB introspector (`catalog/introspect/database.py`) | DB | `[x]` | PR1 — reuses Phase 1 `database_client_service`, `db_credential_encryption`, `db_pipeline_service.engine_scope`, `extractor.get_schema/profile_column/get_row_count`. PR2a wired FK extraction (was discarded before). |
-| 6 | Tabular introspector (`catalog/introspect/tabular.py`) | TAB | `[x]` | PR1-tab — downloads original blob (CSV/XLSX/Parquet), one Table per sheet (XLSX) or one Table (CSV/Parquet). `source_id = document_id`. `fetch_doc`/`fetch_blob` injectable for unit tests (no Settings). **2026-06-10**: sample cap now imports the shared `SAMPLE_LIMIT` (=3) from `catalog/introspect/base.py` — single source of truth across the tabular + DB introspection paths (commit 6d46ba5). |
-| 7 | `BaseIntrospector` ABC (`catalog/introspect/base.py`) | B | `[x]` | Pre-existing; signature locked |
-
-### Ingestion — shared catalog plumbing
-
-| # | Item | Owner | Status | Notes |
-|---|---|---|---|---|
-| 8 | ~~Catalog enricher + prompt~~ | B | **REMOVED in KM-557** | Cost optimization — planner reads stats + sample rows + column names directly. `catalog/enricher.py` + `config/prompts/catalog_enricher.md` deleted. `render_source` (the only piece still needed) moved to `src/catalog/render.py`. Tests moved to `tests/catalog/test_render.py`. |
-| 9 | Catalog validator (`catalog/validator.py`) | B | `[x]` | PR1 (DB owner picked up) — uniqueness invariants |
-| 10 | Catalog store — Postgres jsonb (`catalog/store.py`) | B | `[x]` | PR1 (DB owner picked up) — `INSERT ... ON CONFLICT` |
-| 11 | Catalog reader (`catalog/reader.py`) | B | `[x]` | PR1 (DB owner picked up) — filters by source_hint, empty on miss |
-| 12 | PII detector (`catalog/pii_detector.py`) | B | `[x]` | PR1 (DB owner picked up) — name + value matching, bias toward over-flag |
-
-### Ingestion — pipelines
-
-| # | Item | Owner | Status | Notes |
-|---|---|---|---|---|
-| 13 | Structured pipeline (`pipeline/structured_pipeline.py`) | B | `[x]` | PR2a (DB owner) — Source-type-agnostic: caller supplies the introspector. `default_structured_pipeline()` factory wires production deps lazily so tests can inject mocks without `Settings()` construction. **KM-557**: enrich step removed; pipeline is now `introspect → merge with existing → validate → upsert`. Constructor no longer takes `enricher`. |
-| 14 | Triggers (`pipeline/triggers.py`) | B | `[x]` | PR2a — `on_db_registered` implemented (DB owner). PR1-tab — `on_tabular_uploaded` implemented (TAB owner). **2026-05-11** — `on_document_uploaded` implemented. **2026-05-12** — `on_catalog_rebuild_requested` implemented: iterates all Sources in current catalog, re-runs `on_db_registered` (schema) or `on_tabular_uploaded` (tabular) per source; per-source errors logged but don't abort. |
-| 15 | Ingestion orchestrator (`pipeline/orchestrator.py`) | B | **DELETED** | Redundant stub — `StructuredPipeline` already takes introspector at run() time. Deleted in Cleanup PR. |
-| 16 | Document pipeline (`pipeline/document_pipeline.py`) | TAB | `[x]` | Flattened `pipeline/document_pipeline/document_pipeline.py` (folder) → `pipeline/document_pipeline.py` (file). Updated import in `api/v1/document.py`. |
-
-### Query — shared spine
-
-| # | Item | Owner | Status | Notes |
-|---|---|---|---|---|
-| 17 | IR validator (`query/ir/validator.py`) | B | `[x]` | PR1 (DB owner) — full rule set; descriptive errors for planner retry |
-| 18 | Planner LLM service (`query/planner/service.py`) | B | `[x]` | PR2b — Azure OpenAI structured output → `QueryIR`. Injectable chain. Supports retry via `previous_error` argument. |
-| 19 | Planner prompt (`query/planner/prompt.py`, `config/prompts/query_planner.md`) | B | `[x]` | PR2b — system prompt with hard constraints + few-shot for DB and tabular sources. `build_planner_prompt(question, catalog, previous_error)` calls `catalog.render.render_source` (renamed from `catalog.enricher.render_source` in KM-557). |
-| 20 | Intent router (`agents/orchestration.py` — class `OrchestratorAgent`; `config/prompts/intent_router.md`) | B | `[x]` | PR2b — single LLM call → `IntentRouterDecision(needs_search, source_hint, rewritten_query)`. Supports conversation history. **NOTE**: source filename + class name were kept from Phase 1 for import-site compatibility; only the body is Phase 2. Prompt file and test file use the `intent_router` name. |
-| 21 | Executor base + `QueryResult` (`query/executor/base.py`) | B | `[x]` | Pre-existing scaffold |
-| 22 | Executor dispatcher (`query/executor/dispatcher.py`) | B | `[x]` | PR4 — picks DbExecutor / TabularExecutor by `source.source_type`. Lazy imports of production executors keep import side-effect-free for tests. Caches per source_type. |
-| 23 | Compiler base ABC (`query/compiler/base.py`) | B | `[x]` | Pre-existing scaffold |
-| 24 | Top-level QueryService (`query/service.py`) | B | `[x]` | PR4+5 — `plan → validate → dispatch → execute → QueryResult`. Retry loop on validation failure (max 3, planner re-prompted with prior error). Catches NotImplementedError from TabularExecutor placeholder gracefully. Never raises. |
-
-### Query — DB path
-
-| # | Item | Status | Notes |
-|---|---|---|---|
-| 25 | SQL compiler (`query/compiler/sql.py`) | `[x]` | PR3-DB — Postgres dialect (Supabase reuses); deterministic IR → (sql, named-params dict); double-quoted identifiers from catalog; all whitelisted ops (=, !=, <, <=, >, >=, in, not_in, is_null, is_not_null, like, between); alias-aware order_by; `CompiledSql.params: dict[str, Any]` (changed from `list`). MySQL/BigQuery/Snowflake compilers later. |
-| 26 | DB executor (`query/executor/db.py`) | `[x]` | PR3-DB — sync engine via `db_pipeline_service.engine_scope` inside `asyncio.to_thread`. sqlglot SELECT-only / no-DML guard. Postgres-only session settings: `default_transaction_read_only=on` + `statement_timeout=30000`. asyncio.wait_for backstop. Never raises — populates `QueryResult.error`. 10k row hard cap. |
-| 27 | Credential encryption (`security/credentials.py`) | `[ ]` | Stub exists; PR1 reused Phase 1 `utils/db_credential_encryption.py` instead. Move in cleanup PR |
-| 28 | User-DB connection management | `[x]` | PR3-DB reused Phase 1 `db_pipeline_service.engine_scope` (same as PR1 introspector); no new helper needed |
-
-### Query — Tabular path
-
-| # | Item | Status | Notes |
-|---|---|---|---|
-| 29 | Pandas compiler (`query/compiler/pandas.py`) | `[x]` | PR3-TAB — `CompiledPandas` dataclass; all 12 filter ops; all 6 aggs; group_by via `pd.concat` of Series; alias-aware order_by; `_like_to_regex` (`%`→`.*`, `_`→`.`); pure module-level helpers. (`polars` for large files still deferred — see Planned dependencies.) |
-| 30 | Tabular executor (`query/executor/tabular.py`) | `[x]` | PR3-TAB — `fetch_blob` injectable for tests; blob path: single-table → `{uid}/{did}.parquet`, multi-table → `{uid}/{did}__{table.name}.parquet`; `asyncio.to_thread`; 10k row hard cap; errors → `QueryResult.error`. Dispatcher routes to it by `source_type`. |
-| 31 | Parquet upload/download wrapper | `[x]` | Moved `knowledge/parquet_service.py` → `storage/parquet.py`. Updated 4 import sites: `pipeline/document_pipeline.py`, `knowledge/processing_service.py`, `query/executor/tabular.py`, `query/executors/tabular.py`. |
-
-### Agents + chat
-
-| # | Item | Status | Notes |
-|---|---|---|---|
-| 32 | Chatbot agent + prompt (`agents/chatbot.py`, `config/prompts/chatbot_system.md`) | `[x]` | PR7-bundle — `ChatbotAgent` (was `AnswerAgent`) streams tokens, accepts `QueryResult` or list[`DocumentChunk`] or neither. **Cleanup PR**: renamed `answer_agent.py` → `chatbot.py`, `AnswerAgent` → `ChatbotAgent`; Phase 1 `agents/chatbot.py` deleted. |
-| 33 | Guardrails prompt (`config/prompts/guardrails.md`) | `[x]` | PR7-bundle — appended to `chatbot_system.md` so guardrails take precedence in conflict. |
-| — | Chat handler / orchestrator (`agents/chat_handler.py`) | `[x]` | PR4-bundle — top-level Phase 2 orchestrator. Routes by `source_hint`: chat → AnswerAgent direct; structured → CatalogReader + QueryService; unstructured → DocumentRetriever placeholder + AnswerAgent. Yields `intent` / `chunk` / `done` / `error` SSE-style events. Phase 1 chat.py NOT touched — cleanup PR rewires the API to call this. **2026-06-09**: gained the gated `structured→slow` branch (`enable_slow_path=False`) + `enable_tracing` (KM-626/631). |
-
-### Tools — slow-path "Tools" component (TAB)
-
-New scope after the original 42-item table; added as the tool layer landed (KM-608/624–631). All TAB-owned (`src/tools/`), all never-throw.
-
-| # | Item | Owner | Status | Notes |
-|---|---|---|---|---|
-| — | Analytics compute fns (`tools/analytics/`) | TAB | `[x]` | KM-608/624/625 — 8 **composite** `analyze_*` fns (descriptive, aggregate, comparison, contribution, profile, correlation, segment, trend) + prompt-style DESCRIPTIONs. Pure pandas, no I/O. JSON-safe outputs (numpy/Decimal/Timestamp normalized — KM-624 + commit 4bb7623). |
-| — | Tool contracts (`tools/contracts.py`) | TAB | `[x]` | KM-627 — canonical `ToolSpec` / `ToolRegistry` / `ToolOutput`. `agents/planner/contracts.py` re-exports them (+ keeps the lead's `BusinessContext` stub). |
-| — | Analytics registry (`tools/registry.py`) | TAB | `[x]` | KM-628 — `analytics_registry()`. `analyze_descriptive.required` = `["data","column_ids"]` (aligned to compute signature, commit 4bb7623). |
-| — | Invoker layer (`tools/invoker.py`) | TAB | `[x]` | KM-629 — `AnalyticsToolInvoker` (Pattern A: `analyze_*` take a `data` `${t<id>}` placeholder from upstream `query_structured`; `_materialize` → DataFrame, `_coerce_decimals` covers the whole family) + `CompositeToolInvoker` (routes data-access vs analytics by name). |
-| — | Data-access tools (`tools/data_access.py`) | TAB | `[x]` | KM-630 — `DataAccessToolInvoker`: `list_sources` / `describe_source` / `query_structured` / `retrieve_documents`. Per-request DI (`user_id` + `CatalogReader`). `query_structured` calls `IRValidator` + `ExecutorDispatcher` (planner skipped — IR pre-built by the agent Planner). **Superseded by KM-642/643** — renamed `data_retrieve`/`knowledge_retrieve` and `list_sources`+`describe_source` merged into `data_check` + new `knowledge_check`; see row below. |
-| — | Tool tests (`tests/unit/tools/`) | TAB | `[x]` | analytics + data-access + invoker tests (gitignored). Incl. regression `test_decimal_columns_coerced_for_analyze_contribution`. |
-| — | Data/knowledge tool taxonomy (`tools/data_access.py`) | TAB | `[x]` | KM-642/643 (commits c38c0c2, 4bd5f1e) — renamed `query_structured`→`data_retrieve`, `retrieve_documents`→`knowledge_retrieve`; merged `list_sources`+`describe_source` → parameterized `data_check` (no arg = list structured sources; `source_id` = that source's schema) + new `knowledge_check` (unstructured/documents). Split mirrors the catalog's structured/unstructured slices. Planner stub/prompt/validator/few-shots synced; `DATA_ACCESS_TOOLS` guard kept in lockstep. Note: dated log entries above (e.g. the 2026-06-09 E2E) keep the old names as historical record. |
-
-### API surface
-
-| # | Item | Owner | Status | Notes |
-|---|---|---|---|---|
-| 34 | DB client endpoints (`api/v1/db_client.py`) | DB | `[x]` | **Cleanup PR** — `/ingest` now calls only `on_db_registered`. Phase 1 `db_pipeline_service.run()` + `decrypt_credentials_dict` removed. Error from catalog build now raises HTTP 500 (was silent log). Response simplified to `{"status": "success", "client_id": ...}`. |
-| 35 | Document/tabular upload endpoints (`api/v1/document.py`) | TAB | `[x]` | Rewired `/document/process` — after processing CSV/XLSX, calls `on_tabular_uploaded(document_id, user_id)`. Catalog ingestion failure is logged but does not fail the request. **2026-05-11** — CSV/XLSX no longer ingested to vector store (`knowledge_processor` skipped for tabular types in `document_pipeline.py`); they go to catalog only. |
-| 36 | Chat stream endpoint (`api/v1/chat.py`) | B | `[x]` | Rewired `/chat/stream` — replaced `query_executor.execute()` (Phase 1) with `CatalogReader + QueryService` (Phase 2). **Cleanup PR**: fully rewired to `ChatHandler.handle()`. Inline intent routing, retrieval, and answer generation removed. Redis cache, fast intent, history loading, and message persistence remain in chat.py. Sources event emits `[]` (retrieval not yet exposed by ChatHandler). |
-| 37 | Room / users endpoints (`api/v1/room.py`, `api/v1/users.py`) | B | `[ ]` | No catalog work; only touch if auth flow changes |
-| — | Data catalog index endpoint (`api/v1/data_catalog.py`) | DB | `[x]` | **KM-557** — `GET /api/v1/data-catalog/{user_id}` → `list[CatalogIndexEntry]`. **Cleanup PR** — added `POST /api/v1/data-catalog/rebuild?user_id=` → calls `on_catalog_rebuild_requested`; per-source errors logged but don't fail the request. |
-
-### Tests + eval
-
-| # | Item | Owner | Status | Notes |
-|---|---|---|---|---|
-| 38 | DB compiler golden tests (`tests/query/compiler/test_sql.py`) | DB | `[x]` | PR3-DB — 36 tests across all whitelisted ops, identifier quoting, agg / count_distinct / count(*), order_by alias resolution, parameter sequencing, error paths. Pure-Python, no LLM, no DB. |
-| 39 | Pandas compiler golden tests (`tests/unit/query/compiler/test_pandas_compiler.py`) | TAB | `[x]` | PR3-TAB — 43 tests: all 12 filter ops, all 6 aggs, group_by, order_by, limit, aliases, empty DataFrame, error paths. `test_tabular_executor.py` adds 12 more (blob name resolution + happy path + error paths). |
-| 40 | IR validator tests (`tests/query/ir/test_validator.py`) | B | `[x]` | PR1 — 19 tests, all rules covered |
-| — | PII detector tests (`tests/catalog/test_pii_detector.py`) | B | `[x]` | PR1 — 26 tests (parametrized) |
-| — | Catalog validator tests (`tests/catalog/test_validator.py`) | B | `[x]` | PR1 — 5 tests |
-| — | Catalog render tests (`tests/catalog/test_render.py`) | B | `[x]` | **KM-557** — 5 tests (renamed from `test_enricher.py`; LLM enrichment tests dropped, render-only tests kept). |
-| — | Catalog store integration test (`tests/catalog/test_store.py`) | DB | `[x]` | PR1 — module-level skip without `RUN_INTEGRATION_TESTS=1` |
-| — | DB introspector test | DB | `[ ]` | Deferred to PR2 — needs Postgres testcontainer or fixture infra |
-| — | Tabular introspector test | TAB | `[x]` | PR1-tab — 31 unit tests (CSV/XLSX/Parquet, stats, PII, error paths). No DB/blob I/O — mocks injected via constructor. |
-| 41 | Planner eval (`tests/query/planner/`) | B | `[x]` | PR6-scaffold — `test_golden_questions.py` with 3 DB-targeting cases. TAB added `test_golden_tabular.py` with 4 tabular cases (group_by+sum, top-N+limit, date range filter, XLSX sheet selection). All 4 passed against real Azure OpenAI. Fix shipped alongside: `query/planner/service.py` replaced `("system", text)` tuple with `SystemMessage` — without this, `{...}` in `query_planner.md` was parsed as f-string variables and crashed on every real invocation. |
-| 42 | E2E smoke tests (`tests/e2e/`) | B | `[ ]` | Defer until Phase 2 endpoints are wired (cleanup PR). Component-level orchestration is already covered by `test_chat_handler.py` + `test_service.py`. |
-| — | Golden IR fixtures (`tests/fixtures/golden_irs.json`) | B | `[~]` | PR1 seeded with 5 DB-targeting examples; TAB extends in PR1-tab |
-| — | Shared `sample_catalog` fixture (`tests/conftest.py`) | B | `[x]` | PR1 — DB-shaped; TAB may add tabular sibling |
-
----
-
-## What just shipped (2026-05-12 — Cleanup PR)
-
-**Phase 1 removal + Phase 2 API rewiring:**
-- `src/api/v1/chat.py` — fully rewired to `ChatHandler.handle()`. Removed inline IntentRouter, retrieval, and ChatbotAgent calls. Redis cache, fast intent, load_history, save_messages stay in chat.py.
-- `src/api/v1/db_client.py` — `/ingest` now calls only `on_db_registered`. Phase 1 `db_pipeline_service.run()` block removed. Catalog build failure now raises HTTP 500.
-- `src/api/v1/data_catalog.py` — added `POST /api/v1/data-catalog/rebuild` endpoint.
-- `src/pipeline/triggers.py` — `on_catalog_rebuild_requested` implemented: iterates catalog sources, re-runs the appropriate trigger per source type, per-source errors logged.
-
-**Dead modules deleted:**
-- `src/agents/chatbot.py` (Phase 1 LangChain chatbot)
-- `src/pipeline/orchestrator.py` (empty stub)
-- `src/query/base.py` (old duplicate of `executor/base.py`)
-- `src/api/v1/knowledge.py` (fake `/knowledge/rebuild` endpoint)
-- `src/config/agents/` (folder — prompts only used by deleted Phase 1 chatbot)
-
-**Renames:**
-- `src/agents/answer_agent.py` → `src/agents/chatbot.py`; `AnswerAgent` → `ChatbotAgent`; updated all import sites (`chat_handler.py`, `chat.py`)
-
-**Fixes + improvements:**
-- `src/agents/chat_handler.py` — `_get_document_retriever()` now returns `RetrievalRouter` (Redis-cached) instead of `DocumentRetriever` directly; retrieval-level cache restored.
-- `src/retrieval/router.py` — removed dead `db: AsyncSession` and `source_hint` parameters + `_UNSTRUCTURED_HINTS` constant from `retrieve()`. Cache key simplified.
-- `src/knowledge/processing_service.py` — removed dead `_build_csv_documents`, `_build_excel_documents`, `_profile_dataframe`, `_to_sheet_document` methods + `pandas` and `upload_parquet` imports.
-- `src/catalog/models.py` — added `top_values: list[Any] | None` to `ColumnStats`.
-- `src/catalog/introspect/tabular.py` — `_to_column` now populates `top_values` for columns with ≤10 distinct values; useful for query planner WHERE clause generation.
-- `main.py` — replaced deprecated `@app.on_event("startup")` with `lifespan` context manager; removed `knowledge_router`.
-
----
-
-## What just shipped (KM-557 — DB owner)
-
-After lead review of the catalog ingestion cost: dropped LLM enrichment,
-renamed the storage table, and exposed a lightweight index endpoint for
-the upcoming catalog refresher.
-
-**Files deleted**:
-- `src/catalog/enricher.py` — entire CatalogEnricher + EnrichmentResponse + apply_descriptions removed
-- `src/config/prompts/catalog_enricher.md` — dead prompt
-- `tests/catalog/test_enricher.py` — replaced by `test_render.py`
-
-**Files added**:
-- `src/catalog/render.py` — new home for `render_source` (the only piece of the old enricher still needed; consumed by `query/planner/prompt.py`)
-- `src/api/v1/data_catalog.py` — `GET /api/v1/data-catalog/{user_id}` returns `list[CatalogIndexEntry]`
-- `tests/catalog/test_render.py` — 5 tests (same coverage as the old render block)
-
-**Files modified**:
-- `src/db/postgres/models.py` — `__tablename__ = "data_catalog"` (was `"catalogs"`). Class name unchanged
-- `src/pipeline/structured_pipeline.py` — `StructuredPipeline(validator, store)` (was `(enricher, validator, store)`); pipeline is now `introspect → merge → validate → upsert`; `default_structured_pipeline()` no longer constructs an enricher
-- `src/pipeline/triggers.py` — docstrings updated; `on_catalog_rebuild_requested` docstring rewritten for the refresher use case
-- `src/query/planner/prompt.py` — import now `from ...catalog.render import render_source`
-- `src/catalog/introspect/{base,database,tabular}.py` — docstring scrubs (no behavior changes)
-- `src/models/api/catalog.py` — added `CatalogIndexEntry`; simplified `CatalogRebuildResponse` to `sources_rebuilt`
-- `main.py` — registered `data_catalog_router`
-- `src/security/README.md` — one stale wording fix
-
-**No migration**: the `data_catalog` table is created from scratch on first `init_db()`. The old `catalogs` table was never deployed against production data, so no rename SQL is needed.
-
-**Tests**: all 4 `test_structured_pipeline.py` tests reworked to construct `StructuredPipeline(validator=, store=)` without `enricher`. 5 `test_render.py` tests cover render_source standalone.
-
-**Lint**: `ruff check` clean on modified Phase 2 paths.
-
-**Open follow-ups left for the lead**:
-- `on_catalog_rebuild_requested` body — the refresher will iterate the index endpoint and call this trigger per source
-- `api/v1/db_client.py` `/ingest` still doesn't call `on_db_registered` — same blocker as before, untouched by KM-557
-
----
-
-## What just shipped (2026-05-11 — retrieval migration + bug fixes)
-
-**Files implemented / migrated**:
-- `src/retrieval/base.py` — `RetrievalResult` dataclass + `BaseRetriever` ABC (was in `src/rag/base.py`)
-- `src/retrieval/document.py` — full `DocumentRetriever` migrated from `src/rag/retrievers/document.py`; all retrieval methods (MMR/cosine/euclidean/inner_product/manhattan). Tabular file types filtered out from results.
-- `src/retrieval/router.py` — `RetrievalRouter` (Redis-cached, unstructured-only). `invalidate_cache(user_id)` clears all `retrieval:{user_id}:*` keys.
-
-**Deleted** (no longer used):
-- `src/rag/` — entire folder (base.py, retriever.py, router.py, retrievers/)
-- `src/tools/` — entire folder (search.py was the only real file; only called by deleted rag/ router)
-
-**Bug fixes**:
-- `src/pipeline/document_pipeline.py` — `retrieval_router.invalidate_cache(user_id)` called after `process()` and `delete()`. Redis failure is caught and logged (does not fail the document op).
-- `src/pipeline/document_pipeline.py` — CSV/XLSX now skips `knowledge_processor` (vector store). Tabular files go to catalog only; no duplicate embeddings.
-- `src/pipeline/triggers.py` — `on_document_uploaded` implemented (was `raise NotImplementedError`).
-- `src/agents/chat_handler.py` — `_normalize_chunks` now handles `RetrievalResult` objects. Previously they were silently dropped, causing empty context for unstructured queries through ChatHandler.
-
-**Import updates** (all changed from `src.rag.*` → `src.retrieval.*`):
-- `src/api/v1/chat.py`, `src/query/base.py`, `src/query/query_executor.py`, `src/query/executors/db_executor.py`, `src/query/executors/tabular.py`
-
----
-
-## What shipped previously (PR2b/4/5/6/7-bundle — DB owner solo, teammate reviews)
-
-**Files implemented**:
-- `src/agents/orchestration.py` — `OrchestratorAgent.classify(message, history) → IntentRouterDecision`. Pydantic model for structured output. History-aware query rewriting. Phase 1 filename + class name preserved; body fully rewritten for Phase 2.
-- `src/agents/answer_agent.py` — `AnswerAgent.astream(...)` streams answer tokens; accepts `QueryResult` and/or `list[DocumentChunk]`. Renames to `chatbot.py` in cleanup PR.
-- `src/agents/chat_handler.py` — `ChatHandler.handle(message, user_id, history)` returns `AsyncIterator[dict]` of `intent` / `chunk` / `done` / `error` SSE events. All deps injectable; lazy default builders.
-- `src/query/planner/prompt.py` — `render_catalog(catalog)` + `build_planner_prompt(question, catalog, previous_error)`. Reuses `catalog.enricher.render_source` for consistency across LLM call sites.
-- `src/query/planner/service.py` — `QueryPlannerService.plan(question, catalog, previous_error)` Azure OpenAI structured output → `QueryIR`.
-- `src/query/executor/dispatcher.py` — `ExecutorDispatcher.pick(ir) → BaseExecutor` by `source.source_type`. Lazy executor imports + per-source-type cache.
-- `src/query/service.py` — `QueryService.run(user_id, question, catalog) → QueryResult`. Plan→validate→retry-on-failure (max 3)→dispatch→execute. Catches NotImplementedError from TabularExecutor placeholder gracefully.
-
-**Prompts written** (filled in placeholders):
-- `src/config/prompts/intent_router.md`
-- `src/config/prompts/query_planner.md`
-- `src/config/prompts/chatbot_system.md`
-- `src/config/prompts/guardrails.md`
-
-**Tests added** (46 new — total now 146 + 2 skipped):
-- `tests/agents/test_intent_router.py` (4)
-- `tests/agents/test_answer_agent.py` (12)
-- `tests/agents/test_chat_handler.py` (6)
-- `tests/query/planner/test_prompt.py` (7)
-- `tests/query/planner/test_service.py` (3)
-- `tests/query/executor/test_dispatcher.py` (5)
-- `tests/query/test_service.py` (8)
-- `tests/query/planner/test_golden_questions.py` (3 — skipped by default; eval harness scaffold)
-
-**Lint**: `ruff check` clean on all Phase 2 paths. Phase 1 files have pre-existing E501/S608 issues — out of scope for this PR.
-
-**Placeholders / blockers for teammate** (status as of DB owner's commit, before merge):
-- `src/query/executor/tabular.py` (TAB) — DB owner's note: "still raises NotImplementedError". **Post-merge**: TAB shipped this in PR3-TAB; dispatcher now routes to the real `TabularExecutor`. The `NotImplementedError` catch in `QueryService` stays as a safety net.
-- `src/retrieval/document.py` — **implemented** (2026-05-11). Full `DocumentRetriever` migrated from `src/rag/retrievers/document.py`; supports MMR/cosine/euclidean/manhattan/inner_product. `_normalize_chunks` in `chat_handler.py` now handles `RetrievalResult` → `DocumentChunk` conversion correctly.
-- `src/api/v1/chat.py` (Phase 1) — NOT touched. Cleanup PR rewires the SSE endpoint to call `ChatHandler.handle(...)`.
-- `src/api/v1/db_client.py` (Phase 1) — NOT touched. Cleanup PR rewires `/database-clients/{id}/ingest` to call `pipeline.triggers.on_db_registered`.
-
----
-
-## What shipped previously (PR3-TAB — TAB owner)
-
-**Files implemented**:
-- `src/query/compiler/pandas.py` — `PandasCompiler` + `CompiledPandas(apply, output_columns)` dataclass. Pure helper functions (easier to test in isolation): `_apply_filters` (all 12 ops, `_like_to_regex` for LIKE), `_apply_select` (column pick + rename), `_apply_agg` (scalar + group_by via `pd.concat` of Series → `reset_index`), `_apply_orderby` (alias-aware via `_resolve_order_col`). Closure captures all IR fields explicitly so `apply(df)` is self-contained.
-- `src/query/executor/tabular.py` — `TabularExecutor` with injectable `fetch_blob` (same testability pattern as `TabularIntrospector`). Resolves Parquet blob path from `az_blob://{uid}/{did}` + table: single-table → `{uid}/{did}.parquet`, multi-table → `{uid}/{did}__{table.name}.parquet`. Runs compile → download → `asyncio.to_thread(_load_and_apply)` → 10k hard cap. Never raises; errors populate `QueryResult.error`. Uses `compiled.output_columns` for column labels (safe on empty DataFrame).
-
-**Tests added** (55 new — total suite was 86 all passing at PR3-TAB time):
-- `tests/unit/query/compiler/test_pandas_compiler.py` — 43 tests across all 12 filter ops (including `is_null`, `not_in`, `like`, `between`), all 6 agg fns, group_by, order_by asc/desc, limit-after-order, alias round-trip, empty DataFrame, error paths.
-- `tests/unit/query/executor/test_tabular_executor.py` — 12 tests: `_resolve_blob_name` (single/multi-table, bad prefix), happy-path `QueryResult` shape (columns, rows, backend, truncated, source_id), wrong source_type → error, blob fetch failure → error, unknown source → error.
-
-**Lint**: `ruff check` clean on both files.
-
----
-
-## What shipped previously (PR1-tab — TAB owner)
-
-**Files implemented**:
-- `src/catalog/introspect/tabular.py` — `TabularIntrospector` reads original blob (CSV/XLSX/Parquet), profiles each column (dtype, stats, sample values), runs PIIDetector. For XLSX: one `Table` per sheet (`Table.name = sheet_name`); for CSV/Parquet: one `Table` (`Table.name = filename stem`). `fetch_doc`/`fetch_blob` are constructor-injectable for unit tests — no `Settings` or DB required at import time.
-- `src/pipeline/triggers.py` — `on_tabular_uploaded` wired (mirrors `on_db_registered` pattern).
-
-**Tests added** (31 new):
-- `tests/unit/catalog/test_introspect_tabular.py` — CSV / XLSX / Parquet shapes, per-column stats, nullable detection, PII name + value matching, sample capping, all error paths. Pure Python, no network I/O.
-
-**Executor contract note**: introspector downloads the *original* blob for schema reading. The tabular executor (PR3-TAB) downloads *Parquet* blobs for query execution. For CSV/Parquet sources (single table), the executor must call `parquet_blob_name(uid, did, sheet_name=None)`; for XLSX (multi-table), `parquet_blob_name(uid, did, table.name)`.
-
----
-
-## What shipped previously (PR3-DB — DB owner)
-
-**Files implemented**:
-- `src/query/compiler/sql.py` — `SqlCompiler` for Postgres dialect; `CompiledSql(sql, params)` dataclass with `params: dict[str, Any]` (changed from `list`); supports all 12 whitelisted filter ops, all 6 aggs, alias-aware order_by; `_qident` escapes embedded double-quotes
-- `src/query/executor/db.py` — `DbExecutor` with sqlglot SELECT-only guard, Postgres session-level read-only + 30s `statement_timeout`, `asyncio.wait_for` backstop, 10k row hard cap; rejects non-`schema` source_type and `dbclient://` URI mismatch; never raises (populates `QueryResult.error`)
-
-**Files extended**:
-- `src/query/compiler/pandas.py` — fixed pre-existing UP035 (Callable import)
-- `pyproject.toml` — added `S608` to `tests/**` ruff ignore (false positive: tests assert literal SQL strings)
-
-**Tests added** (36 new, all passing — total now 100):
-- `tests/query/compiler/test_sql.py` — every filter op, every agg, count(*), count_distinct, order_by alias vs column, multi-filter AND, identifier quoting escape, error paths
-
-**Lint**: `ruff check` clean on Phase 2 paths.
-
-**Hand-off note for teammate**: `CompiledSql.params` is now `dict[str, Any]` not `list`. The pandas compiler will follow the same convention (or document its own) — coordinate when PR3-TAB lands.
-
----
-
-## What shipped previously (PR2a — DB owner)
-
-**Files implemented**:
-- `src/catalog/enricher.py` — Azure OpenAI GPT-4o + structured output (`EnrichmentResponse`), `render_source` (reusable by planner prompt later), `apply_descriptions` merger, injectable `structured_chain` for tests
-- `src/pipeline/structured_pipeline.py` — `StructuredPipeline` orchestrator + `default_structured_pipeline()` factory with lazy production-dep imports
-- `src/pipeline/triggers.py` — `on_db_registered` wired; tabular/document/rebuild stubs preserved with implementation notes
-
-**Files extended**:
-- `src/catalog/models.py` — added `ForeignKey` model, `Table.foreign_keys: list[ForeignKey] = []`
-- `src/catalog/introspect/database.py` — `_extract_foreign_keys` populates `Table.foreign_keys` from extractor data
-- `src/config/prompts/catalog_enricher.md` — full system prompt with style rules and one few-shot example
-
-**Tests added** (14 new, all passing — total now 64):
-- `tests/catalog/test_enricher.py` — render / apply / end-to-end with fake chain (10 tests)
-- `tests/pipeline/test_structured_pipeline.py` — orchestration with stub deps (4 tests)
-
-**Lint**: `ruff check` clean on all Phase 2 paths. Phase 1 files (`pipeline/db_pipeline/`, `pipeline/document_pipeline/`) have pre-existing ruff issues — out of scope for this PR.
-
----
-
-## What shipped previously (PR1 — DB owner's first chunk)
-
-**Files implemented** (was `NotImplementedError`):
-- `src/catalog/pii_detector.py`, `src/catalog/validator.py`, `src/catalog/store.py`, `src/catalog/reader.py`
-- `src/catalog/introspect/database.py` (FK extraction added in PR2a)
-- `src/query/ir/validator.py`
-
-**Files extended**:
-- `src/query/ir/operators.py` — `TYPE_COMPATIBILITY` matrix
-- `src/catalog/models.py` — `location_ref` URI-scheme docstring
-- `src/db/postgres/models.py` — `Catalog` SQLAlchemy table; `init_db.py` imports it
-
-**Tests**: 50 unit tests + 1 integration (gated on `RUN_INTEGRATION_TESTS=1`).
-
-**Reused Phase 1 utilities** (cleanup deferred):
-- `src/database_client/database_client_service.py:get`
-- `src/utils/db_credential_encryption.py:decrypt_credentials_dict`
-- `src/pipeline/db_pipeline/db_pipeline_service.py:engine_scope`
-- `src/pipeline/db_pipeline/extractor.py:get_schema/profile_column/get_row_count`
-
----
-
-## Open contract items (not yet locked)
-
-- **Joins in IR** — currently single-table only (ARCHITECTURE.md §7); DB owner accepted the constraint for v1, will revisit in PR3 if it's blocking real queries
-- **`updated_at` on Source vs `generated_at` on Catalog** — Pydantic models have both; introspector sets per-Source; CatalogStore preserves both
-- **Catalog refresh trigger** (open question §3) — default policy is rebuild-on-upload-or-connect; auto-refresh deferred
-- **Unstructured catalog entries** (open question §2) — currently empty filter for `source_hint="unstructured"`; revisit when adding doc descriptions
-- **PII handling for `sample_values`** (open question §5) — currently nulls them out (skip); mask/synthesize deferred
-- **Dialect priority for SQL compiler** — PR3 will land Postgres first, MySQL second; BigQuery/Snowflake/SQL Server later
-
----
-
-## How to update this file
-
-When a PR lands:
-1. Flip status from `[ ]` or `[~]` to `[x]`
-2. Add a short note (file paths, scope cuts, surprises)
-3. Bump "Last updated" at the top
-4. If a new contract decision lands, move it from "Open contract items" to the relevant inline note
-
-When opening a PR:
-1. Flip status to `[~]` and add yourself as the active owner in the PR row
-2. Don't promise items in the PR description that aren't in the table

REPO_CONTEXT.md

@@ -1,494 +0,0 @@
-# Repo Context — Agentic Service Data Eyond Catalog
-
-Orientation file for future Claude Code sessions. Cross-reference `ARCHITECTURE.md` for the full design rationale and decision log.
-
----
-
-## Product vision — Data Eyond, your AI data scientist
-
-Data Eyond is positioned as an *AI data scientist* that supports business analytics. It is built around the **CRISP-DM** framework (Business Understanding → Data Understanding → Data Preparation → Modeling → Evaluation → Deployment) — the agent works through data problems the way a real analyst would, not as a one-shot Q&A bot.
-
-**Target users:**
-- **Executives** — deep-dive into their own data and extract insight to drive business decisions without needing a data team in the loop.
-- **Data analysts / scientists** — offload routine analysis so they can focus on heavier work.
-
-**Envisioned user flow:**
-1. **Discovery interview** — a short conversation with a Data Eyond *interview agent* that draws out goal, business context, and what the user is actually trying to learn (CRISP-DM Business Understanding).
-2. **Connect data** — DB connection or file upload (DB, CSV, XLSX, Parquet, documents).
-3. **Ask Data Eyond** — natural-language analytical question.
-4. **CRISP-DM-structured analytical response** — exportable as a **presentation deliverable** or a **notebook-style report**.
-
-North star: less "chatbot over a database", more "junior data scientist that hands back a polished, decision-ready deliverable."
-
-The current repo (Phase 2, below) is the *foundation* — IntentRouter → QueryPlanner → Executor → ChatbotAgent gives us a reliable structured-query spine. The next evolution is the agentic layer that turns this into an end-to-end CRISP-DM workflow (see *Roadmap — agentic evolution* further down).
-
----
-
-## TL;DR
-
-FastAPI multi-agent backend for data analysis. Users upload documents and register databases / tabular files; they ask natural-language questions and get answers grounded in their data, streamed via SSE.
-
-The architecture has two paths:
-
-- **Unstructured** (PDF, DOCX, TXT) — dense similarity over prose chunks (PGVector).
-- **Structured** (databases, XLSX, CSV, Parquet) — a per-user **data catalog** describes what tables/columns exist; an LLM produces a **JSON IR** of intent; a deterministic Python compiler turns the IR into SQL or pandas; the executor runs it.
-
-The LLM produces *intent*, not query syntax. Deterministic code does the rest.
-
-The Phase 2 end-to-end flow is **wired and runnable** as of 2026-05-12. See *Implementation status* below for the per-file matrix. `PROGRESS.md` is the authoritative line-by-line tracker; this file is the orientation.
-
----
-
-## Stack
-
-- Python 3.12, FastAPI 0.115, uvicorn, sse-starlette
-- Async SQLAlchemy 2.0 + asyncpg (Postgres), psycopg3 (PGVector multi-statement workaround)
-- LangChain 0.3 + langchain-postgres (PGVector) + langchain-openai (Azure OpenAI GPT-4o + embeddings)
-- LangGraph 0.2 + langgraph-checkpoint-postgres
-- Redis 5 (response + retrieval cache)
-- Azure Blob Storage (uploads + Parquet)
-- pandas, pyarrow, polars-ready (deferred), sqlglot, pydantic v2, structlog, slowapi, langfuse
-- presidio-analyzer + spaCy `en_core_web_lg` (PII), pytesseract + pdf2image (PDF OCR)
-- DB connectors: psycopg2, pymysql, pymssql, sqlalchemy-bigquery, snowflake-sqlalchemy
-
-Run: `uv run --no-sync uvicorn main:app --host 0.0.0.0 --port 7860`. On Windows use `uv run --no-sync python run.py` (sets `WindowsSelectorEventLoopPolicy` for psycopg3 async).
-
----
-
-## Top-level layout
-
-```
-main.py                — FastAPI app + middleware + router wiring + init_db() on startup
-run.py                 — Windows-safe local entry point
-ARCHITECTURE.md        — design intent (source of truth for shape + invariants)
-README.md
-Dockerfile             — python:3.12-slim, installs spaCy en_core_web_lg, tesseract, poppler
-pyproject.toml / uv.lock
-scripts/               — backfill scripts (build_initial_catalogs, enrich_all_sources)
-src/                   — all application code
-```
-
----
-
-## src/ map
-
-### Core data shapes (only files with real content)
-
-| Path | Role |
-|---|---|
-| `catalog/models.py` | Pydantic: `Catalog → Source[] → Table[] → Column[]` |
-| `query/ir/models.py` | `QueryIR` (select / filters / group_by / order_by / limit) |
-| `query/ir/operators.py` | `ALLOWED_FILTER_OPS`, `ALLOWED_AGG_FNS`, `LIMIT_HARD_CAP=10000` |
-| `security/pii_patterns.py` | name patterns + email/phone regex for PII detection |
-
-### Catalog — identity layer for structured sources (Cs ∪ Ct)
-
-| Path | Role |
-|---|---|
-| `catalog/introspect/base.py` | `BaseIntrospector.introspect(location_ref) -> Source` |
-| `catalog/introspect/database.py` | `information_schema` + ~100 row sample → draft Source |
-| `catalog/introspect/tabular.py` | Parquet/CSV/XLSX header reader + sample (one Table per sheet for XLSX) |
-| `catalog/render.py` | renders a `Source` as the canonical text block consumed by the planner (KM-557; LLM enrichment removed — planner reads stats + samples directly) |
-| `catalog/validator.py` | invariants beyond Pydantic shape (unique IDs, FK refs) |
-| `catalog/store.py` | persist as Postgres `jsonb` row keyed by user_id (`get/upsert/delete`) — table `data_catalog` |
-| `catalog/reader.py` | load + filter catalog by source_hint (returns full catalog for ≤50 tables) |
-| `catalog/pii_detector.py` | flag PII columns at ingestion → suppresses `sample_values` |
-
-### Query — catalog-driven structured path
-
-| Path | Role |
-|---|---|
-| `query/service.py` | `QueryService.run(user_id, question, catalog) -> QueryResult` (top-level) |
-| `query/planner/service.py` | LLM call: question + catalog → QueryIR (structured output) |
-| `query/planner/prompt.py` | renders catalog into the planner prompt |
-| `query/ir/validator.py` | catalog-aware IR validation: column_ids exist, ops whitelisted, value_type matches data_type, limit ≤ cap |
-| `query/compiler/base.py` | `BaseCompiler.compile(ir) -> object` |
-| `query/compiler/sql.py` | IR → `(sql, params)`; identifiers from catalog, values parameterized |
-| `query/compiler/pandas.py` | IR → callable that runs against a DataFrame |
-| `query/executor/base.py` | `BaseExecutor.run(ir) -> QueryResult` (uniform across backends) |
-| `query/executor/db.py` | runs compiled SQL via asyncpg/pymysql in read-only txn (sqlglot second-line defence) |
-| `query/executor/tabular.py` | runs pandas/polars chain on a Parquet file (eager pandas → pyarrow pushdown → polars lazy by file size) |
-| `query/executor/dispatcher.py` | picks DB vs Tabular executor based on `source.source_type` of the IR's source |
-
-### Retrieval — unstructured path (Cu)
-
-| Path | Role |
-|---|---|
-| `retrieval/document.py` | `DocumentRetriever` over PGVector chunks |
-| `retrieval/router.py` | dispatches the `unstructured` route (the `chat` and `structured` routes do not pass through here) |
-
-### Agents — the three LLM call sites
-
-| Path | Role |
-|---|---|
-| `agents/orchestration.py` | `OrchestratorAgent` — classifies message → `needs_search`, `source_hint ∈ {chat, unstructured, structured}`, `rewritten_query`. Filename + class name kept from Phase 1; body replaced with Phase 2 logic. Output model is `IntentRouterDecision` |
-| `agents/chatbot.py` | `ChatbotAgent` — final answer formation (receives Cu chunks or QueryResult); SSE-streamed via `astream` |
-| `agents/chat_handler.py` | `ChatHandler` — top-level orchestrator; routes to chat / unstructured / structured and yields SSE-style `intent`/`chunk`/`done`/`error` events |
-
-(`QueryPlanner` is the third LLM call site, under `query/planner/`. The
-fourth — `CatalogEnricher` — was removed in KM-557; ingestion no longer
-makes any LLM calls.)
-
-### Pipelines — ingestion coordinators
-
-| Path | Role |
-|---|---|
-| `pipeline/structured_pipeline.py` | DB / tabular: introspect → merge → validate → store (no enrich step since KM-557) |
-| `pipeline/document_pipeline.py` | unstructured: extract → chunk → embed → PGVector. CSV/XLSX skip vector store (catalog only). Invalidates retrieval cache on process/delete. |
-| `pipeline/triggers.py` | event entry points called by API routes: `on_db_registered`, `on_tabular_uploaded`, `on_document_uploaded`, `on_catalog_rebuild_requested` |
-
-(`pipeline/orchestrator.py` was deleted in the Cleanup PR — it was a redundant stub; `StructuredPipeline` already takes the introspector at `run()` time.)
-
-### Security — cross-cutting
-
-| Path | Role |
-|---|---|
-| `security/auth.py` | bcrypt password hash/verify, JWT encode/decode, get_user |
-| `security/credentials.py` | Fernet encrypt/decrypt for stored DB credentials |
-| `security/pii_patterns.py` | (already listed) |
-
-### API + infra + config
-
-| Path | Role |
-|---|---|
-| `api/v1/*.py` | FastAPI routers — thin endpoints delegating to `pipeline/triggers` and `query/service` |
-| `models/api/{catalog,chat,document}.py` | request/response Pydantic models |
-| `db/postgres/connection.py` | two async engines: `engine` (app) and `_pgvector_engine` (PGVector) |
-| `db/postgres/init_db.py` | startup: creates `vector` extension, all tables, HNSW + GIN indexes |
-| `db/postgres/models.py` | SQLAlchemy app tables (users, rooms, chat messages, …) |
-| `db/postgres/vector_store.py` | shared PGVector instance (collection `documents` — written by Go ingestion service) |
-| `db/redis/connection.py` | async Redis client |
-| `storage/az_blob/az_blob.py` | Azure Blob async wrapper (uploads + Parquet) |
-| `middlewares/{cors,logging,rate_limit}.py` | CORS allow-all (POC), structlog JSON, slowapi |
-| `observability/langfuse/langfuse.py` | trace helper |
-| `config/settings.py` | pydantic-settings; `.env` uses double-underscore aliases |
-| `config/env_constant.py` | env file path constant |
-| `config/prompts/*.md` | prompt templates: `intent_router`, `query_planner`, `chatbot_system`, `guardrails` (KM-557 removed `catalog_enricher`) |
-
----
-
-## Core architectural decisions
-
-1. **Catalog as primary context, not retrieval.** For ≤50 tables (typical), the entire catalog is rendered into the planner prompt verbatim (~3–5k tokens). No vector search, no BM25, no top-k for structured data. Catalog-level retrieval (BM25 + table-level vectors with RRF) is the *deferred* upgrade for users with hundreds of tables.
-
-2. **JSON IR over raw SQL.** The planner LLM emits a Pydantic-validated intent, never a SQL string. The compiler is deterministic Python. Benefits: validatable before execution, dialect-portable (one IR → SQL of any dialect / pandas / polars), cheaper tokens, trivially testable without an LLM, and the LLM literally cannot emit invalid SQL syntax.
-
-3. **Deterministic compiler, not LLM SQL writer.** All actual query construction happens in pure code. Compiler bugs are reproducible and fixable. Same IR → same query.
-
-4. **Pipeline stage isolation.** Each stage (`IntentRouter`, `CatalogReader`, `QueryPlanner`, `IRValidator`, `QueryCompiler`, `QueryExecutor`, `ChatbotAgent`) is its own module with typed input and typed output. No god classes.
-
-5. **Minimal LLM surface.** Only three LLM call sites in the system (KM-557 dropped `CatalogEnricher` — ingestion is now LLM-free; the planner reads stats + sample rows + column names directly):
-   - `IntentRouter` — once per user message
-   - `QueryPlanner` — once per structured query
-   - `ChatbotAgent` — once per answer (formatting)
-
-6. **Three-way routing**: `chat` / `unstructured` / `structured`. The router commits to one path. Cross-source questions ("compare DB sales vs uploaded customer file") are handled inside the structured path because the planner sees Cs ∪ Ct in one prompt. **DB vs tabular is not a routing concern** — it's a per-source attribute (`source_type`) that only matters at execution time.
-
-7. **Stable IDs.** `source_id`, `table_id`, `column_id` are stable internal references. Renaming a column in the source DB does not invalidate cached IRs.
-
-8. **PII suppression at the boundary.** Columns flagged with `pii_flag=true` have `sample_values: null` — real PII never enters LLM prompts. Auto-detected at ingestion via name patterns + value regex (`security/pii_patterns.py`). When in doubt, flag — false positives cost nothing; false negatives leak data.
-
----
-
-## End-to-end flows
-
-### Ingestion (when user uploads a file or connects a DB)
-
-```
-source upload / DB connect
-    │
-    ├── unstructured (pdf/docx/txt)
-    │     → DocumentPipeline: extract → chunk → embed → PGVector
-    │
-    └── structured (DB schema or tabular file)
-          → introspect (information_schema or file headers + sample rows)
-          → CatalogValidator (Pydantic + unique-IDs + FK refs)
-          → CatalogStore.upsert(user_id jsonb row in `data_catalog`)
-```
-
-### Query (per user message)
-
-```
-user message
-    │
-    → Redis cache check (24h TTL)  ── miss ─→ continue
-    →
-    → IntentRouter LLM   →  needs_search? source_hint?
-    │
-    ├── chat          → ChatbotAgent → SSE stream
-    ├── unstructured  → DocumentRetriever (Cu) → ChatbotAgent → SSE stream
-    └── structured    →
-          CatalogReader.read(user_id, "structured")          # full Cs ∪ Ct
-              ↓
-          QueryPlanner LLM(question, catalog) → QueryIR
-              ↓
-          IRValidator.validate(ir, catalog)
-              (source_id ∈ catalog, table_id ∈ source, column_ids ∈ table,
-               ops/aggs whitelisted, value_type matches data_type, limit ≤ 10000)
-              fail → re-prompt planner with error context (max 3 retries)
-              ↓
-          ExecutorDispatcher.pick(ir)              # by source.source_type
-              ├─ DbExecutor       → SqlCompiler → sqlglot guard → asyncpg/pymysql
-              │                     (read-only txn, 30s timeout)
-              └─ TabularExecutor  → PandasCompiler → eager pandas (≤100 MB)
-                                    or pyarrow pushdown (100 MB–1 GB)
-                                    or polars lazy scan (>1 GB)
-              ↓
-          QueryResult
-              ↓
-          ChatbotAgent → SSE stream
-```
-
----
-
-## Catalog schema (per-user `jsonb` row)
-
-```
-Catalog
-├── user_id, schema_version, generated_at
-└── sources[]
-    └── Source { source_id, source_type, name, description, location_ref, updated_at }
-        └── tables[]
-            └── Table { table_id, name, description, row_count, foreign_keys[] }
-                ├── columns[]
-                │   └── Column { column_id, name, data_type, description,
-                │                 nullable, pii_flag, sample_values[]|null, stats|null }
-                └── foreign_keys[]
-                    └── ForeignKey { column_id, target_table_id, target_column_id }
-```
-
-`source_type ∈ {schema, tabular, unstructured}`.
-`data_type ∈ {int, decimal, string, datetime, date, bool, json}`.
-`ForeignKey` references are within the SAME `Source` only; cross-source FKs are not modeled.
-
-Deferred Column fields (add when justified): `description_human`, `synonyms[]`, `tags[]`, `primary_key`, `unit`, `semantic_type`, `example_questions[]`, `schema_hash`, `enrichment_status`.
-
----
-
-## JSON IR schema
-
-```jsonc
-{
-  "ir_version": "1.0",
-  "source_id":  "...",
-  "table_id":   "...",
-  "select": [
-    {"kind": "column", "column_id": "...", "alias": "..."},
-    {"kind": "agg",    "fn": "count|count_distinct|sum|avg|min|max",
-                       "column_id": "...?", "alias": "..."}
-  ],
-  "filters": [
-    {"column_id": "...",
-     "op":    "= | != | < | <= | > | >= | in | not_in | is_null | is_not_null | like | between",
-     "value": ...,
-     "value_type": "int|decimal|string|datetime|date|bool"}
-  ],
-  "group_by": ["column_id", ...],
-  "order_by": [{"column_id": "...", "dir": "asc|desc"}],
-  "limit": 100
-}
-```
-
-Single-table only in v1. `having`, `offset`, boolean filter trees, `distinct`, joins, window functions are deferred until user demand proves the limitation.
-
----
-
-## Implementation status
-
-**As of 2026-05-12 — Phase 2 end-to-end flow is wired.** `PROGRESS.md` has the per-PR line-item table; this section is the high-level snapshot. Stub files (`raise NotImplementedError`) are now the exception, not the rule.
-
-| Area | Status | Notes |
-|---|---|---|
-| Catalog Pydantic models | ✅ | `catalog/models.py` — incl. `ForeignKey`, `ColumnStats.top_values` |
-| JSON IR Pydantic models | ✅ | `query/ir/models.py` + `operators.py` (TYPE_COMPATIBILITY filled) |
-| Catalog ingestion — DB | ✅ | introspect → validate → upsert. `on_db_registered` wired; `/api/v1/db-clients/{id}/ingest` calls it |
-| Catalog ingestion — tabular | ✅ | CSV/XLSX/Parquet; `on_tabular_uploaded` wired into `/api/v1/document/process`. XLSX → one Table per sheet. CSV/XLSX skip vector store |
-| Catalog ingestion — unstructured | ✅ | `on_document_uploaded` implemented; full DocumentPipeline (extract → chunk → embed → PGVector) |
-| Catalog store / reader / validator / PII detector | ✅ | `data_catalog` jsonb table (renamed from `catalogs` in KM-557) |
-| LLM enrichment | ❌ removed (KM-557) | Cost cut — planner reads `column.stats` + `sample_values` + `top_values` + `column.name` directly. `catalog/render.py` keeps the source-rendering helper |
-| `IntentRouter` (lives as `OrchestratorAgent` in `agents/orchestration.py`) | ✅ | 3-way `source_hint`, history-aware query rewriting. Filename + class name kept from Phase 1; Phase 2 body |
-| `CatalogReader` | ✅ | Loads full catalog; filters by `source_hint` |
-| `QueryPlanner` LLM call | ✅ | Azure OpenAI structured output → `QueryIR`; supports retry with `previous_error` |
-| IR validator | ✅ | Catalog-aware; full rule set; descriptive errors |
-| SQL compiler (Postgres) | ✅ | All 12 filter ops, all 6 aggs, alias-aware order_by, parameterized values, quoted identifiers |
-| DbExecutor | ✅ | sqlglot SELECT-only guard, RO txn, `statement_timeout=30000`, 10k row cap, never raises |
-| Pandas compiler | ✅ | Same op coverage as SQL; pure module-level helpers |
-| TabularExecutor | ✅ | Parquet blob path resolution, `asyncio.to_thread`, 10k cap, never raises |
-| ExecutorDispatcher | ✅ | Routes by `source.source_type`; lazy imports + cache |
-| QueryService | ✅ | plan → validate → retry-on-fail (max 3) → dispatch → execute → `QueryResult` |
-| `ChatbotAgent` + prompt + guardrails | ✅ | Renamed from `AnswerAgent` in Cleanup PR. Guardrails appended to `chatbot_system.md` |
-| `ChatHandler` (top-level chat orchestrator) | ✅ | SSE events: `intent` / `chunk` / `done` / `error` |
-| `DocumentRetriever` + `RetrievalRouter` (Redis-cached) | ✅ | Migrated from `src/rag/` (now deleted). Mentor commit `61c746f` rewrote to raw SQL (pgvector `<=>` cosine, `<+>` manhattan) to dodge asyncpg type-mapping issues with Go-ingested schema. Methods reduced to `cosine | manhattan`. Collection: `documents`. |
-| `/api/v1/chat/stream` | ✅ | Rewired to `ChatHandler`; Redis cache + fast intent + history + message persistence remain in chat.py |
-| `/api/v1/db-clients/{id}/ingest` | ✅ | Calls only `on_db_registered`; Phase 1 dual-write removed |
-| `/api/v1/document/{upload,process,delete}` | ✅ | `/process` triggers `on_tabular_uploaded` for CSV/XLSX |
-| `GET /api/v1/data-catalog/{user_id}` | ✅ | Index endpoint (KM-557) |
-| `POST /api/v1/data-catalog/rebuild` | ✅ | Iterates sources, re-runs per-source trigger |
-| Credential encryption | ⚠️ stub | `security/credentials.py` not migrated; runtime reuses Phase 1 `utils/db_credential_encryption.py` |
-| Tests | ✅ 146+ unit | Compilers (DB 36, Pandas 43), validators, introspectors, agents, chat handler, dispatcher, planner |
-| Planner eval harness | 🟡 scaffold | 3 DB + 4 tabular golden cases. Gated on `RUN_PLANNER_EVAL=1`. Real Azure OpenAI passing |
-| E2E smoke tests | ❌ not started | Component-level orchestration is covered |
-| DB introspector unit test | ❌ deferred | Needs Postgres testcontainer |
-| Sources event in `/chat/stream` | ⚠️ emits `[]` | `ChatHandler` doesn't surface retrieval sources yet; same gap reflected in `save_messages` |
-
-**Deferred to later phases**: joins in IR, schema drift detection, hybrid catalog search (BM25 + RRF for 100+ table users), polars lazy scan for >1GB tabular files, MySQL/BigQuery/Snowflake SQL dialects, mask/synthesize PII strategies.
-
----
-
-## Team — division of work
-
-The service is built by two engineers; many modules are source-type-agnostic and shared.
-
-- **DB** owns SQL paths: introspection, SQL compiler, DB executor, credential storage.
-- **TAB** owns tabular paths: CSV/XLSX/Parquet introspection, pandas compiler, tabular executor, blob/Parquet plumbing.
-- **B** = both — shared contracts and source-type-agnostic plumbing. Pair-program or split with explicit hand-off.
-
-### Step-by-step ownership
-
-| # | Step | File / area | Owner | Notes |
-|---|---|---|---|---|
-| 0 | **Lock contracts before coding** | — | B | See "Decisions to lock" below; block until aligned |
-| 1 | Catalog Pydantic models | `catalog/models.py` | B | Already done; only touch if both agree |
-| 2 | IR Pydantic models | `query/ir/models.py` | B | Already done; joins/window fns require joint sign-off |
-| 3 | IR operator whitelists | `query/ir/operators.py` | B | Already done; both compilers rely on these |
-| 4 | PII patterns / regex | `security/pii_patterns.py` | B | Already done; extend together as gaps appear |
-| **Ingestion — introspection** | | | | |
-| 5 | DB introspector (information_schema, sample, FKs) | `catalog/introspect/database.py` | DB | Use SQLAlchemy `inspect()`; dialect-aware quoting |
-| 6 | Tabular introspector (CSV/XLSX/Parquet headers + sample) | `catalog/introspect/tabular.py` | TAB | Each XLSX sheet → one Table |
-| 7 | `BaseIntrospector` ABC | `catalog/introspect/base.py` | B | Confirm signature returns the same `Source` shape |
-| **Ingestion — shared catalog plumbing** | | | | |
-| 8 | ~~Catalog enricher + prompt~~ | — | **REMOVED in KM-557.** Cost optimization — planner reads stats + sample rows directly. `catalog/render.py` keeps the source-rendering helper. |
-| 9 | Catalog validator | `catalog/validator.py` | B | Type-agnostic |
-| 10 | Catalog store (Postgres jsonb) | `catalog/store.py` | B | Recommend DB (Postgres expertise) |
-| 11 | Catalog reader | `catalog/reader.py` | B | Type-agnostic |
-| 12 | PII detector | `catalog/pii_detector.py` | B | Either; uses `pii_patterns.py` |
-| **Ingestion — pipelines** | | | | |
-| 13 | Structured pipeline (introspect → enrich → validate → store) | `pipeline/structured_pipeline.py` | B | Pair on this — calls both introspectors via dispatcher |
-| 14 | Triggers (`on_db_registered`, `on_tabular_uploaded`) | `pipeline/triggers.py` | B | Each owns their trigger function |
-| 15 | Ingestion orchestrator | `pipeline/orchestrator.py` | B | Routes by source_type; pair |
-| 16 | Document pipeline (PDF/DOCX/TXT) | `pipeline/document_pipeline.py` | TAB | Tabular-adjacent (file uploads) |
-| **Query — shared spine** | | | | |
-| 17 | IR validator (catalog-aware) | `query/ir/validator.py` | B | Recommend DB; both must agree on exact error messages so retry-prompt is consistent |
-| 18 | Planner LLM service | `query/planner/service.py` | B | Type-agnostic |
-| 19 | Planner prompt (catalog → text) | `query/planner/prompt.py`, `config/prompts/query_planner.md` | B | **Pair-program**. Must describe DB tables and tabular files in one consistent format |
-| 20 | Intent router (chat/unstructured/structured) | `agents/orchestration.py` (class `OrchestratorAgent` — Phase 1 filename + class name preserved; Phase 2 body), `config/prompts/intent_router.md` | B | Type-agnostic. The prompt file uses `intent_router.md`, but the source module is still `orchestration.py` |
-| 21 | Executor base + `QueryResult` | `query/executor/base.py` | B | Lock the shape before either implements an executor |
-| 22 | Executor dispatcher | `query/executor/dispatcher.py` | B | Reads `source.source_type` from catalog; pair |
-| 23 | Compiler base ABC | `query/compiler/base.py` | B | Already done |
-| 24 | Top-level QueryService | `query/service.py` | B | Wires planner → validator → compiler → executor; pair |
-| **Query — DB path** | | | | |
-| 25 | SQL compiler (IR → SQL + params, per dialect) | `query/compiler/sql.py` | DB | Identifiers from catalog (quoted), values parameterized |
-| 26 | DB executor (asyncpg/pymysql, sqlglot guard, RO txn, 30s timeout) | `query/executor/db.py` | DB | |
-| 27 | Credential encryption (Fernet) | `security/credentials.py` | DB | Needed for stored user DB creds |
-| 28 | User-DB connection management | helper in pipelines | DB | engine_scope context manager pattern |
-| **Query — Tabular path** | | | | |
-| 29 | Pandas compiler (IR → callable on DataFrame) | `query/compiler/pandas.py` | TAB | Same IR, different backend |
-| 30 | Tabular executor (eager pandas first; pyarrow / polars later) | `query/executor/tabular.py` | TAB | Initial scope: eager pandas only |
-| 31 | Parquet upload/download + Azure Blob wrapper | `storage/az_blob/az_blob.py` (+ helper) | TAB | XLSX sheet → one Parquet per sheet (deterministic blob name) |
-| **Agents + chat** | | | | |
-| 32 | Chatbot agent + prompt | `agents/chatbot.py`, `config/prompts/chatbot_system.md` | B | Receives QueryResult or Cu chunks |
-| 33 | Guardrails prompt | `config/prompts/guardrails.md` | B | |
-| **API surface** | | | | |
-| 34 | DB client endpoints (register/ingest/list/delete) | `api/v1/db_client.py` | DB | |
-| 35 | Document/tabular upload endpoints | `api/v1/document.py` | TAB | |
-| 36 | Chat stream endpoint (SSE) | `api/v1/chat.py` | B | Dispatches both paths; pair |
-| 37 | Room / users endpoints | `api/v1/room.py`, `api/v1/users.py` | B | Whoever has bandwidth |
-| **Tests + eval** | | | | |
-| 38 | DB compiler golden tests (IR → SQL fixtures) | `tests/query/compiler/test_sql.py` | DB | Pure-Python, no LLM |
-| 39 | Pandas compiler golden tests (IR → expected DataFrame) | `tests/query/compiler/test_pandas.py` | TAB | Pure-Python, no LLM |
-| 40 | IR validator tests (catalog × IR error matrix) | `tests/query/ir/test_validator.py` | B | Each contributes test cases for their source type |
-| 41 | Planner eval (golden question → IR examples) | `tests/query/planner/` | B | Each contributes ~10 question→IR examples |
-| 42 | E2E smoke tests | `tests/e2e/` | B | Pair |
-
-### Decisions to lock before coding
-
-If made unilaterally these create silent contract drift. Lock them in a 30-min sync first.
-
-| Decision | Why it matters | Recommended call |
-|---|---|---|
-| `QueryResult` shape (current scaffold: `source_id, backend, rows, row_count, truncated, elapsed_ms, error`) | Both executors return this; chatbot consumes it | Lock as-is unless either side needs more (e.g. `column_types` for formatting) |
-| `Source.location_ref` format (`az_blob://...` vs `dbclient://{id}` etc.) | Dispatcher and executors both parse this | Pick a convention now; document in `catalog/models.py` docstring |
-| Where do user DB credentials live? | DB executor needs creds to run queries; Source has `location_ref` but creds are encrypted separately | Recommend: `location_ref="dbclient://{client_id}"`; executor looks up creds by ID |
-| How does dispatcher pick the executor? | Routes by `source.source_type` — but where does dispatcher get it (catalog reload, or IR carries it)? | Recommend: dispatcher takes `(Catalog, IR)`, looks up source by `IR.source_id` |
-| Joins in v1 IR? | Excluded per ARCHITECTURE.md §7. DB path is most affected — real DB use often needs joins. | Recommend: ship single-table; revisit in PR 2. **DB owner must accept the constraint or push back early** |
-| Planner prompt — render tabular vs DB sources uniformly | If described differently, planner gets confused | Pair-program. Render both as `Table: name (n rows) — Columns: ...` regardless of source_type |
-| Error contract — raise or return `QueryResult.error`? | Both executors must behave the same so chatbot branches consistently | Recommend: never raise from `executor.run()`; populate `QueryResult.error` |
-| PII handling for tabular `sample_values` | DB samples come from `information_schema`; tabular from file reads. Same `pii_flag` rule must apply both sides | Confirm tabular introspector calls `pii_detector` |
-| Catalog refresh trigger (open question §3) | Affects both pipelines symmetrically | Default: rebuild on every upload/connect; defer auto-refresh |
-| `updated_at` semantics — per-Source vs per-Catalog | Affects how each pipeline writes | Recommend: per-Source `updated_at` + Catalog-level `generated_at` |
-| Dialect support scope for v1 | DB compiler must implement at least one dialect well | Recommend: Postgres first (matches app DB); MySQL second |
-| Test-fixture format for golden IRs | Both compilers test against golden IR → expected output | Recommend: shared `tests/fixtures/golden_irs.json`; each side adds expected SQL or DataFrame |
-| Logging conventions | structlog is already in place; both should log the same fields | Quick agreement: log `source_id`, `table_id`, `ir_version`, `elapsed_ms` |
-
-### Working rhythm (suggested)
-
-1. **Day 1** — 30-min sync to lock the decisions table. PR any contract/docstring changes that fall out.
-2. **Week 1** — both build introspectors + agree on the planner prompt format. PR in parallel; review each other's.
-3. **Week 2** — DB builds SQL compiler + DB executor; TAB builds pandas compiler + tabular executor. Both write golden tests against shared IR fixtures.
-4. **Week 3** — pair on dispatcher, QueryService, and chat endpoint integration. End-to-end smoke test.
-5. **Ongoing** — short daily standup, mostly to flag IR-shape questions and catalog-field additions *before* either side implements against an unconfirmed contract.
-
-Biggest risk: **silent contract drift** — one side adds a `QueryResult` field or assumes a new IR op exists, the other ships without it, and integration breaks at the dispatcher. The §0 lock + shared golden-IR fixtures are what prevent that.
-
-### Onboarding to Claude Code
-
-If you're new to Claude Code, before you start:
-
-1. Read `ARCHITECTURE.md` end-to-end (~10 min) — this is the source of truth.
-2. Skim this file (`REPO_CONTEXT.md`) — find your section in the ownership table.
-3. Read your owned files' docstrings — every stub explains its contract.
-4. Open Claude Code in this repo. When you ask Claude to implement a stub:
-   - Reference the file path + the contract it should follow
-   - Point it at `ARCHITECTURE.md` section if relevant (e.g. §7 for IR validation)
-   - Ask it to write the test first (golden IR fixtures), then the implementation
-   - Always review the diff — don't auto-accept
-
-Useful slash commands while working: `/review` (PR review), `/security-review` (audit pending changes).
-
----
-
-## Conventions & gotchas
-
-- **Async event loop on Windows**: `run.py` sets `WindowsSelectorEventLoopPolicy` because psycopg3 async needs it. Don't call `uvicorn` directly on Windows.
-- **Two Postgres engines**: `engine` (app tables) and `_pgvector_engine` (asyncpg with `prepared_statement_cache_size=0`) — the latter is required because PGVector emits `advisory_lock + CREATE EXTENSION` as a multi-statement string and asyncpg rejects multi-statement prepared queries. `init_db.py` creates the extension explicitly so `PGVector(create_extension=False)` skips that path.
-- **Read-only at every layer for user DBs**: IR validation + compiler whitelists + sqlglot SELECT-only check + read-only DB credentials + LIMIT enforcement + 30s timeout. Five layers; no single point of failure.
-- **Identifiers vs values**: identifiers (table/column names) come from the catalog and are inlined as quoted identifiers — they were verified at validation time so this is safe. Values from `IR.filters` are *always* parameterized, never inlined as strings.
-- **Credential encryption**: Fernet via `dataeyond__db__credential__key` env var; lives in `security/credentials.py`. Sensitive fields = `{"password", "service_account_json"}`.
-- **Settings env-var aliases**: `.env` uses double-underscore names (`azureai__api_key__4o`); `Settings` exposes them as `azureai_api_key_4o` via `Field(alias=...)`. Mind both forms when adding settings.
-- **Prompts**: `src/config/prompts/*.md` — `intent_router`, `query_planner`, `chatbot_system`, `guardrails` are all written. `chatbot_system` has `guardrails` appended so guardrails take precedence in conflict. `catalog_enricher.md` was deleted in KM-557. `config/agents/` folder deleted in Cleanup PR.
-- **Planner prompt parsing gotcha**: `query/planner/service.py` uses `SystemMessage(content=...)` not `("system", text)`. The tuple form causes LangChain to interpret `{...}` in `query_planner.md` as f-string variables and crash on every real invocation. Don't refactor back to tuples.
-- **Tests**: 146+ unit tests in place. Run with `uv run pytest`. Planner eval gated on `RUN_PLANNER_EVAL=1`; catalog store integration test gated on `RUN_INTEGRATION_TESTS=1`.
-
----
-
-## Recommended reading order
-
-1. `ARCHITECTURE.md` — design intent (the source of truth)
-2. `src/catalog/models.py` + `src/query/ir/models.py` — the two data shapes everything else moves between
-3. `src/query/ir/operators.py` + `src/security/pii_patterns.py` — the explicit whitelists / patterns
-4. Skim every `__init__.py`-level docstring under `src/catalog/`, `src/query/`, `src/agents/`, `src/pipeline/` — each describes the contract its module enforces
-5. `main.py` + `src/db/postgres/{connection,init_db}.py` — runtime bootstrap
-6. `ARCHITECTURE.md §10` — five open questions that haven't been decided yet
-
----
-
-## Open questions
-
-Resolved as Phase 2 landed:
-
-1. ✅ Catalog storage shape — Postgres `jsonb` row in `data_catalog` table, keyed by `user_id`.
-2. ❌ Unstructured files in catalog — still not modeled; router uses `source_hint` from the LLM instead.
-3. 🟡 Catalog refresh trigger — rebuild-on-upload-or-connect is the default. Explicit endpoint `POST /api/v1/data-catalog/rebuild` exists. Background TTL deferred.
-4. ✅ Joins out of v1 IR — confirmed; single-table only. Revisit when real queries need it.
-5. 🟡 PII `sample_values` — currently nulled out (skip). Mask/synthesize deferred.
-
----
-
-## Glossary
-
-- **Cu** — unstructured context (prose chunks)
-- **Cs** — schema context (DB tables/columns from catalog)
-- **Ct** — tabular context (file sheets/columns from catalog)
-- **IR** — intermediate representation (the JSON query shape)
-- **PII** — personally identifiable information
-- **ABC** — abstract base class