docs: ADR-025 (single-Chroma-collection rationale) + KI-018/019 backlog

ADR-025 documents the architectural decision to keep one Chroma
collection partitioned by doc_type metadata rather than splitting into
4 collections (policies, regulatory, reviews, profiles). Records the
trade-off: joint retrieval simplicity wins over per-type independence,
with explicit revisit triggers for when to split later.

KI-018 — rag/policies.duckdb is a 132-byte empty stub from ADR-004's
hybrid design that was never populated. Marketplace reads JSON
directly instead. Options: (A) populate from data/policy_facts/*.json
for SQL filtering, (B) remove dead code. Backlog.

KI-019 — kb/policies/*.md (224 hand-written policy summaries) are NOT
embedded. Retrieval would benefit from natural-language summary chunks
for "what does X cover?" style questions. Backlog.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

docs/40-evaluation/known-issues.md

@@ -352,3 +352,49 @@ filter or boost by facet when intent is clear. Live count:
 
 Verified locally; pushed to HF Dataset; live HF Space picks up
 on next rebuild.
+
+### KI-018 — `rag/policies.duckdb` stub is dead code (132 bytes, never populated)
+
+**Severity:** P3
+**Source:** `rag/policies.duckdb` is in the repo but empty (132-byte sqlite header only)
+**Discovered:** Data architecture audit 2026-05-14
+
+ADR-004 designed a hybrid structured (DuckDB) + vector (Chroma) split. The
+DuckDB half was never populated; the marketplace UI ended up reading
+`data/policy_facts/*.json` directly (one file per policy) at request time
+via the `/api/policies/all` endpoint. The 132-byte file in the repo is
+visually misleading — looks like a real store, isn't.
+
+**Fix plan (two options):**
+- (A) Populate it from `data/policy_facts/*.json` so the marketplace can
+  do SQL filters (sum-insured ≥ X AND room-rent-cap = no AND restoration
+  = unlimited). Worth ~30 min.
+- (B) Remove the file + the `import duckdb` + the `policies.duckdb` ADR
+  reference + add a note in ADR-025 explaining why structured filtering
+  reads JSON directly. Cleaner architecturally.
+
+**Recommendation:** (A) — once activated, the marketplace tab gets
+proper SQL filtering that's faster than the current N-JSON-load pattern.
+
+### KI-019 — `kb/policies/*.md` not embedded (224 human-readable summaries left out of retrieval)
+
+**Severity:** P3
+**Source:** `kb/policies/<policy_id>.md` files exist (224 markdown summaries) but the ingest pipeline only reads from `rag/extracted/*.json`. The natural-language summary text is never embedded.
+**Discovered:** Data architecture audit 2026-05-14
+
+Each policy has a hand-written or auto-generated `kb/policies/<id>.md`
+that summarizes the policy in 1-2 pages of natural English (e.g.
+"Care Supreme covers in-patient hospitalization across India with a
+36-month PED waiting…"). These would be EXCELLENT retrieval targets
+for "what does X cover?" style questions because they're already
+phrased for human reading. Currently retrieval matches against raw
+PDF-extracted prose which is denser + less semantic.
+
+**Fix plan:** Add a kb-markdown ingestion path in `rag/ingest.py` that
+walks `kb/policies/*.md`, splits at H2 boundaries (each H2 ≈ one
+schema-field discussion), embeds each section, and writes with
+`doc_type="summary"` so retrieval can prefer summary chunks when the
+intent is high-level Q&A.
+
+Expected impact: better grounding for natural-language "tell me about X"
+questions; cleaner citations (one summary chunk vs. 3-4 raw chunks).

docs/60-decisions/ADR-025-single-chroma-collection-with-metadata-partitioning.md

@@ -0,0 +1,77 @@
+# ADR-025: Single Chroma collection with metadata partitioning (not 4 separate collections)
+
+**Status:** Locked
+**Date:** 2026-05-14
+
+## Context
+
+The bot retrieves four logically distinct kinds of content at request time:
+
+1. **Policy chunks** (PDF wordings, brochures, CIS, prospectuses) — what the policy says
+2. **Regulatory chunks** (IRDAI master circulars, Insurance Act, etc.) — what the regulator mandates
+3. **Review chunks** (per-insurer claim metrics, aggregator ratings, Reddit/YouTube sentiment, news) — what users say
+4. **Profile chunks** (one chunk per active session, holding the user's age / dependents / income / conditions / budget) — what we know about THIS user
+
+A reasonable mental model is "four separate databases". The literal Chroma equivalent would be four collections (`policies`, `regulatory`, `reviews`, `profiles`) queried separately, with results merged in the orchestrator.
+
+## Decision
+
+**Keep ONE Chroma collection (`policies`) partitioned by a `doc_type` metadata field.** All four logical types co-exist in the same vector index. Retrieval is a single cosine-similarity call; the orchestrator applies metadata filters only when the user's intent is clearly type-specific (e.g., a regulatory-only question).
+
+```
+collection: policies (Chroma; 7,366 chunks; BGE-small 384-d cosine)
+  doc_type=wordings    (5,401 chunks)  — policy wordings
+  doc_type=brochure    (  611 chunks)
+  doc_type=cis         (  303 chunks)
+  doc_type=prospectus  (  483 chunks)
+  doc_type=regulatory  (  498 chunks)  — IRDAI + Insurance Act
+  doc_type=review      (   60 chunks)  — 10 insurers × 6 facets (KI-017)
+  doc_type=profile     (variable)      — one per active session_id
+```
+
+## Alternatives considered
+
+| Approach | Why rejected |
+|---|---|
+| 4 separate collections (`policies`, `regulatory`, `reviews`, `profiles`) | Joint retrieval requires N queries + manual merge + N embedding-pool warmups. Loses the natural "user profile boosted in the same retrieval as the policy text" trick. ~2× higher per-turn latency for marginal lifecycle benefit. |
+| Per-insurer collections | Cross-insurer comparison (the killer feature) becomes N queries. No upside. |
+| Per-doc_type with shared embedder | Same as 4-collection but with one embedder. Still N queries. |
+
+## Why one-collection wins for THIS bot
+
+1. **Joint retrieval is the headline RAG move.** The bot's most cited feature is "the brain sees policy text + regulatory mandate + user's own profile in the same context window". One Chroma query returns all three. Splitting would require an explicit fan-out + merge.
+
+2. **Profile-chunk boost is organic.** When `session_id` is passed to `retrieve()`, the chunk with `policy_id="profile_<session_id>"` is fetched separately and prepended to the top-k. This works cleanly because the profile lives in the same collection — it's a metadata lookup, not a cross-collection fetch.
+
+3. **Free-tier resource discipline.** Each Chroma collection in the deployment has its own HNSW index (~50 MB of RAM warmup per collection on first use). The HF Space free tier has 16 GB RAM total; 4 collections means ~200 MB of cold-start memory pressure for negligible UX gain.
+
+4. **Metadata partitioning is functionally equivalent.** When intent IS type-specific (e.g., the user asks about IRDAI), the orchestrator can apply `where={"doc_type": "regulatory"}` filters. The retrieval is single-query, same speed, same ergonomics as a separate-collection design — just with a 0.1ms metadata filter step.
+
+## Consequences
+
+**Positive:**
+
+- Joint retrieval stays cheap (one query, one warmup).
+- Profile chunk boost is a simple Chroma `get(ids=[profile_<session>])` call.
+- Single backup target (`rag/vectors/chroma.sqlite3` + HNSW binaries) — simpler dataset hygiene.
+- Mental model: "one vector store, one schema, doc_type tells you what kind".
+
+**Negative:**
+
+- A bad embedding for one doc_type can pollute global recall on unrelated queries. We mitigate with retrieval-floor + faithfulness gates.
+- We cannot iterate embedding models per type (e.g., a legal-domain embedder for regulatory only) without changing the architecture.
+
+## When we'd revisit
+
+| Condition | Migration |
+|---|---|
+| Per-type embedding models become necessary | Split off the heaviest type (regulatory) into its own collection with its own embedder. |
+| Multi-tenant deployment (profiles for thousands of concurrent users) | Move profiles to a separate `profiles` collection with TTL-based eviction policy. |
+| Review content grows to >5,000 chunks | Move reviews to a dedicated collection so policy retrieval isn't diluted. |
+
+## Related KIs and ADRs
+
+- **KI-017** — Enriched per-facet review chunks (10 → 60). Validates that doc_type partitioning scales for richer content without requiring a separate collection.
+- **KI-018** — Empty `rag/policies.duckdb` cleanup. The structured store from ADR-004 is currently dormant; structured filtering reads `data/policy_facts/*.json` directly.
+- **ADR-004** — Hybrid structured + vector retrieval (the original design — DuckDB for filtering, Chroma for citing).
+- **ADR-011** — Local BGE-small embeddings.