feat: KI-044 PCM pre-roll via AudioWorklet + 11 folder READMEs

KI-044 — Proper first-word capture in Live mode. Previous MediaRecorder
approach started recording only AFTER VAD declared speech-start, missing
the first 50-100 ms of the opening word. User report: "ensure that the
first word uttered is also captured, especially in live."

New approach (frontend/src/lib/useLiveConversation.ts):
• Inline AudioWorkletProcessor (Blob-URL registered, no static asset
routing) taps the raw PCM from the mic source. Every render quantum
(128 samples) is posted to the main thread as Float32.
• Main thread maintains a circular preroll buffer (~300 ms at the
AudioContext sample rate) while no utterance is in progress.
• When VAD fires speech-start: snapshot the preroll into the active
utterance buffer; continue accumulating subsequent samples.
• When VAD fires silence-end: encode utterance as 16-bit PCM WAV
(Sarvam Saarika native format) and post to onUtterance.
• Worklet stays running via source → workletNode → silentSink
(zero-gain GainNode) → ctx.destination so the audio graph doesn't GC
the node while keeping output silent — user's voice does NOT play
back through speakers.

PTT path (page.tsx::startRecording) is unaffected — PTT input is already
primed when the user clicks the 🎤 button.

11 folder READMEs added by a parallel agent:
• backend/, backend/providers/, rag/, tools/, tools/audit/, eval/,
tests/, data/, data/profiles/, audit_results/, frontend/ (replaces
the create-next-app boilerplate).
• Skipped folders that already had meaningful indices (docs/, kb/).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

audit_results/README.md

@@ -0,0 +1,67 @@
+# `audit_results/` — Audit output + production-readiness register
+
+Two artefact classes coexist here:
+
+1. **`ENTERPRISE_AUDIT.md`** — the **master defect log**. A living, hand-curated, severity-tagged register of every P0/P1/P2/P3 issue discovered during the readiness sprint, with evidence + fix status. Start here before any deploy decision.
+2. **`<run_id>/` subdirectories** — per-run output of the multi-persona audit framework (`tools/audit/`). Each is a complete, immutable snapshot of one full or partial 100-persona pass.
+
+## `ENTERPRISE_AUDIT.md`
+
+| Section | What's there |
+| --- | --- |
+| Executive scorecard | One-line status per domain: disk stability, data pipeline, observability, accuracy, latency, profile capture, language fairness, code hygiene, test coverage, voice UX, fact-find tone, secrets. |
+| Defect Register | D-001 … D-NNN. Each row: severity (P0–P3) · title · symptom · root cause · impact · fix status. |
+
+Severity legend: **P0** blocks production deployment · **P1** blocks enterprise procurement · **P2** quality / hygiene · **P3** nice-to-have.
+
+Status emoji: ✅ fixed · ⚠️ partial · 🟡 improving · 🔴 open.
+
+## Run-directory layout
+
+```
+audit_results/
+├── ENTERPRISE_AUDIT.md
+└── <run_id>/
+    ├── transcripts/
+    │   ├── P001.json
+    │   ├── P002.json
+    │   ├── P003.partial.json     (in-flight or interrupted persona)
+    │   └── …
+    ├── report.md                 (analyze.py rollup)
+    └── summary.json              (machine-readable rollup)
+```
+
+| Run-ID prefix | Meaning |
+| --- | --- |
+| `full_YYYYMMDD_HHMMSS` | Full 100-persona pass against the live HF Space. |
+| `postfix_YYYYMMDD_HHMMSS` | Targeted re-run after shipping a specific fix, used to confirm the defect is gone. |
+
+## Each persona transcript
+
+`transcripts/P###.json` captures the 30-turn flow for one persona. Per turn:
+
+| Field | Source |
+| --- | --- |
+| `user_text` | `tools/audit/flows.py` |
+| `reply_text` | live API |
+| `intent`, `brain_used` | `backend/orchestrator.py` |
+| `citations` | `backend/main.py` response |
+| `profile_updates` | `backend/profile_extractor.py` |
+| `faithfulness_passed`, `blocked` | `backend/faithfulness.py` |
+| `latency_ms` | wall-clock around the HTTP call |
+
+`.partial.json` files indicate the audit was interrupted (rate-limit pile-up, deploy mid-run, etc.); the runner is resumable and will pick up from the next persona on restart.
+
+## How runs flow into the register
+
+1. `python tools/audit/run_audit.py` writes per-persona transcripts.
+2. `python tools/audit/analyze.py audit_results/<run_id>/` rolls up `report.md` + `summary.json`.
+3. New defects → new `D-NNN` row in `ENTERPRISE_AUDIT.md`.
+4. Fixes → status flipped, run re-executed as `postfix_…`, evidence linked.
+
+## Related
+
+- `tools/audit/README.md` — framework that produces the run dirs
+- `tools/audit/personas.json`, `tools/audit/flows.json` — the deterministic inputs each run replays
+- Root `CLAUDE.md` — high-level project state; defers detail to this file for change history
+- [`kb/AUDIT_TRAIL.md`](../kb/AUDIT_TRAIL.md) — data-lineage doc that pairs with the behaviour-audit log

backend/README.md

@@ -0,0 +1,45 @@
+# `backend/` — FastAPI orchestrator
+
+FastAPI + Pydantic service that fronts every chat turn. The HTTP entry point is `main.py`; everything else is a typed module the orchestrator composes per turn.
+
+## Entry points
+
+| File | Role |
+| --- | --- |
+| `main.py` | FastAPI app, all HTTP routes (`/api/chat`, `/api/profile`, `/api/policies`, admin endpoints). Wires CORS, request validation, returns the typed response that the frontend's `openapi-typescript` codegen consumes. |
+| `orchestrator.py` | The brain of a turn: `classify_intent`, `pick_brain`, fact-find routing, profile-RAG injection, faithfulness gate dispatch. Pinned by `tests/test_routing_regression.py`. |
+| `config.py` | Pydantic-settings — single source of truth for env-vars, model IDs, chunk sizes, chain budgets. |
+
+## Per-turn helpers
+
+| File | Role | Related ADR |
+| --- | --- | --- |
+| `needs_finder.py` | 9-slot fact-find graph (`GRAPH`) + slot detection from free text. | [ADR-027](../docs/60-decisions/ADR-027-fact-find-llm-paraphraser.md) |
+| `question_paraphraser.py` | LLM rewrite of the canonical slot question so each session sounds fresh; verifier rejects off-slot drift. Cached per `(session_id, slot_id)`. | ADR-027 |
+| `fact_find_normalizer.py` | LLM-driven free-text → slot-value coercion (e.g. "32 lakh" → `3200000`). Goes through `NimChainLLM`, not a single client (KI-033). | — |
+| `profile_extractor.py` | LLM extractor that pulls profile updates out of conversational asides ("by the way, my dad has diabetes"). Chain-pattern, never a hardcoded model. | [ADR-022](../docs/60-decisions/ADR-022-conversational-profile-updates.md) |
+| `profile_store.py` | **NEW (KI-040).** Persistent named-profile JSON store under `data/profiles/`. O(1) name-keyed lookup; mirrors into `profile_rag` on every save. | — |
+| `profile_rag.py` | Embeds the user's profile as a Chroma chunk so the brain sees it alongside policy chunks for "what's best for me?" turns. | — |
+| `session_state.py` | In-memory session map; tracks fact-find progress + chat history per `session_id`. |
+| `faithfulness.py` | 4-gate hallucination guard (retrieval floor → citation integrity → regex numeric grounding → LLM judge). Blocks land in `logs/hallucinations.jsonl`. | — |
+| `scorecard.py` | Pure-function 6-sub-score scorer over the 62-field extracted JSON. No LLM. | — |
+| `translator.py` | Sarvam-M Indic ↔ English translator wrapper. | [ADR-006](../docs/60-decisions/ADR-006-sarvam-first-stack.md) |
+| `translation_check.py` | Post-hoc detector for mixed-script replies; flags Hinglish leakage. | — |
+| `persona.py` | The consultative-advisor system prompt + view-aware prompt overlays. | [ADR-008](../docs/60-decisions/ADR-008-consultative-advisor-persona.md), [ADR-021](../docs/60-decisions/ADR-021-view-aware-system-prompt.md) |
+| `voice_format.py` | Strips markdown / lists / bullet glyphs so TTS sounds natural. | — |
+| `premium_calculator.py` | Looks up `data/premiums/illustrative_premiums.json` + applies the documented scaling factors. Never claims a real quote. | [ADR-007](../docs/60-decisions/ADR-007-illustrative-pricing.md) |
+| `security.py` | Request rate-limiting, input sanitisation, admin-IP allowlist. | [ADR-023](../docs/60-decisions/ADR-023-admin-panel-ip-gated.md) |
+| `admin.py` | Admin-only routes (live LLM-health, usage rollups, hallucination tail). | ADR-023 |
+| `llm_health.py` | Lightweight probe that pings each provider and writes `data/llm_health.json` for the admin tab. | — |
+
+## Subdirectory
+
+`providers/` — concrete STT / TTS / LLM / embeddings client implementations. All LLM access goes through `NimChainLLM(chain=...)` from `providers/nvidia_nim_llm.py` — never instantiate a single-provider client directly. See `backend/providers/README.md`.
+
+## Where to read for what
+
+- **System tour:** root `README.md`
+- **Stable contracts a new contributor must know:** root `CLAUDE.md`
+- **Decisions with alternatives:** `docs/60-decisions/ADR-*.md`
+- **Routing invariants:** `tests/test_routing_regression.py`
+- **Defect register:** `audit_results/ENTERPRISE_AUDIT.md`

backend/providers/README.md

@@ -0,0 +1,40 @@
+# `backend/providers/` — STT / TTS / LLM / embedding clients
+
+Every external model is fronted by a small typed client here. The orchestrator and helper modules **only** ever import provider symbols from this folder — single import surface so a provider swap is local.
+
+## Files
+
+| File | Provider | Role | Notes |
+| --- | --- | --- | --- |
+| `base.py` | — | Abstract `LLM`, `STT`, `TTS`, `Embeddings` Protocols. Every concrete client conforms. | — |
+| `nvidia_nim_llm.py` | NVIDIA NIM | Core chain runner — `NimChainLLM(chain=[...])` walks a fallback ladder under a wall-clock budget. Exposes `get_brain_llm()`, `get_fast_brain_llm()`, `get_judge_llm()`. Also home of `_balanced_brain_chain()` (50/50 NIM ↔ Groq rotator). | [ADR-019](../../docs/60-decisions/ADR-019-nim-single-provider-consolidation.md), [ADR-026](../../docs/60-decisions/ADR-026-provider-load-balancing.md) |
+| `groq_llm.py` | Groq | Single-call Llama-3.3-70B client. Used as the 50% load-balance primary for the brain chain, never standalone. | [ADR-026](../../docs/60-decisions/ADR-026-provider-load-balancing.md) |
+| `openrouter_llm.py` | OpenRouter | Multi-model fallback rung (DeepSeek-V3 etc.) for chains; rarely the primary in production. | — |
+| `sarvam_llm.py` | Sarvam-M | Indic-aware LLM; on the judge / translator fallback chains and used by `backend/translator.py`. | [ADR-006](../../docs/60-decisions/ADR-006-sarvam-first-stack.md) |
+| `sarvam_stt.py` | Sarvam Saarika v2.5 | Speech-to-text (10 Indic languages + English). | ADR-006 |
+| `sarvam_tts.py` | Sarvam Bulbul v2 | Text-to-speech; returns base64 WAV the frontend mounts in the in-DOM `<audio>` element. | ADR-006 |
+| `voyage_embeddings.py` | Voyage AI | Original ingest-time embedder. **Not on the hot path** — query-time uses Chroma vectors directly. Configured in `.env` for occasional re-ingest. | [ADR-011](../../docs/60-decisions/ADR-011-bge-local-embeddings.md) |
+| `local_embeddings.py` | BGE-small-en-v1.5 (local) | The actual production embedder. 384-dim, free, no rate cap. | ADR-011 |
+| `_smoke_test.py` | — | Stand-alone connectivity probe — hits every provider, prints latency + first 80 chars. Run before a long audit. | — |
+
+## Invariants
+
+- **Never instantiate `NvidiaNimLLM(model=...)` directly.** Always go through `NimChainLLM(chain=...)` so the call survives single-pool rate limits. KI-033 migrated the last two stragglers (`profile_extractor`, `fact_find_normalizer`).
+- **Brain chains preserve family diversity.** Qwen brain ↔ Mistral judge — failovers can't accidentally collapse to a single family and produce circular grading.
+- **Per-call random for load-balance, not a shared counter.** `random.random()` is evaluated at chain-construction time; a shared `itertools.cycle` breaks under async concurrency (see ADR-026 "Why per-call random").
+
+## Chain budgets
+
+| Chain | Per-link timeout (s) | Total budget (s) | Where set |
+| --- | --- | --- | --- |
+| Brain | 20 | 35 | `nvidia_nim_llm.py::get_brain_llm` |
+| Fast brain | 12 | 22 | `nvidia_nim_llm.py::get_fast_brain_llm` |
+| Judge | 30 | 75 | `nvidia_nim_llm.py::get_judge_llm` |
+
+Per-link timeout is dynamically clipped to remaining budget.
+
+## Related
+
+- [ADR-006](../../docs/60-decisions/ADR-006-sarvam-first-stack.md), [ADR-011](../../docs/60-decisions/ADR-011-bge-local-embeddings.md), [ADR-019](../../docs/60-decisions/ADR-019-nim-single-provider-consolidation.md), [ADR-026](../../docs/60-decisions/ADR-026-provider-load-balancing.md)
+- `tests/test_routing_regression.py::TestProviderLoadBalancing` — pins the 50/50 split
+- `data/llm_health.json` — last health-probe snapshot surfaced in the admin tab

data/README.md

@@ -0,0 +1,45 @@
+# `data/` — Runtime + marketplace data
+
+Three classes of file live here, intentionally side-by-side:
+
+1. **Runtime state** — written by the live server during normal operation (`profiles/`, `sessions/`, `llm_health.json`, `llm_usage.jsonl`).
+2. **Pre-computed marketplace data** — curated artefacts the server reads on every relevant turn (`policy_facts/`, `premiums/`, `reviews/`).
+3. **Source/lineage maps** — human-readable manifests of where every claim traces back to (`corpus_urls.md`, `regulatory_urls.md`, `information_source_map.md`).
+
+The structured policy schema and PDFs themselves live under `rag/`. This folder is downstream.
+
+## Top-level files
+
+| File | What it is | Owner |
+| --- | --- | --- |
+| `corpus_urls.md` | Discovery manifest — every PDF URL ingested into `rag/corpus/`. | discovery agent / `tools/check_link_rot.py` |
+| `regulatory_urls.md` | IRDAI / regulatory PDF URLs. See [ADR-017](../docs/60-decisions/ADR-017-irdai-corpus-playwright-rescue.md). | discovery agent |
+| `information_source_map.md` | Human-readable claim → URL → verdict map. Master audit doc for the Source Methodology directive. Mirror of `eval/info_source_map.json`. | `tools/info_source_map.py` |
+| `llm_health.json` | Last per-provider health-probe snapshot (latency, success, last error). Powers the admin tab. | `backend/llm_health.py` |
+| `llm_usage.jsonl` | Append-only per-call log: provider, model, tokens, latency, success. Aggregated in the admin tab. | `backend/main.py` |
+
+## Subdirectories
+
+| Path | Class | Contents |
+| --- | --- | --- |
+| `profiles/` | runtime | Persistent named-profile JSON store (KI-040). One file per user, normalised-name slug. See `data/profiles/README.md`. |
+| `sessions/` | runtime | Per-session conversation state JSONs. Ephemeral — pruned periodically. Currently includes `anonymous.json` (no-name fallback). |
+| `policy_facts/` | pre-computed | **256 curated JSONs**, one per policy variant. Each field carries `{value, unit?, source_pdf_path, source_quote}` provenance. The Indian-BFSI-audit-grade machine source; `kb/policies/*.md` are the human-readable mirror. See `_curation_report.md` for the three batches that built it. |
+| `policies/` | pre-computed | Subfolder per insurer with PDFs / supplementary text used for one-off lookups outside the main ingest pipeline. |
+| `premiums/` | pre-computed | `illustrative_premiums.json` — sample starting premiums pulled from PolicyBazaar / JoinDitto / Beshak + insurer rate cards (2026-05-13). Refreshed by `tools/refresh_premiums.py`. **Illustrative only** per [ADR-007](../docs/60-decisions/ADR-007-illustrative-pricing.md). |
+| `reviews/` | pre-computed | One JSON per insurer with IRDAI claim-settlement metrics, complaints/10K, aggregator sentiment, news tone. Index + leaderboard in `reviews/INDEX.md`. Source: IRDAI Annual Report 2023-24. |
+
+## Provenance + KPIs
+
+| Metric | Value (2026-05-14) | Where to verify |
+| --- | --- | --- |
+| Curated policy variants | 256 | `data/policy_facts/` file count |
+| Per-policy avg field completeness | 83.5% (Batch 1) | `data/policy_facts/_curation_report.md` |
+| Information-source-map verdicts | ✅ 798 · ⚠️ 321 · ❌ 0 · ⏳ 1385 | `eval/info_source_map.json` |
+
+## Related
+
+- [`kb/AUDIT_TRAIL.md`](../kb/AUDIT_TRAIL.md) — end-to-end lineage; `data/policy_facts/` is stage 8 output
+- [`kb/INDEX.md`](../kb/INDEX.md) — policy index with completeness % per file
+- [ADR-007](../docs/60-decisions/ADR-007-illustrative-pricing.md) — pricing is illustrative, never a real quote
+- [ADR-009](../docs/60-decisions/ADR-009-19-insurer-comprehensive-schema.md) — 19-insurer scope + 48-field schema

data/profiles/README.md

@@ -0,0 +1,67 @@
+# `data/profiles/` — Named-profile JSON store
+
+Persistent name-keyed profile store introduced by **KI-040 (2026-05-14)**. Lets a returning visitor say their name and have the bot recognise them + auto-load the stored profile, so they don't have to walk the 9-slot fact-find again.
+
+Canonical store: `backend/profile_store.py`. ADR (deferred — code self-documents).
+
+## File layout
+
+```
+data/profiles/
+└── <normalised-name>.json     (one file per user)
+```
+
+- **Filename slug:** lowercase + alpha-only of the user's first name. `"Rohit"` and `"rohit."` both resolve to `rohit.json`.
+- **Inside the file:** the original capitalised display name is preserved.
+
+## JSON shape
+
+```json
+{
+  "display_name": "Rohit",
+  "first_seen": "2026-05-14T09:12:33Z",
+  "last_seen":  "2026-05-14T22:41:08Z",
+  "sessions":   ["sess_abc…", "sess_def…"],
+  "profile": {
+    "age": 32,
+    "dependents": "spouse+1_child",
+    "city_tier": "metro",
+    "...": "..."
+  }
+}
+```
+
+Schema: `profile` mirrors the 9-slot `GRAPH` in `backend/needs_finder.py`. Everything else is bookkeeping.
+
+## Two-layer sync
+
+| Layer | Purpose | Where |
+| --- | --- | --- |
+| JSON (this folder) | Canonical, O(1) name-keyed lookup, deterministic, human-readable, manually editable. | `backend/profile_store.py::save_profile` |
+| Chroma vector chunk | Re-embedded on every save so the brain sees the profile alongside policy chunks at retrieval time — powers "what's best for me?" questions. | `backend/profile_rag.py::upsert_profile_chunk` |
+
+Both stay in sync: `save_profile()` fires the Chroma upsert in the same call. Embedding cost is once per update, not per query.
+
+## Why JSON, not Chroma-only
+
+The original design considered embedding-only. The "why JSON" trade-offs:
+
+- **Deterministic name lookup.** `Rohit` → `rohit.json` is exact; vector search is approximate and can collide on common first names.
+- **Human-readable.** A BFSI auditor can `cat` the file and see the full profile.
+- **Manually editable.** Quick repair without a re-embed pipeline.
+- **No HNSW bloat exposure.** Profile updates do not touch the policy vector store ([ADR-029](../docs/60-decisions/ADR-029-disk-storage-hardening.md)).
+
+The Chroma chunk is purely a retrieval-time view of the canonical JSON.
+
+## Privacy + retention
+
+- Profiles are local to the deployed instance. No third-party share.
+- Per [ADR-010](../docs/60-decisions/ADR-010-secret-handling.md), the folder is not exposed via the HTTP API except through the user's own `session_id`.
+- The folder is committed empty (placeholder) — actual profiles are runtime artefacts.
+
+## Related
+
+- `backend/profile_store.py` — the canonical store implementation
+- `backend/profile_extractor.py` + [ADR-022](../docs/60-decisions/ADR-022-conversational-profile-updates.md) — how conversational asides flow into the profile
+- `backend/profile_rag.py` — the embedding mirror
+- `backend/needs_finder.py::GRAPH` — the 9-slot schema the `profile` block conforms to

eval/README.md

@@ -0,0 +1,54 @@
+# `eval/` — Gold-QA harness
+
+Numerical accuracy + grounding eval. Walks a fixed list of curated questions through the orchestrator in-process (fast, no HTTP) and grades each reply with both a regex hard-facts checker and an LLM-judge of a different family from the brain.
+
+`eval/` is the **objective** quality bar; `tools/audit/` is the **behavioural** stress test. Both feed the readiness register in `audit_results/ENTERPRISE_AUDIT.md`.
+
+## Files
+
+| File | Role |
+| --- | --- |
+| `generate_gold.py` | Builds `gold_qa.json` from `data/policy_facts/`: for each curated field with a verbatim quote, emits a natural-language question + expected answer + expected citation. |
+| `gold_qa.json` | 96-Q gold set. Each entry: `{policy_id, question, expected_answer, expected_regex, source_quote, source_pdf}`. |
+| `run.py` | Runner. For each pair: calls `backend.orchestrator.handle_turn` with `policy_filter_ids=[pair.policy_id]` so retrieval is scoped to the policy under test. Grades each reply twice — regex hard-facts + Groq Llama judge (different family from the NIM brain → non-circular). |
+| `results.json` | Machine-readable last-run results. |
+| `results.md` | Human-readable last-run report — per-question pass/fail, per-category rollup, hallucination breakdown. |
+| `info_source_map.json` | Generated by `tools/info_source_map.py`. Claim → URL → verdict (✅ 798 / ⚠️ 321 / ❌ 0 / ⏳ 1385 as of 2026-05-14). The canonical source-grounding KPI. |
+| `verified_urls.json` | HEAD-check verdict on every URL in the corpus / facts. Generated by `tools/verify_urls.py`. |
+| `reviews_url_verification.json` | URL-validation output for `data/reviews/<insurer>.json`. |
+| `chunk_sweep_results.json`, `chunk_diagnostic.json` | Outputs of `tools/chunk_sweep.py` — chunk-size / overlap grid. See [ADR-018](../docs/60-decisions/ADR-018-chunk-size-sweep-deferred.md). |
+
+## Usage
+
+```bash
+# Full 96-Q eval
+python -m eval.run
+
+# Smoke
+python -m eval.run --limit 30
+
+# Scoped to one policy
+python -m eval.run --policy hdfc-ergo__optima-secure
+
+# Regenerate the gold set after curation refresh
+python -m eval.generate_gold
+```
+
+## Grading
+
+| Layer | What it checks | Output field |
+| --- | --- | --- |
+| Regex hard-facts | Expected numeric / boolean / waiting-period value appears verbatim in `reply_text`. | `regex_pass` |
+| LLM judge (Groq Llama-3.3-70B) | Reply *semantically* answers the question and cites the expected source. Different family from the NIM Qwen brain → non-circular. | `judge_pass` |
+| Faithfulness gate | The 4-gate guard in `backend/faithfulness.py` already ran during `handle_turn`; eval records whether it blocked. | `blocked` |
+
+## Acceptance bar
+
+Per `audit_results/ENTERPRISE_AUDIT.md` D-003: **≥ 90% on the 96-Q gold-QA set is the P0 gate for enterprise deployment.** Current state lives in `results.md`.
+
+## Related
+
+- [`backend/faithfulness.py`](../backend/faithfulness.py) — the 4-gate guard that produces the `blocked` verdict
+- [`kb/AUDIT_TRAIL.md`](../kb/AUDIT_TRAIL.md) § "Live bot replies" — provenance trail for any flagged reply
+- `tools/info_source_map.py` — generator of `info_source_map.json`
+- [ADR-014](../docs/60-decisions/ADR-014-groq-llama-grader.md) (superseded — but the Groq-judge-of-different-family principle survives)

frontend/README.md

@@ -1,36 +1,50 @@
-This is a [Next.js](https://nextjs.org) project bootstrapped with [`create-next-app`](https://nextjs.org/docs/app/api-reference/cli/create-next-app).
+# `frontend/` — Next.js 14 (App Router) UI
 
-## Getting Started
+The chat UI, profile builder, scorecard, and admin panel. Single-page-ish — almost everything lives in `src/app/page.tsx` with `src/lib/` carrying the API client, the EN ↔ हिं i18n table, and the live-conversation hook.
 
-First, run the development server:
+For Claude / agent-specific rules (e.g. the "this is NOT the Next.js you know" note) see `AGENTS.md` in this folder — `CLAUDE.md` is an alias of it.
 
-```bash
-npm run dev
-# or
-yarn dev
-# or
-pnpm dev
-# or
-bun dev
-```
+## Entry points
 
-Open [http://localhost:3000](http://localhost:3000) with your browser to see the result.
+| File | Role |
+| --- | --- |
+| `src/app/page.tsx` | The chat surface. Hosts the `Message` component (which mounts the in-DOM `<audio>` element for TTS), the toolbar (Live toggle + push-to-talk), the tab switcher (Chat / Profile Builder / Scorecard / Admin), and every view's render. |
+| `src/app/layout.tsx` | Root layout + font wiring (Geist via `next/font`). |
+| `src/app/globals.css` | Tailwind v4 entrypoint + the shadcn/ui token layer ([ADR-013](../docs/60-decisions/ADR-013-tailwind-shadcn-ui.md)). |
+| `src/lib/api.ts` | Typed API client. Generated types via `openapi-typescript` from the FastAPI OpenAPI schema ([ADR-015](../docs/60-decisions/ADR-015-openapi-typescript-codegen.md)). |
+| `src/lib/useLiveConversation.ts` | The continuously-open-mic VAD hook that powers Live mode + barge-in. State persists in `localStorage.insurance_live_pref` ([ADR-028](../docs/60-decisions/ADR-028-voice-ux-live-default.md)). |
+| `src/lib/i18n.ts` | EN ↔ हिं strings + the 13-term `GLOSSARY` mirrored to [`kb/methodology/glossary.json`](../kb/methodology/glossary.json). |
 
-You can start editing the page by modifying `app/page.tsx`. The page auto-updates as you edit the file.
+## Voice UX invariants ([ADR-028](../docs/60-decisions/ADR-028-voice-ux-live-default.md))
 
-This project uses [`next/font`](https://nextjs.org/docs/app/building-your-application/optimizing/fonts) to automatically optimize and load [Geist](https://vercel.com/font), a new font family for Vercel.
+- **One default voice mode + one fallback.** Live mode is the default; the toolbar pill toggles it (green = on, red = off). The 🎤 push-to-talk button suspends Live for one turn → captures with VAD silence-cutoff → resumes Live if the user preference is still on.
+- **Hands-free mode was removed in KI-027.** Any reference to it in code or docs is stale.
+- **Bot TTS plays via the in-DOM `<audio>` element inside `Message`** (autoplay-on-mount via ref'd `useEffect`). **Never use `new Audio(url).play()`** — those detached instances are invisible to the `document.querySelectorAll("audio").pause()` call in the barge-in handler, so they keep playing under the user's speech.
 
-## Learn More
+## Tooling
 
-To learn more about Next.js, take a look at the following resources:
+| Concern | Where |
+| --- | --- |
+| Build / dev / lint | `package.json` scripts (`dev`, `build`, `lint`). |
+| Lint config | `eslint.config.mjs`. |
+| TS config | `tsconfig.json`. |
+| Postcss / Tailwind | `postcss.config.mjs`. |
+| Next config | `next.config.ts`. |
+| Static assets | `public/`. |
 
-- [Next.js Documentation](https://nextjs.org/docs) - learn about Next.js features and API.
-- [Learn Next.js](https://nextjs.org/learn) - an interactive Next.js tutorial.
+## Develop locally
 
-You can check out [the Next.js GitHub repository](https://github.com/vercel/next.js) - your feedback and contributions are welcome!
+```bash
+cd frontend
+npm install        # or pnpm / bun
+npm run dev        # localhost:3000 — points at the FastAPI on :8000 by default
+```
 
-## Deploy on Vercel
+Point at a different backend with `NEXT_PUBLIC_API_BASE=...` (see `src/lib/api.ts`).
 
-The easiest way to deploy your Next.js app is to use the [Vercel Platform](https://vercel.com/new?utm_medium=default-template&filter=next.js&utm_source=create-next-app&utm_campaign=create-next-app-readme) from the creators of Next.js.
+## Related
 
-Check out our [Next.js deployment documentation](https://nextjs.org/docs/app/building-your-application/deploying) for more details.
+- `AGENTS.md` (this folder) — required reading for AI agents touching this code
+- Root `CLAUDE.md` § Voice UX (ADR-028) — the canonical voice-UX cheat-sheet
+- [ADR-005](../docs/60-decisions/ADR-005-nextjs-fastapi-frontend.md) · [ADR-013](../docs/60-decisions/ADR-013-tailwind-shadcn-ui.md) · [ADR-015](../docs/60-decisions/ADR-015-openapi-typescript-codegen.md) · [ADR-021](../docs/60-decisions/ADR-021-view-aware-system-prompt.md) · [ADR-028](../docs/60-decisions/ADR-028-voice-ux-live-default.md)
+- [`backend/main.py`](../backend/main.py) — the API the frontend talks to

frontend/src/lib/useLiveConversation.ts

@@ -3,39 +3,41 @@
 /**
  * useLiveConversation — full-duplex voice mode with barge-in.
  *
- * Differences from the existing push-to-talk + Hands-free toggle:
+ * KI-044 (2026-05-14) — PCM pre-roll via AudioWorklet.
+ * --------------------------------------------------------------------
+ * Previous implementation started a MediaRecorder ONLY when VAD declared
+ * "speech started" — by which point ~50-80 ms of the first word was
+ * already past the mic. Users reported the bot only hearing "ello" / "i
+ * am" rather than "hello" / "hi i am".
  *
- *   - Mic is OPEN continuously while live mode is on (single getUserMedia
- *     stream, not opened/closed per turn).
- *   - VAD (RMS on AnalyserNode + frame counters) detects when the user
- *     starts and stops speaking, with no button press.
- *   - When VAD detects speech start, we immediately:
- *       * pause + clear every <audio> currently playing (kill the bot's
- *         in-progress TTS reply mid-sentence),
- *       * abort the in-flight /api/chat fetch via AbortController,
- *       * start a new MediaRecorder for the user's utterance.
- *   - When VAD detects silence for >600ms after speech, we stop the
- *     recorder and POST the blob to the supplied `onUtterance` handler
- *     (which the page wires to transcribe → chat).
+ * Current implementation:
+ *   - Single getUserMedia stream + AudioContext stay open while Live is on.
+ *   - An AudioWorkletNode taps the raw PCM from the source — every render
+ *     quantum (128 samples) is posted back to the main thread as Float32.
+ *   - The main thread keeps a circular preroll buffer (~300 ms / 4800
+ *     samples at 16 kHz) when no utterance is in progress.
+ *   - When VAD fires speech-start, the preroll is snapshotted into the
+ *     active utterance buffer and subsequent samples are appended.
+ *   - When VAD fires silence-end, we encode the full utterance (preroll +
+ *     speech + small post-roll) as a 16-bit PCM WAV (Sarvam Saarika's
+ *     native format) and post it to `onUtterance`.
+ *   - VAD itself still runs off the AnalyserNode (separate path) so its
+ *     sensitivity tuning is independent from the PCM capture rate.
  *
- * The hook returns an AbortController slot the caller assigns to its
- * own in-flight fetches so barge-in can cancel them.
+ * Result: the user's first phoneme is in the blob. No more "ello".
+ *
+ * Push-to-talk path (page.tsx::startRecording) is unaffected — PTT
+ * recording starts when the user clicks, the input is already primed.
  */
 
 import { useCallback, useEffect, useRef, useState } from "react";
 
 export type LiveConversationOptions = {
-  /** Called when the user finishes an utterance — pass it the audio blob. */
   onUtterance: (blob: Blob, abort: AbortController) => Promise<void>;
-  /** Called when VAD detects speech start (so the UI can show "listening…"). */
   onSpeechStart?: () => void;
-  /** Called when VAD detects speech end (so the UI can show "thinking…"). */
   onSpeechEnd?: () => void;
-  /** RMS threshold above which we declare "speech". Tune in browser. */
   rmsThreshold?: number;
-  /** Consecutive loud frames needed to start recording (debounce). */
   speechStartFrames?: number;
-  /** Consecutive quiet frames needed to stop recording (~16 ms/frame). */
   silenceEndFrames?: number;
 };
 
@@ -44,24 +46,75 @@ export type LiveConversationState = {
   recording: boolean;
   micPermissionDenied: boolean;
   setLive: (v: boolean) => void;
-  /** Caller-managed abort slot for in-flight fetches; VAD aborts it on speech. */
   inflightAbortRef: React.MutableRefObject<AbortController | null>;
 };
 
 const DEFAULTS = {
-  // KI-041 (2026-05-14) — sensitivity bumped. Previous threshold 28 was high
-  // enough that normal speaking volume in a moderately-quiet room didn't
-  // trigger barge-in detection, so users reported "speaking over the bot
-  // doesn't interrupt it". 18 catches typical conversational volume reliably
-  // while still rejecting room hum / breathing / keyboard clatter.
+  // KI-041 — sensitivity bumped to catch normal conversational volume.
   rmsThreshold: 18,
-  // ~48ms of speech to fire. Previous 5 frames (~80ms) added perceptible
-  // latency on barge-in; 3 frames keeps false-positive immunity but cuts
-  // the response time by ~32ms.
-  speechStartFrames: 3,
+  // KI-043/044 — fire on first loud frame (~16 ms) so the preroll buffer
+  // snapshot captures as much pre-trigger audio as possible.
+  speechStartFrames: 1,
   silenceEndFrames: 40, // ~640 ms of silence to declare utterance end
+  minUtteranceMs: 400,
+  // KI-044 — How much pre-trigger PCM we keep in the rolling buffer.
+  // 300 ms is generous; covers the ~80 ms VAD latency + ~100 ms of
+  // user onset before the first detectable frame, with margin.
+  prerollMs: 300,
 };
 
+// AudioWorklet processor source — inlined as a Blob URL so we don't need
+// a separate static asset route. Runs on the audio thread; posts each
+// 128-sample mono Float32Array back to the main thread.
+const WORKLET_SOURCE = `
+class PCMCaptureProcessor extends AudioWorkletProcessor {
+  process(inputs) {
+    const input = inputs[0];
+    if (input && input[0]) {
+      // Clone the buffer so it survives the transfer; the original is
+      // a view onto the audio thread's internal buffer.
+      this.port.postMessage(input[0].slice(0));
+    }
+    return true;
+  }
+}
+registerProcessor('pcm-capture', PCMCaptureProcessor);
+`;
+
+// Encode Float32 samples as a 16-bit PCM WAV file (mono). Returns a Blob
+// suitable for `<input type=file>` upload to /api/transcribe.
+function encodeWAV(samples: Float32Array, sampleRate: number): Blob {
+  const headerSize = 44;
+  const dataSize = samples.length * 2; // 16-bit
+  const buffer = new ArrayBuffer(headerSize + dataSize);
+  const view = new DataView(buffer);
+
+  const writeString = (offset: number, str: string) => {
+    for (let i = 0; i < str.length; i++) view.setUint8(offset + i, str.charCodeAt(i));
+  };
+
+  writeString(0, "RIFF");
+  view.setUint32(4, 36 + dataSize, true);
+  writeString(8, "WAVE");
+  writeString(12, "fmt ");
+  view.setUint32(16, 16, true);          // PCM chunk size
+  view.setUint16(20, 1, true);           // PCM format
+  view.setUint16(22, 1, true);           // mono
+  view.setUint32(24, sampleRate, true);
+  view.setUint32(28, sampleRate * 2, true); // byte rate
+  view.setUint16(32, 2, true);           // block align
+  view.setUint16(34, 16, true);          // bits per sample
+  writeString(36, "data");
+  view.setUint32(40, dataSize, true);
+
+  let offset = headerSize;
+  for (let i = 0; i < samples.length; i++, offset += 2) {
+    const s = Math.max(-1, Math.min(1, samples[i]));
+    view.setInt16(offset, s < 0 ? s * 0x8000 : s * 0x7fff, true);
+  }
+  return new Blob([buffer], { type: "audio/wav" });
+}
+
 export function useLiveConversation(opts: LiveConversationOptions): LiveConversationState {
   const [live, setLive] = useState(false);
   const [recording, setRecording] = useState(false);
@@ -71,11 +124,18 @@ export function useLiveConversation(opts: LiveConversationOptions): LiveConversa
   const audioCtxRef = useRef<AudioContext | null>(null);
   const analyserRef = useRef<AnalyserNode | null>(null);
   const sourceRef = useRef<MediaStreamAudioSourceNode | null>(null);
-  const recorderRef = useRef<MediaRecorder | null>(null);
-  const chunksRef = useRef<Blob[]>([]);
-  const recordingRef = useRef(false); // sync ref for VAD loop
+  const workletRef = useRef<AudioWorkletNode | null>(null);
+  const workletUrlRef = useRef<string | null>(null);
+  const sampleRateRef = useRef<number>(48000);
+
+  // KI-044 — sample-level capture buffers
+  const prerollRef = useRef<Float32Array[]>([]);
+  const speechBufferRef = useRef<Float32Array[]>([]);
+
+  const recordingRef = useRef(false);
   const rafIdRef = useRef<number | null>(null);
   const inflightAbortRef = useRef<AbortController | null>(null);
+  const recStartTsRef = useRef<number>(0);
 
   const onUtteranceRef = useRef(opts.onUtterance);
   const onSpeechStartRef = useRef(opts.onSpeechStart);
@@ -90,75 +150,80 @@ export function useLiveConversation(opts: LiveConversationOptions): LiveConversa
     rmsThreshold: opts.rmsThreshold ?? DEFAULTS.rmsThreshold,
     speechStartFrames: opts.speechStartFrames ?? DEFAULTS.speechStartFrames,
     silenceEndFrames: opts.silenceEndFrames ?? DEFAULTS.silenceEndFrames,
+    minUtteranceMs: DEFAULTS.minUtteranceMs,
+    prerollMs: DEFAULTS.prerollMs,
   };
 
-  const stopRecording = useCallback(() => {
-    if (recorderRef.current && recorderRef.current.state !== "inactive") {
-      try { recorderRef.current.stop(); } catch {}
-    }
-  }, []);
-
   const interruptBotAudio = useCallback(() => {
-    // Pause + reset every audio element in the DOM. Bot replies use plain
-    // <audio> elements; killing src forces the loaded buffer to drop.
     if (typeof document !== "undefined") {
       document.querySelectorAll("audio").forEach((a) => {
         try {
           a.pause();
-          // Don't blank src — let the existing buffer GC but leave the
-          // element so the chat history scroll position doesn't jump.
           a.currentTime = a.duration || 0;
         } catch {}
       });
     }
   }, []);
 
-  const startRecording = useCallback(() => {
-    if (!streamRef.current) return;
-    const mime = MediaRecorder.isTypeSupported("audio/webm")
-      ? "audio/webm"
-      : "";
-    const rec = mime
-      ? new MediaRecorder(streamRef.current, { mimeType: mime })
-      : new MediaRecorder(streamRef.current);
-    chunksRef.current = [];
-    rec.ondataavailable = (e) => {
-      if (e.data && e.data.size > 0) chunksRef.current.push(e.data);
-    };
-    rec.onstop = async () => {
-      recordingRef.current = false;
-      setRecording(false);
-      if (chunksRef.current.length === 0) return;
-      const blob = new Blob(chunksRef.current, {
-        type: rec.mimeType || "audio/webm",
-      });
-      // Reject blobs that are almost certainly silence or VAD false-trips.
-      if (blob.size < 3000) return;
-      onSpeechEndRef.current?.();
-      const abort = new AbortController();
-      inflightAbortRef.current = abort;
-      try {
-        await onUtteranceRef.current(blob, abort);
-      } catch (e) {
-        const name = (e as { name?: string })?.name;
-        if (name !== "AbortError") {
-          // surface to console; the UI's existing error toast will fire too
-          // eslint-disable-next-line no-console
-          console.error("[live-mode] utterance handler failed:", e);
-        }
-      } finally {
-        if (inflightAbortRef.current === abort) {
-          inflightAbortRef.current = null;
-        }
-      }
-    };
-    recorderRef.current = rec;
+  // KI-044 — open speech capture: snapshot the preroll into speechBuffer,
+  // flag recording, fire callbacks. The PCM keeps flowing via the worklet
+  // port; we just toggle where it lands.
+  const beginSpeechCapture = useCallback(() => {
+    speechBufferRef.current = [...prerollRef.current];
+    prerollRef.current = [];
     recordingRef.current = true;
+    recStartTsRef.current = Date.now();
     setRecording(true);
     onSpeechStartRef.current?.();
-    rec.start();
   }, []);
 
+  // KI-044 — close speech capture: encode WAV, run guards, fire onUtterance.
+  const endSpeechCapture = useCallback(async () => {
+    if (!recordingRef.current) return;
+    recordingRef.current = false;
+    setRecording(false);
+    const durationMs = Date.now() - (recStartTsRef.current || Date.now());
+    const chunks = speechBufferRef.current;
+    speechBufferRef.current = [];
+
+    if (chunks.length === 0) return;
+    if (durationMs < cfg.minUtteranceMs) {
+      // eslint-disable-next-line no-console
+      console.debug("[live-mode] dropped short utterance", durationMs, "ms");
+      return;
+    }
+
+    // Concatenate Float32Array chunks
+    let totalSamples = 0;
+    for (const c of chunks) totalSamples += c.length;
+    const merged = new Float32Array(totalSamples);
+    let offset = 0;
+    for (const c of chunks) {
+      merged.set(c, offset);
+      offset += c.length;
+    }
+
+    const wav = encodeWAV(merged, sampleRateRef.current);
+    if (wav.size < 3000) return; // floor (matches prior heuristic)
+
+    onSpeechEndRef.current?.();
+    const abort = new AbortController();
+    inflightAbortRef.current = abort;
+    try {
+      await onUtteranceRef.current(wav, abort);
+    } catch (e) {
+      const name = (e as { name?: string })?.name;
+      if (name !== "AbortError") {
+        // eslint-disable-next-line no-console
+        console.error("[live-mode] utterance handler failed:", e);
+      }
+    } finally {
+      if (inflightAbortRef.current === abort) {
+        inflightAbortRef.current = null;
+      }
+    }
+  }, [cfg.minUtteranceMs]);
+
   // VAD loop — runs while `live` is true.
   const tickVAD = useCallback(() => {
     if (!analyserRef.current) return;
@@ -177,26 +242,33 @@ export function useLiveConversation(opts: LiveConversationOptions): LiveConversa
         loud++;
         quiet = 0;
         if (loud === cfg.speechStartFrames && !recordingRef.current) {
-          // Barge in: kill bot audio + cancel in-flight chat + start recording.
+          // Barge in: kill bot audio + cancel in-flight chat + begin capture.
           interruptBotAudio();
           if (inflightAbortRef.current) {
             try { inflightAbortRef.current.abort(); } catch {}
             inflightAbortRef.current = null;
           }
-          startRecording();
+          beginSpeechCapture();
         }
       } else {
         quiet++;
         loud = 0;
         if (quiet === cfg.silenceEndFrames && recordingRef.current) {
-          stopRecording();
+          void endSpeechCapture();
         }
       }
 
       rafIdRef.current = requestAnimationFrame(loop);
     };
     rafIdRef.current = requestAnimationFrame(loop);
-  }, [cfg.rmsThreshold, cfg.silenceEndFrames, cfg.speechStartFrames, interruptBotAudio, startRecording, stopRecording]);
+  }, [
+    cfg.rmsThreshold,
+    cfg.silenceEndFrames,
+    cfg.speechStartFrames,
+    interruptBotAudio,
+    beginSpeechCapture,
+    endSpeechCapture,
+  ]);
 
   useEffect(() => {
     let cancelled = false;
@@ -206,7 +278,18 @@ export function useLiveConversation(opts: LiveConversationOptions): LiveConversa
         cancelAnimationFrame(rafIdRef.current);
         rafIdRef.current = null;
       }
-      stopRecording();
+      // Drop any pending capture without firing onUtterance
+      recordingRef.current = false;
+      speechBufferRef.current = [];
+      prerollRef.current = [];
+      if (workletRef.current) {
+        try { workletRef.current.disconnect(); } catch {}
+        workletRef.current = null;
+      }
+      if (workletUrlRef.current) {
+        try { URL.revokeObjectURL(workletUrlRef.current); } catch {}
+        workletUrlRef.current = null;
+      }
       if (streamRef.current) {
         streamRef.current.getTracks().forEach((t) => t.stop());
         streamRef.current = null;
@@ -242,6 +325,8 @@ export function useLiveConversation(opts: LiveConversationOptions): LiveConversa
           window.AudioContext;
         const ctx = new AudioCtx();
         audioCtxRef.current = ctx;
+        sampleRateRef.current = ctx.sampleRate;
+
         const source = ctx.createMediaStreamSource(stream);
         const analyser = ctx.createAnalyser();
         analyser.fftSize = 512;
@@ -249,6 +334,48 @@ export function useLiveConversation(opts: LiveConversationOptions): LiveConversa
         source.connect(analyser);
         sourceRef.current = source;
         analyserRef.current = analyser;
+
+        // KI-044 — register the inline PCM-capture worklet + tap the source.
+        const blob = new Blob([WORKLET_SOURCE], { type: "application/javascript" });
+        const url = URL.createObjectURL(blob);
+        workletUrlRef.current = url;
+        try {
+          await ctx.audioWorklet.addModule(url);
+          const node = new AudioWorkletNode(ctx, "pcm-capture");
+          workletRef.current = node;
+
+          const prerollSamplesCap = Math.ceil((cfg.prerollMs / 1000) * ctx.sampleRate);
+
+          node.port.onmessage = (ev: MessageEvent<Float32Array>) => {
+            const chunk = ev.data;
+            if (recordingRef.current) {
+              speechBufferRef.current.push(chunk);
+            } else {
+              prerollRef.current.push(chunk);
+              // Trim oldest chunks to keep total length under prerollSamplesCap.
+              let total = 0;
+              for (const c of prerollRef.current) total += c.length;
+              while (total > prerollSamplesCap && prerollRef.current.length > 1) {
+                total -= prerollRef.current[0].length;
+                prerollRef.current.shift();
+              }
+            }
+          };
+
+          source.connect(node);
+          // Worklet's process() only runs while the node is connected to a
+          // destination (directly or via the graph). But we DON'T want the
+          // user's mic playing back through speakers — route via a zero-gain
+          // GainNode so the graph stays "live" but output is silent.
+          const silentSink = ctx.createGain();
+          silentSink.gain.value = 0;
+          node.connect(silentSink);
+          silentSink.connect(ctx.destination);
+        } catch (e) {
+          // eslint-disable-next-line no-console
+          console.error("[live-mode] AudioWorklet setup failed; fallback to silence", e);
+        }
+
         setMicPermissionDenied(false);
         tickVAD();
       } catch (e) {
@@ -263,7 +390,7 @@ export function useLiveConversation(opts: LiveConversationOptions): LiveConversa
       cancelled = true;
       tearDown();
     };
-  }, [live, stopRecording, tickVAD]);
+  }, [live, tickVAD, cfg.prerollMs]);
 
   return {
     live,

rag/README.md

@@ -0,0 +1,66 @@
+# `rag/` — Retrieval pipeline (corpus → extraction → embeddings → Chroma)
+
+The end-to-end RAG pipeline: download insurer PDFs, parse + chunk, embed with local BGE-small, persist to Chroma, run structured extraction with a Pydantic schema, and serve top-k retrieval at query time.
+
+Lineage for every artefact below is documented in [`kb/AUDIT_TRAIL.md`](../kb/AUDIT_TRAIL.md).
+
+## Build-time scripts
+
+| File | Pipeline stage | What it produces |
+| --- | --- | --- |
+| `download_corpus.py` | 1. SOURCE → 3. DOWNLOAD | `rag/corpus/<insurer>/*.pdf` + `_manifest.json`. HEAD-checks + magic-byte sniff + retry on 403/timeout. |
+| `download_retry.py` | 3. DOWNLOAD | Retries the failures from the previous run. |
+| `download_regulatory.py` | 3. DOWNLOAD | IRDAI / regulatory PDFs (deferred from v1; see [ADR-017](../docs/60-decisions/ADR-017-irdai-corpus-playwright-rescue.md)). |
+| `ingest.py` | 4. PARSE → 5. CHUNK → 6. EMBED → 7. INDEX | The big one. `read_pdf_pages` (pdfplumber) → `chunk_pages` (800-tok / 120 overlap, sentence-aware) → BGE embed → `chromadb.PersistentClient.add(...)`. Carries the in-process HNSW bloat tripwire ([ADR-029](../docs/60-decisions/ADR-029-disk-storage-hardening.md)). |
+| `extract.py` | 8. STRUCTURED EXTRACTION | LLM extraction over each PDF using `schema.py::HealthPolicy` (62 fields). Sarvam-M primary → DeepSeek-V3 fallback. Writes `rag/extracted/<policy_id>.json` + upserts `policies.duckdb`. |
+| `build_kb.py` | 9. SCORECARD → KB MIRROR | Runs `backend/scorecard.py` per policy and regenerates the human-readable `kb/policies/<id>.md` tree. |
+| `source_map.py` | post-build | Builds `source_map.json` — every chunk → (PDF path, page, span) for citation rendering. |
+
+## Runtime modules
+
+| File | Stage | Notes |
+| --- | --- | --- |
+| `retrieve.py` | 10. RETRIEVAL | Top-k Chroma query with policy-id / insurer-slug filters. In-process LRU cache (cap 256) keyed by `(query_norm, top_k, sorted policy_ids, sorted insurer_slugs)`. |
+| `schema.py` | 8. STRUCTURED EXTRACTION | The 62-field `HealthPolicy` Pydantic schema — single source of truth for the extracted JSON shape. See `rag/SCHEMA.md` for the field-by-field doc. |
+
+## Persistent artefacts
+
+| Path | Source of truth | Notes |
+| --- | --- | --- |
+| `rag/corpus/<insurer>/*.pdf` | insurer CDNs | 208 PDFs across 19 insurers + IRDAI. Not in git — hydrated at Docker build from the companion HF dataset. |
+| `rag/extracted/<policy_id>.json` | `extract.py` | 203 JSONs, one per policy, conforming to `schema.HealthPolicy`. Generated; never hand-edit. |
+| `rag/vectors/chroma.sqlite3` + HNSW binaries | `ingest.py` | Persistent Chroma store. Symlinked to `rag/_hf_dataset_backup/rag/vectors/` for the offline canonical copy. |
+| `rag/policies.duckdb` | `extract.py` | DuckDB rollup of the 62-field JSONs; used for SQL-style filters in `backend/main.py`. |
+| `rag/source_map.json` | `source_map.py` | chunk_id → (pdf_path, page, span) for the citation links shown in the UI. |
+| `rag/SCHEMA.md` | hand-written | Field-by-field documentation of the Pydantic schema. |
+
+## Subdirectories
+
+- `corpus/` — raw PDFs (one folder per insurer slug). Generated by `download_corpus.py`. Per-PDF folders intentionally do not carry READMEs.
+- `extracted/` — 62-field JSONs. Auto-generated by `extract.py`. Do not edit by hand.
+- `vectors/` — Chroma persistent store. Treat as opaque — re-build with `python -m rag.ingest` if corrupted.
+- `_hf_dataset_backup/rag/{corpus,extracted,vectors}/` — offline canonical mirror of the companion HF Dataset (`rohitsar567/insurance-bot-data`). See [ADR-020](../docs/60-decisions/ADR-020-code-data-split-hf-dataset.md).
+
+## Cold-rebuild
+
+```bash
+python -m rag.download_corpus
+python -m rag.download_retry
+python -m rag.extract
+rm -rf rag/vectors
+python -m rag.ingest
+python -m rag.build_kb
+python -m eval.generate_gold
+python -m eval.run
+```
+
+Total cost from cold: < $2 (BGE local + ~80 LLM extractions). Wall-time: ~30-40 min on a modern laptop.
+
+## Related
+
+- [ADR-004](../docs/60-decisions/ADR-004-hybrid-structured-vector.md) — hybrid structured + vector retrieval rationale
+- [ADR-011](../docs/60-decisions/ADR-011-bge-local-embeddings.md) — why local BGE replaced Voyage
+- [ADR-018](../docs/60-decisions/ADR-018-chunk-size-sweep-deferred.md) — 800/120 chunk-size baseline
+- [ADR-020](../docs/60-decisions/ADR-020-code-data-split-hf-dataset.md) — code-vs-data repo split
+- [ADR-029](../docs/60-decisions/ADR-029-disk-storage-hardening.md) — HNSW bloat tripwire (3-layer defence)
+- [`kb/AUDIT_TRAIL.md`](../kb/AUDIT_TRAIL.md) — end-to-end lineage doc

tests/README.md

@@ -0,0 +1,39 @@
+# `tests/` — Unit + live-verification tests
+
+Deliberately small. The bulk of behavioural quality lives in `eval/` (gold-QA accuracy) and `tools/audit/` (multi-persona stress). This folder pins the **invariants** — the specific bugs we have ever shipped and never want back.
+
+## Files
+
+| File | Role |
+| --- | --- |
+| `test_routing_regression.py` | 15 `unittest` cases pinning the KI-018 / KI-023 / KI-025 fixes — see "Routing invariants" in the root `CLAUDE.md`. Includes `TestProviderLoadBalancing` which asserts the 50/50 NIM ↔ Groq split holds over 1000 seeded calls ([ADR-026](../docs/60-decisions/ADR-026-provider-load-balancing.md)). |
+| `live_verify.py` | End-to-end production drift detector. Hits the **deployed** API with a 20-Q gold subset and asserts HTTP 200, non-empty `reply_text`, ≥1 citation, faithfulness pass, and Doc-01 latency budget (p95 ≤ 7000ms). Writes `tests/live_results_<ts>.md`. Cron-able for nightly. |
+
+## What each test pins
+
+| KI / ADR | Assertion | Why it matters |
+| --- | --- | --- |
+| KI-018 (D-003) | `classify_intent("What is the waiting period for PED in Activ Assure?")` returns `"qa"` and `should_route_to_fact_find` returns `False` on empty profile. | Headline 30% gold-QA accuracy bug — direct QA was force-routed to fact-find. |
+| KI-018 | `CONTEXT_DEPENDENT_INTENTS = {"recommendation", "comparison"}` — no `"qa"`. | Adding `"qa"` re-introduces the headline bug. |
+| KI-023 | `FACT_FIND_TRIGGERS` uses word-boundary regex, not substring. | Stops `"hi"` firing on `"which"` / `"this"` / `"high"`. |
+| ADR-026 / KI-025 | `_balanced_brain_chain(..., groq_first_probability=0.5)` lands Groq-primary between 400 and 600 of 1000 seeded calls. | Catches the shared-counter pathology where every brain call lands on one provider. |
+
+## Running
+
+```bash
+# Unit (in-process, no API)
+.venv/bin/python -m unittest tests.test_routing_regression -v
+
+# Live (against deployed HF Space)
+python tests/live_verify.py
+
+# Live (against any other deploy)
+TARGET_URL=http://localhost:8000 python tests/live_verify.py
+```
+
+## Related
+
+- Root `CLAUDE.md` § Routing invariants — the four lines this folder protects
+- `audit_results/ENTERPRISE_AUDIT.md` D-003 — full incident report for KI-018
+- [ADR-026](../docs/60-decisions/ADR-026-provider-load-balancing.md) — load-balance behaviour pinned by `TestProviderLoadBalancing`
+- `eval/run.py` — the broader 96-Q accuracy eval (different surface, same underlying orchestrator)

tools/README.md

@@ -0,0 +1,80 @@
+# `tools/` — Operational scripts
+
+Loose collection of CLI scripts: corpus operations, data uploads, probes, KB regeneration, scheduled-job runners. Nothing under `tools/` is imported by the live server — `backend/` and `rag/` are the runtime surface.
+
+Scheduling for the long-running ones is wired via macOS LaunchAgents — see `CRON_README.md` in this folder for cadence + script paths, and [ADR-029](../docs/60-decisions/ADR-029-disk-storage-hardening.md) for the disk-safety LaunchAgents.
+
+## Corpus + extraction batch ops
+
+| Script | Purpose |
+| --- | --- |
+| `extract_all_corpus.py`, `extract_batch_5.py`, `extract_failed.py`, `extract_pdf_range.py`, `reextract_all.py` | Batch re-extractions over `rag/corpus/`. Useful when the schema or extraction prompt changes. |
+| `extract_pdf_text.py`, `extract_policy_text.py`, `extract_policy_text_batch2.py` | Raw text dumps for manual inspection / regex curation. |
+| `curate_batch2.py`, `curate_remaining.py`, `clear_batch2.py` | Verbatim-quote curation passes that produced `data/policy_facts/`. See [`data/policy_facts/_curation_report.md`](../data/policy_facts/_curation_report.md). |
+| `generate_policy_facts.py` | Convert extraction outputs to the `data/policy_facts/<id>.json` shape with `{value, unit, source_pdf_path, source_quote}` provenance. |
+| `pydantic_validate_batch_5.py`, `validate_batch_5.py`, `validate_json.py`, `validate_schema.py` | Schema validators for the 62-field `HealthPolicy`. |
+| `count_fields.py` | Per-policy completeness scorer that feeds the `kb/INDEX.md` completeness % column. |
+
+## Source-map + verification
+
+| Script | Purpose |
+| --- | --- |
+| `info_source_map.py` | Builds `eval/info_source_map.json` + `data/information_source_map.md` — claim → URL → verdict (✅ / ⚠️ / ❌ / ⏳). The canonical KPI for source-grounding quality. |
+| `verify_urls.py` | HEAD-checks every URL in the corpus / facts; writes `eval/verified_urls.json`. |
+| `verify_review_urls.py`, `verify_new_corpus.py` | Sub-verifiers for the reviews dataset and freshly-added corpus URLs. |
+| `browser_verify.py` | Playwright-backed verifier for URLs that block HEAD requests. Output: `tools/browser_verified.json`. |
+| `check_link_rot.py`, `check_pdf_etags.py` | LaunchAgent-driven freshness checks — corpus URL rot + PDF eTag drift. |
+| `refresh_premiums.py` | LaunchAgent-driven refresh of `data/premiums/illustrative_premiums.json`. |
+
+## KB + dataset builders
+
+| Script | Purpose |
+| --- | --- |
+| `build_kb_mirror.py` | Regenerates the entire `kb/policies/<id>.md` tree from `data/policy_facts/`. Idempotent. |
+| `ingest_kb_summaries.py` | Ingests `kb/policies/*.md` summaries into Chroma so policy meta is retrievable. Carries the HNSW bloat tripwire. |
+| `ingest_reviews.py` | Ingests `data/reviews/<insurer>.json` into Chroma. Carries the HNSW bloat tripwire. |
+| `build_readme_pdf.py` | Renders the master `README.md` to PDF for offline review. |
+
+## HF Hub uploads (data-side mirror)
+
+| Script | Target |
+| --- | --- |
+| `upload_to_hf.py` | Code-side push to the HF Space repo (`huggingface.co/spaces/rohitsar567/InsuranceBot`). |
+| `upload_corpus_to_dataset.py`, `upload_extracted_to_dataset.py`, `upload_vectors_to_dataset.py`, `upload_all_to_dataset.py` | Push specific slices of `rag/` to the companion HF Dataset `rohitsar567/insurance-bot-data`. See [ADR-020](../docs/60-decisions/ADR-020-code-data-split-hf-dataset.md) and [ADR-024](../docs/60-decisions/ADR-024-triple-mirror-code-and-data.md). |
+| `set_hf_secrets.py` | One-shot helper that pushes the runtime secrets into the HF Space (idempotent). |
+
+## Probes + diagnostics
+
+| Script | Provider it pokes |
+| --- | --- |
+| `sarvam_probe.py`, `sarvam_nothink_probe.py` | Sarvam-M / Saarika / Bulbul connectivity + latency. |
+| `groq_probe.py`, `groq_long_probe.py` | Groq Llama free-tier latency + sustained-rate test. |
+| `openrouter_probe.py`, `or_models.py` | OpenRouter routing + model-list inspection. |
+| `pdf_probe.py` | pdfplumber parse on a single PDF — first stop when extraction silently produces empty text. |
+| `heavy_smoke_test.py` | End-to-end smoke against the live HF Space (every provider in one call). |
+
+## Chunk-size & retrieval sweeps
+
+| Script | Purpose |
+| --- | --- |
+| `chunk_sweep.py`, `chunk_sweep_diagnostic.py` | Grid-search over chunk size / overlap. Output: `eval/chunk_sweep_results.json`. See [ADR-018](../docs/60-decisions/ADR-018-chunk-size-sweep-deferred.md). |
+| `sweep_retrieval.py` | Retrieval-strategy A/B (filter vs no-filter, top-k variants). |
+
+## Scheduled jobs / shell wrappers
+
+| Path | Purpose |
+| --- | --- |
+| `install_crons.sh`, `CRON_README.md` | Install the LaunchAgents; the README is the canonical cadence + path reference. |
+| `install_git_hooks.sh`, `git-hooks/` | Pre-commit hooks (decimal grep, secret scan, schema validation). |
+| `full_pipeline.sh`, `pipeline_finish_all.sh`, `post_extract_deploy.sh`, `reextract_then_deploy.sh`, `quarterly_rebuild.sh` | Multi-step orchestrations (download → extract → ingest → push → smoke). |
+| `reconcile_manifest.py` | Drift check between `rag/corpus/_manifest.json` and what's actually on disk. |
+
+## Subdirectory
+
+`audit/` — multi-persona conversational audit framework. See `tools/audit/README.md`.
+
+## Related
+
+- `CRON_README.md` (this folder) — LaunchAgent cadence reference
+- [ADR-020](../docs/60-decisions/ADR-020-code-data-split-hf-dataset.md), [ADR-024](../docs/60-decisions/ADR-024-triple-mirror-code-and-data.md), [ADR-029](../docs/60-decisions/ADR-029-disk-storage-hardening.md)
+- `audit_results/ENTERPRISE_AUDIT.md` — defect register, including silent-LaunchAgent regressions (D-002)

tools/audit/README.md

@@ -0,0 +1,56 @@
+# `tools/audit/` — Multi-persona conversational audit
+
+End-to-end conversational stress test: walks 100 distinct personas through 30-turn flows against the live API, captures every turn's reply / latency / faithfulness verdict / blocked status, and rolls up to a defect-counting report.
+
+This is the framework that surfaced the headline KI-018 (QA→fact-find misrouting) and KI-021 (latency p95 blow-out) defects in the readiness audit.
+
+## Files
+
+| File | Role |
+| --- | --- |
+| `run_audit.py` | Entry point. Walks each persona × 30 turns against the live `/api/chat`. Resumable (per-persona transcripts land as they finish). Concurrent (`--workers W`) but rate-aware — global NIM 40 req/min cap enforced via per-request sleep. Retries 5xx with exponential backoff. |
+| `personas.py` | Generator: 10 archetypes × 10 demographic profiles × 1 deterministic style = **100 unique personas**. Stable order = stable persona IDs across runs, so diffs are regressions not shuffle noise. Run as a script to (re)generate `personas.json`. |
+| `personas.json` | Materialised 100-persona list. Stable input to `run_audit.py`. |
+| `flows.py` | Generator: per persona produces a 30-turn user-text sequence in 5 phases — opening (1) · fact-find answers (9) · free-form Qs (10) · edge-case probes (5) · adversarial + close (5). |
+| `flows.json` | Materialised flows. `dict[persona_id, list[str]]` of the 30 turns each persona sends. |
+| `analyze.py` | Post-run aggregator: reads `audit_results/<run_id>/transcripts/*.json`, computes per-archetype / per-language / per-style breakdowns of faithfulness, blocked rate, p95 latency. Emits `report.md` + `summary.json` into the run dir. |
+
+## Output layout
+
+```
+audit_results/<run_id>/
+├── transcripts/
+│   ├── P001.json          (complete persona)
+│   ├── P002.json
+│   ├── P003.partial.json  (in-flight or interrupted)
+│   └── …
+├── report.md              (analyze.py output — defect breakdown)
+└── summary.json           (machine-readable rollup)
+```
+
+`<run_id>` convention is `full_YYYYMMDD_HHMMSS` for the full 100-persona pass and `postfix_YYYYMMDD_HHMMSS` for a post-fix re-run targeting a specific defect.
+
+## Typical run
+
+```bash
+# Full audit against the live HF Space
+python tools/audit/run_audit.py --workers 4
+
+# Smoke (5 personas) for a config change
+python tools/audit/run_audit.py --max-personas 5 --base http://localhost:8000
+
+# Aggregate after
+python tools/audit/analyze.py audit_results/full_20260514_145243/
+```
+
+## Watch-outs
+
+- **HF Space rebuild is 5-8 min.** Don't start an audit until the desired image is stably deployed, or transcripts span multiple builds and become useless for A/B.
+- **The 40 req/min NIM cap is global.** Bumping `--workers` past 4 will not help — the per-request sleep clamps dispatch rate.
+- **Personas are stable by index.** If you change the `ARCHETYPES` / demographic lists, P037 is no longer the same person — call it out in the run notes.
+
+## Related
+
+- `audit_results/ENTERPRISE_AUDIT.md` — defect register fed by audit output
+- `audit_results/README.md` — output-folder layout reference
+- Root `CLAUDE.md` § Routing invariants — the KI-018 / KI-023 regressions the audit catches