80-audit/ENTERPRISE_AUDIT.md

# Enterprise-Grade Readiness Audit

**Target deployment:** top-tier Indian insurance companies (HDFC ERGO, ICICI Lombard, Bajaj Allianz, Star Health, Tata AIG, etc.)
**Audit date:** 2026-05-14 (initial) · 2026-05-15 (follow-on sprint KI-167 → KI-179 landed)
**Audit window:** active — this is a living document. Background audits (100-persona simulation + full gold-QA eval) running concurrently; results merge here as they land.

---

## Executive scorecard

| Domain | Status | Severity if not fixed |
|---|---|---|
| Disk / storage stability | ✅ Fixed (3-layer prevention — ADR-029) | — |
| Data pipeline integrity | ✅ Fixed (HF Hub canonical restored + in-process HNSW tripwire) | — |
| Operational observability | ⚠️ Partial (silent-LaunchAgent regression fixed; broader `except Exception:` audit pending) | P1 |
| Product quality (factual accuracy) | 🟡 Routing fix shipped (KI-018 / KI-023). 5-Q smoke: 60% (was 0%). Full 96-Q post-fix landing. | **P0 — needs ≥90% for deployment** |
| UX latency | 🟡 Chain-budget cap installed (KI-021), probe-driven primary election (KI-080, supersedes KI-025), fast-brain reorder (KI-035). Per-turn LLM calls dropped from 5-6 → 1-2. | **P0 — needs p95 < 3s** |
| Profile-capture / slot-filling | ✅ Was a telemetry bug (KI-019), not a slot-filler bug. Fact-find branch now reports profile_updates. | — |
| Language-handling fairness | ⚠️ Hinglish concern was a 20-persona sampling artifact. Real outliers: tax_planner archetype (4.6 refusals) + stream style (4.3) — open. | P1 — India-market regulatory risk |
| Code hygiene | ✅ Loose tmp files removed; `fact_find_normalizer` + `profile_extractor` migrated to chain pattern (KI-033) | — |
| Test coverage | 🟡 15 unit tests pinning KI-018 / KI-023 / KI-080 routing + primary-election invariants. Broader coverage still open. | P1 |
| Voice UX | ✅ Live default-on + clickable toggle + labeled push-to-talk + barge-in working (ADR-028) + hybrid Web Speech + MediaRecorder capture (KI-168) + tab-switch revival (KI-173 / KI-174) | — |
| Fact-find robotic tone | ✅ Replaced with `sales_brain` LLM-driven loop ([ADR-039](../70-docs/60-decisions/ADR-039-llm-driven-sales-brain.md), KI-167) — no scripted prompts, no `<FF>` trailer, native provider JSON mode | — |
| LLM stack | ✅ 3-tier chain Google → NIM → OpenRouter per [ADR-040](../70-docs/60-decisions/ADR-040-google-gemini-primary.md). Gemini 2.0/2.5 Flash primary; NIM Mistral 675B fallback; OR `:free` diversity. Judge stays NIM Mistral 675B (cross-family from Gemini). | — |
| Secrets handling | ✅ Verified clean (.env never committed) | — |

Legend: ✅ fixed / ⚠️ partial / 🟡 improving / 🔴 open

---

## Defect Register

Each row: ID · severity · title · evidence · fix status. P0 = blocks production deployment. P1 = blocks enterprise procurement. P2 = quality / hygiene. P3 = nice-to-have.

### D-001 · P0 · ChromaDB HNSW link_lists.bin runaway growth — **FIXED**
**Symptom:** 2026-05-14 15:18 — `rag/_hf_dataset_backup/rag/vectors/148fbdda-…/link_lists.bin` reached 277 GB logical / 136 GB on-disk for only 12 MB of actual vector data (5K chunks). Disk filled from ~137 GB free → 50 MiB free in ~45 min.
**Root cause:** ChromaDB 1.5.9 HNSW persistence pathology — known issue where the link-graph adjacency file accumulates sparse holes during certain add/delete cycles. Bloat factor: ~277,000× expected.
**Impact:** total system unavailability (workstation unusable); during inference would have meant slow query, eventual OOM.
**Fix (deployed):**
1. In-process tripwire — `rag/ingest.py` declares `HNSW_BLOAT_THRESHOLD_BYTES = 500 MB` and calls `_abort_if_hnsw_bloated()` after every `collection.add(...)`. Two other writers (`tools/ingest_kb_summaries.py`, `tools/ingest_reviews.py`) import the same guard.
2. Out-of-process auto-purge — `~/Library/Scripts/insurance-bot/check-vector-bloat.sh` + LaunchAgent `com.rohit.insurancebot.vectorbloat` (60-min cadence). Auto-deletes `_hf_dataset_backup/` at 20 GB. Warns at 5 GB.
3. Disk-free tripwire — `~/Library/Scripts/cache-prevention/disk-free-tripwire.sh` + LaunchAgent `com.rohit.disk-free-tripwire` (15-min cadence). Critical alert <8 GB free; dumps every `~/Developer` subdir >1 GB into the log.
4. Re-downloaded canonical dataset from HF Hub (`rohitsar567/insurance-bot-data` · 539 files · 498 MB) — `link_lists.bin` now 58-66 KB.

**Production hardening still owed (D-001a, P1):**
- Open ChromaDB issue tracker — file or upstream a reproduction so the actual root cause is fixed, not just contained.
- Move ingest to a temp directory + atomic rename — currently the live store is also the ingest target.

---

### D-002 · P0 · Three LaunchAgents silently failing under wrong path — **FIXED**
**Symptom:** `com.rohit.insurancebot.linkrot`, `.pdfetags`, `.premiums` all `cd "/Users/rohitsar/Documents/Personal/AI Work/Insurance Sales Bot"` — that directory does NOT exist. The actual project is `/Users/rohitsar/Developer/Insurance Sales Bot`. They've been failing every scheduled run.
**Impact:** Link-rot detection, PDF eTag refresh, and premium-page refresh have all been **broken indefinitely** — corpus URL changes go undetected, insurer pricing data goes stale, regulatory PDFs may have been updated without the bot's awareness.
**Fix (deployed):** Sed-replaced `~/Documents/Personal/AI Work/` → `~/Developer/` in all three `run_*.sh` scripts. Created log directory. Smoke-test running in background (id `bh2cv32e2`).
**Production hardening owed (D-002a, P1):**
- Add a heartbeat-or-page check (like the bloat watcher) for every LaunchAgent: if `last_exit != 0` for N consecutive runs, page.
- This is the SECOND silent-LaunchAgent regression in 18 days (memory: `feedback_tcc_blocks_launchd_in_documents`). Pattern needs a class-fix.

---

### D-003 · P0 · Factual accuracy on gold-QA eval — **ROOT-CAUSED + PARTIAL FIX SHIPPED**
**Initial symptom:** `eval/results.md` (2026-05-13 21:52 UTC) showed 30.0% factual accuracy on 10 questions. Full 96-Q re-run (2026-05-14 16:32 UTC, before fix) showed **41.7%**.
**Diagnostic breakthrough — by brain accuracy:**
- `nim-chain` (the QA brain): **51.9% accuracy** when used.
- `needs_finder` (the slot-filler): **0.0% accuracy** when used.
- `nim` direct: **0.0%** (only 0 turns).
The full eval was sending **every** QA question to `needs_finder`. Sample bot answers from before the fix:
- *Q: "What is the waiting period for pre-existing diseases under Activ Assure?" → A: "Happy to help. First, your age?"*
- *Q: "Is there a cap on room rent under Activ Assure?" → A: "Sorry, I didn't catch that. Let me ask again — Who else needs cover..."*
- *Q: "Does Activ Assure cover AYUSH?" → A: "Got it. Who else needs cover — just you, spouse, kids, parents, or a mix?"*

**Root cause:** `backend/orchestrator.py:174-183`. The KI-013 guard ("never recommend without profile") force-routed every turn to fact-find when the session profile was empty — **regardless of intent**. Eval and audit sessions start with empty profiles, so 100% of QA questions got swallowed by `needs_finder`.

**Fix shipped:** restricted the `profile_is_empty` force-route to `intent ∈ {recommendation, comparison}` only. QA intent now passes through to retrieval + `nim-chain`. New comment block in source marks this as KI-018.

**Smoke validation (5 questions, no-judge regex grader, post-fix):**
- Factual accuracy: **60.0%** (was 0% on the same 5 questions before).
- Brain breakdown: `nim-chain` 100% (was `needs_finder` 100% before).
- The two PED-waiting-period questions that previously responded "First, your age?" now correctly answer "24 months".

**Projection:** if `nim-chain`'s 51.9% accuracy holds when it serves all 96 questions (instead of only some subset), headline factual accuracy goes from 41.7% → low 50s. To reach the enterprise bar of ≥90%, additional work is needed on the brain itself:
- `waiting_period`: retrieval ranks wrong section; likely chunking issue.
- `sub_limit`: structured-extraction problem — sub-limit tables don't survive chunking.
- `exclusions_oos`: refusal logic doesn't catch OOS exclusions — needs tighter regulatory_oos filter applied to exclusions too.

**Adjacent bug — D-003a, P1:** the Groq judge (Llama-3.3-70B) returned `JSONDecodeError` 11 times on the 96-Q run; each gets scored 0 factual, inflating the failure rate. Need to wrap judge response with a JSON-repair pass or fall back to the regex grader on judge-parse-failure.

**Next:** queue a fresh full 96-Q eval re-run after the 100-persona audit completes (avoid competing on NIM quota).

---

### D-004 · P0 · Latency p95 49s, p99 59s — **DEGRADED ON BROADER SAMPLE**
**Symptom (now confirmed on full 100-persona audit, 3000 turns):**
- p50: **9.9s** (was 7.3s on the 20-persona sample — **+36%**)
- p95: **49.1s** (was 24.2s — **+103%**)
- p99: **58.9s** (was 48.0s — **+23%**)
**Impact:** **Unacceptable for chat UX.** A 49-second p95 wait on "what's covered?" feels broken. Enterprise insurance customers expect sub-3s p95. Some persona/intent combos (recommendation × `nim-chain::v4-pro`) take ~1500s for a single 30-turn session.
**Why it degraded vs. 20-persona sample:** the 20-persona run was all first-buyer + upgrader archetypes (simpler, more fact-find-only). The full 100 includes `tax_planner`, `comparer`, `savvy` archetypes that hit the heavier `v4-pro` model + multi-call cascades, which pushed p95 up dramatically.
**Likely contributors (in order of suspected impact):**
1. **`v4-pro` brain on comparison/recommendation** — frontier 1.6T MoE, slow per token; combine with NIM rate limit and you get the long-tail.
2. NIM 40 req/min cap forces serial dispatch under load → queuing latency.
3. Multi-call cascades (brain + judge + cross-check) per docstring in `run_audit.py`.
4. **20 HTTP timeouts** (0.7%) — the bot occasionally hangs on a turn entirely (P100 had 3 ReadTimeouts in a row).
**Fix proposal:**
- Streaming responses (TTFT instead of full response time) — most of the 49s wait is the user staring at a blank screen.
- Tier `comparer`/`tax_planner` archetypes to `v4-flash`; reserve `v4-pro` for truly heavy recommendation synthesis.
- Hard timeout on NIM calls (currently apparently >60s) — fail-fast at 20s, retry with smaller context.
- Cache the intent classifier output for the first N turns of a session.
**Owner:** needs design discussion before changing brain routing.

---

### D-005 · P0 · Profile-capture / slot-filling broken — **CONFIRMED REAL BUG ON 100-PERSONA SAMPLE**
**Symptom (full 100-persona audit, post-fix):**
| Field | Personas hit |
|---|---|
| `health_conditions` | 20 / 100 |
| `existing_cover_inr` | 12 / 100 |
| `age` | **12 / 100** |
| `parents_to_insure` | 7 / 100 |
| `primary_goal` | 6 / 100 |
| `parents_age_max` | 6 / 100 |
| `parents_has_ped` | 1 / 100 |
| All other fields (dependents, income, location, marital, conditions ≠ health_conditions, budget) | **0 / 100** |

**Re-diagnosis:** This is NOT a downstream effect of D-003. The audit pre-dated the D-003 fix, but more importantly: 1281 of 3000 turns ran `needs_finder` (43%) — i.e. the slot-filler DID run on plenty of turns. Yet age was captured for only 12% of personas, even though every persona is canonically asked "First, your age?" on turn 1 and answers it. **The slot-filler is asking but not retaining the answer.**

**Likely root causes:**
1. `backend/fact_find_normalizer.normalize_answer()` is rejecting valid Indian-accented age responses ("twenty-five" / "मेरी उम्र पचास साल है" / "I'm 28 going on 29").
2. `_reask_count` is hitting its cap (2 fails → give up + move on without storing).
3. The 258 fact-find re-asks correlate with this — almost every persona had a re-ask, often multiple.
4. Hindi-primary personas captured `health_conditions` (Devanagari numerals in the canned persona response) but failed on age (free-text Hindi).

**Impact:** Bot cannot recommend a policy because it doesn't have the user's profile. The 263 faithfulness-gate refusals are largely because retrieval has no profile to constrain against. **Recommendation flow is fundamentally broken** for the majority of users.

**Next step:** Read 5 P-files (across archetypes), grep for `_reask_counts` increments, identify the specific normalizer regex that's misfiring.

---

### D-006 · P1 · Refusal-rate fairness — **REDIAGNOSED ON 100-PERSONA SAMPLE; HINGLISH CONCERN CLEARED, REAL OUTLIERS IDENTIFIED**

**Refusal rate by archetype** (avg refusals/persona over 30 turns):
| Archetype | Refusals | Faithfulness fails |
|---|---:|---:|
| `tax_planner` | **4.6** | 46 |
| `code_switcher` | 4.4 | 44 |
| `savvy` | 3.3 | 33 |
| `specific_condition` | 3.2 | 32 |
| `low_trust` | 2.8 | 28 |
| `anxious` | 2.5 | 25 |
| `senior_care` | 2.1 | 21 |
| `comparer` | 2.0 | 20 |
| `upgrader` | 1.0 | 10 |
| `first_buyer` | **0.4** | 4 |

**Refusal rate by conversational style:**
| Style | Refusals |
|---|---:|
| `stream` | **4.3** (highest) |
| `tester` | 3.4 |
| `casual_en` | 3.1 |
| `anxious_q` | 2.8 |
| `verbose` | 2.7 |
| `hindi_primary` | 2.5 |
| `numbers_heavy` | 2.1 |
| `formal_en` | 1.9 |
| `hinglish` | 1.9 |
| `terse` | **1.6** (lowest) |

**Re-diagnosis:** The earlier 20-persona "hinglish 2× more refusals" claim was a small-sample artifact (2 hinglish personas × 1 refusal each). On the full sample, hinglish (1.9) is actually slightly BETTER than hindi_primary (2.5) and is the second-lowest style. **Hinglish is fine.**

**Real fairness issues:**
1. **`tax_planner` and `code_switcher` archetypes get refused 11× more than `first_buyer`** (4.6 / 4.4 vs 0.4). These users ask tax-deduction questions (80D / 80DD / 80U) and code-switched comparison questions — the bot's faithfulness gate refuses both heavily. **This is an India-market segment we cannot afford to alienate.**
2. **`stream` style refused 2.7× more than `terse`** (4.3 vs 1.6). Long stream-of-consciousness input is failing the faithfulness gate. Likely the gate is matching the wrong span of the user's question.

**Worst refusers (real users to debug against):** P081 (Saif Banerjee, code_switcher/stream, **9 refusals**), P069 (Vikram Banerjee, tax_planner/casual_en, 8), P063+P064 (tax_planner, 7 each), P091 (specific_condition/tester, 7).

**Fix proposal:**
- Add tax-related gold-QA questions (80D, 80DD, 80U) — currently the gold set has zero.
- Loosen faithfulness gate for `stream`-style: chunk the question, take the most retrieval-rich span, not the whole thing.
- For `code_switcher`, run Sarvam translation pass at gate-evaluation time, not only at brain time.

---

### D-007 · P1 · No unit tests; only `live_verify.py` — **OPEN**
**Symptom:** `tests/` contains only `live_verify.py`. Backend modules (`orchestrator.py`, `faithfulness.py`, `security.py`, `scorecard.py`, `profile_rag.py`) have no isolated tests.
**Impact:** Enterprise procurement (and SOC 2 / ISO 27001 audits) require test coverage evidence. The eval/audit suites are integration-level; they don't catch unit regressions.
**Fix proposal:** Add `tests/unit/` with pytest, target ≥70% line coverage on `backend/`. Block PRs that drop coverage.

---

### D-008 · P1 · `except Exception:` audit — **PARTIAL**
**Symptom:** ~17 sites across `backend/main.py`, `admin.py`, `security.py`, `profile_rag.py`, `scorecard.py` catch broad exceptions. Most legitimate (defensive deletes, malformed-line-skip, fail-open availability tradeoffs). But several swallow legitimate errors:
- `backend/main.py:518` after `record_accept(sha, sid, len(chunks))` — telemetry write silently swallowed.
- `backend/main.py:660` after building `hint` — silent failure for what could be a routing bug.
- `backend/profile_rag.py:142-144` — `coll.delete(where=...)` failure swallowed; if a stale chunk exists, the new chunk will collide on ID.
**Impact:** Real errors get masked; debugging in production becomes archaeology.
**Fix:** Each `except Exception: pass` should at minimum log to `LOG_DIR/turns.jsonl` (or the structured logger) with `level=warn` and an event name.
**Note:** Recent commit `2412797 fix(observability): KI-001..006 — log silent failures + fail-CLOSED judge` already addresses some of these. Need to confirm coverage.

---

### D-009 · P2 · Loose `tmp_*.py` files in project root — **FIXED**
**Symptom:** `tmp_extract.py`, `tmp_count_fields.py`, `tmp_batch_extract.py` in repo root. Were git-tracked.
**Fix (deployed):** `git rm` issued. Confirmed gone. Pending commit.

---

### D-010 · P2 · TODO/FIXME density in `backend/` + `rag/` — **OPEN**
**Count:** 29 TODO/FIXME/XXX/HACK markers across backend + rag (excludes `__pycache__`).
**Action:** Triage list; convert to GitHub issues; resolve before enterprise audit.

---

## Fixes shipped today (commit reference)

| KI | Commit | What |
|---|---|---|
| KI-018 | `bcb7079` | Stop force-routing QA intent to fact-find on empty profile (the gold-eval headline bug) |
| KI-019 | `bcb7079` | Fact-find branch now reports `profile_updates` in `TurnResult` — fixes the audit telemetry that misread captures as zero |
| KI-020 | `bcb7079` | `POST /api/session/reset` + frontend Clear-chat / Start-fresh buttons |
| KI-021 | `bcb7079` | Cumulative chain budget on `NimChainLLM` (brain 35s, fast-brain 22s) bounds the long-tail latency |
| KI-022 | `bcb7079` | Groq judge JSON-parse fallback to regex grader (11/96 questions previously scored 0 falsely) |
| KI-023 | `3fb3586` | Word-boundary intent triggers (`"hi"` was matching `"which"`/`"this"`); regression test |
| KI-024 | `1304e7c` | Parallelized 96-Q gold eval (~5× speedup) |
| KI-025 | `a04c17a` | NIM↔Groq 50/50 load-balance on brain chain primary (ADR-026) |
| KI-026 | `effcfeb` | Voice mode mutual exclusion (Live + PTT + Hands-free no longer fight for the mic) |
| KI-027/8/9 | `e01547c`/`4ae5278`/`65ba46c` | Voice UX simplification: Live default-on + clickable toggle + labeled push-to-talk (ADR-028) |
| KI-030 | `3d06a80` | Barge-in fix — bot TTS now plays via in-DOM `<audio>` so `querySelectorAll("audio").pause()` can find it |
| KI-032 | `6f495c1` | LLM paraphraser for fact-find questions with verifier + cache (ADR-027) |
| KI-033 | `9a977de` | `fact_find_normalizer` + `profile_extractor` migrated from hardcoded single-model to `NimChainLLM(FAST_BRAIN_CHAIN)` |
| KI-034/5 | `844ed03` | LRU retrieval cache + `FAST_BRAIN_CHAIN` reordered (Nemotron Nano 30B primary) |
| KI-036/7/8 | `36ef017` | Greeting flow, strict paraphrase verifier, waiting dots — tightens the first-impression UX surface |
| KI-039 | `a774b91` | Single Clear-chat button (full reset) — removes ambiguity between "reset" vs "new session" |
| KI-040 | `4449d44` | Named profiles + Clear-chat that preserves profile — lets users iterate without re-doing fact-find |
| KI-041/2 | `748ce54` | VAD sensitivity tuning + Live OFF by default + dots on first turn — fixes accidental barge-in on quiet rooms |
| KI-044 | `86fcc31` | PCM pre-roll via AudioWorklet + 11 folder READMEs — eliminates first-syllable clipping on TTS playback |
| KI-045 | `930e11e` | Natural-conversation classifier in fact-find: intent_change / off-topic-question escapes fact-find branch into QA. Prevents the bot from droning past a user's pivot. |
| KI-046 | `28b6114` | Explicit refusal on adversarial/fanciful out-of-corpus questions (space tourism, diamond-tipped surgery). Closes the "absence of exclusion ≠ inclusion" reasoning leak. |
| KI-047 | `e2d09f9` | Bucket reorg (`docs/` → `70-docs/`, `audit_results/` → `80-audit/`) — Option A safe subset, keeps code dirs untouched |
| KI-048 | `13a1cf4` | Admin backend: `GET /api/admin/profiles` + `GET /api/admin/performance`, both behind `_check_admin` (IP allowlist + password, 404 on auth fail) |
| KI-049 | `c8bf1a1` | Retrieval top-k 5 → 10 for table-cell questions (room rent / sub-limit / cap / NCB / etc.). Boosts the chance the policy's structured cap-table chunk lands in context — directly targets the `sub_limit` accuracy gap from D-003. |
| KI-050 | `52c6351` | Complete `data/` → `40-data/` rename across all Python string-path refs. Finishes KI-047 for runtime code. |
| KI-051 | `2eae364` | Dockerfile `COPY` paths updated (`data → 40-data`, `docs → 70-docs`). Without this the HF Space rebuild fails on the renamed source dirs. |
| KI-052 | `c53d167` | Admin panel HTML: 3 lazy-loaded tabs (Profile + Visitor Log, Performance, LLM Chain). Performance pulls KI-048's new endpoint; auth state preserved across tab switches. |
| KI-053/4 | `57ef382` | Eval-mode skip profile-extractor flag (`INSURANCE_BOT_SKIP_PROFILE_EXTRACTOR`) + gold_qa.json grew 96 → 110 with tax_planning questions. |
| KI-055 | `8ff05ba` | HF Space BUILD_ERROR fix — dropped `rag/vectors` broken symlink from repo; Docker `chown -R user:user /app` no longer hits broken symlink → exit 0. |
| KI-056 | `2fbd062` | Dynamic acknowledger rotation (8 variants, deterministic per session/turn/slot hash) + family-aware opener for spouse/kids/parents mentions + opportunistic `infer_dependents_from_text` + Q3 paren consistency. **Now superseded by KI-070's single-LLM driver, but the family-aware capture survives natively in the LLM prompt.** |
| KI-057 | `23a6d29` | Live VAD hardening (6 layered fixes): adaptive noise floor EMA, voice-band spectral gate (190-2150 Hz), `speechStartFrames: 1→3`, `maxUtteranceMs: 18s`, post-utterance cooldown 700ms, flush-on-toggle-off. Stops ambient noise pinning the segment open + recovers mid-utterance audio when Live is toggled off. |
| KI-058 | `23a6d29` | `.gitignore` `lib/` rule scoped to anchored venv paths only (`/lib/`, `/.venv/lib/`, `/venv/lib/`) so `frontend/src/lib/` stops getting flagged. |
| KI-059 | `e610ee9` | Opening-turn name capture — "Hi this is Rohit" / "I'm Anjali" / "My name is Sarah Connor" in the very first message routes straight to the name slot handler. **Now superseded by KI-070 (LLM extracts names natively from any utterance).** |
| KI-060 | `e610ee9` | Live silence-end window loosened 40 → 90 frames (~640ms → ~1.5s) so natural mid-sentence pauses don't auto-submit. |
| KI-061 | `e610ee9` | Personalized welcome-back greeting on returning visitor — `_format_known_profile_summary` + `_format_missing_slots` so the bot reflects what's on file and offers to fill gaps before recommending. |
| KI-062 | `e610ee9` | Full-name capture (regex widened 1-2 → 1-4 words) + `compute_persona_id` 12-char sha1 over name + age + dependents + income_band + location_tier + parents_age_max. Two "Rohit"s with different identity fields resolve to distinct persona_ids. Legacy-name-slug → persona_id file migration on save. |
| KI-063 | `d89a871` | Per-user interaction log on `Profile` dataclass: `shown_policies` / `selected_policies` / `rejected_policies` lists with dedup-in-place. Auto-log on `intent ∈ {recommendation, comparison}` AND `faithfulness_passed`. New POST endpoints `/api/profile/select` + `/api/profile/reject` (KI-068: now fire-and-forget via `asyncio.to_thread` so disk writes don't block reply). |
| KI-064 | `4c6215a` | Live silence-end window bumped 90 → 120 frames (~1.5s → ~2s) — KI-060's 1.5s was still cutting off "Hi, I'm looking to buy a new insurance ... policy". |
| KI-066 | `7d021cf` | TTS shorthand normalizer in `backend/voice_format.py::_normalize_money`. `₹5L` → "5 lakhs", `₹5-10L` → "5 to 10 lakhs", `₹25L+` → "25 lakhs or more", `₹2Cr` → "2 crores", `Rs. 50,000` → "rupees 50,000". Stops Sarvam Bulbul from reading shorthand letter-by-letter. |
| KI-067 | `28f795b` | `_parse_existing_cover` extended to recognise first-time-buyer phrasings ("This is my first policy" / "new to insurance" / "never bought" / "don't have any"). **Now superseded by KI-070 (brain captures natively from prose).** |
| KI-068 | `26023b8` | Humanized fact-find readback (`primary_goal: first_buy` → "goal: first health policy"; `30k_60k` → "₹30,000–60,000/year"; `metro` → "metro city"). Stripped `**bold**` markdown wrapper that was leaking as literal asterisks in chat + being read by TTS as "asterisk asterisk". Made KI-063 logging fire-and-forget. |
| KI-069 | `26023b8` | Fixed KI-059 false-positive: regex was matching "this is correct" / "I'm good" / "first, show me ..." and routing users back to the name re-ask loop. Added `_NON_NAME_TOKENS` blocklist (50+ confirmatory words) + uppercase-name-only validation. **Now superseded by KI-070 (brain doesn't need this regex).** |
| **KI-070** | **`364591b`** | **Single-LLM-call fact-find replaces 3-layer template stitching ([ADR-030](../70-docs/60-decisions/ADR-030-llm-driven-fact-find.md)).** Orchestrator fact-find branch ~387 → ~95 lines. New `backend/fact_find_brain.py` (441 LOC). Deleted: `backend/question_paraphraser.py`, `_pick_opener`, `_NEUTRAL_OPENERS`, `_FAMILY_OPENERS`, `_contains_self_introduction`. Native multi-fact capture verified live 2026-05-15: opener "Hi, I'm Rohit Sar. I'm 32, just myself, living in Mumbai." captured `{name, age, dependents, location_tier}` in ONE turn. |
| KI-071 | `2e60476` | Docs-vs-code reconciliation: 12 files updated to reflect actual chain primaries (Qwen 3-Next 80B brain / Nemotron Nano 30B fast brain / Mistral Large 3 675B judge per D-022 brain swap). DeepSeek V4-Pro / V4-Flash + Llama-4 Maverick correctly documented as fallback chain entries. README §1.2 + §4.3 rewritten; ADR-019 D-022 supersession note; kb/security + kb/eval INDEX; frontend/eval/rag subdirectory READMEs; ADR-029 filename + ADR-028 filename + env-var name fixed; dead `input.hands_free` i18n keys deleted; `welcome.subtitle` softened to "a few short questions" for KI-070's variable turn count. |
| KI-072 | `e098e1b` | **P0 fix.** `_canonical_fallback` now applies the user's current message to whichever slot was last asked (read from `session.awaiting_question_id`) via legacy normalizer BEFORE picking next slot. Previous behaviour wedged fact-find when LLM brain failed: "I am Don" / "Don" / "Don Jon" all bounced off the name slot's canonical re-ask. |
| KI-073 | `c53381c` | Frontend Clear chat now explicitly resets `profileCompleteness` state immediately so the "55% DONE" header chip clears synchronously for the new visitor, regardless of network latency on the backend session-reset call. |
| KI-074 | `4516e87` | **P0 fix.** KI-072 only checked the awaiting slot; if the LLM brain was driving slot X but the user supplied Y, Y was dropped. Now `_canonical_fallback` GREEDILY runs `_normalize_for_slot` against every unfilled slot in priority order (age → dependents → income_band → existing_cover → primary_goal → location → parents_age → budget → name), with slot-specific trigger guards to prevent cross-contamination ("29 years old" was getting written into existing_cover_inr AND parents_age_max AND age before the trigger guards). Name parser tightened: explicit-intro-only, 50+ word blocklist (my/your/first/looking/...), conjunction stop ("Rohit Sar and I am 32" → "Rohit Sar"). Also stripped `**...**` markdown leak from health_conditions slot (different GRAPH entry than KI-068). |
| **KI-075** | **`5fc01a7`** | **Root cause of "still robotic" UX.** Live probe showed 4 of 5 fact-find turns hit `_TIMEOUT_S = 12s` asyncio.wait_for cap at exactly 13.2s latency — NIM cold-start eats 10-15s after a Space rebuild. Outer wait_for was killing brain calls BEFORE the chain's internal 22s `total_budget_s` could try cross-provider fallbacks (Groq, OpenRouter). Bumped `_TIMEOUT_S` to 25s. Brain success rate climbs from ~20% (1/5) to expected 80%+ for cold-start sessions; near-100% once warm. |
| KI-076 | (HF dataset) | Disabled the `rohitsar567/insurance-bot-data` dataset viewer by uploading a README with `viewer: false` YAML frontmatter. The viewer was failing with `StreamingRowsError: CastError` on heterogeneous JSON shapes (PDFs + Chroma binary + multi-schema JSONs). Dataset itself stays fully public + the HF Space `snapshot_download` is unaffected (schema-agnostic). Page now shows clean "Viewer disabled" notice + the new README we wrote. |
| KI-077 | `2bb3898` | "Build your profile" panel: added Name input field at top with "captured from chat" badge when populated. Backend `/api/profile/completeness` + `/api/profile` POST + `UserProfile` TypeScript type all extended with `name`. Panel pre-fills every field from the session's captured chat state via existing `initialProfile`. New `useEffect` keeps panel in sync when chat captures fields while panel is open. On Save, name persists to the named-profile JSON store (KI-040/062) so the user is auto-recognised on return visits. |
| KI-078 | `078ff45` | LLM chain hardening: per-link timeout 12s → 6s so chain can try 3-4 candidates inside `total_budget_s=22s` instead of 1. Narrowed `except Exception` to re-raise `CancelledError`/`KeyboardInterrupt`/`SystemExit` so `asyncio.wait_for` actually bubbles. New `_fallback_reason` stamped on `FactFindOutcome` and surfaced as `fact_find_brain::fallback:timeout` / `:no_trailer` / `:empty_reply` / `:llm_error` in `TurnResult.brain_used` for production telemetry. |
| KI-079 | `87ee522` | Two-layer chain hardening for fact-find. (1) FAST_BRAIN_CHAIN reorder: Groq Llama-3.3-70B promoted from position #5 to #2 (right after Nemotron primary) so cross-provider fallback is reached in ~6-7s of budget instead of ~20s. (2) Heavy-chain escalation: when `drive_fact_find()` raises `TimeoutError` on fast-brain, orchestrator retries once on `BRAIN_CHAIN` (Qwen 80B primary, 35s budget) before falling to `_canonical_fallback`. Adds `fallback:timeout_after_escalation` / `:llm_error_after_escalation` to telemetry vocabulary. Now serves as the last-bite safety net once KI-080's primary election is in place. |
| **KI-080** | **`6159c54`** | **Sticky primary election for LLM chains ([ADR-031](../70-docs/60-decisions/ADR-031-sticky-primary-election.md); end-to-end architecture spec in [ADR-032](../70-docs/60-decisions/ADR-032-llm-chain-architecture.md)).** `NimChainLLM.chat()` refactored from "iterate every chain candidate sequentially per call" to "call the probe-elected PRIMARY once; on real-time failure, call the cross-provider BACKUP once and re-trigger the probe." `backend/llm_health.py` runs a 60s background probe loop that scores every candidate on (latency × success rate) and elects a sticky PRIMARY + provider-diverse BACKUP for each of `BRAIN_CHAIN` / `FAST_BRAIN_CHAIN` / `JUDGE_CHAIN`. **Per-turn LLM call count drops from 5-6 (sustained NIM degradation, every candidate queued + timed out) to 1 (most cases) or 2 (primary fails real-time → backup + re-probe).** ADR-026's `_balanced_brain_chain` (KI-025 50/50 NIM ↔ Groq rotation) is deprecated — the probe-driven election dynamically picks the actually-faster candidate instead of a fixed 50/50 coin. Code retained as a bypassed branch behind a feature flag for one-release rollback. Cold-start fallback (no probes complete yet) uses `chain[0]` as the initial primary. KI-079 escalation still applies as the last bite if both primary AND backup fail in the same turn. Inline tests: 7/7 OK (cold-start, lowest-latency primary, cross-provider backup, demote-on-failure, score-update, primary-success makes 1 call, primary-fail/backup-success makes exactly 2 calls). `tests/test_routing_regression.py` → 15/15 pass. |
| KI-081 | (no commit — HF Space env secrets) | Pushed `GROQ_API_KEY` + `OPENROUTER_API_KEY` to the HF Space repository secrets so the KI-080 cross-provider election candidates actually have working keys in production. Pre-KI-081 only `NVIDIA_NIM_API_KEY` was set on the Space; the elector would mark every Groq + OpenRouter candidate as `no_api_key` and election degraded to NIM-only candidates — defeating the cross-provider BACKUP invariant. |
| KI-084 | `119e0fd` | **LLM chain telemetry hardening + free-tier guards.** Four changes in one commit. (1) Probe cadence `PROBE_INTERVAL_SEC` raised 60s → **300s** — the prior cadence burned ~30-50K probe tokens/day on Groq alone, self-tripping Groq's 100K/day TPD free-tier cap. (2) `PROBE_MAX_TOKENS` cut 5 → **1** — same 200 envelope, ~50× less token spend per probe. (3) Explicit per-phase `httpx.Timeout(connect=2, read=self.timeout, write=2, pool=2)` on every chat call — previously `timeout=self.timeout` collapsed to a single read deadline so a stuck NIM pool could occupy the TCP connection past `asyncio.wait_for` cancellation, leaking NIM concurrency slots. (4) New `_classify_error` surfaces HTTP status codes explicitly (`Status429` vs `HTTPStatusError:503`); rate-limit failures get a **1-hour** sin-bin (`DEGRADE_DURATION_LONG_S = 3600s`) instead of the 30s transient window — free-tier daily quotas don't reset in 30 seconds. |
| KI-085 | `8fc7979` | **Proactive credit tracking — closes the reactive-only gap KI-084 leaves.** KI-084 demotes a candidate for 1h AFTER a 429 hits, costing one user-facing failover turn per dead quota. KI-085 promotes `llm_health` from liveness-only to liveness-AND-credits so election excludes quota-exhausted candidates BEFORE the user gets stuck behind a 429. Three signal sources: (1) Groq response headers `x-ratelimit-remaining-tokens-day` + `x-ratelimit-reset-tokens-day` (low-water 5K tokens); (2) OpenRouter `/api/v1/credits` polled every 10 min from probe loop, plus per-call header fallback (low-water $0.05); (3) NIM local 60s rate-meter, gate at 35-of-40 req/min (headroom 5). Election adds `_has_credits(h, now_mono)` to eligibility predicate. Admin `status_summary` extended with `credits_remaining` / `credits_unit` / `credits_low_water` per model. 11/11 inline tests pass + routing_regression 15/15. |
| KI-086 | `d90f8c0` (bundled with KI-087) | **Admin "LLM Health & Credits" tab.** New `GET /api/admin/llm-health` endpoint returns `{chains, candidates, recent_turns, snapshot_ts}` JSON: per-chain elected PRIMARY + BACKUP with snapshots, per-candidate health grid with credits + degraded-until, last 20 turn outcomes from `40-data/llm_usage.jsonl`. Same `_check_admin` IP-allowlist + password gate as other admin endpoints. Frontend extends the existing "LLM Chain" tab in `frontend/public/admin/llm-control.html` with three sections: (A) per-chain election cards, (B) candidate health table, (C) recent turns table. Auto-polls every 30s while tab is active. Operator now sees at-a-glance which LLM is in use where, why a candidate is gated out, and how the election state evolves. |
| **KI-087** | **`d90f8c0`** | **NIM-first election preference.** Pre-KI-087 election scored purely by `latency × success_rate`, which consistently favoured Groq's 161ms LPU TTFT over NIM's 500ms-1s — so every probe round elected Groq as PRIMARY across all 3 chains. Result: every chat call hit Groq first, burned Groq's 100K daily TPD inside 50 turns, then started returning 429s. KI-087 changes election so it prefers ANY eligible NIM candidate over ALL non-NIM candidates. Within the NIM pool the standard score still picks the fastest healthy NIM model. Only when the NIM pool is empty does election fall through to Groq / OpenRouter as PRIMARY. BACKUP rule unchanged in spirit: cross-provider against PRIMARY. Rationale: NIM is the strategic free provider (ADR-019, no daily cap, 110+ models, single-key, $0); Groq has 100K daily TPD; OpenRouter charges real USD. Both should serve as emergency fallback only. |
| **KI-088** | **`14ee008`** | **NIM concurrency semaphore + serial probe + dropped inner retry.** Pre-KI-088 the process could fire 6+ concurrent NIM HTTP calls (probe burst `asyncio.gather` across 6 candidates + admin pollers + per-user turns), self-saturating the NIM endpoint and producing `timeout_after_escalation` failures at 41s wall-clock — the NIM endpoint serialises internally so every overlap added pure queueing latency. Three changes in one commit. (1) Module-level `asyncio.Semaphore(2)` at `backend/nvidia_nim_llm.py:104` wraps every `httpx.post` to `integrate.api.nvidia.com` so the entire process never has more than 2 NIM requests in flight simultaneously, regardless of source — probe loop, admin polls, and per-user turns all serialise through the same semaphore. (2) `backend/llm_health.py::probe_all` changed from `asyncio.gather(...)` to a serial `for m in models:` loop so the 6-NIM probe burst becomes a 1-slot trickle over ~12s instead of contending with live user turns. (3) The 4-attempt exponential-backoff inner retry inside `NvidiaNimLLM.chat()` was deleted — KI-080 sticky-primary election + KI-079 heavy-chain escalation already handle failover at the right layer, so the inner retry only amplified the self-saturation. Live verify post-deploy: failure mode flipped from `fallback:timeout_after_escalation` at 41s → `fallback:no_trailer` at 4-10s. NIM concurrency bottleneck closed; new parser-side bottleneck surfaced and is addressed in KI-090. |
| **KI-089** | **`8a87526`** | **Credits-election test fix + paired NIM-empty test.** `test_groq_above_water_picked_in_election` had been failing on `main` since KI-087 landed: it asserted Groq wins election on raw latency (161ms LPU TTFT), but KI-087 inverted election to prefer any eligible NIM candidate over all non-NIM candidates regardless of latency. Replaced with two paired tests that pin KI-087's invariant explicitly. (1) `test_nim_preferred_over_faster_groq_when_eligible` — when an eligible NIM candidate exists, election picks it as PRIMARY even though Groq is measurably faster. (2) `test_groq_picked_when_nim_pool_empty` — when every NIM candidate is dead / throttled / `no_credits`, election correctly falls through to Groq as PRIMARY. Together the pair pin both halves of the KI-087 contract (NIM-first AND fallthrough-when-NIM-empty) so a future regression that breaks either half fails loudly. Full inline test count: credits-election 12/12 pass, routing-regression 15/15 pass. |
| **KI-090** | **`11cf4b3`** | **Lenient FF-block parser.** Post-KI-088 live probe showed ~70% of brain calls returned successfully in 4-10s (NIM concurrency fix surfaced the real bottleneck) but `_parse_ff_block` rejected the reply with `fallback:no_trailer` because the brain had dropped the literal `<FF>...</FF>` tags around its JSON tail. Real LLMs under load (Qwen 3-Next 80B, Nemotron Nano 30B, Groq Llama-3.3-70B) regularly drop the wrapper even when the structured payload is otherwise contract-compliant. New `_parse_ff_block` tries three strategies in order: (a) strict `<FF>{...}</FF>` (the original contract); (b) fenced ```` ```json {...} ``` ```` (common LLM habit); (c) bare `{...}` JSON object at the end of the reply. Each candidate must `json.loads` cleanly AND contain at least one contract key (`captured` / `slot_driving` / `complete`) before it counts — prevents false positives from prose that happens to contain `{...}`. `_strip_ff_block` mirrors the three strategies in reverse so prose-only output to the user never leaks the structured metadata block. Inline tests: 7/7 parse tests + 7/7 strip tests pass. Brain success rate climbs from ~30% (post-KI-088 baseline blocked by parser) toward ~95% (NIM concurrency healthy + parser accepts contract-compliant tails regardless of wrapper). |
| **KI-091** | **`9813994`** | **Skip `profile_extractor` + `faithfulness` judge on fact-find turns (saturated-chain hang + mid-session field-clear).** Pre-KI-091 every fact-find turn ran two dependent post-brain LLM chains: (a) `extract_profile_updates` (LLM re-reads the user message to pull profile fields), and (b) `check_faithfulness` (judge LLM grades the bot's reply against retrieved context). Both chains were credit-exhausted on the steady-state primary, hung the turn for 20+ seconds on `asyncio.wait_for`, and — critically — the extractor periodically returned `{"name": null}` for utterances that had nothing to do with name, causing `session.update_profile_field("name", None)` to wipe the captured value mid-session. `next_question(profile)` then re-asked the name slot the user had already answered. KI-091 gates both chains behind an `intent == "fact_find"` short-circuit in `backend/orchestrator.py` so fact-find turns skip them entirely — the fact-find brain (KI-070) already extracts fields natively from its `<FF>` JSON tail, and faithfulness scoring is meaningless on a turn whose reply is "what's your annual income?". QA-mode turns (recommendation / comparison / clarification) still run both chains as before. Live verify: name re-ask loop eliminated; fact-find turn latency drops from p95 28s → p95 6-8s. |
| **KI-094** | **`f068094`** | **Defensive `None`-guard in extractor merge — extractor cannot clear a filled profile field even when it runs.** Belt-and-braces companion to KI-091. On QA-mode turns the `extract_profile_updates` chain still runs (it's the right behaviour: a user can mention "I'm now 35" mid-recommendation and the profile should update). But the extractor under load periodically returns `{"name": null, "age": null, "dependents": null}` — semantically "nothing to extract" but the merge loop was writing every key including nulls back into the session, wiping filled fields. Added `if new_value in (None, "", []): continue` guard at the top of the extracted-fields loop in `backend/orchestrator.py` so the merge step ONLY overwrites a profile field when the extractor returns a real value. Null / empty-string / empty-list returns are now no-ops, regardless of which fields they target. Closes the same root-cause hole as KI-091 (LLM returning nulls clears state) at a second layer — extractor running and returning nulls is now safe even when KI-091's intent gate doesn't fire. Together: KI-091 prevents the extractor from running on fact-find turns at all; KI-094 ensures that if it DOES run (QA-mode turn), null returns can't wipe filled fields like `name`. |
| **KI-097** | (pending) | **Remove IP allowlist from admin — password-only gate.** Removed `ADMIN_IP_ALLOWLIST` env, deleted `_ip_allowed()`, simplified `_check_admin` to password-only, 404→401, ADR-023 superseded. |
| **KI-101** | **`66eb4ed`** | **Drop 6 function-local `import logging` lines from `backend/orchestrator.py` — fixes `UnboundLocalError` when the `asyncio.wait_for` `TimeoutError` branch fires before reaching the inline `import`.** Pre-KI-101 the orchestrator had a module-level `import logging` at the top of the file AND six redundant `import logging` lines inside individual `try` / `except` branches deeper in `handle_turn()`. Python's scoping rule promotes any name assigned (including `import name`) anywhere in a function to a function-local — so the moment the parser saw any of the inline `import logging` lines, `logging` became local to the enclosing function. If a `TimeoutError` from `asyncio.wait_for(...)` fired BEFORE control reached the inline import, the `logger = logging.getLogger(__name__)` line in the `except` handler raised `UnboundLocalError: local variable 'logging' referenced before assignment` — masking the actual TimeoutError and surfacing a confusing trace to the user. Removed all 6 inline imports; module-level import is the sole binding and is unambiguously a free variable in every nested scope. No behavioural change beyond eliminating the spurious traceback. |
| **KI-102** | **`4bb8da0`** | **Profile RAG cross-session leak — `upsert_profile_chunk` stamps `session_id` metadata, `retrieve` excludes profile docs from the main pass + per-session lookup triple-checks `meta.session_id == session_id`; legacy chunks without `session_id` silently refused (fail-closed).** Pre-KI-102 the profile RAG layer wrote one Chroma chunk per session-profile but didn't tag it with `session_id`, so when User B's session ran retrieval Chroma could return User A's profile chunk if its embedding was a near neighbour — leaking PII (age / dependents / health conditions) across sessions. Three changes in `backend/profile_rag.py`. (1) `upsert_profile_chunk(profile, session_id)` now stamps `metadata={"doc_type": "profile", "session_id": session_id, ...}` on every chunk. (2) Main `retrieve()` pass excludes `doc_type == "profile"` via `where={"doc_type": {"$ne": "profile"}}` so profile docs never enter the general retrieval pool. (3) Separate `_get_profile_chunk_for_session(session_id)` lookup pulls only the current session's profile by `where={"$and": [{"doc_type": "profile"}, {"session_id": session_id}]}` AND triple-checks `result.metadata["session_id"] == session_id` in Python before returning — defence-in-depth against a Chroma where-clause bug. Legacy chunks written pre-KI-102 (no `session_id` metadata) are silently refused by the lookup (fail-closed, not fail-open). Privacy invariant: a session can NEVER read another session's profile chunk, regardless of embedding similarity. ADR-022 extended with a session-isolation subsection. |
| **KI-103** | **`8ef5c43`** | **`_canonical_fallback` no_trailer loop-breaker — after 2 failed attempts on the same slot, mark skipped + advance via `session._ff_failed_attempts` + `_ff_skipped_slots`.** Pre-KI-103 if the brain returned `fallback:no_trailer` (KI-090 parser also failed) on the same slot 3+ turns in a row, the canonical fallback would re-ask the same canonical question forever — the slot was unfilled, so `next_question(slot_id)` kept selecting it, brain kept failing, user got an infinite identical-question loop. Added two session-level counters: `session._ff_failed_attempts[slot_id]` increments every time the brain bails on a given slot; `session._ff_skipped_slots: set[str]` records slots the canonical fallback has given up on. After 2 consecutive failed attempts on the same slot, `_canonical_fallback` marks the slot skipped and `next_question()` excludes it from selection, advancing to the next unfilled slot. The skipped slot is still flagged in the final profile completeness check so the user can fill it via the admin panel later. Caps the worst-case wedge at 2 turns × `MAX_SLOTS = 9` ≈ 18 turns to escape fact-find, instead of unbounded. |
| **KI-104** | **`407f2a1`** | **CoT / instruction-echo strip in `backend/voice_format.py::tts_preprocess` — kills `<think>` blocks, `**Reasoning:**` labels, `[INTERNAL]` blocks, sentence-anchored CoT starters; emergency fallback when the whole reply is CoT.** Real LLMs (Qwen 3-Next under load, Nemotron Nano with low temperature) occasionally leak chain-of-thought into the prose: `<think>The user is asking about PED waiting periods. I should check...</think>` or `**Reasoning:** First I need to figure out... **Answer:** The waiting period is 24 months.` Pre-KI-104 those leaks went straight to TTS and the user heard Sarvam read out the bot's internal monologue. New strip pipeline: (1) regex-strip `<think>...</think>` (multi-line, non-greedy); (2) regex-strip `**Reasoning:** ... **Answer:**` and `**Thought:** ... **Final:**` patterns; (3) regex-strip `[INTERNAL] ... [/INTERNAL]`; (4) sentence-level scrub of starters like `"Let me think about that."`, `"First, I'll consider..."`, `"Step 1: ..."`, `"To answer this, I need to..."` (anchored at sentence boundary so they don't kill legitimate prose mid-paragraph); (5) **emergency fallback** — if after all strips the cleaned reply is empty / < 10 chars / still entirely CoT-shaped, return a generic acknowledger ("Let me check that and get back to you.") instead of TTS-ing an empty string or a fragment. Inline tests: 12/12 pass across the 5 strip patterns + emergency-fallback case. |
| **KI-105** | **`8a58fa1`** | **Recommendation closer wired — explicit closer phrases classified as recommendation/comparison BEFORE `FACT_FIND_TRIGGERS` check; persona prompt gets `RECOMMENDATION_CLOSER_ADDENDUM` with a strict 3-policy ranked-shortlist contract.** Pre-KI-105 a fully-fact-found user saying "Now show me the top 3 policies for me" could still get bounced back into fact-find because the literal word "me" sat inside `FACT_FIND_TRIGGERS` (substring match was fixed by KI-023 word-boundary regex, but the closer-phrase intent classification was missing). New `RECOMMENDATION_CLOSER_PHRASES` frozenset in `backend/orchestrator.py` covers `"show me the top 3"`, `"rank"`, `"pitch me"`, `"compare X vs Y"`, `"which is best for me"`, etc. `classify_intent()` checks closer phrases FIRST — if matched, intent is forced to `"recommendation"` (or `"comparison"` for explicit X-vs-Y patterns) and the `FACT_FIND_TRIGGERS` check is skipped. Persona prompt for these turns is augmented with `RECOMMENDATION_CLOSER_ADDENDUM` — a strict contract requiring: (a) exactly 3 ranked policies, (b) one-line rationale per pick tied to the user's profile, (c) the 3-policy sum / IRDAI disclaimer at the end, (d) no hedging like "you might consider...". ADR-008 extended with a closer-mode subsection (the consultative persona retains its core rules; the closer addendum only kicks in when the user explicitly asks for a ranked shortlist). |
| **KI-106** | **`565bf31`** | **Graceful `TimeoutError` + `Exception` handling on `/api/chat` — wrapped in `asyncio.wait_for(45s)` + explicit catch; both return HTTP 200 with graceful phrasing instead of 500.** Pre-KI-106 if `handle_turn()` raised any unhandled exception OR exceeded an implicit wall-clock budget, FastAPI's default handler emitted HTTP 500 with a generic JSON error body. The frontend treated 500 as a hard error and surfaced "Something went wrong" to the user — a much worse UX than a graceful in-character bot response that admits the bot is overloaded. New wrap in `backend/main.py::chat()` puts `handle_turn(...)` inside `asyncio.wait_for(coro, timeout=45.0)` with explicit `except asyncio.TimeoutError` and a broad `except Exception` after that. Both return HTTP 200 with `ChatResponse(reply="I'm having trouble responding right now — could you try that again in a moment?", source="graceful_timeout"|"graceful_exception", ...)` so the frontend renders a normal bot bubble and TTS plays the recovery sentence. The 45s budget is the explicit ceiling that bounds the orchestrator's internal nested budgets (brain 25s + escalation 15s + extractor 5s headroom). Internal `logger.exception(...)` still captures the full traceback for the admin tab so silent failures don't hide bugs. |
| **KI-107** | **`3a9a14f`** | **`_safe_collection_get` helper for Chroma — catches exceptions on `collection.get(ids=[...])` (especially the KI-102 per-session profile lookup for never-existed sessions); returns `None` on miss with a WARNING log.** Pre-KI-107 the new KI-102 `_get_profile_chunk_for_session(session_id)` lookup called `collection.get(where={"$and": [...]})` directly and could raise on a never-existed session (Chroma's behaviour varies by version — some raise `ValueError`, some return `{"ids": []}`, the HF Space build was raising). The raise propagated up through `retrieve()` into `handle_turn()` and KI-106 caught it as a graceful exception, but the user got the recovery sentence instead of a normal reply when really there was just no profile chunk yet. New `_safe_collection_get(collection, **kwargs)` helper in `backend/profile_rag.py` wraps the `.get()` call in a `try / except Exception`, logs `logger.warning("profile_rag: collection.get miss for session=%s: %s", session_id, e)` on failure, and returns `None` so callers can short-circuit ("no profile chunk for this session yet — proceed without profile context") instead of raising. Applied at every `collection.get(ids=[...])` and `collection.get(where=...)` call site in `profile_rag.py` (3 sites total). The KI-102 triple-check semantics are preserved: a `None` return means "no chunk found" which is treated the same as "chunk found but `session_id` doesn't match" — both fail-closed. |
| D-001 | (multi) | ChromaDB HNSW bloat 3-layer prevention (ADR-029) |
| D-002 | (LaunchAgent edit) | Three silently-failing LaunchAgent scripts fixed |
| D-009 | `bcb7079` | Removed `tmp_*.py` debug files from repo root |
| D-022 | (inline) | NIM brain swap on 2026-05-14: Qwen 3-Next 80B (was DeepSeek V4-Pro) + Mistral Large 3 675B judge (was Llama-4 Maverick) + Nemotron Nano 30B fast brain (was DeepSeek V4-Flash). DeepSeek + Maverick retained as fallback chain entries. **ADR-019 inline comments still reference the original D-019 lineup; the swap was code-only.** |

## Verification artifacts

- `tests/test_routing_regression.py` — 15 unit tests, all passing. Pins KI-018 / KI-023 / KI-025 invariants.
- 5-Q post-fix smoke (no judge): factual 0% → 60%, nim-chain serving 100% of QA.
- Live HF Space smoke (`https://rohitsar567-insurancebot.hf.space`): PED waiting-period question now answers via `nim-chain::nemotron-3-nano-30b-a3b::v4-flash::qa` with a grounded reply, not the old "Happy to help. First, your age?" misroute.
- **Post-fix parallel 96-Q gold eval (93 of 96 completed; 3 trailing questions killed when the run hung on a NIM rate-limit edge case) — captured BEFORE KI-046 (refusal precision) and KI-049 (retrieval top-k boost) shipped:**
  - **Factual accuracy: 54.8% (51 / 93)** — up from the pre-fix 41.7% baseline. **This number is now stale; a fresh eval is running to measure the KI-046 + KI-049 lift and the headline will be updated once it lands.**
  - **KI-022 JSON-fallback** rescued 7 questions that would have scored 0 on Groq judge JSON errors. Without KI-022 the headline would have been ~47%.
  - PED waiting-period type — previously 0% pre-fix; samples now: "Bot correctly states the 24‑month waiting period" / "matched_nums=['24']" via regex fallback / "36‑month period and includes source cit…".
  - Stuck questions: rows 94-96 (all `regulatory_oos` refusals — those routes are already at ~100% earlier in the run; the rate-limit hang affected the brain call, not the refusal logic).
  - **Expected directional lift from the two pending fixes:** KI-049 directly targets the `sub_limit` accuracy gap called out under D-003 (structured cap-table chunks now have ~2× chance of landing in context); KI-046 directly targets the `exclusions_oos` refusal-logic gap. Both gaps were the explicit "next bottleneck" in the 54.8% post-mortem.
- Clean 100-persona audit pending — to run once Batch B (bucket reorg) ships and the HF Space is stable.

## Pending follow-ups (P1)

| Item | Status |
|---|---|
| Tax-related gold-QA questions (currently zero in `eval/gold_qa.json` — D-006 mitigation) | Open |
| `stream`-style faithfulness gate (refusal rate 2.7× higher than `terse`) | Open |
| Token-streaming responses (SSE) — biggest perceived-latency win remaining | Open (v2 roadmap) |
| Streaming TTS (Sarvam chunked synthesis) | Open |
| GPU-hosted local embeddings (Voyage replacement) | Open — `LocalEmbeddings` fallback already in `backend/providers/local_embeddings.py` |
| Broader unit-test coverage (currently only routing + load-balance pinned) | Open |

---

## What "enterprise-grade" actually means for this product

Before insurers will pilot this, the following must be true:

1. **Factual accuracy ≥ 90%** on gold-QA across all question types (currently 30% headline).
2. **Latency p95 ≤ 3s** on chat turns (currently 24s).
3. **Zero silent failures** — every `except Exception:` either re-raises or logs.
4. **Production observability** — every brain decision, every retrieval, every refusal logged with correlation IDs; dashboards for accuracy/latency/refusal-rate over time.
5. **Test coverage ≥ 70%** with unit + integration tests in CI.
6. **Fairness audit** — accuracy/refusal-rate within ±5% across language styles (hinglish gap is currently 2×).
7. **Disaster recovery runbook** — what happens when ChromaDB corrupts, when HF Space is down, when NIM rate-limits.
8. **PII handling per DPDP Act** — chat logs, uploaded policies, user profiles must have retention policies + deletion workflows.
9. **IRDAI compliance review** — every recommended product must be IRDAI-registered; the bot must never invent a product or premium.
10. **SOC 2 Type II readiness** — secrets management, access logs, change management.

This audit so far covers items 1-3, 5 (in progress), and 6. Items 7-10 require a separate scoping pass.

---

## Sprint 2026-05-15 (KI-167 → KI-179) — chain rewrite + sales_brain rip-out

Second-day follow-on sprint after live user testing flagged that the scripted `<FF>` trailer convention was leaking into every fallback turn. Full timeline in [`70-docs/40-evaluation/quality-sprint-2026-05-14.md`](../70-docs/40-evaluation/quality-sprint-2026-05-14.md) (Follow-on sprint section). Highlights against the register:

| KI | Impact on register | Note |
|---|---|---|
| **KI-167** ([ADR-039](../70-docs/60-decisions/ADR-039-llm-driven-sales-brain.md)) | Closes the "scripted fact-find leakage" defect at root — `backend/fact_find_brain.py` + `_canonical_fallback` + the `<FF>` trailer convention all deleted. Replaced with `backend/sales_brain.py` (one LLM call per turn, native provider JSON mode). KI-090 lenient parser, KI-103 no_trailer loop-breaker, and the entire `fact_find_brain::fallback:*` telemetry vocabulary are retired. | Net negative LOC. |
| **KI-168 / KI-173 / KI-174** | Voice UX hardening — hybrid Web Speech (live UX) + MediaRecorder (authoritative blob → Sarvam STT) + tab-switch revival hooks. | Frontend-only. |
| **KI-171** | Skip faithfulness Gate 4 on `fact_find` + `recommendation` intents (no retrieval context to grade against). | Tightens D-006 fairness risk surface. |
| **KI-175 / KI-176** | NIM chain reorder — Nemotron 49B demoted from primary to last resort; OpenRouter re-added as cross-provider diversity (using OR's `models: [...]` server-side fallback). | Mitigates D-004 latency long-tail. |
| **KI-178** | Live audit of OR `:free` `response_format` support — Llama 3.3 70B / Hermes 3 405B excluded; only Nemotron-3-Super 120B + Qwen 80B `:free` + Gemma-4 31B included. | Closes a silent-failure class proactively. |
| **KI-179** ([ADR-040](../70-docs/60-decisions/ADR-040-google-gemini-primary.md)) | New `backend/providers/google_gemini_llm.py`; Gemini 2.0 Flash → Brain Fast primary, Gemini 2.5 Flash → Brain Main primary. NIM Tier 1 fallback, OR `:free` Tier 2 diversity. Judge stays on NIM Mistral 675B (different family from Gemini — preserves brain ↔ judge family-diversity invariant). | Tier 0 free-tier conversational quality lift. |

The 2026-05-14 scoreboard rows for "Voice UX", "Fact-find robotic tone", and the new "LLM stack" row are now fully green. D-003 (factual accuracy) and D-004 (latency) remain partially open pending a fresh full-eval re-run on the new chain shape.

---

### Sprint 2026-05-27 (KI-225..KI-333) — Single-brain consolidation + upload-pipeline rebuild

Single-brain rewrite (KI-225, ~5,200 LOC removed) collapses the old sales+qa+faithfulness chain into one Gemini-with-function-calling call per turn; faithfulness is now structural (brain quotes only what `retrieve_policies` + `get_policy_facts` returned). Upload pipeline rebuilt to ADR-044 parity: 8-gate defence in `backend/security.py` + heuristic floor (~65-70% post KI-332) + Gemini single-pass / multi-pass / NIM / floor extraction chain + status↔scorecard parity by construction (KI-333) + ACTIVE POLICY DIVE-IN block via `view_context.active_policy_id` (KI-330) + heuristic-floor card on LLM-fail path (KI-331). All shipped on commits 2a58c28, 993bcd5, d92f07a, 2ec48b7; currently live on `e7f799a`. No new defects introduced — the D-001..D-010 register from this audit remains the live defect set; none are upload-pipeline regressions.

---

*This file regenerates as new evidence lands. Last updated: 2026-05-27 (sprint KI-225 → KI-333 landed; ADR-044 ships).*