fix(fact-find): keyword fast-path + re-ask cap (KI-011 critical)

CRITICAL bug discovered via the 100-persona audit on the first persona:
under audit concurrency, NIM rate-limit causes the Llama-3.3-70B normalizer
LLM call to fail. Normalizer returns None. Orchestrator KEEPS
awaiting_question_id set. Bot re-asks the SAME question. User answers
something — but it's an answer to the NEXT question we never moved on
to. Normalizer fails again. Bot re-asks again. Infinite loop.

Concretely on persona P002 (verbose style): turn 7 user said "so
basically, bangalore, let me know what you think" → location normalizer
LLM call hit NIM rate limit → returned None → reask_clarify. Then turns
8-20+ all received reask_clarify response to "Which city?" while the
persona moved on with answers to dependents/income/etc. Bot stayed stuck
asking about city forever.

Fix layered in two places:

[A] backend/fact_find_normalizer.py — KEYWORD FAST PATH
New `_keyword_normalize()` function tries deterministic substring
matches BEFORE the LLM. Hand-curated patterns cover ~80% of common
answers:
- dependents: spouse/wife/kids/parents combinations → enum
- income_band: "around 8 lakh", "more than 25" → enum
- primary_goal: "first policy", "compare", "tax planning" → enum
- location: 9 metros + 15 tier1 + 14 tier2 cities by name → tier
- budget: numeric "k" suffixes → band
- health_conditions: keyword extraction from "diabetes, BP" → list
LLM is fallback only for nuanced phrasing. Zero NIM calls for common
answers means audit concurrency can't break the fact-find flow.

[B] backend/orchestrator.py — RE-ASK CAP
Session now tracks _reask_counts[question_id]. After 2 failed
normalizations on the SAME question, the orchestrator gives up:
marks question as asked, clears awaiting_question_id, lets
next_question() move on. Better to have an incomplete profile than
an infinite re-ask trap.

[C] docs/40-evaluation/known-issues.md — 10-issue Quality Sprint Log
Documents 10 P0/P1/P2 issues found in the same code-review sweep
(faithfulness fail-open, session disk swallows, translator silent
fall-throughs, etc.) with fix plans. KI-011 is the one this commit
closes; others are tracked for future sprints.

Unit-tested: 14/14 keyword-path cases pass including the exact P002
verbose-style input "so basically, bangalore, let me know what you think".

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

backend/fact_find_normalizer.py

@@ -112,7 +112,6 @@ async def normalize_answer(question_id: str, raw_text: str) -> Any:
 
     schema = _FIELD_SCHEMA.get(question_id)
     if schema is None:
-        # Unknown question id — defensive pass-through
         return raw_text.strip() or None
 
     # Fast paths — no LLM needed for plain integers / cover-amount parsing.
@@ -121,10 +120,122 @@ async def normalize_answer(question_id: str, raw_text: str) -> Any:
     if question_id == "existing_cover":
         return _parse_existing_cover(raw_text)
 
+    # KEYWORD FAST PATH — robust to NIM rate-limit and to LLM hiccups.
+    # Try matching common patterns BEFORE the LLM call. Catches ~80% of
+    # answers without consuming a NIM request and is deterministic under
+    # load. The LLM is the fall-back for nuanced/edge phrasings.
+    kw = _keyword_normalize(question_id, raw_text)
+    if kw is not None:
+        validated = _validate(kw, schema)
+        if validated is not None:
+            return validated
+
     # Enum + list fields — let the LLM map natural language to canonical value.
     return await _llm_normalize(question_id, raw_text, schema)
 
 
+# ----------------------------------------------------------------------------
+# Keyword fast-path — hand-curated common phrasing → schema value.
+# Order matters within each field: more-specific patterns first.
+# Case-insensitive substring matches on a normalized version of the text.
+# ----------------------------------------------------------------------------
+
+def _keyword_normalize(question_id: str, raw_text: str) -> Any:
+    s = raw_text.lower()
+
+    if question_id == "dependents":
+        if any(k in s for k in ["spouse", "wife", "husband"]) and "kid" in s and "parent" in s:
+            return "self+spouse+kids+parents"
+        if any(k in s for k in ["spouse", "wife", "husband"]) and "kid" in s:
+            return "self+spouse+kids"
+        if any(k in s for k in ["spouse", "wife", "husband"]) and "parent" in s:
+            return "self+spouse+kids+parents"
+        if any(k in s for k in ["spouse", "wife", "husband"]):
+            return "self+spouse"
+        if "parent" in s and "no" not in s.split():
+            return "self+parents"
+        if "just me" in s or "only me" in s or "myself" in s or "only self" in s or s.strip() in {"me", "self"}:
+            return "self"
+
+    elif question_id == "income_band":
+        import re as _re
+        if _re.search(r"(more than|above|over|>=?|>)\s*25", s) or "25l+" in s or "25 lakh+" in s:
+            return "25L+"
+        m = _re.search(r"(\d+(?:\.\d+)?)\s*(?:l|lakh|lac)", s)
+        if m:
+            val = float(m.group(1))
+            if val >= 25: return "25L+"
+            if val >= 10: return "10L-25L"
+            if val >= 5:  return "5L-10L"
+            return "under_5L"
+        if "10-25" in s or "10l-25l" in s: return "10L-25L"
+        if "5-10"  in s or "5l-10l"  in s: return "5L-10L"
+        if "under 5" in s or "<5" in s or "below 5" in s: return "under_5L"
+
+    elif question_id == "primary_goal":
+        if any(k in s for k in ["first policy", "first one", "first time", "first buy", "new policy", "buying my first"]):
+            return "first_buy"
+        if any(k in s for k in ["upgrade", "upgrading", "better cover", "more cover", "increase cover"]):
+            return "upgrade"
+        if any(k in s for k in ["compare", "comparison", " vs ", " vs.", "versus"]):
+            return "compare_specific"
+        if any(k in s for k in [" tax ", "80d", "deduction", "tax planning"]):
+            return "tax_planning"
+
+    elif question_id == "location":
+        metro = ["bangalore", "bengaluru", "mumbai", "delhi", "new delhi", "chennai", "kolkata", "hyderabad", "pune"]
+        tier1 = ["ahmedabad", "jaipur", "lucknow", "kanpur", "nagpur", "indore", "thane", "bhopal", "visakhapatnam", "patna", "vadodara", "ghaziabad", "ludhiana", "agra", "nashik"]
+        tier2 = ["surat", "kochi", "trivandrum", "thiruvananthapuram", "coimbatore", "vijayawada", "madurai", "rajkot", "ranchi", "amritsar", "allahabad", "prayagraj", "jodhpur", "raipur"]
+        for c in metro:
+            if c in s: return "metro"
+        for c in tier1:
+            if c in s: return "tier1"
+        for c in tier2:
+            if c in s: return "tier2"
+        if "metro" in s: return "metro"
+        if "tier 1" in s or "tier1" in s: return "tier1"
+        if "tier 2" in s or "tier2" in s: return "tier2"
+        if "tier 3" in s or "tier3" in s or "village" in s or "small town" in s: return "tier3"
+
+    elif question_id == "budget":
+        import re as _re
+        if "60k+" in s or ">60k" in s or "more than 60" in s or "above 60" in s:
+            return "60k+"
+        if "30-60" in s or "30k_60k" in s or "30k-60k" in s:
+            return "30k_60k"
+        if "15-30" in s or "15k_30k" in s or "15k-30k" in s:
+            return "15k_30k"
+        if "under 15" in s or "<15" in s or "below 15" in s or "under_15" in s:
+            return "under_15k"
+        m = _re.search(r"(\d+)\s*k", s)
+        if m:
+            v = int(m.group(1))
+            if v >= 60: return "60k+"
+            if v >= 30: return "30k_60k"
+            if v >= 15: return "15k_30k"
+            return "under_15k"
+
+    elif question_id == "health_conditions":
+        if any(p in s for p in ["none", "no condition", "nothing", "no pre-exist", "no health", "no chronic"]):
+            return []
+        canonical = []
+        cond_keywords = {
+            "diabetes": ["diabetes", "diabetic", "sugar"],
+            "hypertension": ["hypertension", " bp ", "blood pressure", "high bp"],
+            "thyroid": ["thyroid", "hypothyroid", "hyperthyroid"],
+            "asthma": ["asthma"],
+            "heart": ["heart problem", "heart disease", "cardiac"],
+            "cancer": ["cancer", "tumor"],
+        }
+        for cond, kws in cond_keywords.items():
+            if any(k in s for k in kws):
+                canonical.append(cond)
+        if canonical:
+            return canonical
+
+    return None
+
+
 # ----------------------------------------------------------------------------
 # Fast-path parsers (no LLM)
 # ----------------------------------------------------------------------------

backend/orchestrator.py

@@ -163,9 +163,14 @@ async def handle_turn(
     if treat_as_fact_find:
         # If we were awaiting an answer, normalize + record it before picking next Q.
         # Uses backend/fact_find_normalizer.py to map free-text → schema enums.
-        # If the input is a non-answer (STT failure / empty / gibberish), or the
-        # LLM can't confidently map it, we DON'T clear awaiting_question_id so
-        # the bot re-asks the same question rather than silently moving on.
+        #
+        # Two safety nets:
+        # (1) Keyword fast-path inside normalize_answer() handles ~80% of
+        #     answers without needing the NIM LLM (no rate-limit risk).
+        # (2) Re-ask cap (`_reask_count` on the session) — after 2 consecutive
+        #     failures on the same question we GIVE UP, skip that question,
+        #     and proceed to the next. Better to have an incomplete profile
+        #     than an infinite reask loop.
         ambiguous_or_failed = False
         if session.awaiting_question_id:
             from backend.fact_find_normalizer import is_valid_answer, normalize_answer
@@ -187,11 +192,27 @@ async def handle_turn(
                         if qid not in session.profile.asked:
                             session.profile.asked.append(qid)
                         session.set_awaiting(None)
+                        # Reset re-ask counter on success
+                        if hasattr(session, "_reask_counts"):
+                            session._reask_counts.pop(qid, None)
                     else:
                         ambiguous_or_failed = True
 
+            # ---- Re-ask cap (safety against infinite loops) ----
+            if ambiguous_or_failed:
+                if not hasattr(session, "_reask_counts"):
+                    session._reask_counts = {}
+                session._reask_counts[qid] = session._reask_counts.get(qid, 0) + 1
+                if session._reask_counts[qid] >= 2:
+                    # Give up on this question; mark it asked so next_question moves on.
+                    if qid not in session.profile.asked:
+                        session.profile.asked.append(qid)
+                    session.set_awaiting(None)
+                    ambiguous_or_failed = False  # no longer a reask situation
+
         # If the answer didn't normalize, pick the SAME question again (re-ask
-        # with a gentle clarifier) instead of moving on with garbage.
+        # with a gentle clarifier) instead of moving on with garbage — UNLESS
+        # the cap above just kicked in, in which case we move on.
         if ambiguous_or_failed and session.awaiting_question_id:
             q = next((qq for qq in __import__('backend.needs_finder', fromlist=['GRAPH']).GRAPH if qq.id == session.awaiting_question_id), None)
         else:

docs/40-evaluation/known-issues.md

@@ -0,0 +1,211 @@
+# Known Issues + Quality Sprint Log
+
+Living document. Every defect we find — whether via code review, eval audit,
+or production observation — lands here with severity, root cause, and the
+plan to fix it. Closed issues stay in the log with a `**FIXED in <sha>**`
+annotation so reviewers can audit the project's quality trajectory.
+
+## Severity scale
+
+- **P0 / Critical** — User-visible incorrect behavior, BFSI compliance risk,
+  or data loss. Block any release.
+- **P1 / High** — Silent degradation; user gets a worse experience but it
+  doesn't visibly break. Ship a fix in the next sprint.
+- **P2 / Medium** — Edge case; cosmetic; non-critical path. Backlog.
+- **P3 / Low** — Code smell or minor inefficiency.
+
+---
+
+## Open issues
+
+### KI-001 — Gate 4 (LLM judge) fails OPEN on judge error
+
+**Severity:** P0
+**Source:** `backend/faithfulness.py:253-255`
+**Discovered:** Code-review sweep 2026-05-14
+
+When the judge LLM call fails (network, rate limit, JSON parse error, NIM
+408/503), Gate 4 currently returns `supported=True, ["judge_error_failopen"]`.
+The reply ships through without grading.
+
+In BFSI this is the *unsafe* default — an unsupported claim that should
+have been blocked by Gate 4 leaks through to the user because the judge
+hiccupped. The audit log preserves `judge_error_failopen` but the user
+never sees the gate failed.
+
+**Fix plan:** Add `FAITHFULNESS_FAIL_CLOSED` env var (default `true` in
+production, `false` in dev / smoke tests). When fail-closed, return
+`supported=False, ["judge_unavailable_failclosed"]` so the orchestrator's
+cross-check retry path or final refusal fires instead.
+
+---
+
+### KI-002 — Session-state disk flush silently swallows errors
+
+**Severity:** P1
+**Source:** `backend/session_state.py:67-68`
+**Discovered:** Code-review sweep 2026-05-14
+
+`SessionState._flush()` writes the profile JSON to disk via a tmp+replace
+pattern. On disk-full, EACCES, or JSON encode error, the bare `except
+Exception: pass` drops the failure with zero observability. The user's
+profile is silently lost on the next Space restart. They redo fact-find.
+
+**Fix plan:** Add `logging.warning("session flush failed for %s: %s",
+session_id, e)` — keep the no-crash behaviour but surface failure rate to
+the HF Space logs so we can detect when it starts happening.
+
+---
+
+### KI-003 — Session-state disk load silently returns None on schema drift
+
+**Severity:** P1
+**Source:** `backend/session_state.py:114-115`
+**Discovered:** Code-review sweep 2026-05-14
+
+When the on-disk session JSON has a schema mismatch (Profile dataclass
+field renamed, type changed), `_load_from_disk` catches the exception and
+returns `None`. The user gets a fresh blank profile and has to redo
+fact-find. No log, no metric.
+
+**Fix plan:** Log the exception with `session_id` so we know schema drift
+is happening. Also: tighten the existing valid-field filter (line 105-106)
+to additionally type-check values so a stringified int doesn't pass through.
+
+---
+
+### KI-004 — Indic translator failure → original Indic text flows into English brain silently
+
+**Severity:** P1
+**Source:** `backend/orchestrator.py:148-149`
+**Discovered:** Code-review sweep 2026-05-14
+
+When Sarvam-M translator fails on an Indic query, the orchestrator falls
+through with the original Indic text, sending it to the English-trained
+DeepSeek/NIM brain. The brain handles it imperfectly. The user gets a
+degraded reply with no indication that the translator failed.
+
+**Fix plan:** Log the failure (`logging.warning("Indic translator failed
+for session %s, lang=%s: %s", session_id, language, e)`). Optionally
+return a soft refusal in Indic ("Sorry, I'm having trouble with the
+translation right now — could you ask in English?") instead of silently
+mis-routing.
+
+---
+
+### KI-005 — Profile-RAG chunk upsert failure silently swallowed
+
+**Severity:** P1
+**Source:** `backend/orchestrator.py:285-287`
+**Discovered:** Code-review sweep 2026-05-14
+
+After the conversational profile-update extractor lands a new field, the
+orchestrator re-upserts the profile chunk into Chroma so retrieval reflects
+the latest state. If that Chroma write fails (lock, disk, schema), the
+exception is swallowed. Subsequent turns retrieve the *stale* profile.
+The user thinks the bot incorporates their new fact ("I just got
+diabetes"); it actually doesn't.
+
+**Fix plan:** Log the failure + record in `TurnResult.profile_updates`
+that the upsert hit a problem so the frontend can show a small warning
+or retry.
+
+---
+
+### KI-006 — Conversational profile extraction failure silently swallowed
+
+**Severity:** P2
+**Source:** `backend/orchestrator.py:288-289`
+**Discovered:** Code-review sweep 2026-05-14
+
+If `extract_profile_updates()` itself raises (rare; NIM unavailable), the
+mid-chat profile-update feature is silently disabled for that turn. User
+won't know why their "I just turned 40" didn't take.
+
+**Fix plan:** Log + add to `TurnResult.profile_updates_meta` so the
+frontend could surface "we missed an update — try mentioning it again".
+
+---
+
+### KI-007 — Indic cascade total failure → English reply with zero log
+
+**Severity:** P2
+**Source:** `backend/orchestrator.py:425-426`
+**Discovered:** Code-review sweep 2026-05-14
+
+When all three Indic drift gates fail (or `translate_to_indic` itself
+raises), we fall back to English. The user asked in Hinglish but gets
+English. No log of which gate failed.
+
+**Fix plan:** Add structured logging of which gate caused the fall-back
+(`anchor` / `llmjudge` / `cosine`) so we can tune thresholds against real
+production drift data.
+
+---
+
+### KI-008 — TTS preprocess can swallow blocking content
+
+**Severity:** P3
+**Source:** `backend/main.py:258-272`
+**Discovered:** Code-review sweep 2026-05-14
+
+`tts_preprocess()` is called inside a `try: … except Exception as e: log
++ return text only` block. If the preprocessor strips the acronym expansion
+incorrectly, the TTS voice would butcher PED / SI / IRDAI etc. No fall-back
+to a hard-coded acronym dict — we just log + skip.
+
+**Fix plan:** Add a regression test for `tts_preprocess()` covering the 20
+most common BFSI acronyms.
+
+---
+
+### KI-009 — Live-mode VAD: no calibration on entry
+
+**Severity:** P2
+**Source:** `frontend/src/lib/useLiveConversation.ts` — `rmsThreshold: 28`
+**Discovered:** Code-review sweep 2026-05-14
+
+The RMS threshold is a constant. Quiet speakers, far-mic users, and noisy
+backgrounds all hit one fixed bar. Some users won't trigger VAD; others
+will trigger it on background noise.
+
+**Fix plan:** Calibrate the threshold by sampling 1 second of ambient
+audio on Live-mode entry. Set threshold to `mean(ambient) + 2 * sigma`.
+
+---
+
+### KI-010 — Audit runner: output unbuffered required `PYTHONUNBUFFERED=1`
+
+**Severity:** P3
+**Source:** `tools/audit/run_audit.py`
+**Discovered:** Self-test 2026-05-14
+
+Initial run had zero progress prints in the captured log because Python's
+default stdout buffering held lines until process exit. Fixed in the same
+session: all `print()` calls now use `flush=True` and we add per-5-turn
+progress prints. Document for future tooling.
+
+**Status:** FIXED in commit during audit framework rollout.
+
+---
+
+## Closed issues (this session)
+
+- **Issue 1: Full-duplex voice barge-in** — shipped in `d31e132`.
+- **Issue 2 + 4: Garbage profile recording + sidebar sync** — shipped in `9a1b321`.
+- **Issue 3: Sarvam STT 400 (webm→wav)** — shipped in `a777198`.
+- **Bug A: Cold-start "Load failed"** — shipped in `f81328f`.
+- **Bug B: Citation chip insurer prefix** — shipped in `f81328f`.
+- **Bug C: "Try again" intent handling** — shipped in `f81328f`.
+
+---
+
+## Quality-sprint cadence
+
+Every batch of fixes ships as one commit referencing the KI numbers it
+closes. The audit run (`audit_results/<run_id>/report.md`) is the
+empirical signal for whether a fix is actually working in production.
+
+The standing ratio target: **for every 1 user-facing bug a reviewer
+catches, we should close 5 internal issues from this log before the next
+review.**