70-docs/40-evaluation/known-issues.md

# Known Issues + Quality Sprint Log

> ⚠️ **Historical defect log — not the present-state map.** Entries below
> are dated incidents; many reference subsystems (`orchestrator`,
> `faithfulness.py` judge, `fact_find_brain` / `sales_brain`,
> `fact_find_normalizer`, the 3-tier Gemini/NIM/OpenRouter chain,
> sticky-primary election) that were **subsequently removed** by the
> single-LLM-with-tools rewrite (one Gemini 2.5-flash call per turn with
> `save_profile_field` / `retrieve_policies` / `mark_recommendation`,
> structured+vector retrieval, small `nim_fallback`). The log is retained
> as the quality trajectory of record; KI/ADR cross-references stay valid as
> history. Present-state authority: [`README.md`](../../README.md) §4.

Living document. Every defect we find — whether via code review, eval audit,
or production observation — lands here with severity, root cause, and the
plan to fix it. Closed issues stay in the log with a `**FIXED in <sha>**`
annotation so reviewers can audit the project's quality trajectory.

## Severity scale

- **P0 / Critical** — User-visible incorrect behavior, BFSI compliance risk,
  or data loss. Block any release.
- **P1 / High** — Silent degradation; user gets a worse experience but it
  doesn't visibly break. Ship a fix in the next sprint.
- **P2 / Medium** — Edge case; cosmetic; non-critical path. Backlog.
- **P3 / Low** — Code smell or minor inefficiency.

---

## Open issues

### KI-001 — Gate 4 (LLM judge) fails OPEN on judge error

**Severity:** P0
**Source:** `backend/faithfulness.py:253-255`
**Discovered:** Code-review sweep 2026-05-14

When the judge LLM call fails (network, rate limit, JSON parse error, NIM
408/503), Gate 4 currently returns `supported=True, ["judge_error_failopen"]`.
The reply ships through without grading.

In BFSI this is the *unsafe* default — an unsupported claim that should
have been blocked by Gate 4 leaks through to the user because the judge
hiccupped. The audit log preserves `judge_error_failopen` but the user
never sees the gate failed.

**Fix plan:** Add `FAITHFULNESS_FAIL_CLOSED` env var (default `true` in
production, `false` in dev / smoke tests). When fail-closed, return
`supported=False, ["judge_unavailable_failclosed"]` so the orchestrator's
cross-check retry path or final refusal fires instead.

---

### KI-002 — Session-state disk flush silently swallows errors

**Severity:** P1
**Source:** `backend/session_state.py:67-68`
**Discovered:** Code-review sweep 2026-05-14

`SessionState._flush()` writes the profile JSON to disk via a tmp+replace
pattern. On disk-full, EACCES, or JSON encode error, the bare `except
Exception: pass` drops the failure with zero observability. The user's
profile is silently lost on the next Space restart. They redo fact-find.

**Fix plan:** Add `logging.warning("session flush failed for %s: %s",
session_id, e)` — keep the no-crash behaviour but surface failure rate to
the HF Space logs so we can detect when it starts happening.

---

### KI-003 — Session-state disk load silently returns None on schema drift

**Severity:** P1
**Source:** `backend/session_state.py:114-115`
**Discovered:** Code-review sweep 2026-05-14

When the on-disk session JSON has a schema mismatch (Profile dataclass
field renamed, type changed), `_load_from_disk` catches the exception and
returns `None`. The user gets a fresh blank profile and has to redo
fact-find. No log, no metric.

**Fix plan:** Log the exception with `session_id` so we know schema drift
is happening. Also: tighten the existing valid-field filter (line 105-106)
to additionally type-check values so a stringified int doesn't pass through.

---

### KI-004 — Indic translator failure → original Indic text flows into English brain silently

**Severity:** P1
**Source:** `backend/orchestrator.py:148-149`
**Discovered:** Code-review sweep 2026-05-14

When Sarvam-M translator fails on an Indic query, the orchestrator falls
through with the original Indic text, sending it to the English-trained
DeepSeek/NIM brain. The brain handles it imperfectly. The user gets a
degraded reply with no indication that the translator failed.

**Fix plan:** Log the failure (`logging.warning("Indic translator failed
for session %s, lang=%s: %s", session_id, language, e)`). Optionally
return a soft refusal in Indic ("Sorry, I'm having trouble with the
translation right now — could you ask in English?") instead of silently
mis-routing.

---

### KI-005 — Profile-RAG chunk upsert failure silently swallowed

**Severity:** P1
**Source:** `backend/orchestrator.py:285-287`
**Discovered:** Code-review sweep 2026-05-14

After the conversational profile-update extractor lands a new field, the
orchestrator re-upserts the profile chunk into Chroma so retrieval reflects
the latest state. If that Chroma write fails (lock, disk, schema), the
exception is swallowed. Subsequent turns retrieve the *stale* profile.
The user thinks the bot incorporates their new fact ("I just got
diabetes"); it actually doesn't.

**Fix plan:** Log the failure + record in `TurnResult.profile_updates`
that the upsert hit a problem so the frontend can show a small warning
or retry.

---

### KI-006 — Conversational profile extraction failure silently swallowed

**Severity:** P2
**Source:** `backend/orchestrator.py:288-289`
**Discovered:** Code-review sweep 2026-05-14

If `extract_profile_updates()` itself raises (rare; NIM unavailable), the
mid-chat profile-update feature is silently disabled for that turn. User
won't know why their "I just turned 40" didn't take.

**Fix plan:** Log + add to `TurnResult.profile_updates_meta` so the
frontend could surface "we missed an update — try mentioning it again".

---

### KI-007 — Indic cascade total failure → English reply with zero log

**Severity:** P2
**Source:** `backend/orchestrator.py:425-426`
**Discovered:** Code-review sweep 2026-05-14

When all three Indic drift gates fail (or `translate_to_indic` itself
raises), we fall back to English. The user asked in Hinglish but gets
English. No log of which gate failed.

**Fix plan:** Add structured logging of which gate caused the fall-back
(`anchor` / `llmjudge` / `cosine`) so we can tune thresholds against real
production drift data.

---

### KI-008 — TTS preprocess can swallow blocking content

**Severity:** P3
**Source:** `backend/main.py:258-272`
**Discovered:** Code-review sweep 2026-05-14

`tts_preprocess()` is called inside a `try: … except Exception as e: log
+ return text only` block. If the preprocessor strips the acronym expansion
incorrectly, the TTS voice would butcher PED / SI / IRDAI etc. No fall-back
to a hard-coded acronym dict — we just log + skip.

**Fix plan:** Add a regression test for `tts_preprocess()` covering the 20
most common BFSI acronyms.

---

### KI-009 — Live-mode VAD: no calibration on entry

**Severity:** P2
**Source:** `frontend/src/lib/useLiveConversation.ts` — `rmsThreshold: 28`
**Discovered:** Code-review sweep 2026-05-14

The RMS threshold is a constant. Quiet speakers, far-mic users, and noisy
backgrounds all hit one fixed bar. Some users won't trigger VAD; others
will trigger it on background noise.

**Fix plan:** Calibrate the threshold by sampling 1 second of ambient
audio on Live-mode entry. Set threshold to `mean(ambient) + 2 * sigma`.

---

### KI-010 — Audit runner: output unbuffered required `PYTHONUNBUFFERED=1`

**Severity:** P3
**Source:** `tools/audit/run_audit.py`
**Discovered:** Self-test 2026-05-14

Initial run had zero progress prints in the captured log because Python's
default stdout buffering held lines until process exit. Fixed in the same
session: all `print()` calls now use `flush=True` and we add per-5-turn
progress prints. Document for future tooling.

**Status:** FIXED in commit during audit framework rollout.

---

## Closed issues (this session)

- **Issue 1: Full-duplex voice barge-in** — shipped in `d31e132`.
- **Issue 2 + 4: Garbage profile recording + sidebar sync** — shipped in `9a1b321`.
- **Issue 3: Sarvam STT 400 (webm→wav)** — shipped in `a777198`.
- **Bug A: Cold-start "Load failed"** — shipped in `f81328f`.
- **Bug B: Citation chip insurer prefix** — shipped in `f81328f`.
- **Bug C: "Try again" intent handling** — shipped in `f81328f`.

---

## Quality-sprint cadence

Every batch of fixes ships as one commit referencing the KI numbers it
closes. The audit run (`80-audit/<run_id>/report.md`) is the
empirical signal for whether a fix is actually working in production.

The standing ratio target: **for every 1 user-facing bug a reviewer
catches, we should close 5 internal issues from this log before the next
review.**

### KI-011 — Fact-find re-ask infinite loop under load — **FIXED in `171f2a4`**

**Severity:** P0
**Source:** `backend/orchestrator.py` fact-find branch + `backend/fact_find_normalizer.py` LLM-only path
**Discovered:** First persona of the 100-persona audit (P002, verbose style)

When NIM rate-limited the Llama-3.3-70B normalizer under audit
concurrency, the LLM call raised, the orchestrator marked the answer
ambiguous, kept `awaiting_question_id` set, and re-asked the same
question on the next turn. User moved on with answers to OTHER
questions. Normalizer rejected them. Bot re-asked again. Infinite loop.

**Fix:** Keyword fast-path in `fact_find_normalizer.py` (hand-curated
substring matchers for 9 metros, 15 tier1 cities, dependents
combinations, income/budget bands, primary goals, common health
conditions) bypasses the LLM for ~80% of answers. Re-ask cap in
`orchestrator.py` gives up after 2 failed normalizations on the same
question and marks it asked. Production audit on persona P002
post-fix: 30/30 turns completed in 85s with 0 refusals (vs. infinite
loop before).

### KI-012 — Bot stuck in fact_find_complete readback loop — **FIXED in next commit**

**Severity:** P0
**Source:** `backend/orchestrator.py` fact-find-complete branch
**Discovered:** Reviewing audit transcript of P002 post-KI-011-fix

After fact-find completes, the orchestrator only calls
`session.set_awaiting(None)` — it does NOT flip
`session.free_form_session = True`. On every subsequent turn, the
classifier still routes through the fact-find branch (because
`free_form_session` is false), `next_question()` returns None (all
fields captured), and the code path emits the readback summary AGAIN
instead of going to retrieval + brain.

**Effect:** P002 used 19 of 30 turns repeating the same readback
"Got it — here's what I've understood: …" instead of answering the
user's real policy questions. Every persona that completes fact-find
hits this.

**Fix:** When fact-find completes, set
`session.free_form_session = True` and flush to disk. Subsequent turns
skip the fact-find branch entirely and go through retrieval + brain
as intended.

### KI-013 — Bot recommends policies on vague openers without fact-find — **FIXED in next commit**

**Severity:** P0
**Source:** `backend/orchestrator.py` intent → fact_find gating
**Discovered:** Real user testing 2026-05-14

User gave a vague opening ("I want a health policy") and bot
immediately retrieved + pitched "Care Senior" — a senior-citizen-only
policy. User is not a senior. The intent classifier routed the message
to "recommendation" / "qa", not to fact_find. The orchestrator then
went straight to retrieval + brain → bot recommended whatever scored
highest, regardless of user demographics.

**Fix:** Force fact-find whenever profile is empty (no age, no
dependents, no income_band). Regardless of what the intent classifier
says. Bot will now always start with "First, your age?" before any
recommendation.

### KI-014 — Vague dependents term "family" auto-mapped to self+spouse+kids — **FIXED in next commit**

**Severity:** P1
**Source:** `backend/fact_find_normalizer.py` keyword fast-path
**Discovered:** Real user testing 2026-05-14

User said "family" as their dependents answer. Bot assumed
"self+spouse+kids". User had intended their joint family (parents +
siblings). All subsequent recommendations were wrong.

**Fix:** Add VAGUE_TERMS list (`family`, `everyone`, `joint family`,
etc.) that explicitly returns None from the keyword fast-path,
forcing either the LLM normalizer (which is more nuanced) or a re-ask
clarifier. Phrases like "family — me and my wife" still parse
correctly because the disambiguating words come through.

### KI-015 — Age in readback summary doesn't match user's stated age

**Severity:** P1
**Source:** Possibly `backend/needs_finder.py::record_answer` for age,
or LLM readback hallucination
**Discovered:** Real user testing 2026-05-14

User said "31" but bot's readback summary said "30". Possible causes:
(a) User's earlier answer contained "30" that the int parser caught
first; (b) The bot is using the LLM to generate the readback and the
LLM is hallucinating numeric values.

**Fix plan:** Add a CONFIRMATION step before the bot transitions to
free-form recommendations. After fact-find readback, the bot should
ask "Does this all look right? Reply 'yes' or correct anything that's
off." Then proceed only if user confirms. Also: log the raw fact-find
inputs vs the captured profile so we can debug mismatches.

### KI-016 — NIM has promoted Qwen3-next-80B over DeepSeek-V4-Flash

**Severity:** P2
**Source:** Live audit log: `brain=nim-chain::qwen3-next-80b-a3b-instruct::v4-flash::qa`
**Discovered:** Audit run 2026-05-14

The NIM chain now tries Qwen3-next-80B BEFORE DeepSeek-V4-Flash for
`qa` intents. NIM-side catalog change (not ours). Latency per turn
~10s — slower than V4-Flash's ~3s. Worth investigating whether
Qwen3-next is empirically a better fit than V4-Flash for our use
case (Indian health-insurance grounded Q&A), or whether we should
explicitly demote it via the admin panel's chain reorder.

**Fix plan:** Run eval/run.py on the gold set with each model
isolated as primary, compare factual/citation/refusal scores. If
V4-Flash wins, reorder via /api/admin/chain.

### KI-017 — Reviews underrepresented in vector store — **FIXED in `next commit`**

**Severity:** P2 → user-facing (sparse review retrieval)
**Source:** `tools/ingest_reviews.py` produced 1 chunk per insurer (~500 chars)
**Discovered:** Architecture audit 2026-05-14

Pre-fix state: 10 review chunks in Chroma vs ~116 KB of structured
review data in `data/reviews/*.json` (10 insurers, each with claim
metrics, aggregator ratings, Reddit sentiment, YouTube coverage,
news, aggregate score). A user asking "what do customers say about
Star Health?" retrieved only ONE generic paragraph per insurer,
losing the nuance of metrics-vs-sentiment-vs-news.

**Fix:** Refactored `review_to_paragraph()` → `review_to_chunks()`
that yields 4-6 semantically distinct chunks per insurer:
  1. CLAIM METRICS (IRDAI primary-source numbers)
  2. AGGREGATOR RATINGS (Policybazaar, InsuranceDekho, MouthShut, Trustpilot)
  3. REDDIT/QUORA SENTIMENT (notable themes + sample post URLs)
  4. YOUTUBE COVERAGE (creator reviews + sentiment)
  5. RECENT NEWS (verified press coverage, one line per item)
  6. OVERALL TRUST SCORE (aggregate + letter grade + computation notes)

Each chunk gets a `review_facet` metadata field so retrieval can
filter or boost by facet when intent is clear. Live count:
60 chunks total (was 10), all 10 insurers × 6 facets.

Verified locally; pushed to HF Dataset; live HF Space picks up
on next rebuild.

### KI-018 — `rag/policies.duckdb` stub is dead code (132 bytes, never populated)

**Severity:** P3
**Source:** `rag/policies.duckdb` is in the repo but empty (132-byte sqlite header only)
**Discovered:** Data architecture audit 2026-05-14

ADR-004 designed a hybrid structured (DuckDB) + vector (Chroma) split. The
DuckDB half was never populated; the marketplace UI ended up reading
`data/policy_facts/*.json` directly (one file per policy) at request time
via the `/api/policies/all` endpoint. The 132-byte file in the repo is
visually misleading — looks like a real store, isn't.

**Fix plan (two options):**
- (A) Populate it from `data/policy_facts/*.json` so the marketplace can
  do SQL filters (sum-insured ≥ X AND room-rent-cap = no AND restoration
  = unlimited). Worth ~30 min.
- (B) Remove the file + the `import duckdb` + the `policies.duckdb` ADR
  reference + add a note in ADR-025 explaining why structured filtering
  reads JSON directly. Cleaner architecturally.

**Recommendation:** (A) — once activated, the marketplace tab gets
proper SQL filtering that's faster than the current N-JSON-load pattern.

### KI-167 — Scripted `<FF>` trailer + canonical_fallback removed in favour of LLM-driven `sales_brain` — **FIXED** (ADR-039)

**Severity:** P0
**Source:** `backend/fact_find_brain.py` + `_canonical_fallback` + `_pick_opener` family in `backend/orchestrator.py`
**Discovered:** 2026-05-15 — user report *"zero natural LLM chat — it always defaults to the script."*

Pre-fix, every fact-find turn ran through ADR-030's one-call brain with a `<FF>...</FF>` trailer convention; on parse failure or contract violation the orchestrator fell to `_canonical_fallback`, which emitted the scripted `Question.prompt_en` of the next unfilled slot prefixed with `"Got that — {slot}."`. Real LLMs (Qwen, Nemotron, Mistral) under load regularly dropped the trailer tags (KI-090 lenient parser bought partial recovery; KI-155 / KI-156 / KI-158 / KI-161 showed it was structural) and silently violated the contract, so the scripted prefix became the dominant user-facing path.

**Fix:** Ripped out `backend/fact_find_brain.py` (441 LOC) + `_canonical_fallback` + `_pick_opener` + `_NEUTRAL_OPENERS` + `_FAMILY_OPENERS` + `_contains_self_introduction`. Replaced with `backend/sales_brain.py` — one LLM call per turn against `FAST_BRAIN_CHAIN` using native provider JSON mode (`response_mime_type=application/json` on Gemini, `response_format={"type":"json_object"}` on NIM). New deterministic post-processor `backend/sales_brain_normalizer.py` validates and normalises the captured fields. KI-091 / KI-094 None-guards retained. See [ADR-039](../60-decisions/ADR-039-llm-driven-sales-brain.md).

---

### KI-168 — Hybrid voice capture (Web Speech + MediaRecorder + Sarvam) — **FIXED**

**Severity:** P1
**Source:** `frontend/src/lib/useStreamingVoice.ts`
**Discovered:** 2026-05-15

Pre-fix voice UX waited for the full Sarvam STT round-trip before showing the user their transcribed text — felt sluggish on the chat surface. New hybrid: Web Speech API streams interim transcripts into the chat input live, while `MediaRecorder` captures the authoritative audio blob in parallel; on browser silence-detect the blob is POSTed to `/api/transcribe` (Sarvam Saarika STT) for the authoritative transcript. Auto-submit falls back to the Web Speech transcript only if Sarvam errors. KI-173 heartbeat + KI-174 `visibilitychange` / `focus` revival hooks keep the mic alive across tab and app switches.

---

### KI-171 — Skip faithfulness judge on `fact_find` + `recommendation` intents — **FIXED**

**Severity:** P1
**Source:** `backend/orchestrator.py`
**Discovered:** 2026-05-15

Faithfulness Gate 4 (LLM judge) was being run on every turn including fact-find and recommendation, where there's no retrieved-chunks context to grade the reply against — burning judge calls and occasionally blocking valid replies. Now gated behind `intent ∈ {qa, comparison}` only. Recommendation replies still go through Gates 1-3 (retrieval floor, citation integrity, regex numeric grounding).

---

### KI-175 / KI-176 — NIM chain reorder + OpenRouter re-added as cross-provider diversity — **FIXED**

**Severity:** P2
**Source:** Chain definitions in `backend/providers/nvidia_nim_llm.py`
**Discovered:** 2026-05-15

KI-175 demoted NIM Nemotron 49B from primary to last resort across all three chains — Mistral Large 3 675B and Qwen 80B consistently outperform on the live audit. KI-176 re-added OpenRouter as a cross-provider diversity tier using OR's `models: [...]` array param for server-side fallback within the OR free pool. The NIM-only lock from [ADR-038](../60-decisions/ADR-038-nim-only-chains.md) was relaxed in scope (the `<FF>` trailer convention that motivated it no longer exists post-KI-167); only OR `:free` candidates with verified native JSON mode are included (Nemotron-3-Super 120B, Qwen 80B `:free`).

---

### KI-178 / KI-179 — Google Gemini Flash added as Tier 0 primary — **FIXED** (ADR-040)

**Severity:** P2
**Source:** `backend/providers/google_gemini_llm.py` (new); chain definitions in `nvidia_nim_llm.py`
**Discovered:** 2026-05-15

Operator obtained a Google AI Studio API key. KI-178 audited which OpenRouter free-tier models support `response_format` (only Nemotron-3-Super 120B + Qwen 80B + Gemma-4 31B). KI-179 added a new `LLMProvider` against Google AI Studio's REST API (`gemini-2.0-flash` + `gemini-2.5-flash`, native JSON mode via `response_mime_type=application/json`, 1500 req/day free quota). Promoted to Tier 0 primary on Brain Fast (Gemini 2.0 Flash) + Brain Main (Gemini 2.5 Flash). NIM stays as Tier 1 fallback with Mistral 675B as the strongest non-Gemini candidate; OpenRouter sits as Tier 2 diversity. Judge stays on NIM Mistral 675B (different family from Gemini — preserves brain ↔ judge family-diversity invariant). See [ADR-040](../60-decisions/ADR-040-google-gemini-primary.md).

---

### KI-019 — `kb/policies/*.md` not embedded (224 human-readable summaries left out of retrieval)

**Severity:** P3
**Source:** `kb/policies/<policy_id>.md` files exist (224 markdown summaries) but the ingest pipeline only reads from `rag/extracted/*.json`. The natural-language summary text is never embedded.
**Discovered:** Data architecture audit 2026-05-14

Each policy has a hand-written or auto-generated `kb/policies/<id>.md`
that summarizes the policy in 1-2 pages of natural English (e.g.
"Care Supreme covers in-patient hospitalization across India with a
36-month PED waiting…"). These would be EXCELLENT retrieval targets
for "what does X cover?" style questions because they're already
phrased for human reading. Currently retrieval matches against raw
PDF-extracted prose which is denser + less semantic.

**Fix plan:** Add a kb-markdown ingestion path in `rag/ingest.py` that
walks `kb/policies/*.md`, splits at H2 boundaries (each H2 ≈ one
schema-field discussion), embeds each section, and writes with
`doc_type="summary"` so retrieval can prefer summary chunks when the
intent is high-level Q&A.

Expected impact: better grounding for natural-language "tell me about X"
questions; cleaner citations (one summary chunk vs. 3-4 raw chunks).