Spaces:
Sleeping
feat(llm+docs): KI-177 + KI-179 + KI-183 + ADR-040 docs cascade
Browse filesTHREE-TIER LLM CHAIN — Google Gemini PRIMARY, NIM Tier 1, OR Tier 2.
Live audit (KI-178) of OpenRouter's free-tier catalog killed my earlier
recommendation: gemini-2.0-flash isn't in OR's catalog, llama-3.3-70b-
:free and hermes-3-405b:free don't expose response_format. So OR's
free-with-JSON pool is only nemotron-3-super-120b / qwen3-next-80b /
gemma-4-31b. User obtained a Google AI Studio key, set up billing
($0 historical spend, $10 cap, free 1500/day quota active), confirmed
working live on gemini-2.5-flash + gemini-2.5-flash-lite.
NEW MODEL TIERS:
Brain Fast (sales_brain):
Tier 0 Google gemini-2.5-flash-lite (KI-183 — 2.0-flash retired)
Tier 1 NIM qwen3-next-80b → mistral-large-3 → llama-4-maverick
Tier 2 OR nemotron-3-super-120b:free → qwen3-next-80b:free → gemma-4-31b:free
Last NIM nemotron-super-49b (only when all else dies)
Brain Main (orchestrator):
Tier 0 Google gemini-2.5-flash
Tier 1+ same NIM/OR cascade
Judge (faithfulness):
NIM-only at top mistral-large-3 → llama-4-maverick → OR qwen-free → nemotron-49b
(cross-family from Gemini brain preserves grading invariant)
CODE (KI-179 + KI-177 + KI-183):
+ backend/providers/google_gemini_llm.py (NEW) — direct REST,
native responseMimeType=application/json, factory get_gemini_llm
+ backend/providers/tiered_brain_llm.py (NEW) — wrapper
walking Gemini → NIM → OR for orchestrator brain path
M backend/sales_brain.py — 3-tier cascade; brain_used now
sales_brain::{gemini,nim,or}:<model>; KI-169 think-strip retry stays
in-tier; KI-183 model defaults updated for retired gemini-2.0-flash
M backend/orchestrator.py — pick_brain returns get_tiered_brain_llm(role)
M backend/providers/openrouter_llm.py — supports models=[...] param
M frontend/public/admin/llm-control.html — KI-177 clearChildren null guard
M tools/set_hf_secrets.py — GOOGLE_API_KEY in SECRETS_TO_SET
DOCS CASCADE (KI-180 + KI-182):
+ 70-docs/60-decisions/ADR-040-google-gemini-primary.md (NEW)
M ARCHITECTURE.md, CLAUDE.md, README.md
M 70-docs/00-overview/{roadmap,problem-statement}.md
M 70-docs/10-architecture/{system-overview,stack-rationale,safety-architecture}.md
M 70-docs/30-engineering/{discovery-script,needs-analysis-flow}.md
M 70-docs/40-evaluation/{eval-methodology,known-issues,quality-sprint-2026-05-14}.md
M 70-docs/60-decisions/ADR-{014,016,019,026,030,031,032,038}.md
→ all marked Superseded by ADR-040 (2026-05-15)
M 80-audit/ENTERPRISE_AUDIT.md
M backend/README.md, frontend/README.md, rag/README.md, eval/README.md
M kb/AUDIT_TRAIL.md, tools/README.md
VERIFICATION:
py_compile all .py — clean
npx tsc --noEmit — clean
Live Gemini probe: HTTP 200 gemini-2.5-flash 943-1205ms; gemini-2.5-
flash-lite 739-1161ms; gemini-2.0-flash HTTP 404 (retired)
Tiered chain mock tests: gemini-up / gemini-down / gemini-nim-down /
all-down — all correct tag emission
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- 70-docs/00-overview/problem-statement.md +2 -2
- 70-docs/00-overview/roadmap.md +11 -2
- 70-docs/10-architecture/safety-architecture.md +6 -6
- 70-docs/10-architecture/stack-rationale.md +9 -6
- 70-docs/10-architecture/system-overview.md +160 -99
- 70-docs/30-engineering/discovery-script.md +9 -5
- 70-docs/30-engineering/needs-analysis-flow.md +76 -59
- 70-docs/40-evaluation/eval-methodology.md +5 -5
- 70-docs/40-evaluation/known-issues.md +52 -0
- 70-docs/40-evaluation/quality-sprint-2026-05-14.md +42 -0
- 70-docs/60-decisions/ADR-014-groq-llama-grader.md +5 -1
- 70-docs/60-decisions/ADR-016-hybrid-brain-router.md +5 -1
- 70-docs/60-decisions/ADR-019-nim-single-provider-consolidation.md +1 -1
- 70-docs/60-decisions/ADR-026-provider-load-balancing.md +3 -2
- 70-docs/60-decisions/ADR-030-llm-driven-fact-find.md +8 -3
- 70-docs/60-decisions/ADR-031-sticky-primary-election.md +5 -1
- 70-docs/60-decisions/ADR-032-llm-chain-architecture.md +5 -5
- 70-docs/60-decisions/ADR-038-nim-only-chains.md +3 -1
- 70-docs/60-decisions/ADR-040-google-gemini-primary.md +90 -0
- 80-audit/ENTERPRISE_AUDIT.md +22 -4
- ARCHITECTURE.md +95 -50
- CLAUDE.md +17 -12
- README.md +57 -36
- backend/README.md +14 -4
- backend/orchestrator.py +12 -8
- backend/providers/google_gemini_llm.py +233 -0
- backend/providers/openrouter_llm.py +63 -1
- backend/providers/tiered_brain_llm.py +229 -0
- backend/sales_brain.py +154 -30
- eval/README.md +4 -3
- frontend/README.md +3 -0
- frontend/public/admin/llm-control.html +1 -0
- kb/AUDIT_TRAIL.md +34 -15
- rag/README.md +1 -1
- tools/README.md +3 -3
- tools/set_hf_secrets.py +1 -0
|
@@ -219,9 +219,9 @@ To protect against scope creep — and to give the interviewer something to grad
|
|
| 219 |
|
| 220 |
### 8.4 Technical envelope
|
| 221 |
|
| 222 |
-
- v1 UI:
|
| 223 |
- v1 corpus: 15–20 health policies across 5 leading Indian insurers (Star Health, HDFC ERGO, Niva Bupa, Care Health, ICICI Lombard — exact list pending availability check).
|
| 224 |
-
- v1 deployment: local development + a single-instance deploy
|
| 225 |
|
| 226 |
---
|
| 227 |
|
|
|
|
| 219 |
|
| 220 |
### 8.4 Technical envelope
|
| 221 |
|
| 222 |
+
- v1 UI: Next.js 14 (App Router) + Tailwind v4 + shadcn/ui, calling a FastAPI backend. (The original draft scoped Streamlit; the migration to Next.js + FastAPI is captured in [ADR-005](../60-decisions/ADR-005-nextjs-fastapi-frontend.md).)
|
| 223 |
- v1 corpus: 15–20 health policies across 5 leading Indian insurers (Star Health, HDFC ERGO, Niva Bupa, Care Health, ICICI Lombard — exact list pending availability check).
|
| 224 |
+
- v1 deployment: local development + a single-instance deploy on Hugging Face Spaces (Docker) for the interviewer to demo.
|
| 225 |
|
| 226 |
---
|
| 227 |
|
|
@@ -10,12 +10,21 @@
|
|
| 10 |
|
| 11 |
v1 is a **vertical slice**: 10 insurers × Health × ~80 policies × voice-first advisor. The architecture is built so v2 is a **data/config change, not a rebuild**. This document maps the path.
|
| 12 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 13 |
## 1. What v1 ships
|
| 14 |
|
| 15 |
**Working product:**
|
| 16 |
- Voice-first chat advisor over a curated corpus of Indian health insurance policies (~76 PDFs from 10 insurers, ingested into Chroma + DuckDB)
|
| 17 |
- Multi-language: English + Hindi/Hinglish via Sarvam Saarika STT + Sarvam Bulbul TTS
|
| 18 |
-
-
|
| 19 |
- 4-gate hallucination defense + auditable refusal log
|
| 20 |
- 62-field structured extraction per policy
|
| 21 |
- Clean Next.js + Tailwind frontend
|
|
@@ -23,7 +32,7 @@ v1 is a **vertical slice**: 10 insurers × Health × ~80 policies × voice-first
|
|
| 23 |
- 8 design / decision documents totaling ~30 pages
|
| 24 |
|
| 25 |
**Eval signal:**
|
| 26 |
-
- Gold Q&A harness (~300 pairs targeted) + automated grader (the
|
| 27 |
- `eval/results.md` versioned table per run
|
| 28 |
- Live audit log `logs/hallucinations.jsonl` for every blocked claim
|
| 29 |
|
|
|
|
| 10 |
|
| 11 |
v1 is a **vertical slice**: 10 insurers × Health × ~80 policies × voice-first advisor. The architecture is built so v2 is a **data/config change, not a rebuild**. This document maps the path.
|
| 12 |
|
| 13 |
+
## 0.1 Recently shipped (2026-05-15)
|
| 14 |
+
|
| 15 |
+
- **KI-167 / [ADR-039](../60-decisions/ADR-039-llm-driven-sales-brain.md)** — Ripped out `backend/fact_find_brain.py` + canonical_fallback + the `<FF>` trailer convention. Replaced with `backend/sales_brain.py` — one LLM call per turn using native provider JSON mode. No scripted prompts, multi-fact capture per turn, ~500 LOC deleted.
|
| 16 |
+
- **KI-168** — Hybrid voice capture: Web Speech API streams interim transcripts for live UX feel while MediaRecorder runs in parallel; Sarvam STT is the authoritative transcript posted on browser silence-detect.
|
| 17 |
+
- **KI-171** — Skip faithfulness judge on `fact_find` + `recommendation` intents (the judge has no retrieved context to grade against on those turns).
|
| 18 |
+
- **KI-173 / KI-174** — Voice heartbeat + `visibilitychange` / `focus` revival hooks keep the mic alive across tab and app switches.
|
| 19 |
+
- **KI-175 / KI-176** — NIM chain reorder (Nemotron 49B demoted to last resort), OpenRouter re-added as cross-provider diversity via `models: [...]` server-side fallback.
|
| 20 |
+
- **KI-178 / KI-179 / [ADR-040](../60-decisions/ADR-040-google-gemini-primary.md)** — OpenRouter free-tier `response_format` audit; added Google AI Studio (Gemini 2.0 / 2.5 Flash, 1500 req/day free) as Tier 0 primary on Brain Fast + Brain Main.
|
| 21 |
+
|
| 22 |
## 1. What v1 ships
|
| 23 |
|
| 24 |
**Working product:**
|
| 25 |
- Voice-first chat advisor over a curated corpus of Indian health insurance policies (~76 PDFs from 10 insurers, ingested into Chroma + DuckDB)
|
| 26 |
- Multi-language: English + Hindi/Hinglish via Sarvam Saarika STT + Sarvam Bulbul TTS
|
| 27 |
+
- 3-tier brain chain (ADR-040, KI-179): Brain Fast (`backend/sales_brain.py`, fact-find) leads with **Gemini 2.0 Flash** (Google AI Studio, 1500 req/day free, native JSON mode) → NIM Qwen 80B / Mistral Large 3 675B / Llama-4 Maverick → OpenRouter `:free` (Nemotron-3-Super 120B, Qwen 80B) → NIM Nemotron 49B last resort. Brain Main (synthesis / comparison / recommendation) leads with **Gemini 2.5 Flash** → NIM Mistral 675B / Maverick / Qwen 80B → OR Nemotron-3-Super → NIM Nemotron 49B. Judge primary stays on **NIM Mistral Large 3 675B** (different family from the Gemini brain — preserves the cross-family invariant). Sticky-primary election (KI-080) per chain. Sarvam scoped to Indic translation + voice only.
|
| 28 |
- 4-gate hallucination defense + auditable refusal log
|
| 29 |
- 62-field structured extraction per policy
|
| 30 |
- Clean Next.js + Tailwind frontend
|
|
|
|
| 32 |
- 8 design / decision documents totaling ~30 pages
|
| 33 |
|
| 34 |
**Eval signal:**
|
| 35 |
+
- Gold Q&A harness (~300 pairs targeted) + automated grader (the JUDGE_CHAIN with NIM Mistral Large 3 675B primary — different family from the Gemini brain so grading stays non-circular)
|
| 36 |
- `eval/results.md` versioned table per run
|
| 37 |
- Live audit log `logs/hallucinations.jsonl` for every blocked claim
|
| 38 |
|
|
@@ -19,7 +19,7 @@ In a regulated BFSI domain, the worst failure isn't "the bot looks slow" — it'
|
|
| 19 |
|
| 20 |
**Description:** Bot claims a policy covers something it doesn't, or quotes a wrong waiting period / sub-limit.
|
| 21 |
**Detection:**
|
| 22 |
-
- Run-time: Faithfulness Gate 4 (LLM-judge
|
| 23 |
- Run-time: Faithfulness Gate 3 (regex grounding — every ₹, %, day/month/year in the reply must appear in retrieved chunks).
|
| 24 |
- Post-hoc: Gold Q&A eval (factual_accuracy metric).
|
| 25 |
**Mitigation:** Block reply → return safe refusal → log to `logs/hallucinations.jsonl`. Audit log enables manual review and corpus improvement.
|
|
@@ -50,7 +50,7 @@ In a regulated BFSI domain, the worst failure isn't "the bot looks slow" — it'
|
|
| 50 |
|
| 51 |
**Description:** Sarvam-M emits `<think>...</think>` reasoning. If `max_tokens` is exhausted before `</think>` is reached, the reply is unusable. This was a recurring issue when Sarvam-M was the primary brain.
|
| 52 |
**Detection:** `strip_think_tags()` checks for `<think>` without matching `</think>`.
|
| 53 |
-
**Resolution (2026-05-14, D-019
|
| 54 |
**Owner:** `backend/orchestrator.py`
|
| 55 |
**Status:** Resolved by architecture change.
|
| 56 |
|
|
@@ -95,9 +95,9 @@ In a regulated BFSI domain, the worst failure isn't "the bot looks slow" — it'
|
|
| 95 |
**Description:** End-to-end (user speech end → bot speech start) p95 > 7s (Doc 01 C1).
|
| 96 |
**Detection:** Per-turn latency logged in `logs/turns.jsonl`. Aggregate via eval harness.
|
| 97 |
**Mitigation v1:**
|
| 98 |
-
- Tiered brain routing (
|
| 99 |
- TTS happens server-side and is streamed via base64 in same response. Future: WebSocket for streaming TTS.
|
| 100 |
-
**Status:**
|
| 101 |
|
| 102 |
### F-10 — Streamlit-Cloud-Free / Render-Free cold start
|
| 103 |
|
|
@@ -171,7 +171,7 @@ Run-time. Auditable. Tested.
|
|
| 171 |
|
| 172 |
### F-16 — Translation cascade introduces drift after faithfulness gate
|
| 173 |
|
| 174 |
-
**Description:** Indic queries use the cascade: Sarvam translates Hinglish → English,
|
| 175 |
**Detection v1:** None automated. Manual spot-check of bilingual eval set.
|
| 176 |
**Mitigation v1:** Sarvam translator system prompt explicitly forbids changing numbers/citations + caps at 60 words; preserves `[Source: ...]` tags.
|
| 177 |
**Mitigation v2:** Back-translate Hinglish→English; compare against original English; block if cosine similarity < 0.85.
|
|
@@ -190,5 +190,5 @@ Run-time. Auditable. Tested.
|
|
| 190 |
| M-07 | Nightly synthetic eval cron | Infra | v2 |
|
| 191 |
| M-08 | Live-traffic spot grading | Eval | v2 |
|
| 192 |
| M-09 | 3-judge consensus for grader | Eval | v2 |
|
| 193 |
-
| M-10 | Sarvam-M no longer brain —
|
| 194 |
| M-11 | Faithfulness 4-gate verifier | Orchestrator | Live ✅ |
|
|
|
|
| 19 |
|
| 20 |
**Description:** Bot claims a policy covers something it doesn't, or quotes a wrong waiting period / sub-limit.
|
| 21 |
**Detection:**
|
| 22 |
+
- Run-time: Faithfulness Gate 4 (LLM-judge — JUDGE_CHAIN with NIM Mistral Large 3 675B primary; different family from the Gemini brain so grading stays non-circular; see [ADR-040](../60-decisions/ADR-040-google-gemini-primary.md)). KI-171 skips Gate 4 on `fact_find` + `recommendation` intents.
|
| 23 |
- Run-time: Faithfulness Gate 3 (regex grounding — every ₹, %, day/month/year in the reply must appear in retrieved chunks).
|
| 24 |
- Post-hoc: Gold Q&A eval (factual_accuracy metric).
|
| 25 |
**Mitigation:** Block reply → return safe refusal → log to `logs/hallucinations.jsonl`. Audit log enables manual review and corpus improvement.
|
|
|
|
| 50 |
|
| 51 |
**Description:** Sarvam-M emits `<think>...</think>` reasoning. If `max_tokens` is exhausted before `</think>` is reached, the reply is unusable. This was a recurring issue when Sarvam-M was the primary brain.
|
| 52 |
**Detection:** `strip_think_tags()` checks for `<think>` without matching `</think>`.
|
| 53 |
+
**Resolution (2026-05-14, D-019 → refined 2026-05-15 across KI-080 / KI-167 / KI-179):** Sarvam-M moved out of the brain role entirely. The current Brain Fast + Brain Main chains lead with Google AI Studio (Gemini 2.0 / 2.5 Flash) with NIM (Qwen 80B / Mistral Large 3 675B / Llama-4 Maverick) and OpenRouter `:free` (Nemotron-3-Super, Qwen 80B) as fallbacks; see [ADR-040](../60-decisions/ADR-040-google-gemini-primary.md). All chain primaries emit direct responses without `<think>` preambles, and native JSON mode (`response_mime_type=application/json` on Gemini, `response_format={"type":"json_object"}` on NIM / OpenRouter) provides server-side parseable output. Per-turn primary is elected from a 300s background probe (KI-080, [ADR-031](../60-decisions/ADR-031-sticky-primary-election.md)) — adapts to live provider degradation. Sarvam-M remains only for Indic translation. F-04 cannot fire on the current stack.
|
| 54 |
**Owner:** `backend/orchestrator.py`
|
| 55 |
**Status:** Resolved by architecture change.
|
| 56 |
|
|
|
|
| 95 |
**Description:** End-to-end (user speech end → bot speech start) p95 > 7s (Doc 01 C1).
|
| 96 |
**Detection:** Per-turn latency logged in `logs/turns.jsonl`. Aggregate via eval harness.
|
| 97 |
**Mitigation v1:**
|
| 98 |
+
- Tiered brain routing (ADR-040): voice turns and fact-find go to Brain Fast (Gemini 2.0 Flash primary, ~1-2s TTFT); `comparison` and `recommendation` intents hit Brain Main (Gemini 2.5 Flash primary, slightly higher latency but stronger synthesis). Sarvam-M no longer in the brain hot path.
|
| 99 |
- TTS happens server-side and is streamed via base64 in same response. Future: WebSocket for streaming TTS.
|
| 100 |
+
**Status:** Brain Fast / Brain Main typically respond in 2-6s; long-tail latency comes from heavy fallback candidates (NIM Mistral 675B) only when Google quota is exhausted.
|
| 101 |
|
| 102 |
### F-10 — Streamlit-Cloud-Free / Render-Free cold start
|
| 103 |
|
|
|
|
| 171 |
|
| 172 |
### F-16 — Translation cascade introduces drift after faithfulness gate
|
| 173 |
|
| 174 |
+
**Description:** Indic queries use the cascade: Sarvam translates Hinglish → English, the elected Brain Main candidate (Gemini 2.5 Flash by default, NIM Mistral 675B / Maverick / Qwen as fallbacks) reasons → English answer, faithfulness gates run on English answer, Sarvam translates English answer → Hinglish. **Faithfulness does NOT re-verify the final Hinglish output.** If Sarvam corrupts the translation (drops a citation, changes a number, invents a benefit), we wouldn't catch it.
|
| 175 |
**Detection v1:** None automated. Manual spot-check of bilingual eval set.
|
| 176 |
**Mitigation v1:** Sarvam translator system prompt explicitly forbids changing numbers/citations + caps at 60 words; preserves `[Source: ...]` tags.
|
| 177 |
**Mitigation v2:** Back-translate Hinglish→English; compare against original English; block if cosine similarity < 0.85.
|
|
|
|
| 190 |
| M-07 | Nightly synthetic eval cron | Infra | v2 |
|
| 191 |
| M-08 | Live-traffic spot grading | Eval | v2 |
|
| 192 |
| M-09 | 3-judge consensus for grader | Eval | v2 |
|
| 193 |
+
| M-10 | Sarvam-M no longer brain — Gemini 2.0 / 2.5 Flash primary on Brain Fast / Brain Main, NIM fallback | Orchestrator (ADR-040) | Live ✅ |
|
| 194 |
| M-11 | Faithfulness 4-gate verifier | Orchestrator | Live ✅ |
|
|
@@ -62,14 +62,15 @@ The product is split into five layers. Each layer has independent technology cho
|
|
| 62 |
| 8 | Backend | FE↔BE type safety | **`openapi-typescript` codegen** from FastAPI's auto-generated OpenAPI | Locked | Single source of truth; backend route changes propagate automatically | tRPC (Node-only), GraphQL, manual TS types |
|
| 63 |
| 9 | Orchestration | STT | **Sarvam Saarika v2.5** | Locked | Sarvam-first; their newer Indic ASR model | Whisper-large-v3, Deepgram Nova-2, Sarvam Saaras v3 |
|
| 64 |
| 10 | Orchestration | TTS | **Sarvam Bulbul** | Locked | Sarvam-first; Indic prosody first-class | ElevenLabs, OpenAI TTS, Coqui local |
|
| 65 |
-
| 11 | Orchestration | LLM (
|
| 66 |
-
|
|
|
|
|
| 67 |
| 13 | Orchestration | Embeddings | **BGE-small-en-v1.5 (local CPU, sentence-transformers)** | Locked | Free, runs offline, 384-dim, BEIR-strong on insurance/legal text. Voyage path kept for v2 swap (~3pp retrieval-quality delta in BEIR-style spot checks). | Voyage AI voyage-3 (rejected for v1 — 3 RPM free-tier blocked 104-PDF ingest), OpenAI text-embedding-3-small (rejected — no GPT), Cohere |
|
| 68 |
| 14 | Storage | Structured DB | **DuckDB** (single file) | Locked | One file, columnar, no server; deploys via `git clone` | SQLite, Postgres |
|
| 69 |
| 15 | Storage | Vector DB | **Chroma** (local persisted) | Locked | Embedded, no infra, supports metadata filtering | FAISS, Pinecone, Qdrant, Weaviate |
|
| 70 |
| 16 | Ingest | PDF parsing | **pdfplumber** | Locked | Clean text + page numbers — critical for citation grammar | PyMuPDF, Unstructured.io |
|
| 71 |
| 17 | Ingest | Chunking | **Custom 800-token chunks, 120-token overlap, page-aware** | Locked | Standard for technical docs; overlap protects clause boundaries | LangChain text splitter, fixed-size |
|
| 72 |
-
| 18 | Ingest | Structured extraction | **
|
| 73 |
| 19 | Pricing | Approach | **Hand-curated illustrative bands** from public PolicyBazaar quotes | Locked | Honest, defensible; "advisor-not-broker" positioning | Live scraping, actuarial model |
|
| 74 |
| 20 | Cross-cutting | Auth | **None for v1** (single-tenant demo) | Locked | Out of scope per `01-requirements.md` §7 | Auth0, Clerk, NextAuth |
|
| 75 |
| 21 | Cross-cutting | Observability | **JSONL turn log + cost tracker** | Locked | Lightweight; one log file, queryable post-hoc | Langfuse, Helicone, custom dashboard |
|
|
@@ -169,10 +170,12 @@ Every pick here corresponds to one or more entries in `decisions.md`:
|
|
| 169 |
| Consultative persona | D-008 |
|
| 170 |
| 10 insurers × all health policies | D-009 |
|
| 171 |
| Secret handling | D-010 |
|
| 172 |
-
| Embeddings provider (pending) | D-011 |
|
| 173 |
-
| Render deployment
|
| 174 |
| Next.js + Tailwind + shadcn UI stack | D-013 |
|
| 175 |
-
|
|
|
|
|
|
|
|
| 176 |
|
| 177 |
---
|
| 178 |
|
|
|
|
| 62 |
| 8 | Backend | FE↔BE type safety | **`openapi-typescript` codegen** from FastAPI's auto-generated OpenAPI | Locked | Single source of truth; backend route changes propagate automatically | tRPC (Node-only), GraphQL, manual TS types |
|
| 63 |
| 9 | Orchestration | STT | **Sarvam Saarika v2.5** | Locked | Sarvam-first; their newer Indic ASR model | Whisper-large-v3, Deepgram Nova-2, Sarvam Saaras v3 |
|
| 64 |
| 10 | Orchestration | TTS | **Sarvam Bulbul** | Locked | Sarvam-first; Indic prosody first-class | ElevenLabs, OpenAI TTS, Coqui local |
|
| 65 |
+
| 11 | Orchestration | LLM — Brain Fast (sales_brain) | **Gemini 2.0 Flash** primary (Google AI Studio, 1500 req/day free, native JSON mode) → NIM Qwen 80B → NIM Mistral Large 3 675B → NIM Llama-4 Maverick → OR `:free` (Nemotron-3-Super 120B, Qwen 80B) → NIM Nemotron 49B last resort | Locked (ADR-040) | Frontier free-tier conversational quality on the fact-find surface; native JSON mode via `response_mime_type=application/json` matches NIM's `response_format` contract validated since [ADR-039](../60-decisions/ADR-039-llm-driven-sales-brain.md) / KI-167. 6-level fallback survives Google quota exhaustion or 429. | NIM-only (rejected — strictly worse free-tier primary post-Gemini), OR Llama 3.3 70B (rejected — no `response_format` on OR free per KI-178), Sarvam-M as brain (rejected long ago for `<think>` truncation) |
|
| 66 |
+
| 11a | Orchestration | LLM — Brain Main (QA / comparison / recommendation) | **Gemini 2.5 Flash** primary → NIM Mistral Large 3 675B → NIM Llama-4 Maverick → NIM Qwen 80B → OR Nemotron-3-Super 120B → NIM Nemotron 49B last resort | Locked (ADR-040) | Higher synthesis quality than 2.0 Flash on long-context recommendation; shares the 1500 req/day Google quota. | (see row 11) |
|
| 67 |
+
| 12 | Orchestration | LLM — Judge (faithfulness Gate 4) | **NIM Mistral Large 3 675B** primary → NIM Llama-4 Maverick → OR Qwen 80B `:free` → NIM Nemotron 49B last resort | Locked (ADR-040) | Different family from the Gemini brain — preserves the brain ↔ judge family-diversity invariant ([ADR-014](../60-decisions/ADR-014-groq-llama-grader.md) lineage). KI-171 skips the judge on `fact_find` + `recommendation` intents (no retrieved context to grade). | Gemini as judge (rejected — same family as brain, circular grading), Groq Llama (rejected — rate cap), Sarvam-M held out |
|
| 68 |
| 13 | Orchestration | Embeddings | **BGE-small-en-v1.5 (local CPU, sentence-transformers)** | Locked | Free, runs offline, 384-dim, BEIR-strong on insurance/legal text. Voyage path kept for v2 swap (~3pp retrieval-quality delta in BEIR-style spot checks). | Voyage AI voyage-3 (rejected for v1 — 3 RPM free-tier blocked 104-PDF ingest), OpenAI text-embedding-3-small (rejected — no GPT), Cohere |
|
| 69 |
| 14 | Storage | Structured DB | **DuckDB** (single file) | Locked | One file, columnar, no server; deploys via `git clone` | SQLite, Postgres |
|
| 70 |
| 15 | Storage | Vector DB | **Chroma** (local persisted) | Locked | Embedded, no infra, supports metadata filtering | FAISS, Pinecone, Qdrant, Weaviate |
|
| 71 |
| 16 | Ingest | PDF parsing | **pdfplumber** | Locked | Clean text + page numbers — critical for citation grammar | PyMuPDF, Unstructured.io |
|
| 72 |
| 17 | Ingest | Chunking | **Custom 800-token chunks, 120-token overlap, page-aware** | Locked | Standard for technical docs; overlap protects clause boundaries | LangChain text splitter, fixed-size |
|
| 73 |
+
| 18 | Ingest | Structured extraction | **`NimChainLLM(BRAIN_CHAIN)` with Pydantic structured output + self-critique pass** | Locked | Reliable JSON-mode extraction via native provider JSON mode; self-critique runs through `NimChainLLM(JUDGE_CHAIN)` for per-field confidence. | Pure regex, LangChain extraction, Instructor lib |
|
| 74 |
| 19 | Pricing | Approach | **Hand-curated illustrative bands** from public PolicyBazaar quotes | Locked | Honest, defensible; "advisor-not-broker" positioning | Live scraping, actuarial model |
|
| 75 |
| 20 | Cross-cutting | Auth | **None for v1** (single-tenant demo) | Locked | Out of scope per `01-requirements.md` §7 | Auth0, Clerk, NextAuth |
|
| 76 |
| 21 | Cross-cutting | Observability | **JSONL turn log + cost tracker** | Locked | Lightweight; one log file, queryable post-hoc | Langfuse, Helicone, custom dashboard |
|
|
|
|
| 170 |
| Consultative persona | D-008 |
|
| 171 |
| 10 insurers × all health policies | D-009 |
|
| 172 |
| Secret handling | D-010 |
|
| 173 |
+
| Embeddings provider (pending → BGE local) | D-011 / ADR-011 |
|
| 174 |
+
| Render deployment (superseded by HF Spaces) | D-012 / ADR-012 |
|
| 175 |
| Next.js + Tailwind + shadcn UI stack | D-013 |
|
| 176 |
+
| Groq Llama grader (superseded → NIM Mistral) | D-014 / ADR-014 / ADR-040 |
|
| 177 |
+
| LLM-driven sales brain (rip-out of `<FF>` trailer) | ADR-039 |
|
| 178 |
+
| Google Gemini Flash as Tier 0 primary | ADR-040 |
|
| 179 |
|
| 180 |
---
|
| 181 |
|
|
@@ -3,10 +3,10 @@
|
|
| 3 |
| Field | Value |
|
| 4 |
| --- | --- |
|
| 5 |
| Project | Insurance Sales Portfolio Expert |
|
| 6 |
-
| Document version | 0.
|
| 7 |
-
| Date | 2026-05-
|
| 8 |
-
| Depends on | `
|
| 9 |
-
| Status |
|
| 10 |
|
| 11 |
---
|
| 12 |
|
|
@@ -27,40 +27,64 @@ This document specifies the technical architecture: the stack, the data flow, th
|
|
| 27 |
│ audio bytes / text / clicks
|
| 28 |
▼
|
| 29 |
┌─────────────────────────────────────────────────────────────────────┐
|
| 30 |
-
│
|
| 31 |
-
│
|
| 32 |
-
│
|
|
|
|
| 33 |
└─────────────────────────────────────────────────────────────────────┘
|
| 34 |
│ │ │
|
| 35 |
-
│ audio
|
| 36 |
▼ ▼ ▼
|
| 37 |
┌──────────────────┐ ┌─────────────────────┐ ┌──────────────────────┐
|
| 38 |
-
│ VOICE
|
| 39 |
-
│ ─ Sarvam
|
| 40 |
-
│
|
| 41 |
-
│ ─ Sarvam Bulbul │ ◄────── │ ─
|
| 42 |
-
│
|
| 43 |
-
└──────────────────┘ │ ─
|
| 44 |
-
│
|
| 45 |
-
│
|
| 46 |
└──────────┬──────────┘ │ at ingest time
|
| 47 |
│ │
|
| 48 |
▼ │
|
| 49 |
-
|
| 50 |
-
|
| 51 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 52 |
▲ │
|
| 53 |
│ retrieved chunks │
|
| 54 |
│ │
|
| 55 |
┌──────────┴──────────┐ │
|
| 56 |
│ VECTOR STORE │ ┌───────────┴──────────┐
|
| 57 |
│ Chroma │ │ INGEST PIPELINE │
|
| 58 |
-
│ ─ chunk +
|
| 59 |
│ ─ metadata: policy, │ │ ─ download PDFs │
|
| 60 |
│ page, clause │ │ ─ chunk + embed │
|
| 61 |
-
|
| 62 |
-
|
| 63 |
-
|
| 64 |
▲
|
| 65 |
│ raw PDFs
|
| 66 |
┌────────────┴─────────┐
|
|
@@ -71,21 +95,26 @@ This document specifies the technical architecture: the stack, the data flow, th
|
|
| 71 |
|
| 72 |
---
|
| 73 |
|
| 74 |
-
## 2. Stack picks (
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 75 |
|
| 76 |
-
|
| 77 |
-
| --- | --- | --- | --- | --- |
|
| 78 |
-
| STT | **Sarvam Saaras** | Sarvam-first per assignment context; strong Indic/Hinglish handling | Whisper-large-v3, Deepgram Nova | D-006 |
|
| 79 |
-
| TTS | **Sarvam Bulbul** | First-party Indic prosody; Sarvam-first | ElevenLabs, OpenAI TTS | D-006 |
|
| 80 |
-
| LLM | **Sarvam-M** (or current Sarvam flagship) | Sarvam-first; Indic-tuned reasoning | GPT-4o, Claude Sonnet | D-006 |
|
| 81 |
-
| Embeddings | **Sarvam embedding API if available, else `text-embedding-3-small`** | Will benchmark; whichever scores higher on retrieval @5 against our gold set | OpenAI embeddings, BGE-large local | D-006 |
|
| 82 |
-
| Vector DB | **Chroma** (local, persisted to disk) | Zero-config, embedded, works on Streamlit Cloud | FAISS (no metadata filtering), Qdrant (overkill) | D-004 |
|
| 83 |
-
| Structured DB | **DuckDB** (single file, embedded) | One-file, columnar, no server, deployable to Streamlit Cloud | SQLite, Postgres | D-004 |
|
| 84 |
-
| UI | **Streamlit** | Fastest v1; business logic decoupled so v2 swaps UI only | FastAPI + React (v2) | D-005 |
|
| 85 |
-
| Audio capture | **`streamlit-mic-recorder`** or `streamlit-webrtc` (whichever proves stable in <1h spike) | Required for browser-side mic | Native HTML5 + WebSocket (too much glue) | — |
|
| 86 |
-
| Deployment | **Streamlit Community Cloud** | Free, GitHub-connected, supports Streamlit out of box, secrets UI for API keys | Render, Fly.io | — |
|
| 87 |
|
| 88 |
-
**Provider abstraction:** every provider
|
| 89 |
|
| 90 |
---
|
| 91 |
|
|
@@ -94,31 +123,46 @@ This document specifies the technical architecture: the stack, the data flow, th
|
|
| 94 |
```
|
| 95 |
user speaks
|
| 96 |
│
|
| 97 |
-
|
| 98 |
-
|
| 99 |
│
|
| 100 |
-
▼
|
| 101 |
-
|
|
|
|
| 102 |
│
|
| 103 |
-
|
|
|
|
| 104 |
│
|
| 105 |
-
├── if
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 106 |
│
|
| 107 |
-
|
| 108 |
-
|
|
|
|
|
|
|
| 109 |
─ returns chunks with (policy_id, page, clause, text, score)
|
| 110 |
-
|
|
|
|
|
|
|
| 111 |
─ enforces citation grammar [Source: ...]
|
| 112 |
-
─
|
| 113 |
-
|
|
|
|
|
|
|
|
|
|
| 114 |
│
|
| 115 |
▼
|
| 116 |
-
voice.synthesize(text) ── Sarvam Bulbul ──►
|
|
|
|
| 117 |
│
|
| 118 |
▼
|
| 119 |
-
|
| 120 |
-
|
| 121 |
-
log_turn({stt_confidence, retrieved_ids, latency_ms, cost_usd, refused})
|
| 122 |
```
|
| 123 |
|
| 124 |
---
|
|
@@ -151,18 +195,18 @@ Full Pydantic model lives in `rag/schema.py` (designed in parallel by a sub-agen
|
|
| 151 |
for pdf in corpus:
|
| 152 |
text_chunks = chunk(pdf, target_tokens=800, overlap=120)
|
| 153 |
# chunks indexed in Chroma with metadata: policy_id, page, clause_path
|
| 154 |
-
|
| 155 |
# one-shot LLM extraction with the schema as a structured-output target
|
| 156 |
-
extracted =
|
| 157 |
full_pdf_text,
|
| 158 |
schema=HealthPolicy,
|
| 159 |
few_shot_examples=[hdfc_ergo_example, niva_bupa_example]
|
| 160 |
)
|
| 161 |
-
|
| 162 |
-
# self-critique pass: LLM checks every field's evidence in the source
|
| 163 |
-
confidence_per_field =
|
| 164 |
extracted.extraction_confidence_pct = aggregate(confidence_per_field)
|
| 165 |
-
|
| 166 |
# write to DuckDB
|
| 167 |
structured_store.upsert(extracted)
|
| 168 |
```
|
|
@@ -246,52 +290,68 @@ Every turn writes one JSON line to `logs/turns.jsonl`:
|
|
| 246 |
|
| 247 |
```
|
| 248 |
insurance-sales-bot/
|
| 249 |
-
├──
|
| 250 |
-
│ ├──
|
| 251 |
-
│ ├── main.py # Streamlit entrypoint
|
| 252 |
│ ├── orchestrator.py # turn handler, persona prompt, intent routing
|
| 253 |
-
│ ├──
|
| 254 |
-
│ ├──
|
| 255 |
-
│ ├──
|
| 256 |
-
│ ├──
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 257 |
│ └── providers/
|
| 258 |
-
│ ├──
|
| 259 |
-
│ ├──
|
| 260 |
-
│ ├──
|
| 261 |
-
│
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 262 |
├── rag/
|
| 263 |
-
│ ├── corpus/ # raw PDFs (
|
| 264 |
-
│ ├── extracted/ # per-policy JSON (
|
| 265 |
-
│ ├── vectors/ # Chroma persistence (
|
| 266 |
-
│ ├── policies.duckdb #
|
| 267 |
│ ├── schema.py # Pydantic HealthPolicy model
|
| 268 |
│ ├── ingest.py # download → chunk → embed → extract → store
|
| 269 |
│ ├── retrieve.py # query → top-k chunks
|
| 270 |
-
│
|
| 271 |
-
|
| 272 |
-
│
|
| 273 |
-
│
|
| 274 |
-
│
|
| 275 |
-
│
|
|
|
|
|
|
|
| 276 |
├── eval/
|
| 277 |
-
│ ├── gold_qa.json #
|
| 278 |
-
│ ├──
|
| 279 |
-
│
|
| 280 |
-
│ └── results.md # last run's accuracy table
|
| 281 |
-
├── data/
|
| 282 |
-
│ ├── insurers.json # 10 insurers metadata
|
| 283 |
-
│ └── corpus_urls.md # output of corpus-discovery sub-agent
|
| 284 |
├── 70-docs/
|
| 285 |
-
│ ├──
|
| 286 |
-
│ ├──
|
| 287 |
-
│ ├──
|
| 288 |
-
│ ├──
|
| 289 |
-
│ ├──
|
| 290 |
-
│
|
| 291 |
-
|
|
|
|
|
|
|
|
|
|
| 292 |
├── tests/
|
| 293 |
-
│
|
| 294 |
-
|
|
|
|
| 295 |
├── requirements.txt
|
| 296 |
├── .env / .env.example
|
| 297 |
├── .gitignore
|
|
@@ -302,11 +362,12 @@ insurance-sales-bot/
|
|
| 302 |
|
| 303 |
## 10. Deployment
|
| 304 |
|
| 305 |
-
- **
|
| 306 |
-
- **
|
| 307 |
-
- **
|
| 308 |
-
- **
|
| 309 |
-
- **
|
|
|
|
| 310 |
|
| 311 |
---
|
| 312 |
|
|
|
|
| 3 |
| Field | Value |
|
| 4 |
| --- | --- |
|
| 5 |
| Project | Insurance Sales Portfolio Expert |
|
| 6 |
+
| Document version | 0.2 |
|
| 7 |
+
| Date | 2026-05-15 |
|
| 8 |
+
| Depends on | `problem-statement.md` |
|
| 9 |
+
| Status | Live |
|
| 10 |
|
| 11 |
---
|
| 12 |
|
|
|
|
| 27 |
│ audio bytes / text / clicks
|
| 28 |
▼
|
| 29 |
┌─────────────────────────────────────────────────────────────────────┐
|
| 30 |
+
│ NEXT.JS 14 UI (frontend/src/app/page.tsx) │
|
| 31 |
+
│ ─ chat surface · Live + push-to-talk toggle · profile builder │
|
| 32 |
+
│ ─ Web Speech (interim) + MediaRecorder (authoritative) hybrid │
|
| 33 |
+
│ ─ in-DOM <audio> for TTS playback (barge-in compatible) │
|
| 34 |
└─────────────────────────────────────────────────────────────────────┘
|
| 35 |
│ │ │
|
| 36 |
+
│ audio blob │ user_query (text) │ filter / select
|
| 37 |
▼ ▼ ▼
|
| 38 |
┌──────────────────┐ ┌─────────────────────┐ ┌──────────────────────┐
|
| 39 |
+
│ VOICE │ │ ORCHESTRATOR │ │ STRUCTURED STORE │
|
| 40 |
+
│ ─ Sarvam Saarika │ ──────► │ (backend/ │ ◄ │ DuckDB │
|
| 41 |
+
│ STT │ text │ orchestrator.py) │ │ ─ 1 row per policy │
|
| 42 |
+
│ ─ Sarvam Bulbul │ ◄────── │ ─ classify_intent │ │ ─ 62-field schema │
|
| 43 |
+
│ TTS │ audio │ ─ profile RAG │ │ ─ filter / compare │
|
| 44 |
+
└──────────────────┘ │ ─ sales_brain / │ └──────────────────────┘
|
| 45 |
+
│ QA brain dispatch │ ▲
|
| 46 |
+
│ ─ faithfulness gates│ │ extracted
|
| 47 |
└──────────┬──────────┘ │ at ingest time
|
| 48 |
│ │
|
| 49 |
▼ │
|
| 50 |
+
┌──────────────────────────────────┐ │
|
| 51 |
+
│ LLM CHAINS (3-tier, ADR-040) │ │
|
| 52 |
+
│ ┌────────────────────────────┐ │ │
|
| 53 |
+
│ │ Brain Fast (sales_brain): │ │ │
|
| 54 |
+
│ │ Gemini 2.0 Flash (Tier 0) │ │ │
|
| 55 |
+
│ │ → NIM Qwen/Mistral/Llama4 │ │ │
|
| 56 |
+
│ │ → OR :free fallback │ │ │
|
| 57 |
+
│ │ → NIM Nemotron 49B (last) │ │ │
|
| 58 |
+
│ ├────────────────────────────┤ │ │
|
| 59 |
+
│ │ Brain Main (synthesis): │ │ │
|
| 60 |
+
│ │ Gemini 2.5 Flash (Tier 0) │ │ │
|
| 61 |
+
│ │ → NIM Mistral/Maverick/ │ │ │
|
| 62 |
+
│ │ Qwen │ │ │
|
| 63 |
+
│ │ → OR Nemotron-3-Super │ │ │
|
| 64 |
+
│ │ → NIM Nemotron 49B (last) │ │ │
|
| 65 |
+
│ ├────────────────────────────┤ │ │
|
| 66 |
+
│ │ Judge (cross-family): │ │ │
|
| 67 |
+
│ │ NIM Mistral Large 3 675B │ │ │
|
| 68 |
+
│ │ → NIM Llama-4 Maverick │ │ │
|
| 69 |
+
│ │ → OR Qwen 80B :free │ │ │
|
| 70 |
+
│ │ → NIM Nemotron 49B (last) │ │ │
|
| 71 |
+
│ └────────────────────────────┘ │ │
|
| 72 |
+
│ Sticky-primary election (KI-080)│ │
|
| 73 |
+
│ per chain via │ │
|
| 74 |
+
│ backend/llm_health.py probe loop│ │
|
| 75 |
+
└──────────────────────────────────┘ │
|
| 76 |
▲ │
|
| 77 |
│ retrieved chunks │
|
| 78 |
│ │
|
| 79 |
┌──────────┴──────────┐ │
|
| 80 |
│ VECTOR STORE │ ┌───────────┴──────────┐
|
| 81 |
│ Chroma │ │ INGEST PIPELINE │
|
| 82 |
+
│ ─ chunk + BGE-small │ ◄ │ (rag/ingest.py) │
|
| 83 |
│ ─ metadata: policy, │ │ ─ download PDFs │
|
| 84 |
│ page, clause │ │ ─ chunk + embed │
|
| 85 |
+
│ ─ profile chunks │ │ ─ LLM extract → DB │
|
| 86 |
+
│ (session-scoped) │ │ ─ per-insurer adapter│
|
| 87 |
+
└─────────────────────┘ └──────────────────────┘
|
| 88 |
▲
|
| 89 |
│ raw PDFs
|
| 90 |
┌────────────┴─────────┐
|
|
|
|
| 95 |
|
| 96 |
---
|
| 97 |
|
| 98 |
+
## 2. Stack picks (current)
|
| 99 |
+
|
| 100 |
+
| Layer | Pick | Why | Decision |
|
| 101 |
+
| --- | --- | --- | --- |
|
| 102 |
+
| STT | **Sarvam Saarika v2.5** | Best-in-class Indian-accent + Hinglish handling. | [ADR-006](../60-decisions/ADR-006-sarvam-first-stack.md) |
|
| 103 |
+
| TTS | **Sarvam Bulbul v2** | First-party Indic prosody; ₹ + lakh shorthand normalised pre-TTS by `voice_format.py`. | [ADR-006](../60-decisions/ADR-006-sarvam-first-stack.md) |
|
| 104 |
+
| LLM — Brain Fast (sales_brain) | **Gemini 2.0 Flash** primary → NIM Qwen 80B / Mistral Large 3 675B / Llama-4 Maverick → OR `:free` (Nemotron-3-Super, Qwen 80B) → NIM Nemotron 49B last resort. Native JSON mode via `response_mime_type=application/json` / `response_format={"type":"json_object"}`. | Frontier free-tier quality on Gemini (1500 req/day) with NIM diversity below it. Sticky-primary election (KI-080) picks one candidate per call. | [ADR-040](../60-decisions/ADR-040-google-gemini-primary.md), [ADR-039](../60-decisions/ADR-039-llm-driven-sales-brain.md) |
|
| 105 |
+
| LLM — Brain Main (QA / comparison / recommendation) | **Gemini 2.5 Flash** primary → NIM Mistral 675B / Maverick / Qwen 80B → OR Nemotron-3-Super → NIM Nemotron 49B last resort. | Higher synthesis quality than 2.0 on long-context recommendation; shares the Google quota with sales_brain. | [ADR-040](../60-decisions/ADR-040-google-gemini-primary.md) |
|
| 106 |
+
| LLM �� Judge (faithfulness Gate 4) | **NIM Mistral Large 3 675B** primary → NIM Llama-4 Maverick → OR Qwen 80B `:free` → NIM Nemotron 49B last resort. | Different family from the Gemini brain — preserves brain ↔ judge family-diversity invariant ([ADR-014](../60-decisions/ADR-014-groq-llama-grader.md) lineage). | [ADR-040](../60-decisions/ADR-040-google-gemini-primary.md) |
|
| 107 |
+
| Indic translation | **Sarvam-M** | Best-in-class Hinglish ↔ English translation. | [ADR-006](../60-decisions/ADR-006-sarvam-first-stack.md) |
|
| 108 |
+
| Embeddings | **BGE-small-en-v1.5 (local CPU, 384-d)** | Free, runs offline, no rate limits. | [ADR-011](../60-decisions/ADR-011-bge-local-embeddings.md) |
|
| 109 |
+
| Vector DB | **Chroma** (local persisted) | Embedded, supports metadata filtering, HNSW bloat guard ([ADR-029](../60-decisions/ADR-029-hnsw-bloat-tripwire.md)) | [ADR-004](../60-decisions/ADR-004-hybrid-structured-vector.md) |
|
| 110 |
+
| Structured DB | **DuckDB** (single file) | One-file, columnar, no server. | [ADR-004](../60-decisions/ADR-004-hybrid-structured-vector.md) |
|
| 111 |
+
| UI | **Next.js 14 (App Router) + Tailwind v4 + shadcn/ui** | Production-pattern, full UI flexibility. | [ADR-005](../60-decisions/ADR-005-nextjs-fastapi-frontend.md), [ADR-013](../60-decisions/ADR-013-tailwind-shadcn-ui.md) |
|
| 112 |
+
| Audio capture | **Web Speech API (interim text) + `MediaRecorder` (authoritative blob), hybrid (KI-168)** | Live UX cadence on the UI + Sarvam-grade STT for the actual transcript. | [ADR-028](../60-decisions/ADR-028-voice-ux-single-default-mode.md) |
|
| 113 |
+
| Deployment | **HF Spaces (Docker) + companion HF Dataset for corpus / vectors** | Free, GitHub-mirrored, snapshot_download for data hydration. | [ADR-012](../60-decisions/ADR-012-render-then-hf-space-deploy.md), [ADR-020](../60-decisions/ADR-020-code-data-split-hf-dataset.md) |
|
| 114 |
|
| 115 |
+
**Provider keys:** `GOOGLE_API_KEY` (Google AI Studio, 1500 req/day free), `NVIDIA_NIM_API_KEY` (free, 40 req/min), `OPENROUTER_API_KEY` ($10 unlocks 1000 req/day on `:free` models), `SARVAM_API_KEY` (Sarvam STT + TTS + translation).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 116 |
|
| 117 |
+
**Provider abstraction:** every LLM provider lives under `backend/providers/` (`google_gemini_llm.py`, `nvidia_nim_llm.py`, `openrouter_llm.py`) behind a common `LLMProvider` interface. `NimChainLLM` is provider-agnostic — chains may freely mix Google / NIM / OpenRouter candidates by URL. **This is non-negotiable** — it's c-readiness commitment #5.
|
| 118 |
|
| 119 |
---
|
| 120 |
|
|
|
|
| 123 |
```
|
| 124 |
user speaks
|
| 125 |
│
|
| 126 |
+
│ Web Speech streams interim transcript to chat input (live UX)
|
| 127 |
+
│ MediaRecorder captures audio blob in parallel
|
| 128 |
│
|
| 129 |
+
▼ (browser silence-detect → audio blob)
|
| 130 |
+
POST /api/transcribe ── Sarvam Saarika v2.5 ──► authoritative text
|
| 131 |
+
│ (fallback to Web Speech transcript if Sarvam errors)
|
| 132 |
│
|
| 133 |
+
▼ (auto-submit)
|
| 134 |
+
POST /api/chat → orchestrator.handle_turn(text, session)
|
| 135 |
│
|
| 136 |
+
├── if intent == fact_find:
|
| 137 |
+
│ sales_brain.drive_sales_turn(profile, history, user_text)
|
| 138 |
+
│ ─ ONE LLM call against FAST_BRAIN_CHAIN (Gemini 2.0 Flash primary)
|
| 139 |
+
│ ─ native JSON mode returns {reply, captures, slot_driving, complete}
|
| 140 |
+
│ ─ sales_brain_normalizer maps loose captures → canonical fields
|
| 141 |
+
│ ─ KI-094 None-guard ensures null captures don't wipe filled fields
|
| 142 |
+
│ ─ KI-171 skips faithfulness judge on fact-find turns
|
| 143 |
│
|
| 144 |
+
├── if intent == comparison: structured_store.query(filters) → rows
|
| 145 |
+
│
|
| 146 |
+
└── if intent ∈ {qa, recommendation}:
|
| 147 |
+
rag.retrieve(query, top_k=10)
|
| 148 |
─ returns chunks with (policy_id, page, clause, text, score)
|
| 149 |
+
─ profile_rag chunks filtered to current session_id (KI-102)
|
| 150 |
+
BRAIN_CHAIN.chat(persona_prompt + history + chunks + user_query)
|
| 151 |
+
─ Gemini 2.5 Flash primary → NIM Mistral 675B fallback
|
| 152 |
─ enforces citation grammar [Source: ...]
|
| 153 |
+
─ refusal returned when retrieval is weak
|
| 154 |
+
faithfulness gates (skipped on recommendation per KI-171):
|
| 155 |
+
Gate 1: retrieval floor Gate 2: citation integrity
|
| 156 |
+
Gate 3: regex numeric Gate 4: cross-family judge
|
| 157 |
+
(NIM Mistral Large 3 675B)
|
| 158 |
│
|
| 159 |
▼
|
| 160 |
+
voice.synthesize(text) ── Sarvam Bulbul v2 ──► base64 audio
|
| 161 |
+
│ (voice_format.tts_preprocess strips CoT leaks + expands ₹ shorthand)
|
| 162 |
│
|
| 163 |
▼
|
| 164 |
+
frontend in-DOM <audio> element plays the response (barge-in compatible)
|
| 165 |
+
log_turn({intent, brain_used, retrieved_ids, latency_ms, faithfulness, refused})
|
|
|
|
| 166 |
```
|
| 167 |
|
| 168 |
---
|
|
|
|
| 195 |
for pdf in corpus:
|
| 196 |
text_chunks = chunk(pdf, target_tokens=800, overlap=120)
|
| 197 |
# chunks indexed in Chroma with metadata: policy_id, page, clause_path
|
| 198 |
+
|
| 199 |
# one-shot LLM extraction with the schema as a structured-output target
|
| 200 |
+
extracted = NimChainLLM(BRAIN_CHAIN).extract_structured(
|
| 201 |
full_pdf_text,
|
| 202 |
schema=HealthPolicy,
|
| 203 |
few_shot_examples=[hdfc_ergo_example, niva_bupa_example]
|
| 204 |
)
|
| 205 |
+
|
| 206 |
+
# self-critique pass: judge LLM checks every field's evidence in the source
|
| 207 |
+
confidence_per_field = NimChainLLM(JUDGE_CHAIN).self_critique(extracted, full_pdf_text)
|
| 208 |
extracted.extraction_confidence_pct = aggregate(confidence_per_field)
|
| 209 |
+
|
| 210 |
# write to DuckDB
|
| 211 |
structured_store.upsert(extracted)
|
| 212 |
```
|
|
|
|
| 290 |
|
| 291 |
```
|
| 292 |
insurance-sales-bot/
|
| 293 |
+
├── backend/
|
| 294 |
+
│ ├── main.py # FastAPI app; HTTP routes (/api/chat, /api/transcribe, /api/profile, admin)
|
|
|
|
| 295 |
│ ├── orchestrator.py # turn handler, persona prompt, intent routing
|
| 296 |
+
│ ├── sales_brain.py # LLM-driven fact-find (one call/turn, native JSON mode)
|
| 297 |
+
│ ├── sales_brain_normalizer.py # deterministic post-processor over LLM captures
|
| 298 |
+
│ ├── needs_finder.py # 9-slot fact-find SCHEMA (data only; prompts retired)
|
| 299 |
+
│ ├── profile_extractor.py # conversational profile-update extractor
|
| 300 |
+
│ ├── profile_store.py # persistent named-profile JSON store (KI-040)
|
| 301 |
+
│ ├── profile_rag.py # session-scoped profile chunk in Chroma (KI-102)
|
| 302 |
+
│ ├── faithfulness.py # 4-gate hallucination guard
|
| 303 |
+
│ ├── persona.py # consultative-advisor system prompt
|
| 304 |
+
│ ├── voice_format.py # TTS pre-processing (CoT strip, ₹ shorthand)
|
| 305 |
+
│ ├── translator.py # Sarvam-M Indic ↔ English translation
|
| 306 |
+
│ ├── llm_health.py # background probe + sticky-primary election
|
| 307 |
+
│ ├── admin.py # /api/admin/* (LLM health, performance, profiles)
|
| 308 |
│ └── providers/
|
| 309 |
+
│ ├── sarvam_stt.py # Sarvam Saarika v2.5 client
|
| 310 |
+
│ ├── sarvam_tts.py # Sarvam Bulbul v2 client
|
| 311 |
+
│ ├── google_gemini_llm.py # Google AI Studio (Tier 0 primary)
|
| 312 |
+
│ ├── nvidia_nim_llm.py # NIM client + NimChainLLM dispatcher
|
| 313 |
+
│ ├── openrouter_llm.py # OpenRouter client (cross-provider diversity)
|
| 314 |
+
│ └── local_embeddings.py # BGE-small-en-v1.5 sentence-transformers
|
| 315 |
+
├── frontend/
|
| 316 |
+
│ ├── src/app/page.tsx # chat surface + voice toolbar + tab switcher
|
| 317 |
+
│ ├── src/lib/api.ts # typed API client (openapi-typescript codegen)
|
| 318 |
+
│ ├── src/lib/useStreamingVoice.ts # hybrid Web Speech + MediaRecorder hook (KI-168)
|
| 319 |
+
│ └── src/lib/useLiveConversation.ts # Live-mode VAD hook
|
| 320 |
├── rag/
|
| 321 |
+
│ ├── corpus/ # raw PDFs (hydrated from HF dataset at Docker build)
|
| 322 |
+
│ ├── extracted/ # per-policy JSON (62-field HealthPolicy)
|
| 323 |
+
│ ├── vectors/ # Chroma persistence (HF dataset side)
|
| 324 |
+
│ ├── policies.duckdb # DuckDB rollup
|
| 325 |
│ ├── schema.py # Pydantic HealthPolicy model
|
| 326 |
│ ├── ingest.py # download → chunk → embed → extract → store
|
| 327 |
│ ├── retrieve.py # query → top-k chunks
|
| 328 |
+
│ └── extract.py # LLM-driven structured extraction
|
| 329 |
+
├── 40-data/
|
| 330 |
+
│ ├── policy_facts/ # curated {value, source_quote} JSONs (256 variants)
|
| 331 |
+
│ ├── premiums/illustrative_premiums.json
|
| 332 |
+
│ ├── reviews/ # per-insurer IRDAI + aggregator + Reddit data
|
| 333 |
+
│ ├── profiles/ # persistent named profiles (KI-040)
|
| 334 |
+
│ ├── llm_health.json # last probe snapshot
|
| 335 |
+
│ └── llm_usage.jsonl # per-call telemetry
|
| 336 |
├── eval/
|
| 337 |
+
│ ├── gold_qa.json # 96+ gold Q&A pairs
|
| 338 |
+
│ ├── run.py # in-process orchestrator runner + regex + judge
|
| 339 |
+
│ └── results.md / results.json
|
|
|
|
|
|
|
|
|
|
|
|
|
| 340 |
├── 70-docs/
|
| 341 |
+
│ ├── 00-overview/ # roadmap + problem-statement
|
| 342 |
+
│ ├── 10-architecture/ # this doc + stack-rationale + safety + scoring
|
| 343 |
+
│ ├── 20-data-pipeline/ # ingestion-policy + info-source-map
|
| 344 |
+
│ ├── 30-engineering/ # discovery-script + needs-analysis-flow
|
| 345 |
+
│ ├── 40-evaluation/ # eval-methodology + known-issues + quality sprints
|
| 346 |
+
│ └── 60-decisions/ # ADR-001 … ADR-040
|
| 347 |
+
├── 80-audit/
|
| 348 |
+
│ ├── ENTERPRISE_AUDIT.md # live defect register
|
| 349 |
+
│ └── <run_id>/ # 100-persona audit transcripts
|
| 350 |
+
├── tools/ # operational scripts (corpus / KB / probes / cron)
|
| 351 |
├── tests/
|
| 352 |
+
│ ├── test_routing_regression.py
|
| 353 |
+
│ └── test_persona_*.py
|
| 354 |
+
├── Dockerfile # HF Space image (Python + snapshot_download)
|
| 355 |
├── requirements.txt
|
| 356 |
├── .env / .env.example
|
| 357 |
├── .gitignore
|
|
|
|
| 362 |
|
| 363 |
## 10. Deployment
|
| 364 |
|
| 365 |
+
- **Code repo:** `github.com/rohitsar567/insurance-sales-bot` (public; mirrored to HF Space `huggingface.co/spaces/rohitsar567/InsuranceBot`).
|
| 366 |
+
- **Data repo:** `huggingface.co/datasets/rohitsar567/insurance-bot-data` — 539 files / ~498 MB (corpus PDFs + Chroma vectors + extracted JSONs). See [ADR-020](../60-decisions/ADR-020-code-data-split-hf-dataset.md).
|
| 367 |
+
- **Hosting:** HF Space Docker image. `Dockerfile` runs `huggingface_hub.snapshot_download` at build time to hydrate `rag/` from the data repo so the Space repo stays code-only (~3 MB).
|
| 368 |
+
- **Secrets (HF Space repository secrets):** `GOOGLE_API_KEY`, `NVIDIA_NIM_API_KEY`, `OPENROUTER_API_KEY`, `SARVAM_API_KEY`, admin password / IP allowlist. Mirrored locally in `.env` (chmod 600, gitignored).
|
| 369 |
+
- **Per-deploy verification:** `tests/live_verify.py` runs a small smoke against the live Space URL post-deploy.
|
| 370 |
+
- **Triple-mirror invariant ([ADR-024](../60-decisions/ADR-024-triple-mirror-code-and-data.md)):** every commit pushed to BOTH GitHub origin AND the HF Space remote; data side mirrors via `tools/upload_*_to_dataset.py`.
|
| 371 |
|
| 372 |
---
|
| 373 |
|
|
@@ -129,10 +129,14 @@ This is the customer-protection framing. It tells the user honesty is **self-pro
|
|
| 129 |
|
| 130 |
## Implementation notes
|
| 131 |
|
| 132 |
-
The
|
| 133 |
|
| 134 |
-
|
| 135 |
-
2. Add a row in `70-docs/scorecard-knowledge-graph.md` Part B showing how the new input shifts weights.
|
| 136 |
-
3. Wire the shift into `_profile_tuned_weights()` in `backend/scorecard.py`.
|
| 137 |
|
| 138 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 129 |
|
| 130 |
## Implementation notes
|
| 131 |
|
| 132 |
+
The 9-slot fact-find SCHEMA still lives in `backend/needs_finder.py::GRAPH` — used now as the schema source for the `sales_brain` LLM system prompt rather than as a scripted question list. Each entry's `id`, `field`, `is_core`, and `condition` are consumed by the LLM as a structured contract; the `prompt_en` / `prompt_hi` strings are no longer rendered to the user post-[ADR-039](../60-decisions/ADR-039-llm-driven-sales-brain.md) / KI-167 (the LLM owns voice + cadence end-to-end via its system prompt + the conversation so far).
|
| 133 |
|
| 134 |
+
To add a new question:
|
|
|
|
|
|
|
| 135 |
|
| 136 |
+
1. Add a `Question(...)` entry with `id`, `field` (which Profile attribute it sets), `is_core` (boolean — counts toward completeness), optional `condition` callable, optional `parser`.
|
| 137 |
+
2. Surface the new slot in the `sales_brain` system prompt's 9-slot schema (alongside accepted value shapes + examples) so the LLM knows to capture it. See `backend/sales_brain.py::_SYSTEM_PROMPT`.
|
| 138 |
+
3. Wire any post-capture validation into `backend/sales_brain_normalizer.py` (enum coercion, INR parsing, bounds).
|
| 139 |
+
4. Add a row in `70-docs/scorecard-knowledge-graph.md` Part B showing how the new input shifts weights.
|
| 140 |
+
5. Wire the shift into `_profile_tuned_weights()` in `backend/scorecard.py`.
|
| 141 |
+
|
| 142 |
+
Drift between these places breaks the transparency promise. Keep them in sync.
|
|
@@ -1,24 +1,31 @@
|
|
| 1 |
-
# 05 — Needs Analysis Flow (
|
| 2 |
|
| 3 |
| Field | Value |
|
| 4 |
| --- | --- |
|
| 5 |
| Project | Insurance Sales Portfolio Expert |
|
| 6 |
-
| Version | 0.
|
| 7 |
-
| Date | 2026-05-
|
| 8 |
-
| Implementation | `backend/needs_finder.py` |
|
| 9 |
|
| 10 |
-
## 0. Why
|
| 11 |
|
| 12 |
-
A good Independent Financial Advisor opens with a stable, repeatable set of
|
| 13 |
|
| 14 |
-
|
| 15 |
|
| 16 |
-
|
| 17 |
-
2. **Testable.** Pure functions of `Profile → Question`. Every branch can be unit-tested.
|
| 18 |
-
3. **Fail-soft.** Even if the LLM brain degrades, the question flow still works.
|
| 19 |
-
4. **Bilingual by construction.** Each node has English + Hindi prompts side by side.
|
| 20 |
|
| 21 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 22 |
|
| 23 |
```
|
| 24 |
┌─────────────────────────────┐
|
|
@@ -74,76 +81,86 @@ Why:
|
|
| 74 |
|
| 75 |
## 2. Termination criteria
|
| 76 |
|
| 77 |
-
The
|
| 78 |
-
|
| 79 |
-
1. All 6 core questions answered (age, dependents, income, existing_cover, primary_goal, location)
|
| 80 |
-
2. All applicable conditional questions answered (parents_age if dependents include parents)
|
| 81 |
-
3. health_conditions and budget_band answered
|
| 82 |
-
|
| 83 |
-
Or when the user sets `profile.free_form_session = True` — the orchestrator skips fact-find and answers free questions directly.
|
| 84 |
|
| 85 |
-
|
| 86 |
|
| 87 |
-
|
| 88 |
-
|
| 89 |
-
| Node | English | Hindi |
|
| 90 |
-
| --- | --- | --- |
|
| 91 |
-
| age | "To start, what's your age?" | "शुरू करते हैं — आपकी उम्र क्या है?" |
|
| 92 |
-
| dependents | "Who else do you want to cover…" | "आपके अलावा किस-किस को cover करना है…" |
|
| 93 |
-
| budget | "What annual premium budget…" | "Premium के लिए सालाना कितना खर्च…" |
|
| 94 |
|
| 95 |
-
|
| 96 |
|
| 97 |
## 4. Parsing user answers
|
| 98 |
|
| 99 |
-
|
| 100 |
|
| 101 |
-
-
|
| 102 |
-
-
|
| 103 |
-
-
|
|
|
|
|
|
|
| 104 |
|
| 105 |
-
|
| 106 |
|
| 107 |
## 5. How the orchestrator uses this
|
| 108 |
|
| 109 |
```python
|
| 110 |
profile = Profile.from_session(session_id)
|
| 111 |
-
|
| 112 |
-
if
|
| 113 |
-
#
|
| 114 |
-
|
| 115 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 116 |
else:
|
| 117 |
-
#
|
| 118 |
-
chunks = retrieve(user_query, top_k=
|
| 119 |
-
reply =
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 120 |
```
|
| 121 |
|
| 122 |
-
The orchestrator can also let the user drive the conversation freely — if the user immediately asks a specific question like "compare Star and HDFC ERGO," we set `free_form_session = True` and skip the fact-find.
|
| 123 |
-
|
| 124 |
## 6. What the user sees
|
| 125 |
|
|
|
|
|
|
|
| 126 |
```
|
| 127 |
-
Bot: "Hi
|
| 128 |
-
|
| 129 |
-
|
| 130 |
-
User: "
|
| 131 |
-
Bot: "
|
| 132 |
-
|
| 133 |
-
|
| 134 |
-
|
| 135 |
-
|
| 136 |
-
|
| 137 |
-
|
| 138 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 139 |
```
|
| 140 |
|
|
|
|
|
|
|
| 141 |
## 7. v2 enhancements
|
| 142 |
|
| 143 |
| # | Enhancement | Why |
|
| 144 |
| --- | --- | --- |
|
| 145 |
-
| 1 |
|
| 146 |
-
| 2 |
|
| 147 |
| 3 | Skip-confirm flow ("you can skip this — say 'skip'") | Buyer autonomy |
|
| 148 |
-
| 4 | Save profile across sessions | Returning user picks up where they left off |
|
| 149 |
-
| 5 | Tone-match the user's energy (formal vs casual) |
|
|
|
|
| 1 |
+
# 05 — Needs Analysis Flow (LLM-driven sales brain)
|
| 2 |
|
| 3 |
| Field | Value |
|
| 4 |
| --- | --- |
|
| 5 |
| Project | Insurance Sales Portfolio Expert |
|
| 6 |
+
| Version | 0.2 |
|
| 7 |
+
| Date | 2026-05-15 |
|
| 8 |
+
| Implementation | `backend/sales_brain.py` (KI-167) — the schema source still lives in `backend/needs_finder.py::GRAPH` |
|
| 9 |
|
| 10 |
+
## 0. Why one LLM call per turn (not "scripted question graph + paraphraser")
|
| 11 |
|
| 12 |
+
A good Independent Financial Advisor opens with a stable, repeatable set of slots to fill — and adapts which slot to drive next based on what the buyer just said, what they've already told us, and what's still missing. We replicate this with **one LLM call per turn** that owns the entire fact-find surface (voice, cadence, slot-selection, multi-fact capture in a single turn), not a hardcoded state machine with a paraphraser glued on top.
|
| 13 |
|
| 14 |
+
The original v0.1 of this doc was an explicit `GRAPH` of canonical questions with `prompt_en` / `prompt_hi` strings. That design shipped with ADR-027 (LLM paraphraser on top of the graph) and ADR-030 (one-call brain with `<FF>` trailer + canonical fallback). Both architectures hit a hard wall: scripted prompts leaked into every fallback turn ("Got that — {slot}.") and produced robotic cadence even when the brain succeeded. KI-167 / [ADR-039](../60-decisions/ADR-039-llm-driven-sales-brain.md) ripped it all out: no canonical_fallback, no scripted prompts, no `<FF>` trailer. Native provider JSON mode (`response_mime_type=application/json` on Gemini, `response_format={"type":"json_object"}` on NIM) provides server-side parseable output.
|
| 15 |
|
| 16 |
+
What we kept and what we changed:
|
|
|
|
|
|
|
|
|
|
| 17 |
|
| 18 |
+
| Concern | v0.1 (scripted graph) | Current (LLM-driven sales_brain) |
|
| 19 |
+
|---|---|---|
|
| 20 |
+
| **Slot schema** | `GRAPH` of 9 `Question(id, prompt_en, prompt_hi, field, is_core, condition, parser)` entries | `GRAPH` data structure retained as the schema source; `prompt_en` / `prompt_hi` are dead strings now |
|
| 21 |
+
| **Question text** | Hardcoded `prompt_en` rendered to user | LLM generates natural prose in the advisor's voice per turn (no template) |
|
| 22 |
+
| **Slot selection** | `next_question(profile)` walked the graph in order | LLM picks `slot_driving` in its JSON response (constrained by the schema to required slots first) |
|
| 23 |
+
| **Multi-fact capture** | One slot per turn (a user saying "I'm 32, just myself, in Mumbai" only filled `age`) | LLM emits a `captures` map per turn — one utterance can fill 2-4 slots |
|
| 24 |
+
| **Auditable behavior** | Graph order was the trace | Per-turn LLM response is logged (intent, brain_used, captures, slot_driving, complete) in `40-data/llm_usage.jsonl` + `logs/turns.jsonl` |
|
| 25 |
+
| **Fail-soft** | Graph survived LLM degradation | Outer 25s `asyncio.wait_for` on the brain call; on total NIM + Google exhaustion the orchestrator returns the [ADR-038](../60-decisions/ADR-038-nim-only-chains.md) graceful error message to the user — no scripted reply |
|
| 26 |
+
| **Bilingual** | Hand-authored `prompt_en` + `prompt_hi` | Sarvam-M translation cascade on the LLM's output (English authoring + Indic translation), same UX, fewer hand-authored strings |
|
| 27 |
+
|
| 28 |
+
## 1. The 9-slot schema (data-only — the LLM consults this through its system prompt)
|
| 29 |
|
| 30 |
```
|
| 31 |
┌─────────────────────────────┐
|
|
|
|
| 81 |
|
| 82 |
## 2. Termination criteria
|
| 83 |
|
| 84 |
+
The LLM sets `complete: true` in its JSON response when all required slots are filled. `backend/sales_brain_normalizer.py` double-checks: if `complete == true` while any required slot is empty, the normalizer overrides to `complete = false` — the LLM cannot prematurely declare fact-find done.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 85 |
|
| 86 |
+
Or when the user opts out — if the user immediately asks "compare Star and HDFC ERGO" or "show me the top 3 policies for me" (KI-105 closer phrases), the intent classifier forces intent to `comparison` / `recommendation` BEFORE the fact-find branch is even reached, and the orchestrator routes straight to retrieval + Brain Main.
|
| 87 |
|
| 88 |
+
## 3. Bilingual handling
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 89 |
|
| 90 |
+
The LLM authors in English; Sarvam-M translates Hinglish ↔ English at the cascade boundary. The "Indic-native, not translated" principle (problem-statement §5.3) is preserved by the LLM's native-quality output rather than by hand-authored Hindi strings — the cascade itself is the bilingual surface.
|
| 91 |
|
| 92 |
## 4. Parsing user answers
|
| 93 |
|
| 94 |
+
The LLM produces a `captures` map per turn (e.g. `{"age": 34, "dependents": "self", "location_tier": "metro"}`). `backend/sales_brain_normalizer.py` is a pure-function post-processor that:
|
| 95 |
|
| 96 |
+
- Resolves field-name aliases (`location` → `location_tier`)
|
| 97 |
+
- Coerces enums (`Bangalore` → `metro`)
|
| 98 |
+
- Parses INR amounts (`"32 lakh"` → `3200000`)
|
| 99 |
+
- Drops null / empty values (KI-094 None-guard — a `None` capture never overwrites a filled field)
|
| 100 |
+
- Validates types and bounds (rejects `age: 250`)
|
| 101 |
|
| 102 |
+
No LLM calls in the normalizer — pure rules. The orchestrator applies the normalized map via `session.update_profile_field()` exactly as before. Profile persistence remains unchanged (`backend/profile_store.save_profile()` + `backend/profile_rag.upsert_profile_chunk()` with session-scoped metadata per KI-102).
|
| 103 |
|
| 104 |
## 5. How the orchestrator uses this
|
| 105 |
|
| 106 |
```python
|
| 107 |
profile = Profile.from_session(session_id)
|
| 108 |
+
|
| 109 |
+
if intent == "fact_find":
|
| 110 |
+
# KI-167: one LLM call per turn against FAST_BRAIN_CHAIN
|
| 111 |
+
outcome = await sales_brain.drive_sales_turn(
|
| 112 |
+
profile=profile,
|
| 113 |
+
history=session.chat_history[-N:],
|
| 114 |
+
user_text=user_text,
|
| 115 |
+
)
|
| 116 |
+
# outcome.reply, outcome.captures, outcome.slot_driving, outcome.complete
|
| 117 |
+
apply_normalized_captures(session, outcome.captures)
|
| 118 |
+
if outcome.complete:
|
| 119 |
+
session.free_form_session = True
|
| 120 |
+
reply = outcome.reply
|
| 121 |
else:
|
| 122 |
+
# QA / comparison / recommendation → Brain Main
|
| 123 |
+
chunks = retrieve(user_query, top_k=10)
|
| 124 |
+
reply = await BRAIN_CHAIN.chat(persona_prompt + history + chunks + user_text)
|
| 125 |
+
if intent not in {"fact_find", "recommendation"}:
|
| 126 |
+
# KI-171: judge skipped on fact-find + recommendation intents
|
| 127 |
+
verdict = await faithfulness.check(reply, chunks)
|
| 128 |
+
if not verdict.supported:
|
| 129 |
+
reply = SAFE_REFUSAL
|
| 130 |
```
|
| 131 |
|
|
|
|
|
|
|
| 132 |
## 6. What the user sees
|
| 133 |
|
| 134 |
+
A natural conversation, not a script. Example (real session, 2026-05-15):
|
| 135 |
+
|
| 136 |
```
|
| 137 |
+
Bot: "Hi! I'm your insurance advisor. To get you the right shortlist I'll need
|
| 138 |
+
to learn a bit about you and your situation. What's your name and a rough
|
| 139 |
+
sense of who we're looking to protect?"
|
| 140 |
+
User: "Hi, I'm Rohit Sar. I'm 32, just myself, living in Mumbai."
|
| 141 |
+
Bot: "Nice to meet you, Rohit. Got it — 32, covering yourself, Mumbai-based.
|
| 142 |
+
Two more quick things and we can look at policies: roughly what's your
|
| 143 |
+
annual income, and have you had any health conditions diagnosed —
|
| 144 |
+
diabetes, BP, anything chronic?"
|
| 145 |
+
User: "around 18 lakhs, no conditions"
|
| 146 |
+
Bot: "Perfect. And what's drawing you to insurance right now — first policy,
|
| 147 |
+
top-up on employer cover, or something specific?"
|
| 148 |
+
User: "first policy, employer cover feels too thin"
|
| 149 |
+
Bot: "Makes sense. Last one: comfortable annual premium range — under 15K,
|
| 150 |
+
15-30K, 30-60K, or higher?"
|
| 151 |
+
User: "15-30 is fine"
|
| 152 |
+
Bot: "Great — that's everything I need. Based on your profile, three policies
|
| 153 |
+
stand out: [...]"
|
| 154 |
```
|
| 155 |
|
| 156 |
+
Note: the LLM captured `{name, age, dependents, location_tier}` in a single turn from the user's opener. Pre-KI-167 each fact required its own turn. The exact turn count varies — some sessions finish in 3-4 turns, others in 6-7 depending on what the user volunteers.
|
| 157 |
+
|
| 158 |
## 7. v2 enhancements
|
| 159 |
|
| 160 |
| # | Enhancement | Why |
|
| 161 |
| --- | --- | --- |
|
| 162 |
+
| 1 | Stream the prose token-by-token to the frontend | The JSON tail is parsed server-side after stream completion — would cut perceived latency below the canonical-fallback path |
|
| 163 |
+
| 2 | Promote the same single-call pattern to the QA path | Orchestrator's QA + fact-find branches collapse into one brain call with a richer schema (intent in the JSON block instead of out-of-band classification) |
|
| 164 |
| 3 | Skip-confirm flow ("you can skip this — say 'skip'") | Buyer autonomy |
|
| 165 |
+
| 4 | Save profile across sessions ✅ shipped KI-040 | Returning user picks up where they left off |
|
| 166 |
+
| 5 | Tone-match the user's energy (formal vs casual) | The LLM already does this implicitly via the system prompt; could be explicit per-session signal |
|
|
@@ -32,7 +32,7 @@ For every successfully extracted policy in DuckDB, generate templated Q&A where
|
|
| 32 |
|
| 33 |
### Pipeline B — LLM-drafted nuanced questions (curated)
|
| 34 |
|
| 35 |
-
For top-priority policies (top 10–20), run
|
| 36 |
|
| 37 |
Target: 5 × 20 = 100 questions covering multi-clause and edge-case reasoning that Pipeline A can't generate.
|
| 38 |
|
|
@@ -56,9 +56,9 @@ Each pair marked with `expected_refusal: bool`. Refusal cases test that the bot
|
|
| 56 |
|
| 57 |
Numbers, dates, currency, durations, percentages are extracted via regex from both gold and bot reply. Exact match (after normalization) is the strongest signal. Catches "the premium is ₹15,000" hallucinations without an LLM.
|
| 58 |
|
| 59 |
-
### Signal 2 — LLM-judge faithfulness (
|
| 60 |
|
| 61 |
-
Different model family from
|
| 62 |
|
| 63 |
> Given GOLD and BOT answers, output strict JSON: `{factual_match: bool, citation_present: bool, score: 0-1, reason: str}`. Be strict on partial matches.
|
| 64 |
|
|
@@ -76,7 +76,7 @@ Different model family from Sarvam-M (the brain) → **non-circular evaluation**
|
|
| 76 |
| Citation accuracy | C3 ≥ 95% | n_correct_citations / n_non_refusal |
|
| 77 |
| Refusal precision | C4 ≥ 90% | n_correct_refusals / n_expected_refusals |
|
| 78 |
| Hindi parity | C8 within 5pp | factual_acc(hi) vs factual_acc(en) |
|
| 79 |
-
| Brain winners | (router config) | factual_acc grouped by brain_used |
|
| 80 |
|
| 81 |
## 4. Output artifacts per run
|
| 82 |
|
|
@@ -100,6 +100,6 @@ Bilingual sub-set: 20 questions translated to Hindi + Hinglish via Sarvam-M with
|
|
| 100 |
## 7. Known limitations (transparent)
|
| 101 |
|
| 102 |
- **Sample bias:** Pipeline A questions are templated, so accuracy on Pipeline A is upper-bound real-world performance. Pipeline B + C provide the harder signal.
|
| 103 |
-
- **Single judge model:**
|
| 104 |
- **No human evaluation:** Cost-prohibitive for 1,100 questions. We rely on the grader; manual spot-check 5%.
|
| 105 |
- **No latency budget enforcement in eval:** Latency is captured per record but doesn't gate the score. Doc 01 C1 (p50 ≤ 4s) is monitored separately.
|
|
|
|
| 32 |
|
| 33 |
### Pipeline B — LLM-drafted nuanced questions (curated)
|
| 34 |
|
| 35 |
+
For top-priority policies (top 10–20), run `NimChainLLM(BRAIN_CHAIN)` on the policy text with prompt: *"generate 5 buyer-style questions whose answers are explicitly in this document; include the source clause."* Each generated pair is **human spot-checked** before commit.
|
| 36 |
|
| 37 |
Target: 5 × 20 = 100 questions covering multi-clause and edge-case reasoning that Pipeline A can't generate.
|
| 38 |
|
|
|
|
| 56 |
|
| 57 |
Numbers, dates, currency, durations, percentages are extracted via regex from both gold and bot reply. Exact match (after normalization) is the strongest signal. Catches "the premium is ₹15,000" hallucinations without an LLM.
|
| 58 |
|
| 59 |
+
### Signal 2 — LLM-judge faithfulness (JUDGE_CHAIN: NIM Mistral Large 3 675B primary)
|
| 60 |
|
| 61 |
+
Different model family from the Gemini-led Brain Fast / Brain Main → **non-circular evaluation**. Falls through NIM Llama-4 Maverick → OR Qwen 80B `:free` → NIM Nemotron 49B as last resort. Judge prompt:
|
| 62 |
|
| 63 |
> Given GOLD and BOT answers, output strict JSON: `{factual_match: bool, citation_present: bool, score: 0-1, reason: str}`. Be strict on partial matches.
|
| 64 |
|
|
|
|
| 76 |
| Citation accuracy | C3 ≥ 95% | n_correct_citations / n_non_refusal |
|
| 77 |
| Refusal precision | C4 ≥ 90% | n_correct_refusals / n_expected_refusals |
|
| 78 |
| Hindi parity | C8 within 5pp | factual_acc(hi) vs factual_acc(en) |
|
| 79 |
+
| Brain winners | (router config) | factual_acc grouped by `brain_used` — emitted values include `sales_brain::continue`, `sales_brain::complete`, `brain_main::<elected_model>`, `brain_main::graceful_error` |
|
| 80 |
|
| 81 |
## 4. Output artifacts per run
|
| 82 |
|
|
|
|
| 100 |
## 7. Known limitations (transparent)
|
| 101 |
|
| 102 |
- **Sample bias:** Pipeline A questions are templated, so accuracy on Pipeline A is upper-bound real-world performance. Pipeline B + C provide the harder signal.
|
| 103 |
+
- **Single judge model:** NIM Mistral Large 3 675B is the elected judge primary; the fallback ladder includes Llama-4 Maverick + OR Qwen 80B + NIM Nemotron 49B for resilience but every grade run uses a single judge per question. Risk of judge-specific bias. v2: 3-judge consensus.
|
| 104 |
- **No human evaluation:** Cost-prohibitive for 1,100 questions. We rely on the grader; manual spot-check 5%.
|
| 105 |
- **No latency budget enforcement in eval:** Latency is captured per record but doesn't gate the score. Doc 01 C1 (p50 ≤ 4s) is monitored separately.
|
|
@@ -376,6 +376,58 @@ visually misleading — looks like a real store, isn't.
|
|
| 376 |
**Recommendation:** (A) — once activated, the marketplace tab gets
|
| 377 |
proper SQL filtering that's faster than the current N-JSON-load pattern.
|
| 378 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 379 |
### KI-019 — `kb/policies/*.md` not embedded (224 human-readable summaries left out of retrieval)
|
| 380 |
|
| 381 |
**Severity:** P3
|
|
|
|
| 376 |
**Recommendation:** (A) — once activated, the marketplace tab gets
|
| 377 |
proper SQL filtering that's faster than the current N-JSON-load pattern.
|
| 378 |
|
| 379 |
+
### KI-167 — Scripted `<FF>` trailer + canonical_fallback removed in favour of LLM-driven `sales_brain` — **FIXED** (ADR-039)
|
| 380 |
+
|
| 381 |
+
**Severity:** P0
|
| 382 |
+
**Source:** `backend/fact_find_brain.py` + `_canonical_fallback` + `_pick_opener` family in `backend/orchestrator.py`
|
| 383 |
+
**Discovered:** 2026-05-15 — user report *"zero natural LLM chat — it always defaults to the script."*
|
| 384 |
+
|
| 385 |
+
Pre-fix, every fact-find turn ran through ADR-030's one-call brain with a `<FF>...</FF>` trailer convention; on parse failure or contract violation the orchestrator fell to `_canonical_fallback`, which emitted the scripted `Question.prompt_en` of the next unfilled slot prefixed with `"Got that — {slot}."`. Real LLMs (Qwen, Nemotron, Mistral) under load regularly dropped the trailer tags (KI-090 lenient parser bought partial recovery; KI-155 / KI-156 / KI-158 / KI-161 showed it was structural) and silently violated the contract, so the scripted prefix became the dominant user-facing path.
|
| 386 |
+
|
| 387 |
+
**Fix:** Ripped out `backend/fact_find_brain.py` (441 LOC) + `_canonical_fallback` + `_pick_opener` + `_NEUTRAL_OPENERS` + `_FAMILY_OPENERS` + `_contains_self_introduction`. Replaced with `backend/sales_brain.py` — one LLM call per turn against `FAST_BRAIN_CHAIN` using native provider JSON mode (`response_mime_type=application/json` on Gemini, `response_format={"type":"json_object"}` on NIM). New deterministic post-processor `backend/sales_brain_normalizer.py` validates and normalises the captured fields. KI-091 / KI-094 None-guards retained. See [ADR-039](../60-decisions/ADR-039-llm-driven-sales-brain.md).
|
| 388 |
+
|
| 389 |
+
---
|
| 390 |
+
|
| 391 |
+
### KI-168 — Hybrid voice capture (Web Speech + MediaRecorder + Sarvam) — **FIXED**
|
| 392 |
+
|
| 393 |
+
**Severity:** P1
|
| 394 |
+
**Source:** `frontend/src/lib/useStreamingVoice.ts`
|
| 395 |
+
**Discovered:** 2026-05-15
|
| 396 |
+
|
| 397 |
+
Pre-fix voice UX waited for the full Sarvam STT round-trip before showing the user their transcribed text — felt sluggish on the chat surface. New hybrid: Web Speech API streams interim transcripts into the chat input live, while `MediaRecorder` captures the authoritative audio blob in parallel; on browser silence-detect the blob is POSTed to `/api/transcribe` (Sarvam Saarika STT) for the authoritative transcript. Auto-submit falls back to the Web Speech transcript only if Sarvam errors. KI-173 heartbeat + KI-174 `visibilitychange` / `focus` revival hooks keep the mic alive across tab and app switches.
|
| 398 |
+
|
| 399 |
+
---
|
| 400 |
+
|
| 401 |
+
### KI-171 — Skip faithfulness judge on `fact_find` + `recommendation` intents — **FIXED**
|
| 402 |
+
|
| 403 |
+
**Severity:** P1
|
| 404 |
+
**Source:** `backend/orchestrator.py`
|
| 405 |
+
**Discovered:** 2026-05-15
|
| 406 |
+
|
| 407 |
+
Faithfulness Gate 4 (LLM judge) was being run on every turn including fact-find and recommendation, where there's no retrieved-chunks context to grade the reply against — burning judge calls and occasionally blocking valid replies. Now gated behind `intent ∈ {qa, comparison}` only. Recommendation replies still go through Gates 1-3 (retrieval floor, citation integrity, regex numeric grounding).
|
| 408 |
+
|
| 409 |
+
---
|
| 410 |
+
|
| 411 |
+
### KI-175 / KI-176 — NIM chain reorder + OpenRouter re-added as cross-provider diversity — **FIXED**
|
| 412 |
+
|
| 413 |
+
**Severity:** P2
|
| 414 |
+
**Source:** Chain definitions in `backend/providers/nvidia_nim_llm.py`
|
| 415 |
+
**Discovered:** 2026-05-15
|
| 416 |
+
|
| 417 |
+
KI-175 demoted NIM Nemotron 49B from primary to last resort across all three chains — Mistral Large 3 675B and Qwen 80B consistently outperform on the live audit. KI-176 re-added OpenRouter as a cross-provider diversity tier using OR's `models: [...]` array param for server-side fallback within the OR free pool. The NIM-only lock from [ADR-038](../60-decisions/ADR-038-nim-only-chains.md) was relaxed in scope (the `<FF>` trailer convention that motivated it no longer exists post-KI-167); only OR `:free` candidates with verified native JSON mode are included (Nemotron-3-Super 120B, Qwen 80B `:free`).
|
| 418 |
+
|
| 419 |
+
---
|
| 420 |
+
|
| 421 |
+
### KI-178 / KI-179 — Google Gemini Flash added as Tier 0 primary — **FIXED** (ADR-040)
|
| 422 |
+
|
| 423 |
+
**Severity:** P2
|
| 424 |
+
**Source:** `backend/providers/google_gemini_llm.py` (new); chain definitions in `nvidia_nim_llm.py`
|
| 425 |
+
**Discovered:** 2026-05-15
|
| 426 |
+
|
| 427 |
+
Operator obtained a Google AI Studio API key. KI-178 audited which OpenRouter free-tier models support `response_format` (only Nemotron-3-Super 120B + Qwen 80B + Gemma-4 31B). KI-179 added a new `LLMProvider` against Google AI Studio's REST API (`gemini-2.0-flash` + `gemini-2.5-flash`, native JSON mode via `response_mime_type=application/json`, 1500 req/day free quota). Promoted to Tier 0 primary on Brain Fast (Gemini 2.0 Flash) + Brain Main (Gemini 2.5 Flash). NIM stays as Tier 1 fallback with Mistral 675B as the strongest non-Gemini candidate; OpenRouter sits as Tier 2 diversity. Judge stays on NIM Mistral 675B (different family from Gemini — preserves brain ↔ judge family-diversity invariant). See [ADR-040](../60-decisions/ADR-040-google-gemini-primary.md).
|
| 428 |
+
|
| 429 |
+
---
|
| 430 |
+
|
| 431 |
### KI-019 — `kb/policies/*.md` not embedded (224 human-readable summaries left out of retrieval)
|
| 432 |
|
| 433 |
**Severity:** P3
|
|
@@ -85,3 +85,45 @@ ffefacf KI-013 reinforcement — persona rules 8 + 9 for demographic-aware recs
|
|
| 85 |
|
| 86 |
Every commit pushed to BOTH `origin` (HF Space) and `github` per the
|
| 87 |
triple-mirror contract in [ADR-024](../60-decisions/ADR-024-triple-mirror-code-and-data.md).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 85 |
|
| 86 |
Every commit pushed to BOTH `origin` (HF Space) and `github` per the
|
| 87 |
triple-mirror contract in [ADR-024](../60-decisions/ADR-024-triple-mirror-code-and-data.md).
|
| 88 |
+
|
| 89 |
+
---
|
| 90 |
+
|
| 91 |
+
## Follow-on sprint — 2026-05-15 (KI-167 → KI-179)
|
| 92 |
+
|
| 93 |
+
A second-day sprint after live testing flagged that the scripted `<FF>` trailer convention was leaking into every fallback turn ("zero natural LLM chat — it always defaults to the script"). The day produced one ADR (ADR-039), one provider integration (Google AI Studio), and a chain-level rewrite (ADR-040).
|
| 94 |
+
|
| 95 |
+
### Findings & fixes by source
|
| 96 |
+
|
| 97 |
+
| KI | Severity | Issue | Resolution |
|
| 98 |
+
|---|---|---|---|
|
| 99 |
+
| KI-167 | P0 | `<FF>` trailer + `_canonical_fallback` dominated user-visible turns when LLM contract violations occurred; bot felt robotic | Ripped out `fact_find_brain.py` + canonical_fallback + scripted prompts; new `backend/sales_brain.py` with one LLM call/turn using native provider JSON mode. ADR-039. |
|
| 100 |
+
| KI-168 | P1 | Voice UX waited for full Sarvam STT round-trip before showing transcript | Hybrid Web Speech (interim) + MediaRecorder (authoritative blob → Sarvam STT on silence-detect). |
|
| 101 |
+
| KI-171 | P1 | Faithfulness judge ran on `fact_find` + `recommendation` turns where there's no retrieval context | Skip Gate 4 on those intents; Gates 1-3 still run. |
|
| 102 |
+
| KI-173 | P2 | Mic died when user switched tabs / minimised the app | Heartbeat keeps mic stream alive. |
|
| 103 |
+
| KI-174 | P2 | Mic state did not recover on `visibilitychange` / `focus` | Revival hooks reattach the mic on tab return. |
|
| 104 |
+
| KI-175 | P2 | NIM Nemotron 49B was a chain primary but consistently underperformed Mistral 675B / Qwen 80B | Demoted Nemotron 49B to last resort across all chains. |
|
| 105 |
+
| KI-176 | P2 | OpenRouter dropped from chains in ADR-038 was no longer needed after ADR-039 retired `<FF>` | Re-added OR `:free` candidates with verified JSON mode as cross-provider diversity. |
|
| 106 |
+
| KI-178 | P2 | Audit of which OR `:free` models support `response_format` | Llama 3.3 70B / Hermes 3 405B excluded (no native JSON mode); Nemotron-3-Super 120B + Qwen 80B `:free` + Gemma-4 31B included. |
|
| 107 |
+
| KI-179 | P2 | Google AI Studio key obtained; need to wire it into chains | New `backend/providers/google_gemini_llm.py`; Gemini 2.0 Flash → Brain Fast primary, Gemini 2.5 Flash → Brain Main primary. ADR-040. |
|
| 108 |
+
|
| 109 |
+
### What changed in the architecture
|
| 110 |
+
|
| 111 |
+
- **Brain Fast (sales_brain):** Gemini 2.0 Flash → NIM Qwen 80B → NIM Mistral Large 3 675B → NIM Llama-4 Maverick → OR Nemotron-3-Super 120B → OR Qwen 80B `:free` → NIM Nemotron 49B (last resort).
|
| 112 |
+
- **Brain Main:** Gemini 2.5 Flash → NIM Mistral Large 3 675B → NIM Llama-4 Maverick → NIM Qwen 80B → OR Nemotron-3-Super → NIM Nemotron 49B (last resort).
|
| 113 |
+
- **Judge:** NIM Mistral Large 3 675B → NIM Llama-4 Maverick → OR Qwen 80B `:free` → NIM Nemotron 49B (last resort).
|
| 114 |
+
|
| 115 |
+
### What was deleted
|
| 116 |
+
|
| 117 |
+
- `backend/fact_find_brain.py` (441 LOC).
|
| 118 |
+
- `_canonical_fallback`, `_normalize_for_slot`, `_pick_opener`, `_NEUTRAL_OPENERS`, `_FAMILY_OPENERS`, `_contains_self_introduction` in `backend/orchestrator.py`.
|
| 119 |
+
- The `<FF>{...}</FF>` trailer convention + lenient parser ladder + `:no_trailer` / `:empty_reply` / `:llm_error` telemetry variants from `TurnResult.brain_used`.
|
| 120 |
+
- Dead `prompt_en` / `prompt_hi` strings in `backend/needs_finder.py::GRAPH` are no longer rendered (data structure retained for the LLM's system-prompt schema).
|
| 121 |
+
|
| 122 |
+
### What stayed
|
| 123 |
+
|
| 124 |
+
- KI-080 sticky-primary election machinery (now scores Google / NIM / OpenRouter candidates uniformly).
|
| 125 |
+
- KI-084 per-phase httpx timeouts.
|
| 126 |
+
- KI-085 proactive credit gating (extended to Google quota).
|
| 127 |
+
- KI-091 / KI-094 None-guards on the profile extractor (still relevant for QA-mode turns).
|
| 128 |
+
- KI-102 / KI-107 session-isolated profile RAG.
|
| 129 |
+
- KI-106 graceful `TimeoutError` + `Exception` handling on `/api/chat`.
|
|
@@ -1,8 +1,12 @@
|
|
| 1 |
# ADR-014: Groq Llama-3.3-70B as eval grader
|
| 2 |
|
| 3 |
-
**Status:** Superseded by [ADR-019](ADR-019-nim-single-provider-consolidation.md)
|
| 4 |
**Date:** 2026-05-13
|
| 5 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 6 |
## Context
|
| 7 |
|
| 8 |
The eval harness needed a grading LLM that was NOT the same family as the brain LLM (to avoid circular evaluation). Available options:
|
|
|
|
| 1 |
# ADR-014: Groq Llama-3.3-70B as eval grader
|
| 2 |
|
| 3 |
+
**Status:** Superseded by [ADR-040](ADR-040-google-gemini-primary.md) (2026-05-15) — Groq is no longer in any chain. The judge primary is NIM Mistral Large 3 675B (different family from the Gemini brain — preserves the non-circular cross-family principle this ADR introduced). Originally superseded by [ADR-019](ADR-019-nim-single-provider-consolidation.md).
|
| 4 |
**Date:** 2026-05-13
|
| 5 |
|
| 6 |
+
## Why supersession (current)
|
| 7 |
+
|
| 8 |
+
Groq was removed from all chains in [ADR-038](ADR-038-nim-only-chains.md) (KI-160) after KI-155 demonstrated Groq Llama-3.3-70B silently ignores structured-output contracts. The brain ↔ judge family-diversity principle this ADR introduced is now expressed differently — the brain is Gemini (Google family), the judge is Mistral (Mistral family) — but the architectural intent (judge does not mark its own homework) is unchanged.
|
| 9 |
+
|
| 10 |
## Context
|
| 11 |
|
| 12 |
The eval harness needed a grading LLM that was NOT the same family as the brain LLM (to avoid circular evaluation). Available options:
|
|
@@ -1,8 +1,12 @@
|
|
| 1 |
# ADR-016: Hybrid brain router (Sarvam primary + Llama / DeepSeek fallback)
|
| 2 |
|
| 3 |
-
**Status:** Superseded by [ADR-019](ADR-019-nim-single-provider-consolidation.md)
|
| 4 |
**Date:** 2026-05-13
|
| 5 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 6 |
## Context
|
| 7 |
|
| 8 |
A single brain LLM was insufficient: Sarvam-M's 2048 output cap + `<think>` tokens caused mid-JSON truncation in extraction and mid-answer truncation in advisory. But Sarvam-first is non-negotiable narrative for the assignment.
|
|
|
|
| 1 |
# ADR-016: Hybrid brain router (Sarvam primary + Llama / DeepSeek fallback)
|
| 2 |
|
| 3 |
+
**Status:** Superseded by [ADR-040](ADR-040-google-gemini-primary.md) (2026-05-15) — the cross-provider router intent survives, but the candidate pool now leads with Google AI Studio (Gemini 2.0 / 2.5 Flash) on Brain Fast / Brain Main with NIM Mistral / Maverick / Qwen and OpenRouter `:free` as cross-provider diversity. Originally superseded by [ADR-019](ADR-019-nim-single-provider-consolidation.md).
|
| 4 |
**Date:** 2026-05-13
|
| 5 |
|
| 6 |
+
## Why superseded
|
| 7 |
+
|
| 8 |
+
The original Sarvam-primary + Llama/DeepSeek-fallback router was retired in two steps. ADR-019 first collapsed cross-provider sprawl onto a single NIM key. ADR-040 then re-opened the cross-provider surface — but on a different basis: native JSON-mode contract validation (`response_mime_type=application/json` / `response_format={"type":"json_object"}`) replaces empirical query-class routing as the candidate-selection rule. The current chain is Google → NIM → OpenRouter, scored by KI-080 probe-driven election rather than a static "Indic → Sarvam, comparison → DeepSeek" heuristic.
|
| 9 |
+
|
| 10 |
## Context
|
| 11 |
|
| 12 |
A single brain LLM was insufficient: Sarvam-M's 2048 output cap + `<think>` tokens caused mid-JSON truncation in extraction and mid-answer truncation in advisory. But Sarvam-first is non-negotiable narrative for the assignment.
|
|
@@ -1,6 +1,6 @@
|
|
| 1 |
# ADR-019: NVIDIA NIM as the single non-Sarvam provider
|
| 2 |
|
| 3 |
-
**Status:**
|
| 4 |
**Date:** 2026-05-14
|
| 5 |
|
| 6 |
**Partial supersession (D-022, 2026-05-14):** NIM's DeepSeek V4-Pro / V4-Flash and Llama-4 Maverick inference pools degraded with hours-long timeouts during a production incident. The chains were re-ordered to put **Qwen 3-Next 80B** (brain primary), **Nemotron Nano 30B** (fast brain primary), and **Mistral Large 3 675B** (judge primary) at the head, with DeepSeek + Maverick retained as fallback chain entries that re-enter rotation when those pools recover. The single-provider consolidation decision in this ADR still stands (NIM remains the primary inference provider); only the *specific model picks* were swapped. See `backend/providers/nvidia_nim_llm.py` for the current chain definitions.
|
|
|
|
| 1 |
# ADR-019: NVIDIA NIM as the single non-Sarvam provider
|
| 2 |
|
| 3 |
+
**Status:** Superseded by [ADR-040](ADR-040-google-gemini-primary.md) (2026-05-15) — the single-provider stance is retired. Google AI Studio (Gemini 2.0 / 2.5 Flash, 1500 req/day free, native JSON mode) is now Tier 0 primary on Brain Fast + Brain Main; NIM stays as Tier 1 fallback (Mistral Large 3 675B + Llama-4 Maverick + Qwen 80B) and is still the judge primary (cross-family from Gemini). OpenRouter `:free` is Tier 2 diversity. ADR-019's "cross-family judge" + "single-key per provider" + "$0 free-tier" principles are preserved across the multi-provider stack.
|
| 4 |
**Date:** 2026-05-14
|
| 5 |
|
| 6 |
**Partial supersession (D-022, 2026-05-14):** NIM's DeepSeek V4-Pro / V4-Flash and Llama-4 Maverick inference pools degraded with hours-long timeouts during a production incident. The chains were re-ordered to put **Qwen 3-Next 80B** (brain primary), **Nemotron Nano 30B** (fast brain primary), and **Mistral Large 3 675B** (judge primary) at the head, with DeepSeek + Maverick retained as fallback chain entries that re-enter rotation when those pools recover. The single-provider consolidation decision in this ADR still stands (NIM remains the primary inference provider); only the *specific model picks* were swapped. See `backend/providers/nvidia_nim_llm.py` for the current chain definitions.
|
|
@@ -1,8 +1,9 @@
|
|
| 1 |
# ADR-026 — Brain provider load-balancing (NIM ↔ Groq, 50/50)
|
| 2 |
|
| 3 |
-
**Status:**
|
|
|
|
| 4 |
**Owner:** Rohit Saraf
|
| 5 |
-
**
|
| 6 |
|
| 7 |
## Context
|
| 8 |
|
|
|
|
| 1 |
# ADR-026 — Brain provider load-balancing (NIM ↔ Groq, 50/50)
|
| 2 |
|
| 3 |
+
**Status:** Superseded by [ADR-040](ADR-040-google-gemini-primary.md) (2026-05-15) — the 50/50 random-coin load balance was first deprecated by [ADR-031](ADR-031-sticky-primary-election.md)'s probe-driven sticky-primary election (KI-080) and is now fully retired. The current chain shape (Google AI Studio Tier 0 primary → NIM Tier 1 fallback → OpenRouter `:free` Tier 2 diversity) is decided by per-call election against probe scores rather than a static probability split.
|
| 4 |
+
**Date:** 2026-05-14
|
| 5 |
**Owner:** Rohit Saraf
|
| 6 |
+
**Originally supersedes:** Partial override of [ADR-019](ADR-019-nim-single-provider-consolidation.md)'s single-provider stance
|
| 7 |
|
| 8 |
## Context
|
| 9 |
|
|
@@ -1,9 +1,14 @@
|
|
| 1 |
# ADR-030 — LLM-driven fact-find (single brain call per turn)
|
| 2 |
|
| 3 |
-
**Status:**
|
|
|
|
| 4 |
**Owner:** Rohit Saraf
|
| 5 |
-
**
|
| 6 |
-
**Related:** [ADR-022](ADR-022-conversational-profile-updates.md) (free-form profile updates — the answer-side LLM that this ADR
|
|
|
|
|
|
|
|
|
|
|
|
|
| 7 |
|
| 8 |
## Context
|
| 9 |
|
|
|
|
| 1 |
# ADR-030 — LLM-driven fact-find (single brain call per turn)
|
| 2 |
|
| 3 |
+
**Status:** Superseded by [ADR-039](ADR-039-llm-driven-sales-brain.md) (2026-05-15, KI-167) — partially superseded by [ADR-040](ADR-040-google-gemini-primary.md) (chain candidates). The "schema-aware single LLM call per turn" intuition is preserved in `backend/sales_brain.py`; the `<FF>...</FF>` trailer convention, the `_canonical_fallback` greedy slot-walker, and the scripted `Question.prompt_en` prefix are all retired in favour of native provider JSON mode (`response_mime_type=application/json` / `response_format={"type":"json_object"}`).
|
| 4 |
+
**Date:** 2026-05-15
|
| 5 |
**Owner:** Rohit Saraf
|
| 6 |
+
**Originally supersedes:** [ADR-027](ADR-027-fact-find-llm-paraphraser.md) (the LLM paraphraser was a band-aid on the hardcoded-graph stitching; this was the full replacement.)
|
| 7 |
+
**Related:** [ADR-022](ADR-022-conversational-profile-updates.md) (free-form profile updates — the answer-side LLM that this ADR generalized into the whole loop)
|
| 8 |
+
|
| 9 |
+
## Why superseded
|
| 10 |
+
|
| 11 |
+
The `<FF>{...}</FF>` trailer convention proved structurally fragile under load — see KI-155 / KI-156 / KI-158 / KI-161. Even with the KI-090 lenient parser ladder, real LLMs (Qwen, Nemotron, Mistral) regularly returned grammatically fine prose with no parseable trailer, causing `_canonical_fallback` to fire and the user to see the scripted `Question.prompt_en` of the next unfilled slot. The scripted prefix became the dominant user-facing path even when the LLM call had "succeeded." ADR-039 retires the trailer convention entirely in favour of native provider JSON mode; ADR-040 expanded the candidate pool to Google AI Studio + NIM + OpenRouter, all of which natively support server-enforced JSON mode.
|
| 12 |
|
| 13 |
## Context
|
| 14 |
|
|
@@ -1,6 +1,10 @@
|
|
| 1 |
# ADR-031 — Sticky primary election for LLM chains
|
| 2 |
|
| 3 |
-
**Status:** Superseded by [ADR-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 4 |
|
| 5 |
**Status (original):** Accepted — 2026-05-15
|
| 6 |
**Owner:** Rohit Saraf
|
|
|
|
| 1 |
# ADR-031 — Sticky primary election for LLM chains
|
| 2 |
|
| 3 |
+
**Status:** Superseded by [ADR-040](ADR-040-google-gemini-primary.md) — 2026-05-15 (KI-179) on the candidate-pool scope. The sticky-primary election machinery (per-chain probe + scored election + provider-diverse BACKUP) is preserved and now scores Google AI Studio / NIM / OpenRouter candidates uniformly. Previously superseded by [ADR-038](ADR-038-nim-only-chains.md) (KI-160) on the same scope. Body retained below for history.
|
| 4 |
+
|
| 5 |
+
## Why superseded
|
| 6 |
+
|
| 7 |
+
ADR-038 had locked the candidate pool to NIM-only after KI-155 showed cross-provider candidates silently violated the `<FF>` trailer contract. ADR-039 / KI-167 retired the `<FF>` trailer entirely in favour of native provider JSON mode, which removes the structural reason for the lock. ADR-040 then added Google AI Studio (Gemini 2.0 / 2.5 Flash) as Tier 0 primary on Brain Fast / Brain Main with NIM as Tier 1 fallback and OpenRouter `:free` as Tier 2 diversity. The election scoring (latency × success_rate) and the provider-diverse BACKUP rule are unchanged.
|
| 8 |
|
| 9 |
**Status (original):** Accepted — 2026-05-15
|
| 10 |
**Owner:** Rohit Saraf
|
|
@@ -1,12 +1,12 @@
|
|
| 1 |
# ADR-032 — LLM Chain Architecture Reference
|
| 2 |
|
| 3 |
-
**Status:** Superseded by [ADR-
|
| 4 |
|
| 5 |
-
- **BRAIN:**
|
| 6 |
-
- **
|
| 7 |
-
- **JUDGE:**
|
| 8 |
|
| 9 |
-
Body below retained as architectural reference for probe / election / timeout machinery — all still in effect
|
| 10 |
|
| 11 |
**Status (original):** Accepted — 2026-05-15
|
| 12 |
**Type:** Architecture reference (not a decision ADR)
|
|
|
|
| 1 |
# ADR-032 — LLM Chain Architecture Reference
|
| 2 |
|
| 3 |
+
**Status:** Superseded by [ADR-040](ADR-040-google-gemini-primary.md) — 2026-05-15 (KI-179) on the candidate-pool scope. The NIM-only lock from ADR-038 was relaxed once ADR-039 / KI-167 retired the `<FF>` trailer convention. Current candidate lists per ADR-040:
|
| 4 |
|
| 5 |
+
- **BRAIN (Fast — sales_brain):** Gemini 2.0 Flash (Google AI Studio Tier 0 primary), then NIM `qwen/qwen3-next-80b-a3b-instruct` → NIM `mistralai/mistral-large-3-675b-instruct-2512` → NIM `meta/llama-4-maverick-17b-128e-instruct` → OpenRouter `nvidia/nemotron-3-super-120b-a12b:free` → OpenRouter `qwen/qwen3-next-80b-a3b-instruct:free` → NIM `nvidia/llama-3.3-nemotron-super-49b-v1.5` (last resort).
|
| 6 |
+
- **BRAIN (Main — synthesis / QA / comparison / recommendation):** Gemini 2.5 Flash (Tier 0 primary), then NIM Mistral 675B → NIM Maverick → NIM Qwen 80B → OpenRouter Nemotron-3-Super → NIM Nemotron 49B (last resort).
|
| 7 |
+
- **JUDGE:** NIM `mistralai/mistral-large-3-675b-instruct-2512` (primary), NIM Maverick (backup), OpenRouter Qwen 80B `:free`, NIM Nemotron 49B (last resort).
|
| 8 |
|
| 9 |
+
Body below retained as architectural reference for probe / election / timeout / credit-gating machinery — all still in effect across the multi-provider pool. Section 4 "Election algorithm" treats Google / NIM / OpenRouter candidates uniformly via `provider_of()` for the cross-provider BACKUP rule.
|
| 10 |
|
| 11 |
**Status (original):** Accepted — 2026-05-15
|
| 12 |
**Type:** Architecture reference (not a decision ADR)
|
|
@@ -1,6 +1,8 @@
|
|
| 1 |
# ADR-038 — NIM-only chains (KI-160)
|
| 2 |
|
| 3 |
-
**Status:**
|
|
|
|
|
|
|
| 4 |
**Owner:** Rohit Saraf
|
| 5 |
**Supersedes:** [ADR-031](ADR-031-sticky-primary-election.md) (cross-provider election scope), [ADR-032](ADR-032-llm-chain-architecture.md) (candidate-pool scope)
|
| 6 |
**Related KIs:** KI-155 (Groq `<FF>` contract violation, root cause), KI-160 (this lock-down), KI-080 / KI-081 / KI-084 / KI-085 / KI-087 (probe + election mechanics, retained within NIM scope)
|
|
|
|
| 1 |
# ADR-038 — NIM-only chains (KI-160)
|
| 2 |
|
| 3 |
+
**Status:** Superseded by [ADR-040](ADR-040-google-gemini-primary.md) (2026-05-15). The NIM-only lock was relaxed once KI-167 ([ADR-039](ADR-039-llm-driven-sales-brain.md))'s sales_brain eliminated the `<FF>` trailer convention that originally motivated the lock. Native provider JSON mode (`response_format={"type":"json_object"}` on NIM, `response_mime_type=application/json` on Gemini) is now the structured-output contract, so cross-provider candidates that natively support JSON mode are safe to re-add to the election pool. The "fail-loud > fail-silent-with-garbage" principle and brain ↔ judge family-diversity invariant from this ADR are preserved in ADR-040.
|
| 4 |
+
|
| 5 |
+
**Original status:** Accepted — 2026-05-15
|
| 6 |
**Owner:** Rohit Saraf
|
| 7 |
**Supersedes:** [ADR-031](ADR-031-sticky-primary-election.md) (cross-provider election scope), [ADR-032](ADR-032-llm-chain-architecture.md) (candidate-pool scope)
|
| 8 |
**Related KIs:** KI-155 (Groq `<FF>` contract violation, root cause), KI-160 (this lock-down), KI-080 / KI-081 / KI-084 / KI-085 / KI-087 (probe + election mechanics, retained within NIM scope)
|
|
@@ -0,0 +1,90 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# ADR-040 — Google Gemini Flash as primary LLM tier (KI-179)
|
| 2 |
+
|
| 3 |
+
**Status:** Accepted — 2026-05-15
|
| 4 |
+
**Owner:** Rohit Saraf
|
| 5 |
+
**Supersedes:** [ADR-038](ADR-038-nim-only-chains.md) (NIM-only chain lock) — partially. The "fail-loud > fail-silent-with-garbage" principle and the brain ↔ judge family-diversity invariant are preserved; the strict NIM-only candidate-pool scope is relaxed now that the `<FF>` trailer convention that motivated the lock no longer exists post-[ADR-039](ADR-039-llm-driven-sales-brain.md) / KI-167.
|
| 6 |
+
**Related KIs:** KI-167 (sales_brain rip-out), KI-171 (judge skip on fact_find + recommendation), KI-175 (NIM chain reorder — nemotron demoted to last), KI-176 (OpenRouter `models: [...]` server-side fallback), KI-178 (live audit of OR free-tier JSON-mode support), KI-179 (this decision — Google AI Studio added as Tier 0).
|
| 7 |
+
|
| 8 |
+
## Context
|
| 9 |
+
|
| 10 |
+
After KI-167 ([ADR-039](ADR-039-llm-driven-sales-brain.md)) eliminated the `<FF>...</FF>` trailer convention in favour of native provider JSON mode (`response_format={"type":"json_object"}` on NIM, `response_mime_type=application/json` on Gemini), the structural reason for the NIM-only lock dissolved. KI-176 added OpenRouter back as a cross-provider fallback using OR's `models: [...]` array param for server-side fallback within the OR free pool.
|
| 11 |
+
|
| 12 |
+
KI-178 then audited which OpenRouter free-tier models actually support `response_format`:
|
| 13 |
+
|
| 14 |
+
- **No Gemini variants exist on OR's free tier.**
|
| 15 |
+
- **Llama 3.3 70B and Hermes 3 405B on OR free do not expose `response_format`** — JSON would have to be prompt-engineered, which carries a non-trivial parse-failure rate (the very class of silent failure KI-160 / ADR-038 was instituted to prevent).
|
| 16 |
+
- **The only OR free-tier models with native JSON mode** are `nvidia/nemotron-3-super-120b-a12b:free`, `qwen/qwen3-next-80b-a3b-instruct:free`, and `google/gemma-4-31b:free`.
|
| 17 |
+
|
| 18 |
+
In parallel, the operator obtained a Google AI Studio API key. Google's documented free tier on Gemini 2.0 / 2.5 Flash is **1,500 requests/day, 15 requests/minute, native JSON mode via `response_mime_type=application/json`**, with conversational quality that is genuinely better than any model in the existing NIM pool for the sales-brain free-form QA + recommendation surfaces. The free quota is adequate for the demo + early-production workload (a typical fact-find conversation is ~10 turns; 1,500 req/day supports ~150 conversations/day before falling to fallbacks).
|
| 19 |
+
|
| 20 |
+
The chain shape needs to reflect this: a frontier-quality free tier exists, and routing past it would be deliberately ignoring a strictly-better primary.
|
| 21 |
+
|
| 22 |
+
## Decision
|
| 23 |
+
|
| 24 |
+
**Add Google AI Studio as Tier 0 (primary) on the Brain Fast and Brain Main chains.** NIM stays as Tier 1 fallback with its 675B Mistral Large 3 as the strongest non-Gemini candidate. OpenRouter ($10 stays parked) sits as Tier 2 diversity pool before the nemotron-49b last resort. The judge stays NIM-primary on Mistral Large 3 675B — different family from Gemini, preserving the brain ↔ judge family-diversity invariant.
|
| 25 |
+
|
| 26 |
+
Concretely, the three chains become:
|
| 27 |
+
|
| 28 |
+
### Brain Fast (fact-find conversation, `backend/sales_brain.py`)
|
| 29 |
+
|
| 30 |
+
1. **PRIMARY — Google AI Studio:** `gemini-2.0-flash` (1500 req/day free, native JSON mode)
|
| 31 |
+
2. Fallback 1 — NIM: `qwen3-next-80b-a3b-instruct`
|
| 32 |
+
3. Fallback 2 — NIM: `mistralai/mistral-large-3-675b-instruct-2512` (675B dense)
|
| 33 |
+
4. Fallback 3 — NIM: `meta/llama-4-maverick-17b-128e-instruct` (128B MoE)
|
| 34 |
+
5. Fallback 4 — OpenRouter: `nvidia/nemotron-3-super-120b-a12b:free`
|
| 35 |
+
6. Fallback 5 — OpenRouter: `qwen/qwen3-next-80b-a3b-instruct:free`
|
| 36 |
+
7. Last resort — NIM: `nvidia/llama-3.3-nemotron-super-49b-v1.5`
|
| 37 |
+
|
| 38 |
+
### Brain Main (orchestrator free-form QA + recommendation synthesis)
|
| 39 |
+
|
| 40 |
+
1. **PRIMARY — Google AI Studio:** `gemini-2.5-flash` (same 1500 req/day quota, higher synthesis quality than 2.0 Flash on long-context recommendation generation)
|
| 41 |
+
2. Fallback 1 — NIM: `mistralai/mistral-large-3-675b-instruct-2512`
|
| 42 |
+
3. Fallback 2 — NIM: `meta/llama-4-maverick-17b-128e-instruct`
|
| 43 |
+
4. Fallback 3 — NIM: `qwen/qwen3-next-80b-a3b-instruct`
|
| 44 |
+
5. Fallback 4 — OpenRouter: `nvidia/nemotron-3-super-120b-a12b:free`
|
| 45 |
+
6. Last resort — NIM: `nvidia/llama-3.3-nemotron-super-49b-v1.5`
|
| 46 |
+
|
| 47 |
+
### Judge (faithfulness Gate 4 — KI-171 skips this on `fact_find` + `recommendation` queries)
|
| 48 |
+
|
| 49 |
+
1. **PRIMARY — NIM:** `mistralai/mistral-large-3-675b-instruct-2512` (different family from the Gemini brain; preserves cross-family invariant)
|
| 50 |
+
2. Fallback 1 — NIM: `meta/llama-4-maverick-17b-128e-instruct`
|
| 51 |
+
3. Fallback 2 — OpenRouter: `qwen/qwen3-next-80b-a3b-instruct:free`
|
| 52 |
+
4. Last resort — NIM: `nvidia/llama-3.3-nemotron-super-49b-v1.5`
|
| 53 |
+
|
| 54 |
+
A new `backend/providers/google_gemini_llm.py` wrapper (KI-179) implements the `LLMProvider` interface against Google AI Studio's REST API, matching the same `chat()` signature as `NvidiaNimLLM` and `OpenRouterLLM`. `NimChainLLM` is provider-agnostic — chains may now mix Google / NIM / OpenRouter candidates by URL.
|
| 55 |
+
|
| 56 |
+
## Consequences
|
| 57 |
+
|
| 58 |
+
### Positive
|
| 59 |
+
|
| 60 |
+
- **Frontier-tier conversational quality on the free path.** Gemini 2.0 Flash + 2.5 Flash are genuinely best-in-class for the sales-brain surface (one-shot JSON-mode replies with tight schemas + natural prose) and the recommendation-synthesis surface (multi-policy comparison + ranked shortlist).
|
| 61 |
+
- **Native JSON mode preserved.** `response_mime_type=application/json` is Google's server-enforced equivalent of NIM's `response_format={"type":"json_object"}` — the [ADR-039](ADR-039-llm-driven-sales-brain.md) structured-output guarantee survives the provider swap.
|
| 62 |
+
- **NIM's massive Mistral Large 3 (675B dense) stays accessible as Fallback 1** on both Brain chains. The judge chain still leads with Mistral 675B, so the rescue-by-judge path remains identical.
|
| 63 |
+
- **OpenRouter $10 diversity pool stays valuable.** The two OR `:free` candidates with verified JSON mode (nemotron-3-super-120b, qwen3-next-80b) cover a third-provider tier under both Google and NIM, so a simultaneous Google + NIM degradation has at least one diverse candidate to elect.
|
| 64 |
+
- **Family diversity preserved at the judge boundary.** Brain primaries are Gemini (Google family); judge primary is Mistral Large 3 675B (Mistral family) — the brain ↔ judge family diversity that [ADR-014](ADR-014-groq-llama-grader.md) / [ADR-032](ADR-032-llm-chain-architecture.md) require is intact.
|
| 65 |
+
|
| 66 |
+
### Negative
|
| 67 |
+
|
| 68 |
+
- **One additional dependency** — Google AI Studio key (`GOOGLE_API_KEY`). The provider list grows from {NIM, OpenRouter, Sarvam} to {Google, NIM, OpenRouter, Sarvam}.
|
| 69 |
+
- **Free-tier quota cap of 1500 req/day on Gemini Flash.** Adequate for demo + early production but not unbounded. A sustained 100+ concurrent-conversation load would exhaust the quota and force fallback to NIM Mistral 675B.
|
| 70 |
+
|
| 71 |
+
### Mitigations
|
| 72 |
+
|
| 73 |
+
- **6-level fallback chain on Brain Fast, 5-level on Brain Main.** A Google AI Studio outage / quota exhaustion / 429 simply falls to NIM Mistral 675B — the same provider family that has been production-validated since KI-160.
|
| 74 |
+
- **Quota visibility.** Daily request counter is visible at [aistudio.google.com](https://aistudio.google.com); admin telemetry (KI-086 `/api/admin/llm-health`) surfaces per-candidate health + degraded-until timestamps, so quota-exhaustion is observable in the existing admin panel.
|
| 75 |
+
- **Key rotation procedure exists.** Google AI Studio keys can be rotated in-console; the `GOOGLE_API_KEY` env in HF Space secrets is one-flip to swap.
|
| 76 |
+
- **Probe loop (KI-080 + KI-088 NIM semaphore) extends naturally** to the Google provider — `backend/llm_health.py` scores every candidate uniformly on `(1 / max(50, latency_ms)) * success_rate`; the elector treats provider-of-origin as opaque.
|
| 77 |
+
|
| 78 |
+
## Alternatives considered
|
| 79 |
+
|
| 80 |
+
- **Ship Llama 3.3 70B / Hermes 3 405B from OpenRouter free tier as the new primary.** Rejected. KI-178 confirmed neither model exposes `response_format` on OR free — JSON would have to be prompt-engineered out of prose, reintroducing the class of silent failure that KI-160 / [ADR-038](ADR-038-nim-only-chains.md) was instituted to prevent. The point of native JSON mode is server-side enforcement; a model without it is structurally worse for our use case regardless of its parameter count.
|
| 81 |
+
|
| 82 |
+
- **Stay NIM-only (KI-175 state — nemotron demoted to last, Mistral 675B as new NIM primary).** Rejected. Live testing showed Gemini Flash conversational quality is genuinely better on the sales-brain surface (more natural opener variety, less repetition, fewer "Got that —" style filler patterns), and the 1500 req/day Google free tier is plentiful for the workload. Routing past a strictly-better free primary would be deliberately ignoring it.
|
| 83 |
+
|
| 84 |
+
- **Buy paid Gemini via OpenRouter's $10 balance.** Rejected. OR's paid Gemini is ~1000–2000 conversations before the $10 refill is needed; direct Google AI Studio is genuinely free at the same quality level. The OR $10 is better spent on the diversity-pool free models (nemotron-3-super-120b, qwen3-next-80b) that round out the fallback chain.
|
| 85 |
+
|
| 86 |
+
- **Make Gemini the judge too (same family across all roles).** Rejected. Brain ↔ judge family diversity ([ADR-014](ADR-014-groq-llama-grader.md) / [ADR-032](ADR-032-llm-chain-architecture.md)) is the structural defence against circular grading — the judge would mark its own homework. Keeping the judge on Mistral Large 3 675B (Mistral family) preserves the non-circular grading invariant when the brain is Gemini.
|
| 87 |
+
|
| 88 |
+
## Reversal trigger
|
| 89 |
+
|
| 90 |
+
Re-open this ADR if (a) Google AI Studio imposes a paid-tier requirement or removes `response_mime_type` support, (b) the 1500 req/day quota is consistently exhausted before noon UTC over a 7-day window (signalling we need either a paid tier or a different primary), (c) live audit shows Gemini Flash silently violating the JSON-mode contract on any sales-brain or recommendation turn (the failure class KI-160 originally locked NIM against), or (d) the brain ↔ judge family-diversity invariant breaks (e.g. Google ships a judge-grade reasoning model and we're tempted to consolidate). Reversal would shift the Brain primaries back to NIM Mistral 675B with OpenRouter as Tier 1 — i.e. the KI-176 state — not the strict NIM-only lock of KI-160 (the `<FF>` trailer convention that motivated that lock is permanently retired).
|
|
@@ -1,7 +1,7 @@
|
|
| 1 |
# Enterprise-Grade Readiness Audit
|
| 2 |
|
| 3 |
**Target deployment:** top-tier Indian insurance companies (HDFC ERGO, ICICI Lombard, Bajaj Allianz, Star Health, Tata AIG, etc.)
|
| 4 |
-
**Audit date:** 2026-05-14
|
| 5 |
**Audit window:** active — this is a living document. Background audits (100-persona simulation + full gold-QA eval) running concurrently; results merge here as they land.
|
| 6 |
|
| 7 |
---
|
|
@@ -19,8 +19,9 @@
|
|
| 19 |
| Language-handling fairness | ⚠️ Hinglish concern was a 20-persona sampling artifact. Real outliers: tax_planner archetype (4.6 refusals) + stream style (4.3) — open. | P1 — India-market regulatory risk |
|
| 20 |
| Code hygiene | ✅ Loose tmp files removed; `fact_find_normalizer` + `profile_extractor` migrated to chain pattern (KI-033) | — |
|
| 21 |
| Test coverage | 🟡 15 unit tests pinning KI-018 / KI-023 / KI-080 routing + primary-election invariants. Broader coverage still open. | P1 |
|
| 22 |
-
| Voice UX | ✅ Live default-on + clickable toggle + labeled push-to-talk + barge-in working (ADR-028) | — |
|
| 23 |
-
| Fact-find robotic tone | ✅
|
|
|
|
| 24 |
| Secrets handling | ✅ Verified clean (.env never committed) | — |
|
| 25 |
|
| 26 |
Legend: ✅ fixed / ⚠️ partial / 🟡 improving / 🔴 open
|
|
@@ -336,4 +337,21 @@ This audit so far covers items 1-3, 5 (in progress), and 6. Items 7-10 require a
|
|
| 336 |
|
| 337 |
---
|
| 338 |
|
| 339 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
# Enterprise-Grade Readiness Audit
|
| 2 |
|
| 3 |
**Target deployment:** top-tier Indian insurance companies (HDFC ERGO, ICICI Lombard, Bajaj Allianz, Star Health, Tata AIG, etc.)
|
| 4 |
+
**Audit date:** 2026-05-14 (initial) · 2026-05-15 (follow-on sprint KI-167 → KI-179 landed)
|
| 5 |
**Audit window:** active — this is a living document. Background audits (100-persona simulation + full gold-QA eval) running concurrently; results merge here as they land.
|
| 6 |
|
| 7 |
---
|
|
|
|
| 19 |
| Language-handling fairness | ⚠️ Hinglish concern was a 20-persona sampling artifact. Real outliers: tax_planner archetype (4.6 refusals) + stream style (4.3) — open. | P1 — India-market regulatory risk |
|
| 20 |
| Code hygiene | ✅ Loose tmp files removed; `fact_find_normalizer` + `profile_extractor` migrated to chain pattern (KI-033) | — |
|
| 21 |
| Test coverage | 🟡 15 unit tests pinning KI-018 / KI-023 / KI-080 routing + primary-election invariants. Broader coverage still open. | P1 |
|
| 22 |
+
| Voice UX | ✅ Live default-on + clickable toggle + labeled push-to-talk + barge-in working (ADR-028) + hybrid Web Speech + MediaRecorder capture (KI-168) + tab-switch revival (KI-173 / KI-174) | — |
|
| 23 |
+
| Fact-find robotic tone | ✅ Replaced with `sales_brain` LLM-driven loop ([ADR-039](../70-docs/60-decisions/ADR-039-llm-driven-sales-brain.md), KI-167) — no scripted prompts, no `<FF>` trailer, native provider JSON mode | — |
|
| 24 |
+
| LLM stack | ✅ 3-tier chain Google → NIM → OpenRouter per [ADR-040](../70-docs/60-decisions/ADR-040-google-gemini-primary.md). Gemini 2.0/2.5 Flash primary; NIM Mistral 675B fallback; OR `:free` diversity. Judge stays NIM Mistral 675B (cross-family from Gemini). | — |
|
| 25 |
| Secrets handling | ✅ Verified clean (.env never committed) | — |
|
| 26 |
|
| 27 |
Legend: ✅ fixed / ⚠️ partial / 🟡 improving / 🔴 open
|
|
|
|
| 337 |
|
| 338 |
---
|
| 339 |
|
| 340 |
+
## Sprint 2026-05-15 (KI-167 → KI-179) — chain rewrite + sales_brain rip-out
|
| 341 |
+
|
| 342 |
+
Second-day follow-on sprint after live user testing flagged that the scripted `<FF>` trailer convention was leaking into every fallback turn. Full timeline in [`70-docs/40-evaluation/quality-sprint-2026-05-14.md`](../70-docs/40-evaluation/quality-sprint-2026-05-14.md) (Follow-on sprint section). Highlights against the register:
|
| 343 |
+
|
| 344 |
+
| KI | Impact on register | Note |
|
| 345 |
+
|---|---|---|
|
| 346 |
+
| **KI-167** ([ADR-039](../70-docs/60-decisions/ADR-039-llm-driven-sales-brain.md)) | Closes the "scripted fact-find leakage" defect at root — `backend/fact_find_brain.py` + `_canonical_fallback` + the `<FF>` trailer convention all deleted. Replaced with `backend/sales_brain.py` (one LLM call per turn, native provider JSON mode). KI-090 lenient parser, KI-103 no_trailer loop-breaker, and the entire `fact_find_brain::fallback:*` telemetry vocabulary are retired. | Net negative LOC. |
|
| 347 |
+
| **KI-168 / KI-173 / KI-174** | Voice UX hardening — hybrid Web Speech (live UX) + MediaRecorder (authoritative blob → Sarvam STT) + tab-switch revival hooks. | Frontend-only. |
|
| 348 |
+
| **KI-171** | Skip faithfulness Gate 4 on `fact_find` + `recommendation` intents (no retrieval context to grade against). | Tightens D-006 fairness risk surface. |
|
| 349 |
+
| **KI-175 / KI-176** | NIM chain reorder — Nemotron 49B demoted from primary to last resort; OpenRouter re-added as cross-provider diversity (using OR's `models: [...]` server-side fallback). | Mitigates D-004 latency long-tail. |
|
| 350 |
+
| **KI-178** | Live audit of OR `:free` `response_format` support — Llama 3.3 70B / Hermes 3 405B excluded; only Nemotron-3-Super 120B + Qwen 80B `:free` + Gemma-4 31B included. | Closes a silent-failure class proactively. |
|
| 351 |
+
| **KI-179** ([ADR-040](../70-docs/60-decisions/ADR-040-google-gemini-primary.md)) | New `backend/providers/google_gemini_llm.py`; Gemini 2.0 Flash → Brain Fast primary, Gemini 2.5 Flash → Brain Main primary. NIM Tier 1 fallback, OR `:free` Tier 2 diversity. Judge stays on NIM Mistral 675B (different family from Gemini — preserves brain ↔ judge family-diversity invariant). | Tier 0 free-tier conversational quality lift. |
|
| 352 |
+
|
| 353 |
+
The 2026-05-14 scoreboard rows for "Voice UX", "Fact-find robotic tone", and the new "LLM stack" row are now fully green. D-003 (factual accuracy) and D-004 (latency) remain partially open pending a fresh full-eval re-run on the new chain shape.
|
| 354 |
+
|
| 355 |
+
---
|
| 356 |
+
|
| 357 |
+
*This file regenerates as new evidence lands. Last updated: 2026-05-15 (sprint KI-167 → KI-179 landed; ADR-039 + ADR-040 ship).*
|
|
@@ -1,50 +1,95 @@
|
|
| 1 |
-
Insurance Sales Portfolio Expert
|
| 2 |
-
|
| 3 |
-
|
| 4 |
-
|
| 5 |
-
|
| 6 |
-
|
| 7 |
-
|
| 8 |
-
|
| 9 |
-
|
| 10 |
-
|
| 11 |
-
|
| 12 |
-
|
| 13 |
-
- **
|
| 14 |
-
|
| 15 |
-
|
| 16 |
-
|
| 17 |
-
|
| 18 |
-
|
| 19 |
-
-
|
| 20 |
-
-
|
| 21 |
-
|
| 22 |
-
|
| 23 |
-
|
| 24 |
-
|
| 25 |
-
|
| 26 |
-
|
| 27 |
-
|
| 28 |
-
-
|
| 29 |
-
|
| 30 |
-
|
| 31 |
-
|
| 32 |
-
-
|
| 33 |
-
|
| 34 |
-
|
| 35 |
-
|
| 36 |
-
|
| 37 |
-
|
| 38 |
-
|
| 39 |
-
---
|
| 40 |
-
|
| 41 |
-
|
| 42 |
-
|
| 43 |
-
|
| 44 |
-
|
| 45 |
-
|
| 46 |
-
|
| 47 |
-
|
| 48 |
-
|
| 49 |
-
|
| 50 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Insurance Sales Portfolio Expert — Architecture & Design
|
| 2 |
+
|
| 3 |
+
A condensed, current architecture map of the Insurance Sales AI Agent. For
|
| 4 |
+
authoritative system-level detail see [`70-docs/10-architecture/system-overview.md`](70-docs/10-architecture/system-overview.md);
|
| 5 |
+
for routing + chain mechanics see
|
| 6 |
+
[`70-docs/60-decisions/ADR-040-google-gemini-primary.md`](70-docs/60-decisions/ADR-040-google-gemini-primary.md).
|
| 7 |
+
|
| 8 |
+
---
|
| 9 |
+
|
| 10 |
+
## System architecture
|
| 11 |
+
|
| 12 |
+
### 1. Frontend
|
| 13 |
+
- **Technology:** Next.js 14 (App Router) + Tailwind v4 + shadcn/ui — deployed
|
| 14 |
+
to the HF Space static surface.
|
| 15 |
+
- **Voice capture (post-KI-168 hybrid):** Web Speech API streams interim
|
| 16 |
+
transcripts into the chat input for live UX feel, while `MediaRecorder` runs
|
| 17 |
+
in parallel and captures the authoritative audio blob. On browser
|
| 18 |
+
silence-detect the blob is POSTed to `/api/transcribe` (Sarvam Saarika STT,
|
| 19 |
+
the authoritative transcript); auto-submit then falls back to the Web Speech
|
| 20 |
+
transcript only if Sarvam errors. KI-173 heartbeat + KI-174
|
| 21 |
+
`visibilitychange` / `focus` revival keep the mic alive across tab and app
|
| 22 |
+
switches.
|
| 23 |
+
|
| 24 |
+
### 2. Backend (FastAPI orchestrator)
|
| 25 |
+
- **Technology:** FastAPI + Pydantic. Single per-turn entry `POST /api/chat`
|
| 26 |
+
fans out through `backend/orchestrator.py`.
|
| 27 |
+
- **Sales brain (`backend/sales_brain.py`, KI-167):** one LLM call per
|
| 28 |
+
fact-find turn against `FAST_BRAIN_CHAIN`. Uses native provider JSON mode
|
| 29 |
+
(`response_mime_type=application/json` on Gemini,
|
| 30 |
+
`response_format={"type":"json_object"}` on NIM / OpenRouter) — no `<FF>`
|
| 31 |
+
trailer convention, no scripted canonical fallback. See
|
| 32 |
+
[ADR-039](70-docs/60-decisions/ADR-039-llm-driven-sales-brain.md).
|
| 33 |
+
- **Faithfulness:** 4-gate guard in `backend/faithfulness.py` (retrieval floor
|
| 34 |
+
→ citation integrity → regex numeric grounding → cross-family judge).
|
| 35 |
+
Skipped on `fact_find` + `recommendation` intents per KI-171.
|
| 36 |
+
|
| 37 |
+
### 3. LLM chains (3-tier, post-ADR-040)
|
| 38 |
+
| Role | Tier 0 (primary) | Tier 1 (NIM fallback) | Tier 2 (OR diversity) | Last resort |
|
| 39 |
+
|---|---|---|---|---|
|
| 40 |
+
| Brain Fast (sales_brain) | Gemini 2.0 Flash | Qwen 3-Next 80B → Mistral Large 3 675B → Llama-4 Maverick | OR Nemotron-3-Super 120B → OR Qwen 80B `:free` | NIM Nemotron 49B |
|
| 41 |
+
| Brain Main (synthesis) | Gemini 2.5 Flash | Mistral Large 3 675B → Llama-4 Maverick → Qwen 80B | OR Nemotron-3-Super 120B | NIM Nemotron 49B |
|
| 42 |
+
| Judge (faithfulness Gate 4) | NIM Mistral Large 3 675B | Llama-4 Maverick | OR Qwen 80B `:free` | NIM Nemotron 49B |
|
| 43 |
+
|
| 44 |
+
Provider keys: `GOOGLE_API_KEY` (1500 req/day free), `NVIDIA_NIM_API_KEY`
|
| 45 |
+
(free, 40 req/min), `OPENROUTER_API_KEY` ($10 unlocks 1000 req/day on `:free`
|
| 46 |
+
models). Brain ↔ judge family-diversity invariant is preserved — Gemini-brain
|
| 47 |
+
calls are graded by Mistral.
|
| 48 |
+
|
| 49 |
+
### 4. Retrieval & data
|
| 50 |
+
- **Vector store:** Chroma persistent client at `rag/vectors/` (BGE-small-en-v1.5
|
| 51 |
+
local 384-d embeddings — see [ADR-011](70-docs/60-decisions/ADR-011-bge-local-embeddings.md)).
|
| 52 |
+
- **Structured store:** DuckDB rollup of 62-field `HealthPolicy` extractions
|
| 53 |
+
(`rag/policies.duckdb`); the source-of-truth JSONs live in
|
| 54 |
+
`rag/extracted/<policy_id>.json` and curated facts in `40-data/policy_facts/`.
|
| 55 |
+
- **Profile RAG:** session profile is embedded as a Chroma chunk with
|
| 56 |
+
`session_id` metadata; KI-102 enforces strict session isolation on lookup.
|
| 57 |
+
|
| 58 |
+
### 5. Voice pipeline
|
| 59 |
+
- **STT:** Sarvam Saarika v2.5 (`/api/transcribe`).
|
| 60 |
+
- **TTS:** Sarvam Bulbul v2 — bot replies normalised by
|
| 61 |
+
`backend/voice_format.py::tts_preprocess` (markdown stripped, CoT leakage
|
| 62 |
+
killed per KI-104, money shorthand expanded per KI-066).
|
| 63 |
+
|
| 64 |
+
---
|
| 65 |
+
|
| 66 |
+
## How a single turn flows
|
| 67 |
+
|
| 68 |
+
1. **Voice input** — browser captures audio with `MediaRecorder` while Web
|
| 69 |
+
Speech provides interim text for the UI.
|
| 70 |
+
2. **Transcription** — silence-detect POSTs the blob to `/api/transcribe`
|
| 71 |
+
(Sarvam Saarika STT); Web Speech transcript is the fallback.
|
| 72 |
+
3. **Chat call** — `POST /api/chat` runs `classify_intent`,
|
| 73 |
+
profile-RAG injection, and dispatches to `sales_brain` (fact-find) or to
|
| 74 |
+
retrieval + Brain Main (QA / comparison / recommendation).
|
| 75 |
+
4. **Brain call** — sticky-primary election (KI-080) picks one candidate from
|
| 76 |
+
the relevant chain; native JSON mode guarantees parseable output.
|
| 77 |
+
5. **Faithfulness gates** — retrieval floor + citation integrity + regex
|
| 78 |
+
numeric grounding + (cross-family) judge LLM. Blocked replies land in
|
| 79 |
+
`logs/hallucinations.jsonl`.
|
| 80 |
+
6. **TTS synthesis** — Sarvam Bulbul returns base64 audio in the same JSON
|
| 81 |
+
response; frontend plays it through the in-DOM `<audio>` element so
|
| 82 |
+
barge-in's `querySelectorAll("audio").pause()` actually finds it.
|
| 83 |
+
7. **Telemetry** — every turn writes a JSONL record (intent, brain_used,
|
| 84 |
+
latency, faithfulness verdict, cost estimate) for replay + audit.
|
| 85 |
+
|
| 86 |
+
---
|
| 87 |
+
|
| 88 |
+
## Where to read next
|
| 89 |
+
|
| 90 |
+
- [`70-docs/10-architecture/system-overview.md`](70-docs/10-architecture/system-overview.md) — full system map with the canonical data flow.
|
| 91 |
+
- [`70-docs/10-architecture/stack-rationale.md`](70-docs/10-architecture/stack-rationale.md) — every technology pick with alternatives + reasoning.
|
| 92 |
+
- [`70-docs/60-decisions/ADR-040-google-gemini-primary.md`](70-docs/60-decisions/ADR-040-google-gemini-primary.md) — the current chain shape.
|
| 93 |
+
- [`70-docs/60-decisions/ADR-039-llm-driven-sales-brain.md`](70-docs/60-decisions/ADR-039-llm-driven-sales-brain.md) — fact-find LLM-driven sales brain.
|
| 94 |
+
- [`backend/README.md`](backend/README.md) — per-module entry points.
|
| 95 |
+
- [`80-audit/ENTERPRISE_AUDIT.md`](80-audit/ENTERPRISE_AUDIT.md) — live defect register + production-readiness scorecard.
|
|
@@ -18,19 +18,23 @@ This file is read by Claude Code (and any compatible AI tool) at the start of a
|
|
| 18 |
- **Hands-free was removed entirely** in KI-027. Anything in the codebase still referring to it is stale.
|
| 19 |
- **Bot TTS plays via the in-DOM `<audio>` element** inside `Message` (autoplay-on-mount via ref'd `useEffect`). Never use `new Audio(url).play()` — those detached instances are invisible to `document.querySelectorAll("audio").pause()` in the barge-in handler.
|
| 20 |
|
| 21 |
-
## LLM stack (ADR-019 + ADR-026 → ADR-031 + ADR-032 → ADR-038) — KI-080 → KI-087, KI-160
|
| 22 |
|
| 23 |
-
Every LLM role is a `NimChainLLM` candidate pool, NOT a hardcoded single model. End-to-end spec: [ADR-032](70-docs/60-decisions/ADR-032-llm-chain-architecture.md).
|
| 24 |
|
| 25 |
-
- **
|
|
|
|
|
|
|
|
|
|
|
|
|
| 26 |
- **Probe-driven sticky primary election (KI-080, [ADR-031](70-docs/60-decisions/ADR-031-sticky-primary-election.md), superseded by ADR-038 for candidate-pool scope).** All three chains elect a sticky PRIMARY + BACKUP from a background probe within the NIM pool. `backend/llm_health.py` scores every candidate on `(1 / max(50, latency_ms)) * success_rate` and writes the current election to process state. `NimChainLLM.chat()` calls PRIMARY once; on real-time failure it falls to BACKUP and triggers an immediate probe refresh. **Per-turn LLM call count: 1 (most cases) or 2 (PRIMARY fails real-time → BACKUP).**
|
| 27 |
- **Probe cadence + per-phase timeouts (KI-084, `119e0fd`).** Probe loop ticks at `PROBE_INTERVAL_SEC = 300s`. Probe `max_tokens` cut `5 → 1`. Every chat call uses explicit `httpx.Timeout(connect=2, read=12, write=2, pool=2)` so a stuck NIM pool releases its TCP socket independently of the outer `asyncio.wait_for`. Rate-limit failures (HTTP 429 / `RateLimit` body) get a **1h sin-bin** (`DEGRADE_DURATION_LONG_S = 3600s`).
|
| 28 |
- **Proactive credit gating (KI-085, `8fc7979`), now NIM-only scope.** Election is gated by `is_alive AND has_credits` so quota-exhausted NIM models are excluded BEFORE the user hits a 429. Signal source within the NIM pool: per-model local 60-second rate-meter (gate at 35-of-40 req/min, headroom 5). Per-model rate-metering applies within the locked NIM pool.
|
| 29 |
-
- **HF Space secrets (KI-081).** `
|
| 30 |
- **Admin telemetry (KI-086, `d90f8c0`).** `GET /api/admin/llm-health` returns `{chains, candidates, recent_turns, snapshot_ts}` with per-chain elected primary/backup, per-candidate health + credits + degraded-until, and last 20 turn outcomes. Admin "LLM Chain" tab auto-refreshes every 30s and renders whatever the chain config exposes (NIM-only post-KI-160).
|
| 31 |
- **KI-025's 50/50 NIM ↔ Groq rotation ([ADR-026](70-docs/60-decisions/ADR-026-provider-load-balancing.md)) is deprecated** — `_balanced_brain_chain` retained behind a feature flag for one-release rollback; the probe-driven NIM-only election picks the actually-faster candidate dynamically.
|
| 32 |
- **Cold-start fallback.** Before the first probe completes (process restart, HF Space rebuild), `chain[0]` is the initial primary and `chain[1]` is the initial backup, both NIM.
|
| 33 |
-
- **Brain / fast-brain / judge primaries in steady state** are **
|
| 34 |
- **KI-079 escalation as last bite (`87ee522`) — modified by KI-167.** If both PRIMARY and BACKUP fail in a single fact-find turn, orchestrator retries once on `BRAIN_CHAIN` (heavy brain, `_TIMEOUT_S_ESCALATION = 15s`, 35s chain budget). Post-KI-167 ([ADR-039](70-docs/60-decisions/ADR-039-llm-driven-sales-brain.md)) the `_canonical_fallback` terminal step is gone — on heavy-brain exhaustion the orchestrator returns the [ADR-038](70-docs/60-decisions/ADR-038-nim-only-chains.md) graceful error message instead. Worst-case wall-clock before graceful error: 25s FAST + 15s heavy = 40s.
|
| 35 |
- **NIM concurrency semaphore + serial probe (KI-088, `14ee008`).** Module-level `asyncio.Semaphore(2)` wraps every NIM HTTP call so our process never has >2 NIM requests in flight simultaneously, regardless of source (probe loop + admin polls + per-user turns all serialise through the same semaphore). Probe loop changed parallel→serial so the 6-NIM probe burst becomes a 1-slot trickle over ~12s. Inner 4-attempt exponential-backoff retry deleted from `NvidiaNimLLM.chat()` — KI-080 election + KI-079 escalation now handle failover. Result: latency-based failures (41s timeouts under self-saturation) dropped to zero; replaced by a parser-side bottleneck (KI-090).
|
| 36 |
- **Lenient FF-block parser (KI-090, `11cf4b3`) — RETIRED by KI-167 ([ADR-039](70-docs/60-decisions/ADR-039-llm-driven-sales-brain.md)).** Was needed because the `<FF>...</FF>` trailer convention was fragile under load. Post-KI-167 the sales brain uses NIM `response_format={"type":"json_object"}` for guaranteed structured output, so the whole `_parse_ff_block` strict → fenced → bare-JSON ladder is gone.
|
|
@@ -47,18 +51,18 @@ Every LLM role is a `NimChainLLM` candidate pool, NOT a hardcoded single model.
|
|
| 47 |
- **Chroma collection re-ingested + profile-write hardening (KI-112).** KI-111 wrapped `.query()` so the bot survived the corruption, but every embedding query was raising `InternalError: Error executing plan: Internal error: Error finding id` and silently returning empty retrieval — the bot was answering 206 policies' worth of Qs without access to any policy chunk. Root cause: a pre-KI-102 deploy wrote a `profile_anonymous` chunk with NO `session_id` metadata; that legacy row poisoned every later `coll.query(where={"doc_type": {"$ne": "profile"}})` and the damage spread across HNSW segments (full collection extraction surfaced 1580 / 7356 chunks across 148 policies as `Error getting embedding`). Fix: full re-ingest from `rag/corpus/` PDFs → clean `rag/vectors/` + two new write-time guards in `backend/profile_rag.py::upsert_profile_chunk` — (a) reject `session_id` that isn't a non-empty `str`, (b) reject any embedding whose length ≠ `embedder.dimension` or that contains `None`. Both guards log a `WARNING` and return without writing, so a future model-drift or bad-input event can't re-poison HNSW. 4 new regression tests in `tests/test_profile_rag_isolation.py::TestUpsertRejectsBadInputs`. Repaired vectors uploaded to HF dataset `rohitsar567/insurance-bot-data` via `tools/upload_vectors_to_dataset.py` so the Space rebuild picks up the clean index. Old corrupted Chroma archived at `rag/_hf_dataset_backup/rag/vectors.corrupted.<ts>/`.
|
| 48 |
- **Chain budgets:** brain 20s × 35s total, fast-brain 12s × 22s total, judge 30s × 75s total. With KI-080 only PRIMARY + BACKUP consume budget in the common case — leaves headroom for KI-079 escalation. KI-084 per-phase httpx timeouts are nested inside these budgets.
|
| 49 |
- **STT/TTS/Translator** = Sarvam (Saarika v2.5 / Bulbul v2 / Sarvam-M). **Embeddings** = local BGE-small-en-v1.5.
|
| 50 |
-
- **Provider keys.** `
|
| 51 |
|
| 52 |
-
## Fact-find loop (ADR-039, supersedes ADR-030 / ADR-027) — KI-167
|
| 53 |
|
| 54 |
-
**One LLM call per turn, structured-output JSON, no scripted prompts, no fallback prose.** KI-167 ripped out `backend/fact_find_brain.py` (the `<FF>{...}</FF>` trailer convention + `_canonical_fallback` greedy slot-walker + scripted `Question.prompt_en` prefixed with `"Got that — {slot}."`) and replaced it with `backend/sales_brain.py` — a single LLM-driven sales agent. The pre-KI-167 architecture had accreted eight patches (KI-090 / KI-091 / KI-094 / KI-103 / KI-150 / KI-155 / KI-156 / KI-158 / KI-161) working around the fragility of pattern-matching structure out of prose; KI-155 and follow-ups showed the canonical fallback was the *dominant* path, not the safety net. User-visible symptom: "zero natural LLM chat — it always defaults to the script."
|
| 55 |
|
| 56 |
-
- **Single
|
| 57 |
- **System prompt** carries the 9-slot schema + current profile state. The LLM is free to ask in any order, in any voice, multi-fact in one turn or one slot at a time. No scripted opener, no acknowledger prefix, no `prompt_en` template.
|
| 58 |
- **Deterministic post-processor** (`backend/sales_brain_normalizer.py`) takes the LLM's loose `captures` dict and emits a `{canonical_field: validated_value}` map: alias resolution (`location` → `location_tier`), enum normalization (`Bangalore` → `metro`), INR-amount parsing, null/empty drop, type/bounds validation. No LLM calls — pure rules. Override: if LLM sets `complete: true` while any required slot is empty, force `complete: false`.
|
| 59 |
- **Profile persistence unchanged.** Captures flow through `session.update_profile_field()` → `backend/profile_store.save_profile()` (disk durability) → `backend/profile_rag.upsert_profile_chunk()` (retrieval-side visibility). KI-102 / KI-107 / KI-112 session-isolation guards remain in force.
|
| 60 |
- **No scripted prompts, no canonical_fallback, no trailer convention.** `backend/needs_finder.py::GRAPH` slot-id data stays as a schema source for the system prompt, but `Question.prompt_en` is dead text — never consulted by the fact-find branch. `_canonical_fallback`, `_normalize_for_slot`, `_pick_opener`, `_NEUTRAL_OPENERS`, `_FAMILY_OPENERS`, `_contains_self_introduction`, the lenient `<FF>` parser ladder, and the `"Got that — {slot}."` prefix logic are deleted.
|
| 61 |
-
- **Outer 25s `asyncio.wait_for` ceiling retained.** On total
|
| 62 |
- **DELETED in KI-167:** `backend/fact_find_brain.py` (441 LOC); `_canonical_fallback` + `_pick_opener` branches in `backend/orchestrator.py`; `_ff_failed_attempts` / `_ff_skipped_slots` session fields; every `fact_find_brain::fallback:*` telemetry variant; the lenient `<FF>` / fenced-`json` / bare-JSON-tail parser ladder.
|
| 63 |
- **Persona-audit fixtures rewritten** to assert on final captured state (which slots are filled with what) rather than per-turn slot order — the LLM may capture in different turn order than the pre-KI-167 rules engine did. See WS4.
|
| 64 |
- **Natural-conversation escape (KI-045):** intent_change phrases / off-topic questions still exit fact-find by routing through `should_route_to_fact_find` upstream of the brain.
|
|
@@ -133,8 +137,9 @@ Three independent safety layers against ChromaDB HNSW bloat:
|
|
| 133 |
- **The `indusind-general` slug did not exist anywhere in the codebase before 2026-05-15.** Reliance General Insurance was rebranded to IndusInd General; **KI-144** migrated insurer slug + policy IDs + Chroma metadata + marketplace alias mapping. Any code referencing `reliance-general` should either be retained as a legacy alias (one card remains under `reliance-general` for back-compat) or migrated to `indusind-general`. Do not silently merge the two — they're tracked as separate slugs.
|
| 134 |
- **Voice mode now defaults OFF (KI-131 / KI-134 / KI-139 / KI-148).** The Live pill renders red by default; the user must opt in. AudioContext.resume() is required to unlock TTS autoplay. VAD thresholds: `rmsThreshold=18`, `voiceBandMinProp=0.20`, `noiseFloor * 1.8`. TTS preprocess now expands `k → thousand`. Anything in the codebase still assuming default-ON Live mode is stale.
|
| 135 |
- **Marketplace dedup is one card per IRDAI-filed product (KI-133 / KI-141 / KI-142 / KI-145).** Aliases handle marketing renames (e.g. Reliance → IndusInd); sub-variants stay separate only when material terms differ. Card count is 166 across 19 real insurers — anything counting 138 or 188 or 206 against the marketplace is stale.
|
| 136 |
-
- **`sales_brain.py` is the fact-find handler (KI-167, [ADR-039](70-docs/60-decisions/ADR-039-llm-driven-sales-brain.md)).** `backend/fact_find_brain.py` is deleted along with the `<FF>` trailer convention, `_canonical_fallback`, scripted `Question.prompt_en`, and the `"Got that — {slot}."` prefix. Never re-introduce a fallback to scripted prompts on the fact-find branch — on total
|
|
|
|
| 137 |
|
| 138 |
---
|
| 139 |
|
| 140 |
-
*Last reviewed 2026-05-15 — KI-101..KI-112 landed (orchestrator stability + profile-RAG session isolation + recommendation closer + graceful chat error handling + Chroma re-ingest + profile-write hardening). Same day: KI-125..KI-150 landed (full corpus rebuild → 7,317 chunks; marketplace dedup → 166 cards; voice default OFF + VAD retune; IndusInd General slug migration from Reliance General). Same day: KI-160 / [ADR-038](70-docs/60-decisions/ADR-038-nim-only-chains.md) locked chains to NIM-only after KI-155's `<FF>` trailer contract violation. Same day: KI-167 / [ADR-039](70-docs/60-decisions/ADR-039-llm-driven-sales-brain.md) replaced `fact_find_brain.py` (
|
|
|
|
| 18 |
- **Hands-free was removed entirely** in KI-027. Anything in the codebase still referring to it is stale.
|
| 19 |
- **Bot TTS plays via the in-DOM `<audio>` element** inside `Message` (autoplay-on-mount via ref'd `useEffect`). Never use `new Audio(url).play()` — those detached instances are invisible to `document.querySelectorAll("audio").pause()` in the barge-in handler.
|
| 20 |
|
| 21 |
+
## LLM stack (ADR-019 + ADR-026 → ADR-031 + ADR-032 → ADR-038 → ADR-040) — KI-080 → KI-087, KI-160, KI-167 → KI-179
|
| 22 |
|
| 23 |
+
Every LLM role is a `NimChainLLM` candidate pool, NOT a hardcoded single model. End-to-end spec: [ADR-032](70-docs/60-decisions/ADR-032-llm-chain-architecture.md). Chains now mix **Google AI Studio (Gemini Flash) as Tier 0 primary, NIM as Tier 1 fallback, OpenRouter free as Tier 2 diversity pool** ([ADR-040](70-docs/60-decisions/ADR-040-google-gemini-primary.md) — supersedes ADR-038's NIM-only lock once KI-167 retired the `<FF>` trailer convention that motivated the lock). Native provider JSON mode (`response_format={"type":"json_object"}` on NIM, `response_mime_type=application/json` on Gemini) is the structured-output contract; cross-provider fallback is safe again as long as every candidate supports JSON mode natively. Chains preserve brain ↔ judge family diversity (Gemini brain ↔ Mistral / Llama-4 judge) so failovers can't produce circular grading.
|
| 24 |
|
| 25 |
+
- **Three-tier election ([ADR-040](70-docs/60-decisions/ADR-040-google-gemini-primary.md)).** Candidate pools per role:
|
| 26 |
+
- **Brain Fast** (sales-brain fact-find per KI-167, fast QA): Google `gemini-2.0-flash` (PRIMARY) → NIM `qwen3-next-80b-a3b-instruct` → NIM `mistralai/mistral-large-3-675b-instruct-2512` (675B dense) → NIM `meta/llama-4-maverick-17b-128e-instruct` (128B MoE) → OpenRouter `nvidia/nemotron-3-super-120b-a12b:free` → OpenRouter `qwen/qwen3-next-80b-a3b-instruct:free` → NIM `nvidia/llama-3.3-nemotron-super-49b-v1.5` (last resort).
|
| 27 |
+
- **Brain Main** (free-form QA + recommendation synthesis): Google `gemini-2.5-flash` (PRIMARY) → NIM `mistralai/mistral-large-3-675b-instruct-2512` → NIM `meta/llama-4-maverick-17b-128e-instruct` → NIM `qwen/qwen3-next-80b-a3b-instruct` → OpenRouter `nvidia/nemotron-3-super-120b-a12b:free` → NIM `nvidia/llama-3.3-nemotron-super-49b-v1.5` (last resort).
|
| 28 |
+
- **Judge** (faithfulness Gate 4, KI-171 skips on `fact_find` + `recommendation` queries): NIM `mistralai/mistral-large-3-675b-instruct-2512` (PRIMARY — different family from Gemini brain) → NIM `meta/llama-4-maverick-17b-128e-instruct` → OpenRouter `qwen/qwen3-next-80b-a3b-instruct:free` → NIM `nvidia/llama-3.3-nemotron-super-49b-v1.5`.
|
| 29 |
+
Google AI Studio free tier: 1500 req/day, 15 req/min on Gemini 2.0/2.5 Flash. On total chain exhaustion, orchestrator returns a graceful error message to the user — **fail-loud > fail-silent-with-garbage** still holds at the chain-tail level.
|
| 30 |
- **Probe-driven sticky primary election (KI-080, [ADR-031](70-docs/60-decisions/ADR-031-sticky-primary-election.md), superseded by ADR-038 for candidate-pool scope).** All three chains elect a sticky PRIMARY + BACKUP from a background probe within the NIM pool. `backend/llm_health.py` scores every candidate on `(1 / max(50, latency_ms)) * success_rate` and writes the current election to process state. `NimChainLLM.chat()` calls PRIMARY once; on real-time failure it falls to BACKUP and triggers an immediate probe refresh. **Per-turn LLM call count: 1 (most cases) or 2 (PRIMARY fails real-time → BACKUP).**
|
| 31 |
- **Probe cadence + per-phase timeouts (KI-084, `119e0fd`).** Probe loop ticks at `PROBE_INTERVAL_SEC = 300s`. Probe `max_tokens` cut `5 → 1`. Every chat call uses explicit `httpx.Timeout(connect=2, read=12, write=2, pool=2)` so a stuck NIM pool releases its TCP socket independently of the outer `asyncio.wait_for`. Rate-limit failures (HTTP 429 / `RateLimit` body) get a **1h sin-bin** (`DEGRADE_DURATION_LONG_S = 3600s`).
|
| 32 |
- **Proactive credit gating (KI-085, `8fc7979`), now NIM-only scope.** Election is gated by `is_alive AND has_credits` so quota-exhausted NIM models are excluded BEFORE the user hits a 429. Signal source within the NIM pool: per-model local 60-second rate-meter (gate at 35-of-40 req/min, headroom 5). Per-model rate-metering applies within the locked NIM pool.
|
| 33 |
+
- **HF Space secrets (KI-081, updated KI-179).** The chain config now consults **`GOOGLE_API_KEY`** (Tier 0, Brain Fast + Brain Main primary), `NVIDIA_NIM_API_KEY` (Tier 1 fallback + Judge primary), and `OPENROUTER_API_KEY` (Tier 2 diversity pool — KI-176 server-side `models: [...]` fallback within the OR free pool). `GROQ_API_KEY` remains in HF Space secrets but is dormant — Groq is no longer in any chain (KI-155 `<FF>` violation root cause remains a guard even though the `<FF>` convention itself is gone).
|
| 34 |
- **Admin telemetry (KI-086, `d90f8c0`).** `GET /api/admin/llm-health` returns `{chains, candidates, recent_turns, snapshot_ts}` with per-chain elected primary/backup, per-candidate health + credits + degraded-until, and last 20 turn outcomes. Admin "LLM Chain" tab auto-refreshes every 30s and renders whatever the chain config exposes (NIM-only post-KI-160).
|
| 35 |
- **KI-025's 50/50 NIM ↔ Groq rotation ([ADR-026](70-docs/60-decisions/ADR-026-provider-load-balancing.md)) is deprecated** — `_balanced_brain_chain` retained behind a feature flag for one-release rollback; the probe-driven NIM-only election picks the actually-faster candidate dynamically.
|
| 36 |
- **Cold-start fallback.** Before the first probe completes (process restart, HF Space rebuild), `chain[0]` is the initial primary and `chain[1]` is the initial backup, both NIM.
|
| 37 |
+
- **Brain / fast-brain / judge primaries in steady state** are **Google `gemini-2.5-flash`** (Brain Main), **Google `gemini-2.0-flash`** (Brain Fast / sales-brain fact-find), and **NIM Mistral Large 3 675B** (Judge), per the ADR-040 chain. Not hardcoded — elected primary follows live `latency × success_rate × credits_available` across the full three-tier pool; on Google quota exhaustion or 429, election falls naturally to NIM Mistral 675B (the Tier 1 fallback) without operator intervention.
|
| 38 |
- **KI-079 escalation as last bite (`87ee522`) — modified by KI-167.** If both PRIMARY and BACKUP fail in a single fact-find turn, orchestrator retries once on `BRAIN_CHAIN` (heavy brain, `_TIMEOUT_S_ESCALATION = 15s`, 35s chain budget). Post-KI-167 ([ADR-039](70-docs/60-decisions/ADR-039-llm-driven-sales-brain.md)) the `_canonical_fallback` terminal step is gone — on heavy-brain exhaustion the orchestrator returns the [ADR-038](70-docs/60-decisions/ADR-038-nim-only-chains.md) graceful error message instead. Worst-case wall-clock before graceful error: 25s FAST + 15s heavy = 40s.
|
| 39 |
- **NIM concurrency semaphore + serial probe (KI-088, `14ee008`).** Module-level `asyncio.Semaphore(2)` wraps every NIM HTTP call so our process never has >2 NIM requests in flight simultaneously, regardless of source (probe loop + admin polls + per-user turns all serialise through the same semaphore). Probe loop changed parallel→serial so the 6-NIM probe burst becomes a 1-slot trickle over ~12s. Inner 4-attempt exponential-backoff retry deleted from `NvidiaNimLLM.chat()` — KI-080 election + KI-079 escalation now handle failover. Result: latency-based failures (41s timeouts under self-saturation) dropped to zero; replaced by a parser-side bottleneck (KI-090).
|
| 40 |
- **Lenient FF-block parser (KI-090, `11cf4b3`) — RETIRED by KI-167 ([ADR-039](70-docs/60-decisions/ADR-039-llm-driven-sales-brain.md)).** Was needed because the `<FF>...</FF>` trailer convention was fragile under load. Post-KI-167 the sales brain uses NIM `response_format={"type":"json_object"}` for guaranteed structured output, so the whole `_parse_ff_block` strict → fenced → bare-JSON ladder is gone.
|
|
|
|
| 51 |
- **Chroma collection re-ingested + profile-write hardening (KI-112).** KI-111 wrapped `.query()` so the bot survived the corruption, but every embedding query was raising `InternalError: Error executing plan: Internal error: Error finding id` and silently returning empty retrieval — the bot was answering 206 policies' worth of Qs without access to any policy chunk. Root cause: a pre-KI-102 deploy wrote a `profile_anonymous` chunk with NO `session_id` metadata; that legacy row poisoned every later `coll.query(where={"doc_type": {"$ne": "profile"}})` and the damage spread across HNSW segments (full collection extraction surfaced 1580 / 7356 chunks across 148 policies as `Error getting embedding`). Fix: full re-ingest from `rag/corpus/` PDFs → clean `rag/vectors/` + two new write-time guards in `backend/profile_rag.py::upsert_profile_chunk` — (a) reject `session_id` that isn't a non-empty `str`, (b) reject any embedding whose length ≠ `embedder.dimension` or that contains `None`. Both guards log a `WARNING` and return without writing, so a future model-drift or bad-input event can't re-poison HNSW. 4 new regression tests in `tests/test_profile_rag_isolation.py::TestUpsertRejectsBadInputs`. Repaired vectors uploaded to HF dataset `rohitsar567/insurance-bot-data` via `tools/upload_vectors_to_dataset.py` so the Space rebuild picks up the clean index. Old corrupted Chroma archived at `rag/_hf_dataset_backup/rag/vectors.corrupted.<ts>/`.
|
| 52 |
- **Chain budgets:** brain 20s × 35s total, fast-brain 12s × 22s total, judge 30s × 75s total. With KI-080 only PRIMARY + BACKUP consume budget in the common case — leaves headroom for KI-079 escalation. KI-084 per-phase httpx timeouts are nested inside these budgets.
|
| 53 |
- **STT/TTS/Translator** = Sarvam (Saarika v2.5 / Bulbul v2 / Sarvam-M). **Embeddings** = local BGE-small-en-v1.5.
|
| 54 |
+
- **Provider keys (post-KI-179).** `GOOGLE_API_KEY` (Tier 0 — Gemini Flash primary), `NVIDIA_NIM_API_KEY` (Tier 1 fallback + Judge primary), `OPENROUTER_API_KEY` (Tier 2 diversity pool) required in `.env` (local) and HF Space environment (production). `GROQ_API_KEY` retained as dormant secret for one-flip re-enable, but no chain references it.
|
| 55 |
|
| 56 |
+
## Fact-find loop (ADR-039 + ADR-040, supersedes ADR-030 / ADR-027) — KI-167 / KI-179
|
| 57 |
|
| 58 |
+
**One LLM call per turn, structured-output JSON, no scripted prompts, no fallback prose.** KI-167 ripped out `backend/fact_find_brain.py` (the `<FF>{...}</FF>` trailer convention + `_canonical_fallback` greedy slot-walker + scripted `Question.prompt_en` prefixed with `"Got that — {slot}."`) and replaced it with `backend/sales_brain.py` — a single LLM-driven sales agent. KI-179 then moved the primary candidate from NIM Qwen to Google `gemini-2.0-flash` ([ADR-040](70-docs/60-decisions/ADR-040-google-gemini-primary.md)) for best-in-class conversational quality on the free path. The pre-KI-167 architecture had accreted eight patches (KI-090 / KI-091 / KI-094 / KI-103 / KI-150 / KI-155 / KI-156 / KI-158 / KI-161) working around the fragility of pattern-matching structure out of prose; KI-155 and follow-ups showed the canonical fallback was the *dominant* path, not the safety net. User-visible symptom: "zero natural LLM chat — it always defaults to the script."
|
| 59 |
|
| 60 |
+
- **Single LLM call per turn** via `NimChainLLM(FAST_BRAIN_CHAIN)` against the [ADR-040](70-docs/60-decisions/ADR-040-google-gemini-primary.md) three-tier pool (Google Tier 0 → NIM Tier 1 → OpenRouter Tier 2). Google uses **`response_mime_type=application/json`**; NIM uses **`response_format={"type":"json_object"}`**; OpenRouter free-tier candidates (`nvidia/nemotron-3-super-120b-a12b:free`, `qwen/qwen3-next-80b-a3b-instruct:free`) also support `response_format` natively (KI-178 audit). Every candidate enforces structured output server-side — same contract already validated in `backend/translation_check.py`, `backend/faithfulness.py`, and `backend/security.py`. Response shape: `{"reply": "<prose for user>", "captures": {<slot_id>: <raw>}, "slot_driving": "<slot_id|null>", "complete": <bool>}`.
|
| 61 |
- **System prompt** carries the 9-slot schema + current profile state. The LLM is free to ask in any order, in any voice, multi-fact in one turn or one slot at a time. No scripted opener, no acknowledger prefix, no `prompt_en` template.
|
| 62 |
- **Deterministic post-processor** (`backend/sales_brain_normalizer.py`) takes the LLM's loose `captures` dict and emits a `{canonical_field: validated_value}` map: alias resolution (`location` → `location_tier`), enum normalization (`Bangalore` → `metro`), INR-amount parsing, null/empty drop, type/bounds validation. No LLM calls — pure rules. Override: if LLM sets `complete: true` while any required slot is empty, force `complete: false`.
|
| 63 |
- **Profile persistence unchanged.** Captures flow through `session.update_profile_field()` → `backend/profile_store.save_profile()` (disk durability) → `backend/profile_rag.upsert_profile_chunk()` (retrieval-side visibility). KI-102 / KI-107 / KI-112 session-isolation guards remain in force.
|
| 64 |
- **No scripted prompts, no canonical_fallback, no trailer convention.** `backend/needs_finder.py::GRAPH` slot-id data stays as a schema source for the system prompt, but `Question.prompt_en` is dead text — never consulted by the fact-find branch. `_canonical_fallback`, `_normalize_for_slot`, `_pick_opener`, `_NEUTRAL_OPENERS`, `_FAMILY_OPENERS`, `_contains_self_introduction`, the lenient `<FF>` parser ladder, and the `"Got that — {slot}."` prefix logic are deleted.
|
| 65 |
+
- **Outer 25s `asyncio.wait_for` ceiling retained.** On total chain exhaustion (all of Google → NIM → OpenRouter fail in a single turn) the orchestrator returns the graceful error to the user (fail-loud) rather than cascading to a scripted reply. There is no scripted safety net — that is intentional.
|
| 66 |
- **DELETED in KI-167:** `backend/fact_find_brain.py` (441 LOC); `_canonical_fallback` + `_pick_opener` branches in `backend/orchestrator.py`; `_ff_failed_attempts` / `_ff_skipped_slots` session fields; every `fact_find_brain::fallback:*` telemetry variant; the lenient `<FF>` / fenced-`json` / bare-JSON-tail parser ladder.
|
| 67 |
- **Persona-audit fixtures rewritten** to assert on final captured state (which slots are filled with what) rather than per-turn slot order — the LLM may capture in different turn order than the pre-KI-167 rules engine did. See WS4.
|
| 68 |
- **Natural-conversation escape (KI-045):** intent_change phrases / off-topic questions still exit fact-find by routing through `should_route_to_fact_find` upstream of the brain.
|
|
|
|
| 137 |
- **The `indusind-general` slug did not exist anywhere in the codebase before 2026-05-15.** Reliance General Insurance was rebranded to IndusInd General; **KI-144** migrated insurer slug + policy IDs + Chroma metadata + marketplace alias mapping. Any code referencing `reliance-general` should either be retained as a legacy alias (one card remains under `reliance-general` for back-compat) or migrated to `indusind-general`. Do not silently merge the two — they're tracked as separate slugs.
|
| 138 |
- **Voice mode now defaults OFF (KI-131 / KI-134 / KI-139 / KI-148).** The Live pill renders red by default; the user must opt in. AudioContext.resume() is required to unlock TTS autoplay. VAD thresholds: `rmsThreshold=18`, `voiceBandMinProp=0.20`, `noiseFloor * 1.8`. TTS preprocess now expands `k → thousand`. Anything in the codebase still assuming default-ON Live mode is stale.
|
| 139 |
- **Marketplace dedup is one card per IRDAI-filed product (KI-133 / KI-141 / KI-142 / KI-145).** Aliases handle marketing renames (e.g. Reliance → IndusInd); sub-variants stay separate only when material terms differ. Card count is 166 across 19 real insurers — anything counting 138 or 188 or 206 against the marketplace is stale.
|
| 140 |
+
- **`sales_brain.py` is the fact-find handler (KI-167, [ADR-039](70-docs/60-decisions/ADR-039-llm-driven-sales-brain.md); primary candidate is Google Gemini 2.0 Flash post-KI-179 / [ADR-040](70-docs/60-decisions/ADR-040-google-gemini-primary.md)).** `backend/fact_find_brain.py` is deleted along with the `<FF>` trailer convention, `_canonical_fallback`, scripted `Question.prompt_en`, and the `"Got that — {slot}."` prefix. Never re-introduce a fallback to scripted prompts on the fact-find branch — on total three-tier chain exhaustion the orchestrator returns a graceful error message (fail-loud > fail-silent-with-script). The KI-150 historical note (`fact_find_brain` `max_tokens` 420 → 700) is moot post-KI-167 since the prose+trailer prompt shape that needed the larger budget no longer exists.
|
| 141 |
+
- **Never hardcode `NvidiaNimLLM(model=...)` post-KI-179** — chains now mix Google (`backend/providers/google_gemini_llm.py`), NIM (`backend/providers/nvidia_nim_llm.py`), and OpenRouter (`backend/providers/openrouter_llm.py`) candidates. Always go through `NimChainLLM(chain=...)` so the call surfs the full three-tier failover; bypassing the chain reintroduces the single-provider single-point-of-failure that KI-080 / KI-160 / KI-179 collectively eliminated.
|
| 142 |
|
| 143 |
---
|
| 144 |
|
| 145 |
+
*Last reviewed 2026-05-15 — KI-101..KI-112 landed (orchestrator stability + profile-RAG session isolation + recommendation closer + graceful chat error handling + Chroma re-ingest + profile-write hardening). Same day: KI-125..KI-150 landed (full corpus rebuild → 7,317 chunks; marketplace dedup → 166 cards; voice default OFF + VAD retune; IndusInd General slug migration from Reliance General). Same day: KI-160 / [ADR-038](70-docs/60-decisions/ADR-038-nim-only-chains.md) locked chains to NIM-only after KI-155's `<FF>` trailer contract violation. Same day: KI-167 / [ADR-039](70-docs/60-decisions/ADR-039-llm-driven-sales-brain.md) replaced `fact_find_brain.py` with `backend/sales_brain.py` (single LLM call per turn, native JSON mode, deterministic post-processor). Same day: KI-171 (judge skip on fact_find + recommendation), KI-175 (NIM chain reorder — nemotron demoted to last), KI-176 (OpenRouter `models: [...]` server-side fallback), KI-178 (live audit of OR free-tier JSON-mode support), KI-179 / [ADR-040](70-docs/60-decisions/ADR-040-google-gemini-primary.md) added Google AI Studio (Gemini 2.0 / 2.5 Flash) as the Tier 0 primary on Brain Fast + Brain Main, with NIM as Tier 1 fallback and OpenRouter as Tier 2 diversity pool. ADR-038 is now superseded — the NIM-only lock was relaxed once the `<FF>` trailer convention that motivated it was retired.*
|
|
@@ -64,7 +64,11 @@ Thirty-plus knowledge increments landed today. The corpus was rebuilt, the marke
|
|
| 64 |
- **KI-149** — budget + income parser captures bare numerals like `"30000"`.
|
| 65 |
- **KI-150** — `fact_find_brain` `max_tokens` 420 → 700 (one of eight band-aids on the prose+`<FF>`-trailer prompt shape; superseded by KI-167).
|
| 66 |
- **KI-160** — chains locked to NIM-only ([ADR-038](70-docs/60-decisions/ADR-038-nim-only-chains.md)) after KI-155 demonstrated Groq Llama-3.3 silently ignores the `<FF>` structured-output trailer contract.
|
| 67 |
-
- **KI-167** — `backend/fact_find_brain.py` ripped out and replaced by `backend/sales_brain.py` ([ADR-039](70-docs/60-decisions/ADR-039-llm-driven-sales-brain.md)). Single LLM call per turn using
|
|
|
|
|
|
|
|
|
|
|
|
|
| 68 |
|
| 69 |
Per-insurer card counts (166 total across 19 real insurers): HDFC ERGO 15 · National Insurance 14 · Niva Bupa 14 · Bajaj Allianz 13 · ICICI Lombard 13 · Star Health 11 · Care Health 10 · New India Assurance 9 · Tata AIG 9 · Acko 7 · Aditya Birla 7 · Royal Sundaram 7 · Cholamandalam MS 6 · Go Digit 6 · IFFCO Tokio 6 · ManipalCigna 6 · SBI General 6 · IndusInd General 3 · Oriental Insurance 3 · Reliance General 1.
|
| 70 |
|
|
@@ -92,15 +96,15 @@ A take-home is a sample of how the engineer thinks under constraint. Three thing
|
|
| 92 |
|
| 93 |
2. **Hallucination defense and refusal as product features.** BFSI deployments get fined for mis-selling; the bot is biased toward refusal over confident wrong answers. The 4 faithfulness gates + cross-check retry + 3 Indic drift checks + audit log are the BFSI-compliance-grade version of "we shipped a chatbot." When the eval shows a headline accuracy below 100% because the gates are aggressive, the right response is to soften the gates carefully — not to ship a higher number by relaxing the verifier.
|
| 94 |
|
| 95 |
-
3. **Honest model picks — Sarvam where Sarvam is uniquely strong,
|
| 96 |
|
| 97 |
-
- **
|
| 98 |
-
- **
|
| 99 |
-
- **
|
| 100 |
|
| 101 |
-
**
|
| 102 |
|
| 103 |
-
The result: a Sarvam customer deploying this stack gets a product that *uses Sarvam exactly where Sarvam beats the world* and uses MIT-licensed open-weights frontier models for everything else — $0 inference,
|
| 104 |
|
| 105 |
The rest is craftsmanship. The 8-section KB ([`kb/`](kb/)) is regeneratable from primary sources in <40 minutes for <$2 cold. Every numeric value in every reviewer-facing artifact traces to a source PDF + page + clause. Every architectural decision is in [`70-docs/60-decisions/`](70-docs/60-decisions/) with alternatives and revisit-at-scale notes. Every production-readiness defect is in [`80-audit/ENTERPRISE_AUDIT.md`](80-audit/ENTERPRISE_AUDIT.md). The repo is structured so a new engineer joining on Monday could ship v1.1 by Friday.
|
| 106 |
|
|
@@ -201,8 +205,8 @@ The bot is two flows running together — the customer's experience and the tech
|
|
| 201 |
- `detect_language(user_text)` → `english / indic`
|
| 202 |
- Indic cascade: if indic, Sarvam-M translates → English for reasoning, response translated back
|
| 203 |
|
| 204 |
-
**Tech 4 — Fact-find branch (KI-167 / [ADR-039](70-docs/60-decisions/ADR-039-llm-driven-sales-brain.md))**
|
| 205 |
-
- If profile empty AND intent ∈ {fact_find, recommendation, comparison}: hand the turn to `backend/sales_brain.py::drive_sales_brain()` — one
|
| 206 |
- Else: continue to retrieval
|
| 207 |
|
| 208 |
</td></tr>
|
|
@@ -254,9 +258,9 @@ The bot is two flows running together — the customer's experience and the tech
|
|
| 254 |
</td><td>
|
| 255 |
|
| 256 |
**Tech 7 — Brain selection, `pick_brain(intent, language)`**
|
| 257 |
-
- `intent ∈ {comparison, recommendation}` →
|
| 258 |
-
- `intent ∈ {qa, fact_find}` →
|
| 259 |
-
- KI-080 election: 1 LLM call per turn (most cases) or 2 (primary fails real-time →
|
| 260 |
|
| 261 |
**Tech 8 — System prompt construction, `build_messages()`**
|
| 262 |
- `[System: ADVISOR_PROMPT + USER PROFILE block + USER IS LOOKING AT (view_context) block]`
|
|
@@ -376,11 +380,13 @@ The bot is two flows running together — the customer's experience and the tech
|
|
| 376 |
│
|
| 377 |
▼
|
| 378 |
┌──────────────────────────────────────────────────────────────────────────┐
|
| 379 |
-
│ TECH 4. Fact-find OR free-form branch (KI-167 / ADR-039)
|
| 380 |
│ ------- │
|
| 381 |
│ · If intent ∈ {fact_find, recommendation, comparison} AND profile │
|
| 382 |
│ incomplete: │
|
| 383 |
-
│ drive_sales_brain() — one
|
|
|
|
|
|
|
| 384 |
│ → {reply, captures, slot_driving, complete} │
|
| 385 |
│ normalize captures → session.update_profile_field() │
|
| 386 |
│ → profile_store.save_profile() + profile_rag.upsert_profile_chunk() │
|
|
@@ -415,9 +421,14 @@ The bot is two flows running together — the customer's experience and the tech
|
|
| 415 |
┌──────────────────────────────────────────────────────────────────────────┐
|
| 416 |
│ TECH 7. Brain selection — pick_brain(intent, language) │
|
| 417 |
│ ------- │
|
| 418 |
-
│ · intent ∈ {comparison, recommendation} →
|
| 419 |
-
│
|
| 420 |
-
│
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 421 |
└──────────────────────────────────────────────────────────────────────────┘
|
| 422 |
│
|
| 423 |
▼
|
|
@@ -435,7 +446,7 @@ The bot is two flows running together — the customer's experience and the tech
|
|
| 435 |
┌──────────────────────────────────────────────────────────────────────────┐
|
| 436 |
│ TECH 9. Brain LLM call → reply text │
|
| 437 |
│ ------- │
|
| 438 |
-
│ ·
|
| 439 |
│ · strip_think_tags() removes <think>…</think> chain-of-thought │
|
| 440 |
│ · Capture brain_model_actual for non-circular judge selection │
|
| 441 |
└──────────────────────────────────────────────────────────────────────────┘
|
|
@@ -521,14 +532,18 @@ The bot is two flows running together — the customer's experience and the tech
|
|
| 521 |
│ ┌───────────────────────┼──────────────────────────┐ │
|
| 522 |
│ ▼ ▼ ▼ │
|
| 523 |
│ ┌──────────────┐ ┌──────────────────┐ ┌──────────────────────┐ │
|
| 524 |
-
│ │ STRUCTURED │ │ VECTOR STORE │ │
|
| 525 |
-
│ │ DuckDB │ │ Chroma 0.5.20 │ │
|
| 526 |
-
│ │ 62 fields │ │ BGE-small (384d) │ │
|
| 527 |
-
│ │ per policy │ │ 800/120 chunk │ │
|
| 528 |
-
│ └──────────────┘ │ +profile chunk │ │
|
| 529 |
-
│ │ per session │ │
|
| 530 |
-
│ └──────────────────┘ │
|
| 531 |
-
│ │
|
|
|
|
|
|
|
|
|
|
|
|
|
| 532 |
│ └──────────────────────┘ │
|
| 533 |
└─────────────────────────────────────────────────────────────────────────┘
|
| 534 |
│
|
|
@@ -573,16 +588,16 @@ The bot is two flows running together — the customer's experience and the tech
|
|
| 573 |
|
| 574 |
### 4.3 Model stack
|
| 575 |
|
| 576 |
-
Every LLM role is served by a **candidate pool**
|
| 577 |
|
| 578 |
| Role | Primary | Fallback chain (in order) | Provider(s) | Why this primary |
|
| 579 |
|---|---|---|---|---|
|
| 580 |
-
| **
|
| 581 |
-
| **
|
| 582 |
-
| **Judge** (faithfulness Gate 4, Hinglish drift, eval grader) | Mistral Large 3 675B
|
| 583 |
-
| **Profile extractor** (free-form profile updates → 9-slot schema, ADR-022) | Fast
|
| 584 |
-
| **Fact-find normalizer** (user answer → typed slot value) | Fast
|
| 585 |
-
| **Sales brain** (fact-find conversation driver — KI-167 / [ADR-039](70-docs/60-decisions/ADR-039-llm-driven-sales-brain.md)) | Fast
|
| 586 |
| **Indic translation** (English ↔ Hindi / Hinglish / vernacular) | Sarvam-M | — (no fallback; Indic is non-substitutable) | Sarvam | Best-in-class on Hinglish and code-mixed Indian languages; no open-weights frontier model is competitive here. |
|
| 587 |
| **STT** (speech-to-text) | Sarvam Saarika v2.5 (`saarika:v2.5`) | — | Sarvam | Best-in-class on Indian-accented English and Hinglish; WebM transcoded to 16kHz mono WAV at the gateway (`backend/providers/sarvam_stt.py`). |
|
| 588 |
| **TTS** (text-to-speech) | Sarvam Bulbul v2 (`bulbul:v2`, speaker `anushka`) | — | Sarvam | Best-in-class on Hinglish prosody; 22.05kHz output, base64 WAV from Sarvam endpoint. |
|
|
@@ -806,9 +821,15 @@ cp -R rag/_hf_dataset_backup/rag/* rag/ # corpus/, extracted/, vectors/
|
|
| 806 |
|
| 807 |
```bash
|
| 808 |
cp .env.example .env
|
| 809 |
-
# Edit .env and fill in the
|
| 810 |
-
#
|
| 811 |
-
#
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 812 |
chmod 600 .env
|
| 813 |
```
|
| 814 |
|
|
|
|
| 64 |
- **KI-149** — budget + income parser captures bare numerals like `"30000"`.
|
| 65 |
- **KI-150** — `fact_find_brain` `max_tokens` 420 → 700 (one of eight band-aids on the prose+`<FF>`-trailer prompt shape; superseded by KI-167).
|
| 66 |
- **KI-160** — chains locked to NIM-only ([ADR-038](70-docs/60-decisions/ADR-038-nim-only-chains.md)) after KI-155 demonstrated Groq Llama-3.3 silently ignores the `<FF>` structured-output trailer contract.
|
| 67 |
+
- **KI-167** — `backend/fact_find_brain.py` ripped out and replaced by `backend/sales_brain.py` ([ADR-039](70-docs/60-decisions/ADR-039-llm-driven-sales-brain.md)). Single LLM call per turn using native JSON mode; no scripted `Question.prompt_en`, no `<FF>` trailer, no `_canonical_fallback`, no `"Got that — {slot}."` prefix. Deterministic post-processor (`backend/sales_brain_normalizer.py`) normalizes captures; profile persists via `profile_store` + `profile_rag` as before.
|
| 68 |
+
- **KI-171** — orchestrator skips the Judge chain on `fact_find` + `recommendation` queries (faithfulness scoring is meaningless on profile-capture turns and on multi-policy ranked shortlists where the rubric is structural, not factual grounding).
|
| 69 |
+
- **KI-175** — NIM chain reorder: Mistral Large 3 675B promoted to primary, nemotron-49b demoted to last resort across all three chains. The old "nemotron primary" ordering predated the 675B model's availability.
|
| 70 |
+
- **KI-176** — OpenRouter re-added to the chain pool with `models: [...]` server-side fallback within the OR free pool. KI-178 live-audit then confirmed which OR free-tier models actually support `response_format` (nemotron-3-super-120b, qwen3-next-80b, gemma-4 — not Llama 3.3 70B or Hermes 3 405B).
|
| 71 |
+
- **KI-179** — Google AI Studio added as Tier 0 primary on Brain Fast + Brain Main ([ADR-040](70-docs/60-decisions/ADR-040-google-gemini-primary.md)). `backend/providers/google_gemini_llm.py` wrapper matches the `LLMProvider` interface; chains now span {Google → NIM → OpenRouter}. ADR-038 (NIM-only lock) is superseded — the lock was relaxed once KI-167 retired the `<FF>` trailer convention that originally motivated it. Brain Fast primary is `gemini-2.0-flash`; Brain Main primary is `gemini-2.5-flash`; Judge stays NIM Mistral Large 3 675B (different family from Gemini, preserving the brain ↔ judge non-circular grading invariant). Google's free tier: 1500 req/day, 15 req/min, native JSON via `response_mime_type=application/json`.
|
| 72 |
|
| 73 |
Per-insurer card counts (166 total across 19 real insurers): HDFC ERGO 15 · National Insurance 14 · Niva Bupa 14 · Bajaj Allianz 13 · ICICI Lombard 13 · Star Health 11 · Care Health 10 · New India Assurance 9 · Tata AIG 9 · Acko 7 · Aditya Birla 7 · Royal Sundaram 7 · Cholamandalam MS 6 · Go Digit 6 · IFFCO Tokio 6 · ManipalCigna 6 · SBI General 6 · IndusInd General 3 · Oriental Insurance 3 · Reliance General 1.
|
| 74 |
|
|
|
|
| 96 |
|
| 97 |
2. **Hallucination defense and refusal as product features.** BFSI deployments get fined for mis-selling; the bot is biased toward refusal over confident wrong answers. The 4 faithfulness gates + cross-check retry + 3 Indic drift checks + audit log are the BFSI-compliance-grade version of "we shipped a chatbot." When the eval shows a headline accuracy below 100% because the gates are aggressive, the right response is to soften the gates carefully — not to ship a higher number by relaxing the verifier.
|
| 98 |
|
| 99 |
+
3. **Honest model picks — Sarvam where Sarvam is uniquely strong, frontier-tier free brains for reasoning.** Voice and Indic are non-substitutable: **Sarvam Saarika v2.5** for speech-to-text, **Sarvam Bulbul v2** (speaker `anushka`) for text-to-speech, and **Sarvam-M** for Hindi/Hinglish/vernacular translation — no closed-source frontier matches Sarvam on Indian accents or code-mixed Hinglish. Reasoning runs on a three-tier candidate pool per role (KI-179 / [ADR-040](70-docs/60-decisions/ADR-040-google-gemini-primary.md)), not a single hardcoded brain. Every candidate supports native JSON mode server-side — Google's `response_mime_type=application/json`, NIM's `response_format={"type":"json_object"}`, OpenRouter's `response_format` pass-through on the audited free-tier models (KI-178). Cross-provider fallback is safe again now that the `<FF>` trailer convention that triggered KI-160 / ADR-038's NIM-only lock has been retired by KI-167 / [ADR-039](70-docs/60-decisions/ADR-039-llm-driven-sales-brain.md):
|
| 100 |
|
| 101 |
+
- **Brain Main** (comparison, recommendation, synthesis) — primary **Google `gemini-2.5-flash`** via Google AI Studio (1500 req/day free, native JSON mode), Tier 1 fallback **NIM Mistral Large 3 675B** (`mistralai/mistral-large-3-675b-instruct-2512`), then **NIM Llama-4 Maverick 17B/128E**, then **NIM Qwen 3-Next 80B**, then **OpenRouter `nvidia/nemotron-3-super-120b-a12b:free`**, last resort **NIM Nemotron-Super 49B v1.5**.
|
| 102 |
+
- **Brain Fast** (sales-brain fact-find turns per KI-167, fast QA) — primary **Google `gemini-2.0-flash`**, Tier 1 fallback **NIM Qwen 3-Next 80B**, then **NIM Mistral Large 3 675B**, then **NIM Llama-4 Maverick 17B/128E**, then **OpenRouter `nvidia/nemotron-3-super-120b-a12b:free`** + `qwen/qwen3-next-80b-a3b-instruct:free`, last resort **NIM Nemotron-Super 49B v1.5**.
|
| 103 |
+
- **Judge** (faithfulness Gate 4, Hinglish drift LLM-judge, eval grader; KI-171 skips this on `fact_find` + `recommendation` queries) — primary **NIM Mistral Large 3 675B** (different family from the Gemini brain — preserves cross-family non-circular grading), Tier 1 fallback **NIM Llama-4 Maverick 17B/128E**, then **OpenRouter `qwen/qwen3-next-80b-a3b-instruct:free`**, last resort **NIM Nemotron-Super 49B v1.5**.
|
| 104 |
|
| 105 |
+
**Three-tier election (KI-179, [ADR-040](70-docs/60-decisions/ADR-040-google-gemini-primary.md)).** Google → NIM → OpenRouter, with the chain falling through naturally when an upper tier 429s, exhausts quota, or fails real-time. On total chain exhaustion the orchestrator returns a graceful error message — fail-loud > fail-silent-with-garbage is preserved. The 50/50 NIM ↔ Groq rotation of KI-025 ([ADR-026](70-docs/60-decisions/ADR-026-provider-load-balancing.md)) and the NIM-only lock of KI-160 ([ADR-038](70-docs/60-decisions/ADR-038-nim-only-chains.md)) are both superseded. KI-085's proactive credit gating applies per-candidate (60-second rate-meter, gate at 35-of-40 req/min). `GROQ_API_KEY` remains in HF Space secrets but is dormant — no chain references it.
|
| 106 |
|
| 107 |
+
The result: a Sarvam customer deploying this stack gets a product that *uses Sarvam exactly where Sarvam beats the world*, uses **Google Gemini Flash on the free path for frontier-tier conversational quality**, and uses MIT-licensed open-weights frontier models (Mistral Large 3 675B, Llama-4 Maverick, Qwen 3-Next 80B, Nemotron-Super 49B) for everything else — $0 inference, three independent free-tier providers, single-key-per-provider for the entire non-voice stack.
|
| 108 |
|
| 109 |
The rest is craftsmanship. The 8-section KB ([`kb/`](kb/)) is regeneratable from primary sources in <40 minutes for <$2 cold. Every numeric value in every reviewer-facing artifact traces to a source PDF + page + clause. Every architectural decision is in [`70-docs/60-decisions/`](70-docs/60-decisions/) with alternatives and revisit-at-scale notes. Every production-readiness defect is in [`80-audit/ENTERPRISE_AUDIT.md`](80-audit/ENTERPRISE_AUDIT.md). The repo is structured so a new engineer joining on Monday could ship v1.1 by Friday.
|
| 110 |
|
|
|
|
| 205 |
- `detect_language(user_text)` → `english / indic`
|
| 206 |
- Indic cascade: if indic, Sarvam-M translates → English for reasoning, response translated back
|
| 207 |
|
| 208 |
+
**Tech 4 — Fact-find branch (KI-167 / [ADR-039](70-docs/60-decisions/ADR-039-llm-driven-sales-brain.md); primary candidate is Google Gemini 2.0 Flash post-KI-179 / [ADR-040](70-docs/60-decisions/ADR-040-google-gemini-primary.md))**
|
| 209 |
+
- If profile empty AND intent ∈ {fact_find, recommendation, comparison}: hand the turn to `backend/sales_brain.py::drive_sales_brain()` — one LLM call against the Brain Fast chain (Google `gemini-2.0-flash` primary → NIM Qwen / Mistral 675B / Llama-4 Maverick → OpenRouter free → NIM Nemotron-49b last resort) using native JSON mode (`response_mime_type=application/json` on Gemini, `response_format={"type":"json_object"}` on NIM / OR). System prompt carries the 9-slot schema + current profile state. Response: `{reply, captures, slot_driving, complete}`. `backend/sales_brain_normalizer.py` normalizes captures → `session.update_profile_field()` → `profile_store.save_profile()` + `profile_rag.upsert_profile_chunk()`. Emit `reply` to the user.
|
| 210 |
- Else: continue to retrieval
|
| 211 |
|
| 212 |
</td></tr>
|
|
|
|
| 258 |
</td><td>
|
| 259 |
|
| 260 |
**Tech 7 — Brain selection, `pick_brain(intent, language)`**
|
| 261 |
+
- `intent ∈ {comparison, recommendation}` → Brain Main chain with probe-elected primary across the three-tier pool ([ADR-040](70-docs/60-decisions/ADR-040-google-gemini-primary.md)): Google `gemini-2.5-flash` → NIM Mistral Large 3 675B → NIM Llama-4 Maverick → NIM Qwen 3-Next 80B → OpenRouter `nemotron-3-super-120b:free` → NIM Nemotron-49b
|
| 262 |
+
- `intent ∈ {qa, fact_find}` → Brain Fast chain: Google `gemini-2.0-flash` → NIM Qwen → NIM Mistral 675B → NIM Llama-4 Maverick → OpenRouter free pool → NIM Nemotron-49b. `fact_find` turns go through `backend/sales_brain.py` with native JSON mode (KI-167 / [ADR-039](70-docs/60-decisions/ADR-039-llm-driven-sales-brain.md))
|
| 263 |
+
- KI-080 election: 1 LLM call per turn (most cases) or 2 (primary fails real-time → next tier). Per-chain primary refreshed every 300s by background probe. On total three-tier exhaustion: graceful error to user (fail-loud — KI-179 preserves the [ADR-038](70-docs/60-decisions/ADR-038-nim-only-chains.md) principle even though the NIM-only candidate-pool scope is superseded).
|
| 264 |
|
| 265 |
**Tech 8 — System prompt construction, `build_messages()`**
|
| 266 |
- `[System: ADVISOR_PROMPT + USER PROFILE block + USER IS LOOKING AT (view_context) block]`
|
|
|
|
| 380 |
│
|
| 381 |
▼
|
| 382 |
┌──────────────────────────────────────────────────────────────────────────┐
|
| 383 |
+
│ TECH 4. Fact-find OR free-form branch (KI-167 / ADR-039 / ADR-040) │
|
| 384 |
│ ------- │
|
| 385 |
│ · If intent ∈ {fact_find, recommendation, comparison} AND profile │
|
| 386 |
│ incomplete: │
|
| 387 |
+
│ drive_sales_brain() — one LLM call against Brain Fast chain │
|
| 388 |
+
│ (Gemini 2.0 Flash → NIM Qwen / Mistral 675B → OR free) │
|
| 389 |
+
│ using native JSON mode (response_mime_type / response_format) │
|
| 390 |
│ → {reply, captures, slot_driving, complete} │
|
| 391 |
│ normalize captures → session.update_profile_field() │
|
| 392 |
│ → profile_store.save_profile() + profile_rag.upsert_profile_chunk() │
|
|
|
|
| 421 |
┌──────────────────────────────────────────────────────────────────────────┐
|
| 422 |
│ TECH 7. Brain selection — pick_brain(intent, language) │
|
| 423 |
│ ------- │
|
| 424 |
+
│ · intent ∈ {comparison, recommendation} → Brain Main chain │
|
| 425 |
+
│ Gemini 2.5 Flash → NIM Mistral 675B → NIM Llama-4 Maverick → │
|
| 426 |
+
│ NIM Qwen 80B → OR nemotron-3-super-120b:free → NIM Nemotron-49b │
|
| 427 |
+
│ · intent ∈ {qa, fact_find} → Brain Fast chain │
|
| 428 |
+
│ Gemini 2.0 Flash → NIM Qwen 80B → NIM Mistral 675B → NIM Llama-4 → │
|
| 429 |
+
│ OR free pool → NIM Nemotron-49b last resort │
|
| 430 |
+
│ · fact_find turns dispatched to sales_brain.py (KI-167 / ADR-039) │
|
| 431 |
+
│ · Three-tier election: Google → NIM → OpenRouter (KI-179 / ADR-040) │
|
| 432 |
└──────────────────────────────────────────────────────────────────────────┘
|
| 433 |
│
|
| 434 |
▼
|
|
|
|
| 446 |
┌──────────────────────────────────────────────────────────────────────────┐
|
| 447 |
│ TECH 9. Brain LLM call → reply text │
|
| 448 |
│ ------- │
|
| 449 |
+
│ · Provider-agnostic chat call via NimChainLLM (Google / NIM / OR) │
|
| 450 |
│ · strip_think_tags() removes <think>…</think> chain-of-thought │
|
| 451 |
│ · Capture brain_model_actual for non-circular judge selection │
|
| 452 |
└──────────────────────────────────────────────────────────────────────────┘
|
|
|
|
| 532 |
│ ┌───────────────────────┼──────────────────────────┐ │
|
| 533 |
│ ▼ ▼ ▼ │
|
| 534 |
│ ┌──────────────┐ ┌──────────────────┐ ┌──────────────────────┐ │
|
| 535 |
+
│ │ STRUCTURED │ │ VECTOR STORE │ │ 3-TIER BRAIN ROUTER │ │
|
| 536 |
+
│ │ DuckDB │ │ Chroma 0.5.20 │ │ Tier 0: Google AI │ │
|
| 537 |
+
│ │ 62 fields │ │ BGE-small (384d) │ │ Studio (Gemini │ │
|
| 538 |
+
│ │ per policy │ │ 800/120 chunk │ │ 2.0 / 2.5 Flash) │ │
|
| 539 |
+
│ └──────────────┘ │ +profile chunk │ │ Tier 1: NIM (Mistral │ │
|
| 540 |
+
│ │ per session │ │ 675B, Llama-4, │ │
|
| 541 |
+
│ └──────────────────┘ │ Qwen 80B, Nemo-49) │ │
|
| 542 |
+
│ │ Tier 2: OR free pool │ │
|
| 543 |
+
│ │ (nemotron-120b, │ │
|
| 544 |
+
│ │ qwen-80b:free) │ │
|
| 545 |
+
│ │ KI-179 / ADR-040 │ │
|
| 546 |
+
│ │ $0 inference path │ │
|
| 547 |
│ └──────────────────────┘ │
|
| 548 |
└─────────────────────────────────────────────────────────────────────────┘
|
| 549 |
│
|
|
|
|
| 588 |
|
| 589 |
### 4.3 Model stack
|
| 590 |
|
| 591 |
+
Every LLM role is served by a **three-tier candidate pool** elected over by a probe-driven sticky-primary scheme ([ADR-031](70-docs/60-decisions/ADR-031-sticky-primary-election.md), end-to-end spec in [ADR-032](70-docs/60-decisions/ADR-032-llm-chain-architecture.md), current candidate-pool shape in [ADR-040](70-docs/60-decisions/ADR-040-google-gemini-primary.md)), never a hardcoded single model. Chains preserve brain ↔ judge family diversity (Gemini brain ↔ Mistral judge) so any failover still produces non-circular grading. Tier 0 (Google AI Studio) is the steady-state primary; Tier 1 (NIM) is the fallback; Tier 2 (OpenRouter free pool) is the diversity tail before the last-resort NIM Nemotron-49b. Every candidate enforces native JSON mode server-side — `response_mime_type=application/json` on Google, `response_format={"type":"json_object"}` on NIM and on the audited OR free-tier models (KI-178).
|
| 592 |
|
| 593 |
| Role | Primary | Fallback chain (in order) | Provider(s) | Why this primary |
|
| 594 |
|---|---|---|---|---|
|
| 595 |
+
| **Brain Main** (comparison, recommendation, synthesis) | **Google `gemini-2.5-flash`** (1500 req/day free, native JSON mode) | NIM Mistral Large 3 675B → NIM Llama-4 Maverick 17B/128E → NIM Qwen 3-Next 80B → OpenRouter `nvidia/nemotron-3-super-120b-a12b:free` → NIM Nemotron-Super 49B v1.5 (last resort) | Google + NIM + OpenRouter | Best-in-class conversational + synthesis quality on the free path; 1500 req/day adequate for demo + early production. NIM Mistral 675B Tier 1 fallback preserves dense 675B reasoning when Google quota / 429 trips. KI-179 / [ADR-040](70-docs/60-decisions/ADR-040-google-gemini-primary.md). |
|
| 596 |
+
| **Brain Fast** (voice turns, sales-brain fact-find KI-167, QA) | **Google `gemini-2.0-flash`** (same 1500 req/day quota, lower-latency tier for fast turns) | NIM Qwen 3-Next 80B → NIM Mistral Large 3 675B → NIM Llama-4 Maverick → OpenRouter `nemotron-3-super-120b:free` + `qwen3-next-80b:free` → NIM Nemotron-Super 49B v1.5 (last resort) | Google + NIM + OpenRouter | Bottleneck on these short jobs is TTFT — Gemini 2.0 Flash optimised for low-latency JSON. Sales brain (KI-167) calls this chain with native JSON mode for guaranteed structured output. KI-179 / [ADR-040](70-docs/60-decisions/ADR-040-google-gemini-primary.md). |
|
| 597 |
+
| **Judge** (faithfulness Gate 4, Hinglish drift, eval grader; KI-171 skips on `fact_find` + `recommendation`) | **NIM Mistral Large 3 675B** (`mistralai/mistral-large-3-675b-instruct-2512`) | NIM Llama-4 Maverick 17B/128E → OpenRouter `qwen/qwen3-next-80b-a3b-instruct:free` → NIM Nemotron-Super 49B v1.5 | NIM + OpenRouter | Different family from the Gemini brain (Mistral, not Google / Llama / Qwen) so the judge sees the brain's output from a genuinely different decision surface. 675B dense, MIT, ~4.3s on NIM. Brain ↔ judge family diversity is the structural defence against circular grading. |
|
| 598 |
+
| **Profile extractor** (free-form profile updates → 9-slot schema, ADR-022) | Brain Fast chain (Gemini 2.0 Flash primary) | inherits Brain Fast chain | Google + NIM + OpenRouter | Short prompt, structured JSON out — same chain as sales brain. |
|
| 599 |
+
| **Fact-find normalizer** (user answer → typed slot value) | Brain Fast chain (Gemini 2.0 Flash primary) | inherits Brain Fast chain | Google + NIM + OpenRouter | Same shape as profile extractor — narrow input, narrow JSON output. |
|
| 600 |
+
| **Sales brain** (fact-find conversation driver — KI-167 / [ADR-039](70-docs/60-decisions/ADR-039-llm-driven-sales-brain.md), KI-179 / [ADR-040](70-docs/60-decisions/ADR-040-google-gemini-primary.md)) | Brain Fast chain (Gemini 2.0 Flash primary) | inherits Brain Fast chain | Google + NIM + OpenRouter | One LLM call per turn using native JSON mode. System prompt carries the 9-slot schema + current profile state; LLM owns voice + flow + slot order, returns `{reply, captures, slot_driving, complete}`. Replaces the scripted-`prompt_en` + paraphraser + opener-rotation + `<FF>` trailer + `_canonical_fallback` machinery (ADR-027 / ADR-030 retired). |
|
| 601 |
| **Indic translation** (English ↔ Hindi / Hinglish / vernacular) | Sarvam-M | — (no fallback; Indic is non-substitutable) | Sarvam | Best-in-class on Hinglish and code-mixed Indian languages; no open-weights frontier model is competitive here. |
|
| 602 |
| **STT** (speech-to-text) | Sarvam Saarika v2.5 (`saarika:v2.5`) | — | Sarvam | Best-in-class on Indian-accented English and Hinglish; WebM transcoded to 16kHz mono WAV at the gateway (`backend/providers/sarvam_stt.py`). |
|
| 603 |
| **TTS** (text-to-speech) | Sarvam Bulbul v2 (`bulbul:v2`, speaker `anushka`) | — | Sarvam | Best-in-class on Hinglish prosody; 22.05kHz output, base64 WAV from Sarvam endpoint. |
|
|
|
|
| 821 |
|
| 822 |
```bash
|
| 823 |
cp .env.example .env
|
| 824 |
+
# Edit .env and fill in the keys (see 70-docs/60-decisions/ADR-010-secret-handling.md):
|
| 825 |
+
# GOOGLE_API_KEY — Tier 0 brain (Gemini 2.0/2.5 Flash, KI-179 / ADR-040)
|
| 826 |
+
# NVIDIA_NIM_API_KEY — Tier 1 brain fallback + Judge primary
|
| 827 |
+
# OPENROUTER_API_KEY — Tier 2 diversity pool (free-tier OR models)
|
| 828 |
+
# SARVAM_API_KEY — STT / TTS / Indic translation (non-substitutable)
|
| 829 |
+
# VOYAGE_API_KEY — offline ingest embeddings only (BGE on hot path)
|
| 830 |
+
# HF_TOKEN — data-dataset pull at boot
|
| 831 |
+
# ADMIN_PASSWORD — /api/admin/* gating (post-KI-097, password-only)
|
| 832 |
+
# GROQ_API_KEY — DORMANT (no chain references it post-KI-160)
|
| 833 |
chmod 600 .env
|
| 834 |
```
|
| 835 |
|
|
@@ -14,9 +14,11 @@ FastAPI + Pydantic service that fronts every chat turn. The HTTP entry point is
|
|
| 14 |
|
| 15 |
| File | Role | Related ADR |
|
| 16 |
| --- | --- | --- |
|
| 17 |
-
| `needs_finder.py` | 9-slot fact-find
|
| 18 |
-
| `
|
| 19 |
-
|
|
|
|
|
|
|
|
| 20 |
| `fact_find_normalizer.py` | LLM-driven free-text → slot-value coercion (e.g. "32 lakh" → `3200000`). Goes through `NimChainLLM`, not a single client (KI-033). | — |
|
| 21 |
| `profile_extractor.py` | LLM extractor that pulls profile updates out of conversational asides ("by the way, my dad has diabetes"). Chain-pattern, never a hardcoded model. | [ADR-022](../70-docs/60-decisions/ADR-022-conversational-profile-updates.md) |
|
| 22 |
| `profile_store.py` | **NEW (KI-040).** Persistent named-profile JSON store under `40-data/profiles/`. O(1) name-keyed lookup; mirrors into `profile_rag` on every save. | — |
|
|
@@ -35,7 +37,15 @@ FastAPI + Pydantic service that fronts every chat turn. The HTTP entry point is
|
|
| 35 |
|
| 36 |
## Subdirectory
|
| 37 |
|
| 38 |
-
`providers/` — concrete STT / TTS / LLM / embeddings client implementations
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 39 |
|
| 40 |
## Where to read for what
|
| 41 |
|
|
|
|
| 14 |
|
| 15 |
| File | Role | Related ADR |
|
| 16 |
| --- | --- | --- |
|
| 17 |
+
| `needs_finder.py` | 9-slot fact-find SCHEMA (`GRAPH`). Post-KI-167 the `prompt_en` / `prompt_hi` strings are dead text — the LLM owns voice + cadence end-to-end via its system prompt. The data structure is retained as the schema source for `sales_brain.py`. | [ADR-039](../70-docs/60-decisions/ADR-039-llm-driven-sales-brain.md), [ADR-040](../70-docs/60-decisions/ADR-040-google-gemini-primary.md) |
|
| 18 |
+
| `sales_brain.py` | **KI-167 LLM-driven fact-find driver.** One brain call per turn against `FAST_BRAIN_CHAIN` (Gemini 2.0 Flash primary) using native provider JSON mode (`response_mime_type=application/json` on Gemini, `response_format={"type":"json_object"}` on NIM / OpenRouter). Returns `{reply, captures, slot_driving, complete}`. No `<FF>` trailer, no scripted fallback. | [ADR-039](../70-docs/60-decisions/ADR-039-llm-driven-sales-brain.md), [ADR-040](../70-docs/60-decisions/ADR-040-google-gemini-primary.md) |
|
| 19 |
+
| `sales_brain_normalizer.py` | **KI-167 deterministic post-processor.** Pure-rule mapping of the LLM's loose `captures` dict to canonical `{field: validated_value}`: alias resolution (`location` → `location_tier`), enum coercion (`Bangalore` → `metro`), INR parsing, type / bounds validation, KI-094 null-drop. No LLM calls. | [ADR-039](../70-docs/60-decisions/ADR-039-llm-driven-sales-brain.md) |
|
| 20 |
+
| ~~`fact_find_brain.py`~~ *(deleted in KI-167)* | Was the ADR-030 one-call brain with `<FF>...</FF>` trailer convention + `_canonical_fallback`. Removed entirely. | superseded by [ADR-039](../70-docs/60-decisions/ADR-039-llm-driven-sales-brain.md) |
|
| 21 |
+
| ~~`question_paraphraser.py`~~ *(deleted in KI-070)* | Was an LLM rewrite of canonical slot questions (ADR-027). Superseded first by `fact_find_brain.py`, now by `sales_brain.py`. | superseded by [ADR-039](../70-docs/60-decisions/ADR-039-llm-driven-sales-brain.md) |
|
| 22 |
| `fact_find_normalizer.py` | LLM-driven free-text → slot-value coercion (e.g. "32 lakh" → `3200000`). Goes through `NimChainLLM`, not a single client (KI-033). | — |
|
| 23 |
| `profile_extractor.py` | LLM extractor that pulls profile updates out of conversational asides ("by the way, my dad has diabetes"). Chain-pattern, never a hardcoded model. | [ADR-022](../70-docs/60-decisions/ADR-022-conversational-profile-updates.md) |
|
| 24 |
| `profile_store.py` | **NEW (KI-040).** Persistent named-profile JSON store under `40-data/profiles/`. O(1) name-keyed lookup; mirrors into `profile_rag` on every save. | — |
|
|
|
|
| 37 |
|
| 38 |
## Subdirectory
|
| 39 |
|
| 40 |
+
`providers/` — concrete STT / TTS / LLM / embeddings client implementations:
|
| 41 |
+
|
| 42 |
+
- `sarvam_stt.py` / `sarvam_tts.py` — Sarvam Saarika v2.5 + Bulbul v2.
|
| 43 |
+
- `google_gemini_llm.py` — Google AI Studio (`gemini-2.0-flash`, `gemini-2.5-flash`); Tier 0 primary on Brain Fast + Brain Main ([ADR-040](../70-docs/60-decisions/ADR-040-google-gemini-primary.md)).
|
| 44 |
+
- `nvidia_nim_llm.py` — NIM client + `NimChainLLM` dispatcher (the elector + cross-provider BACKUP machinery from KI-080). Tier 1 fallback across all chains; judge primary stays on NIM Mistral Large 3 675B.
|
| 45 |
+
- `openrouter_llm.py` — OpenRouter client; Tier 2 cross-provider diversity using OR's `models: [...]` server-side fallback (KI-176).
|
| 46 |
+
- `local_embeddings.py` — BGE-small-en-v1.5 sentence-transformers.
|
| 47 |
+
|
| 48 |
+
All LLM access goes through `NimChainLLM(chain=...)` — never instantiate a single-provider client directly. See `backend/providers/README.md`.
|
| 49 |
|
| 50 |
## Where to read for what
|
| 51 |
|
|
@@ -28,6 +28,7 @@ from backend.providers.nvidia_nim_llm import (
|
|
| 28 |
get_brain_llm,
|
| 29 |
get_fast_brain_llm,
|
| 30 |
)
|
|
|
|
| 31 |
from rag.retrieve import RetrievedChunk, format_for_llm_context, retrieve
|
| 32 |
|
| 33 |
|
|
@@ -297,12 +298,15 @@ def should_route_to_fact_find(
|
|
| 297 |
def pick_brain(intent: str, language: str) -> BrainPick:
|
| 298 |
"""Route to the reasoning brain per D-019 (2026-05-14, tiered routing).
|
| 299 |
|
| 300 |
-
|
| 301 |
-
|
| 302 |
-
|
| 303 |
-
|
| 304 |
-
|
| 305 |
-
|
|
|
|
|
|
|
|
|
|
| 306 |
|
| 307 |
Indic queries get a Sarvam-M translation pass in `handle_turn` before
|
| 308 |
retrieval, then another after reasoning to convert the English reply back
|
|
@@ -311,8 +315,8 @@ def pick_brain(intent: str, language: str) -> BrainPick:
|
|
| 311 |
"""
|
| 312 |
HEAVY_INTENTS = {"comparison", "recommendation"}
|
| 313 |
if intent in HEAVY_INTENTS:
|
| 314 |
-
return BrainPick(
|
| 315 |
-
return BrainPick(
|
| 316 |
|
| 317 |
|
| 318 |
# ---------- welcome-back greeting helpers ----------
|
|
|
|
| 28 |
get_brain_llm,
|
| 29 |
get_fast_brain_llm,
|
| 30 |
)
|
| 31 |
+
from backend.providers.tiered_brain_llm import get_tiered_brain_llm
|
| 32 |
from rag.retrieve import RetrievedChunk, format_for_llm_context, retrieve
|
| 33 |
|
| 34 |
|
|
|
|
| 298 |
def pick_brain(intent: str, language: str) -> BrainPick:
|
| 299 |
"""Route to the reasoning brain per D-019 (2026-05-14, tiered routing).
|
| 300 |
|
| 301 |
+
KI-179 (2026-05-15) — every route now goes through a 3-tier wrapper
|
| 302 |
+
(TieredBrainLLM) that prefers Google Gemini Flash, falls back to NIM,
|
| 303 |
+
then OpenRouter free-tier. The intent-based heavy vs fast split is
|
| 304 |
+
preserved at the wrapper level — heavy intents get `gemini-2.5-flash`
|
| 305 |
+
+ BRAIN_CHAIN, light intents get `gemini-2.0-flash` + FAST_BRAIN_CHAIN.
|
| 306 |
+
|
| 307 |
+
Original D-019 design (kept for context):
|
| 308 |
+
- 'comparison' / 'recommendation' → heavy brain (quality > latency)
|
| 309 |
+
- 'fact_find' / 'qa' → fast brain (latency > quality)
|
| 310 |
|
| 311 |
Indic queries get a Sarvam-M translation pass in `handle_turn` before
|
| 312 |
retrieval, then another after reasoning to convert the English reply back
|
|
|
|
| 315 |
"""
|
| 316 |
HEAVY_INTENTS = {"comparison", "recommendation"}
|
| 317 |
if intent in HEAVY_INTENTS:
|
| 318 |
+
return BrainPick(get_tiered_brain_llm(role="brain"), f"v4-pro::{intent}")
|
| 319 |
+
return BrainPick(get_tiered_brain_llm(role="fast_brain"), f"v4-flash::{intent}")
|
| 320 |
|
| 321 |
|
| 322 |
# ---------- welcome-back greeting helpers ----------
|
|
@@ -0,0 +1,233 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Google Gemini — primary LLM tier for sales_brain + orchestrator brain (KI-179).
|
| 2 |
+
|
| 3 |
+
Google AI Studio's free tier on `gemini-2.0-flash` and `gemini-2.5-flash` gives
|
| 4 |
+
1500 req/day with native JSON mode (`responseMimeType: "application/json"`).
|
| 5 |
+
This is best-in-class free quality for conversation and beats every model in
|
| 6 |
+
NIM or OpenRouter free tiers.
|
| 7 |
+
|
| 8 |
+
Wire-shape: REST API direct via httpx (no SDK dependency — matches the pattern
|
| 9 |
+
of nvidia_nim_llm.py and openrouter_llm.py).
|
| 10 |
+
|
| 11 |
+
Endpoint:
|
| 12 |
+
POST https://generativelanguage.googleapis.com/v1beta/models/{model}:generateContent?key={api_key}
|
| 13 |
+
|
| 14 |
+
Request body:
|
| 15 |
+
{
|
| 16 |
+
"contents": [
|
| 17 |
+
{"role": "user", "parts": [{"text": "..."}]},
|
| 18 |
+
{"role": "model", "parts": [{"text": "..."}]}
|
| 19 |
+
],
|
| 20 |
+
"systemInstruction": {"parts": [{"text": "..."}]},
|
| 21 |
+
"generationConfig": {
|
| 22 |
+
"temperature": 0.6,
|
| 23 |
+
"maxOutputTokens": 700,
|
| 24 |
+
"responseMimeType": "application/json"
|
| 25 |
+
}
|
| 26 |
+
}
|
| 27 |
+
|
| 28 |
+
Role mapping:
|
| 29 |
+
OpenAI shape → Gemini shape
|
| 30 |
+
role="system" → systemInstruction (separate top-level field)
|
| 31 |
+
role="user" → contents[i].role = "user"
|
| 32 |
+
role="assistant" → contents[i].role = "model"
|
| 33 |
+
|
| 34 |
+
JSON mode:
|
| 35 |
+
response_format == {"type": "json_object"} →
|
| 36 |
+
generationConfig.responseMimeType = "application/json"
|
| 37 |
+
|
| 38 |
+
The provider matches the existing `LLMProvider` interface so it slots
|
| 39 |
+
straight into both sales_brain.py (Tier 0) and the orchestrator brain main
|
| 40 |
+
path (Tier 0, via TieredBrainLLM wrapper).
|
| 41 |
+
"""
|
| 42 |
+
|
| 43 |
+
from __future__ import annotations
|
| 44 |
+
|
| 45 |
+
import asyncio
|
| 46 |
+
import os
|
| 47 |
+
from typing import Optional
|
| 48 |
+
|
| 49 |
+
import httpx
|
| 50 |
+
|
| 51 |
+
from backend.providers.base import ChatMessage, LLMProvider, LLMResult
|
| 52 |
+
|
| 53 |
+
|
| 54 |
+
GEMINI_BASE_URL = "https://generativelanguage.googleapis.com/v1beta/models"
|
| 55 |
+
DEFAULT_MODEL = "gemini-2.5-flash-lite" # KI-183 — gemini-2.0-flash retired for new accounts
|
| 56 |
+
|
| 57 |
+
|
| 58 |
+
def _to_gemini_contents(messages: list[ChatMessage]) -> tuple[Optional[str], list[dict]]:
|
| 59 |
+
"""Split an OpenAI-style message list into (systemInstruction, contents).
|
| 60 |
+
|
| 61 |
+
Gemini's API expects:
|
| 62 |
+
- `systemInstruction` as a separate top-level field (one block, the
|
| 63 |
+
concatenation of every system message in order)
|
| 64 |
+
- `contents` as a list of `{role: "user" | "model", parts: [{text: ...}]}`
|
| 65 |
+
entries (no "system" role allowed inside contents)
|
| 66 |
+
|
| 67 |
+
OpenAI's role names map cleanly:
|
| 68 |
+
- "system" → folded into systemInstruction
|
| 69 |
+
- "user" → contents[i].role = "user"
|
| 70 |
+
- "assistant" → contents[i].role = "model"
|
| 71 |
+
|
| 72 |
+
Empty / non-string content is coerced to a safe empty string so a stray
|
| 73 |
+
None never breaks the wire payload.
|
| 74 |
+
"""
|
| 75 |
+
system_chunks: list[str] = []
|
| 76 |
+
contents: list[dict] = []
|
| 77 |
+
for m in messages:
|
| 78 |
+
text = m.content if isinstance(m.content, str) else str(m.content or "")
|
| 79 |
+
if m.role == "system":
|
| 80 |
+
if text:
|
| 81 |
+
system_chunks.append(text)
|
| 82 |
+
continue
|
| 83 |
+
if m.role == "user":
|
| 84 |
+
contents.append({"role": "user", "parts": [{"text": text}]})
|
| 85 |
+
elif m.role == "assistant":
|
| 86 |
+
contents.append({"role": "model", "parts": [{"text": text}]})
|
| 87 |
+
else:
|
| 88 |
+
# Unknown role — treat as user input rather than dropping it.
|
| 89 |
+
contents.append({"role": "user", "parts": [{"text": text}]})
|
| 90 |
+
system_instruction = "\n\n".join(system_chunks) if system_chunks else None
|
| 91 |
+
return system_instruction, contents
|
| 92 |
+
|
| 93 |
+
|
| 94 |
+
class GoogleGeminiLLM(LLMProvider):
|
| 95 |
+
"""Google Gemini Flash via the AI Studio REST API.
|
| 96 |
+
|
| 97 |
+
`api_key` is read from the `GOOGLE_API_KEY` env var at *chat-time* — NOT
|
| 98 |
+
at module import time — so missing-key errors only surface when this
|
| 99 |
+
provider is actually called. That keeps cold imports cheap and lets the
|
| 100 |
+
Tier 0 layer cleanly fall through to Tier 1 (NIM) when the key is
|
| 101 |
+
unavailable (e.g., on the eval harness machine).
|
| 102 |
+
"""
|
| 103 |
+
|
| 104 |
+
name = "gemini"
|
| 105 |
+
|
| 106 |
+
def __init__(
|
| 107 |
+
self,
|
| 108 |
+
model: str = DEFAULT_MODEL,
|
| 109 |
+
api_key: Optional[str] = None,
|
| 110 |
+
timeout: float = 25.0,
|
| 111 |
+
):
|
| 112 |
+
# Defer the key-presence check to chat() — see class docstring.
|
| 113 |
+
self.api_key = api_key or os.environ.get("GOOGLE_API_KEY", "")
|
| 114 |
+
self.model = model
|
| 115 |
+
self.timeout = timeout
|
| 116 |
+
self.name = f"gemini::{model}"
|
| 117 |
+
|
| 118 |
+
async def chat(
|
| 119 |
+
self,
|
| 120 |
+
messages: list[ChatMessage],
|
| 121 |
+
temperature: float = 0.6,
|
| 122 |
+
max_tokens: int = 700,
|
| 123 |
+
response_format: Optional[dict] = None,
|
| 124 |
+
**kwargs, # absorb OR-specific kwargs like `models=[...]` — ignored here
|
| 125 |
+
) -> LLMResult:
|
| 126 |
+
if not self.api_key:
|
| 127 |
+
raise RuntimeError(
|
| 128 |
+
"GOOGLE_API_KEY not set. Get a key at "
|
| 129 |
+
"https://aistudio.google.com/app/apikey and add "
|
| 130 |
+
"GOOGLE_API_KEY=... to .env"
|
| 131 |
+
)
|
| 132 |
+
|
| 133 |
+
system_instruction, contents = _to_gemini_contents(messages)
|
| 134 |
+
|
| 135 |
+
generation_config: dict = {
|
| 136 |
+
"temperature": temperature,
|
| 137 |
+
"maxOutputTokens": max_tokens,
|
| 138 |
+
}
|
| 139 |
+
# JSON mode — Gemini's native equivalent of OpenAI response_format.
|
| 140 |
+
if response_format and response_format.get("type") == "json_object":
|
| 141 |
+
generation_config["responseMimeType"] = "application/json"
|
| 142 |
+
|
| 143 |
+
body: dict = {
|
| 144 |
+
"contents": contents,
|
| 145 |
+
"generationConfig": generation_config,
|
| 146 |
+
}
|
| 147 |
+
if system_instruction:
|
| 148 |
+
body["systemInstruction"] = {"parts": [{"text": system_instruction}]}
|
| 149 |
+
|
| 150 |
+
# API key goes in the URL query param — the Google AI Studio default.
|
| 151 |
+
# The header form (`x-goog-api-key`) also works but is undocumented for
|
| 152 |
+
# the v1beta generativelanguage endpoint.
|
| 153 |
+
url = f"{GEMINI_BASE_URL}/{self.model}:generateContent?key={self.api_key}"
|
| 154 |
+
headers = {"Content-Type": "application/json"}
|
| 155 |
+
|
| 156 |
+
# Per-phase timeouts mirroring the NIM/OpenRouter pattern: a stuck
|
| 157 |
+
# connection releases its slot on its own deadline rather than holding
|
| 158 |
+
# past the outer wait_for cancellation.
|
| 159 |
+
client_timeout = httpx.Timeout(
|
| 160 |
+
connect=2.0,
|
| 161 |
+
read=self.timeout,
|
| 162 |
+
write=2.0,
|
| 163 |
+
pool=2.0,
|
| 164 |
+
)
|
| 165 |
+
|
| 166 |
+
async with httpx.AsyncClient(timeout=client_timeout) as client:
|
| 167 |
+
resp = await client.post(url, headers=headers, json=body)
|
| 168 |
+
if resp.status_code >= 400:
|
| 169 |
+
# Surface the Google error body so the caller's log makes the
|
| 170 |
+
# root cause visible (typical failure: 429 quota exceeded or
|
| 171 |
+
# 400 prompt-block).
|
| 172 |
+
detail = ""
|
| 173 |
+
try:
|
| 174 |
+
detail = resp.text[:500]
|
| 175 |
+
except Exception:
|
| 176 |
+
pass
|
| 177 |
+
raise httpx.HTTPStatusError(
|
| 178 |
+
f"Gemini API {resp.status_code}: {detail}",
|
| 179 |
+
request=resp.request,
|
| 180 |
+
response=resp,
|
| 181 |
+
)
|
| 182 |
+
payload = resp.json()
|
| 183 |
+
|
| 184 |
+
# Response shape:
|
| 185 |
+
# {"candidates": [{"content": {"parts": [{"text": "..."}], "role": "model"},
|
| 186 |
+
# "finishReason": "STOP"}],
|
| 187 |
+
# "usageMetadata": {"promptTokenCount": N, "candidatesTokenCount": M, ...}}
|
| 188 |
+
text = ""
|
| 189 |
+
try:
|
| 190 |
+
candidates = payload.get("candidates") or []
|
| 191 |
+
if candidates:
|
| 192 |
+
parts = (candidates[0].get("content") or {}).get("parts") or []
|
| 193 |
+
# Concatenate every text part — Gemini sometimes returns
|
| 194 |
+
# multiple chunks per candidate when streaming-shape leaks
|
| 195 |
+
# into a non-streaming response.
|
| 196 |
+
text = "".join(
|
| 197 |
+
p.get("text", "") for p in parts if isinstance(p, dict)
|
| 198 |
+
)
|
| 199 |
+
except Exception:
|
| 200 |
+
text = ""
|
| 201 |
+
|
| 202 |
+
usage = payload.get("usageMetadata") or {}
|
| 203 |
+
return LLMResult(
|
| 204 |
+
text=text,
|
| 205 |
+
model=self.model,
|
| 206 |
+
prompt_tokens=usage.get("promptTokenCount"),
|
| 207 |
+
completion_tokens=usage.get("candidatesTokenCount"),
|
| 208 |
+
raw=payload,
|
| 209 |
+
)
|
| 210 |
+
|
| 211 |
+
|
| 212 |
+
# ----------------------------------------------------------------------------
|
| 213 |
+
# Factory
|
| 214 |
+
# ----------------------------------------------------------------------------
|
| 215 |
+
|
| 216 |
+
def get_gemini_llm(
|
| 217 |
+
model: str = DEFAULT_MODEL,
|
| 218 |
+
timeout: float = 25.0,
|
| 219 |
+
) -> GoogleGeminiLLM:
|
| 220 |
+
"""Return a fresh GoogleGeminiLLM client.
|
| 221 |
+
|
| 222 |
+
25s timeout matches the KI-179 brief — Gemini Flash typically responds in
|
| 223 |
+
1-3s so 25s is a generous outer bound that still bails fast on a quota /
|
| 224 |
+
network stall.
|
| 225 |
+
|
| 226 |
+
Common models:
|
| 227 |
+
- "gemini-2.0-flash" → fast, conversational, sales_brain Tier 0
|
| 228 |
+
- "gemini-2.5-flash" → slightly heavier reasoning, brain main Tier 0
|
| 229 |
+
"""
|
| 230 |
+
return GoogleGeminiLLM(model=model, timeout=timeout)
|
| 231 |
+
|
| 232 |
+
|
| 233 |
+
__all__ = ["GoogleGeminiLLM", "get_gemini_llm", "DEFAULT_MODEL"]
|
|
@@ -68,13 +68,26 @@ class OpenRouterLLM(LLMProvider):
|
|
| 68 |
temperature: float = 0.2,
|
| 69 |
max_tokens: int = 1024,
|
| 70 |
response_format: Optional[dict] = None,
|
|
|
|
| 71 |
) -> LLMResult:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 72 |
body: dict = {
|
| 73 |
-
"model":
|
| 74 |
"messages": [{"role": m.role, "content": m.content} for m in messages],
|
| 75 |
"temperature": temperature,
|
| 76 |
"max_tokens": max_tokens,
|
| 77 |
}
|
|
|
|
|
|
|
| 78 |
if response_format:
|
| 79 |
body["response_format"] = response_format
|
| 80 |
|
|
@@ -138,3 +151,52 @@ class OpenRouterLLM(LLMProvider):
|
|
| 138 |
completion_tokens=usage.get("completion_tokens"),
|
| 139 |
raw=payload,
|
| 140 |
)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 68 |
temperature: float = 0.2,
|
| 69 |
max_tokens: int = 1024,
|
| 70 |
response_format: Optional[dict] = None,
|
| 71 |
+
models: Optional[list[str]] = None,
|
| 72 |
) -> LLMResult:
|
| 73 |
+
"""Send a chat completion to OpenRouter.
|
| 74 |
+
|
| 75 |
+
KI-176 — `models` (optional list of model ids) activates OpenRouter's
|
| 76 |
+
native server-side fallback: the server tries each id in order and
|
| 77 |
+
returns the first that succeeds. When provided, `models[0]` is also
|
| 78 |
+
set as the primary `model` field (OpenRouter requires both for
|
| 79 |
+
routing; if `models` is omitted the lone `model` field is used).
|
| 80 |
+
Reference: https://openrouter.ai/docs/features/model-routing
|
| 81 |
+
"""
|
| 82 |
+
primary_model = models[0] if models else self.model
|
| 83 |
body: dict = {
|
| 84 |
+
"model": primary_model,
|
| 85 |
"messages": [{"role": m.role, "content": m.content} for m in messages],
|
| 86 |
"temperature": temperature,
|
| 87 |
"max_tokens": max_tokens,
|
| 88 |
}
|
| 89 |
+
if models:
|
| 90 |
+
body["models"] = list(models)
|
| 91 |
if response_format:
|
| 92 |
body["response_format"] = response_format
|
| 93 |
|
|
|
|
| 151 |
completion_tokens=usage.get("completion_tokens"),
|
| 152 |
raw=payload,
|
| 153 |
)
|
| 154 |
+
|
| 155 |
+
|
| 156 |
+
# ----------------------------------------------------------------------------
|
| 157 |
+
# KI-176 — Factory for the OpenRouter free-tier frontier pool.
|
| 158 |
+
#
|
| 159 |
+
# User loaded $10 on OpenRouter; that unlocks 1000 req/day across all `:free`
|
| 160 |
+
# suffix models. This factory returns an OpenRouterLLM whose `model` field is
|
| 161 |
+
# a sane single-model default; callers that want server-side fallback should
|
| 162 |
+
# pass `models=[...]` to `chat()`.
|
| 163 |
+
#
|
| 164 |
+
# Model IDs verified against the LIVE OpenRouter catalog
|
| 165 |
+
# (GET /api/v1/models, 2026-05-15). The model IDs from the KI-176 brief
|
| 166 |
+
# (gemini-2.0-flash-exp:free, llama-3.3-70b-instruct:free,
|
| 167 |
+
# hermes-3-llama-3.1-405b:free) were checked:
|
| 168 |
+
# - gemini-2.0-flash-exp:free — NOT in catalog
|
| 169 |
+
# - llama-3.3-70b-instruct:free — PRESENT but no response_format
|
| 170 |
+
# - hermes-3-llama-3.1-405b:free — PRESENT but no response_format
|
| 171 |
+
# So those would silently lose JSON mode. The IDs below are the actual
|
| 172 |
+
# free-tier set that declare `response_format` (and most also
|
| 173 |
+
# `structured_outputs`) in their `supported_parameters`:
|
| 174 |
+
# - nvidia/nemotron-3-super-120b-a12b:free (120B, structured_outputs)
|
| 175 |
+
# - qwen/qwen3-next-80b-a3b-instruct:free (80B, structured_outputs)
|
| 176 |
+
# - google/gemma-4-31b-it:free (31B, response_format)
|
| 177 |
+
# - minimax/minimax-m2.5:free (?, response_format)
|
| 178 |
+
# - google/gemma-4-26b-a4b-it:free (26B, response_format)
|
| 179 |
+
# ----------------------------------------------------------------------------
|
| 180 |
+
|
| 181 |
+
# Default JSON-mode-capable free-tier model. Callers usually override this by
|
| 182 |
+
# passing `models=[...]` to chat() — this is just the lone-model fallback.
|
| 183 |
+
DEFAULT_FREE_BRAIN_MODEL = "nvidia/nemotron-3-super-120b-a12b:free"
|
| 184 |
+
|
| 185 |
+
|
| 186 |
+
def get_openrouter_llm(
|
| 187 |
+
chain_name: str = "sales_brain",
|
| 188 |
+
model: str = DEFAULT_FREE_BRAIN_MODEL,
|
| 189 |
+
timeout: float = 60.0,
|
| 190 |
+
) -> "OpenRouterLLM":
|
| 191 |
+
"""Return a fresh OpenRouterLLM ready for sales_brain.
|
| 192 |
+
|
| 193 |
+
The returned client is single-use-shaped (one chat() call per turn) and
|
| 194 |
+
intended to be passed `models=[primary, fallback1, fallback2]` so the
|
| 195 |
+
OpenRouter server handles in-pool failover before the caller falls back
|
| 196 |
+
to the NIM chain.
|
| 197 |
+
"""
|
| 198 |
+
return OpenRouterLLM(
|
| 199 |
+
model=model,
|
| 200 |
+
timeout=timeout,
|
| 201 |
+
chain_name=chain_name,
|
| 202 |
+
)
|
|
@@ -0,0 +1,229 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""TieredBrainLLM — 3-tier wrapper for the orchestrator's brain main path (KI-179).
|
| 2 |
+
|
| 3 |
+
Architecture:
|
| 4 |
+
|
| 5 |
+
Tier 0 — Google Gemini Flash (best free quality, 1500 req/day)
|
| 6 |
+
Tier 1 — NIM brain chain (NimChainLLM, multi-candidate election)
|
| 7 |
+
Tier 2 — OpenRouter free pool (cross-provider safety net)
|
| 8 |
+
|
| 9 |
+
Why a wrapper class instead of inlining the tiers in orchestrator.py?
|
| 10 |
+
|
| 11 |
+
The orchestrator's brain main path is one of the busiest call-sites in the
|
| 12 |
+
system — faithfulness gates, cross-check retry, drift recovery, eval logging
|
| 13 |
+
all hang off it. Inlining a 3-tier try/except cascade there would explode
|
| 14 |
+
the cyclomatic complexity of an already-long function. Wrapping the
|
| 15 |
+
3-tier dance behind the SAME `LLMProvider.chat()` signature as `NimChainLLM`
|
| 16 |
+
lets us swap in the wrapper at the construction site with zero downstream
|
| 17 |
+
changes — the existing `await pick.provider.chat(...)` call shape is
|
| 18 |
+
preserved.
|
| 19 |
+
|
| 20 |
+
Match the existing `LLMProvider` interface so this drops into the same
|
| 21 |
+
slot the orchestrator currently fills with `get_brain_llm()` /
|
| 22 |
+
`get_fast_brain_llm()`. `self.model` and `self.name` are updated after a
|
| 23 |
+
successful call to reflect which tier actually served — preserves the
|
| 24 |
+
KI-080 audit trail.
|
| 25 |
+
"""
|
| 26 |
+
|
| 27 |
+
from __future__ import annotations
|
| 28 |
+
|
| 29 |
+
import asyncio
|
| 30 |
+
import logging
|
| 31 |
+
from typing import Optional
|
| 32 |
+
|
| 33 |
+
from backend.providers.base import ChatMessage, LLMProvider, LLMResult
|
| 34 |
+
from backend.providers.google_gemini_llm import get_gemini_llm
|
| 35 |
+
from backend.providers.nvidia_nim_llm import (
|
| 36 |
+
get_brain_llm,
|
| 37 |
+
get_fast_brain_llm,
|
| 38 |
+
)
|
| 39 |
+
from backend.providers.openrouter_llm import get_openrouter_llm
|
| 40 |
+
|
| 41 |
+
|
| 42 |
+
# Mirrors sales_brain.py's free-tier pool: every model declares
|
| 43 |
+
# response_format in its supported_parameters so the OR server's native
|
| 44 |
+
# fallback walks an JSON-mode-aware list.
|
| 45 |
+
_OR_MODELS: list[str] = [
|
| 46 |
+
"nvidia/nemotron-3-super-120b-a12b:free",
|
| 47 |
+
"qwen/qwen3-next-80b-a3b-instruct:free",
|
| 48 |
+
"google/gemma-4-31b-it:free",
|
| 49 |
+
]
|
| 50 |
+
|
| 51 |
+
|
| 52 |
+
class TieredBrainLLM(LLMProvider):
|
| 53 |
+
"""3-tier brain wrapper: Gemini → NIM → OpenRouter.
|
| 54 |
+
|
| 55 |
+
`role` is one of "brain" | "fast_brain" — controls which NIM chain
|
| 56 |
+
factory is used for Tier 1:
|
| 57 |
+
- "brain" → get_brain_llm() (BRAIN_CHAIN, 20s per-link)
|
| 58 |
+
- "fast_brain" → get_fast_brain_llm() (FAST_BRAIN_CHAIN, 6s per-link)
|
| 59 |
+
|
| 60 |
+
`gemini_model` defaults to "gemini-2.5-flash" for the heavy brain (slightly
|
| 61 |
+
higher quality on complex synthesis) and can be overridden to
|
| 62 |
+
"gemini-2.0-flash" for the fast brain (lower latency).
|
| 63 |
+
|
| 64 |
+
`per_tier_timeout` is the outer asyncio.wait_for cap applied to EACH
|
| 65 |
+
tier individually. Mirrors the 30s budget the orchestrator's existing
|
| 66 |
+
brain call uses (see orchestrator.py KI-098).
|
| 67 |
+
"""
|
| 68 |
+
|
| 69 |
+
name = "tiered-brain"
|
| 70 |
+
|
| 71 |
+
def __init__(
|
| 72 |
+
self,
|
| 73 |
+
role: str = "brain",
|
| 74 |
+
gemini_model: str = "gemini-2.5-flash",
|
| 75 |
+
per_tier_timeout: float = 30.0,
|
| 76 |
+
):
|
| 77 |
+
if role not in ("brain", "fast_brain"):
|
| 78 |
+
raise ValueError(f"role must be 'brain' or 'fast_brain', got {role!r}")
|
| 79 |
+
self.role = role
|
| 80 |
+
self.gemini_model = gemini_model
|
| 81 |
+
self.per_tier_timeout = per_tier_timeout
|
| 82 |
+
# Start with the Tier 0 model as the "current" model — `chat()`
|
| 83 |
+
# rewrites this to whichever tier actually served the request.
|
| 84 |
+
self.model = gemini_model
|
| 85 |
+
self.name = f"tiered-brain::{role}::gemini:{gemini_model}"
|
| 86 |
+
|
| 87 |
+
async def chat(
|
| 88 |
+
self,
|
| 89 |
+
messages: list[ChatMessage],
|
| 90 |
+
temperature: float = 0.2,
|
| 91 |
+
max_tokens: int = 1024,
|
| 92 |
+
response_format: Optional[dict] = None,
|
| 93 |
+
**kwargs, # absorb provider-specific kwargs (e.g. OR's `models=[...]`)
|
| 94 |
+
) -> LLMResult:
|
| 95 |
+
gemini_exc: Optional[BaseException] = None
|
| 96 |
+
nim_exc: Optional[BaseException] = None
|
| 97 |
+
or_exc: Optional[BaseException] = None
|
| 98 |
+
|
| 99 |
+
# ---- Tier 0 — Google Gemini ----
|
| 100 |
+
try:
|
| 101 |
+
gemini_llm = get_gemini_llm(model=self.gemini_model, timeout=25.0)
|
| 102 |
+
result = await asyncio.wait_for(
|
| 103 |
+
gemini_llm.chat(
|
| 104 |
+
messages=messages,
|
| 105 |
+
temperature=temperature,
|
| 106 |
+
max_tokens=max_tokens,
|
| 107 |
+
response_format=response_format,
|
| 108 |
+
),
|
| 109 |
+
timeout=self.per_tier_timeout,
|
| 110 |
+
)
|
| 111 |
+
text = (result.text or "").strip()
|
| 112 |
+
if text:
|
| 113 |
+
# Stamp the actually-served model so faithfulness can exclude it
|
| 114 |
+
# from the judge chain (cross-family-grading invariant).
|
| 115 |
+
self.model = result.model or self.gemini_model
|
| 116 |
+
self.name = f"tiered-brain::{self.role}::gemini:{self.model}"
|
| 117 |
+
return result
|
| 118 |
+
# Empty text from Gemini → treat as failure, fall through.
|
| 119 |
+
logging.info(
|
| 120 |
+
"TieredBrainLLM[%s]: Gemini returned empty text, falling to NIM",
|
| 121 |
+
self.role,
|
| 122 |
+
)
|
| 123 |
+
except (asyncio.CancelledError, KeyboardInterrupt, SystemExit):
|
| 124 |
+
raise
|
| 125 |
+
except asyncio.TimeoutError as e:
|
| 126 |
+
gemini_exc = e
|
| 127 |
+
logging.info(
|
| 128 |
+
"TieredBrainLLM[%s]: Gemini timed out, falling to NIM",
|
| 129 |
+
self.role,
|
| 130 |
+
)
|
| 131 |
+
except Exception as e: # noqa: BLE001 — fall through on ANY provider error
|
| 132 |
+
gemini_exc = e
|
| 133 |
+
logging.info(
|
| 134 |
+
"TieredBrainLLM[%s]: Gemini raised %s, falling to NIM: %s",
|
| 135 |
+
self.role, type(e).__name__, str(e)[:200],
|
| 136 |
+
)
|
| 137 |
+
|
| 138 |
+
# ---- Tier 1 — NIM chain ----
|
| 139 |
+
try:
|
| 140 |
+
nim_llm = (
|
| 141 |
+
get_brain_llm() if self.role == "brain" else get_fast_brain_llm()
|
| 142 |
+
)
|
| 143 |
+
result = await asyncio.wait_for(
|
| 144 |
+
nim_llm.chat(
|
| 145 |
+
messages=messages,
|
| 146 |
+
temperature=temperature,
|
| 147 |
+
max_tokens=max_tokens,
|
| 148 |
+
response_format=response_format,
|
| 149 |
+
),
|
| 150 |
+
timeout=self.per_tier_timeout,
|
| 151 |
+
)
|
| 152 |
+
text = (result.text or "").strip()
|
| 153 |
+
if text:
|
| 154 |
+
self.model = result.model or getattr(nim_llm, "model", "nim")
|
| 155 |
+
self.name = f"tiered-brain::{self.role}::nim:{self.model}"
|
| 156 |
+
return result
|
| 157 |
+
logging.info(
|
| 158 |
+
"TieredBrainLLM[%s]: NIM returned empty text, falling to OpenRouter",
|
| 159 |
+
self.role,
|
| 160 |
+
)
|
| 161 |
+
except (asyncio.CancelledError, KeyboardInterrupt, SystemExit):
|
| 162 |
+
raise
|
| 163 |
+
except asyncio.TimeoutError as e:
|
| 164 |
+
nim_exc = e
|
| 165 |
+
logging.info(
|
| 166 |
+
"TieredBrainLLM[%s]: NIM timed out, falling to OpenRouter",
|
| 167 |
+
self.role,
|
| 168 |
+
)
|
| 169 |
+
except Exception as e: # noqa: BLE001
|
| 170 |
+
nim_exc = e
|
| 171 |
+
logging.info(
|
| 172 |
+
"TieredBrainLLM[%s]: NIM raised %s, falling to OpenRouter: %s",
|
| 173 |
+
self.role, type(e).__name__, str(e)[:200],
|
| 174 |
+
)
|
| 175 |
+
|
| 176 |
+
# ---- Tier 2 — OpenRouter free pool ----
|
| 177 |
+
try:
|
| 178 |
+
or_llm = get_openrouter_llm(chain_name=f"orchestrator_{self.role}")
|
| 179 |
+
result = await asyncio.wait_for(
|
| 180 |
+
or_llm.chat(
|
| 181 |
+
messages=messages,
|
| 182 |
+
temperature=temperature,
|
| 183 |
+
max_tokens=max_tokens,
|
| 184 |
+
response_format=response_format,
|
| 185 |
+
models=_OR_MODELS,
|
| 186 |
+
),
|
| 187 |
+
timeout=self.per_tier_timeout,
|
| 188 |
+
)
|
| 189 |
+
self.model = result.model or "openrouter"
|
| 190 |
+
self.name = f"tiered-brain::{self.role}::or:{self.model}"
|
| 191 |
+
return result
|
| 192 |
+
except (asyncio.CancelledError, KeyboardInterrupt, SystemExit):
|
| 193 |
+
raise
|
| 194 |
+
except Exception as e: # noqa: BLE001
|
| 195 |
+
or_exc = e
|
| 196 |
+
|
| 197 |
+
# All three tiers failed — raise a single combined RuntimeError so
|
| 198 |
+
# the orchestrator's existing outer-error handling treats it as a
|
| 199 |
+
# standard brain failure (it already handles `pick.provider.chat`
|
| 200 |
+
# raising — see the wait_for cap at orchestrator.py KI-098).
|
| 201 |
+
raise RuntimeError(
|
| 202 |
+
f"TieredBrainLLM[{self.role}] all tiers failed: "
|
| 203 |
+
f"gemini={type(gemini_exc).__name__ if gemini_exc else 'n/a'}, "
|
| 204 |
+
f"nim={type(nim_exc).__name__ if nim_exc else 'n/a'}, "
|
| 205 |
+
f"or={type(or_exc).__name__ if or_exc else 'n/a'}"
|
| 206 |
+
) from or_exc
|
| 207 |
+
|
| 208 |
+
|
| 209 |
+
def get_tiered_brain_llm(
|
| 210 |
+
role: str = "brain",
|
| 211 |
+
gemini_model: Optional[str] = None,
|
| 212 |
+
) -> TieredBrainLLM:
|
| 213 |
+
"""Factory for the orchestrator's brain main path.
|
| 214 |
+
|
| 215 |
+
Defaults:
|
| 216 |
+
role="brain" → gemini-2.5-flash + BRAIN_CHAIN (heavy synthesis)
|
| 217 |
+
role="fast_brain" → gemini-2.5-flash-lite + FAST_BRAIN_CHAIN (voice latency)
|
| 218 |
+
|
| 219 |
+
KI-183 (2026-05-15) — gemini-2.0-flash retired for new accounts;
|
| 220 |
+
gemini-2.5-flash-lite is the replacement (faster, cleaner JSON).
|
| 221 |
+
"""
|
| 222 |
+
if gemini_model is None:
|
| 223 |
+
gemini_model = (
|
| 224 |
+
"gemini-2.5-flash" if role == "brain" else "gemini-2.5-flash-lite"
|
| 225 |
+
)
|
| 226 |
+
return TieredBrainLLM(role=role, gemini_model=gemini_model)
|
| 227 |
+
|
| 228 |
+
|
| 229 |
+
__all__ = ["TieredBrainLLM", "get_tiered_brain_llm"]
|
|
@@ -56,10 +56,28 @@ from typing import Any, Optional
|
|
| 56 |
|
| 57 |
from backend.needs_finder import Profile
|
| 58 |
from backend.providers.base import ChatMessage
|
|
|
|
| 59 |
from backend.providers.nvidia_nim_llm import get_fast_brain_llm
|
|
|
|
| 60 |
from backend.sales_brain_normalizer import normalize_captures
|
| 61 |
|
| 62 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 63 |
# ----------------------------------------------------------------------------
|
| 64 |
# Public result dataclass — the contract WS2 will consume.
|
| 65 |
# ----------------------------------------------------------------------------
|
|
@@ -376,44 +394,150 @@ async def drive_sales_brain(
|
|
| 376 |
messages.append(ChatMessage(role=role, content=str(content)))
|
| 377 |
messages.append(ChatMessage(role="user", content=user_text or ""))
|
| 378 |
|
| 379 |
-
#
|
| 380 |
-
# (
|
| 381 |
-
#
|
| 382 |
-
#
|
| 383 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 384 |
try:
|
| 385 |
-
|
| 386 |
-
|
| 387 |
-
|
| 388 |
-
|
| 389 |
-
|
| 390 |
-
|
| 391 |
-
),
|
| 392 |
-
timeout=_TIMEOUT_S,
|
| 393 |
)
|
| 394 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 395 |
elapsed = time.time() - t0
|
| 396 |
logging.warning(
|
| 397 |
-
"sales_brain
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 398 |
)
|
| 399 |
-
|
| 400 |
-
|
| 401 |
-
|
| 402 |
-
|
| 403 |
-
brain_used="sales_brain::error:timeout",
|
| 404 |
-
error_reason=f"timeout_after_{elapsed:.1f}s",
|
| 405 |
-
)
|
| 406 |
-
except Exception as e: # noqa: BLE001 — broad catch is intentional here
|
| 407 |
-
logging.warning(
|
| 408 |
-
"sales_brain LLM call failed (session=%s): %s: %s",
|
| 409 |
-
session_id, type(e).__name__, str(e)[:200],
|
| 410 |
)
|
|
|
|
| 411 |
return SalesBrainResult(
|
| 412 |
reply_text="",
|
| 413 |
captured_updates={},
|
| 414 |
ready_for_recommendations=False,
|
| 415 |
-
brain_used=f"sales_brain::error:
|
| 416 |
-
error_reason=
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 417 |
)
|
| 418 |
|
| 419 |
# Parse the JSON object from the LLM response.
|
|
@@ -510,7 +634,7 @@ async def drive_sales_brain(
|
|
| 510 |
reply_text=retry_reply,
|
| 511 |
captured_updates=captured_updates,
|
| 512 |
ready_for_recommendations=ready_for_recommendations,
|
| 513 |
-
brain_used=f"sales_brain::
|
| 514 |
raw_json=retry_parsed,
|
| 515 |
error_reason=None,
|
| 516 |
)
|
|
@@ -533,7 +657,7 @@ async def drive_sales_brain(
|
|
| 533 |
reply_text=reply_text,
|
| 534 |
captured_updates=captured_updates,
|
| 535 |
ready_for_recommendations=ready_for_recommendations,
|
| 536 |
-
brain_used=f"sales_brain::
|
| 537 |
raw_json=parsed,
|
| 538 |
error_reason=None,
|
| 539 |
)
|
|
|
|
| 56 |
|
| 57 |
from backend.needs_finder import Profile
|
| 58 |
from backend.providers.base import ChatMessage
|
| 59 |
+
from backend.providers.google_gemini_llm import get_gemini_llm
|
| 60 |
from backend.providers.nvidia_nim_llm import get_fast_brain_llm
|
| 61 |
+
from backend.providers.openrouter_llm import get_openrouter_llm
|
| 62 |
from backend.sales_brain_normalizer import normalize_captures
|
| 63 |
|
| 64 |
|
| 65 |
+
# ----------------------------------------------------------------------------
|
| 66 |
+
# KI-176 — OpenRouter free-tier frontier pool (PRIMARY tier).
|
| 67 |
+
#
|
| 68 |
+
# User loaded $10 on OpenRouter which unlocks 1000 req/day on `:free` models.
|
| 69 |
+
# These IDs were verified against the LIVE OpenRouter catalog and ALL declare
|
| 70 |
+
# `response_format` in their supported_parameters (the first two also declare
|
| 71 |
+
# `structured_outputs`). The list is passed to OpenRouter's native `models`
|
| 72 |
+
# array for server-side in-pool fallback BEFORE we fall through to NIM.
|
| 73 |
+
# ----------------------------------------------------------------------------
|
| 74 |
+
_OR_MODELS: list[str] = [
|
| 75 |
+
"nvidia/nemotron-3-super-120b-a12b:free", # 120B, structured_outputs
|
| 76 |
+
"qwen/qwen3-next-80b-a3b-instruct:free", # 80B, structured_outputs
|
| 77 |
+
"google/gemma-4-31b-it:free", # 31B, response_format
|
| 78 |
+
]
|
| 79 |
+
|
| 80 |
+
|
| 81 |
# ----------------------------------------------------------------------------
|
| 82 |
# Public result dataclass — the contract WS2 will consume.
|
| 83 |
# ----------------------------------------------------------------------------
|
|
|
|
| 394 |
messages.append(ChatMessage(role=role, content=str(content)))
|
| 395 |
messages.append(ChatMessage(role="user", content=user_text or ""))
|
| 396 |
|
| 397 |
+
# KI-179 (2026-05-15) — 3-tier LLM stack with Google Gemini 2.0 Flash
|
| 398 |
+
# PRIMARY (Tier 0), NIM fast-brain chain Tier 1, OpenRouter free-tier
|
| 399 |
+
# pool Tier 2. Google AI Studio's free tier is 1500 req/day with native
|
| 400 |
+
# JSON mode; quality > NIM/OR free tiers for conversational fact-find.
|
| 401 |
+
#
|
| 402 |
+
# Order: Gemini → NIM → OR → final error.
|
| 403 |
+
# `llm` is rebound to whichever provider served so the empty-reply
|
| 404 |
+
# retry path (KI-169, further down) re-uses the SAME provider that
|
| 405 |
+
# just answered — no re-tier on retry.
|
| 406 |
+
llm = None
|
| 407 |
+
result = None
|
| 408 |
+
served_tier: str = "" # "gemini" | "nim" | "or"
|
| 409 |
+
gemini_exc: Optional[BaseException] = None
|
| 410 |
+
nim_exc: Optional[BaseException] = None
|
| 411 |
+
or_exc: Optional[BaseException] = None
|
| 412 |
+
|
| 413 |
+
# ---- Tier 0 — Google Gemini 2.5 Flash Lite ----
|
| 414 |
+
# KI-183 (2026-05-15) — gemini-2.0-flash is retired for new accounts
|
| 415 |
+
# (HTTP 404 "no longer available to new users"). gemini-2.5-flash-lite
|
| 416 |
+
# is the supported replacement: faster (~900ms vs ~1100ms for 2.5-flash),
|
| 417 |
+
# cleaner JSON-mode output, same free quota.
|
| 418 |
+
gemini_llm = None
|
| 419 |
try:
|
| 420 |
+
gemini_llm = get_gemini_llm(model="gemini-2.5-flash-lite", timeout=25.0)
|
| 421 |
+
except Exception as e: # noqa: BLE001
|
| 422 |
+
gemini_exc = e
|
| 423 |
+
logging.info(
|
| 424 |
+
"sales_brain: Gemini unavailable, skipping to NIM (session=%s): %s: %s",
|
| 425 |
+
session_id, type(e).__name__, str(e)[:200],
|
|
|
|
|
|
|
| 426 |
)
|
| 427 |
+
|
| 428 |
+
if gemini_llm is not None:
|
| 429 |
+
try:
|
| 430 |
+
result = await asyncio.wait_for(
|
| 431 |
+
gemini_llm.chat(
|
| 432 |
+
messages=messages,
|
| 433 |
+
temperature=_TEMPERATURE,
|
| 434 |
+
max_tokens=_MAX_TOKENS,
|
| 435 |
+
response_format={"type": "json_object"},
|
| 436 |
+
),
|
| 437 |
+
timeout=_TIMEOUT_S,
|
| 438 |
+
)
|
| 439 |
+
llm = gemini_llm
|
| 440 |
+
served_tier = "gemini"
|
| 441 |
+
except asyncio.TimeoutError as e:
|
| 442 |
+
gemini_exc = e
|
| 443 |
+
logging.info(
|
| 444 |
+
"sales_brain: Gemini timed out, falling back to NIM (session=%s)",
|
| 445 |
+
session_id,
|
| 446 |
+
)
|
| 447 |
+
except Exception as e: # noqa: BLE001
|
| 448 |
+
gemini_exc = e
|
| 449 |
+
logging.info(
|
| 450 |
+
"sales_brain: Gemini raised %s, falling back to NIM (session=%s): %s",
|
| 451 |
+
type(e).__name__, session_id, str(e)[:200],
|
| 452 |
+
)
|
| 453 |
+
|
| 454 |
+
# ---- Tier 1 — NIM fast-brain chain ----
|
| 455 |
+
if result is None:
|
| 456 |
+
nim_llm = get_fast_brain_llm()
|
| 457 |
+
try:
|
| 458 |
+
result = await asyncio.wait_for(
|
| 459 |
+
nim_llm.chat(
|
| 460 |
+
messages=messages,
|
| 461 |
+
temperature=_TEMPERATURE,
|
| 462 |
+
max_tokens=_MAX_TOKENS,
|
| 463 |
+
response_format={"type": "json_object"},
|
| 464 |
+
),
|
| 465 |
+
timeout=_TIMEOUT_S,
|
| 466 |
+
)
|
| 467 |
+
llm = nim_llm
|
| 468 |
+
served_tier = "nim"
|
| 469 |
+
except asyncio.TimeoutError as e:
|
| 470 |
+
nim_exc = e
|
| 471 |
+
logging.info(
|
| 472 |
+
"sales_brain: NIM timed out, falling back to OpenRouter (session=%s)",
|
| 473 |
+
session_id,
|
| 474 |
+
)
|
| 475 |
+
except Exception as e: # noqa: BLE001
|
| 476 |
+
nim_exc = e
|
| 477 |
+
logging.info(
|
| 478 |
+
"sales_brain: NIM raised %s, falling back to OpenRouter (session=%s): %s",
|
| 479 |
+
type(e).__name__, session_id, str(e)[:200],
|
| 480 |
+
)
|
| 481 |
+
|
| 482 |
+
# ---- Tier 2 — OpenRouter free-tier frontier pool ----
|
| 483 |
+
if result is None:
|
| 484 |
+
or_llm = None
|
| 485 |
+
try:
|
| 486 |
+
or_llm = get_openrouter_llm(chain_name="sales_brain")
|
| 487 |
+
except Exception as e: # noqa: BLE001 — missing API key etc.
|
| 488 |
+
or_exc = e
|
| 489 |
+
logging.info(
|
| 490 |
+
"sales_brain: OpenRouter unavailable (session=%s): %s: %s",
|
| 491 |
+
session_id, type(e).__name__, str(e)[:200],
|
| 492 |
+
)
|
| 493 |
+
|
| 494 |
+
if or_llm is not None:
|
| 495 |
+
try:
|
| 496 |
+
result = await asyncio.wait_for(
|
| 497 |
+
or_llm.chat(
|
| 498 |
+
messages=messages,
|
| 499 |
+
temperature=_TEMPERATURE,
|
| 500 |
+
max_tokens=_MAX_TOKENS,
|
| 501 |
+
response_format={"type": "json_object"},
|
| 502 |
+
models=_OR_MODELS,
|
| 503 |
+
),
|
| 504 |
+
timeout=_TIMEOUT_S,
|
| 505 |
+
)
|
| 506 |
+
llm = or_llm
|
| 507 |
+
served_tier = "or"
|
| 508 |
+
except asyncio.TimeoutError as e:
|
| 509 |
+
or_exc = e
|
| 510 |
+
except Exception as e: # noqa: BLE001
|
| 511 |
+
or_exc = e
|
| 512 |
+
|
| 513 |
+
# ---- All three tiers exhausted — final error ----
|
| 514 |
+
if result is None:
|
| 515 |
elapsed = time.time() - t0
|
| 516 |
logging.warning(
|
| 517 |
+
"sales_brain ALL TIERS FAILED (session=%s, %.1fs, "
|
| 518 |
+
"gemini_exc=%s, nim_exc=%s, or_exc=%s)",
|
| 519 |
+
session_id, elapsed,
|
| 520 |
+
f"{type(gemini_exc).__name__}" if gemini_exc else "n/a",
|
| 521 |
+
f"{type(nim_exc).__name__}" if nim_exc else "n/a",
|
| 522 |
+
f"{type(or_exc).__name__}" if or_exc else "n/a",
|
| 523 |
)
|
| 524 |
+
# Classify reason: timeout-everywhere is its own bucket vs mixed errors.
|
| 525 |
+
all_timeouts = all(
|
| 526 |
+
isinstance(e, asyncio.TimeoutError) for e in (gemini_exc, nim_exc, or_exc)
|
| 527 |
+
if e is not None
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 528 |
)
|
| 529 |
+
reason_tag = "timeout_all_tiers" if all_timeouts else "llm_error_all_tiers"
|
| 530 |
return SalesBrainResult(
|
| 531 |
reply_text="",
|
| 532 |
captured_updates={},
|
| 533 |
ready_for_recommendations=False,
|
| 534 |
+
brain_used=f"sales_brain::error:{reason_tag}",
|
| 535 |
+
error_reason=(
|
| 536 |
+
f"gemini_exc={type(gemini_exc).__name__ if gemini_exc else 'n/a'}; "
|
| 537 |
+
f"nim_exc={type(nim_exc).__name__ if nim_exc else 'n/a'}; "
|
| 538 |
+
f"or_exc={type(or_exc).__name__ if or_exc else 'n/a'}; "
|
| 539 |
+
f"elapsed={elapsed:.1f}s"
|
| 540 |
+
),
|
| 541 |
)
|
| 542 |
|
| 543 |
# Parse the JSON object from the LLM response.
|
|
|
|
| 634 |
reply_text=retry_reply,
|
| 635 |
captured_updates=captured_updates,
|
| 636 |
ready_for_recommendations=ready_for_recommendations,
|
| 637 |
+
brain_used=f"sales_brain::{served_tier}:{retry_model}::retry",
|
| 638 |
raw_json=retry_parsed,
|
| 639 |
error_reason=None,
|
| 640 |
)
|
|
|
|
| 657 |
reply_text=reply_text,
|
| 658 |
captured_updates=captured_updates,
|
| 659 |
ready_for_recommendations=ready_for_recommendations,
|
| 660 |
+
brain_used=f"sales_brain::{served_tier}:{served_model}",
|
| 661 |
raw_json=parsed,
|
| 662 |
error_reason=None,
|
| 663 |
)
|
|
@@ -10,7 +10,7 @@ Numerical accuracy + grounding eval. Walks a fixed list of curated questions thr
|
|
| 10 |
| --- | --- |
|
| 11 |
| `generate_gold.py` | Builds `gold_qa.json` from `40-data/policy_facts/`: for each curated field with a verbatim quote, emits a natural-language question + expected answer + expected citation. |
|
| 12 |
| `gold_qa.json` | 96-Q gold set. Each entry: `{policy_id, question, expected_answer, expected_regex, source_quote, source_pdf}`. |
|
| 13 |
-
| `run.py` | Runner. For each pair: calls `backend.orchestrator.handle_turn` with `policy_filter_ids=[pair.policy_id]` so retrieval is scoped to the policy under test. Grades each reply twice — regex hard-facts + NIM Mistral Large 3
|
| 14 |
| `results.json` | Machine-readable last-run results. |
|
| 15 |
| `results.md` | Human-readable last-run report — per-question pass/fail, per-category rollup, hallucination breakdown. |
|
| 16 |
| `info_source_map.json` | Generated by `tools/info_source_map.py`. Claim → URL → verdict (✅ 798 / ⚠️ 321 / ❌ 0 / ⏳ 1385 as of 2026-05-14). The canonical source-grounding KPI. |
|
|
@@ -39,7 +39,7 @@ python -m eval.generate_gold
|
|
| 39 |
| Layer | What it checks | Output field |
|
| 40 |
| --- | --- | --- |
|
| 41 |
| Regex hard-facts | Expected numeric / boolean / waiting-period value appears verbatim in `reply_text`. | `regex_pass` |
|
| 42 |
-
| LLM judge (NIM Mistral Large 3 675B primary;
|
| 43 |
| Faithfulness gate | The 4-gate guard in `backend/faithfulness.py` already ran during `handle_turn`; eval records whether it blocked. | `blocked` |
|
| 44 |
|
| 45 |
## Acceptance bar
|
|
@@ -51,4 +51,5 @@ Per `80-audit/ENTERPRISE_AUDIT.md` D-003: **≥ 90% on the 96-Q gold-QA set is t
|
|
| 51 |
- [`backend/faithfulness.py`](../backend/faithfulness.py) — the 4-gate guard that produces the `blocked` verdict
|
| 52 |
- [`kb/AUDIT_TRAIL.md`](../kb/AUDIT_TRAIL.md) § "Live bot replies" — provenance trail for any flagged reply
|
| 53 |
- `tools/info_source_map.py` — generator of `info_source_map.json`
|
| 54 |
-
- [ADR-014](../70-docs/60-decisions/ADR-014-groq-llama-grader.md) (superseded — but the
|
|
|
|
|
|
| 10 |
| --- | --- |
|
| 11 |
| `generate_gold.py` | Builds `gold_qa.json` from `40-data/policy_facts/`: for each curated field with a verbatim quote, emits a natural-language question + expected answer + expected citation. |
|
| 12 |
| `gold_qa.json` | 96-Q gold set. Each entry: `{policy_id, question, expected_answer, expected_regex, source_quote, source_pdf}`. |
|
| 13 |
+
| `run.py` | Runner. For each pair: calls `backend.orchestrator.handle_turn` with `policy_filter_ids=[pair.policy_id]` so retrieval is scoped to the policy under test. Grades each reply twice — regex hard-facts + `JUDGE_CHAIN` (NIM Mistral Large 3 675B primary; different family from the Gemini brain → non-circular). Falls through to NIM Llama-4 Maverick → OpenRouter Qwen 80B `:free` → NIM Nemotron 49B at the tail. Emits `brain_used` like `sales_brain::continue` / `sales_brain::complete` / `brain_main::<elected_model>` / `brain_main::graceful_error`. |
|
| 14 |
| `results.json` | Machine-readable last-run results. |
|
| 15 |
| `results.md` | Human-readable last-run report — per-question pass/fail, per-category rollup, hallucination breakdown. |
|
| 16 |
| `info_source_map.json` | Generated by `tools/info_source_map.py`. Claim → URL → verdict (✅ 798 / ⚠️ 321 / ❌ 0 / ⏳ 1385 as of 2026-05-14). The canonical source-grounding KPI. |
|
|
|
|
| 39 |
| Layer | What it checks | Output field |
|
| 40 |
| --- | --- | --- |
|
| 41 |
| Regex hard-facts | Expected numeric / boolean / waiting-period value appears verbatim in `reply_text`. | `regex_pass` |
|
| 42 |
+
| LLM judge (`JUDGE_CHAIN`: NIM Mistral Large 3 675B primary; NIM Llama-4 Maverick / OR Qwen 80B `:free` / NIM Nemotron 49B fallbacks per [ADR-040](../70-docs/60-decisions/ADR-040-google-gemini-primary.md)) | Reply *semantically* answers the question and cites the expected source. Different family from the Gemini brain → non-circular. | `judge_pass` |
|
| 43 |
| Faithfulness gate | The 4-gate guard in `backend/faithfulness.py` already ran during `handle_turn`; eval records whether it blocked. | `blocked` |
|
| 44 |
|
| 45 |
## Acceptance bar
|
|
|
|
| 51 |
- [`backend/faithfulness.py`](../backend/faithfulness.py) — the 4-gate guard that produces the `blocked` verdict
|
| 52 |
- [`kb/AUDIT_TRAIL.md`](../kb/AUDIT_TRAIL.md) § "Live bot replies" — provenance trail for any flagged reply
|
| 53 |
- `tools/info_source_map.py` — generator of `info_source_map.json`
|
| 54 |
+
- [ADR-014](../70-docs/60-decisions/ADR-014-groq-llama-grader.md) (superseded — but the cross-family judge principle survives; current judge is NIM Mistral 675B, brain is Gemini)
|
| 55 |
+
- [ADR-040](../70-docs/60-decisions/ADR-040-google-gemini-primary.md) — current chain shape (Google → NIM → OpenRouter)
|
|
@@ -13,12 +13,15 @@ For Claude / agent-specific rules (e.g. the "this is NOT the Next.js you know" n
|
|
| 13 |
| `src/app/globals.css` | Tailwind v4 entrypoint + the shadcn/ui token layer ([ADR-013](../70-docs/60-decisions/ADR-013-tailwind-shadcn-ui.md)). |
|
| 14 |
| `src/lib/api.ts` | Typed API client. Generated types via `openapi-typescript` from the FastAPI OpenAPI schema ([ADR-015](../70-docs/60-decisions/ADR-015-openapi-typescript-codegen.md)). |
|
| 15 |
| `src/lib/useLiveConversation.ts` | The continuously-open-mic VAD hook that powers Live mode + barge-in. State persists in `localStorage.insurance_live_pref` ([ADR-028](../70-docs/60-decisions/ADR-028-voice-ux-single-default-mode.md)). |
|
|
|
|
| 16 |
| `src/lib/i18n.ts` | EN ↔ हिं strings + the 13-term `GLOSSARY` mirrored to [`kb/methodology/glossary.json`](../kb/methodology/glossary.json). |
|
| 17 |
|
| 18 |
## Voice UX invariants ([ADR-028](../70-docs/60-decisions/ADR-028-voice-ux-single-default-mode.md))
|
| 19 |
|
| 20 |
- **One default voice mode + one fallback.** Live mode is the default; the toolbar pill toggles it (green = on, red = off). The 🎤 push-to-talk button suspends Live for one turn → captures with VAD silence-cutoff → resumes Live if the user preference is still on.
|
| 21 |
- **Hands-free mode was removed in KI-027.** Any reference to it in code or docs is stale.
|
|
|
|
|
|
|
| 22 |
- **Bot TTS plays via the in-DOM `<audio>` element inside `Message`** (autoplay-on-mount via ref'd `useEffect`). **Never use `new Audio(url).play()`** — those detached instances are invisible to the `document.querySelectorAll("audio").pause()` call in the barge-in handler, so they keep playing under the user's speech.
|
| 23 |
|
| 24 |
## Tooling
|
|
|
|
| 13 |
| `src/app/globals.css` | Tailwind v4 entrypoint + the shadcn/ui token layer ([ADR-013](../70-docs/60-decisions/ADR-013-tailwind-shadcn-ui.md)). |
|
| 14 |
| `src/lib/api.ts` | Typed API client. Generated types via `openapi-typescript` from the FastAPI OpenAPI schema ([ADR-015](../70-docs/60-decisions/ADR-015-openapi-typescript-codegen.md)). |
|
| 15 |
| `src/lib/useLiveConversation.ts` | The continuously-open-mic VAD hook that powers Live mode + barge-in. State persists in `localStorage.insurance_live_pref` ([ADR-028](../70-docs/60-decisions/ADR-028-voice-ux-single-default-mode.md)). |
|
| 16 |
+
| `src/lib/useStreamingVoice.ts` | **KI-168 hybrid voice capture.** Runs Web Speech API (interim transcripts streamed live to the chat input) and `MediaRecorder` (authoritative audio blob) in parallel; on browser silence-detect the blob is POSTed to `/api/transcribe` (Sarvam Saarika STT). Auto-submits the Sarvam transcript; falls back to the Web Speech transcript only if Sarvam errors. KI-173 heartbeat + KI-174 `visibilitychange` / `focus` revival hooks keep the mic alive across tab / app switches. |
|
| 17 |
| `src/lib/i18n.ts` | EN ↔ हिं strings + the 13-term `GLOSSARY` mirrored to [`kb/methodology/glossary.json`](../kb/methodology/glossary.json). |
|
| 18 |
|
| 19 |
## Voice UX invariants ([ADR-028](../70-docs/60-decisions/ADR-028-voice-ux-single-default-mode.md))
|
| 20 |
|
| 21 |
- **One default voice mode + one fallback.** Live mode is the default; the toolbar pill toggles it (green = on, red = off). The 🎤 push-to-talk button suspends Live for one turn → captures with VAD silence-cutoff → resumes Live if the user preference is still on.
|
| 22 |
- **Hands-free mode was removed in KI-027.** Any reference to it in code or docs is stale.
|
| 23 |
+
- **Hybrid capture (KI-168).** Web Speech API drives the live interim transcript on the chat input; `MediaRecorder` captures the authoritative blob in parallel. On browser silence-detect the blob is POSTed to `/api/transcribe` (Sarvam) for the authoritative transcript; Web Speech is the fallback only if Sarvam errors. See `src/lib/useStreamingVoice.ts`.
|
| 24 |
+
- **Mic survives tab / app switches (KI-173 / KI-174).** Heartbeat keeps the mic stream alive while the tab is backgrounded; `visibilitychange` / `focus` revival hooks reattach the mic on tab return.
|
| 25 |
- **Bot TTS plays via the in-DOM `<audio>` element inside `Message`** (autoplay-on-mount via ref'd `useEffect`). **Never use `new Audio(url).play()`** — those detached instances are invisible to the `document.querySelectorAll("audio").pause()` call in the barge-in handler, so they keep playing under the user's speech.
|
| 26 |
|
| 27 |
## Tooling
|
|
@@ -727,6 +727,7 @@
|
|
| 727 |
function $(id) { return document.getElementById(id); }
|
| 728 |
|
| 729 |
function clearChildren(el) {
|
|
|
|
| 730 |
while (el.firstChild) el.removeChild(el.firstChild);
|
| 731 |
}
|
| 732 |
|
|
|
|
| 727 |
function $(id) { return document.getElementById(id); }
|
| 728 |
|
| 729 |
function clearChildren(el) {
|
| 730 |
+
if (!el) return;
|
| 731 |
while (el.firstChild) el.removeChild(el.firstChild);
|
| 732 |
}
|
| 733 |
|
|
@@ -30,27 +30,32 @@ PARSE CHUNK EMBED
|
|
| 30 |
|
| 31 |
[7] [8] [9]
|
| 32 |
VECTOR INDEX STRUCTURED EXTRACTION SCORECARD
|
| 33 |
-
- Chroma persistent -
|
| 34 |
client with HealthPolicy weighted aggregation
|
| 35 |
- metadata per chunk Pydantic schema - A-F grade
|
| 36 |
-
- rag/vectors/*.sqlite3 -
|
| 37 |
→ rag/extracted/*.json
|
| 38 |
→ rag/policies.duckdb
|
| 39 |
|
| 40 |
[10] [11] [12]
|
| 41 |
RETRIEVAL ORCHESTRATION FAITHFULNESS GATES
|
| 42 |
-
-
|
| 43 |
-
(
|
| 44 |
-
- top-k cosine
|
| 45 |
-
- per-policy filter - persona prompt
|
| 46 |
-
|
|
|
|
|
|
|
|
|
|
| 47 |
|
| 48 |
[13] [14] [15]
|
| 49 |
RESPONSE TTS UI RENDER
|
| 50 |
-
- reply_text + citations - Sarvam Bulbul
|
| 51 |
-
- brain used
|
| 52 |
-
|
| 53 |
-
|
|
|
|
|
|
|
| 54 |
```
|
| 55 |
|
| 56 |
## 2. Every artifact at every stage
|
|
@@ -65,7 +70,7 @@ RESPONSE TTS UI RENDER
|
|
| 65 |
| 5 CHUNK | chunks (in-memory) | _ephemeral_ | `rag/ingest.py:chunk_pages` | run ingest with `--dry-run` |
|
| 66 |
| 6 EMBED | 384-dim vectors | _embedded in Chroma_ | `backend/providers/local_embeddings.py` | re-encode the same text |
|
| 67 |
| 7 VECTOR INDEX | persistent sqlite | `rag/vectors/chroma.sqlite3` + HNSW binaries | `chromadb.PersistentClient` | open with chromadb client |
|
| 68 |
-
| 8 STRUCTURED EXTRACTION | 62-field JSON per policy | `rag/extracted/<policy_id>.json` | `rag/extract.py` (
|
| 69 |
| 8 STRUCTURED EXTRACTION | aggregate table | `rag/policies.duckdb` | upsert in `rag/extract.py` | `duckdb` CLI query |
|
| 70 |
| 9 SCORECARD | per-policy grade | `kb/policies/<policy_id>.md` (live in code via `backend/scorecard.py`) | `rag/build_kb.py` + `backend/scorecard.py` | re-run `build_kb` |
|
| 71 |
| 10 RETRIEVAL (runtime) | top-k chunks | _ephemeral, logged_ | `rag/retrieve.py` | replay query against Chroma |
|
|
@@ -88,7 +93,7 @@ RESPONSE TTS UI RENDER
|
|
| 88 |
- **Risk if tampered**: silent retrieval poisoning. Mitigation: chunk content stored alongside the embedding; can be re-verified against source PDF on demand.
|
| 89 |
|
| 90 |
### Extracted structured data (Stage 8)
|
| 91 |
-
- **Provenance**:
|
| 92 |
- **Verifiable**: `extraction_confidence_pct` field per record + re-runnable in <30s with `python -m rag.extract --policy <id>`
|
| 93 |
- **Risk if tampered**: bad downstream scorecard + filter results. Mitigation: schema validates types; manual spot-check 5%.
|
| 94 |
|
|
@@ -134,7 +139,7 @@ Every architectural decision in `70-docs/decisions.md` produces a specific artif
|
|
| 134 |
Honesty about gaps:
|
| 135 |
|
| 136 |
- **Insurer-side PDF tampering** — we trust the insurer's published PDF was real at download time. We don't have a re-fetch + diff pipeline yet. v2 enhancement.
|
| 137 |
-
- **LLM determinism** —
|
| 138 |
- **Embedding model drift** — if BGE updates, our cached vectors become stale relative to query-time embeddings. v2: pin model commit hash.
|
| 139 |
- **Chroma version compatibility** — different `chromadb` releases store metadata differently (we hit `KeyError: '_type'` on first HF deploy). Pinned to `0.5.20` in `requirements.txt`.
|
| 140 |
|
|
@@ -194,5 +199,19 @@ Three back-to-back curation passes brought the `40-data/policy_facts/` directory
|
|
| 194 |
- **EN ↔ हिं i18n** — full bilingual UI with the 13-term jargon glossary at `frontend/src/lib/i18n.ts` (mirrored to `kb/methodology/glossary.json`).
|
| 195 |
- **Scorecard methodology expander** — every grade opens a transparency panel sourced from `METHODOLOGY_BLUEPRINT` (mirrored to `kb/methodology/scorecard.json`).
|
| 196 |
- **Source-quote popovers** — hovering a fact on a policy card surfaces the verbatim PDF quote that backed it.
|
| 197 |
-
- **2026-05-14 Stack A consolidation (D-019)** — 4 third-party LLM providers (Groq, OpenRouter, direct DeepSeek, Cerebras) collapsed onto a single NVIDIA NIM endpoint. Brain = Qwen 80B (
|
| 198 |
- **2026-05-14 Space/data split (D-020)** — `rag/corpus/` (188 MB), `rag/vectors/` (129 MB), and `rag/extracted/` moved to companion HF dataset `rohitsar567/insurance-bot-data` (50 GB free quota). Dockerfile pulls them at image build time. Space repo collapses from 286 MB → ~3 MB code-only.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 30 |
|
| 31 |
[7] [8] [9]
|
| 32 |
VECTOR INDEX STRUCTURED EXTRACTION SCORECARD
|
| 33 |
+
- Chroma persistent - NimChainLLM(BRAIN_CHAIN)- rules-based 6-sub-score
|
| 34 |
client with HealthPolicy weighted aggregation
|
| 35 |
- metadata per chunk Pydantic schema - A-F grade
|
| 36 |
+
- rag/vectors/*.sqlite3 - native JSON mode - 24 of 62 fields used
|
| 37 |
→ rag/extracted/*.json
|
| 38 |
→ rag/policies.duckdb
|
| 39 |
|
| 40 |
[10] [11] [12]
|
| 41 |
RETRIEVAL ORCHESTRATION FAITHFULNESS GATES
|
| 42 |
+
- BGE-small at query time - intent classifier - Gate 1: retrieval floor
|
| 43 |
+
(local CPU, 384-d) - sales_brain (fact-find) - Gate 2: citation integrity
|
| 44 |
+
- top-k cosine / BRAIN_CHAIN (QA, etc) - Gate 3: regex numeric
|
| 45 |
+
- per-policy filter - persona prompt grounding
|
| 46 |
+
- profile_rag (session- - chat history - Gate 4: LLM-judge
|
| 47 |
+
scoped, KI-102) (NIM Mistral 675B,
|
| 48 |
+
skipped on fact-find +
|
| 49 |
+
recommendation, KI-171)
|
| 50 |
|
| 51 |
[13] [14] [15]
|
| 52 |
RESPONSE TTS UI RENDER
|
| 53 |
+
- reply_text + citations - Sarvam Bulbul v2 - chat bubble
|
| 54 |
+
- brain used (Gemini / - voice_format strips - source links per
|
| 55 |
+
NIM / OR + model id), CoT + ₹ shorthand citation
|
| 56 |
+
latency - base64 in same JSON - scorecard (P1)
|
| 57 |
+
- faithfulness verdict response - in-DOM <audio> for
|
| 58 |
+
barge-in compatibility
|
| 59 |
```
|
| 60 |
|
| 61 |
## 2. Every artifact at every stage
|
|
|
|
| 70 |
| 5 CHUNK | chunks (in-memory) | _ephemeral_ | `rag/ingest.py:chunk_pages` | run ingest with `--dry-run` |
|
| 71 |
| 6 EMBED | 384-dim vectors | _embedded in Chroma_ | `backend/providers/local_embeddings.py` | re-encode the same text |
|
| 72 |
| 7 VECTOR INDEX | persistent sqlite | `rag/vectors/chroma.sqlite3` + HNSW binaries | `chromadb.PersistentClient` | open with chromadb client |
|
| 73 |
+
| 8 STRUCTURED EXTRACTION | 62-field JSON per policy | `rag/extracted/<policy_id>.json` | `rag/extract.py` (`NimChainLLM(BRAIN_CHAIN)` with native JSON mode) | re-run extraction; compare |
|
| 74 |
| 8 STRUCTURED EXTRACTION | aggregate table | `rag/policies.duckdb` | upsert in `rag/extract.py` | `duckdb` CLI query |
|
| 75 |
| 9 SCORECARD | per-policy grade | `kb/policies/<policy_id>.md` (live in code via `backend/scorecard.py`) | `rag/build_kb.py` + `backend/scorecard.py` | re-run `build_kb` |
|
| 76 |
| 10 RETRIEVAL (runtime) | top-k chunks | _ephemeral, logged_ | `rag/retrieve.py` | replay query against Chroma |
|
|
|
|
| 93 |
- **Risk if tampered**: silent retrieval poisoning. Mitigation: chunk content stored alongside the embedding; can be re-verified against source PDF on demand.
|
| 94 |
|
| 95 |
### Extracted structured data (Stage 8)
|
| 96 |
+
- **Provenance**: `NimChainLLM(BRAIN_CHAIN)` over full PDF text + Pydantic schema, native JSON mode
|
| 97 |
- **Verifiable**: `extraction_confidence_pct` field per record + re-runnable in <30s with `python -m rag.extract --policy <id>`
|
| 98 |
- **Risk if tampered**: bad downstream scorecard + filter results. Mitigation: schema validates types; manual spot-check 5%.
|
| 99 |
|
|
|
|
| 139 |
Honesty about gaps:
|
| 140 |
|
| 141 |
- **Insurer-side PDF tampering** — we trust the insurer's published PDF was real at download time. We don't have a re-fetch + diff pipeline yet. v2 enhancement.
|
| 142 |
+
- **LLM determinism** — every brain candidate (Gemini, Qwen, Mistral, Maverick) can produce slightly different extraction across runs even at `temperature=0`. Model versions are pinned in `backend/config.py`; for full reproducibility we'd cache extraction outputs. v2 enhancement.
|
| 143 |
- **Embedding model drift** — if BGE updates, our cached vectors become stale relative to query-time embeddings. v2: pin model commit hash.
|
| 144 |
- **Chroma version compatibility** — different `chromadb` releases store metadata differently (we hit `KeyError: '_type'` on first HF deploy). Pinned to `0.5.20` in `requirements.txt`.
|
| 145 |
|
|
|
|
| 199 |
- **EN ↔ हिं i18n** — full bilingual UI with the 13-term jargon glossary at `frontend/src/lib/i18n.ts` (mirrored to `kb/methodology/glossary.json`).
|
| 200 |
- **Scorecard methodology expander** — every grade opens a transparency panel sourced from `METHODOLOGY_BLUEPRINT` (mirrored to `kb/methodology/scorecard.json`).
|
| 201 |
- **Source-quote popovers** — hovering a fact on a policy card surfaces the verbatim PDF quote that backed it.
|
| 202 |
+
- **2026-05-14 Stack A consolidation (D-019)** — 4 third-party LLM providers (Groq, OpenRouter, direct DeepSeek, Cerebras) collapsed onto a single NVIDIA NIM endpoint. Brain = Qwen 80B (heavy intents) + Nemotron Nano 30B (voice + fact-find). Judge = Mistral Large 3 675B (different family). Sarvam scoped to voice STT/TTS + Indic translation only. Free tier, 40 req/min, no daily cap. ~600 LOC of legacy provider wiring deleted.
|
| 203 |
- **2026-05-14 Space/data split (D-020)** — `rag/corpus/` (188 MB), `rag/vectors/` (129 MB), and `rag/extracted/` moved to companion HF dataset `rohitsar567/insurance-bot-data` (50 GB free quota). Dockerfile pulls them at image build time. Space repo collapses from 286 MB → ~3 MB code-only.
|
| 204 |
+
|
| 205 |
+
## Batch — 2026-05-15
|
| 206 |
+
|
| 207 |
+
The day's sprint produced a chain-level rewrite, a provider integration, and a fact-find rip-out. See [`70-docs/40-evaluation/quality-sprint-2026-05-14.md`](../70-docs/40-evaluation/quality-sprint-2026-05-14.md) (Follow-on sprint section) for the full KI list.
|
| 208 |
+
|
| 209 |
+
- **KI-167 / [ADR-039](../70-docs/60-decisions/ADR-039-llm-driven-sales-brain.md)** — Ripped out `backend/fact_find_brain.py` (441 LOC) + `_canonical_fallback` + the `<FF>{...}</FF>` trailer convention. Replaced with `backend/sales_brain.py` — one LLM call per turn against `FAST_BRAIN_CHAIN` using native provider JSON mode (`response_mime_type=application/json` on Gemini, `response_format={"type":"json_object"}` on NIM). New deterministic post-processor `backend/sales_brain_normalizer.py` validates and normalises captured fields.
|
| 210 |
+
- **KI-168** — Hybrid voice capture: Web Speech API streams interim transcripts to chat input (live UX feel) while `MediaRecorder` runs in parallel; on browser silence-detect the audio blob is POSTed to `/api/transcribe` (Sarvam Saarika v2.5) for the authoritative transcript. Auto-submits the Sarvam result; falls back to Web Speech only if Sarvam errors.
|
| 211 |
+
- **KI-171** — Skip faithfulness Gate 4 (LLM judge) on `fact_find` + `recommendation` intents — no retrieved context to grade against on those turns.
|
| 212 |
+
- **KI-173 / KI-174** — Voice heartbeat + `visibilitychange` / `focus` revival keeps the mic alive across tab / app switches.
|
| 213 |
+
- **KI-175 / KI-176** — Chain reorder: NIM Nemotron 49B demoted from primary to last resort across all three chains (Mistral 675B / Qwen 80B consistently outperform on live audit). OpenRouter re-added as cross-provider diversity using OR's `models: [...]` server-side fallback within the OR free pool.
|
| 214 |
+
- **KI-178** — Live audit of which OpenRouter free-tier models support `response_format`. Only NIM-style native JSON mode is supported on `nvidia/nemotron-3-super-120b-a12b:free`, `qwen/qwen3-next-80b-a3b-instruct:free`, and `google/gemma-4-31b:free`. Llama 3.3 70B / Hermes 3 405B on OR free do NOT — excluded from chains.
|
| 215 |
+
- **KI-179 / [ADR-040](../70-docs/60-decisions/ADR-040-google-gemini-primary.md)** — Operator obtained a Google AI Studio API key. New `backend/providers/google_gemini_llm.py` wraps Google's REST API (`gemini-2.0-flash` for Brain Fast / sales_brain, `gemini-2.5-flash` for Brain Main / synthesis; native JSON mode via `response_mime_type=application/json`; 1500 req/day free quota). Promoted to **Tier 0 primary** on Brain Fast + Brain Main. NIM stays as Tier 1 fallback (Mistral 675B / Maverick / Qwen 80B); OpenRouter `:free` sits as Tier 2 diversity. **Judge stays on NIM Mistral Large 3 675B** (different family from Gemini — preserves brain ↔ judge family-diversity invariant). NIM-only lock from [ADR-038](../70-docs/60-decisions/ADR-038-nim-only-chains.md) relaxed (the `<FF>` trailer convention that motivated the lock is permanently retired post-KI-167). KI-080 sticky-primary election machinery extends naturally to the Google provider.
|
| 216 |
+
|
| 217 |
+
**Net architectural shape after the sprint:** 3-tier chains across Google → NIM → OpenRouter for Brain Fast + Brain Main; NIM-only with OR `:free` diversity for Judge. Sarvam unchanged (STT + TTS + Indic translation). All structured-output traffic uses native provider JSON mode; no trailer conventions remain.
|
|
@@ -12,7 +12,7 @@ Lineage for every artefact below is documented in [`kb/AUDIT_TRAIL.md`](../kb/AU
|
|
| 12 |
| `download_retry.py` | 3. DOWNLOAD | Retries the failures from the previous run. |
|
| 13 |
| `download_regulatory.py` | 3. DOWNLOAD | IRDAI / regulatory PDFs (deferred from v1; see [ADR-017](../70-docs/60-decisions/ADR-017-irdai-corpus-playwright-rescue.md)). |
|
| 14 |
| `ingest.py` | 4. PARSE → 5. CHUNK → 6. EMBED → 7. INDEX | The big one. `read_pdf_pages` (pdfplumber) → `chunk_pages` (800-tok / 120 overlap, sentence-aware) → BGE embed → `chromadb.PersistentClient.add(...)`. Carries the in-process HNSW bloat tripwire ([ADR-029](../70-docs/60-decisions/ADR-029-hnsw-bloat-tripwire.md)). |
|
| 15 |
-
| `extract.py` | 8. STRUCTURED EXTRACTION | LLM extraction over each PDF using `schema.py::HealthPolicy` (62 fields). `get_brain_llm()` (
|
| 16 |
| `build_kb.py` | 9. SCORECARD → KB MIRROR | Runs `backend/scorecard.py` per policy and regenerates the human-readable `kb/policies/<id>.md` tree. |
|
| 17 |
| `source_map.py` | post-build | Builds `source_map.json` — every chunk → (PDF path, page, span) for citation rendering. |
|
| 18 |
|
|
|
|
| 12 |
| `download_retry.py` | 3. DOWNLOAD | Retries the failures from the previous run. |
|
| 13 |
| `download_regulatory.py` | 3. DOWNLOAD | IRDAI / regulatory PDFs (deferred from v1; see [ADR-017](../70-docs/60-decisions/ADR-017-irdai-corpus-playwright-rescue.md)). |
|
| 14 |
| `ingest.py` | 4. PARSE → 5. CHUNK → 6. EMBED → 7. INDEX | The big one. `read_pdf_pages` (pdfplumber) → `chunk_pages` (800-tok / 120 overlap, sentence-aware) → BGE embed → `chromadb.PersistentClient.add(...)`. Carries the in-process HNSW bloat tripwire ([ADR-029](../70-docs/60-decisions/ADR-029-hnsw-bloat-tripwire.md)). |
|
| 15 |
+
| `extract.py` | 8. STRUCTURED EXTRACTION | LLM extraction over each PDF using `schema.py::HealthPolicy` (62 fields). `get_brain_llm()` (Brain Main: Gemini 2.5 Flash primary per [ADR-040](../70-docs/60-decisions/ADR-040-google-gemini-primary.md)) + `get_fast_brain_llm()` (Brain Fast: Gemini 2.0 Flash primary) — both `NimChainLLM` fallback chains across Google → NIM → OpenRouter. Native JSON mode (`response_mime_type=application/json` on Gemini, `response_format={"type":"json_object"}` on NIM). Writes `rag/extracted/<policy_id>.json` + upserts `policies.duckdb`. |
|
| 16 |
| `build_kb.py` | 9. SCORECARD → KB MIRROR | Runs `backend/scorecard.py` per policy and regenerates the human-readable `kb/policies/<id>.md` tree. |
|
| 17 |
| `source_map.py` | post-build | Builds `source_map.json` — every chunk → (PDF path, page, span) for citation rendering. |
|
| 18 |
|
|
@@ -41,15 +41,15 @@ Scheduling for the long-running ones is wired via macOS LaunchAgents — see `CR
|
|
| 41 |
| --- | --- |
|
| 42 |
| `upload_to_hf.py` | Code-side push to the HF Space repo (`huggingface.co/spaces/rohitsar567/InsuranceBot`). |
|
| 43 |
| `upload_corpus_to_dataset.py`, `upload_extracted_to_dataset.py`, `upload_vectors_to_dataset.py`, `upload_all_to_dataset.py` | Push specific slices of `rag/` to the companion HF Dataset `rohitsar567/insurance-bot-data`. See [ADR-020](../70-docs/60-decisions/ADR-020-code-data-split-hf-dataset.md) and [ADR-024](../70-docs/60-decisions/ADR-024-triple-mirror-code-and-data.md). |
|
| 44 |
-
| `set_hf_secrets.py` | One-shot helper that pushes the runtime secrets into the HF Space (idempotent). |
|
| 45 |
|
| 46 |
## Probes + diagnostics
|
| 47 |
|
| 48 |
| Script | Provider it pokes |
|
| 49 |
| --- | --- |
|
| 50 |
| `sarvam_probe.py`, `sarvam_nothink_probe.py` | Sarvam-M / Saarika / Bulbul connectivity + latency. |
|
| 51 |
-
| `groq_probe.py`, `groq_long_probe.py` | Groq Llama free-tier
|
| 52 |
-
| `openrouter_probe.py`, `or_models.py` | OpenRouter routing + model-list inspection. |
|
| 53 |
| `pdf_probe.py` | pdfplumber parse on a single PDF — first stop when extraction silently produces empty text. |
|
| 54 |
| `heavy_smoke_test.py` | End-to-end smoke against the live HF Space (every provider in one call). |
|
| 55 |
|
|
|
|
| 41 |
| --- | --- |
|
| 42 |
| `upload_to_hf.py` | Code-side push to the HF Space repo (`huggingface.co/spaces/rohitsar567/InsuranceBot`). |
|
| 43 |
| `upload_corpus_to_dataset.py`, `upload_extracted_to_dataset.py`, `upload_vectors_to_dataset.py`, `upload_all_to_dataset.py` | Push specific slices of `rag/` to the companion HF Dataset `rohitsar567/insurance-bot-data`. See [ADR-020](../70-docs/60-decisions/ADR-020-code-data-split-hf-dataset.md) and [ADR-024](../70-docs/60-decisions/ADR-024-triple-mirror-code-and-data.md). |
|
| 44 |
+
| `set_hf_secrets.py` | One-shot helper that pushes the runtime secrets into the HF Space (idempotent). Current secret set: `GOOGLE_API_KEY` (Google AI Studio, per [ADR-040](../70-docs/60-decisions/ADR-040-google-gemini-primary.md)), `NVIDIA_NIM_API_KEY`, `OPENROUTER_API_KEY`, `SARVAM_API_KEY`, plus admin password / IP allowlist. |
|
| 45 |
|
| 46 |
## Probes + diagnostics
|
| 47 |
|
| 48 |
| Script | Provider it pokes |
|
| 49 |
| --- | --- |
|
| 50 |
| `sarvam_probe.py`, `sarvam_nothink_probe.py` | Sarvam-M / Saarika / Bulbul connectivity + latency. |
|
| 51 |
+
| `groq_probe.py`, `groq_long_probe.py` | Historical Groq Llama free-tier probe — Groq is no longer in any production chain (removed in [ADR-038](../70-docs/60-decisions/ADR-038-nim-only-chains.md), not re-added in [ADR-040](../70-docs/60-decisions/ADR-040-google-gemini-primary.md)). Kept for benchmarking. |
|
| 52 |
+
| `openrouter_probe.py`, `or_models.py` | OpenRouter routing + model-list inspection. Used by KI-178 to audit which `:free` models expose `response_format`. |
|
| 53 |
| `pdf_probe.py` | pdfplumber parse on a single PDF — first stop when extraction silently produces empty text. |
|
| 54 |
| `heavy_smoke_test.py` | End-to-end smoke against the live HF Space (every provider in one call). |
|
| 55 |
|
|
@@ -31,6 +31,7 @@ REPO_ID = "rohitsar567/InsuranceBot"
|
|
| 31 |
SECRETS_TO_SET = [
|
| 32 |
"SARVAM_API_KEY", "VOYAGE_API_KEY", "NVIDIA_NIM_API_KEY",
|
| 33 |
"OPENROUTER_API_KEY", "GROQ_API_KEY",
|
|
|
|
| 34 |
"ADMIN_IP_ALLOWLIST", "ADMIN_PASSWORD",
|
| 35 |
]
|
| 36 |
# Truly retired providers — delete from Space to prevent confusion.
|
|
|
|
| 31 |
SECRETS_TO_SET = [
|
| 32 |
"SARVAM_API_KEY", "VOYAGE_API_KEY", "NVIDIA_NIM_API_KEY",
|
| 33 |
"OPENROUTER_API_KEY", "GROQ_API_KEY",
|
| 34 |
+
"GOOGLE_API_KEY", # KI-179 (2026-05-15) — Gemini Flash via Google AI Studio
|
| 35 |
"ADMIN_IP_ALLOWLIST", "ADMIN_PASSWORD",
|
| 36 |
]
|
| 37 |
# Truly retired providers — delete from Space to prevent confusion.
|