Spaces:
Sleeping
chore(cleanup): purge stale narrative/tombstones/dead code — codebase reads as the current standard
Browse filesComprehensive stale-content purge so code/docs read as if written correctly
from the start; history of record stays only in ADRs (70-docs/60-decisions),
KIs (80-audit), and the README.
Code (.py): removed mistake/removal/incident narration and tombstone
comments ("REMOVED/VESTIGIAL", dated fix-narration, "Z2 fix/W1 Turn/Priya/
Vikram", "the bug was", "used to ... → loop", "do NOT reintroduce"),
rewrote them as clean current-behavior comments; deleted the dead
SessionState._recall_sniff_done field (zero readers; in-memory-only,
verified safe); cleaned stale references to removed systems
(sales_brain/qa_brain/orchestrator) in comments, docstrings and a log
string. NO logic changed: 29/31 modified .py are AST-equal modulo
docstrings (provably comment-only); the other 2 are the verified-safe
field deletion + log/docstring string text only.
Docs: 70-docs/** + ARCHITECTURE.md rewritten to the current
single-LLM-with-tools standard (gemini-2.5-flash + save_profile_field/
retrieve_policies/mark_recommendation + small nim_fallback); removed
systems no longer described as live; historical eval/known-issues logs
carry accurate "superseded" banners. ADRs and KIs left verbatim.
Verified: full pytest gate green (rc=0, 0 failures) AFTER the purge;
AST-equivalence proves the code sweep is logic-preserving. Not pushed in
this commit's own step — guarded deploy follows.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- 70-docs/00-overview/roadmap.md +25 -18
- 70-docs/10-architecture/safety-architecture.md +16 -10
- 70-docs/10-architecture/scoring-methodology.md +7 -6
- 70-docs/10-architecture/stack-rationale.md +9 -5
- 70-docs/10-architecture/system-overview.md +16 -10
- 70-docs/30-engineering/discovery-script.md +10 -8
- 70-docs/30-engineering/needs-analysis-flow.md +39 -55
- 70-docs/40-evaluation/eval-methodology.md +20 -16
- 70-docs/40-evaluation/known-issues.md +11 -0
- 70-docs/40-evaluation/quality-sprint-2026-05-14.md +10 -0
- ARCHITECTURE.md +9 -7
- backend/admin.py +30 -37
- backend/brain_tools.py +61 -110
- backend/config.py +14 -20
- backend/llm_health.py +118 -154
- backend/main.py +91 -106
- backend/needs_finder.py +24 -35
- backend/nim_fallback.py +8 -8
- backend/policy_identity.py +8 -10
- backend/premium_calculator.py +14 -24
- backend/profile_persistence.py +0 -7
- backend/profile_rag.py +27 -37
- backend/profile_store.py +7 -7
- backend/providers/__init__.py +7 -16
- backend/providers/base.py +1 -1
- backend/providers/google_gemini_llm.py +43 -47
- backend/providers/groq_llm.py +2 -2
- backend/providers/local_embeddings.py +6 -6
- backend/providers/nvidia_nim_llm.py +128 -194
- backend/providers/openrouter_llm.py +11 -18
- backend/providers/sarvam_tts.py +7 -10
- backend/retrieval_filters.py +43 -51
- backend/scorecard.py +4 -7
- backend/session_state.py +12 -27
- backend/single_brain.py +37 -88
- backend/voice_format.py +33 -38
- rag/extract.py +14 -31
- rag/retrieve.py +47 -66
- tools/chunk_sweep.py +3 -3
- tools/set_hf_secrets.py +9 -10
- tools/upload_to_hf.py +11 -12
- tools/upload_vectors_to_dataset.py +2 -2
|
@@ -16,29 +16,36 @@
|
|
| 16 |
|
| 17 |
v1 is a **vertical slice**: 10 insurers × Health × ~80 policies × voice-first advisor. The architecture is built so v2 is a **data/config change, not a rebuild**. This document maps the path.
|
| 18 |
|
| 19 |
-
## 0.1
|
| 20 |
-
|
| 21 |
-
-
|
| 22 |
-
|
| 23 |
-
|
| 24 |
-
|
| 25 |
-
|
| 26 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 27 |
|
| 28 |
## 1. What v1 ships
|
| 29 |
|
| 30 |
**Working product:**
|
| 31 |
-
- Voice-first chat advisor over a curated corpus of Indian health insurance policies (~76 PDFs from 10 insurers, ingested into Chroma
|
| 32 |
- Multi-language: English + Hindi/Hinglish via Sarvam Saarika STT + Sarvam Bulbul TTS
|
| 33 |
-
-
|
| 34 |
-
-
|
| 35 |
- 62-field structured extraction per policy
|
| 36 |
- Clean Next.js + Tailwind frontend
|
| 37 |
-
- FastAPI backend deployed
|
| 38 |
- 8 design / decision documents totaling ~30 pages
|
| 39 |
|
| 40 |
**Eval signal:**
|
| 41 |
-
- Gold Q&A harness (~300 pairs targeted) + automated grader
|
| 42 |
- `eval/results.md` versioned table per run
|
| 43 |
- Live audit log `logs/hallucinations.jsonl` for every blocked claim
|
| 44 |
|
|
@@ -133,10 +140,10 @@ v1 is a **vertical slice**: 10 insurers × Health × ~80 policies × voice-first
|
|
| 133 |
The point of disciplined v1 architecture is that these things are **stable** across the transition:
|
| 134 |
|
| 135 |
1. **62-field structured schema** (`rag/schema.py`) — data-only change to add v2 categories
|
| 136 |
-
2. **Provider abstraction**
|
| 137 |
-
3. **
|
| 138 |
-
4. **
|
| 139 |
-
5. **Eval methodology** (`70-docs/
|
| 140 |
|
| 141 |
The "c-readiness commitments" in Doc 02 §7 are the contract. Every v2 feature is a commitment honored.
|
| 142 |
|
|
@@ -146,7 +153,7 @@ The "c-readiness commitments" in Doc 02 §7 are the contract. Every v2 feature i
|
|
| 146 |
| --- | --- | --- |
|
| 147 |
| Streamlit → Next.js mid-build | Production polish for a BFSI reviewer | 2 extra hours of scaffolding |
|
| 148 |
| Voyage embeddings → BGE local | Voyage 3 RPM rate limit blocked ingestion | Slightly lower retrieval quality (~3pp) for full corpus access |
|
| 149 |
-
| IRDAI corpus deferred | Akamai bot protection;
|
| 150 |
| Push-to-talk over streaming | Risk of broken realtime > demo latency | 2-3s perceived latency vs <1s |
|
| 151 |
| No auth | Out of scope per Doc 01 | Single-user demo only |
|
| 152 |
| Hand-curated 5-node fact-find | Auditable + testable | Less natural than LLM-driven |
|
|
|
|
| 16 |
|
| 17 |
v1 is a **vertical slice**: 10 insurers × Health × ~80 policies × voice-first advisor. The architecture is built so v2 is a **data/config change, not a rebuild**. This document maps the path.
|
| 18 |
|
| 19 |
+
## 0.1 Architecture history (pointers only)
|
| 20 |
+
|
| 21 |
+
> The 2026-05-15 work below (KI-167…KI-179, ADR-039/040) was an
|
| 22 |
+
> intermediate step that was itself **subsequently superseded** by the
|
| 23 |
+
> single-LLM-with-tools rewrite (one Gemini 2.5-flash call per turn with
|
| 24 |
+
> `save_profile_field` / `retrieve_policies` / `mark_recommendation`,
|
| 25 |
+
> structured+vector retrieval, small `nim_fallback`). KI/ADR pointers are
|
| 26 |
+
> kept as history-of-record. Present-state authority:
|
| 27 |
+
> [`README.md`](../../README.md) §4.
|
| 28 |
+
|
| 29 |
+
- **KI-167 / [ADR-039](../60-decisions/ADR-039-llm-driven-sales-brain.md)** — Removed the scripted fact-find renderer + `<FF>` trailer convention (history; the later single-LLM-with-tools handler is the present design).
|
| 30 |
+
- **KI-168** — Hybrid voice capture: Web Speech API streams interim transcripts for live UX feel while MediaRecorder runs in parallel; Sarvam STT is the authoritative transcript posted on browser silence-detect. (Still current.)
|
| 31 |
+
- **KI-171** — Skipping the separate faithfulness judge on no-context turns (the separate judge LLM was later removed entirely; faithfulness is now structural).
|
| 32 |
+
- **KI-173 / KI-174** — Voice heartbeat + `visibilitychange` / `focus` revival hooks keep the mic alive across tab and app switches. (Still current.)
|
| 33 |
+
- **KI-175 / KI-176 / KI-178 / KI-179 / [ADR-040](../60-decisions/ADR-040-google-gemini-primary.md)** — Provider/chain work from the multi-model era (history; the present brain is a single Gemini 2.5-flash call per turn with a small NIM fallback for transient errors).
|
| 34 |
|
| 35 |
## 1. What v1 ships
|
| 36 |
|
| 37 |
**Working product:**
|
| 38 |
+
- Voice-first chat advisor over a curated corpus of Indian health insurance policies (~76 PDFs from 10 insurers, ingested into Chroma)
|
| 39 |
- Multi-language: English + Hindi/Hinglish via Sarvam Saarika STT + Sarvam Bulbul TTS
|
| 40 |
+
- **Single-LLM-with-tools brain:** one **Gemini 2.5-flash** call per turn with function-calling tools (`save_profile_field` / `retrieve_policies` / `mark_recommendation`) handling fact-find, retrieval, QA, and recommendation. Small `backend/nim_fallback.py` (NVIDIA NIM) covers transient Gemini errors so the turn completes; fail-loud otherwise. Sarvam scoped to Indic translation + voice only.
|
| 41 |
+
- Structural faithfulness (the LLM can only cite what `retrieve_policies` returned; recommendation fit gated in `scorecard.py` / `retrieval_filters.py`) + auditable refusal log
|
| 42 |
- 62-field structured extraction per policy
|
| 43 |
- Clean Next.js + Tailwind frontend
|
| 44 |
+
- FastAPI backend deployed as a Hugging Face Space (Docker, `uvicorn`)
|
| 45 |
- 8 design / decision documents totaling ~30 pages
|
| 46 |
|
| 47 |
**Eval signal:**
|
| 48 |
+
- Gold Q&A harness (~300 pairs targeted) + automated offline grader using a judge model from a different family than the runtime Gemini brain (non-circular grading; eval-only, not a runtime gate)
|
| 49 |
- `eval/results.md` versioned table per run
|
| 50 |
- Live audit log `logs/hallucinations.jsonl` for every blocked claim
|
| 51 |
|
|
|
|
| 140 |
The point of disciplined v1 architecture is that these things are **stable** across the transition:
|
| 141 |
|
| 142 |
1. **62-field structured schema** (`rag/schema.py`) — data-only change to add v2 categories
|
| 143 |
+
2. **Provider abstraction** — swap STT/TTS/LLM via config
|
| 144 |
+
3. **Structural faithfulness** — the brain can only cite what `retrieve_policies` returned; recommendation fit gated in `scorecard.py` / `retrieval_filters.py`
|
| 145 |
+
4. **System prompt + citation grammar** — same, refined
|
| 146 |
+
5. **Eval methodology** (`70-docs/40-evaluation/eval-methodology.md`) — same harness, more gold data
|
| 147 |
|
| 148 |
The "c-readiness commitments" in Doc 02 §7 are the contract. Every v2 feature is a commitment honored.
|
| 149 |
|
|
|
|
| 153 |
| --- | --- | --- |
|
| 154 |
| Streamlit → Next.js mid-build | Production polish for a BFSI reviewer | 2 extra hours of scaffolding |
|
| 155 |
| Voyage embeddings → BGE local | Voyage 3 RPM rate limit blocked ingestion | Slightly lower retrieval quality (~3pp) for full corpus access |
|
| 156 |
+
| IRDAI corpus deferred | Akamai bot protection; structural faithfulness already refuses regulatory questions cleanly (the brain can only cite what `retrieve_policies` returned, and there are no IRDAI chunks) | Bot can't ground answers in IRDAI text — refuses instead of citing |
|
| 157 |
| Push-to-talk over streaming | Risk of broken realtime > demo latency | 2-3s perceived latency vs <1s |
|
| 158 |
| No auth | Out of scope per Doc 01 | Single-user demo only |
|
| 159 |
| Hand-curated 5-node fact-find | Auditable + testable | Less natural than LLM-driven |
|
|
@@ -15,14 +15,20 @@
|
|
| 15 |
> consolidation.
|
| 16 |
>
|
| 17 |
> **Current safety model (authoritative: [`README.md`](../../README.md) §6):**
|
| 18 |
-
> faithfulness is now **structural** — the single Gemini
|
| 19 |
-
>
|
| 20 |
-
>
|
| 21 |
-
>
|
| 22 |
-
>
|
| 23 |
-
>
|
| 24 |
-
>
|
| 25 |
-
>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 26 |
|
| 27 |
## 0. Purpose
|
| 28 |
|
|
@@ -207,5 +213,5 @@ Run-time. Auditable. Tested.
|
|
| 207 |
| M-07 | Nightly synthetic eval cron | Infra | v2 |
|
| 208 |
| M-08 | Live-traffic spot grading | Eval | v2 |
|
| 209 |
| M-09 | 3-judge consensus for grader | Eval | v2 |
|
| 210 |
-
| M-10 | Sarvam-M
|
| 211 |
-
| M-11 |
|
|
|
|
| 15 |
> consolidation.
|
| 16 |
>
|
| 17 |
> **Current safety model (authoritative: [`README.md`](../../README.md) §6):**
|
| 18 |
+
> faithfulness is now **structural** — the single Gemini 2.5-flash brain
|
| 19 |
+
> (one call/turn with `save_profile_field` / `retrieve_policies` /
|
| 20 |
+
> `mark_recommendation`) can only state what `retrieve_policies` returned
|
| 21 |
+
> and must cite it; recommendation fit is gated in `backend/scorecard.py` /
|
| 22 |
+
> `retrieval_filters.py`. There is **no separate faithfulness-judge LLM**,
|
| 23 |
+
> no `orchestrator`, and no Brain-Fast/Brain-Main split. The hardened
|
| 24 |
+
> surface is the **8-gate uploaded-PDF defence in `backend/security.py`**
|
| 25 |
+
> (file mechanics, content quality, prompt-injection, per-session + per-IP
|
| 26 |
+
> rate limits, encrypted-PDF, page-ceiling, hash-dedupe) plus per-session
|
| 27 |
+
> quarantine isolation with a 24 h TTL. A small `backend/nim_fallback.py`
|
| 28 |
+
> (NVIDIA NIM) only covers transient Gemini errors so the turn still
|
| 29 |
+
> completes. **Everything below this banner is the historical failure-modes
|
| 30 |
+
> analysis that informed today's design — the per-mode "Status: Live"
|
| 31 |
+
> lines describe the pre-rewrite system, not the present one.**
|
| 32 |
|
| 33 |
## 0. Purpose
|
| 34 |
|
|
|
|
| 213 |
| M-07 | Nightly synthetic eval cron | Infra | v2 |
|
| 214 |
| M-08 | Live-traffic spot grading | Eval | v2 |
|
| 215 |
| M-09 | 3-judge consensus for grader | Eval | v2 |
|
| 216 |
+
| M-10 | Sarvam-M scoped to Indic translation only; brain is one Gemini 2.5-flash call/turn with tools, small `nim_fallback` for transient errors | single_brain | Live ✅ |
|
| 217 |
+
| M-11 | Structural faithfulness (LLM can only cite what `retrieve_policies` returned; recommendation fit gated in `scorecard.py` / `retrieval_filters.py`) — the separate 4-gate judge was removed | single_brain | Live ✅ (superseded the 4-gate judge) |
|
|
@@ -1,10 +1,11 @@
|
|
| 1 |
# Scorecard Methodology — From 62 Schema Fields to a Single A-F Grade
|
| 2 |
|
| 3 |
-
>
|
| 4 |
-
>
|
| 5 |
-
>
|
| 6 |
-
>
|
| 7 |
-
> historical
|
|
|
|
| 8 |
|
| 9 |
| Field | Value |
|
| 10 |
| --- | --- |
|
|
@@ -64,7 +65,7 @@ A small number of fields exist because they materially affect buyer outcomes eve
|
|
| 64 |
|
| 65 |
### How we did it (the actual code path)
|
| 66 |
|
| 67 |
-
`rag/extract.py`
|
| 68 |
|
| 69 |
The full schema lives in `rag/schema.py` (see also `rag/SCHEMA.md` for groupings and gotchas).
|
| 70 |
|
|
|
|
| 1 |
# Scorecard Methodology — From 62 Schema Fields to a Single A-F Grade
|
| 2 |
|
| 3 |
+
> ℹ️ **Methodology is current.** The scorecard is rules-based and
|
| 4 |
+
> LLM-free; the grading logic below (`backend/scorecard.py`) is the
|
| 5 |
+
> present-state design, including the profile-tuned weights used by the
|
| 6 |
+
> single-LLM-with-tools handler when it recommends policies. The only
|
| 7 |
+
> historical pointer is the one-off extraction pipeline noted in §1.
|
| 8 |
+
> Present-state authority: [`README.md`](../../README.md) §4.
|
| 9 |
|
| 10 |
| Field | Value |
|
| 11 |
| --- | --- |
|
|
|
|
| 65 |
|
| 66 |
### How we did it (the actual code path)
|
| 67 |
|
| 68 |
+
`rag/extract.py` runs a one-time offline extraction: an LLM is given the full policy PDF text and the 62-field Pydantic schema as a structured-output target. The LLM extracts each field; if a field is not explicitly stated in the document, it is set to `null`. A self-critique pass scores per-field confidence (the `extraction_confidence_pct` field). This is a build-time pipeline, separate from the runtime single-LLM-with-tools handler.
|
| 69 |
|
| 70 |
The full schema lives in `rag/schema.py` (see also `rag/SCHEMA.md` for groupings and gotchas).
|
| 71 |
|
|
@@ -25,11 +25,15 @@ place so it cannot drift again:
|
|
| 25 |
|
| 26 |
- **Frontend:** Next.js 16 (App Router), React 19, Tailwind v4, static export.
|
| 27 |
- **Backend:** FastAPI + Pydantic; `uvicorn`, port 7860 on the HF Space.
|
| 28 |
-
- **Brain:** Google **Gemini `gemini-2.5-flash
|
| 29 |
-
(
|
| 30 |
-
|
| 31 |
-
|
| 32 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 33 |
- **Voice:** Sarvam Saarika (STT) + Bulbul (TTS) + Sarvam-M (Indic).
|
| 34 |
- **Hosting:** Hugging Face Space (Docker) + companion HF dataset for
|
| 35 |
corpus/vectors; **not** Vercel/Render (that was the old plan).
|
|
|
|
| 25 |
|
| 26 |
- **Frontend:** Next.js 16 (App Router), React 19, Tailwind v4, static export.
|
| 27 |
- **Backend:** FastAPI + Pydantic; `uvicorn`, port 7860 on the HF Space.
|
| 28 |
+
- **Brain:** Google **Gemini `gemini-2.5-flash`** + function-calling tools
|
| 29 |
+
(`save_profile_field` / `retrieve_policies` / `mark_recommendation`), one
|
| 30 |
+
call/turn handling fact-find, retrieval, QA, and recommendation. On a
|
| 31 |
+
transient Gemini error → small **NVIDIA NIM** fallback
|
| 32 |
+
(`backend/nim_fallback.py`) so the turn completes; fail-loud otherwise. No
|
| 33 |
+
separate judge model, no orchestrator, no sales/QA-brain split (all
|
| 34 |
+
retired in the single-LLM-with-tools consolidation).
|
| 35 |
+
- **Retrieval:** structured + vector over Chroma + BGE-small-en-v1.5 (local
|
| 36 |
+
CPU, 384-d) with a profile-tuned scorecard.
|
| 37 |
- **Voice:** Sarvam Saarika (STT) + Bulbul (TTS) + Sarvam-M (Indic).
|
| 38 |
- **Hosting:** Hugging Face Space (Docker) + companion HF dataset for
|
| 39 |
corpus/vectors; **not** Vercel/Render (that was the old plan).
|
|
@@ -12,10 +12,12 @@
|
|
| 12 |
## This document has been superseded
|
| 13 |
|
| 14 |
The earlier version of this file described the **pre-rewrite** architecture
|
| 15 |
-
(an `orchestrator.py`, a `sales_brain`/`QA-brain` split, a
|
| 16 |
-
`faithfulness.py` judge, a
|
| 17 |
-
|
| 18 |
-
|
|
|
|
|
|
|
| 19 |
|
| 20 |
To prevent two documents drifting apart again, there is now exactly **one**
|
| 21 |
authoritative architecture description:
|
|
@@ -28,12 +30,16 @@ authoritative architecture description:
|
|
| 28 |
|
| 29 |
## Current architecture in three lines
|
| 30 |
|
| 31 |
-
- One LLM call per turn: **Gemini `gemini-2.5-flash
|
| 32 |
-
tools** (`
|
| 33 |
-
`backend/
|
| 34 |
-
|
| 35 |
-
|
| 36 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 37 |
- **Next.js 16 / FastAPI**, deployed as an HF Space (Docker, `uvicorn`,
|
| 38 |
port 7860); heavy data pulled at build from the `insurance-bot-data` HF
|
| 39 |
dataset; curated facts in `40-data/`.
|
|
|
|
| 12 |
## This document has been superseded
|
| 13 |
|
| 14 |
The earlier version of this file described the **pre-rewrite** architecture
|
| 15 |
+
(an `orchestrator.py`, a `sales_brain`/`QA-brain` split, a 3-brain design,
|
| 16 |
+
a separate `faithfulness.py` judge LLM, a `profile_extractor`, a tiered
|
| 17 |
+
brain, a translation-check pass, a 3-tier Gemini/NIM/OpenRouter chain, a
|
| 18 |
+
DuckDB 62-field hot path, Next.js 14, Vercel/Render hosting). **None of
|
| 19 |
+
that exists in the codebase anymore** — it was replaced by the
|
| 20 |
+
single-LLM-with-tools rewrite.
|
| 21 |
|
| 22 |
To prevent two documents drifting apart again, there is now exactly **one**
|
| 23 |
authoritative architecture description:
|
|
|
|
| 30 |
|
| 31 |
## Current architecture in three lines
|
| 32 |
|
| 33 |
+
- One LLM call per turn: **Gemini `gemini-2.5-flash` + function-calling
|
| 34 |
+
tools** (`save_profile_field` / `retrieve_policies` /
|
| 35 |
+
`mark_recommendation`) in `backend/single_brain.py` / `brain_tools.py`.
|
| 36 |
+
A single call drives fact-find, retrieval, QA, and recommendation. On a
|
| 37 |
+
transient Gemini error → small `backend/nim_fallback.py` (NVIDIA NIM) so
|
| 38 |
+
the turn completes. Fail-loud.
|
| 39 |
+
- Retrieval: **structured + vector over Chroma + BGE-small local 384-d**
|
| 40 |
+
(`rag/retrieve.py`) with a profile-tuned scorecard; shared policies
|
| 41 |
+
collection + per-session 24h quarantine for user-uploaded PDFs (8
|
| 42 |
+
security gates in `backend/security.py`).
|
| 43 |
- **Next.js 16 / FastAPI**, deployed as an HF Space (Docker, `uvicorn`,
|
| 44 |
port 7860); heavy data pulled at build from the `insurance-bot-data` HF
|
| 45 |
dataset; curated facts in `40-data/`.
|
|
@@ -1,10 +1,12 @@
|
|
| 1 |
# Discovery Conversation Script
|
| 2 |
|
| 3 |
-
> ⚠️ **
|
| 4 |
-
>
|
| 5 |
-
>
|
| 6 |
-
>
|
| 7 |
-
>
|
|
|
|
|
|
|
| 8 |
|
| 9 |
The 10-turn fact-find that turns a stranger into a profiled buyer.
|
| 10 |
|
|
@@ -135,13 +137,13 @@ This is the customer-protection framing. It tells the user honesty is **self-pro
|
|
| 135 |
|
| 136 |
## Implementation notes
|
| 137 |
|
| 138 |
-
The 9-slot fact-find SCHEMA still lives in `backend/needs_finder.py::GRAPH` — used now as the schema source
|
| 139 |
|
| 140 |
To add a new question:
|
| 141 |
|
| 142 |
1. Add a `Question(...)` entry with `id`, `field` (which Profile attribute it sets), `is_core` (boolean — counts toward completeness), optional `condition` callable, optional `parser`.
|
| 143 |
-
2. Surface the new slot in the
|
| 144 |
-
3. Wire any post-capture validation into `backend/
|
| 145 |
4. Add a row in `70-docs/scorecard-knowledge-graph.md` Part B showing how the new input shifts weights.
|
| 146 |
5. Wire the shift into `_profile_tuned_weights()` in `backend/scorecard.py`.
|
| 147 |
|
|
|
|
| 1 |
# Discovery Conversation Script
|
| 2 |
|
| 3 |
+
> ⚠️ **Design intent, not the present-state map.** The turn-by-turn script
|
| 4 |
+
> and adaptive rules below capture the discovery *intent*; the implementation
|
| 5 |
+
> is now a single Gemini 2.5-flash call per turn with function-calling tools
|
| 6 |
+
> (`backend/single_brain.py` / `brain_tools.py`) — there is no scripted
|
| 7 |
+
> question renderer, no `orchestrator`, no `sales_brain` / `qa_brain` split,
|
| 8 |
+
> and no separate faithfulness judge. Present-state authority:
|
| 9 |
+
> [`README.md`](../../README.md) §4.
|
| 10 |
|
| 11 |
The 10-turn fact-find that turns a stranger into a profiled buyer.
|
| 12 |
|
|
|
|
| 137 |
|
| 138 |
## Implementation notes
|
| 139 |
|
| 140 |
+
The 9-slot fact-find SCHEMA still lives in `backend/needs_finder.py::GRAPH` — used now as the schema source embedded in the single-LLM system prompt rather than as a scripted question list. Each entry's `id`, `field`, `is_core`, and `condition` are consumed by the LLM as a structured contract; the `prompt_en` / `prompt_hi` strings are no longer rendered to the user (the LLM owns voice + cadence end-to-end via its system prompt + the conversation so far, and records facts through the `save_profile_field` tool).
|
| 141 |
|
| 142 |
To add a new question:
|
| 143 |
|
| 144 |
1. Add a `Question(...)` entry with `id`, `field` (which Profile attribute it sets), `is_core` (boolean — counts toward completeness), optional `condition` callable, optional `parser`.
|
| 145 |
+
2. Surface the new slot in the single-brain system prompt's 9-slot schema (alongside accepted value shapes + examples) so the LLM knows to capture it. See the system prompt in `backend/single_brain.py`.
|
| 146 |
+
3. Wire any post-capture validation into the `save_profile_field` handler in `backend/brain_tools.py` (enum coercion, INR parsing, bounds).
|
| 147 |
4. Add a row in `70-docs/scorecard-knowledge-graph.md` Part B showing how the new input shifts weights.
|
| 148 |
5. Wire the shift into `_profile_tuned_weights()` in `backend/scorecard.py`.
|
| 149 |
|
|
@@ -1,35 +1,38 @@
|
|
| 1 |
-
# 05 — Needs Analysis Flow (LLM-
|
| 2 |
|
| 3 |
| Field | Value |
|
| 4 |
| --- | --- |
|
| 5 |
| Project | Insurance Sales Portfolio Expert |
|
| 6 |
-
| Version | 0.
|
| 7 |
| Date | 2026-05-17 |
|
| 8 |
-
| Implementation | `backend/single_brain.py`
|
| 9 |
|
| 10 |
-
> ℹ️ **
|
| 11 |
-
>
|
| 12 |
-
>
|
| 13 |
-
>
|
| 14 |
-
>
|
|
|
|
|
|
|
|
|
|
| 15 |
|
| 16 |
## 0. Why one LLM call per turn (not "scripted question graph + paraphraser")
|
| 17 |
|
| 18 |
A good Independent Financial Advisor opens with a stable, repeatable set of slots to fill — and adapts which slot to drive next based on what the buyer just said, what they've already told us, and what's still missing. We replicate this with **one LLM call per turn** that owns the entire fact-find surface (voice, cadence, slot-selection, multi-fact capture in a single turn), not a hardcoded state machine with a paraphraser glued on top.
|
| 19 |
|
| 20 |
-
The original v0.1 of this doc was an explicit `GRAPH` of canonical questions with `prompt_en` / `prompt_hi` strings
|
| 21 |
|
| 22 |
What we kept and what we changed:
|
| 23 |
|
| 24 |
-
| Concern | v0.1 (scripted graph) | Current (LLM
|
| 25 |
|---|---|---|
|
| 26 |
| **Slot schema** | `GRAPH` of 9 `Question(id, prompt_en, prompt_hi, field, is_core, condition, parser)` entries | `GRAPH` data structure retained as the schema source; `prompt_en` / `prompt_hi` are dead strings now |
|
| 27 |
| **Question text** | Hardcoded `prompt_en` rendered to user | LLM generates natural prose in the advisor's voice per turn (no template) |
|
| 28 |
-
| **Slot selection** | `next_question(profile)` walked the graph in order |
|
| 29 |
-
| **Multi-fact capture** | One slot per turn (a user saying "I'm 32, just myself, in Mumbai" only filled `age`) |
|
| 30 |
-
| **Auditable behavior** | Graph order was the trace | Per-turn LLM
|
| 31 |
-
| **Fail-soft** | Graph survived LLM degradation |
|
| 32 |
-
| **Bilingual** | Hand-authored `prompt_en` + `prompt_hi` | Sarvam-M translation
|
| 33 |
|
| 34 |
## 1. The 9-slot schema (data-only — the LLM consults this through its system prompt)
|
| 35 |
|
|
@@ -87,53 +90,35 @@ What we kept and what we changed:
|
|
| 87 |
|
| 88 |
## 2. Termination criteria
|
| 89 |
|
| 90 |
-
|
| 91 |
|
| 92 |
-
|
| 93 |
|
| 94 |
## 3. Bilingual handling
|
| 95 |
|
| 96 |
The LLM authors in English; Sarvam-M translates Hinglish ↔ English at the cascade boundary. The "Indic-native, not translated" principle (problem-statement §5.3) is preserved by the LLM's native-quality output rather than by hand-authored Hindi strings — the cascade itself is the bilingual surface.
|
| 97 |
|
| 98 |
-
## 4.
|
| 99 |
|
| 100 |
-
|
| 101 |
|
| 102 |
- Resolves field-name aliases (`location` → `location_tier`)
|
| 103 |
- Coerces enums (`Bangalore` → `metro`)
|
| 104 |
- Parses INR amounts (`"32 lakh"` → `3200000`)
|
| 105 |
-
- Drops null / empty values (
|
| 106 |
- Validates types and bounds (rejects `age: 250`)
|
| 107 |
|
| 108 |
-
|
| 109 |
-
|
| 110 |
-
## 5. How the
|
| 111 |
-
|
| 112 |
-
``
|
| 113 |
-
|
| 114 |
-
|
| 115 |
-
|
| 116 |
-
|
| 117 |
-
|
| 118 |
-
|
| 119 |
-
history=session.chat_history[-N:],
|
| 120 |
-
user_text=user_text,
|
| 121 |
-
)
|
| 122 |
-
# outcome.reply, outcome.captures, outcome.slot_driving, outcome.complete
|
| 123 |
-
apply_normalized_captures(session, outcome.captures)
|
| 124 |
-
if outcome.complete:
|
| 125 |
-
session.free_form_session = True
|
| 126 |
-
reply = outcome.reply
|
| 127 |
-
else:
|
| 128 |
-
# QA / comparison / recommendation → Brain Main
|
| 129 |
-
chunks = retrieve(user_query, top_k=10)
|
| 130 |
-
reply = await BRAIN_CHAIN.chat(persona_prompt + history + chunks + user_text)
|
| 131 |
-
if intent not in {"fact_find", "recommendation"}:
|
| 132 |
-
# KI-171: judge skipped on fact-find + recommendation intents
|
| 133 |
-
verdict = await faithfulness.check(reply, chunks)
|
| 134 |
-
if not verdict.supported:
|
| 135 |
-
reply = SAFE_REFUSAL
|
| 136 |
-
```
|
| 137 |
|
| 138 |
## 6. What the user sees
|
| 139 |
|
|
@@ -159,14 +144,13 @@ Bot: "Great — that's everything I need. Based on your profile, three policies
|
|
| 159 |
stand out: [...]"
|
| 160 |
```
|
| 161 |
|
| 162 |
-
Note: the LLM captured `{name, age, dependents, location_tier}` in a single turn from the user's opener
|
| 163 |
|
| 164 |
## 7. v2 enhancements
|
| 165 |
|
| 166 |
| # | Enhancement | Why |
|
| 167 |
| --- | --- | --- |
|
| 168 |
-
| 1 | Stream the prose token-by-token to the frontend |
|
| 169 |
-
| 2 |
|
| 170 |
-
| 3 |
|
| 171 |
-
| 4 |
|
| 172 |
-
| 5 | Tone-match the user's energy (formal vs casual) | The LLM already does this implicitly via the system prompt; could be explicit per-session signal |
|
|
|
|
| 1 |
+
# 05 — Needs Analysis Flow (single-LLM-with-tools)
|
| 2 |
|
| 3 |
| Field | Value |
|
| 4 |
| --- | --- |
|
| 5 |
| Project | Insurance Sales Portfolio Expert |
|
| 6 |
+
| Version | 0.3 |
|
| 7 |
| Date | 2026-05-17 |
|
| 8 |
+
| Implementation | `backend/single_brain.py` + `backend/brain_tools.py`. The slot schema still lives in `backend/needs_finder.py::GRAPH`. |
|
| 9 |
|
| 10 |
+
> ℹ️ **Design intent below is still accurate; implementation pointers
|
| 11 |
+
> updated.** The core idea — *one LLM call per turn owns the entire
|
| 12 |
+
> fact-find surface* — is what the single-LLM-with-tools handler does
|
| 13 |
+
> today. Fact-find, retrieval, QA, and recommendation are all handled in
|
| 14 |
+
> that one call via function-calling tools; the separate `sales_brain` /
|
| 15 |
+
> `qa_brain` split, the orchestrator, the profile_extractor, and the
|
| 16 |
+
> separate faithfulness judge were removed. Present-state authority:
|
| 17 |
+
> [`README.md`](../../README.md) §4.
|
| 18 |
|
| 19 |
## 0. Why one LLM call per turn (not "scripted question graph + paraphraser")
|
| 20 |
|
| 21 |
A good Independent Financial Advisor opens with a stable, repeatable set of slots to fill — and adapts which slot to drive next based on what the buyer just said, what they've already told us, and what's still missing. We replicate this with **one LLM call per turn** that owns the entire fact-find surface (voice, cadence, slot-selection, multi-fact capture in a single turn), not a hardcoded state machine with a paraphraser glued on top.
|
| 22 |
|
| 23 |
+
The original v0.1 of this doc was an explicit `GRAPH` of canonical questions with `prompt_en` / `prompt_hi` strings, rendered by a scripted state machine. That scripted approach (and a later one-call-brain variant with a structured trailer + canonical fallback) produced robotic cadence: scripted prompts leaked into fallback turns. The current design removes all of that. There are no scripted prompts and no canonical fallback. One **Gemini 2.5-flash call per turn with function-calling** owns voice, cadence, and slot selection; the `GRAPH` is consulted only as the slot *schema*.
|
| 24 |
|
| 25 |
What we kept and what we changed:
|
| 26 |
|
| 27 |
+
| Concern | v0.1 (scripted graph) | Current (single LLM + tools) |
|
| 28 |
|---|---|---|
|
| 29 |
| **Slot schema** | `GRAPH` of 9 `Question(id, prompt_en, prompt_hi, field, is_core, condition, parser)` entries | `GRAPH` data structure retained as the schema source; `prompt_en` / `prompt_hi` are dead strings now |
|
| 30 |
| **Question text** | Hardcoded `prompt_en` rendered to user | LLM generates natural prose in the advisor's voice per turn (no template) |
|
| 31 |
+
| **Slot selection** | `next_question(profile)` walked the graph in order | The single LLM decides which slot to drive next from the schema + conversation so far (required slots first) |
|
| 32 |
+
| **Multi-fact capture** | One slot per turn (a user saying "I'm 32, just myself, in Mumbai" only filled `age`) | One utterance can fill 2-4 slots; the LLM calls `save_profile_field` per captured fact |
|
| 33 |
+
| **Auditable behavior** | Graph order was the trace | Per-turn LLM call + tool calls are logged (`save_profile_field` / `retrieve_policies` / `mark_recommendation`) in `40-data/llm_usage.jsonl` + `logs/turns.jsonl` |
|
| 34 |
+
| **Fail-soft** | Graph survived LLM degradation | On a transient Gemini error the small `backend/nim_fallback.py` (NVIDIA NIM) completes the turn; fail-loud otherwise — no scripted reply |
|
| 35 |
+
| **Bilingual** | Hand-authored `prompt_en` + `prompt_hi` | Sarvam-M translation on the LLM's output (English authoring + Indic translation), same UX, fewer hand-authored strings |
|
| 36 |
|
| 37 |
## 1. The 9-slot schema (data-only — the LLM consults this through its system prompt)
|
| 38 |
|
|
|
|
| 90 |
|
| 91 |
## 2. Termination criteria
|
| 92 |
|
| 93 |
+
Fact-find is "done" when all required slots are filled. Because the single LLM owns the whole turn, it stops asking discovery questions and moves to retrieval/recommendation once the profile is sufficiently complete; required-slot checks against the `GRAPH` schema keep it from declaring done prematurely.
|
| 94 |
|
| 95 |
+
The user can also opt out — if the user immediately asks "compare Star and HDFC ERGO" or "show me the top 3 policies for me", the same LLM call can skip remaining discovery and go straight to `retrieve_policies` / `mark_recommendation` within that turn.
|
| 96 |
|
| 97 |
## 3. Bilingual handling
|
| 98 |
|
| 99 |
The LLM authors in English; Sarvam-M translates Hinglish ↔ English at the cascade boundary. The "Indic-native, not translated" principle (problem-statement §5.3) is preserved by the LLM's native-quality output rather than by hand-authored Hindi strings — the cascade itself is the bilingual surface.
|
| 100 |
|
| 101 |
+
## 4. Capturing user answers
|
| 102 |
|
| 103 |
+
When the user states a fact, the LLM calls the `save_profile_field` tool (e.g. `save_profile_field(field="age", value=34)`). The tool implementation in `backend/brain_tools.py` is a pure-function handler that:
|
| 104 |
|
| 105 |
- Resolves field-name aliases (`location` → `location_tier`)
|
| 106 |
- Coerces enums (`Bangalore` → `metro`)
|
| 107 |
- Parses INR amounts (`"32 lakh"` → `3200000`)
|
| 108 |
+
- Drops null / empty values (a `None`/empty value never overwrites a filled field)
|
| 109 |
- Validates types and bounds (rejects `age: 250`)
|
| 110 |
|
| 111 |
+
The handler is pure rules — no nested LLM call. Captured fields are persisted to the session profile.
|
| 112 |
+
|
| 113 |
+
## 5. How the single LLM uses this
|
| 114 |
+
|
| 115 |
+
There is no separate orchestrator / intent-classifier / sales-brain split. One `single_brain` call per turn receives the system prompt (which embeds the slot schema), the recent history, and the user's text, and uses function-calling tools to do everything in that one turn:
|
| 116 |
+
|
| 117 |
+
- `save_profile_field` — record a captured fact (fact-find).
|
| 118 |
+
- `retrieve_policies` — structured + vector retrieval over the corpus when the turn needs policy evidence (QA / comparison / recommendation).
|
| 119 |
+
- `mark_recommendation` — record the policies the LLM recommends, surfaced with the profile-tuned scorecard.
|
| 120 |
+
|
| 121 |
+
Because the LLM can only answer from what `retrieve_policies` returned and must cite it, grounding is structural — there is no separate faithfulness-judge LLM in the loop. On a transient Gemini error, `backend/nim_fallback.py` completes the turn.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 122 |
|
| 123 |
## 6. What the user sees
|
| 124 |
|
|
|
|
| 144 |
stand out: [...]"
|
| 145 |
```
|
| 146 |
|
| 147 |
+
Note: the LLM captured `{name, age, dependents, location_tier}` in a single turn from the user's opener via `save_profile_field` calls. The exact turn count varies — some sessions finish in 3-4 turns, others in 6-7 depending on what the user volunteers.
|
| 148 |
|
| 149 |
## 7. v2 enhancements
|
| 150 |
|
| 151 |
| # | Enhancement | Why |
|
| 152 |
| --- | --- | --- |
|
| 153 |
+
| 1 | Stream the prose token-by-token to the frontend | Cuts perceived latency on longer replies |
|
| 154 |
+
| 2 | Skip-confirm flow ("you can skip this — say 'skip'") | Buyer autonomy |
|
| 155 |
+
| 3 | Save profile across sessions ✅ shipped KI-040 | Returning user picks up where they left off |
|
| 156 |
+
| 4 | Tone-match the user's energy (formal vs casual) | The LLM already does this implicitly via the system prompt; could be explicit per-session signal |
|
|
|
|
@@ -1,17 +1,21 @@
|
|
| 1 |
# 03 — Evaluation Plan
|
| 2 |
|
| 3 |
-
> ⚠️ **
|
| 4 |
-
>
|
| 5 |
-
>
|
| 6 |
-
>
|
| 7 |
-
>
|
|
|
|
|
|
|
|
|
|
|
|
|
| 8 |
|
| 9 |
| Field | Value |
|
| 10 |
| --- | --- |
|
| 11 |
| Project | Insurance Sales Portfolio Expert |
|
| 12 |
| Version | 0.1 |
|
| 13 |
| Date | 2026-05-13 |
|
| 14 |
-
| Depends on | `01-requirements.md` §6 (success criteria)
|
| 15 |
| Status | Pipeline implemented; full run pending corpus completion |
|
| 16 |
|
| 17 |
## 0. Purpose
|
|
@@ -28,7 +32,7 @@ These directly back the success criteria in Doc 01 §6: C2 (≥95% factual), C3
|
|
| 28 |
|
| 29 |
### Pipeline A — auto-generated from structured extraction (the bulk)
|
| 30 |
|
| 31 |
-
For every successfully extracted policy
|
| 32 |
|
| 33 |
- ~15 question templates × ~80 policies = ~1,100 candidate pairs
|
| 34 |
- Each pair is **fully reproducible** — the answer traces to a specific field which traces to a specific clause
|
|
@@ -38,7 +42,7 @@ For every successfully extracted policy in DuckDB, generate templated Q&A where
|
|
| 38 |
|
| 39 |
### Pipeline B — LLM-drafted nuanced questions (curated)
|
| 40 |
|
| 41 |
-
For top-priority policies (top 10–20), run
|
| 42 |
|
| 43 |
Target: 5 × 20 = 100 questions covering multi-clause and edge-case reasoning that Pipeline A can't generate.
|
| 44 |
|
|
@@ -56,15 +60,15 @@ Each pair marked with `expected_refusal: bool`. Refusal cases test that the bot
|
|
| 56 |
|
| 57 |
## 2. Grader design
|
| 58 |
|
| 59 |
-
**Single grading endpoint:** `eval/run.py` calls `backend.
|
| 60 |
|
| 61 |
### Signal 1 — Regex hard-checks (deterministic)
|
| 62 |
|
| 63 |
Numbers, dates, currency, durations, percentages are extracted via regex from both gold and bot reply. Exact match (after normalization) is the strongest signal. Catches "the premium is ₹15,000" hallucinations without an LLM.
|
| 64 |
|
| 65 |
-
### Signal 2 — LLM-judge faithfulness (
|
| 66 |
|
| 67 |
-
|
| 68 |
|
| 69 |
> Given GOLD and BOT answers, output strict JSON: `{factual_match: bool, citation_present: bool, score: 0-1, reason: str}`. Be strict on partial matches.
|
| 70 |
|
|
@@ -82,11 +86,11 @@ Different model family from the Gemini-led Brain Fast / Brain Main → **non-cir
|
|
| 82 |
| Citation accuracy | C3 ≥ 95% | n_correct_citations / n_non_refusal |
|
| 83 |
| Refusal precision | C4 ≥ 90% | n_correct_refusals / n_expected_refusals |
|
| 84 |
| Hindi parity | C8 within 5pp | factual_acc(hi) vs factual_acc(en) |
|
| 85 |
-
|
|
| 86 |
|
| 87 |
## 4. Output artifacts per run
|
| 88 |
|
| 89 |
-
- `eval/results.md` — human-readable summary with per-type, per-
|
| 90 |
- `eval/results.json` — machine-readable full per-question record
|
| 91 |
- `logs/hallucinations.jsonl` — every blocked reply with its reason (audit log)
|
| 92 |
|
|
@@ -96,8 +100,8 @@ Different model family from the Gemini-led Brain Fast / Brain Main → **non-cir
|
|
| 96 |
| --- | --- | --- |
|
| 97 |
| Development | Manual, after meaningful changes | `python -m eval.run` |
|
| 98 |
| Pre-deploy | Every PR | GitHub Actions runs eval, blocks merge if accuracy regresses |
|
| 99 |
-
| Production | Nightly synthetic + spot-grading |
|
| 100 |
-
| Post-deploy verification | Per deploy | `tests/live_verify.py` runs full eval against live
|
| 101 |
|
| 102 |
## 6. Bilingual eval
|
| 103 |
|
|
@@ -106,6 +110,6 @@ Bilingual sub-set: 20 questions translated to Hindi + Hinglish via Sarvam-M with
|
|
| 106 |
## 7. Known limitations (transparent)
|
| 107 |
|
| 108 |
- **Sample bias:** Pipeline A questions are templated, so accuracy on Pipeline A is upper-bound real-world performance. Pipeline B + C provide the harder signal.
|
| 109 |
-
- **Single judge model:**
|
| 110 |
- **No human evaluation:** Cost-prohibitive for 1,100 questions. We rely on the grader; manual spot-check 5%.
|
| 111 |
- **No latency budget enforcement in eval:** Latency is captured per record but doesn't gate the score. Doc 01 C1 (p50 ≤ 4s) is monitored separately.
|
|
|
|
| 1 |
# 03 — Evaluation Plan
|
| 2 |
|
| 3 |
+
> ⚠️ **Methodology is current; some implementation pointers are historical.**
|
| 4 |
+
> The gold-set construction, grader signals, and metrics below still
|
| 5 |
+
> describe how we evaluate the bot. But the *system under test* is now the
|
| 6 |
+
> single-LLM-with-tools handler (`backend/single_brain.py` — one Gemini
|
| 7 |
+
> 2.5-flash call per turn with `save_profile_field` / `retrieve_policies` /
|
| 8 |
+
> `mark_recommendation`, structured+vector retrieval, small
|
| 9 |
+
> `nim_fallback.py` for transient errors). There is no `orchestrator`, no
|
| 10 |
+
> `sales_brain`/`qa_brain` split, no separate faithfulness-judge LLM, and no
|
| 11 |
+
> DuckDB hot path. Present-state authority: [`README.md`](../../README.md) §4.
|
| 12 |
|
| 13 |
| Field | Value |
|
| 14 |
| --- | --- |
|
| 15 |
| Project | Insurance Sales Portfolio Expert |
|
| 16 |
| Version | 0.1 |
|
| 17 |
| Date | 2026-05-13 |
|
| 18 |
+
| Depends on | `01-requirements.md` §6 (success criteria); README §4 (system under test) |
|
| 19 |
| Status | Pipeline implemented; full run pending corpus completion |
|
| 20 |
|
| 21 |
## 0. Purpose
|
|
|
|
| 32 |
|
| 33 |
### Pipeline A — auto-generated from structured extraction (the bulk)
|
| 34 |
|
| 35 |
+
For every successfully extracted policy, generate templated Q&A where the answer comes directly from the 62-field structured extraction. Per `eval/generate_gold.py`:
|
| 36 |
|
| 37 |
- ~15 question templates × ~80 policies = ~1,100 candidate pairs
|
| 38 |
- Each pair is **fully reproducible** — the answer traces to a specific field which traces to a specific clause
|
|
|
|
| 42 |
|
| 43 |
### Pipeline B — LLM-drafted nuanced questions (curated)
|
| 44 |
|
| 45 |
+
For top-priority policies (top 10–20), run an LLM on the policy text with prompt: *"generate 5 buyer-style questions whose answers are explicitly in this document; include the source clause."* Each generated pair is **human spot-checked** before commit.
|
| 46 |
|
| 47 |
Target: 5 × 20 = 100 questions covering multi-clause and edge-case reasoning that Pipeline A can't generate.
|
| 48 |
|
|
|
|
| 60 |
|
| 61 |
## 2. Grader design
|
| 62 |
|
| 63 |
+
**Single grading endpoint:** `eval/run.py` calls the single-LLM-with-tools turn handler (`backend/single_brain.py`) in-process, takes the reply, scores against gold with three signals:
|
| 64 |
|
| 65 |
### Signal 1 — Regex hard-checks (deterministic)
|
| 66 |
|
| 67 |
Numbers, dates, currency, durations, percentages are extracted via regex from both gold and bot reply. Exact match (after normalization) is the strongest signal. Catches "the premium is ₹15,000" hallucinations without an LLM.
|
| 68 |
|
| 69 |
+
### Signal 2 — LLM-judge faithfulness (offline eval grader only)
|
| 70 |
|
| 71 |
+
The eval grader uses a **different model family** from the runtime Gemini 2.5-flash brain → **non-circular evaluation**. This judge is part of the *offline eval harness only* — it is not a runtime gate in the bot (runtime grounding is structural: the single LLM can only state what `retrieve_policies` returned and must cite it). Judge prompt:
|
| 72 |
|
| 73 |
> Given GOLD and BOT answers, output strict JSON: `{factual_match: bool, citation_present: bool, score: 0-1, reason: str}`. Be strict on partial matches.
|
| 74 |
|
|
|
|
| 86 |
| Citation accuracy | C3 ≥ 95% | n_correct_citations / n_non_refusal |
|
| 87 |
| Refusal precision | C4 ≥ 90% | n_correct_refusals / n_expected_refusals |
|
| 88 |
| Hindi parity | C8 within 5pp | factual_acc(hi) vs factual_acc(en) |
|
| 89 |
+
| Path winners | (config) | factual_acc grouped by which path produced the turn — `gemini` (primary single-LLM call) vs `nim_fallback` (transient-error fallback) |
|
| 90 |
|
| 91 |
## 4. Output artifacts per run
|
| 92 |
|
| 93 |
+
- `eval/results.md` — human-readable summary with per-type, per-path accuracy + sample misses
|
| 94 |
- `eval/results.json` — machine-readable full per-question record
|
| 95 |
- `logs/hallucinations.jsonl` — every blocked reply with its reason (audit log)
|
| 96 |
|
|
|
|
| 100 |
| --- | --- | --- |
|
| 101 |
| Development | Manual, after meaningful changes | `python -m eval.run` |
|
| 102 |
| Pre-deploy | Every PR | GitHub Actions runs eval, blocks merge if accuracy regresses |
|
| 103 |
+
| Production | Nightly synthetic + spot-grading | Scheduled job; live-traffic sampling via Playwright |
|
| 104 |
+
| Post-deploy verification | Per deploy | `tests/live_verify.py` runs full eval against the live HF Space URL |
|
| 105 |
|
| 106 |
## 6. Bilingual eval
|
| 107 |
|
|
|
|
| 110 |
## 7. Known limitations (transparent)
|
| 111 |
|
| 112 |
- **Sample bias:** Pipeline A questions are templated, so accuracy on Pipeline A is upper-bound real-world performance. Pipeline B + C provide the harder signal.
|
| 113 |
+
- **Single judge model:** the offline eval grader uses one judge model (a different family from the runtime Gemini brain) per question. Risk of judge-specific bias. v2: 3-judge consensus.
|
| 114 |
- **No human evaluation:** Cost-prohibitive for 1,100 questions. We rely on the grader; manual spot-check 5%.
|
| 115 |
- **No latency budget enforcement in eval:** Latency is captured per record but doesn't gate the score. Doc 01 C1 (p50 ≤ 4s) is monitored separately.
|
|
@@ -1,5 +1,16 @@
|
|
| 1 |
# Known Issues + Quality Sprint Log
|
| 2 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3 |
Living document. Every defect we find — whether via code review, eval audit,
|
| 4 |
or production observation — lands here with severity, root cause, and the
|
| 5 |
plan to fix it. Closed issues stay in the log with a `**FIXED in <sha>**`
|
|
|
|
| 1 |
# Known Issues + Quality Sprint Log
|
| 2 |
|
| 3 |
+
> ⚠️ **Historical defect log — not the present-state map.** Entries below
|
| 4 |
+
> are dated incidents; many reference subsystems (`orchestrator`,
|
| 5 |
+
> `faithfulness.py` judge, `fact_find_brain` / `sales_brain`,
|
| 6 |
+
> `fact_find_normalizer`, the 3-tier Gemini/NIM/OpenRouter chain,
|
| 7 |
+
> sticky-primary election) that were **subsequently removed** by the
|
| 8 |
+
> single-LLM-with-tools rewrite (one Gemini 2.5-flash call per turn with
|
| 9 |
+
> `save_profile_field` / `retrieve_policies` / `mark_recommendation`,
|
| 10 |
+
> structured+vector retrieval, small `nim_fallback`). The log is retained
|
| 11 |
+
> as the quality trajectory of record; KI/ADR cross-references stay valid as
|
| 12 |
+
> history. Present-state authority: [`README.md`](../../README.md) §4.
|
| 13 |
+
|
| 14 |
Living document. Every defect we find — whether via code review, eval audit,
|
| 15 |
or production observation — lands here with severity, root cause, and the
|
| 16 |
plan to fix it. Closed issues stay in the log with a `**FIXED in <sha>**`
|
|
@@ -1,5 +1,15 @@
|
|
| 1 |
# Quality Sprint — 2026-05-14
|
| 2 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3 |
One-day sprint focused on hardening the non-speech parts of the bot via
|
| 4 |
code review, large-scale audit simulation, and real user-testing
|
| 5 |
feedback. Every defect found in the same day shipped a fix in the same
|
|
|
|
| 1 |
# Quality Sprint — 2026-05-14
|
| 2 |
|
| 3 |
+
> ⚠️ **Historical sprint log — not the present-state map.** This records
|
| 4 |
+
> dated quality work; the architecture described mid-log (`sales_brain` /
|
| 5 |
+
> `fact_find_brain` / orchestrator / separate faithfulness judge / 3-tier
|
| 6 |
+
> Gemini-2.0/2.5 + NIM/OpenRouter chain / sticky-primary election) was
|
| 7 |
+
> subsequently superseded by the single-LLM-with-tools rewrite (one Gemini
|
| 8 |
+
> 2.5-flash call per turn with `save_profile_field` / `retrieve_policies` /
|
| 9 |
+
> `mark_recommendation`, structured+vector retrieval, small `nim_fallback`).
|
| 10 |
+
> KI/ADR pointers below remain valid as history-of-record. Present-state
|
| 11 |
+
> authority: [`README.md`](../../README.md) §4.
|
| 12 |
+
|
| 13 |
One-day sprint focused on hardening the non-speech parts of the bot via
|
| 14 |
code review, large-scale audit simulation, and real user-testing
|
| 15 |
feedback. Every defect found in the same day shipped a fix in the same
|
|
@@ -19,13 +19,15 @@ authority.
|
|
| 19 |
`/api/coverage`, `/api/profile*`, `/api/scorecard`, `/api/session*`,
|
| 20 |
`/api/admin/*`.
|
| 21 |
- **Brain** — one LLM call per turn: Google Gemini
|
| 22 |
-
(`gemini-2.5-flash
|
| 23 |
-
(`
|
| 24 |
-
|
| 25 |
-
|
| 26 |
-
|
| 27 |
-
(
|
| 28 |
-
|
|
|
|
|
|
|
| 29 |
- **Retrieval** — Chroma vector store, BGE-small-en-v1.5 local 384-d
|
| 30 |
embeddings (`rag/retrieve.py`). Shared "policies" collection (~150 plans,
|
| 31 |
~7.3k chunks, 20 insurers) + a per-session "quarantine" collection for
|
|
|
|
| 19 |
`/api/coverage`, `/api/profile*`, `/api/scorecard`, `/api/session*`,
|
| 20 |
`/api/admin/*`.
|
| 21 |
- **Brain** — one LLM call per turn: Google Gemini
|
| 22 |
+
(`gemini-2.5-flash`) + function-calling tools
|
| 23 |
+
(`save_profile_field`, `retrieve_policies`, `mark_recommendation`) in
|
| 24 |
+
`backend/single_brain.py` / `backend/brain_tools.py`. A single call owns
|
| 25 |
+
the whole turn: fact-find, retrieval, QA, and recommendation. On a
|
| 26 |
+
transient Gemini error / cold-start 503 → small `backend/nim_fallback.py`
|
| 27 |
+
(NVIDIA NIM) so the turn still completes. Fail-loud, never silently wrong.
|
| 28 |
+
The legacy multi-pass design (orchestrator / sales-brain / QA-brain /
|
| 29 |
+
separate faithfulness judge / profile_extractor / tiered brain) was
|
| 30 |
+
removed — it does not exist in the codebase.
|
| 31 |
- **Retrieval** — Chroma vector store, BGE-small-en-v1.5 local 384-d
|
| 32 |
embeddings (`rag/retrieve.py`). Shared "policies" collection (~150 plans,
|
| 33 |
~7.3k chunks, 20 insurers) + a per-session "quarantine" collection for
|
|
@@ -13,10 +13,10 @@ Setup:
|
|
| 13 |
Add to Space secrets (Settings → Variables and secrets):
|
| 14 |
ADMIN_PASSWORD = your-strong-password-here
|
| 15 |
|
| 16 |
-
Returns 401 Unauthorized on bad/missing password.
|
| 17 |
-
|
| 18 |
-
|
| 19 |
-
|
| 20 |
"""
|
| 21 |
from __future__ import annotations
|
| 22 |
|
|
@@ -67,18 +67,15 @@ def _password_ok(supplied: Optional[str]) -> bool:
|
|
| 67 |
|
| 68 |
|
| 69 |
def _check_admin(request: Request, password: Optional[str]) -> None:
|
| 70 |
-
"""
|
| 71 |
|
| 72 |
-
`request` is kept in the signature so callsites don't churn
|
| 73 |
-
function
|
| 74 |
-
(IP allowlist + 404-to-hide-existence) was removed in KI-097.
|
| 75 |
"""
|
| 76 |
-
# TODO: enforce IP allowlist
|
| 77 |
-
#
|
| 78 |
-
#
|
| 79 |
-
#
|
| 80 |
-
# re-introduce an IP allowlist as a second factor (with a documented
|
| 81 |
-
# break-glass procedure so a network change doesn't lock ops out).
|
| 82 |
if not _password_ok(password):
|
| 83 |
raise HTTPException(status_code=401, detail="Unauthorized")
|
| 84 |
|
|
@@ -547,14 +544,12 @@ def _read_usage_24h() -> Optional[dict]:
|
|
| 547 |
last USAGE_TAIL_LINES rows (typically covers ≈24h of activity at current
|
| 548 |
traffic). Keeping the name aligns with the admin UI label.
|
| 549 |
|
| 550 |
-
|
| 551 |
-
|
| 552 |
-
|
| 553 |
-
|
| 554 |
-
|
| 555 |
-
|
| 556 |
-
/api/admin/usage already does via its `chains` dict — instead of blindly
|
| 557 |
-
surfacing every historical role value. llm_health.ROLES is the single
|
| 558 |
source of truth (also used as _LLM_HEALTH_CHAIN_ROLES below).
|
| 559 |
"""
|
| 560 |
usage_path = settings.DATA_DIR / "llm_usage.jsonl"
|
|
@@ -633,9 +628,9 @@ _LLM_HEALTH_CHAIN_ROLES = llm_health.ROLES
|
|
| 633 |
|
| 634 |
|
| 635 |
def _chain_names_map() -> dict[str, list[str]]:
|
| 636 |
-
"""Live (post-admin-override) chain config — read off the module so
|
| 637 |
-
reorders applied earlier in the same process are reflected
|
| 638 |
-
|
| 639 |
from backend.providers import nvidia_nim_llm as nim
|
| 640 |
return {
|
| 641 |
"brain": list(getattr(nim, "BRAIN_CHAIN", [])),
|
|
@@ -758,20 +753,20 @@ def _candidate_available_for_calls(h, now_mono: float) -> bool:
|
|
| 758 |
|
| 759 |
|
| 760 |
def _candidate_credit_exhausted_strict(h, now_mono: float) -> bool:
|
| 761 |
-
"""
|
| 762 |
|
| 763 |
True ONLY when ALL three hold:
|
| 764 |
(1) credits_remaining is NOT None (we have a real signal)
|
| 765 |
(2) credits_remaining <= credits_low_water
|
| 766 |
(3) credits_reset_at is set AND in the FUTURE (we're inside an
|
| 767 |
active gating window — stale or absent reset means the
|
| 768 |
-
snapshot is
|
| 769 |
|
| 770 |
This is intentionally stricter than `not _has_credits()` because the
|
| 771 |
banner is louder than the elector. The elector falls through cheaply
|
| 772 |
-
on a single bad candidate;
|
| 773 |
-
when one quota-exhausted backup co-exists with a perfectly
|
| 774 |
-
|
| 775 |
if h is None:
|
| 776 |
return False
|
| 777 |
if h.credits_remaining is None:
|
|
@@ -987,13 +982,11 @@ async def admin_llm_health(
|
|
| 987 |
# ---------------------------------------------------------------------------
|
| 988 |
|
| 989 |
# The seven canonical "ready-to-recommend" fact-find slots. Mirrors
|
| 990 |
-
#
|
| 991 |
-
#
|
| 992 |
-
#
|
| 993 |
-
#
|
| 994 |
-
#
|
| 995 |
-
# captured as a list (empty list counts as "asked but no conditions" → still
|
| 996 |
-
# a valid captured signal once the `asked` array contains the field name).
|
| 997 |
_PERSONA_DRIFT_SLOTS = ("name", "age", "dependents", "location_tier",
|
| 998 |
"income_band", "primary_goal", "health_conditions")
|
| 999 |
|
|
|
|
| 13 |
Add to Space secrets (Settings → Variables and secrets):
|
| 14 |
ADMIN_PASSWORD = your-strong-password-here
|
| 15 |
|
| 16 |
+
Returns 401 Unauthorized on bad/missing password. Access is gated by a
|
| 17 |
+
strong password only; there is no IP allowlist (it would add operational
|
| 18 |
+
complexity — changing networks would lock the operator out — without
|
| 19 |
+
meaningful additional security).
|
| 20 |
"""
|
| 21 |
from __future__ import annotations
|
| 22 |
|
|
|
|
| 67 |
|
| 68 |
|
| 69 |
def _check_admin(request: Request, password: Optional[str]) -> None:
|
| 70 |
+
"""Password-only gate. Raises 401 on bad/missing password.
|
| 71 |
|
| 72 |
+
`request` is kept in the signature so callsites don't churn; the
|
| 73 |
+
function does not inspect the client IP.
|
|
|
|
| 74 |
"""
|
| 75 |
+
# TODO: enforce an IP allowlist as a second factor for hardening (with
|
| 76 |
+
# a documented break-glass procedure so a network change doesn't lock
|
| 77 |
+
# ops out). The gate is password-only, acceptable for the current
|
| 78 |
+
# threat model.
|
|
|
|
|
|
|
| 79 |
if not _password_ok(password):
|
| 80 |
raise HTTPException(status_code=401, detail="Unauthorized")
|
| 81 |
|
|
|
|
| 544 |
last USAGE_TAIL_LINES rows (typically covers ≈24h of activity at current
|
| 545 |
traffic). Keeping the name aligns with the admin UI label.
|
| 546 |
|
| 547 |
+
The live stack emits the `brain` role (see
|
| 548 |
+
providers.nvidia_nim_llm.get_brain_llm + llm_health.ROLES). The
|
| 549 |
+
append-only llm_usage.jsonl can contain rows tagged with non-canonical
|
| 550 |
+
role values, so we filter to the canonical role set — the same way
|
| 551 |
+
/api/admin/usage does via its `chains` dict — instead of surfacing
|
| 552 |
+
every role value present in the file. llm_health.ROLES is the single
|
|
|
|
|
|
|
| 553 |
source of truth (also used as _LLM_HEALTH_CHAIN_ROLES below).
|
| 554 |
"""
|
| 555 |
usage_path = settings.DATA_DIR / "llm_usage.jsonl"
|
|
|
|
| 628 |
|
| 629 |
|
| 630 |
def _chain_names_map() -> dict[str, list[str]]:
|
| 631 |
+
"""Live (post-admin-override) chain config — read off the module so
|
| 632 |
+
admin reorders applied earlier in the same process are reflected
|
| 633 |
+
immediately. Only the brain role is present."""
|
| 634 |
from backend.providers import nvidia_nim_llm as nim
|
| 635 |
return {
|
| 636 |
"brain": list(getattr(nim, "BRAIN_CHAIN", [])),
|
|
|
|
| 753 |
|
| 754 |
|
| 755 |
def _candidate_credit_exhausted_strict(h, now_mono: float) -> bool:
|
| 756 |
+
"""STRICT credit-exhausted rule for a single candidate.
|
| 757 |
|
| 758 |
True ONLY when ALL three hold:
|
| 759 |
(1) credits_remaining is NOT None (we have a real signal)
|
| 760 |
(2) credits_remaining <= credits_low_water
|
| 761 |
(3) credits_reset_at is set AND in the FUTURE (we're inside an
|
| 762 |
active gating window — stale or absent reset means the
|
| 763 |
+
snapshot is not authoritative).
|
| 764 |
|
| 765 |
This is intentionally stricter than `not _has_credits()` because the
|
| 766 |
banner is louder than the elector. The elector falls through cheaply
|
| 767 |
+
on a single bad candidate; this strictness avoids falsely scaring the
|
| 768 |
+
operator when one quota-exhausted backup co-exists with a perfectly
|
| 769 |
+
healthy primary."""
|
| 770 |
if h is None:
|
| 771 |
return False
|
| 772 |
if h.credits_remaining is None:
|
|
|
|
| 982 |
# ---------------------------------------------------------------------------
|
| 983 |
|
| 984 |
# The seven canonical "ready-to-recommend" fact-find slots. Mirrors
|
| 985 |
+
# brain_tools._REQUIRED_FOR_READY (the recommendation-ready required set)
|
| 986 |
+
# and needs_finder.Profile's persisted fields. Adding/removing a slot must
|
| 987 |
+
# stay in sync with that required set. Health is captured as a list (empty
|
| 988 |
+
# list counts as "asked but no conditions" → still a valid captured signal
|
| 989 |
+
# once the `asked` array contains the field name).
|
|
|
|
|
|
|
| 990 |
_PERSONA_DRIFT_SLOTS = ("name", "age", "dependents", "location_tier",
|
| 991 |
"income_band", "primary_goal", "health_conditions")
|
| 992 |
|
|
@@ -88,41 +88,28 @@ _log = logging.getLogger(__name__)
|
|
| 88 |
|
| 89 |
_FACTS_DIR = settings.DATA_DIR / "policy_facts"
|
| 90 |
_DOCTYPE_SUFFIXES = ("__wordings", "__brochure", "__cis", "__prospectus")
|
| 91 |
-
#
|
| 92 |
-
#
|
| 93 |
-
#
|
| 94 |
-
#
|
| 95 |
-
# surface it). The LLM still only ever sees the top-k best-fit survivors.
|
| 96 |
_RECALL_POOL = 40
|
| 97 |
_FACT_KEYS = (
|
| 98 |
"policy_type_indemnity_or_fixed",
|
| 99 |
-
#
|
| 100 |
-
# (e.g.
|
| 101 |
-
#
|
| 102 |
-
# `policy_type_indemnity_or_fixed` key. Without surfacing this, the
|
| 103 |
-
# retrieval_filters fixed-benefit gate is blind and a daily-cash plan
|
| 104 |
-
# ranks #1 for a comprehensive-indemnity buyer. Read-only; STABLE key.
|
| 105 |
"policy_type",
|
| 106 |
"deductible_amount",
|
| 107 |
"co_payment_pct",
|
| 108 |
"sum_insured_options",
|
| 109 |
-
#
|
| 110 |
-
# gate needs these STABLE policy_facts keys (read-only):
|
| 111 |
# uin_code → canonical dedup identity (1 UIN = 1 product;
|
| 112 |
-
# collapses
|
| 113 |
-
#
|
| 114 |
-
#
|
| 115 |
-
#
|
| 116 |
-
# whose max entry age is 65 cannot accept them
|
| 117 |
-
# (audit P7). The chunk already carries
|
| 118 |
-
# max_entry_age from Chroma metadata, but the
|
| 119 |
-
# curated facts file is the authoritative value
|
| 120 |
-
# and overrides a missing/None chunk field.
|
| 121 |
# maternity_coverage → required-feature gate: an explicit maternity /
|
| 122 |
-
# newborn_coverage newborn need
|
| 123 |
-
# unconfirmed ones (audit P3).
|
| 124 |
-
# max_renewal_age is intentionally NOT read — it was removed from the
|
| 125 |
-
# scorecard (lifelong renewal is the IRDAI norm; scorecard.py:606).
|
| 126 |
"uin_code",
|
| 127 |
"max_entry_age",
|
| 128 |
"maternity_coverage",
|
|
@@ -214,21 +201,12 @@ _reviews_cache: dict = {}
|
|
| 214 |
|
| 215 |
|
| 216 |
def _curated_facts_all() -> dict:
|
| 217 |
-
"""The
|
| 218 |
-
|
| 219 |
-
|
| 220 |
-
|
| 221 |
-
|
| 222 |
-
|
| 223 |
-
/ overall 0 → the Bug #71 ≥70 gate dropped 100% of them → no cards.
|
| 224 |
-
Meanwhile /api/policies/all scored the SAME policies correctly
|
| 225 |
-
(HDFC Ergo Optima Restore = A/78) because it feeds the FULL curated
|
| 226 |
-
dict from main._load_curated_facts (KI-219/251 canonical precedence).
|
| 227 |
-
We now reuse that EXACT function (lazy import — main is loaded at
|
| 228 |
-
request time; dodges the import cycle) so a policy's recommendation
|
| 229 |
-
grade is identical to its marketplace grade by construction. The
|
| 230 |
-
7-key `_load_policy_facts` stays as-is — it is the correct input for
|
| 231 |
-
the eligibility/dedup gate, NOT for grading."""
|
| 232 |
if "d" not in _curated_all_cache:
|
| 233 |
from backend.main import _load_curated_facts # lazy: avoids cycle
|
| 234 |
_curated_all_cache["d"] = _load_curated_facts()
|
|
@@ -236,9 +214,9 @@ def _curated_facts_all() -> dict:
|
|
| 236 |
|
| 237 |
|
| 238 |
def _insurer_reviews(slug: str) -> Optional[dict]:
|
| 239 |
-
"""
|
| 240 |
-
|
| 241 |
-
|
| 242 |
if not slug:
|
| 243 |
return None
|
| 244 |
if slug in _reviews_cache:
|
|
@@ -255,12 +233,11 @@ def _insurer_reviews(slug: str) -> Optional[dict]:
|
|
| 255 |
|
| 256 |
|
| 257 |
def _scorecard_signal(policy_id: str, profile=None) -> dict:
|
| 258 |
-
"""
|
| 259 |
-
|
| 260 |
-
|
| 261 |
-
|
| 262 |
-
|
| 263 |
-
co-pay + SI headroom + cosine)."""
|
| 264 |
try:
|
| 265 |
cur = _curated_facts_all()
|
| 266 |
data = None
|
|
@@ -288,15 +265,8 @@ def _scorecard_signal(policy_id: str, profile=None) -> dict:
|
|
| 288 |
)
|
| 289 |
}
|
| 290 |
sc = build_scorecard(data, insurer_reviews=ir, profile=prof)
|
| 291 |
-
#
|
| 292 |
-
#
|
| 293 |
-
# getattr(sc, "overall", ...) ALWAYS returned None, so every
|
| 294 |
-
# enriched chunk reached retrieval_filters with _overall_score=None
|
| 295 |
-
# and _fit_score fell back to the coarse letter-grade table where
|
| 296 |
-
# every "B" == 70.0 exactly. Within-grade order then collapsed to
|
| 297 |
-
# raw cosine — the live audit's grade/rank inversion (P1 C/65 above
|
| 298 |
-
# B/75, P2 A/77 ranked LAST). Read the real numeric field; keep a
|
| 299 |
-
# belt-and-braces `overall` alias for forward-compat.
|
| 300 |
_overall = getattr(sc, "overall_score", None)
|
| 301 |
if _overall is None:
|
| 302 |
_overall = getattr(sc, "overall", None)
|
|
@@ -647,18 +617,12 @@ _qseed_cache: dict = {} # (profile_sig) -> [seed chunk dicts]
|
|
| 647 |
|
| 648 |
|
| 649 |
def _quality_seed_candidates(profile, limit: int = 25) -> list[dict]:
|
| 650 |
-
"""
|
| 651 |
-
|
| 652 |
-
|
| 653 |
-
|
| 654 |
-
|
| 655 |
-
|
| 656 |
-
filter_pipeline still does precise eligibility + profile-fit ranking on
|
| 657 |
-
the union (proven: it eligibility-passes + ranks a present A-policy #1),
|
| 658 |
-
so this ONLY guarantees CANDIDACY — it never bypasses eligibility or
|
| 659 |
-
fabricates a recommendation. Pure logic layer: no re-ingest, no vector
|
| 660 |
-
DB change. Cached per profile-signature (the catalogue + a fixed profile
|
| 661 |
-
yield a fixed ranking)."""
|
| 662 |
out: list[dict] = []
|
| 663 |
try:
|
| 664 |
prof_sig = repr(sorted(
|
|
@@ -882,18 +846,11 @@ async def retrieve_policies(
|
|
| 882 |
try:
|
| 883 |
from rag.retrieve import retrieve as _retrieve
|
| 884 |
|
| 885 |
-
#
|
| 886 |
-
#
|
| 887 |
-
#
|
| 888 |
-
#
|
| 889 |
-
#
|
| 890 |
-
# downstream quality-aware filter_pipeline (_fit_score, now fed REAL
|
| 891 |
-
# scorecard grades after KI-FULLFACTS) can't surface them — it can
|
| 892 |
-
# only re-order what was retrieved. Fix: retrieve a WIDE pool, let
|
| 893 |
-
# eligibility + scorecard-aware ranking choose, then return the
|
| 894 |
-
# top-k best-fit survivors (truncation happens after filter_pipeline
|
| 895 |
-
# below). For an explicit known-policy follow-up (policy_filter_ids)
|
| 896 |
-
# the caller already knows the policy — keep the narrow top_k.
|
| 897 |
_recall_pool = (
|
| 898 |
(int(top_k) if top_k else 8)
|
| 899 |
if policy_filter_ids
|
|
@@ -953,13 +910,12 @@ async def retrieve_policies(
|
|
| 953 |
guard_signal = None
|
| 954 |
filtered = raw
|
| 955 |
if not policy_filter_ids:
|
| 956 |
-
#
|
| 957 |
-
#
|
| 958 |
-
#
|
| 959 |
-
#
|
| 960 |
-
#
|
| 961 |
-
#
|
| 962 |
-
# carries real retrieved text); seeds only ADD missing candidates.
|
| 963 |
if profile is not None:
|
| 964 |
_seen_pids = {
|
| 965 |
(c.get("policy_id") or "").strip() for c in raw
|
|
@@ -978,28 +934,23 @@ async def retrieve_policies(
|
|
| 978 |
_log.warning("retrieval_filters.filter_pipeline failed: %s", e)
|
| 979 |
filtered = raw
|
| 980 |
|
| 981 |
-
#
|
| 982 |
-
#
|
| 983 |
-
#
|
| 984 |
-
#
|
| 985 |
-
#
|
| 986 |
-
#
|
| 987 |
-
#
|
| 988 |
-
# Health A/87) are what the LLM and the cards both see. Set on the
|
| 989 |
-
# session cache too so prose ⇄ cited cards are gated on the SAME set.
|
| 990 |
filtered = filtered[: max((int(top_k) if top_k else 8), 12)]
|
| 991 |
|
| 992 |
-
#
|
| 993 |
-
#
|
| 994 |
-
#
|
| 995 |
-
#
|
| 996 |
-
#
|
| 997 |
-
#
|
| 998 |
-
#
|
| 999 |
-
#
|
| 1000 |
-
# with a sub-floor policy (also the requirement). qa / follow-up
|
| 1001 |
-
# intents are untouched: they legitimately cite supporting source
|
| 1002 |
-
# chunks regardless of recommendation grade.
|
| 1003 |
if (intent or "").lower() == "recommendation" and filtered:
|
| 1004 |
try:
|
| 1005 |
from backend.single_brain import _recommendation_fit
|
|
|
|
| 88 |
|
| 89 |
_FACTS_DIR = settings.DATA_DIR / "policy_facts"
|
| 90 |
_DOCTYPE_SUFFIXES = ("__wordings", "__brochure", "__cis", "__prospectus")
|
| 91 |
+
# Width of the Chroma candidate pool pulled before eligibility +
|
| 92 |
+
# scorecard-aware ranking — wide enough that a strong policy ranked only
|
| 93 |
+
# mid-cosine for a generic query still enters contention. The LLM still
|
| 94 |
+
# only sees the top-k best-fit survivors.
|
|
|
|
| 95 |
_RECALL_POOL = 40
|
| 96 |
_FACT_KEYS = (
|
| 97 |
"policy_type_indemnity_or_fixed",
|
| 98 |
+
# Raw catalog type key — some curated files carry the product type
|
| 99 |
+
# only under `policy_type` (e.g. "hospital_cash") with no
|
| 100 |
+
# `policy_type_indemnity_or_fixed`; the fixed-benefit gate needs it.
|
|
|
|
|
|
|
|
|
|
| 101 |
"policy_type",
|
| 102 |
"deductible_amount",
|
| 103 |
"co_payment_pct",
|
| 104 |
"sum_insured_options",
|
| 105 |
+
# Eligibility-gate keys (read-only):
|
|
|
|
| 106 |
# uin_code → canonical dedup identity (1 UIN = 1 product;
|
| 107 |
+
# collapses doctype-sibling/rename duplicates).
|
| 108 |
+
# max_entry_age → hard eligibility for the insured person; the
|
| 109 |
+
# curated value is authoritative and overrides
|
| 110 |
+
# a missing/None Chroma chunk field.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 111 |
# maternity_coverage → required-feature gate: an explicit maternity /
|
| 112 |
+
# newborn_coverage newborn need ranks confirmed plans first.
|
|
|
|
|
|
|
|
|
|
| 113 |
"uin_code",
|
| 114 |
"max_entry_age",
|
| 115 |
"maternity_coverage",
|
|
|
|
| 201 |
|
| 202 |
|
| 203 |
def _curated_facts_all() -> dict:
|
| 204 |
+
"""The full curated-facts layer — the same source the marketplace
|
| 205 |
+
scores from (main._load_curated_facts, KI-219/251 canonical
|
| 206 |
+
precedence; lazy import since main is loaded at request time). Using
|
| 207 |
+
it here makes a policy's recommendation grade identical to its
|
| 208 |
+
marketplace grade. The 7-key `_load_policy_facts` is the input for the
|
| 209 |
+
eligibility/dedup gate, not for grading."""
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 210 |
if "d" not in _curated_all_cache:
|
| 211 |
from backend.main import _load_curated_facts # lazy: avoids cycle
|
| 212 |
_curated_all_cache["d"] = _load_curated_facts()
|
|
|
|
| 214 |
|
| 215 |
|
| 216 |
def _insurer_reviews(slug: str) -> Optional[dict]:
|
| 217 |
+
"""Insurer reviews (40-data/reviews/<slug>.json) passed to
|
| 218 |
+
build_scorecard — drives the claim-experience sub-score; the same
|
| 219 |
+
source the marketplace uses."""
|
| 220 |
if not slug:
|
| 221 |
return None
|
| 222 |
if slug in _reviews_cache:
|
|
|
|
| 233 |
|
| 234 |
|
| 235 |
def _scorecard_signal(policy_id: str, profile=None) -> dict:
|
| 236 |
+
"""{_grade, _overall_score} from backend.scorecard using the same
|
| 237 |
+
inputs as the marketplace, so a policy's recommendation grade equals
|
| 238 |
+
its marketplace grade (asserted by tests/test_scorecard_parity.py).
|
| 239 |
+
Read-only; returns {} on any failure (ranking still works off co-pay
|
| 240 |
+
+ SI headroom + cosine)."""
|
|
|
|
| 241 |
try:
|
| 242 |
cur = _curated_facts_all()
|
| 243 |
data = None
|
|
|
|
| 265 |
)
|
| 266 |
}
|
| 267 |
sc = build_scorecard(data, insurer_reviews=ir, profile=prof)
|
| 268 |
+
# Scorecard.overall_score is the numeric field; `overall` is a
|
| 269 |
+
# back-compat alias.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 270 |
_overall = getattr(sc, "overall_score", None)
|
| 271 |
if _overall is None:
|
| 272 |
_overall = getattr(sc, "overall", None)
|
|
|
|
| 617 |
|
| 618 |
|
| 619 |
def _quality_seed_candidates(profile, limit: int = 25) -> list[dict]:
|
| 620 |
+
"""Seed the candidate pool with the catalogue's top policies by the
|
| 621 |
+
profile-tuned scorecard overall, so a strong policy that isn't
|
| 622 |
+
cosine-similar to a generic query still enters contention. This only
|
| 623 |
+
ADDS candidates — filter_pipeline still applies precise eligibility +
|
| 624 |
+
profile-fit ranking on the union; it never bypasses eligibility or
|
| 625 |
+
fabricates a recommendation. Cached per profile-signature."""
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 626 |
out: list[dict] = []
|
| 627 |
try:
|
| 628 |
prof_sig = repr(sorted(
|
|
|
|
| 846 |
try:
|
| 847 |
from rag.retrieve import retrieve as _retrieve
|
| 848 |
|
| 849 |
+
# Decouple the Chroma recall pool from the caller's top_k: pull a
|
| 850 |
+
# wide pool so eligibility + scorecard-aware ranking choose from a
|
| 851 |
+
# broad set, then return the top-k best-fit survivors (truncated
|
| 852 |
+
# after filter_pipeline below). For an explicit known-policy
|
| 853 |
+
# follow-up (policy_filter_ids) keep the narrow top_k.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 854 |
_recall_pool = (
|
| 855 |
(int(top_k) if top_k else 8)
|
| 856 |
if policy_filter_ids
|
|
|
|
| 910 |
guard_signal = None
|
| 911 |
filtered = raw
|
| 912 |
if not policy_filter_ids:
|
| 913 |
+
# Union the cosine pool with the catalogue's top profile-graded
|
| 914 |
+
# policies so a strong policy that isn't cosine-similar to a
|
| 915 |
+
# generic needs-query still enters contention. filter_pipeline
|
| 916 |
+
# then applies eligibility + profile-fit ranking on the union. A
|
| 917 |
+
# cosine chunk wins on a dup (it carries real retrieved text);
|
| 918 |
+
# seeds only add missing candidates.
|
|
|
|
| 919 |
if profile is not None:
|
| 920 |
_seen_pids = {
|
| 921 |
(c.get("policy_id") or "").strip() for c in raw
|
|
|
|
| 934 |
_log.warning("retrieval_filters.filter_pipeline failed: %s", e)
|
| 935 |
filtered = raw
|
| 936 |
|
| 937 |
+
# filter_pipeline has eligibility-filtered + ranked the wide pool
|
| 938 |
+
# best-fit-first (scorecard-aware, deduped to ~1 chunk/policy).
|
| 939 |
+
# Return only the top-N best-fit survivors — enough for the LLM to
|
| 940 |
+
# pick 2-4 recommendations and for the citation builder to select
|
| 941 |
+
# from, without the wide tail as context. This is the only
|
| 942 |
+
# truncation, done after quality ranking, so the LLM and the cards
|
| 943 |
+
# see the same best-fit set.
|
|
|
|
|
|
|
| 944 |
filtered = filtered[: max((int(top_k) if top_k else 8), 12)]
|
| 945 |
|
| 946 |
+
# For a recommendation turn, apply the same fitness floor the
|
| 947 |
+
# citation builder uses (_recommendation_fit) here, so the LLM only
|
| 948 |
+
# sees cardable policies — prose and cards stay 1:1 (it cannot name
|
| 949 |
+
# a policy it won't card). If nothing clears the floor the set is
|
| 950 |
+
# empty and the LLM says there is no strong match rather than
|
| 951 |
+
# padding with a sub-floor policy. qa / follow-up intents are
|
| 952 |
+
# untouched (they cite supporting source chunks regardless of
|
| 953 |
+
# recommendation grade).
|
|
|
|
|
|
|
|
|
|
| 954 |
if (intent or "").lower() == "recommendation" and filtered:
|
| 955 |
try:
|
| 956 |
from backend.single_brain import _recommendation_fit
|
|
@@ -19,12 +19,9 @@ class Settings:
|
|
| 19 |
# Provider keys
|
| 20 |
SARVAM_API_KEY: str = os.environ.get("SARVAM_API_KEY", "")
|
| 21 |
VOYAGE_API_KEY: str = os.environ.get("VOYAGE_API_KEY", "")
|
| 22 |
-
# NVIDIA NIM — single provider hosting the
|
| 23 |
-
#
|
| 24 |
-
#
|
| 25 |
-
# Free tier: 40 req/min, no daily cap, no card. Replaces OpenRouter +
|
| 26 |
-
# direct DeepSeek + Cerebras + Groq (four legacy providers retired
|
| 27 |
-
# 2026-05-14 in favor of single-provider consolidation — see D-019).
|
| 28 |
NVIDIA_NIM_API_KEY: str = os.environ.get("NVIDIA_NIM_API_KEY", "")
|
| 29 |
|
| 30 |
# CROSS-PROVIDER FALLBACKS — last-resort entries appended to BRAIN_CHAIN +
|
|
@@ -38,7 +35,7 @@ class Settings:
|
|
| 38 |
OPENROUTER_API_KEY: str = os.environ.get("OPENROUTER_API_KEY", "")
|
| 39 |
GROQ_API_KEY: str = os.environ.get("GROQ_API_KEY", "")
|
| 40 |
|
| 41 |
-
# Sarvam endpoints (voice STT/TTS + Indic translation only
|
| 42 |
SARVAM_BASE_URL: str = "https://api.sarvam.ai"
|
| 43 |
SARVAM_STT_PATH: str = "/speech-to-text"
|
| 44 |
SARVAM_TTS_PATH: str = "/text-to-speech"
|
|
@@ -48,15 +45,15 @@ class Settings:
|
|
| 48 |
SARVAM_STT_MODEL: str = "saarika:v2.5"
|
| 49 |
SARVAM_TTS_MODEL: str = "bulbul:v2"
|
| 50 |
SARVAM_TTS_SPEAKER: str = "anushka" # natural female advisor voice
|
| 51 |
-
SARVAM_LLM_MODEL: str = "sarvam-m" #
|
| 52 |
|
| 53 |
-
# Voyage
|
|
|
|
| 54 |
VOYAGE_MODEL: str = "voyage-3"
|
| 55 |
|
| 56 |
-
# NVIDIA NIM (single source of truth for brain + judge — tiered
|
| 57 |
-
#
|
| 58 |
-
#
|
| 59 |
-
# models on NIM free tier as of 2026-05-14.
|
| 60 |
NVIDIA_NIM_BASE_URL: str = "https://integrate.api.nvidia.com/v1"
|
| 61 |
NVIDIA_NIM_BRAIN_MODEL: str = "qwen/qwen3-next-80b-a3b-instruct"
|
| 62 |
NVIDIA_NIM_FAST_BRAIN_MODEL: str = "qwen/qwen3-next-80b-a3b-instruct"
|
|
@@ -67,12 +64,9 @@ class Settings:
|
|
| 67 |
EXTRACTED_DIR: Path = ROOT / "rag" / "extracted"
|
| 68 |
VECTORS_DIR: Path = ROOT / "rag" / "vectors"
|
| 69 |
STRUCTURED_DB: Path = ROOT / "rag" / "policies.duckdb"
|
| 70 |
-
# Single source of truth for the curated-facts directory.
|
| 71 |
-
#
|
| 72 |
-
# (
|
| 73 |
-
# Every one of those expressions resolved to <repo_root>/40-data, so this
|
| 74 |
-
# constant is a behaviour-preserving consolidation, not a path change.
|
| 75 |
-
# The directory name is intentionally kept (parallel to 70-docs/80-audit).
|
| 76 |
DATA_DIR: Path = ROOT / "40-data"
|
| 77 |
|
| 78 |
# Tunables (overrideable via env vars so the hyperparameter sweep can iterate)
|
|
@@ -80,7 +74,7 @@ class Settings:
|
|
| 80 |
CHUNK_OVERLAP_TOKENS: int = int(os.environ.get("CHUNK_OVERLAP_TOKENS", "120"))
|
| 81 |
RAG_TOP_K: int = int(os.environ.get("RAG_TOP_K", "5"))
|
| 82 |
|
| 83 |
-
# Quarantine TTL
|
| 84 |
# `user_uploads_quarantine` Chroma collection. They are NOT durable
|
| 85 |
# corpus; a session's upload is auto-purged after this many seconds of
|
| 86 |
# no further uploads from that session, so the quarantine index can't
|
|
|
|
| 19 |
# Provider keys
|
| 20 |
SARVAM_API_KEY: str = os.environ.get("SARVAM_API_KEY", "")
|
| 21 |
VOYAGE_API_KEY: str = os.environ.get("VOYAGE_API_KEY", "")
|
| 22 |
+
# NVIDIA NIM — single provider hosting the reasoning stack (brain +
|
| 23 |
+
# judge; concrete model IDs are set on NVIDIA_NIM_*_MODEL below).
|
| 24 |
+
# Free tier: 40 req/min, no daily cap, no card.
|
|
|
|
|
|
|
|
|
|
| 25 |
NVIDIA_NIM_API_KEY: str = os.environ.get("NVIDIA_NIM_API_KEY", "")
|
| 26 |
|
| 27 |
# CROSS-PROVIDER FALLBACKS — last-resort entries appended to BRAIN_CHAIN +
|
|
|
|
| 35 |
OPENROUTER_API_KEY: str = os.environ.get("OPENROUTER_API_KEY", "")
|
| 36 |
GROQ_API_KEY: str = os.environ.get("GROQ_API_KEY", "")
|
| 37 |
|
| 38 |
+
# Sarvam endpoints (voice STT/TTS + Indic translation only)
|
| 39 |
SARVAM_BASE_URL: str = "https://api.sarvam.ai"
|
| 40 |
SARVAM_STT_PATH: str = "/speech-to-text"
|
| 41 |
SARVAM_TTS_PATH: str = "/text-to-speech"
|
|
|
|
| 45 |
SARVAM_STT_MODEL: str = "saarika:v2.5"
|
| 46 |
SARVAM_TTS_MODEL: str = "bulbul:v2"
|
| 47 |
SARVAM_TTS_SPEAKER: str = "anushka" # natural female advisor voice
|
| 48 |
+
SARVAM_LLM_MODEL: str = "sarvam-m" # Sarvam model for Indic translation
|
| 49 |
|
| 50 |
+
# Voyage — embeddings run on local BGE; this is kept for back-compat
|
| 51 |
+
# with existing extracted/ artifacts.
|
| 52 |
VOYAGE_MODEL: str = "voyage-3"
|
| 53 |
|
| 54 |
+
# NVIDIA NIM (single source of truth for brain + judge — tiered
|
| 55 |
+
# routing). Qwen 3-Next 80B + Mistral Large 3 are the production
|
| 56 |
+
# models on NIM free tier.
|
|
|
|
| 57 |
NVIDIA_NIM_BASE_URL: str = "https://integrate.api.nvidia.com/v1"
|
| 58 |
NVIDIA_NIM_BRAIN_MODEL: str = "qwen/qwen3-next-80b-a3b-instruct"
|
| 59 |
NVIDIA_NIM_FAST_BRAIN_MODEL: str = "qwen/qwen3-next-80b-a3b-instruct"
|
|
|
|
| 64 |
EXTRACTED_DIR: Path = ROOT / "rag" / "extracted"
|
| 65 |
VECTORS_DIR: Path = ROOT / "rag" / "vectors"
|
| 66 |
STRUCTURED_DB: Path = ROOT / "rag" / "policies.duckdb"
|
| 67 |
+
# Single source of truth for the curated-facts directory. Resolves to
|
| 68 |
+
# <repo_root>/40-data; the directory name is intentionally kept
|
| 69 |
+
# (parallel to 70-docs/80-audit).
|
|
|
|
|
|
|
|
|
|
| 70 |
DATA_DIR: Path = ROOT / "40-data"
|
| 71 |
|
| 72 |
# Tunables (overrideable via env vars so the hyperparameter sweep can iterate)
|
|
|
|
| 74 |
CHUNK_OVERLAP_TOKENS: int = int(os.environ.get("CHUNK_OVERLAP_TOKENS", "120"))
|
| 75 |
RAG_TOP_K: int = int(os.environ.get("RAG_TOP_K", "5"))
|
| 76 |
|
| 77 |
+
# Quarantine TTL — user-uploaded PDFs live in the SEPARATE
|
| 78 |
# `user_uploads_quarantine` Chroma collection. They are NOT durable
|
| 79 |
# corpus; a session's upload is auto-purged after this many seconds of
|
| 80 |
# no further uploads from that session, so the quarantine index can't
|
|
@@ -1,19 +1,12 @@
|
|
| 1 |
-
"""Live NIM model health monitor —
|
| 2 |
-
|
| 3 |
-
|
| 4 |
-
|
| 5 |
-
|
| 6 |
-
|
| 7 |
-
|
| 8 |
-
|
| 9 |
-
|
| 10 |
-
OpenRouter) were ever reached. The 10-turn live probe at commit 078ff45
|
| 11 |
-
showed 7/10 fact-find turns timing out at exactly 26.6s.
|
| 12 |
-
|
| 13 |
-
KI-080 inverts the model: the background probe loop ELECTS a primary +
|
| 14 |
-
backup per chain based on real probe latencies; chat() calls the elected
|
| 15 |
-
primary ONCE per turn, with at most ONE real-time fallback to the elected
|
| 16 |
-
backup. Worst case per turn drops from 5-6 LLM calls to 1-2.
|
| 17 |
|
| 18 |
Probes every PROBE_INTERVAL_SEC tick (300s) with a tiny ping ("Reply with
|
| 19 |
exactly: ok"), `max_tokens=1` so probe-driven token spend is negligible.
|
|
@@ -27,24 +20,23 @@ Records per model:
|
|
| 27 |
- probe_history: last PROBE_HISTORY_LEN (ok, latency_ms) tuples; powers
|
| 28 |
the success_rate signal in the election score.
|
| 29 |
|
| 30 |
-
Election
|
| 31 |
-
- Walk the chain definition (BRAIN_CHAIN — the only chain
|
| 32 |
-
|
| 33 |
CURRENT_PRIMARY = first election-eligible model.
|
| 34 |
CURRENT_BACKUP = next election-eligible model after primary.
|
| 35 |
- Chain hierarchy IS the truth — nemotron-49b is LAST in BRAIN_CHAIN so
|
| 36 |
it only serves when qwen + mistral + maverick are all unavailable.
|
| 37 |
-
- Latency / success_rate
|
| 38 |
-
sin-bin, credits above water) but
|
| 39 |
-
|
| 40 |
-
on a faster probe, violating the operator-defined hierarchy.
|
| 41 |
- DEGRADED window: when chat() calls report_failure(model), that model is
|
| 42 |
sidelined for either DEGRADED_WINDOW_SEC (transient, 30s) or
|
| 43 |
-
DEGRADE_DURATION_LONG_S (rate-limit / HTTP 429, 1h
|
| 44 |
-
|
| 45 |
-
|
| 46 |
-
|
| 47 |
-
|
| 48 |
|
| 49 |
Persistence: 40-data/llm_health.json (atomic write via temp+rename).
|
| 50 |
|
|
@@ -87,57 +79,49 @@ ROOT = Path(__file__).resolve().parent.parent
|
|
| 87 |
HEALTH_FILE = settings.DATA_DIR / "llm_health.json"
|
| 88 |
HEALTH_FILE.parent.mkdir(parents=True, exist_ok=True)
|
| 89 |
|
| 90 |
-
#
|
| 91 |
-
#
|
| 92 |
-
#
|
| 93 |
-
#
|
| 94 |
-
#
|
| 95 |
-
# every chat call. 300s cadence keeps election responsive within 5 min
|
| 96 |
-
# of a pool degradation (more than fast enough at our chat volume) and
|
| 97 |
-
# drops probe-driven Groq spend to ~3K/day baseline (well inside quota).
|
| 98 |
PROBE_INTERVAL_SEC = 300
|
| 99 |
-
#
|
| 100 |
-
#
|
| 101 |
-
#
|
| 102 |
-
#
|
| 103 |
-
#
|
| 104 |
-
|
| 105 |
-
|
| 106 |
-
#
|
| 107 |
-
#
|
| 108 |
-
# parse the body content. max_tokens=1 keeps the same response shape +
|
| 109 |
-
# ~50× less token spend per probe.
|
| 110 |
PROBE_MAX_TOKENS = 1
|
| 111 |
PROBE_TIMEOUT_SEC = 8 # per-probe HTTP timeout
|
| 112 |
DOWN_AFTER_CONSECUTIVE_FAILS = 3 # 3 fails in a row = mark down
|
| 113 |
PROBE_HISTORY_LEN = 5 # rolling window for success_rate signal
|
| 114 |
-
HEALTHY_PROBE_AGE_SEC = 600 #
|
| 115 |
-
#
|
| 116 |
-
#
|
| 117 |
-
#
|
| 118 |
-
#
|
| 119 |
-
# router treats it as untested rather than trusting
|
| 120 |
# "healthy"/"unhealthy" verdict from minutes/hours ago.
|
| 121 |
STALE_AGE_SEC = HEALTHY_PROBE_AGE_SEC # alias for clarity; same threshold.
|
| 122 |
DEGRADED_WINDOW_SEC = 30 # report_failure sidelines a model this long
|
| 123 |
# for transient failures (timeout / 5xx).
|
| 124 |
|
| 125 |
-
#
|
| 126 |
-
#
|
| 127 |
-
# consumer that used to iterate ("brain", "fast_brain", "judge") now iterates
|
| 128 |
-
# ROLES so we only have one place to add a future role.
|
| 129 |
ROLES: tuple[str, ...] = ("brain",)
|
| 130 |
-
#
|
| 131 |
# almost always the daily quota on free tiers (Groq TPD, etc.) — they do
|
| 132 |
# NOT reset in 30s. Demote the model from election for an hour so the
|
| 133 |
# elector falls through to a non-rate-limited provider instead of
|
| 134 |
# bouncing back to the dead candidate on every chat turn.
|
| 135 |
DEGRADE_DURATION_LONG_S = 3600.0
|
| 136 |
|
| 137 |
-
#
|
| 138 |
-
#
|
| 139 |
-
#
|
| 140 |
-
# quota-exhausted candidates BEFORE the user gets stuck behind a 429.
|
| 141 |
#
|
| 142 |
# Three signal sources:
|
| 143 |
# 1) GROQ — response headers (x-ratelimit-remaining-tokens-day etc.) on
|
|
@@ -203,8 +187,8 @@ def _headers_for(model_id: str, api_key: str) -> dict[str, str]:
|
|
| 203 |
return h
|
| 204 |
|
| 205 |
|
| 206 |
-
#
|
| 207 |
-
#
|
| 208 |
# Health columns. Centralised here so backend and any other consumer agree
|
| 209 |
# on the same vocabulary ("network issue" / "rate limit (429)" / etc.).
|
| 210 |
# Use lookbehind/lookahead on non-digit so we still match e.g. "Status429",
|
|
@@ -309,8 +293,8 @@ def provider_of(model_id: str) -> str:
|
|
| 309 |
@dataclass
|
| 310 |
class ModelHealth:
|
| 311 |
model: str
|
| 312 |
-
#
|
| 313 |
-
#
|
| 314 |
# as untested instead of trusting a last-known healthy/unhealthy verdict
|
| 315 |
# from minutes/hours ago. Persisted records may still carry the older
|
| 316 |
# status — the elector calls effective_status() at decision time.
|
|
@@ -318,8 +302,8 @@ class ModelHealth:
|
|
| 318 |
last_success_at: Optional[str] = None
|
| 319 |
last_failure_at: Optional[str] = None
|
| 320 |
last_error: Optional[str] = None
|
| 321 |
-
#
|
| 322 |
-
#
|
| 323 |
# "http_429", "net: TimeoutException: ..."); `last_status_code` is the
|
| 324 |
# parsed integer (or None when the failure wasn't HTTP-shaped). Surfaced
|
| 325 |
# in admin Health columns so the operator can see "Off — rate limit (429)"
|
|
@@ -335,7 +319,7 @@ class ModelHealth:
|
|
| 335 |
# KI-080 — set by report_failure(); model is excluded from election
|
| 336 |
# while monotonic time < degraded_until_monotonic.
|
| 337 |
degraded_until_monotonic: float = 0.0
|
| 338 |
-
#
|
| 339 |
# update_credits_from_groq / update_credits_from_openrouter (response
|
| 340 |
# headers + account endpoint) and by the NIM local rate-meter. The
|
| 341 |
# elector gates on `credits_remaining is None OR > credits_low_water`
|
|
@@ -364,8 +348,7 @@ def _iso_age_seconds(iso_ts: Optional[str]) -> Optional[float]:
|
|
| 364 |
|
| 365 |
|
| 366 |
def _all_known_models() -> list[str]:
|
| 367 |
-
"""Pull every model name from BRAIN_CHAIN (the only
|
| 368 |
-
after the three-chain collapse on 2026-05-15)."""
|
| 369 |
from backend.providers.nvidia_nim_llm import BRAIN_CHAIN
|
| 370 |
seen: list[str] = []
|
| 371 |
for m in BRAIN_CHAIN:
|
|
@@ -378,9 +361,8 @@ def _chain_for(chain_name: str) -> list[str]:
|
|
| 378 |
"""Resolve a chain name → live chain list. Reads off the module so
|
| 379 |
runtime admin reorderings are respected by the elector.
|
| 380 |
|
| 381 |
-
|
| 382 |
-
|
| 383 |
-
that still pass "fast_brain"/"judge" (CLEAN1 nim_fallback callers etc.).
|
| 384 |
"""
|
| 385 |
from backend.providers import nvidia_nim_llm as nim
|
| 386 |
return list(getattr(nim, "BRAIN_CHAIN", []))
|
|
@@ -408,9 +390,9 @@ def _load_into_memory() -> None:
|
|
| 408 |
try:
|
| 409 |
raw = json.loads(HEALTH_FILE.read_text())
|
| 410 |
for k, v in raw.get("models", {}).items():
|
| 411 |
-
# Tolerate older
|
| 412 |
-
# probe_history / degraded_until_monotonic
|
| 413 |
-
#
|
| 414 |
v.setdefault("probe_history", [])
|
| 415 |
v.setdefault("degraded_until_monotonic", 0.0)
|
| 416 |
v.setdefault("last_status_code", None)
|
|
@@ -437,7 +419,7 @@ def save(state: Optional[dict[str, ModelHealth]] = None) -> None:
|
|
| 437 |
"""Atomic write — temp file then rename so concurrent readers never see partial.
|
| 438 |
|
| 439 |
If `state` is None, persists the in-memory _STATE. The optional arg is
|
| 440 |
-
kept for backward compatibility with
|
| 441 |
if state is None:
|
| 442 |
_load_into_memory()
|
| 443 |
with _STATE_LOCK:
|
|
@@ -452,7 +434,7 @@ def save(state: Optional[dict[str, ModelHealth]] = None) -> None:
|
|
| 452 |
|
| 453 |
|
| 454 |
# ---------------------------------------------------------------------------
|
| 455 |
-
#
|
| 456 |
# ---------------------------------------------------------------------------
|
| 457 |
|
| 458 |
def _success_rate(h: ModelHealth) -> float:
|
|
@@ -478,8 +460,8 @@ def _score(h: ModelHealth) -> float:
|
|
| 478 |
|
| 479 |
|
| 480 |
def effective_status(h: ModelHealth) -> str:
|
| 481 |
-
"""
|
| 482 |
-
|
| 483 |
|
| 484 |
A stored `status` of "healthy" can mean "the last probe N hours ago
|
| 485 |
said this was healthy" — which the router must NOT trust. When
|
|
@@ -501,11 +483,10 @@ def _is_election_eligible(h: ModelHealth, now_mono: float) -> bool:
|
|
| 501 |
- status is healthy (or degraded with a recent success)
|
| 502 |
- last probe was within HEALTHY_PROBE_AGE_SEC
|
| 503 |
- it is NOT currently in the degraded-window sin-bin
|
| 504 |
-
-
|
| 505 |
-
|
| 506 |
-
-
|
| 507 |
-
|
| 508 |
-
STALE_AGE_SEC).
|
| 509 |
"""
|
| 510 |
if h.degraded_until_monotonic > now_mono:
|
| 511 |
return False
|
|
@@ -528,7 +509,7 @@ def _is_election_eligible(h: ModelHealth, now_mono: float) -> bool:
|
|
| 528 |
|
| 529 |
|
| 530 |
def _has_credits(h: ModelHealth, now_mono: float) -> bool:
|
| 531 |
-
"""
|
| 532 |
|
| 533 |
Rules:
|
| 534 |
- If `credits_reset_at` has elapsed, treat the signal as stale and
|
|
@@ -565,7 +546,7 @@ def _ranked_candidates(chain_name: str) -> list[ModelHealth]:
|
|
| 565 |
return [h for _, h in eligible]
|
| 566 |
|
| 567 |
|
| 568 |
-
#
|
| 569 |
# backup per chain so we can emit a structured promotion/demotion log line
|
| 570 |
# when the elected model changes. Stored in-memory only (cheap; resets on
|
| 571 |
# process restart, which is fine — first post-restart election re-emits).
|
|
@@ -575,8 +556,8 @@ _LAST_ELECTION: dict[str, dict[str, Optional[str]]] = {}
|
|
| 575 |
|
| 576 |
def _emit_election_event(chain_name: str, role: str, from_m: Optional[str],
|
| 577 |
to_m: Optional[str], reason: str) -> None:
|
| 578 |
-
"""
|
| 579 |
-
|
| 580 |
{event, chain, role, from, to, reason, ts}
|
| 581 |
The router / admin UI / log-shipper can grep on `event=election` to
|
| 582 |
rebuild the timeline of who served what when.
|
|
@@ -617,28 +598,22 @@ def _record_election(chain_name: str, role: str, new_model: Optional[str],
|
|
| 617 |
|
| 618 |
|
| 619 |
def get_primary(chain_name: str) -> Optional[str]:
|
| 620 |
-
"""
|
| 621 |
|
| 622 |
Walk the chain definition in priority order. Return the first model
|
| 623 |
that's election-eligible. Chain hierarchy IS the truth — e.g. in
|
| 624 |
BRAIN_CHAIN, nemotron-49b is LAST so it only serves when every
|
| 625 |
-
higher-priority model is unavailable (
|
| 626 |
-
|
| 627 |
-
|
| 628 |
-
|
| 629 |
-
|
| 630 |
-
|
| 631 |
-
|
| 632 |
-
|
| 633 |
-
|
| 634 |
-
|
| 635 |
-
|
| 636 |
-
purely chain-positional.
|
| 637 |
-
|
| 638 |
-
A4 (2026-05-15) — emits a structured `event=election` log line via
|
| 639 |
-
`_record_election` whenever the elected primary changes for this
|
| 640 |
-
chain, so the operator/admin/log-shipper can rebuild the promotion/
|
| 641 |
-
demotion timeline. No log emit when the value is unchanged.
|
| 642 |
"""
|
| 643 |
chain = _chain_for(chain_name)
|
| 644 |
if not chain:
|
|
@@ -657,19 +632,18 @@ def get_primary(chain_name: str) -> Optional[str]:
|
|
| 657 |
|
| 658 |
|
| 659 |
def get_backup(chain_name: str) -> Optional[str]:
|
| 660 |
-
"""
|
| 661 |
-
|
| 662 |
|
| 663 |
Walks the chain definition in priority order, skips the elected
|
| 664 |
primary, and returns the next eligible model. Same chain-as-truth
|
| 665 |
-
philosophy as get_primary: the chains in nvidia_nim_llm.py
|
| 666 |
-
designed with family/provider diversity
|
| 667 |
-
|
| 668 |
-
|
| 669 |
-
|
| 670 |
-
|
| 671 |
-
|
| 672 |
-
`_record_election` whenever the elected backup changes.
|
| 673 |
"""
|
| 674 |
chain = _chain_for(chain_name)
|
| 675 |
if not chain:
|
|
@@ -875,7 +849,7 @@ def update_credits_from_groq(chain_name: str, model: str, headers: dict) -> None
|
|
| 875 |
Called from GroqLLM.chat() after a successful HTTP response. The
|
| 876 |
`headers` dict is the response's headers (case-insensitive via httpx).
|
| 877 |
We prefer the DAILY tokens signal (`x-ratelimit-remaining-tokens-day`)
|
| 878 |
-
because Groq's free-tier daily TPD cap is
|
| 879 |
|
| 880 |
Missing header → no-op. Malformed value → log warning + no-op.
|
| 881 |
"""
|
|
@@ -1097,8 +1071,8 @@ async def poll_openrouter_credits() -> Optional[dict]:
|
|
| 1097 |
|
| 1098 |
|
| 1099 |
# ---------------------------------------------------------------------------
|
| 1100 |
-
# Probing
|
| 1101 |
-
#
|
| 1102 |
# ---------------------------------------------------------------------------
|
| 1103 |
|
| 1104 |
async def probe_one(client: httpx.AsyncClient, model: str, api_key: str) -> tuple[bool, str, Optional[int]]:
|
|
@@ -1198,29 +1172,19 @@ async def probe_all() -> dict[str, ModelHealth]:
|
|
| 1198 |
"""One-shot probe of every known model. Updates persisted state + returns it."""
|
| 1199 |
_load_into_memory()
|
| 1200 |
models = _all_known_models()
|
| 1201 |
-
# Per-provider keys are resolved inside probe_one();
|
| 1202 |
-
#
|
| 1203 |
-
#
|
| 1204 |
-
|
| 1205 |
-
|
| 1206 |
-
#
|
| 1207 |
-
#
|
| 1208 |
-
#
|
| 1209 |
-
#
|
| 1210 |
-
# *model pools*, they all share the same per-API-key concurrency
|
| 1211 |
-
# quota (~3-5 slots free-tier). A 6-candidate parallel burst every
|
| 1212 |
-
# 300s would queue inside NIM and steal slots from in-flight user
|
| 1213 |
-
# turns — exactly the saturation pattern the global outbound
|
| 1214 |
-
# semaphore in nvidia_nim_llm.py is sized to prevent.
|
| 1215 |
-
#
|
| 1216 |
-
# With KI-088's semaphore in place, parallel gather would still
|
| 1217 |
-
# be hard-capped at 2 in-flight but introduce no benefit over a
|
| 1218 |
-
# serial loop. Serial is simpler, easier to reason about, and
|
| 1219 |
-
# naturally yields control to user-traffic between candidates.
|
| 1220 |
#
|
| 1221 |
-
# Cost: 6 NIM candidates × ~2s healthy probe = ~12s, well under
|
| 1222 |
-
#
|
| 1223 |
-
#
|
| 1224 |
async with httpx.AsyncClient() as client:
|
| 1225 |
results = []
|
| 1226 |
for m in models:
|
|
@@ -1260,8 +1224,8 @@ def status_summary() -> dict:
|
|
| 1260 |
state = load()
|
| 1261 |
summary = {"updated_at": None, "by_status": {"healthy": 0, "degraded": 0, "down": 0, "stale": 0, "unknown": 0}, "models": []}
|
| 1262 |
for m, h in state.items():
|
| 1263 |
-
#
|
| 1264 |
-
#
|
| 1265 |
# rows whose stored 'healthy' verdict is older than STALE_AGE_SEC.
|
| 1266 |
eff = effective_status(h)
|
| 1267 |
summary["by_status"][eff] = summary["by_status"].get(eff, 0) + 1
|
|
@@ -1294,9 +1258,9 @@ def status_summary() -> dict:
|
|
| 1294 |
summary["models"].sort(key=lambda x: (x["status"] != "healthy", x["model"]))
|
| 1295 |
if summary["models"]:
|
| 1296 |
summary["updated_at"] = max((m.get("tested_at") or "") for m in summary["models"])
|
| 1297 |
-
#
|
| 1298 |
-
#
|
| 1299 |
-
#
|
| 1300 |
summary["elections"] = {
|
| 1301 |
role: {"primary": get_primary(role), "backup": get_backup(role)}
|
| 1302 |
for role in ROLES
|
|
@@ -1305,14 +1269,14 @@ def status_summary() -> dict:
|
|
| 1305 |
|
| 1306 |
|
| 1307 |
async def background_probe_loop() -> None:
|
| 1308 |
-
"""Long-running task — probes every PROBE_INTERVAL_SEC (300s
|
| 1309 |
Started from main.py.
|
| 1310 |
|
| 1311 |
-
|
| 1312 |
-
|
| 1313 |
-
|
| 1314 |
-
|
| 1315 |
-
|
| 1316 |
"""
|
| 1317 |
tick = 0
|
| 1318 |
# Initial credits poll on startup so the elector has a non-None
|
|
|
|
| 1 |
+
"""Live NIM model health monitor — availability filter + sticky
|
| 2 |
+
primary/backup election.
|
| 3 |
+
|
| 4 |
+
The background probe loop ELECTS a primary + backup per chain; chat()
|
| 5 |
+
calls the elected primary ONCE per turn, with at most ONE real-time
|
| 6 |
+
fallback to the elected backup (worst case 1-2 LLM calls per turn).
|
| 7 |
+
Electing out-of-band, rather than iterating the whole chain on every
|
| 8 |
+
turn, keeps a single turn from queuing many NIM-hosted candidates under
|
| 9 |
+
per-key concurrency throttling and burning the turn budget.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 10 |
|
| 11 |
Probes every PROBE_INTERVAL_SEC tick (300s) with a tiny ping ("Reply with
|
| 12 |
exactly: ok"), `max_tokens=1` so probe-driven token spend is negligible.
|
|
|
|
| 20 |
- probe_history: last PROBE_HISTORY_LEN (ok, latency_ms) tuples; powers
|
| 21 |
the success_rate signal in the election score.
|
| 22 |
|
| 23 |
+
Election:
|
| 24 |
+
- Walk the chain definition (BRAIN_CHAIN — the only chain) in priority
|
| 25 |
+
order.
|
| 26 |
CURRENT_PRIMARY = first election-eligible model.
|
| 27 |
CURRENT_BACKUP = next election-eligible model after primary.
|
| 28 |
- Chain hierarchy IS the truth — nemotron-49b is LAST in BRAIN_CHAIN so
|
| 29 |
it only serves when qwen + mistral + maverick are all unavailable.
|
| 30 |
+
- Latency / success_rate gate ELIGIBILITY (probe-fresh, not in
|
| 31 |
+
sin-bin, credits above water) but do not drive ORDERING, so a faster
|
| 32 |
+
lower-priority model never beats a higher-priority one.
|
|
|
|
| 33 |
- DEGRADED window: when chat() calls report_failure(model), that model is
|
| 34 |
sidelined for either DEGRADED_WINDOW_SEC (transient, 30s) or
|
| 35 |
+
DEGRADE_DURATION_LONG_S (rate-limit / HTTP 429, 1h) so the same turn's
|
| 36 |
+
failure doesn't recycle to the same broken primary on the next turn.
|
| 37 |
+
The next probe tick reconsiders the model normally for the short
|
| 38 |
+
window; rate-limit demotions persist past several probe ticks so the
|
| 39 |
+
elector doesn't keep bouncing back to a quota-exhausted model.
|
| 40 |
|
| 41 |
Persistence: 40-data/llm_health.json (atomic write via temp+rename).
|
| 42 |
|
|
|
|
| 79 |
HEALTH_FILE = settings.DATA_DIR / "llm_health.json"
|
| 80 |
HEALTH_FILE.parent.mkdir(parents=True, exist_ok=True)
|
| 81 |
|
| 82 |
+
# Probe cadence 300s. With ~25 candidates per tick × a 4-token round-trip
|
| 83 |
+
# each, a tighter cadence would burn enough tokens/day on Groq to push the
|
| 84 |
+
# brain to HTTP 429 on Groq's free-tier TPD cap. 300s keeps election
|
| 85 |
+
# responsive within 5 min of a pool degradation (fast enough at our chat
|
| 86 |
+
# volume) and keeps probe-driven Groq spend well inside quota.
|
|
|
|
|
|
|
|
|
|
| 87 |
PROBE_INTERVAL_SEC = 300
|
| 88 |
+
# 300s for steady-state probes. The post-failure tighter cadence is
|
| 89 |
+
# implemented OUT-OF-BAND in `report_failure()`, which schedules a
|
| 90 |
+
# `_reprobe_one()` immediately after every chat failure (effective 0s
|
| 91 |
+
# after-failure cadence on the loop). The 300s cadence then resumes for
|
| 92 |
+
# the long-run probe stream so the steady state stays cheap.
|
| 93 |
+
PROBE_INTERVAL_SEC_FAILING = 30 # post-failure cadence ceiling
|
| 94 |
+
# The probe only needs a non-empty 200 response to mark a candidate
|
| 95 |
+
# healthy; the body content is never parsed. max_tokens=1 keeps the same
|
| 96 |
+
# response shape with minimal token spend per probe.
|
|
|
|
|
|
|
| 97 |
PROBE_MAX_TOKENS = 1
|
| 98 |
PROBE_TIMEOUT_SEC = 8 # per-probe HTTP timeout
|
| 99 |
DOWN_AFTER_CONSECUTIVE_FAILS = 3 # 3 fails in a row = mark down
|
| 100 |
PROBE_HISTORY_LEN = 5 # rolling window for success_rate signal
|
| 101 |
+
HEALTHY_PROBE_AGE_SEC = 600 # election candidates need a probe within
|
| 102 |
+
# the last 600s (tracks 300s cadence plus
|
| 103 |
+
# headroom for one missed tick).
|
| 104 |
+
# Explicit STALE window. If a candidate hasn't been probed in
|
| 105 |
+
# >STALE_AGE_SEC, its on-record status is rewritten to "stale" so the
|
| 106 |
+
# router treats it as untested rather than trusting a last-known
|
| 107 |
# "healthy"/"unhealthy" verdict from minutes/hours ago.
|
| 108 |
STALE_AGE_SEC = HEALTHY_PROBE_AGE_SEC # alias for clarity; same threshold.
|
| 109 |
DEGRADED_WINDOW_SEC = 30 # report_failure sidelines a model this long
|
| 110 |
# for transient failures (timeout / 5xx).
|
| 111 |
|
| 112 |
+
# "brain" is the sole election role; consumers iterate ROLES so a future
|
| 113 |
+
# role only needs to be added in one place.
|
|
|
|
|
|
|
| 114 |
ROLES: tuple[str, ...] = ("brain",)
|
| 115 |
+
# Rate-limit failures (HTTP 429 + provider 'RateLimit' bodies) are
|
| 116 |
# almost always the daily quota on free tiers (Groq TPD, etc.) — they do
|
| 117 |
# NOT reset in 30s. Demote the model from election for an hour so the
|
| 118 |
# elector falls through to a non-rate-limited provider instead of
|
| 119 |
# bouncing back to the dead candidate on every chat turn.
|
| 120 |
DEGRADE_DURATION_LONG_S = 3600.0
|
| 121 |
|
| 122 |
+
# Proactive credit tracking: election excludes quota-exhausted candidates
|
| 123 |
+
# BEFORE the user gets stuck behind a 429 (rather than only demoting a
|
| 124 |
+
# candidate for 1h after a 429 hits).
|
|
|
|
| 125 |
#
|
| 126 |
# Three signal sources:
|
| 127 |
# 1) GROQ — response headers (x-ratelimit-remaining-tokens-day etc.) on
|
|
|
|
| 187 |
return h
|
| 188 |
|
| 189 |
|
| 190 |
+
# Categorize a `last_error` string + optional HTTP status code into the
|
| 191 |
+
# short operator-facing reason rendered by the admin
|
| 192 |
# Health columns. Centralised here so backend and any other consumer agree
|
| 193 |
# on the same vocabulary ("network issue" / "rate limit (429)" / etc.).
|
| 194 |
# Use lookbehind/lookahead on non-digit so we still match e.g. "Status429",
|
|
|
|
| 293 |
@dataclass
|
| 294 |
class ModelHealth:
|
| 295 |
model: str
|
| 296 |
+
# 'stale' is set by `effective_status()` when a row hasn't been pinged
|
| 297 |
+
# in > STALE_AGE_SEC; the router then treats it
|
| 298 |
# as untested instead of trusting a last-known healthy/unhealthy verdict
|
| 299 |
# from minutes/hours ago. Persisted records may still carry the older
|
| 300 |
# status — the elector calls effective_status() at decision time.
|
|
|
|
| 302 |
last_success_at: Optional[str] = None
|
| 303 |
last_failure_at: Optional[str] = None
|
| 304 |
last_error: Optional[str] = None
|
| 305 |
+
# Explicit HTTP status code from the most recent failed probe / chat
|
| 306 |
+
# call. `last_error` is a free-form string ("timeout",
|
| 307 |
# "http_429", "net: TimeoutException: ..."); `last_status_code` is the
|
| 308 |
# parsed integer (or None when the failure wasn't HTTP-shaped). Surfaced
|
| 309 |
# in admin Health columns so the operator can see "Off — rate limit (429)"
|
|
|
|
| 319 |
# KI-080 — set by report_failure(); model is excluded from election
|
| 320 |
# while monotonic time < degraded_until_monotonic.
|
| 321 |
degraded_until_monotonic: float = 0.0
|
| 322 |
+
# Proactive credit tracking. Stamped by
|
| 323 |
# update_credits_from_groq / update_credits_from_openrouter (response
|
| 324 |
# headers + account endpoint) and by the NIM local rate-meter. The
|
| 325 |
# elector gates on `credits_remaining is None OR > credits_low_water`
|
|
|
|
| 348 |
|
| 349 |
|
| 350 |
def _all_known_models() -> list[str]:
|
| 351 |
+
"""Pull every model name from BRAIN_CHAIN (the only chain)."""
|
|
|
|
| 352 |
from backend.providers.nvidia_nim_llm import BRAIN_CHAIN
|
| 353 |
seen: list[str] = []
|
| 354 |
for m in BRAIN_CHAIN:
|
|
|
|
| 361 |
"""Resolve a chain name → live chain list. Reads off the module so
|
| 362 |
runtime admin reorderings are respected by the elector.
|
| 363 |
|
| 364 |
+
"brain" is the only valid role. Any other input is coerced to "brain"
|
| 365 |
+
for back-compat with callers that pass another role name.
|
|
|
|
| 366 |
"""
|
| 367 |
from backend.providers import nvidia_nim_llm as nim
|
| 368 |
return list(getattr(nim, "BRAIN_CHAIN", []))
|
|
|
|
| 390 |
try:
|
| 391 |
raw = json.loads(HEALTH_FILE.read_text())
|
| 392 |
for k, v in raw.get("models", {}).items():
|
| 393 |
+
# Tolerate older on-disk schema: records may be missing
|
| 394 |
+
# probe_history / degraded_until_monotonic or the five
|
| 395 |
+
# credits_* fields.
|
| 396 |
v.setdefault("probe_history", [])
|
| 397 |
v.setdefault("degraded_until_monotonic", 0.0)
|
| 398 |
v.setdefault("last_status_code", None)
|
|
|
|
| 419 |
"""Atomic write — temp file then rename so concurrent readers never see partial.
|
| 420 |
|
| 421 |
If `state` is None, persists the in-memory _STATE. The optional arg is
|
| 422 |
+
kept for backward compatibility with call sites in admin.py."""
|
| 423 |
if state is None:
|
| 424 |
_load_into_memory()
|
| 425 |
with _STATE_LOCK:
|
|
|
|
| 434 |
|
| 435 |
|
| 436 |
# ---------------------------------------------------------------------------
|
| 437 |
+
# Public API: primary/backup election + failure/success reporting
|
| 438 |
# ---------------------------------------------------------------------------
|
| 439 |
|
| 440 |
def _success_rate(h: ModelHealth) -> float:
|
|
|
|
| 460 |
|
| 461 |
|
| 462 |
def effective_status(h: ModelHealth) -> str:
|
| 463 |
+
"""Return the routing-relevant status, applying the STALE_AGE_SEC
|
| 464 |
+
override at read time.
|
| 465 |
|
| 466 |
A stored `status` of "healthy" can mean "the last probe N hours ago
|
| 467 |
said this was healthy" — which the router must NOT trust. When
|
|
|
|
| 483 |
- status is healthy (or degraded with a recent success)
|
| 484 |
- last probe was within HEALTHY_PROBE_AGE_SEC
|
| 485 |
- it is NOT currently in the degraded-window sin-bin
|
| 486 |
+
- it has credits remaining above its low-water mark, OR no credit
|
| 487 |
+
signal yet (cold-start = permissive).
|
| 488 |
+
- effective_status != "stale" (catches the case where the status
|
| 489 |
+
field is 'healthy' but the probe is older than STALE_AGE_SEC).
|
|
|
|
| 490 |
"""
|
| 491 |
if h.degraded_until_monotonic > now_mono:
|
| 492 |
return False
|
|
|
|
| 509 |
|
| 510 |
|
| 511 |
def _has_credits(h: ModelHealth, now_mono: float) -> bool:
|
| 512 |
+
"""Credit-gate predicate for election eligibility.
|
| 513 |
|
| 514 |
Rules:
|
| 515 |
- If `credits_reset_at` has elapsed, treat the signal as stale and
|
|
|
|
| 546 |
return [h for _, h in eligible]
|
| 547 |
|
| 548 |
|
| 549 |
+
# Election event log. Tracks the last elected primary/
|
| 550 |
# backup per chain so we can emit a structured promotion/demotion log line
|
| 551 |
# when the elected model changes. Stored in-memory only (cheap; resets on
|
| 552 |
# process restart, which is fine — first post-restart election re-emits).
|
|
|
|
| 556 |
|
| 557 |
def _emit_election_event(chain_name: str, role: str, from_m: Optional[str],
|
| 558 |
to_m: Optional[str], reason: str) -> None:
|
| 559 |
+
"""Structured log line for every primary/backup promotion or
|
| 560 |
+
demotion. Format:
|
| 561 |
{event, chain, role, from, to, reason, ts}
|
| 562 |
The router / admin UI / log-shipper can grep on `event=election` to
|
| 563 |
rebuild the timeline of who served what when.
|
|
|
|
| 598 |
|
| 599 |
|
| 600 |
def get_primary(chain_name: str) -> Optional[str]:
|
| 601 |
+
"""Elect by CHAIN ORDER, not latency score.
|
| 602 |
|
| 603 |
Walk the chain definition in priority order. Return the first model
|
| 604 |
that's election-eligible. Chain hierarchy IS the truth — e.g. in
|
| 605 |
BRAIN_CHAIN, nemotron-49b is LAST so it only serves when every
|
| 606 |
+
higher-priority model is unavailable (last-resort rule).
|
| 607 |
+
|
| 608 |
+
Latency is not used for primary/backup selection — eligibility
|
| 609 |
+
filtering uses the rolling probe state, but ordering is purely
|
| 610 |
+
chain-positional, so the hierarchy is always honoured when models
|
| 611 |
+
are live.
|
| 612 |
+
|
| 613 |
+
Emits a structured `event=election` log line via `_record_election`
|
| 614 |
+
whenever the elected primary changes for this chain, so the
|
| 615 |
+
operator/admin/log-shipper can rebuild the promotion/demotion
|
| 616 |
+
timeline. No log emit when the value is unchanged.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 617 |
"""
|
| 618 |
chain = _chain_for(chain_name)
|
| 619 |
if not chain:
|
|
|
|
| 632 |
|
| 633 |
|
| 634 |
def get_backup(chain_name: str) -> Optional[str]:
|
| 635 |
+
"""Backup is the SECOND eligible model in chain order, skipping the
|
| 636 |
+
elected primary.
|
| 637 |
|
| 638 |
Walks the chain definition in priority order, skips the elected
|
| 639 |
primary, and returns the next eligible model. Same chain-as-truth
|
| 640 |
+
philosophy as get_primary: the chains in nvidia_nim_llm.py are
|
| 641 |
+
designed with family/provider diversity (Qwen → Mistral → Meta →
|
| 642 |
+
NVIDIA), so walking past the primary in chain order naturally keeps
|
| 643 |
+
the backup on a different family from the primary.
|
| 644 |
+
|
| 645 |
+
Emits a structured `event=election` log line via `_record_election`
|
| 646 |
+
whenever the elected backup changes.
|
|
|
|
| 647 |
"""
|
| 648 |
chain = _chain_for(chain_name)
|
| 649 |
if not chain:
|
|
|
|
| 849 |
Called from GroqLLM.chat() after a successful HTTP response. The
|
| 850 |
`headers` dict is the response's headers (case-insensitive via httpx).
|
| 851 |
We prefer the DAILY tokens signal (`x-ratelimit-remaining-tokens-day`)
|
| 852 |
+
because Groq's free-tier daily TPD cap is the binding limit.
|
| 853 |
|
| 854 |
Missing header → no-op. Malformed value → log warning + no-op.
|
| 855 |
"""
|
|
|
|
| 1071 |
|
| 1072 |
|
| 1073 |
# ---------------------------------------------------------------------------
|
| 1074 |
+
# Probing — records probe_history + skips degraded-window models on the
|
| 1075 |
+
# regular tick.
|
| 1076 |
# ---------------------------------------------------------------------------
|
| 1077 |
|
| 1078 |
async def probe_one(client: httpx.AsyncClient, model: str, api_key: str) -> tuple[bool, str, Optional[int]]:
|
|
|
|
| 1172 |
"""One-shot probe of every known model. Updates persisted state + returns it."""
|
| 1173 |
_load_into_memory()
|
| 1174 |
models = _all_known_models()
|
| 1175 |
+
# Per-provider keys are resolved inside probe_one(); the loop is not
|
| 1176 |
+
# gated on the NIM key so cross-provider candidates keep getting
|
| 1177 |
+
# probed even when NVIDIA_NIM_API_KEY is missing.
|
| 1178 |
+
|
| 1179 |
+
# Probe candidates serially, not in parallel. Candidates share the
|
| 1180 |
+
# same per-API-key concurrency quota (~3-5 slots free-tier), so a
|
| 1181 |
+
# parallel burst every 300s would queue inside NIM and steal slots
|
| 1182 |
+
# from in-flight user turns. A serial loop is simple to reason about
|
| 1183 |
+
# and naturally yields control to user traffic between candidates.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1184 |
#
|
| 1185 |
+
# Cost: ~6 NIM candidates × ~2s healthy probe = ~12s, well under the
|
| 1186 |
+
# 300s probe cadence. No sleep between candidates — the outbound
|
| 1187 |
+
# semaphore handles pacing.
|
| 1188 |
async with httpx.AsyncClient() as client:
|
| 1189 |
results = []
|
| 1190 |
for m in models:
|
|
|
|
| 1224 |
state = load()
|
| 1225 |
summary = {"updated_at": None, "by_status": {"healthy": 0, "degraded": 0, "down": 0, "stale": 0, "unknown": 0}, "models": []}
|
| 1226 |
for m, h in state.items():
|
| 1227 |
+
# `effective_status` applies the STALE_AGE_SEC override at read
|
| 1228 |
+
# time so the admin UI / router see "stale" for
|
| 1229 |
# rows whose stored 'healthy' verdict is older than STALE_AGE_SEC.
|
| 1230 |
eff = effective_status(h)
|
| 1231 |
summary["by_status"][eff] = summary["by_status"].get(eff, 0) + 1
|
|
|
|
| 1258 |
summary["models"].sort(key=lambda x: (x["status"] != "healthy", x["model"]))
|
| 1259 |
if summary["models"]:
|
| 1260 |
summary["updated_at"] = max((m.get("tested_at") or "") for m in summary["models"])
|
| 1261 |
+
# Surface currently-elected primary/backup per chain for the admin UI.
|
| 1262 |
+
# Cheap (a couple of dict lookups + sort over <=10 entries). "brain"
|
| 1263 |
+
# is the only role.
|
| 1264 |
summary["elections"] = {
|
| 1265 |
role: {"primary": get_primary(role), "backup": get_backup(role)}
|
| 1266 |
for role in ROLES
|
|
|
|
| 1269 |
|
| 1270 |
|
| 1271 |
async def background_probe_loop() -> None:
|
| 1272 |
+
"""Long-running task — probes every PROBE_INTERVAL_SEC (300s).
|
| 1273 |
Started from main.py.
|
| 1274 |
|
| 1275 |
+
Also polls OpenRouter's account-level credits endpoint every
|
| 1276 |
+
OPENROUTER_CREDITS_POLL_EVERY_N_TICKS ticks (10 min by default at the
|
| 1277 |
+
300s probe cadence). Groq + NIM signals come from the chat hot path
|
| 1278 |
+
(response headers + local rate-meter respectively) so only OpenRouter
|
| 1279 |
+
needs an out-of-band poll.
|
| 1280 |
"""
|
| 1281 |
tick = 0
|
| 1282 |
# Initial credits poll on startup so the elector has a non-None
|
|
@@ -32,21 +32,22 @@ from backend import sum_insured as _si # SI rationalisation (D1/D3) — source-
|
|
| 32 |
from backend.providers.sarvam_stt import SarvamSTT
|
| 33 |
from backend.providers.sarvam_tts import SarvamTTS
|
| 34 |
|
| 35 |
-
#
|
| 36 |
-
#
|
| 37 |
-
#
|
| 38 |
-
#
|
|
|
|
| 39 |
import os as _os # local alias to avoid stomping any later `import os`
|
| 40 |
|
| 41 |
USE_SINGLE_BRAIN = _os.environ.get("USE_SINGLE_BRAIN", "false").lower() in (
|
| 42 |
"1", "true", "yes", "on",
|
| 43 |
)
|
| 44 |
|
| 45 |
-
#
|
| 46 |
-
#
|
| 47 |
-
#
|
| 48 |
-
#
|
| 49 |
-
#
|
| 50 |
# Word-boundary anchored; case-insensitive at match-time.
|
| 51 |
_CLOSER_KEYWORD_RE = re.compile(
|
| 52 |
r"\b(go with|i'?ll take|i will take|let'?s do|let me get|sign me up|"
|
|
@@ -144,15 +145,10 @@ class ChatResponse(BaseModel):
|
|
| 144 |
session_id: str
|
| 145 |
audio_base64: Optional[str] = None
|
| 146 |
audio_mime: Optional[str] = None # X8 — "audio/wav" | "audio/mp4" | "audio/webm"
|
| 147 |
-
#
|
| 148 |
-
#
|
| 149 |
-
#
|
| 150 |
-
#
|
| 151 |
-
# ("no voice in reply. wtf?"). These two fields make the failure LOUD:
|
| 152 |
-
# the frontend renders a small inline "voice unavailable" notice under
|
| 153 |
-
# the bot bubble (text reply is unaffected). tts_error_code is a closed
|
| 154 |
-
# enum mirroring the STT path's contract so the client never parses raw
|
| 155 |
-
# httpx text:
|
| 156 |
# rate_limit — Sarvam 429 / insufficient_quota / no credits
|
| 157 |
# service_unavailable — Sarvam 5xx / 503
|
| 158 |
# auth — Sarvam 401/403 / missing SARVAM_API_KEY
|
|
@@ -171,13 +167,12 @@ class ChatResponse(BaseModel):
|
|
| 171 |
"flash an acknowledgment + refresh the completeness panel."
|
| 172 |
),
|
| 173 |
)
|
| 174 |
-
#
|
| 175 |
-
# second roundtrip to /api/profile/completeness after every chat turn to
|
| 176 |
-
# learn whether the 7 required slots are now captured. Surface it in the
|
| 177 |
# primary chat response so the UI can flip to 100% in the same render
|
| 178 |
-
# cycle
|
| 179 |
-
#
|
| 180 |
-
#
|
|
|
|
| 181 |
profile_complete: bool = Field(
|
| 182 |
False,
|
| 183 |
description=(
|
|
@@ -343,13 +338,12 @@ async def _quarantine_purge_loop() -> None:
|
|
| 343 |
)
|
| 344 |
|
| 345 |
|
| 346 |
-
#
|
| 347 |
-
#
|
| 348 |
-
#
|
| 349 |
-
#
|
| 350 |
-
#
|
| 351 |
-
#
|
| 352 |
-
# on turn 1" from "stored profile recalled on turn 1". Both end with a
|
| 353 |
# non-empty Profile on the session; only the latter should flip
|
| 354 |
# returning_user_recalled. Heuristic: if EVERY currently-filled slot was
|
| 355 |
# written THIS turn (i.e. lives in `profile_updates`), the user just typed
|
|
@@ -461,11 +455,10 @@ async def _validation_exception_handler(request: Request, exc: RequestValidation
|
|
| 461 |
|
| 462 |
|
| 463 |
# ---------- Admin panel + LLM health background loop ----------
|
| 464 |
-
# Mount the password-gated admin endpoints
|
| 465 |
-
#
|
| 466 |
-
#
|
| 467 |
-
#
|
| 468 |
-
# beyond a strong password.
|
| 469 |
from backend import admin as _admin_router_module
|
| 470 |
app.include_router(_admin_router_module.router)
|
| 471 |
|
|
@@ -797,24 +790,21 @@ async def chat(req: ChatRequest, request: Request):
|
|
| 797 |
if preferred_codec not in _allowed_codecs:
|
| 798 |
preferred_codec = "audio/wav"
|
| 799 |
t_chat0 = time.time()
|
| 800 |
-
#
|
| 801 |
-
#
|
| 802 |
-
# detector simply doesn't fire on the legacy orchestrator path.
|
| 803 |
_pre_turn_name: str = ""
|
| 804 |
_pre_turn_idx: int = 0
|
| 805 |
-
#
|
| 806 |
-
#
|
| 807 |
-
#
|
| 808 |
-
#
|
| 809 |
-
#
|
| 810 |
-
#
|
| 811 |
-
# not a connection-reset to the user. 45s is generous but tighter than
|
| 812 |
-
# HF Space's gateway timeout, so the user always gets a response.
|
| 813 |
try:
|
| 814 |
if USE_SINGLE_BRAIN:
|
| 815 |
-
#
|
| 816 |
-
# Falls back to
|
| 817 |
-
#
|
| 818 |
from backend import single_brain
|
| 819 |
from backend.session_state import get_session
|
| 820 |
|
|
@@ -826,13 +816,11 @@ async def chat(req: ChatRequest, request: Request):
|
|
| 826 |
getattr(_sb_session.profile, "name", None) or ""
|
| 827 |
).strip()
|
| 828 |
_pre_turn_idx = int(getattr(_sb_session, "turn_idx", 0) or 0)
|
| 829 |
-
#
|
| 830 |
-
#
|
| 831 |
-
#
|
| 832 |
-
#
|
| 833 |
-
#
|
| 834 |
-
# in last_recommendation_ids / last_retrieved_chunks /
|
| 835 |
-
# slug_to_insurer and confuses the user. Sticky check below.
|
| 836 |
_sb_was_sticky = getattr(_sb_session, "single_brain_sticky", False)
|
| 837 |
try:
|
| 838 |
turn = await asyncio.wait_for(
|
|
@@ -851,14 +839,13 @@ async def chat(req: ChatRequest, request: Request):
|
|
| 851 |
pass
|
| 852 |
except single_brain.SingleBrainError as _sb_err:
|
| 853 |
if _sb_was_sticky:
|
| 854 |
-
#
|
| 855 |
-
#
|
| 856 |
-
#
|
| 857 |
-
#
|
| 858 |
logging.warning(
|
| 859 |
"single_brain failed on STICKY session (session=%s); "
|
| 860 |
-
"emitting graceful retry, NOT falling back
|
| 861 |
-
"orchestrator: %s",
|
| 862 |
session_id, _sb_err,
|
| 863 |
)
|
| 864 |
turn = single_brain.TurnResult(
|
|
@@ -893,9 +880,8 @@ async def chat(req: ChatRequest, request: Request):
|
|
| 893 |
timeout=20.0,
|
| 894 |
)
|
| 895 |
else:
|
| 896 |
-
#
|
| 897 |
-
#
|
| 898 |
-
# NIM fallback so the bot still serves a reply.
|
| 899 |
from backend.session_state import get_session as _get_session
|
| 900 |
turn = await asyncio.wait_for(
|
| 901 |
nim_fallback.handle_turn_fallback(
|
|
@@ -962,7 +948,7 @@ async def chat(req: ChatRequest, request: Request):
|
|
| 962 |
profile_complete=_compute_profile_complete(session_id),
|
| 963 |
)
|
| 964 |
|
| 965 |
-
#
|
| 966 |
# If the user clearly committed to a policy this turn but Gemini did
|
| 967 |
# NOT call mark_recommendation (single_brain stamps "mark_recommendation"
|
| 968 |
# into turn.brain_used when the tool fires — see single_brain.py:1052),
|
|
@@ -1172,11 +1158,11 @@ async def chat(req: ChatRequest, request: Request):
|
|
| 1172 |
|
| 1173 |
audio_b64 = None
|
| 1174 |
audio_mime: Optional[str] = None
|
| 1175 |
-
#
|
| 1176 |
-
#
|
| 1177 |
-
#
|
| 1178 |
-
#
|
| 1179 |
-
#
|
| 1180 |
tts_error_code: Optional[str] = None
|
| 1181 |
tts_user_message: Optional[str] = None
|
| 1182 |
if req.return_audio and turn.reply_text:
|
|
@@ -1932,9 +1918,9 @@ class ProfileUpdateRequest(BaseModel):
|
|
| 1932 |
parents_has_ped: Optional[bool] = None
|
| 1933 |
health_conditions: Optional[list[str]] = None
|
| 1934 |
budget_band: Optional[str] = None
|
| 1935 |
-
#
|
| 1936 |
-
# dataclass + chat-path save_profile_field
|
| 1937 |
-
#
|
| 1938 |
desired_sum_insured_inr: Optional[int] = None
|
| 1939 |
copay_pct: Optional[int] = None
|
| 1940 |
family_medical_history: Optional[list[str]] = None
|
|
@@ -1971,8 +1957,8 @@ async def session_clear(req: SessionClearRequest):
|
|
| 1971 |
The on-disk profile JSON under `40-data/profiles/` is intentionally NOT
|
| 1972 |
touched — it remains durable user data keyed by persona_id / name slug.
|
| 1973 |
The next time the user volunteers their name in conversation, the
|
| 1974 |
-
confirmation-gated recall flow (see
|
| 1975 |
-
will ask before merging the prior captures.
|
| 1976 |
|
| 1977 |
Body : {session_id: str}
|
| 1978 |
Reply: {cleared: bool, new_session_id: str}
|
|
@@ -2136,9 +2122,9 @@ async def profile_completeness_view(session_id: Optional[str] = None):
|
|
| 2136 |
missing = [k for k, v in profile_dict.items() if k not in answered or v in (None, "", [])]
|
| 2137 |
hint = None
|
| 2138 |
try:
|
| 2139 |
-
#
|
| 2140 |
-
#
|
| 2141 |
-
#
|
| 2142 |
hint = next_question(p)
|
| 2143 |
except Exception:
|
| 2144 |
pass
|
|
@@ -2844,12 +2830,12 @@ def _policy_corroborated_si(policy_id: str | None) -> "_si.SumInsuredView":
|
|
| 2844 |
return _rationalise_si(data, si)
|
| 2845 |
|
| 2846 |
|
| 2847 |
-
#
|
| 2848 |
-
#
|
| 2849 |
-
#
|
| 2850 |
-
#
|
| 2851 |
-
#
|
| 2852 |
-
#
|
| 2853 |
_KI145_DIFF_FIELDS: tuple[str, ...] = (
|
| 2854 |
"copayment_pct",
|
| 2855 |
"pre_existing_disease_waiting_months",
|
|
@@ -3229,14 +3215,14 @@ async def policies_all(session_id: Optional[str] = None):
|
|
| 3229 |
data = _merge_curated(data, curated_for_this)
|
| 3230 |
seen_policy_ids.add(policy_id_local)
|
| 3231 |
slug = data.get("insurer_slug", "")
|
| 3232 |
-
#
|
| 3233 |
-
#
|
| 3234 |
-
#
|
| 3235 |
if slug == "regulatory":
|
| 3236 |
continue
|
| 3237 |
-
#
|
| 3238 |
-
#
|
| 3239 |
-
#
|
| 3240 |
product_key = _product_key_of(policy_id_local)
|
| 3241 |
if product_key in seen_product_keys:
|
| 3242 |
continue
|
|
@@ -4254,10 +4240,10 @@ class RecallByNameRequest(BaseModel):
|
|
| 4254 |
|
| 4255 |
class RecallByNameResponse(BaseModel):
|
| 4256 |
found: bool
|
| 4257 |
-
#
|
| 4258 |
-
#
|
| 4259 |
-
#
|
| 4260 |
-
#
|
| 4261 |
requires_confirmation: bool = False
|
| 4262 |
name: Optional[str] = None # stored display name, for the prompt
|
| 4263 |
summary: Optional[dict] = None # non-PII identity hints, for the prompt
|
|
@@ -4270,16 +4256,15 @@ class RecallByNameResponse(BaseModel):
|
|
| 4270 |
async def recall_profile_by_name(req: RecallByNameRequest):
|
| 4271 |
"""Stage a stored named-profile match for `session_id` (if any).
|
| 4272 |
|
| 4273 |
-
|
| 4274 |
-
|
| 4275 |
-
|
| 4276 |
-
|
| 4277 |
-
|
| 4278 |
-
|
| 4279 |
-
|
| 4280 |
-
|
| 4281 |
-
|
| 4282 |
-
slug doesn't resolve to a stored file.
|
| 4283 |
"""
|
| 4284 |
from backend.profile_persistence import recall_by_name_payload
|
| 4285 |
|
|
|
|
| 32 |
from backend.providers.sarvam_stt import SarvamSTT
|
| 33 |
from backend.providers.sarvam_tts import SarvamTTS
|
| 34 |
|
| 35 |
+
# Single-LLM brain toggle. Off by default; flip via env var. When on, the
|
| 36 |
+
# /api/chat hot path runs single_brain.handle_turn and falls back to
|
| 37 |
+
# nim_fallback.handle_turn_fallback on any SingleBrainError (so users
|
| 38 |
+
# always get a reply). When off, /api/chat routes directly through
|
| 39 |
+
# nim_fallback.handle_turn_fallback.
|
| 40 |
import os as _os # local alias to avoid stomping any later `import os`
|
| 41 |
|
| 42 |
USE_SINGLE_BRAIN = _os.environ.get("USE_SINGLE_BRAIN", "false").lower() in (
|
| 43 |
"1", "true", "yes", "on",
|
| 44 |
)
|
| 45 |
|
| 46 |
+
# Safety net for RULE 7. If Gemini does not call mark_recommendation when
|
| 47 |
+
# the user clearly commits to a policy ("I'll go with that one", "let's do
|
| 48 |
+
# #2", "buy this"), the post-turn detector below auto-calls
|
| 49 |
+
# mark_recommendation against session.last_recommendation_ids[:1] so the
|
| 50 |
+
# closure event is recorded for analytics.
|
| 51 |
# Word-boundary anchored; case-insensitive at match-time.
|
| 52 |
_CLOSER_KEYWORD_RE = re.compile(
|
| 53 |
r"\b(go with|i'?ll take|i will take|let'?s do|let me get|sign me up|"
|
|
|
|
| 145 |
session_id: str
|
| 146 |
audio_base64: Optional[str] = None
|
| 147 |
audio_mime: Optional[str] = None # X8 — "audio/wav" | "audio/mp4" | "audio/webm"
|
| 148 |
+
# TTS voice-output failures are surfaced to the client (text reply is
|
| 149 |
+
# unaffected): the frontend renders a small inline "voice unavailable"
|
| 150 |
+
# notice under the bot bubble. tts_error_code is a closed enum mirroring
|
| 151 |
+
# the STT path's contract so the client never parses raw httpx text:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 152 |
# rate_limit — Sarvam 429 / insufficient_quota / no credits
|
| 153 |
# service_unavailable — Sarvam 5xx / 503
|
| 154 |
# auth — Sarvam 401/403 / missing SARVAM_API_KEY
|
|
|
|
| 167 |
"flash an acknowledgment + refresh the completeness panel."
|
| 168 |
),
|
| 169 |
)
|
| 170 |
+
# Whether the 7 required profile slots are captured. Surfaced in the
|
|
|
|
|
|
|
| 171 |
# primary chat response so the UI can flip to 100% in the same render
|
| 172 |
+
# cycle without a second roundtrip to /api/profile/completeness.
|
| 173 |
+
# Computed via brain_tools._REQUIRED_FOR_READY (same slot list used by
|
| 174 |
+
# retrieve_policies' profile-complete gate) so client + server never
|
| 175 |
+
# disagree.
|
| 176 |
profile_complete: bool = Field(
|
| 177 |
False,
|
| 178 |
description=(
|
|
|
|
| 338 |
)
|
| 339 |
|
| 340 |
|
| 341 |
+
# Single source of truth for "is this profile ready to recommend against".
|
| 342 |
+
# brain_tools._profile_complete uses the same _REQUIRED_FOR_READY tuple; we
|
| 343 |
+
# mirror the call here instead of duplicating the slot list so a future
|
| 344 |
+
# addition (e.g. risk_appetite) only needs to be applied in brain_tools.
|
| 345 |
+
# Distinguishes "first-time capture on turn 1" from "stored profile
|
| 346 |
+
# recalled on turn 1". Both end with a
|
|
|
|
| 347 |
# non-empty Profile on the session; only the latter should flip
|
| 348 |
# returning_user_recalled. Heuristic: if EVERY currently-filled slot was
|
| 349 |
# written THIS turn (i.e. lives in `profile_updates`), the user just typed
|
|
|
|
| 455 |
|
| 456 |
|
| 457 |
# ---------- Admin panel + LLM health background loop ----------
|
| 458 |
+
# Mount the password-gated admin endpoints. Unauthorized callers get
|
| 459 |
+
# 401 Unauthorized. Access is gated by a strong password only (no IP
|
| 460 |
+
# allowlist, which would lock the operator out when switching networks
|
| 461 |
+
# without adding real security).
|
|
|
|
| 462 |
from backend import admin as _admin_router_module
|
| 463 |
app.include_router(_admin_router_module.router)
|
| 464 |
|
|
|
|
| 790 |
if preferred_codec not in _allowed_codecs:
|
| 791 |
preferred_codec = "audio/wav"
|
| 792 |
t_chat0 = time.time()
|
| 793 |
+
# Pre-turn snapshot for the returning-user-recall detector below.
|
| 794 |
+
# Defaults match the "no session yet" case.
|
|
|
|
| 795 |
_pre_turn_name: str = ""
|
| 796 |
_pre_turn_idx: int = 0
|
| 797 |
+
# Never let an inner TimeoutError / unhandled exception bubble out of
|
| 798 |
+
# handle_turn as a 500. The whole call is wrapped in an outer 45s
|
| 799 |
+
# budget so even a pathological hang inside handle_turn surfaces as a
|
| 800 |
+
# graceful reply, not a connection-reset to the user. 45s is generous
|
| 801 |
+
# but tighter than HF Space's gateway timeout, so the user always gets
|
| 802 |
+
# a response.
|
|
|
|
|
|
|
| 803 |
try:
|
| 804 |
if USE_SINGLE_BRAIN:
|
| 805 |
+
# One Gemini call per turn with native function-calling.
|
| 806 |
+
# Falls back to nim_fallback on SingleBrainError so a missing
|
| 807 |
+
# GOOGLE_API_KEY / model outage never breaks the chat.
|
| 808 |
from backend import single_brain
|
| 809 |
from backend.session_state import get_session
|
| 810 |
|
|
|
|
| 816 |
getattr(_sb_session.profile, "name", None) or ""
|
| 817 |
).strip()
|
| 818 |
_pre_turn_idx = int(getattr(_sb_session, "turn_idx", 0) or 0)
|
| 819 |
+
# Once a session has had ANY successful single_brain turn, it
|
| 820 |
+
# must stay on single_brain for the rest of its lifetime.
|
| 821 |
+
# Switching brains mid-stream would discard everything
|
| 822 |
+
# single_brain captured in last_recommendation_ids /
|
| 823 |
+
# last_retrieved_chunks / slug_to_insurer. Sticky check below.
|
|
|
|
|
|
|
| 824 |
_sb_was_sticky = getattr(_sb_session, "single_brain_sticky", False)
|
| 825 |
try:
|
| 826 |
turn = await asyncio.wait_for(
|
|
|
|
| 839 |
pass
|
| 840 |
except single_brain.SingleBrainError as _sb_err:
|
| 841 |
if _sb_was_sticky:
|
| 842 |
+
# Session already had a clean single_brain turn. Do NOT
|
| 843 |
+
# cross-fade to the fallback brain (loses turn state +
|
| 844 |
+
# frontend sees the brain hop). Emit a graceful retry
|
| 845 |
+
# prompt instead.
|
| 846 |
logging.warning(
|
| 847 |
"single_brain failed on STICKY session (session=%s); "
|
| 848 |
+
"emitting graceful retry, NOT falling back: %s",
|
|
|
|
| 849 |
session_id, _sb_err,
|
| 850 |
)
|
| 851 |
turn = single_brain.TurnResult(
|
|
|
|
| 880 |
timeout=20.0,
|
| 881 |
)
|
| 882 |
else:
|
| 883 |
+
# When USE_SINGLE_BRAIN is off, route directly through the
|
| 884 |
+
# minimal NIM fallback so the bot still serves a reply.
|
|
|
|
| 885 |
from backend.session_state import get_session as _get_session
|
| 886 |
turn = await asyncio.wait_for(
|
| 887 |
nim_fallback.handle_turn_fallback(
|
|
|
|
| 948 |
profile_complete=_compute_profile_complete(session_id),
|
| 949 |
)
|
| 950 |
|
| 951 |
+
# Server-side closer-keyword safety net for RULE 7.
|
| 952 |
# If the user clearly committed to a policy this turn but Gemini did
|
| 953 |
# NOT call mark_recommendation (single_brain stamps "mark_recommendation"
|
| 954 |
# into turn.brain_used when the tool fires — see single_brain.py:1052),
|
|
|
|
| 1158 |
|
| 1159 |
audio_b64 = None
|
| 1160 |
audio_mime: Optional[str] = None
|
| 1161 |
+
# When TTS fails we propagate a STRUCTURED, user-facing notice instead
|
| 1162 |
+
# of silently dropping the audio. The text reply is still returned in
|
| 1163 |
+
# full; the frontend renders a small inline "voice unavailable" line
|
| 1164 |
+
# under the bot bubble so the user understands why there's no voice
|
| 1165 |
+
# (e.g. Sarvam 429 / no credits).
|
| 1166 |
tts_error_code: Optional[str] = None
|
| 1167 |
tts_user_message: Optional[str] = None
|
| 1168 |
if req.return_audio and turn.reply_text:
|
|
|
|
| 1918 |
parents_has_ped: Optional[bool] = None
|
| 1919 |
health_conditions: Optional[list[str]] = None
|
| 1920 |
budget_band: Optional[str] = None
|
| 1921 |
+
# Collected by the profile-builder UI; also present on the Profile
|
| 1922 |
+
# dataclass + chat-path save_profile_field. Whitelisted here so
|
| 1923 |
+
# POST /api/profile accepts them.
|
| 1924 |
desired_sum_insured_inr: Optional[int] = None
|
| 1925 |
copay_pct: Optional[int] = None
|
| 1926 |
family_medical_history: Optional[list[str]] = None
|
|
|
|
| 1957 |
The on-disk profile JSON under `40-data/profiles/` is intentionally NOT
|
| 1958 |
touched — it remains durable user data keyed by persona_id / name slug.
|
| 1959 |
The next time the user volunteers their name in conversation, the
|
| 1960 |
+
confirmation-gated recall flow (see session_state's
|
| 1961 |
+
`pending_profile_recall`) will ask before merging the prior captures.
|
| 1962 |
|
| 1963 |
Body : {session_id: str}
|
| 1964 |
Reply: {cleared: bool, new_session_id: str}
|
|
|
|
| 2122 |
missing = [k for k, v in profile_dict.items() if k not in answered or v in (None, "", [])]
|
| 2123 |
hint = None
|
| 2124 |
try:
|
| 2125 |
+
# next_question returns the field name (str) of the next missing
|
| 2126 |
+
# slot. The frontend uses it as a slot-hint; the actual phrasing
|
| 2127 |
+
# is generated by the single-brain LLM.
|
| 2128 |
hint = next_question(p)
|
| 2129 |
except Exception:
|
| 2130 |
pass
|
|
|
|
| 2830 |
return _rationalise_si(data, si)
|
| 2831 |
|
| 2832 |
|
| 2833 |
+
# Decision-critical fields that distinguish a RENAME (curated entry folds
|
| 2834 |
+
# onto extracted parent) from a VARIANT (same UIN but materially different
|
| 2835 |
+
# product — must stay as its own card). Same UIN ≠ same product:
|
| 2836 |
+
# regulators file one "wordings" PDF that covers multiple marketed variants
|
| 2837 |
+
# (e.g. ProHealth Prime vs ProHealth Protect both filed under
|
| 2838 |
+
# MCIHLIP24011V072324; copay/PED/maternity/NCB differ).
|
| 2839 |
_KI145_DIFF_FIELDS: tuple[str, ...] = (
|
| 2840 |
"copayment_pct",
|
| 2841 |
"pre_existing_disease_waiting_months",
|
|
|
|
| 3215 |
data = _merge_curated(data, curated_for_this)
|
| 3216 |
seen_policy_ids.add(policy_id_local)
|
| 3217 |
slug = data.get("insurer_slug", "")
|
| 3218 |
+
# Regulatory is not an insurer; drop entirely from the marketplace.
|
| 3219 |
+
# IRDAI/NHA docs are still retrieved and cited in chat answers, they
|
| 3220 |
+
# just don't appear as marketplace cards.
|
| 3221 |
if slug == "regulatory":
|
| 3222 |
continue
|
| 3223 |
+
# Dedup by product (insurer__product), so the wordings PDF wins and
|
| 3224 |
+
# the brochure/cis variants don't generate duplicate cards. Pass-1
|
| 3225 |
+
# sort order guarantees wordings comes first.
|
| 3226 |
product_key = _product_key_of(policy_id_local)
|
| 3227 |
if product_key in seen_product_keys:
|
| 3228 |
continue
|
|
|
|
| 4240 |
|
| 4241 |
class RecallByNameResponse(BaseModel):
|
| 4242 |
found: bool
|
| 4243 |
+
# The endpoint does not hydrate the session off a bare name (weak/shared
|
| 4244 |
+
# key → stranger-PII leak). When a stored profile matches, the response
|
| 4245 |
+
# asks the UI to confirm identity FIRST. `profile`/`predicted_band` are
|
| 4246 |
+
# never returned pre-confirmation.
|
| 4247 |
requires_confirmation: bool = False
|
| 4248 |
name: Optional[str] = None # stored display name, for the prompt
|
| 4249 |
summary: Optional[dict] = None # non-PII identity hints, for the prompt
|
|
|
|
| 4256 |
async def recall_profile_by_name(req: RecallByNameRequest):
|
| 4257 |
"""Stage a stored named-profile match for `session_id` (if any).
|
| 4258 |
|
| 4259 |
+
A bare name is a weak, shared, guessable key, so this endpoint does not
|
| 4260 |
+
auto-hydrate the session — a second user on a shared browser/IP, or
|
| 4261 |
+
anyone stating a common first name, must not be silently served a
|
| 4262 |
+
stranger's stored profile. When a match exists the response carries
|
| 4263 |
+
`found=True, requires_confirmation=True, name, summary` so the UI
|
| 4264 |
+
renders an explicit "are you <name>?" prompt. The stored fields are
|
| 4265 |
+
applied to the session only after the user confirms (handled on the
|
| 4266 |
+
chat affirmation path via session_state.apply_pending_recall). Returns
|
| 4267 |
+
`found=False` when the slug doesn't resolve to a stored file.
|
|
|
|
| 4268 |
"""
|
| 4269 |
from backend.profile_persistence import recall_by_name_payload
|
| 4270 |
|
|
@@ -1,13 +1,11 @@
|
|
| 1 |
"""User profile state + slot-order hint for the fact-find.
|
| 2 |
|
| 3 |
-
|
| 4 |
-
|
| 5 |
-
lives in `backend/sales_brain.py` (the single-LLM-call brain); this module
|
| 6 |
-
keeps only:
|
| 7 |
|
| 8 |
-
- `Profile` — accumulated user state (
|
| 9 |
- `record_answer` — slot-write helper used by session_state
|
| 10 |
-
- INR / budget / income parsers — used by
|
| 11 |
- `is_field_set` — local helper for record_answer + next_question
|
| 12 |
- `next_question` — returns the field NAME (str) of the next
|
| 13 |
missing slot in canonical order. Used by
|
|
@@ -63,11 +61,9 @@ class Profile:
|
|
| 63 |
|
| 64 |
|
| 65 |
# ----------------------------------------------------------------------------
|
| 66 |
-
#
|
| 67 |
-
#
|
| 68 |
-
#
|
| 69 |
-
# capture it. Bare digits ("30000"), "30 thousand", "30 grand", "₹30,000",
|
| 70 |
-
# "1 lakh", "1.5L" must all map cleanly to a rupee amount.
|
| 71 |
# ----------------------------------------------------------------------------
|
| 72 |
|
| 73 |
def _parse_inr_amount(text: str) -> Optional[int]:
|
|
@@ -82,13 +78,10 @@ def _parse_inr_amount(text: str) -> Optional[int]:
|
|
| 82 |
- strips fluff: "maximum 30000", "I can pay 30000", "around 25000"
|
| 83 |
- tolerates per-year qualifiers: "/year", "per year", "p.a."
|
| 84 |
|
| 85 |
-
|
| 86 |
-
|
| 87 |
-
|
| 88 |
-
|
| 89 |
-
years old" and the parser wrote ₹29 into both budget_band and
|
| 90 |
-
income_band, leading the bot to claim it captured age + income + budget
|
| 91 |
-
from a single utterance.
|
| 92 |
|
| 93 |
Returns the integer rupee amount, or None if no number is recognisable
|
| 94 |
or if the only numbers in the text are clearly not currency.
|
|
@@ -209,10 +202,10 @@ def _parse_income_band(text: str) -> Optional[str]:
|
|
| 209 |
# Engine
|
| 210 |
# ----------------------------------------------------------------------------
|
| 211 |
|
| 212 |
-
#
|
| 213 |
-
#
|
| 214 |
-
#
|
| 215 |
-
#
|
| 216 |
_SLOT_ORDER: list[str] = [
|
| 217 |
"name",
|
| 218 |
"age",
|
|
@@ -238,11 +231,9 @@ def is_field_set(profile: Profile, field_name: str) -> bool:
|
|
| 238 |
def next_question(profile: Profile) -> Optional[str]:
|
| 239 |
"""Return the field NAME of the next missing slot, or None if complete.
|
| 240 |
|
| 241 |
-
|
| 242 |
-
|
| 243 |
-
|
| 244 |
-
frontend which slot to ask next; the actual question phrasing is now
|
| 245 |
-
produced by `sales_brain.py`.
|
| 246 |
|
| 247 |
A free-form session (user driving free questions) returns None so the
|
| 248 |
hint endpoint reports "nothing to ask".
|
|
@@ -256,10 +247,9 @@ def next_question(profile: Profile) -> Optional[str]:
|
|
| 256 |
return None
|
| 257 |
|
| 258 |
|
| 259 |
-
#
|
| 260 |
-
#
|
| 261 |
-
#
|
| 262 |
-
# keeps that call path working without resurrecting the GRAPH.
|
| 263 |
_QID_TO_FIELD: dict[str, str] = {
|
| 264 |
"existing_cover": "existing_cover_inr",
|
| 265 |
"location": "location_tier",
|
|
@@ -271,10 +261,9 @@ _QID_TO_FIELD: dict[str, str] = {
|
|
| 271 |
def record_answer(profile: Profile, question_id: str, raw_answer: Any) -> Profile:
|
| 272 |
"""Mutate profile in place with a raw answer for a named slot.
|
| 273 |
|
| 274 |
-
|
| 275 |
-
|
| 276 |
-
|
| 277 |
-
(preferred) or one of the legacy GRAPH question IDs from `_QID_TO_FIELD`.
|
| 278 |
"""
|
| 279 |
field_name = _QID_TO_FIELD.get(question_id, question_id)
|
| 280 |
if not hasattr(profile, field_name):
|
|
|
|
| 1 |
"""User profile state + slot-order hint for the fact-find.
|
| 2 |
|
| 3 |
+
Fact-find phrasing lives in `backend/single_brain.py` (the single-LLM-call
|
| 4 |
+
brain); this module provides:
|
|
|
|
|
|
|
| 5 |
|
| 6 |
+
- `Profile` — accumulated user state (imported widely)
|
| 7 |
- `record_answer` — slot-write helper used by session_state
|
| 8 |
+
- INR / budget / income parsers — used by the brain tool layer
|
| 9 |
- `is_field_set` — local helper for record_answer + next_question
|
| 10 |
- `next_question` — returns the field NAME (str) of the next
|
| 11 |
missing slot in canonical order. Used by
|
|
|
|
| 61 |
|
| 62 |
|
| 63 |
# ----------------------------------------------------------------------------
|
| 64 |
+
# Free-text INR amount parser for budget + income. Bare digits ("30000"),
|
| 65 |
+
# "30 thousand", "30 grand", "₹30,000", "1 lakh", "1.5L" all map cleanly
|
| 66 |
+
# to a rupee amount.
|
|
|
|
|
|
|
| 67 |
# ----------------------------------------------------------------------------
|
| 68 |
|
| 69 |
def _parse_inr_amount(text: str) -> Optional[int]:
|
|
|
|
| 78 |
- strips fluff: "maximum 30000", "I can pay 30000", "around 25000"
|
| 79 |
- tolerates per-year qualifiers: "/year", "per year", "p.a."
|
| 80 |
|
| 81 |
+
Rejects bare digits below ₹1000 (no plausible annual health insurance
|
| 82 |
+
budget/income falls there) and rejects any text whose only number is
|
| 83 |
+
in an age context ("29 years old", "age 29", "I am 29"), so an age
|
| 84 |
+
answer is never misread as a rupee amount.
|
|
|
|
|
|
|
|
|
|
| 85 |
|
| 86 |
Returns the integer rupee amount, or None if no number is recognisable
|
| 87 |
or if the only numbers in the text are clearly not currency.
|
|
|
|
| 202 |
# Engine
|
| 203 |
# ----------------------------------------------------------------------------
|
| 204 |
|
| 205 |
+
# Canonical slot order for the fact-find hint API. The actual question
|
| 206 |
+
# phrasing lives in `single_brain.py`; this list only encodes which
|
| 207 |
+
# Profile attribute to fill next when nothing else is driving the
|
| 208 |
+
# conversation.
|
| 209 |
_SLOT_ORDER: list[str] = [
|
| 210 |
"name",
|
| 211 |
"age",
|
|
|
|
| 231 |
def next_question(profile: Profile) -> Optional[str]:
|
| 232 |
"""Return the field NAME of the next missing slot, or None if complete.
|
| 233 |
|
| 234 |
+
The caller in `backend/main.py:/api/profile/completeness` uses this
|
| 235 |
+
only to hint to the frontend which slot to ask next; the actual
|
| 236 |
+
question phrasing is produced by `single_brain.py`.
|
|
|
|
|
|
|
| 237 |
|
| 238 |
A free-form session (user driving free questions) returns None so the
|
| 239 |
hint endpoint reports "nothing to ask".
|
|
|
|
| 247 |
return None
|
| 248 |
|
| 249 |
|
| 250 |
+
# Question-id → Profile field-name aliases. Some callers (notably
|
| 251 |
+
# `session_state.record_answer` driven by `awaiting_question_id`) pass a
|
| 252 |
+
# question ID rather than a Profile attribute name; this maps them.
|
|
|
|
| 253 |
_QID_TO_FIELD: dict[str, str] = {
|
| 254 |
"existing_cover": "existing_cover_inr",
|
| 255 |
"location": "location_tier",
|
|
|
|
| 261 |
def record_answer(profile: Profile, question_id: str, raw_answer: Any) -> Profile:
|
| 262 |
"""Mutate profile in place with a raw answer for a named slot.
|
| 263 |
|
| 264 |
+
Applies the parser map inline. `question_id` may be either a Profile
|
| 265 |
+
attribute name (preferred) or one of the question IDs in
|
| 266 |
+
`_QID_TO_FIELD`.
|
|
|
|
| 267 |
"""
|
| 268 |
field_name = _QID_TO_FIELD.get(question_id, question_id)
|
| 269 |
if not hasattr(profile, field_name):
|
|
@@ -1,16 +1,16 @@
|
|
| 1 |
"""Minimal NIM fallback for first-turn Gemini 503.
|
| 2 |
|
| 3 |
-
|
| 4 |
-
|
| 5 |
-
|
| 6 |
|
| 7 |
No tools, no function-calling, no faithfulness gate, no normalizer —
|
| 8 |
just a single NIM chat call with a tiny system prompt. The user gets a
|
| 9 |
polite "let me think out loud while I reconnect" reply; the next turn,
|
| 10 |
single_brain (Gemini) takes over and the session sticks.
|
| 11 |
|
| 12 |
-
|
| 13 |
-
|
| 14 |
If the elector has nothing eligible, we fall back to BRAIN_CHAIN[0] as a
|
| 15 |
last resort so the user still gets a reply.
|
| 16 |
|
|
@@ -96,9 +96,9 @@ def _pick_model() -> str:
|
|
| 96 |
|
| 97 |
|
| 98 |
def _flatten_history(chat_history: Optional[list[dict]]) -> list[ChatMessage]:
|
| 99 |
-
"""Translate the
|
| 100 |
-
|
| 101 |
-
|
| 102 |
"""
|
| 103 |
out: list[ChatMessage] = []
|
| 104 |
for msg in chat_history or []:
|
|
|
|
| 1 |
"""Minimal NIM fallback for first-turn Gemini 503.
|
| 2 |
|
| 3 |
+
Does ONE thing: when single_brain raises SingleBrainError on the first
|
| 4 |
+
turn (no single_brain_sticky yet), give the user a working reply using
|
| 5 |
+
NIM.
|
| 6 |
|
| 7 |
No tools, no function-calling, no faithfulness gate, no normalizer —
|
| 8 |
just a single NIM chat call with a tiny system prompt. The user gets a
|
| 9 |
polite "let me think out loud while I reconnect" reply; the next turn,
|
| 10 |
single_brain (Gemini) takes over and the session sticks.
|
| 11 |
|
| 12 |
+
NIM-only. We elect via llm_health.get_primary("brain") which walks
|
| 13 |
+
BRAIN_CHAIN in priority order over election-eligible models.
|
| 14 |
If the elector has nothing eligible, we fall back to BRAIN_CHAIN[0] as a
|
| 15 |
last resort so the user still gets a reply.
|
| 16 |
|
|
|
|
| 96 |
|
| 97 |
|
| 98 |
def _flatten_history(chat_history: Optional[list[dict]]) -> list[ChatMessage]:
|
| 99 |
+
"""Translate the {role, content} chat_history into the NIM
|
| 100 |
+
`ChatMessage` shape. Roles other than user/assistant are dropped;
|
| 101 |
+
assistant aliases collapse to `assistant`.
|
| 102 |
"""
|
| 103 |
out: list[ChatMessage] = []
|
| 104 |
for msg in chat_history or []:
|
|
@@ -1,20 +1,18 @@
|
|
| 1 |
-
"""Canonical policy identity + dedup — SINGLE SOURCE OF TRUTH
|
| 2 |
|
| 3 |
-
The
|
| 4 |
-
canonical-identity rule (KI-133 / KI-141 / KI-142 / KI-145):
|
| 5 |
|
| 6 |
1 unique IRDAI UIN = 1 unique product (UIN-primary invariant).
|
| 7 |
When no UIN is present, the product_key = the policy_id with any
|
| 8 |
trailing doctype suffix (`__wordings` / `__brochure` / `__cis` /
|
| 9 |
`__prospectus`) stripped.
|
| 10 |
|
| 11 |
-
|
| 12 |
-
|
| 13 |
-
different ids — a marketing
|
| 14 |
-
"my:Optima Secure (older variant)") or two
|
| 15 |
-
(`...__wordings` vs `...__brochure`) —
|
| 16 |
-
|
| 17 |
-
disagreed on what "the same policy" means.
|
| 18 |
|
| 19 |
This module factors that one rule so BOTH surfaces share it. It is
|
| 20 |
deliberately dependency-free (no FastAPI / Chroma import) so
|
|
|
|
| 1 |
+
"""Canonical policy identity + dedup — SINGLE SOURCE OF TRUTH.
|
| 2 |
|
| 3 |
+
The canonical-identity rule:
|
|
|
|
| 4 |
|
| 5 |
1 unique IRDAI UIN = 1 unique product (UIN-primary invariant).
|
| 6 |
When no UIN is present, the product_key = the policy_id with any
|
| 7 |
trailing doctype suffix (`__wordings` / `__brochure` / `__cis` /
|
| 8 |
`__prospectus`) stripped.
|
| 9 |
|
| 10 |
+
Both the marketplace endpoint (`main.py /api/policies/all`) and the
|
| 11 |
+
recommendation-fit gate (retrieval_filters.dedup_by_policy) key on this
|
| 12 |
+
rule, so two chunks for the SAME product with different ids — a marketing
|
| 13 |
+
rename ("my:Optima Secure" vs "my:Optima Secure (older variant)") or two
|
| 14 |
+
doctype siblings (`...__wordings` vs `...__brochure`) — collapse to one
|
| 15 |
+
identity and the two surfaces agree on what "the same policy" means.
|
|
|
|
| 16 |
|
| 17 |
This module factors that one rule so BOTH surfaces share it. It is
|
| 18 |
deliberately dependency-free (no FastAPI / Chroma import) so
|
|
@@ -817,35 +817,25 @@ _DEFAULT_BAND_POLICY_IDS: list[str] = [
|
|
| 817 |
|
| 818 |
|
| 819 |
# ───────────────────────────────────────────────────────────────────────────
|
| 820 |
-
#
|
| 821 |
-
#
|
| 822 |
-
#
|
| 823 |
-
#
|
| 824 |
-
#
|
| 825 |
-
# estimate_premium_band() used to hard-code sum_insured_default=₹10L and
|
| 826 |
-
# IGNORE the profile entirely. So a user who stated a ₹25L target saw the
|
| 827 |
-
# header chip priced at ₹10L (₹6,500–₹26,500) while the panel priced the
|
| 828 |
-
# same profile at ₹25L (point ₹19,100) — contradictory numbers for one
|
| 829 |
-
# profile. Resolving BOTH surfaces' SI from this one function makes them
|
| 830 |
-
# reconcile by construction: the panel's point estimate now falls inside
|
| 831 |
-
# the header band because the header band is the SAME basket priced at the
|
| 832 |
-
# SAME profile-resolved SI.
|
| 833 |
#
|
| 834 |
# Precedence MUST stay byte-identical to PremiumCalculatorPanel's
|
| 835 |
# useState initialiser (frontend/src/app/page.tsx ~L2417) and
|
| 836 |
# PolicyPremiumWidget's initialSumInsured contract:
|
| 837 |
# 1. profile.desired_sum_insured_inr (user's stated target SI)
|
| 838 |
# 2. profile.existing_cover_inr (closest available signal)
|
| 839 |
-
# 3. fallback default (
|
| 840 |
# ───────────────────────────────────────────────────────────────────────────
|
| 841 |
|
| 842 |
-
#
|
| 843 |
-
#
|
| 844 |
-
#
|
| 845 |
-
#
|
| 846 |
-
#
|
| 847 |
-
# has no published SI the caller prices against the user's
|
| 848 |
-
# desired_sum_insured_inr (else ₹10 L default) and surfaces a disclosure.
|
| 849 |
|
| 850 |
|
| 851 |
def resolve_profile_sum_insured(
|
|
@@ -864,9 +854,9 @@ def resolve_profile_sum_insured(
|
|
| 864 |
string/None gracefully and snaps to the nearest ₹50k so the resolved SI
|
| 865 |
lands on a representable slider stop.
|
| 866 |
|
| 867 |
-
|
| 868 |
-
|
| 869 |
-
|
| 870 |
"""
|
| 871 |
profile = profile or {}
|
| 872 |
|
|
|
|
| 817 |
|
| 818 |
|
| 819 |
# ───────────────────────────────────────────────────────────────────────────
|
| 820 |
+
# Single source of truth for the sum-insured the header band AND the
|
| 821 |
+
# per-settings panel both price at. Resolving both surfaces' SI from this
|
| 822 |
+
# one function makes them reconcile by construction: the panel's point
|
| 823 |
+
# estimate falls inside the header band because the header band is the
|
| 824 |
+
# SAME basket priced at the SAME profile-resolved SI.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 825 |
#
|
| 826 |
# Precedence MUST stay byte-identical to PremiumCalculatorPanel's
|
| 827 |
# useState initialiser (frontend/src/app/page.tsx ~L2417) and
|
| 828 |
# PolicyPremiumWidget's initialSumInsured contract:
|
| 829 |
# 1. profile.desired_sum_insured_inr (user's stated target SI)
|
| 830 |
# 2. profile.existing_cover_inr (closest available signal)
|
| 831 |
+
# 3. fallback default (₹10L)
|
| 832 |
# ───────────────────────────────────────────────────────────────────────────
|
| 833 |
|
| 834 |
+
# Pricing respects each policy's own real SI bounds and the user's actual
|
| 835 |
+
# stated target rather than a global clamp: a ₹2 Cr aspiration prices at
|
| 836 |
+
# ₹2 Cr; a ₹1 L corporate top-up prices at ₹1 L. When a policy has no
|
| 837 |
+
# published SI the caller prices against the user's desired_sum_insured_inr
|
| 838 |
+
# (else ₹10 L default) and surfaces a disclosure.
|
|
|
|
|
|
|
| 839 |
|
| 840 |
|
| 841 |
def resolve_profile_sum_insured(
|
|
|
|
| 854 |
string/None gracefully and snaps to the nearest ₹50k so the resolved SI
|
| 855 |
lands on a representable slider stop.
|
| 856 |
|
| 857 |
+
The user's actual stated target is honoured (a ₹2 Cr aspiration prices
|
| 858 |
+
at ₹2 Cr, a ₹1 L top-up at ₹1 L) rather than clamped to a synthetic
|
| 859 |
+
envelope.
|
| 860 |
"""
|
| 861 |
profile = profile or {}
|
| 862 |
|
|
@@ -230,13 +230,6 @@ def extract_potential_name(text: str) -> Optional[str]:
|
|
| 230 |
return raw
|
| 231 |
|
| 232 |
|
| 233 |
-
# NOTE 2026-05-17: `extract_name_in_context` + its `_ASSISTANT_ASKED_NAME_RE`
|
| 234 |
-
# / `_BARE_NAME_ANSWER_RE` regexes (b87bd2d, KI-RECALL-FIX) were REMOVED with
|
| 235 |
-
# the deterministic in-conversation recall scaffolding. `extract_potential_name`
|
| 236 |
-
# (above) and `try_recall_by_name` (below) remain — they back the explicit
|
| 237 |
-
# POST /api/profile/recall-by-name endpoint, not an in-chat short-circuit.
|
| 238 |
-
|
| 239 |
-
|
| 240 |
def try_recall_by_name(session, name: str) -> bool:
|
| 241 |
"""Look up a stored profile by name and STAGE it for confirmation.
|
| 242 |
|
|
|
|
| 230 |
return raw
|
| 231 |
|
| 232 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 233 |
def try_recall_by_name(session, name: str) -> bool:
|
| 234 |
"""Look up a stored profile by name and STAGE it for confirmation.
|
| 235 |
|
|
@@ -1,11 +1,10 @@
|
|
| 1 |
"""Customer-profile-as-RAG layer.
|
| 2 |
|
| 3 |
-
|
| 4 |
-
|
| 5 |
-
|
| 6 |
-
|
| 7 |
-
|
| 8 |
-
every subsequent retrieval query — see KI-117 boot cleanup).
|
| 9 |
|
| 10 |
At retrieval time, `rag/retrieve.py::retrieve(..., profile_name_slug=...)`
|
| 11 |
boosts the user's profile chunk so the LLM sees the user's context inline
|
|
@@ -39,9 +38,8 @@ _log = logging.getLogger(__name__)
|
|
| 39 |
def profile_to_chunk_text(profile: dict) -> str:
|
| 40 |
"""Render the profile dict as a natural-language paragraph for the LLM.
|
| 41 |
|
| 42 |
-
|
| 43 |
-
|
| 44 |
-
coherent "USER CONTEXT" block.
|
| 45 |
"""
|
| 46 |
parts: list[str] = ["USER CONTEXT — facts about the person asking this question:"]
|
| 47 |
|
|
@@ -124,10 +122,8 @@ def _get_collection():
|
|
| 124 |
async def upsert_profile_chunk(name_slug: str, profile_dict: dict) -> None:
|
| 125 |
"""Embed the profile paragraph and store as a single chunk in Chroma.
|
| 126 |
|
| 127 |
-
|
| 128 |
-
|
| 129 |
-
never write to Chroma, which eliminates the corruption surface that the
|
| 130 |
-
session_id keying introduced.
|
| 131 |
|
| 132 |
Idempotent — calling this on every profile update is safe; existing
|
| 133 |
chunks for the same name_slug get replaced.
|
|
@@ -138,14 +134,10 @@ async def upsert_profile_chunk(name_slug: str, profile_dict: dict) -> None:
|
|
| 138 |
if not text or len(text) < 30:
|
| 139 |
return
|
| 140 |
|
| 141 |
-
#
|
| 142 |
-
#
|
| 143 |
-
#
|
| 144 |
-
#
|
| 145 |
-
# session_id metadata field that poisoned every subsequent query whose
|
| 146 |
-
# `where` clause referenced session_id. Anonymous users no longer reach
|
| 147 |
-
# this function at all (the orchestrator gates on
|
| 148 |
-
# `session.profile.name` before calling) — this guard is belt-and-braces.
|
| 149 |
if not isinstance(name_slug, str) or not name_slug.strip():
|
| 150 |
_log.warning(
|
| 151 |
"profile_rag.upsert_profile_chunk: refusing to write — name_slug "
|
|
@@ -157,11 +149,11 @@ async def upsert_profile_chunk(name_slug: str, profile_dict: dict) -> None:
|
|
| 157 |
embedder = LocalEmbeddings()
|
| 158 |
[vec] = await embedder.embed([text], input_type="document")
|
| 159 |
|
| 160 |
-
#
|
| 161 |
-
#
|
| 162 |
-
#
|
| 163 |
-
#
|
| 164 |
-
#
|
| 165 |
expected_dim = getattr(embedder, "dimension", None) or 384
|
| 166 |
if (
|
| 167 |
not isinstance(vec, (list, tuple))
|
|
@@ -184,20 +176,18 @@ async def upsert_profile_chunk(name_slug: str, profile_dict: dict) -> None:
|
|
| 184 |
try:
|
| 185 |
coll.delete(where={"policy_id": chunk_id})
|
| 186 |
except Exception as e:
|
| 187 |
-
# Non-fatal: chunk may not exist yet.
|
| 188 |
-
#
|
| 189 |
_log.debug(
|
| 190 |
"profile_rag.upsert_profile_chunk: delete(where=policy_id=%s) "
|
| 191 |
"non-fatal failure: %s: %s",
|
| 192 |
chunk_id, type(e).__name__, str(e)[:200],
|
| 193 |
)
|
| 194 |
|
| 195 |
-
#
|
| 196 |
-
#
|
| 197 |
-
#
|
| 198 |
-
#
|
| 199 |
-
# the chat reply still returns even if the profile-chunk write fails.
|
| 200 |
-
# On next upsert (next profile field change), the retry will succeed.
|
| 201 |
try:
|
| 202 |
coll.add(
|
| 203 |
ids=[chunk_id],
|
|
@@ -208,8 +198,8 @@ async def upsert_profile_chunk(name_slug: str, profile_dict: dict) -> None:
|
|
| 208 |
"insurer_slug": "profile",
|
| 209 |
"policy_name": f"User profile ({name_slug[:16]})",
|
| 210 |
"doc_type": "profile",
|
| 211 |
-
#
|
| 212 |
-
#
|
| 213 |
"name_slug": name_slug,
|
| 214 |
"source_url": "",
|
| 215 |
"page_start": 0,
|
|
@@ -228,7 +218,7 @@ async def upsert_profile_chunk(name_slug: str, profile_dict: dict) -> None:
|
|
| 228 |
|
| 229 |
|
| 230 |
def remove_profile_chunk(name_slug: str) -> None:
|
| 231 |
-
"""Optional cleanup.
|
| 232 |
if not name_slug:
|
| 233 |
return
|
| 234 |
try:
|
|
|
|
| 1 |
"""Customer-profile-as-RAG layer.
|
| 2 |
|
| 3 |
+
Profile chunks are keyed by `name_slug` (the canonicalised user name),
|
| 4 |
+
NOT by session_id. Only NAMED users ever get embedded; anonymous sessions
|
| 5 |
+
never write to Chroma. Keying by name_slug avoids the corruption surface
|
| 6 |
+
of session_id-keyed chunks (an anonymous row missing a session_id
|
| 7 |
+
metadata field could poison session_id-scoped retrieval queries).
|
|
|
|
| 8 |
|
| 9 |
At retrieval time, `rag/retrieve.py::retrieve(..., profile_name_slug=...)`
|
| 10 |
boosts the user's profile chunk so the LLM sees the user's context inline
|
|
|
|
| 38 |
def profile_to_chunk_text(profile: dict) -> str:
|
| 39 |
"""Render the profile dict as a natural-language paragraph for the LLM.
|
| 40 |
|
| 41 |
+
Rendered as a coherent "USER CONTEXT" block so that when this chunk is
|
| 42 |
+
retrieved alongside policy chunks, the LLM sees the user's facts inline.
|
|
|
|
| 43 |
"""
|
| 44 |
parts: list[str] = ["USER CONTEXT — facts about the person asking this question:"]
|
| 45 |
|
|
|
|
| 122 |
async def upsert_profile_chunk(name_slug: str, profile_dict: dict) -> None:
|
| 123 |
"""Embed the profile paragraph and store as a single chunk in Chroma.
|
| 124 |
|
| 125 |
+
Keyed by `name_slug` (canonical user name), not `session_id`. Only
|
| 126 |
+
NAMED users ever get embedded — anonymous chats never write to Chroma.
|
|
|
|
|
|
|
| 127 |
|
| 128 |
Idempotent — calling this on every profile update is safe; existing
|
| 129 |
chunks for the same name_slug get replaced.
|
|
|
|
| 134 |
if not text or len(text) < 30:
|
| 135 |
return
|
| 136 |
|
| 137 |
+
# Input guard 1: name_slug must be a non-empty str. An empty key would
|
| 138 |
+
# upsert under id "profile_" and collide across anonymous sessions.
|
| 139 |
+
# Callers gate on `session.profile.name` before calling, so anonymous
|
| 140 |
+
# users do not reach this function; this guard is belt-and-braces.
|
|
|
|
|
|
|
|
|
|
|
|
|
| 141 |
if not isinstance(name_slug, str) or not name_slug.strip():
|
| 142 |
_log.warning(
|
| 143 |
"profile_rag.upsert_profile_chunk: refusing to write — name_slug "
|
|
|
|
| 149 |
embedder = LocalEmbeddings()
|
| 150 |
[vec] = await embedder.embed([text], input_type="document")
|
| 151 |
|
| 152 |
+
# Input guard 2: embedding must be a list of finite floats whose length
|
| 153 |
+
# matches the embedder's declared dimension. An empty / None / mis-shaped
|
| 154 |
+
# embedding added to Chroma would silently corrupt HNSW (dangling
|
| 155 |
+
# pointer or shape mismatch). The corpus uses 384-dim
|
| 156 |
+
# BAAI/bge-small-en-v1.5; any other shape is rejected.
|
| 157 |
expected_dim = getattr(embedder, "dimension", None) or 384
|
| 158 |
if (
|
| 159 |
not isinstance(vec, (list, tuple))
|
|
|
|
| 176 |
try:
|
| 177 |
coll.delete(where={"policy_id": chunk_id})
|
| 178 |
except Exception as e:
|
| 179 |
+
# Non-fatal: chunk may not exist yet. Log so a silent failure of
|
| 180 |
+
# the profile store is observable.
|
| 181 |
_log.debug(
|
| 182 |
"profile_rag.upsert_profile_chunk: delete(where=policy_id=%s) "
|
| 183 |
"non-fatal failure: %s: %s",
|
| 184 |
chunk_id, type(e).__name__, str(e)[:200],
|
| 185 |
)
|
| 186 |
|
| 187 |
+
# Wrap coll.add() in try/except so upsert is non-fatal: a transient
|
| 188 |
+
# Chroma sqlite lock during HNSW compaction (add() interleaving with
|
| 189 |
+
# the retrieve path's get()) must not break the chat reply. The next
|
| 190 |
+
# upsert (next profile field change) retries the write.
|
|
|
|
|
|
|
| 191 |
try:
|
| 192 |
coll.add(
|
| 193 |
ids=[chunk_id],
|
|
|
|
| 198 |
"insurer_slug": "profile",
|
| 199 |
"policy_name": f"User profile ({name_slug[:16]})",
|
| 200 |
"doc_type": "profile",
|
| 201 |
+
# Stamp name_slug; the retrieve path filters profile
|
| 202 |
+
# chunks via this field.
|
| 203 |
"name_slug": name_slug,
|
| 204 |
"source_url": "",
|
| 205 |
"page_start": 0,
|
|
|
|
| 218 |
|
| 219 |
|
| 220 |
def remove_profile_chunk(name_slug: str) -> None:
|
| 221 |
+
"""Optional cleanup. Keyed by name_slug."""
|
| 222 |
if not name_slug:
|
| 223 |
return
|
| 224 |
try:
|
|
@@ -14,8 +14,8 @@ Architectural answer to the "embed or JSON?" question:
|
|
| 14 |
style questions. Embedding cost = once per update, not per query.
|
| 15 |
|
| 16 |
Both layers stay in sync: when `save_profile()` is called here, the
|
| 17 |
-
|
| 18 |
-
|
| 19 |
|
| 20 |
Files live under `40-data/profiles/<normalised-name>.json`. Names are
|
| 21 |
normalised to lowercase + alpha-only for the filename so "Rohit" and
|
|
@@ -48,8 +48,8 @@ def _normalise_name(name: str) -> str:
|
|
| 48 |
return cleaned[:60] # cap filename length
|
| 49 |
|
| 50 |
|
| 51 |
-
#
|
| 52 |
-
#
|
| 53 |
_PERSONA_ID_FIELDS: tuple[str, ...] = (
|
| 54 |
"age", "dependents", "income_band", "location_tier", "parents_age_max",
|
| 55 |
)
|
|
@@ -170,7 +170,7 @@ def load_profile(name: str, *, persona_id: Optional[str] = None) -> Optional[Pro
|
|
| 170 |
p = _path_for(name)
|
| 171 |
if p and p.exists():
|
| 172 |
return _load_from_path(p)
|
| 173 |
-
#
|
| 174 |
return None
|
| 175 |
|
| 176 |
|
|
@@ -266,10 +266,10 @@ def save_profile(name: str, profile: Profile, *, session_id: Optional[str] = Non
|
|
| 266 |
|
| 267 |
|
| 268 |
# ---------------------------------------------------------------------------
|
| 269 |
-
#
|
| 270 |
#
|
| 271 |
# Three event types are tracked on the Profile:
|
| 272 |
-
# shown — auto-logged by
|
| 273 |
# recommendation / comparison turn that passed faithfulness.
|
| 274 |
# selected — user clicked "save / shortlist" on a policy card (frontend
|
| 275 |
# POSTs to /api/profile/select).
|
|
|
|
| 14 |
style questions. Embedding cost = once per update, not per query.
|
| 15 |
|
| 16 |
Both layers stay in sync: when `save_profile()` is called here, the
|
| 17 |
+
save path also fires `profile_rag.upsert_profile_chunk()` so the Chroma
|
| 18 |
+
side reflects the new state.
|
| 19 |
|
| 20 |
Files live under `40-data/profiles/<normalised-name>.json`. Names are
|
| 21 |
normalised to lowercase + alpha-only for the filename so "Rohit" and
|
|
|
|
| 48 |
return cleaned[:60] # cap filename length
|
| 49 |
|
| 50 |
|
| 51 |
+
# Identity-defining fields used to disambiguate two users with the same
|
| 52 |
+
# display name. Order matters for hash stability.
|
| 53 |
_PERSONA_ID_FIELDS: tuple[str, ...] = (
|
| 54 |
"age", "dependents", "income_band", "location_tier", "parents_age_max",
|
| 55 |
)
|
|
|
|
| 170 |
p = _path_for(name)
|
| 171 |
if p and p.exists():
|
| 172 |
return _load_from_path(p)
|
| 173 |
+
# No cross-identity display-name directory scan (would be a leak vector).
|
| 174 |
return None
|
| 175 |
|
| 176 |
|
|
|
|
| 266 |
|
| 267 |
|
| 268 |
# ---------------------------------------------------------------------------
|
| 269 |
+
# Per-user policy interaction tracking.
|
| 270 |
#
|
| 271 |
# Three event types are tracked on the Profile:
|
| 272 |
+
# shown — auto-logged by the brain when a policy is cited in a
|
| 273 |
# recommendation / comparison turn that passed faithfulness.
|
| 274 |
# selected — user clicked "save / shortlist" on a policy card (frontend
|
| 275 |
# POSTs to /api/profile/select).
|
|
@@ -1,26 +1,17 @@
|
|
| 1 |
"""Provider clients — thin, async, behind a common interface.
|
| 2 |
|
| 3 |
-
|
| 4 |
-
D-022 on 2026-05-14 — NIM brain pool swap), the provider stack is:
|
| 5 |
|
| 6 |
sarvam_stt.py — Sarvam Saarika v2.5 (speech-to-text)
|
| 7 |
sarvam_tts.py — Sarvam Bulbul v2 (text-to-speech, speaker anushka)
|
| 8 |
sarvam_llm.py — Sarvam-M (Indic translation IN and OUT; not brain)
|
| 9 |
local_embeddings.py — BGE-small-en-v1.5 (local CPU embeddings)
|
| 10 |
-
nvidia_nim_llm.py — Chain pattern via NimChainLLM.
|
| 11 |
-
|
| 12 |
-
|
| 13 |
-
|
| 14 |
-
|
| 15 |
-
|
| 16 |
-
entries that re-enter rotation when NIM's pools
|
| 17 |
-
recover from the 2026-05-14 outage.
|
| 18 |
-
openrouter_llm.py — OpenRouter cross-provider fallback (bottom of every chain).
|
| 19 |
-
groq_llm.py — Groq LPU; brain primary rotates 50/50 NIM ↔ Groq
|
| 20 |
-
per call (KI-025 / ADR-026).
|
| 21 |
-
|
| 22 |
-
See 70-docs/60-decisions/ADR-019, ADR-026, ADR-030 and 80-audit/ENTERPRISE_AUDIT.md
|
| 23 |
-
(D-022 row) for the full provenance.
|
| 24 |
|
| 25 |
All clients are async (use httpx.AsyncClient) so the FastAPI handlers can
|
| 26 |
parallelize provider calls without blocking the event loop.
|
|
|
|
| 1 |
"""Provider clients — thin, async, behind a common interface.
|
| 2 |
|
| 3 |
+
The provider stack:
|
|
|
|
| 4 |
|
| 5 |
sarvam_stt.py — Sarvam Saarika v2.5 (speech-to-text)
|
| 6 |
sarvam_tts.py — Sarvam Bulbul v2 (text-to-speech, speaker anushka)
|
| 7 |
sarvam_llm.py — Sarvam-M (Indic translation IN and OUT; not brain)
|
| 8 |
local_embeddings.py — BGE-small-en-v1.5 (local CPU embeddings)
|
| 9 |
+
nvidia_nim_llm.py — Chain pattern via NimChainLLM. BRAIN_CHAIN is the
|
| 10 |
+
only chain: Qwen 3-Next 80B → Mistral Large 3
|
| 11 |
+
675B → Llama-4 Maverick 17B → Nemotron-Super 49B
|
| 12 |
+
(last resort).
|
| 13 |
+
openrouter_llm.py — OpenRouter client (cross-provider option).
|
| 14 |
+
groq_llm.py — Groq LPU client.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 15 |
|
| 16 |
All clients are async (use httpx.AsyncClient) so the FastAPI handlers can
|
| 17 |
parallelize provider calls without blocking the event loop.
|
|
@@ -7,7 +7,7 @@ The interfaces are deliberately narrow:
|
|
| 7 |
- An embeddings provider takes a list of strings and returns vectors.
|
| 8 |
|
| 9 |
Concrete implementations live in sibling modules (sarvam_*, groq_*, voyage_*,
|
| 10 |
-
openrouter_*).
|
| 11 |
class — so swapping providers is a config change.
|
| 12 |
"""
|
| 13 |
|
|
|
|
| 7 |
- An embeddings provider takes a list of strings and returns vectors.
|
| 8 |
|
| 9 |
Concrete implementations live in sibling modules (sarvam_*, groq_*, voyage_*,
|
| 10 |
+
openrouter_*). Callers import these by interface, never by concrete
|
| 11 |
class — so swapping providers is a config change.
|
| 12 |
"""
|
| 13 |
|
|
@@ -1,9 +1,9 @@
|
|
| 1 |
-
"""Google Gemini — primary LLM tier for
|
| 2 |
|
| 3 |
-
Google AI Studio's free tier on `gemini-2.
|
| 4 |
-
|
| 5 |
-
|
| 6 |
-
|
| 7 |
|
| 8 |
Wire-shape: REST API direct via httpx (no SDK dependency — matches the pattern
|
| 9 |
of nvidia_nim_llm.py and openrouter_llm.py).
|
|
@@ -35,9 +35,8 @@ JSON mode:
|
|
| 35 |
response_format == {"type": "json_object"} →
|
| 36 |
generationConfig.responseMimeType = "application/json"
|
| 37 |
|
| 38 |
-
The provider matches the
|
| 39 |
-
|
| 40 |
-
path (Tier 0, via TieredBrainLLM wrapper).
|
| 41 |
"""
|
| 42 |
|
| 43 |
from __future__ import annotations
|
|
@@ -57,13 +56,13 @@ from backend.providers.base import ChatMessage, LLMProvider, LLMResult
|
|
| 57 |
|
| 58 |
GEMINI_BASE_URL = "https://generativelanguage.googleapis.com/v1beta/models"
|
| 59 |
GEMINI_CACHE_URL = "https://generativelanguage.googleapis.com/v1beta/cachedContents"
|
| 60 |
-
DEFAULT_MODEL = "gemini-2.5-flash" #
|
| 61 |
|
| 62 |
# ----------------------------------------------------------------------------
|
| 63 |
-
#
|
| 64 |
#
|
| 65 |
# Keyed by `(model, sha256(system_text))` so the same base preamble shared
|
| 66 |
-
# across
|
| 67 |
# dict with the cache `name` (the server-side resource id used in subsequent
|
| 68 |
# generateContent bodies as `cachedContent`) and the local-clock `expires_at`
|
| 69 |
# wall-time so we can self-evict before issuing a guaranteed-miss request.
|
|
@@ -77,7 +76,7 @@ DEFAULT_MODEL = "gemini-2.5-flash" # KI-183 retired gemini-2.0-flash → 2.5-fl
|
|
| 77 |
_CACHE_REGISTRY: dict[tuple[str, str, str], dict] = {}
|
| 78 |
_CACHE_REGISTRY_LOCK = threading.Lock()
|
| 79 |
|
| 80 |
-
#
|
| 81 |
# `cachedContents` resources live up to ~60min on free tier; we refresh
|
| 82 |
# before that ceiling so an in-flight call never lands on an expired cache.
|
| 83 |
# `CACHE_REFRESH_AGE_SEC` is the wall-clock age at which we proactively
|
|
@@ -89,11 +88,11 @@ CACHE_REFRESH_AGE_SEC = 50 * 60 # 50min — refresh BEFORE 60min server ceiling
|
|
| 89 |
def _cache_key(model: str, system_text: str, dynamic_prefix: str = "") -> tuple[str, str, str]:
|
| 90 |
"""Build the registry key for a (model, system_text, dynamic_prefix) tuple.
|
| 91 |
|
| 92 |
-
|
| 93 |
-
|
| 94 |
-
|
| 95 |
-
|
| 96 |
-
|
| 97 |
|
| 98 |
Hashing inputs rather than storing the raw string keeps the registry
|
| 99 |
footprint tiny even when the preamble is multi-KB.
|
|
@@ -104,18 +103,18 @@ def _cache_key(model: str, system_text: str, dynamic_prefix: str = "") -> tuple[
|
|
| 104 |
|
| 105 |
|
| 106 |
# ---------------------------------------------------------------------------
|
| 107 |
-
#
|
| 108 |
#
|
| 109 |
# Gemini's REST surface returns different error shapes for different failure
|
| 110 |
# modes (429 rate-limit, 400 BlockedReason / SafetyRating, 404 cache not
|
| 111 |
-
# found, 500/503 server errors). The tier
|
| 112 |
# `BrainProviderError(retryable=bool)` where `retryable=True` signals the
|
| 113 |
# tier should fall through to the next provider, and `retryable=False`
|
| 114 |
# signals a hard error (auth / content blocked) that should surface to the
|
| 115 |
# caller without burning fallback budget.
|
| 116 |
# ---------------------------------------------------------------------------
|
| 117 |
class BrainProviderError(RuntimeError):
|
| 118 |
-
"""Stable provider-error envelope consumed by
|
| 119 |
|
| 120 |
Attributes:
|
| 121 |
provider: short name ("gemini" / "nim" / ...)
|
|
@@ -142,7 +141,7 @@ class BrainProviderError(RuntimeError):
|
|
| 142 |
def _classify_gemini_error(status_code: int, detail: str) -> BrainProviderError:
|
| 143 |
"""Map a Gemini REST response into BrainProviderError(retryable=bool).
|
| 144 |
|
| 145 |
-
Routing rules (
|
| 146 |
- 429 (rate-limit / quota) → retryable (next tier)
|
| 147 |
- 5xx (server errors) → retryable (next tier)
|
| 148 |
- 408 / 504 (timeout) → retryable
|
|
@@ -184,9 +183,9 @@ def invalidate_cache(model: str, system_text: str, dynamic_prefix: str = "") ->
|
|
| 184 |
alive (it will lapse on TTL), but our reference is gone so the next
|
| 185 |
chat() call provisions a fresh one.
|
| 186 |
|
| 187 |
-
|
| 188 |
-
|
| 189 |
-
|
| 190 |
"""
|
| 191 |
key = _cache_key(model, system_text, dynamic_prefix)
|
| 192 |
with _CACHE_REGISTRY_LOCK:
|
|
@@ -270,8 +269,7 @@ class GoogleGeminiLLM(LLMProvider):
|
|
| 270 |
Re-uses an existing live cache from the module registry when the
|
| 271 |
(model, system_text) pair matches AND the local `expires_at` is still
|
| 272 |
in the future. Cache misses + creation failures are silent (logged at
|
| 273 |
-
INFO) so a caching outage never breaks the main path
|
| 274 |
-
requires fail-safe behaviour.
|
| 275 |
"""
|
| 276 |
if not self.api_key or not system_text:
|
| 277 |
return None
|
|
@@ -280,8 +278,8 @@ class GoogleGeminiLLM(LLMProvider):
|
|
| 280 |
now = time.time()
|
| 281 |
with _CACHE_REGISTRY_LOCK:
|
| 282 |
entry = _CACHE_REGISTRY.get(key)
|
| 283 |
-
#
|
| 284 |
-
# (a) self-evict ~10s before LOCAL expires_at
|
| 285 |
# (b) PROACTIVELY refresh if the entry is older than
|
| 286 |
# CACHE_REFRESH_AGE_SEC (50min) regardless of expires_at —
|
| 287 |
# guards against server-side TTL drift on long-lived
|
|
@@ -349,8 +347,8 @@ class GoogleGeminiLLM(LLMProvider):
|
|
| 349 |
# truth, but we shadow it locally so we self-evict before
|
| 350 |
# the inevitable 4xx on an expired reference.
|
| 351 |
"expires_at": now_create + ttl_seconds,
|
| 352 |
-
#
|
| 353 |
-
#
|
| 354 |
# when caller set a longer TTL than Google honours.
|
| 355 |
"created_at": now_create,
|
| 356 |
}
|
|
@@ -410,12 +408,11 @@ class GoogleGeminiLLM(LLMProvider):
|
|
| 410 |
# connection releases its slot on its own deadline rather than holding
|
| 411 |
# past the outer wait_for cancellation.
|
| 412 |
#
|
| 413 |
-
#
|
| 414 |
-
#
|
| 415 |
-
#
|
| 416 |
-
#
|
| 417 |
-
# asyncio.CancelledError leaking up through the cancellation chain
|
| 418 |
-
# (which the tier wrapper has historically misclassified).
|
| 419 |
read_timeout = max(2.0, self.timeout - 2.0)
|
| 420 |
client_timeout = httpx.Timeout(
|
| 421 |
connect=2.0,
|
|
@@ -430,9 +427,9 @@ class GoogleGeminiLLM(LLMProvider):
|
|
| 430 |
except (asyncio.CancelledError, KeyboardInterrupt, SystemExit):
|
| 431 |
raise
|
| 432 |
except httpx.TimeoutException as e:
|
| 433 |
-
#
|
| 434 |
-
#
|
| 435 |
-
#
|
| 436 |
raise BrainProviderError(
|
| 437 |
f"Gemini timeout after {read_timeout:.1f}s (model={self.model})",
|
| 438 |
provider="gemini", retryable=True, status=None,
|
|
@@ -443,7 +440,7 @@ class GoogleGeminiLLM(LLMProvider):
|
|
| 443 |
f"Gemini transport error ({type(e).__name__}): {str(e)[:200]}",
|
| 444 |
provider="gemini", retryable=True, status=None,
|
| 445 |
) from e
|
| 446 |
-
#
|
| 447 |
# Symptoms: 400/404 with body mentioning "cachedContent" /
|
| 448 |
# "cache" / "not found". Strip the reference, re-add the inline
|
| 449 |
# systemInstruction, drop the registry entry, retry once.
|
|
@@ -484,8 +481,8 @@ class GoogleGeminiLLM(LLMProvider):
|
|
| 484 |
provider="gemini", retryable=True, status=None,
|
| 485 |
) from e
|
| 486 |
if resp.status_code >= 400:
|
| 487 |
-
#
|
| 488 |
-
#
|
| 489 |
# retryable=True → fall through to next tier
|
| 490 |
# retryable=False → surface to caller (auth/content-block)
|
| 491 |
# The original httpx.HTTPStatusError is preserved as __cause__
|
|
@@ -542,13 +539,12 @@ def get_gemini_llm(
|
|
| 542 |
) -> GoogleGeminiLLM:
|
| 543 |
"""Return a fresh GoogleGeminiLLM client.
|
| 544 |
|
| 545 |
-
|
| 546 |
-
|
| 547 |
-
|
| 548 |
|
| 549 |
Common models:
|
| 550 |
-
- "gemini-2.
|
| 551 |
-
- "gemini-2.5-flash" → slightly heavier reasoning, brain main Tier 0
|
| 552 |
"""
|
| 553 |
return GoogleGeminiLLM(model=model, timeout=timeout)
|
| 554 |
|
|
|
|
| 1 |
+
"""Google Gemini — primary LLM tier for the single-brain stack.
|
| 2 |
|
| 3 |
+
Google AI Studio's free tier on `gemini-2.5-flash` gives 1500 req/day
|
| 4 |
+
with native JSON mode (`responseMimeType: "application/json"`). This is
|
| 5 |
+
best-in-class free quality for conversation and beats every model in NIM
|
| 6 |
+
or OpenRouter free tiers.
|
| 7 |
|
| 8 |
Wire-shape: REST API direct via httpx (no SDK dependency — matches the pattern
|
| 9 |
of nvidia_nim_llm.py and openrouter_llm.py).
|
|
|
|
| 35 |
response_format == {"type": "json_object"} →
|
| 36 |
generationConfig.responseMimeType = "application/json"
|
| 37 |
|
| 38 |
+
The provider matches the `LLMProvider` interface so it slots straight
|
| 39 |
+
into the single-brain stack (single_brain.py) as the primary tier.
|
|
|
|
| 40 |
"""
|
| 41 |
|
| 42 |
from __future__ import annotations
|
|
|
|
| 56 |
|
| 57 |
GEMINI_BASE_URL = "https://generativelanguage.googleapis.com/v1beta/models"
|
| 58 |
GEMINI_CACHE_URL = "https://generativelanguage.googleapis.com/v1beta/cachedContents"
|
| 59 |
+
DEFAULT_MODEL = "gemini-2.5-flash" # free-tier, brain-grade. NOT the -lite tier: it does not reliably support single-brain tool-calling (save_profile_field).
|
| 60 |
|
| 61 |
# ----------------------------------------------------------------------------
|
| 62 |
+
# Module-level cachedContents registry.
|
| 63 |
#
|
| 64 |
# Keyed by `(model, sha256(system_text))` so the same base preamble shared
|
| 65 |
+
# across brain calls deduplicates onto one cache. Each value is a small
|
| 66 |
# dict with the cache `name` (the server-side resource id used in subsequent
|
| 67 |
# generateContent bodies as `cachedContent`) and the local-clock `expires_at`
|
| 68 |
# wall-time so we can self-evict before issuing a guaranteed-miss request.
|
|
|
|
| 76 |
_CACHE_REGISTRY: dict[tuple[str, str, str], dict] = {}
|
| 77 |
_CACHE_REGISTRY_LOCK = threading.Lock()
|
| 78 |
|
| 79 |
+
# Gemini cachedContents server-side TTL ceiling. Google's
|
| 80 |
# `cachedContents` resources live up to ~60min on free tier; we refresh
|
| 81 |
# before that ceiling so an in-flight call never lands on an expired cache.
|
| 82 |
# `CACHE_REFRESH_AGE_SEC` is the wall-clock age at which we proactively
|
|
|
|
| 88 |
def _cache_key(model: str, system_text: str, dynamic_prefix: str = "") -> tuple[str, str, str]:
|
| 89 |
"""Build the registry key for a (model, system_text, dynamic_prefix) tuple.
|
| 90 |
|
| 91 |
+
The key partitions on a separate `dynamic_prefix` hash (in addition to
|
| 92 |
+
the preamble SHA256) so per-persona caches don't collide on the same
|
| 93 |
+
static preamble hash when `_dynamic_profile_block` varies per persona.
|
| 94 |
+
`dynamic_prefix` defaults to "" so callers that don't pass it get the
|
| 95 |
+
preamble-only key.
|
| 96 |
|
| 97 |
Hashing inputs rather than storing the raw string keeps the registry
|
| 98 |
footprint tiny even when the preamble is multi-KB.
|
|
|
|
| 103 |
|
| 104 |
|
| 105 |
# ---------------------------------------------------------------------------
|
| 106 |
+
# Normalized provider error class.
|
| 107 |
#
|
| 108 |
# Gemini's REST surface returns different error shapes for different failure
|
| 109 |
# modes (429 rate-limit, 400 BlockedReason / SafetyRating, 404 cache not
|
| 110 |
+
# found, 500/503 server errors). The brain tier needs a stable contract:
|
| 111 |
# `BrainProviderError(retryable=bool)` where `retryable=True` signals the
|
| 112 |
# tier should fall through to the next provider, and `retryable=False`
|
| 113 |
# signals a hard error (auth / content blocked) that should surface to the
|
| 114 |
# caller without burning fallback budget.
|
| 115 |
# ---------------------------------------------------------------------------
|
| 116 |
class BrainProviderError(RuntimeError):
|
| 117 |
+
"""Stable provider-error envelope consumed by the brain tier.
|
| 118 |
|
| 119 |
Attributes:
|
| 120 |
provider: short name ("gemini" / "nim" / ...)
|
|
|
|
| 141 |
def _classify_gemini_error(status_code: int, detail: str) -> BrainProviderError:
|
| 142 |
"""Map a Gemini REST response into BrainProviderError(retryable=bool).
|
| 143 |
|
| 144 |
+
Routing rules (the brain tier's expectations):
|
| 145 |
- 429 (rate-limit / quota) → retryable (next tier)
|
| 146 |
- 5xx (server errors) → retryable (next tier)
|
| 147 |
- 408 / 504 (timeout) → retryable
|
|
|
|
| 183 |
alive (it will lapse on TTL), but our reference is gone so the next
|
| 184 |
chat() call provisions a fresh one.
|
| 185 |
|
| 186 |
+
`dynamic_prefix` is an optional partition arg; defaults to "". When
|
| 187 |
+
supplied, only the matching (model, system_text, dynamic_prefix)
|
| 188 |
+
entry is dropped.
|
| 189 |
"""
|
| 190 |
key = _cache_key(model, system_text, dynamic_prefix)
|
| 191 |
with _CACHE_REGISTRY_LOCK:
|
|
|
|
| 269 |
Re-uses an existing live cache from the module registry when the
|
| 270 |
(model, system_text) pair matches AND the local `expires_at` is still
|
| 271 |
in the future. Cache misses + creation failures are silent (logged at
|
| 272 |
+
INFO) so a caching outage never breaks the main path (fail-safe).
|
|
|
|
| 273 |
"""
|
| 274 |
if not self.api_key or not system_text:
|
| 275 |
return None
|
|
|
|
| 278 |
now = time.time()
|
| 279 |
with _CACHE_REGISTRY_LOCK:
|
| 280 |
entry = _CACHE_REGISTRY.get(key)
|
| 281 |
+
# TWO-stage refresh:
|
| 282 |
+
# (a) self-evict ~10s before LOCAL expires_at.
|
| 283 |
# (b) PROACTIVELY refresh if the entry is older than
|
| 284 |
# CACHE_REFRESH_AGE_SEC (50min) regardless of expires_at —
|
| 285 |
# guards against server-side TTL drift on long-lived
|
|
|
|
| 347 |
# truth, but we shadow it locally so we self-evict before
|
| 348 |
# the inevitable 4xx on an expired reference.
|
| 349 |
"expires_at": now_create + ttl_seconds,
|
| 350 |
+
# created_at lets us proactively refresh entries that have
|
| 351 |
+
# lived past CACHE_REFRESH_AGE_SEC even
|
| 352 |
# when caller set a longer TTL than Google honours.
|
| 353 |
"created_at": now_create,
|
| 354 |
}
|
|
|
|
| 408 |
# connection releases its slot on its own deadline rather than holding
|
| 409 |
# past the outer wait_for cancellation.
|
| 410 |
#
|
| 411 |
+
# Timeout binding: bind the httpx read timeout to
|
| 412 |
+
# `self.timeout - 2.0` so the underlying connection times out ~2s
|
| 413 |
+
# BEFORE the outer wait_for cancellation. This surfaces a clean
|
| 414 |
+
# BrainProviderError(retryable=True) instead of an
|
| 415 |
+
# asyncio.CancelledError leaking up through the cancellation chain.
|
|
|
|
| 416 |
read_timeout = max(2.0, self.timeout - 2.0)
|
| 417 |
client_timeout = httpx.Timeout(
|
| 418 |
connect=2.0,
|
|
|
|
| 427 |
except (asyncio.CancelledError, KeyboardInterrupt, SystemExit):
|
| 428 |
raise
|
| 429 |
except httpx.TimeoutException as e:
|
| 430 |
+
# TimeoutException → retryable BrainProviderError so the
|
| 431 |
+
# tier falls through cleanly instead of seeing
|
| 432 |
+
# asyncio.CancelledError.
|
| 433 |
raise BrainProviderError(
|
| 434 |
f"Gemini timeout after {read_timeout:.1f}s (model={self.model})",
|
| 435 |
provider="gemini", retryable=True, status=None,
|
|
|
|
| 440 |
f"Gemini transport error ({type(e).__name__}): {str(e)[:200]}",
|
| 441 |
provider="gemini", retryable=True, status=None,
|
| 442 |
) from e
|
| 443 |
+
# Graceful fallback when a cache reference is stale.
|
| 444 |
# Symptoms: 400/404 with body mentioning "cachedContent" /
|
| 445 |
# "cache" / "not found". Strip the reference, re-add the inline
|
| 446 |
# systemInstruction, drop the registry entry, retry once.
|
|
|
|
| 481 |
provider="gemini", retryable=True, status=None,
|
| 482 |
) from e
|
| 483 |
if resp.status_code >= 400:
|
| 484 |
+
# Normalize the error into BrainProviderError so the tier
|
| 485 |
+
# sees a stable contract:
|
| 486 |
# retryable=True → fall through to next tier
|
| 487 |
# retryable=False → surface to caller (auth/content-block)
|
| 488 |
# The original httpx.HTTPStatusError is preserved as __cause__
|
|
|
|
| 539 |
) -> GoogleGeminiLLM:
|
| 540 |
"""Return a fresh GoogleGeminiLLM client.
|
| 541 |
|
| 542 |
+
Gemini Flash typically responds in 1-3s so the 25s default timeout is
|
| 543 |
+
a generous outer bound that still bails fast on a quota / network
|
| 544 |
+
stall.
|
| 545 |
|
| 546 |
Common models:
|
| 547 |
+
- "gemini-2.5-flash" → conversational brain tier (default)
|
|
|
|
| 548 |
"""
|
| 549 |
return GoogleGeminiLLM(model=model, timeout=timeout)
|
| 550 |
|
|
@@ -104,8 +104,8 @@ class GroqLLM(LLMProvider):
|
|
| 104 |
continue
|
| 105 |
resp.raise_for_status()
|
| 106 |
break
|
| 107 |
-
#
|
| 108 |
-
#
|
| 109 |
# the highest-fidelity provider on this front: every successful
|
| 110 |
# call returns the daily-tokens remaining, so we get a
|
| 111 |
# continuously-updated election signal at zero extra cost.
|
|
|
|
| 104 |
continue
|
| 105 |
resp.raise_for_status()
|
| 106 |
break
|
| 107 |
+
# Stamp credits_remaining from Groq's x-ratelimit-* headers
|
| 108 |
+
# BEFORE we drop the response. Groq is
|
| 109 |
# the highest-fidelity provider on this front: every successful
|
| 110 |
# call returns the daily-tokens remaining, so we get a
|
| 111 |
# continuously-updated election signal at zero extra cost.
|
|
@@ -7,8 +7,8 @@ Default model: BAAI/bge-small-en-v1.5 — 384-dim, ~110MB, top-of-class on
|
|
| 7 |
MTEB English benchmark for its size. Faster than 1024-dim cloud models on
|
| 8 |
CPU and trivially small on GPU.
|
| 9 |
|
| 10 |
-
The interface matches EmbeddingsProvider exactly, so the
|
| 11 |
-
|
| 12 |
"""
|
| 13 |
|
| 14 |
from __future__ import annotations
|
|
@@ -73,10 +73,10 @@ class LocalEmbeddings(EmbeddingsProvider):
|
|
| 73 |
texts = [f"Represent this sentence for searching relevant passages: {t}" for t in texts]
|
| 74 |
# Batch size scales by device: MPS / CUDA throughput benefits from
|
| 75 |
# bigger batches; CPU prefers smaller to avoid memory pressure on M1.
|
| 76 |
-
#
|
| 77 |
-
#
|
| 78 |
-
#
|
| 79 |
-
#
|
| 80 |
batch = 128 if self.device in ("mps", "cuda") else 32
|
| 81 |
vectors = self.model.encode(
|
| 82 |
texts,
|
|
|
|
| 7 |
MTEB English benchmark for its size. Faster than 1024-dim cloud models on
|
| 8 |
CPU and trivially small on GPU.
|
| 9 |
|
| 10 |
+
The interface matches EmbeddingsProvider exactly, so the RAG pipeline
|
| 11 |
+
doesn't change — only the import in ingest/retrieve.
|
| 12 |
"""
|
| 13 |
|
| 14 |
from __future__ import annotations
|
|
|
|
| 73 |
texts = [f"Represent this sentence for searching relevant passages: {t}" for t in texts]
|
| 74 |
# Batch size scales by device: MPS / CUDA throughput benefits from
|
| 75 |
# bigger batches; CPU prefers smaller to avoid memory pressure on M1.
|
| 76 |
+
# MPS/CUDA use batch 128 to minimise GPU kernel launches during
|
| 77 |
+
# bulk re-ingest (800-token chunks at 128 ≈ 100 MB per batch, well
|
| 78 |
+
# within Mac M-series unified memory). CPU stays at 32 to keep peak
|
| 79 |
+
# RSS bounded on machines without a GPU.
|
| 80 |
batch = 128 if self.device in ("mps", "cuda") else 32
|
| 81 |
vectors = self.model.encode(
|
| 82 |
texts,
|
|
@@ -5,20 +5,18 @@ POST https://integrate.api.nvidia.com/v1/chat/completions.
|
|
| 5 |
|
| 6 |
Why NIM:
|
| 7 |
- Frontier open-weights models hosted free (no card, no daily cap, 40 req/min)
|
| 8 |
-
- Single provider
|
| 9 |
- Same Bearer-auth + OpenAI request shape — drop-in retry/backoff
|
| 10 |
|
| 11 |
-
Role
|
| 12 |
- Brain (every reasoning turn): one elected candidate from BRAIN_CHAIN.
|
| 13 |
The chain is ordered Qwen 3-Next → Mistral Large 3 → Llama-4 Maverick
|
| 14 |
-
→ Nemotron-Super (last resort). Election
|
| 15 |
healthy/fastest candidate per turn from background probe data.
|
| 16 |
|
| 17 |
-
|
| 18 |
-
|
| 19 |
-
|
| 20 |
-
faithfulness checks. Sarvam stays for voice STT/TTS + Hindi/Hinglish/
|
| 21 |
-
vernacular translation.
|
| 22 |
"""
|
| 23 |
|
| 24 |
from __future__ import annotations
|
|
@@ -74,18 +72,12 @@ async def _append_usage(record: dict) -> None:
|
|
| 74 |
pass
|
| 75 |
|
| 76 |
|
| 77 |
-
#
|
| 78 |
#
|
| 79 |
-
#
|
| 80 |
-
#
|
| 81 |
-
#
|
| 82 |
-
#
|
| 83 |
-
#
|
| 84 |
-
# We have 5 sources of NIM traffic that all stack on the same key with no
|
| 85 |
-
# global throttle: probe loop (6-slot parallel burst every 300s), admin
|
| 86 |
-
# tab polling (every 30s), per-user chat turn (1-2 calls), concurrent
|
| 87 |
-
# users (N × 1-2), and the inner 4-attempt retry loop (holds a slot up
|
| 88 |
-
# to 15s on 429/5xx backoff).
|
| 89 |
#
|
| 90 |
# 6+ in-flight → queue → 15-25s response times → bot's 12s outer cap
|
| 91 |
# fires → user sees fallback.
|
|
@@ -99,16 +91,11 @@ async def _append_usage(record: dict) -> None:
|
|
| 99 |
_NIM_OUTBOUND_SEMAPHORE = asyncio.Semaphore(2)
|
| 100 |
|
| 101 |
|
| 102 |
-
#
|
| 103 |
-
#
|
| 104 |
-
# Pre-A4: every NvidiaNimLLM.chat() call did `async with httpx.AsyncClient(...)
|
| 105 |
-
# as client: ...`, which constructed a fresh connection pool per request and
|
| 106 |
-
# tore it down on exit. Under brain/judge concurrency this churned TCP
|
| 107 |
-
# connections, dropped TLS sessions, and (per audit) leaked pool slots when
|
| 108 |
-
# an outer wait_for cancellation interrupted the async-with __aexit__ before
|
| 109 |
-
# the underlying transport finalised.
|
| 110 |
#
|
| 111 |
-
#
|
|
|
|
|
|
|
| 112 |
# - max_connections=10 caps total outbound (NIM + probes share)
|
| 113 |
# - max_keepalive_connections=5 keeps a warm pool for the hot path
|
| 114 |
# The client is constructed lazily (first .chat() call) so cold imports
|
|
@@ -142,12 +129,6 @@ def _get_shared_nim_client() -> httpx.AsyncClient:
|
|
| 142 |
|
| 143 |
|
| 144 |
NVIDIA_NIM_BASE_URL = "https://integrate.api.nvidia.com/v1"
|
| 145 |
-
# 2026-05-14 brain swap (D-022): NIM's DeepSeek-V4 + Meta Llama inference pools
|
| 146 |
-
# are repeatedly timing out (15-120s on chat completions, no response). Qwen
|
| 147 |
-
# pool is consistently fast (2s response, clean structured output). Mistral
|
| 148 |
-
# Large 3 (Reddit benchmark: 4.3s, works on free tier) is the cross-family
|
| 149 |
-
# judge replacing the timing-out Llama-4 Maverick.
|
| 150 |
-
#
|
| 151 |
# Heavy brain (complex queries — comparison, recommendation, synthesis):
|
| 152 |
# Qwen 3-Next 80B — 80B / 3B active MoE, frontier multilingual, very fast.
|
| 153 |
NIM_BRAIN_MODEL = "qwen/qwen3-next-80b-a3b-instruct"
|
|
@@ -156,9 +137,9 @@ NIM_BRAIN_MODEL = "qwen/qwen3-next-80b-a3b-instruct"
|
|
| 156 |
# different intents to the same pool. Tiered routing kept for forward-compat.
|
| 157 |
NIM_FAST_BRAIN_MODEL = "qwen/qwen3-next-80b-a3b-instruct"
|
| 158 |
# Judge: Mistral Large 3 — 675B dense, MIT license. Different family from
|
| 159 |
-
# Qwen brain (Mistral vs Alibaba) so the judge sees the brain's output
|
| 160 |
-
# a genuinely different decision surface ("the brain doesn't mark its
|
| 161 |
-
# homework").
|
| 162 |
NIM_JUDGE_MODEL = "mistralai/mistral-large-3-675b-instruct-2512"
|
| 163 |
|
| 164 |
|
|
@@ -196,8 +177,8 @@ class NvidiaNimLLM(LLMProvider):
|
|
| 196 |
"max_tokens": max_tokens,
|
| 197 |
}
|
| 198 |
if response_format:
|
| 199 |
-
#
|
| 200 |
-
#
|
| 201 |
# (NIM's stricter constrained-decoding surface), surface that
|
| 202 |
# too when the caller passes a JSON schema (response_format
|
| 203 |
# with `json_schema`). The plain `{"type": "json_object"}` flag
|
|
@@ -227,31 +208,22 @@ class NvidiaNimLLM(LLMProvider):
|
|
| 227 |
write=2.0,
|
| 228 |
pool=2.0,
|
| 229 |
)
|
| 230 |
-
#
|
| 231 |
-
#
|
| 232 |
-
#
|
| 233 |
-
#
|
| 234 |
-
#
|
| 235 |
-
#
|
| 236 |
-
#
|
| 237 |
-
#
|
| 238 |
-
#
|
| 239 |
-
#
|
| 240 |
-
# (2) calling report_failure() so the elector demotes the
|
| 241 |
-
# slot-blocked candidate for ~30s, and
|
| 242 |
-
# (3) falling through to the elected backup (cross-provider
|
| 243 |
-
# by preference) within the same turn.
|
| 244 |
-
# KI-079 then provides one more bite via BRAIN_CHAIN if both
|
| 245 |
-
# elected models fail. Per-call retries inside this method
|
| 246 |
-
# would only re-queue against the same slot-starved pool,
|
| 247 |
-
# multiplying the queueing problem the semaphore fixes.
|
| 248 |
#
|
| 249 |
# The semaphore wraps ONLY the HTTP round-trip — not response
|
| 250 |
# parsing or usage logging — so we cap concurrent network
|
| 251 |
# traffic without serialising the rest of the pipeline.
|
| 252 |
-
#
|
| 253 |
-
#
|
| 254 |
-
# a fresh client per request. Per-request timeout is applied at
|
| 255 |
# .post() call time so the shared pool serves both long-timeout
|
| 256 |
# chat() and short-timeout probe() calls without contention.
|
| 257 |
client = _get_shared_nim_client()
|
|
@@ -295,40 +267,32 @@ class NvidiaNimLLM(LLMProvider):
|
|
| 295 |
# cross-family-grading invariant survives any failover.
|
| 296 |
|
| 297 |
BRAIN_CHAIN = [
|
| 298 |
-
#
|
| 299 |
-
#
|
| 300 |
-
#
|
| 301 |
-
#
|
| 302 |
-
#
|
| 303 |
-
#
|
| 304 |
-
# that have been "down" for 48+ consecutive probes (qwen3.5-122b,
|
| 305 |
-
# gpt-oss-120b, deepseek-v4-pro) so the election pool only contains
|
| 306 |
-
# demonstrably-healthy NIM models.
|
| 307 |
-
# KI-175 (2026-05-15) — Nemotron demoted to LAST RESORT per operator
|
| 308 |
-
# preference: nemotron is the weakest practical NIM model and should
|
| 309 |
-
# only serve when every higher-quality model is simultaneously down.
|
| 310 |
-
# Primary: Qwen 3-Next 80B — 5/5 recent probes ok, clean JSON, multilingual
|
| 311 |
"qwen/qwen3-next-80b-a3b-instruct",
|
| 312 |
-
# 1st fallback: Mistral Large 3 675B —
|
| 313 |
-
#
|
| 314 |
"mistralai/mistral-large-3-675b-instruct-2512",
|
| 315 |
-
# 2nd fallback: Meta Llama-4 Maverick 17B —
|
| 316 |
-
#
|
| 317 |
"meta/llama-4-maverick-17b-128e-instruct",
|
| 318 |
-
# 3rd / LAST RESORT: NVIDIA Nemotron-Super 49B —
|
| 319 |
-
#
|
| 320 |
-
#
|
| 321 |
"nvidia/llama-3.3-nemotron-super-49b-v1.5",
|
| 322 |
]
|
| 323 |
|
| 324 |
-
#
|
| 325 |
-
#
|
| 326 |
-
#
|
| 327 |
-
# gate (Gate 4 / Hinglish drift) was retired in favour of in-process checks.
|
| 328 |
|
| 329 |
|
| 330 |
def _classify_error(e: BaseException) -> str:
|
| 331 |
-
"""
|
| 332 |
llm_health.report_failure(). The string drives degradation duration:
|
| 333 |
rate-limit failures (HTTP 429) get a 1h sin-bin; everything else 30s.
|
| 334 |
|
|
@@ -338,7 +302,7 @@ def _classify_error(e: BaseException) -> str:
|
|
| 338 |
carries `.response.status_code == 429` we tag it `"Status429"`; for
|
| 339 |
other HTTP statuses we surface e.g. `"HTTPStatusError:503"`; for
|
| 340 |
non-HTTP exceptions we keep the class name (`TimeoutException`,
|
| 341 |
-
`ReadTimeout`, etc.)
|
| 342 |
"""
|
| 343 |
cls = type(e).__name__
|
| 344 |
try:
|
|
@@ -354,32 +318,22 @@ def _classify_error(e: BaseException) -> str:
|
|
| 354 |
|
| 355 |
|
| 356 |
class NimChainLLM(LLMProvider):
|
| 357 |
-
"""
|
| 358 |
-
|
| 359 |
-
|
| 360 |
-
|
| 361 |
-
|
| 362 |
-
|
| 363 |
-
|
| 364 |
-
|
| 365 |
-
|
| 366 |
-
|
| 367 |
-
|
| 368 |
-
provider backup per chain based on real probe latencies. chat()
|
| 369 |
-
calls the elected primary ONCE per turn (1 LLM call). On failure,
|
| 370 |
-
we demote the primary for ~30s and fall through to the elected
|
| 371 |
-
backup (still 2 LLM calls max). The chain list is now the
|
| 372 |
-
CANDIDATE POOL for election, not a per-call sequence.
|
| 373 |
-
|
| 374 |
-
Worst case per turn: 2 LLM calls (was 5-6). Plus a final filter_chain
|
| 375 |
-
refresh path is kept for the rare double-failure edge case so the
|
| 376 |
-
pre-KI-080 graceful-degradation behaviour is preserved.
|
| 377 |
|
| 378 |
`name` after a successful call reflects which model actually answered
|
| 379 |
-
so downstream callers (
|
| 380 |
-
|
| 381 |
-
fact_find_brain
|
| 382 |
-
one turn, fact_find_brain gets one more bite via BRAIN_CHAIN.
|
| 383 |
"""
|
| 384 |
def __init__(self, chain: list[str], api_key: Optional[str] = None,
|
| 385 |
timeout: float = 30.0, per_model_attempts: int = 1,
|
|
@@ -389,15 +343,13 @@ class NimChainLLM(LLMProvider):
|
|
| 389 |
self.chain = chain
|
| 390 |
self.api_key = api_key or getattr(settings, "NVIDIA_NIM_API_KEY", "")
|
| 391 |
self.timeout = timeout
|
| 392 |
-
#
|
| 393 |
-
# fallbacks at 30s each
|
| 394 |
-
#
|
| 395 |
-
#
|
| 396 |
-
#
|
| 397 |
-
#
|
| 398 |
-
#
|
| 399 |
-
# budget is rarely the binding constraint anymore. Kept for the
|
| 400 |
-
# final filter_chain-refresh fallback path + cold-start edge cases.
|
| 401 |
self.total_budget_s = total_budget_s if total_budget_s is not None else max(timeout * 2.5, 30.0)
|
| 402 |
self.per_model_attempts = per_model_attempts
|
| 403 |
self.role = role # 'brain' | 'unknown' — flows into usage log
|
|
@@ -424,17 +376,15 @@ class NimChainLLM(LLMProvider):
|
|
| 424 |
def _get_worker_for(self, model_id: str, timeout: float) -> LLMProvider:
|
| 425 |
"""Dispatch a chain entry to the NIM provider client.
|
| 426 |
|
| 427 |
-
|
| 428 |
-
|
| 429 |
-
|
| 430 |
-
|
| 431 |
-
|
| 432 |
-
|
| 433 |
-
|
| 434 |
-
|
| 435 |
-
|
| 436 |
-
trackers in the provider clients route their response-header
|
| 437 |
-
signals to the right chain state.
|
| 438 |
"""
|
| 439 |
if model_id.startswith(("openrouter:", "groq:", "or:")):
|
| 440 |
raise RuntimeError(
|
|
@@ -454,17 +404,16 @@ class NimChainLLM(LLMProvider):
|
|
| 454 |
'deepseek', 'moonshot', 'minimax', 'nvidia', 'nemotron', 'llama3',
|
| 455 |
'google', 'unknown'.
|
| 456 |
|
| 457 |
-
|
| 458 |
-
|
| 459 |
-
|
| 460 |
-
|
| 461 |
"""
|
| 462 |
m = (model_id or "").lower()
|
| 463 |
-
#
|
| 464 |
-
#
|
| 465 |
-
# the "google" bucket
|
| 466 |
-
#
|
| 467 |
-
# invariant couldn't exclude Gemini brains from a Gemini judge.
|
| 468 |
if "gemini" in m or m.startswith("google/"):
|
| 469 |
return "google"
|
| 470 |
# Strip provider prefix first so 'groq:llama-3.3-70b' → 'meta' (it IS Meta Llama)
|
|
@@ -534,8 +483,8 @@ class NimChainLLM(LLMProvider):
|
|
| 534 |
exclude_models: Optional[list[str]] = None,
|
| 535 |
exclude_families: Optional[list[str]] = None,
|
| 536 |
) -> LLMResult:
|
| 537 |
-
"""
|
| 538 |
-
|
| 539 |
primary fails. The chain list is the candidate POOL for election —
|
| 540 |
NOT a per-call sequence.
|
| 541 |
|
|
@@ -546,8 +495,8 @@ class NimChainLLM(LLMProvider):
|
|
| 546 |
the chain in order.
|
| 547 |
Final fallback (both elected models fail): trigger a synchronous
|
| 548 |
probe refresh + try whatever filter_chain now offers, walking the
|
| 549 |
-
chain in order. This path is the
|
| 550 |
-
|
| 551 |
"""
|
| 552 |
from backend import llm_health
|
| 553 |
|
|
@@ -650,8 +599,8 @@ class NimChainLLM(LLMProvider):
|
|
| 650 |
llm_health.report_success(self._chain_name, model, latency_ms)
|
| 651 |
except Exception:
|
| 652 |
pass
|
| 653 |
-
#
|
| 654 |
-
#
|
| 655 |
# candidate (i.e. anything without an openrouter:/groq: prefix).
|
| 656 |
# Groq + OpenRouter stamp credits from response headers inside
|
| 657 |
# their own .chat() methods.
|
|
@@ -680,7 +629,7 @@ class NimChainLLM(LLMProvider):
|
|
| 680 |
if res is not None:
|
| 681 |
return res
|
| 682 |
|
| 683 |
-
# --- Backup attempt (
|
| 684 |
if elected_backup and elected_backup != elected_primary:
|
| 685 |
res = await _try(elected_backup)
|
| 686 |
if res is not None:
|
|
@@ -688,9 +637,9 @@ class NimChainLLM(LLMProvider):
|
|
| 688 |
|
| 689 |
# --- Both elected failed — final safety net --------------------------
|
| 690 |
# Trigger ONE synchronous probe refresh; whatever the refreshed
|
| 691 |
-
# filter_chain offers, walk it in order and try anything
|
| 692 |
-
# touched this turn.
|
| 693 |
-
#
|
| 694 |
try:
|
| 695 |
# KI-099 — bound the probe-refresh cost. On a hot user-facing turn, a
|
| 696 |
# 60-80s probe walk through ~10 candidates is unacceptable. If we've
|
|
@@ -793,19 +742,15 @@ class NimChainLLM(LLMProvider):
|
|
| 793 |
) from last_err
|
| 794 |
|
| 795 |
|
| 796 |
-
#
|
| 797 |
-
#
|
| 798 |
-
#
|
| 799 |
-
#
|
| 800 |
-
# backup is chosen with explicit cross-provider preference — both signals
|
| 801 |
-
# the rotation was approximating heuristically. Kept around (not deleted)
|
| 802 |
-
# because the regression suite still pins its statistical behaviour. The
|
| 803 |
-
# get_*_llm factories no longer call it.
|
| 804 |
import random as _random
|
| 805 |
|
| 806 |
|
| 807 |
def _balanced_brain_chain(base: list[str], *, groq_first_probability: float = 0.5) -> list[str]:
|
| 808 |
-
"""
|
| 809 |
|
| 810 |
With `groq_first_probability` (default 50%), hoist the Groq Llama entry
|
| 811 |
to the head of the chain so it serves as the primary instead of the
|
|
@@ -813,17 +758,14 @@ def _balanced_brain_chain(base: list[str], *, groq_first_probability: float = 0.
|
|
| 813 |
fallback order — Groq calls that fail (rare; LPU is very reliable)
|
| 814 |
still get the full NIM fallback chain.
|
| 815 |
|
| 816 |
-
|
| 817 |
-
picks
|
| 818 |
-
|
| 819 |
-
|
| 820 |
-
|
| 821 |
-
|
| 822 |
-
|
| 823 |
-
|
| 824 |
-
Uses per-call `random.random()` so concurrent async workers (the
|
| 825 |
-
100-persona audit's 4 workers, the parallel 96-Q eval's 6 workers) each
|
| 826 |
-
flip independently — no shared mutable cycle state, no GIL races, fair
|
| 827 |
distribution in aggregate. `groq_first_probability` is overrideable
|
| 828 |
primarily for testing."""
|
| 829 |
if _random.random() >= groq_first_probability:
|
|
@@ -837,18 +779,16 @@ def _balanced_brain_chain(base: list[str], *, groq_first_probability: float = 0.
|
|
| 837 |
|
| 838 |
|
| 839 |
def assert_family_coverage() -> dict[str, str]:
|
| 840 |
-
"""
|
| 841 |
-
|
| 842 |
-
|
| 843 |
|
| 844 |
-
|
| 845 |
-
|
| 846 |
|
| 847 |
-
|
| 848 |
-
|
| 849 |
-
|
| 850 |
-
Call from admin diagnostics or tests; not invoked on import to keep
|
| 851 |
-
cold-start cheap. Safe to run from a probe loop tick.
|
| 852 |
"""
|
| 853 |
coverage: dict[str, str] = {}
|
| 854 |
unknown: list[str] = []
|
|
@@ -877,25 +817,19 @@ def assert_family_coverage() -> dict[str, str]:
|
|
| 877 |
|
| 878 |
|
| 879 |
def get_brain_llm() -> NimChainLLM:
|
| 880 |
-
"""Heavy brain —
|
| 881 |
-
|
| 882 |
-
Pre-KI-080: per-call _balanced_brain_chain rotation between NIM Qwen
|
| 883 |
-
and Groq Llama (KI-025 50/50 heuristic) → 5-6 LLM calls per turn
|
| 884 |
-
under degraded conditions.
|
| 885 |
|
| 886 |
-
|
| 887 |
-
|
| 888 |
-
|
| 889 |
-
|
| 890 |
-
|
| 891 |
|
| 892 |
-
|
| 893 |
-
|
| 894 |
"""
|
| 895 |
return NimChainLLM(chain=BRAIN_CHAIN, timeout=20.0,
|
| 896 |
role="brain", total_budget_s=35.0)
|
| 897 |
|
| 898 |
|
| 899 |
-
#
|
| 900 |
-
# factories removed. Any straggler caller should import get_brain_llm()
|
| 901 |
-
# directly. The FAST_BRAIN_CHAIN + JUDGE_CHAIN constants are also gone.
|
|
|
|
| 5 |
|
| 6 |
Why NIM:
|
| 7 |
- Frontier open-weights models hosted free (no card, no daily cap, 40 req/min)
|
| 8 |
+
- Single provider for the reasoning stack
|
| 9 |
- Same Bearer-auth + OpenAI request shape — drop-in retry/backoff
|
| 10 |
|
| 11 |
+
Role:
|
| 12 |
- Brain (every reasoning turn): one elected candidate from BRAIN_CHAIN.
|
| 13 |
The chain is ordered Qwen 3-Next → Mistral Large 3 → Llama-4 Maverick
|
| 14 |
+
→ Nemotron-Super (last resort). Election picks the actually
|
| 15 |
healthy/fastest candidate per turn from background probe data.
|
| 16 |
|
| 17 |
+
BRAIN_CHAIN is the only chain: voice fact-find reuses it with its own
|
| 18 |
+
budget, and faithfulness is enforced by in-process checks. Sarvam stays
|
| 19 |
+
for voice STT/TTS + Hindi/Hinglish/vernacular translation.
|
|
|
|
|
|
|
| 20 |
"""
|
| 21 |
|
| 22 |
from __future__ import annotations
|
|
|
|
| 72 |
pass
|
| 73 |
|
| 74 |
|
| 75 |
+
# Global outbound NIM concurrency cap.
|
| 76 |
#
|
| 77 |
+
# The binding constraint is NIM's free-tier per-key concurrency (~3-5
|
| 78 |
+
# slots). Multiple sources of NIM traffic stack on the same key with no
|
| 79 |
+
# global throttle: the probe loop, admin tab polling, per-user chat turns
|
| 80 |
+
# (1-2 calls), concurrent users (N × 1-2), and retry backoff.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 81 |
#
|
| 82 |
# 6+ in-flight → queue → 15-25s response times → bot's 12s outer cap
|
| 83 |
# fires → user sees fallback.
|
|
|
|
| 91 |
_NIM_OUTBOUND_SEMAPHORE = asyncio.Semaphore(2)
|
| 92 |
|
| 93 |
|
| 94 |
+
# Module-level httpx.AsyncClient singleton.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 95 |
#
|
| 96 |
+
# A single shared client (rather than a fresh pool per chat() call) avoids
|
| 97 |
+
# churning TCP connections / dropping TLS sessions / leaking pool slots
|
| 98 |
+
# under concurrency. Explicit Limits:
|
| 99 |
# - max_connections=10 caps total outbound (NIM + probes share)
|
| 100 |
# - max_keepalive_connections=5 keeps a warm pool for the hot path
|
| 101 |
# The client is constructed lazily (first .chat() call) so cold imports
|
|
|
|
| 129 |
|
| 130 |
|
| 131 |
NVIDIA_NIM_BASE_URL = "https://integrate.api.nvidia.com/v1"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 132 |
# Heavy brain (complex queries — comparison, recommendation, synthesis):
|
| 133 |
# Qwen 3-Next 80B — 80B / 3B active MoE, frontier multilingual, very fast.
|
| 134 |
NIM_BRAIN_MODEL = "qwen/qwen3-next-80b-a3b-instruct"
|
|
|
|
| 137 |
# different intents to the same pool. Tiered routing kept for forward-compat.
|
| 138 |
NIM_FAST_BRAIN_MODEL = "qwen/qwen3-next-80b-a3b-instruct"
|
| 139 |
# Judge: Mistral Large 3 — 675B dense, MIT license. Different family from
|
| 140 |
+
# the Qwen brain (Mistral vs Alibaba) so the judge sees the brain's output
|
| 141 |
+
# from a genuinely different decision surface ("the brain doesn't mark its
|
| 142 |
+
# own homework").
|
| 143 |
NIM_JUDGE_MODEL = "mistralai/mistral-large-3-675b-instruct-2512"
|
| 144 |
|
| 145 |
|
|
|
|
| 177 |
"max_tokens": max_tokens,
|
| 178 |
}
|
| 179 |
if response_format:
|
| 180 |
+
# NIM forwards OpenAI-shape response_format to the upstream
|
| 181 |
+
# model. For models that support `nvext.guided_json`
|
| 182 |
# (NIM's stricter constrained-decoding surface), surface that
|
| 183 |
# too when the caller passes a JSON schema (response_format
|
| 184 |
# with `json_schema`). The plain `{"type": "json_object"}` flag
|
|
|
|
| 208 |
write=2.0,
|
| 209 |
pool=2.0,
|
| 210 |
)
|
| 211 |
+
# No inner retry loop. 429/5xx failover is handled by NimChainLLM:
|
| 212 |
+
# (1) the exception is caught in _try(),
|
| 213 |
+
# (2) report_failure() demotes the slot-blocked candidate for
|
| 214 |
+
# ~30s, and
|
| 215 |
+
# (3) the elected backup (cross-provider by preference) is tried
|
| 216 |
+
# within the same turn.
|
| 217 |
+
# If both elected models fail, BRAIN_CHAIN provides one more bite.
|
| 218 |
+
# Per-call retries inside this method would only re-queue against
|
| 219 |
+
# the same slot-starved pool, multiplying the queueing the
|
| 220 |
+
# semaphore prevents.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 221 |
#
|
| 222 |
# The semaphore wraps ONLY the HTTP round-trip — not response
|
| 223 |
# parsing or usage logging — so we cap concurrent network
|
| 224 |
# traffic without serialising the rest of the pipeline.
|
| 225 |
+
# Use the module-level shared httpx.AsyncClient (singleton with a
|
| 226 |
+
# bounded connection pool). Per-request timeout is applied at
|
|
|
|
| 227 |
# .post() call time so the shared pool serves both long-timeout
|
| 228 |
# chat() and short-timeout probe() calls without contention.
|
| 229 |
client = _get_shared_nim_client()
|
|
|
|
| 267 |
# cross-family-grading invariant survives any failover.
|
| 268 |
|
| 269 |
BRAIN_CHAIN = [
|
| 270 |
+
# NIM-only: the chain contains only NIM-hosted candidates whose `<FF>`
|
| 271 |
+
# trailer adherence is verified or structurally safer (different
|
| 272 |
+
# family / smaller routing surface). Nemotron is LAST RESORT — it is
|
| 273 |
+
# the weakest practical NIM model and only serves when every
|
| 274 |
+
# higher-quality model is simultaneously down.
|
| 275 |
+
# Primary: Qwen 3-Next 80B — clean JSON, multilingual.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 276 |
"qwen/qwen3-next-80b-a3b-instruct",
|
| 277 |
+
# 1st fallback: Mistral Large 3 675B — different family (mistral).
|
| 278 |
+
# Strongest fallback.
|
| 279 |
"mistralai/mistral-large-3-675b-instruct-2512",
|
| 280 |
+
# 2nd fallback: Meta Llama-4 Maverick 17B — different family (meta),
|
| 281 |
+
# keeps the chain alive through a Qwen+Mistral outage.
|
| 282 |
"meta/llama-4-maverick-17b-128e-instruct",
|
| 283 |
+
# 3rd / LAST RESORT: NVIDIA Nemotron-Super 49B — only reached when
|
| 284 |
+
# Qwen + Mistral + Maverick are ALL down simultaneously. Kept in the
|
| 285 |
+
# pool for fail-loud > down.
|
| 286 |
"nvidia/llama-3.3-nemotron-super-49b-v1.5",
|
| 287 |
]
|
| 288 |
|
| 289 |
+
# BRAIN_CHAIN is the only chain — every reasoning role resolves to it.
|
| 290 |
+
# Voice fact-find reuses get_brain_llm() with a tighter outer wait_for
|
| 291 |
+
# cap; faithfulness is enforced by in-process checks.
|
|
|
|
| 292 |
|
| 293 |
|
| 294 |
def _classify_error(e: BaseException) -> str:
|
| 295 |
+
"""Classify a chat exception into a stable string passed to
|
| 296 |
llm_health.report_failure(). The string drives degradation duration:
|
| 297 |
rate-limit failures (HTTP 429) get a 1h sin-bin; everything else 30s.
|
| 298 |
|
|
|
|
| 302 |
carries `.response.status_code == 429` we tag it `"Status429"`; for
|
| 303 |
other HTTP statuses we surface e.g. `"HTTPStatusError:503"`; for
|
| 304 |
non-HTTP exceptions we keep the class name (`TimeoutException`,
|
| 305 |
+
`ReadTimeout`, etc.).
|
| 306 |
"""
|
| 307 |
cls = type(e).__name__
|
| 308 |
try:
|
|
|
|
| 318 |
|
| 319 |
|
| 320 |
class NimChainLLM(LLMProvider):
|
| 321 |
+
"""Sticky-primary router across multiple candidate models.
|
| 322 |
+
|
| 323 |
+
A background probe loop ELECTS a primary + backup per chain based on
|
| 324 |
+
real probe latencies. chat() calls the elected primary ONCE per turn
|
| 325 |
+
(1 LLM call). On failure, it demotes the primary for ~30s and falls
|
| 326 |
+
through to the elected backup (2 LLM calls max). The chain list is
|
| 327 |
+
the CANDIDATE POOL for election, not a per-call sequence.
|
| 328 |
+
|
| 329 |
+
Worst case per turn: 2 LLM calls. A final filter_chain refresh path
|
| 330 |
+
handles the rare double-failure edge case so the call still
|
| 331 |
+
degrades gracefully.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 332 |
|
| 333 |
`name` after a successful call reflects which model actually answered
|
| 334 |
+
so downstream callers (brain_used tag, eval logs) can audit which
|
| 335 |
+
candidate produced the output. If primary+backup both fail inside one
|
| 336 |
+
turn, fact_find_brain gets one more bite via BRAIN_CHAIN.
|
|
|
|
| 337 |
"""
|
| 338 |
def __init__(self, chain: list[str], api_key: Optional[str] = None,
|
| 339 |
timeout: float = 30.0, per_model_attempts: int = 1,
|
|
|
|
| 343 |
self.chain = chain
|
| 344 |
self.api_key = api_key or getattr(settings, "NVIDIA_NIM_API_KEY", "")
|
| 345 |
self.timeout = timeout
|
| 346 |
+
# Cumulative chain budget. Without a ceiling, a chain of N
|
| 347 |
+
# fallbacks at 30s each could run for minutes per turn. Default
|
| 348 |
+
# to ~2.5× the per-link timeout so a healthy primary always
|
| 349 |
+
# completes but a cascading-failure chain bails fast. With
|
| 350 |
+
# election doing only 1-2 LLM calls per turn the budget is
|
| 351 |
+
# rarely binding; it still guards the final filter_chain-refresh
|
| 352 |
+
# fallback path + cold-start edge cases.
|
|
|
|
|
|
|
| 353 |
self.total_budget_s = total_budget_s if total_budget_s is not None else max(timeout * 2.5, 30.0)
|
| 354 |
self.per_model_attempts = per_model_attempts
|
| 355 |
self.role = role # 'brain' | 'unknown' — flows into usage log
|
|
|
|
| 376 |
def _get_worker_for(self, model_id: str, timeout: float) -> LLMProvider:
|
| 377 |
"""Dispatch a chain entry to the NIM provider client.
|
| 378 |
|
| 379 |
+
NIM-only: cross-provider (`openrouter:` / `groq:`) prefixes are
|
| 380 |
+
explicitly rejected here. This is a defense-in-depth short-circuit
|
| 381 |
+
— if anyone (admin override, monkeypatch, future drift) injects a
|
| 382 |
+
non-NIM candidate into a chain, the dispatcher raises rather than
|
| 383 |
+
routing to a provider outside the validated pool.
|
| 384 |
+
|
| 385 |
+
Passes `chain_name=self._chain_name` so the credit trackers in the
|
| 386 |
+
provider clients route their response-header signals to the right
|
| 387 |
+
chain state.
|
|
|
|
|
|
|
| 388 |
"""
|
| 389 |
if model_id.startswith(("openrouter:", "groq:", "or:")):
|
| 390 |
raise RuntimeError(
|
|
|
|
| 404 |
'deepseek', 'moonshot', 'minimax', 'nvidia', 'nemotron', 'llama3',
|
| 405 |
'google', 'unknown'.
|
| 406 |
|
| 407 |
+
EVERY model in BRAIN_CHAIN must map to a known family. The
|
| 408 |
+
`assert_family_coverage()` helper walks the chain on demand (from
|
| 409 |
+
tests / admin / probe loop) and raises if any candidate falls
|
| 410 |
+
through to "unknown".
|
| 411 |
"""
|
| 412 |
m = (model_id or "").lower()
|
| 413 |
+
# Recognise Google's Gemini family BEFORE prefix-stripping so ids
|
| 414 |
+
# like "google/gemini-2.5-flash" and bare "gemini-..." both land
|
| 415 |
+
# in the "google" bucket, so the judge-vs-brain cross-family
|
| 416 |
+
# invariant can exclude a Gemini brain from a Gemini judge.
|
|
|
|
| 417 |
if "gemini" in m or m.startswith("google/"):
|
| 418 |
return "google"
|
| 419 |
# Strip provider prefix first so 'groq:llama-3.3-70b' → 'meta' (it IS Meta Llama)
|
|
|
|
| 483 |
exclude_models: Optional[list[str]] = None,
|
| 484 |
exclude_families: Optional[list[str]] = None,
|
| 485 |
) -> LLMResult:
|
| 486 |
+
"""Sticky primary election. Call ONE elected candidate per turn,
|
| 487 |
+
with at most ONE real-time fallback to the elected backup if
|
| 488 |
primary fails. The chain list is the candidate POOL for election —
|
| 489 |
NOT a per-call sequence.
|
| 490 |
|
|
|
|
| 495 |
the chain in order.
|
| 496 |
Final fallback (both elected models fail): trigger a synchronous
|
| 497 |
probe refresh + try whatever filter_chain now offers, walking the
|
| 498 |
+
chain in order. This path is the graceful-degradation safety net
|
| 499 |
+
for the double-failure edge case.
|
| 500 |
"""
|
| 501 |
from backend import llm_health
|
| 502 |
|
|
|
|
| 599 |
llm_health.report_success(self._chain_name, model, latency_ms)
|
| 600 |
except Exception:
|
| 601 |
pass
|
| 602 |
+
# NIM has no clean rate-limit header, so we maintain a local
|
| 603 |
+
# 60s rate-meter for every NIM-prefixed
|
| 604 |
# candidate (i.e. anything without an openrouter:/groq: prefix).
|
| 605 |
# Groq + OpenRouter stamp credits from response headers inside
|
| 606 |
# their own .chat() methods.
|
|
|
|
| 629 |
if res is not None:
|
| 630 |
return res
|
| 631 |
|
| 632 |
+
# --- Backup attempt (single real-time fallback) ----------------------
|
| 633 |
if elected_backup and elected_backup != elected_primary:
|
| 634 |
res = await _try(elected_backup)
|
| 635 |
if res is not None:
|
|
|
|
| 637 |
|
| 638 |
# --- Both elected failed — final safety net --------------------------
|
| 639 |
# Trigger ONE synchronous probe refresh; whatever the refreshed
|
| 640 |
+
# filter_chain offers, walk it in order and try anything not yet
|
| 641 |
+
# touched this turn. Graceful-degradation path for the (rare)
|
| 642 |
+
# double-failure case.
|
| 643 |
try:
|
| 644 |
# KI-099 — bound the probe-refresh cost. On a hot user-facing turn, a
|
| 645 |
# 60-80s probe walk through ~10 candidates is unacceptable. If we've
|
|
|
|
| 742 |
) from last_err
|
| 743 |
|
| 744 |
|
| 745 |
+
# Provider load-balancing pure-function. Not wired into the chat hot path
|
| 746 |
+
# (the probe-driven elector in backend.llm_health selects candidates);
|
| 747 |
+
# retained because the routing regression suite pins its statistical
|
| 748 |
+
# behaviour and it is a useful pure-function for ops / sim / overrides.
|
|
|
|
|
|
|
|
|
|
|
|
|
| 749 |
import random as _random
|
| 750 |
|
| 751 |
|
| 752 |
def _balanced_brain_chain(base: list[str], *, groq_first_probability: float = 0.5) -> list[str]:
|
| 753 |
+
"""Provider load-balancing pure-function.
|
| 754 |
|
| 755 |
With `groq_first_probability` (default 50%), hoist the Groq Llama entry
|
| 756 |
to the head of the chain so it serves as the primary instead of the
|
|
|
|
| 758 |
fallback order — Groq calls that fail (rare; LPU is very reliable)
|
| 759 |
still get the full NIM fallback chain.
|
| 760 |
|
| 761 |
+
Not wired into the chat hot path (the elector in `backend.llm_health`
|
| 762 |
+
picks candidates dynamically from background probe data). Retained
|
| 763 |
+
because the routing regression suite pins its statistical behaviour
|
| 764 |
+
and it is a useful pure-function for ops / sim / overrides (e.g.
|
| 765 |
+
forcing a non-elected order in a debug script).
|
| 766 |
+
|
| 767 |
+
Uses per-call `random.random()` so concurrent async workers each flip
|
| 768 |
+
independently — no shared mutable cycle state, no GIL races, fair
|
|
|
|
|
|
|
|
|
|
| 769 |
distribution in aggregate. `groq_first_probability` is overrideable
|
| 770 |
primarily for testing."""
|
| 771 |
if _random.random() >= groq_first_probability:
|
|
|
|
| 779 |
|
| 780 |
|
| 781 |
def assert_family_coverage() -> dict[str, str]:
|
| 782 |
+
"""Verify every model in BRAIN_CHAIN maps to a known family (i.e. NOT
|
| 783 |
+
'unknown'). Returns a {model: family} dict on success; raises
|
| 784 |
+
RuntimeError listing any 'unknown'-mapped models on failure.
|
| 785 |
|
| 786 |
+
Also verifies nemotron is the TAIL (last entry) of BRAIN_CHAIN so the
|
| 787 |
+
last-resort ordering invariant survives any chain edit.
|
| 788 |
|
| 789 |
+
Walks BRAIN_CHAIN (the only chain). Call from admin diagnostics or
|
| 790 |
+
tests; not invoked on import to keep cold-start cheap. Safe to run
|
| 791 |
+
from a probe loop tick.
|
|
|
|
|
|
|
| 792 |
"""
|
| 793 |
coverage: dict[str, str] = {}
|
| 794 |
unknown: list[str] = []
|
|
|
|
| 817 |
|
| 818 |
|
| 819 |
def get_brain_llm() -> NimChainLLM:
|
| 820 |
+
"""Heavy brain — sticky-primary election over BRAIN_CHAIN.
|
|
|
|
|
|
|
|
|
|
|
|
|
| 821 |
|
| 822 |
+
The background probe loop in backend.llm_health elects the
|
| 823 |
+
actually-fastest candidate dynamically (probe-driven) plus a backup.
|
| 824 |
+
chat() calls 1-2 candidates max per turn. (The _balanced_brain_chain
|
| 825 |
+
rotation pure-function is available for overrides but not wired in
|
| 826 |
+
here.)
|
| 827 |
|
| 828 |
+
Per-link 20s timeout, total chain budget 35s (only binding for the
|
| 829 |
+
final filter_chain-refresh safety net).
|
| 830 |
"""
|
| 831 |
return NimChainLLM(chain=BRAIN_CHAIN, timeout=20.0,
|
| 832 |
role="brain", total_budget_s=35.0)
|
| 833 |
|
| 834 |
|
| 835 |
+
# get_brain_llm() is the brain factory; BRAIN_CHAIN is the only chain.
|
|
|
|
|
|
|
@@ -40,7 +40,7 @@ OPENROUTER_URL = "https://openrouter.ai/api/v1/chat/completions"
|
|
| 40 |
DEFAULT_MODEL = "openai/gpt-oss-120b"
|
| 41 |
|
| 42 |
|
| 43 |
-
#
|
| 44 |
#
|
| 45 |
# `:free` suffix is OpenRouter's documented marker for zero-cost models.
|
| 46 |
# We use this both to:
|
|
@@ -61,8 +61,8 @@ def is_free_model(model_id: str) -> bool:
|
|
| 61 |
|
| 62 |
|
| 63 |
def order_free_pool(models: list[str]) -> list[str]:
|
| 64 |
-
"""
|
| 65 |
-
|
| 66 |
|
| 67 |
Sort key: (length-of-id ascending, id ascending). Shorter ids tend to
|
| 68 |
correlate with smaller / cheaper models on OpenRouter's catalogue
|
|
@@ -77,7 +77,7 @@ def order_free_pool(models: list[str]) -> list[str]:
|
|
| 77 |
|
| 78 |
|
| 79 |
def enforce_free_pool(models: list[str]) -> list[str]:
|
| 80 |
-
"""
|
| 81 |
|
| 82 |
Raises ValueError listing any non-`:free` entries. Returns the
|
| 83 |
deterministically-ordered free pool on success.
|
|
@@ -131,7 +131,7 @@ class OpenRouterLLM(LLMProvider):
|
|
| 131 |
routing; if `models` is omitted the lone `model` field is used).
|
| 132 |
Reference: https://openrouter.ai/docs/features/model-routing
|
| 133 |
|
| 134 |
-
|
| 135 |
- rejects any non-`:free` model in `models` (raises ValueError)
|
| 136 |
- reorders the survivors via `order_free_pool` for stable, smaller-
|
| 137 |
first preference. The single-model `self.model` is checked too
|
|
@@ -196,8 +196,8 @@ class OpenRouterLLM(LLMProvider):
|
|
| 196 |
continue
|
| 197 |
resp.raise_for_status()
|
| 198 |
break
|
| 199 |
-
#
|
| 200 |
-
#
|
| 201 |
# comes from poll_openrouter_credits() every 10 min; this is a
|
| 202 |
# finer-grained per-call backup that catches per-minute bucket
|
| 203 |
# exhaustion before the next poll tick.
|
|
@@ -232,16 +232,9 @@ class OpenRouterLLM(LLMProvider):
|
|
| 232 |
# a sane single-model default; callers that want server-side fallback should
|
| 233 |
# pass `models=[...]` to `chat()`.
|
| 234 |
#
|
| 235 |
-
#
|
| 236 |
-
# (
|
| 237 |
-
#
|
| 238 |
-
# hermes-3-llama-3.1-405b:free) were checked:
|
| 239 |
-
# - gemini-2.0-flash-exp:free — NOT in catalog
|
| 240 |
-
# - llama-3.3-70b-instruct:free — PRESENT but no response_format
|
| 241 |
-
# - hermes-3-llama-3.1-405b:free — PRESENT but no response_format
|
| 242 |
-
# So those would silently lose JSON mode. The IDs below are the actual
|
| 243 |
-
# free-tier set that declare `response_format` (and most also
|
| 244 |
-
# `structured_outputs`) in their `supported_parameters`:
|
| 245 |
# - nvidia/nemotron-3-super-120b-a12b:free (120B, structured_outputs)
|
| 246 |
# - qwen/qwen3-next-80b-a3b-instruct:free (80B, structured_outputs)
|
| 247 |
# - google/gemma-4-31b-it:free (31B, response_format)
|
|
@@ -259,7 +252,7 @@ def get_openrouter_llm(
|
|
| 259 |
model: str = DEFAULT_FREE_BRAIN_MODEL,
|
| 260 |
timeout: float = 60.0,
|
| 261 |
) -> "OpenRouterLLM":
|
| 262 |
-
"""Return a fresh OpenRouterLLM
|
| 263 |
|
| 264 |
The returned client is single-use-shaped (one chat() call per turn) and
|
| 265 |
intended to be passed `models=[primary, fallback1, fallback2]` so the
|
|
|
|
| 40 |
DEFAULT_MODEL = "openai/gpt-oss-120b"
|
| 41 |
|
| 42 |
|
| 43 |
+
# OpenRouter free-pool helpers.
|
| 44 |
#
|
| 45 |
# `:free` suffix is OpenRouter's documented marker for zero-cost models.
|
| 46 |
# We use this both to:
|
|
|
|
| 61 |
|
| 62 |
|
| 63 |
def order_free_pool(models: list[str]) -> list[str]:
|
| 64 |
+
"""Return `models` in a STABLE, deterministic order suitable for
|
| 65 |
+
OpenRouter's `models=[...]` server-side fallback list.
|
| 66 |
|
| 67 |
Sort key: (length-of-id ascending, id ascending). Shorter ids tend to
|
| 68 |
correlate with smaller / cheaper models on OpenRouter's catalogue
|
|
|
|
| 77 |
|
| 78 |
|
| 79 |
def enforce_free_pool(models: list[str]) -> list[str]:
|
| 80 |
+
"""Cost guard: reject paid models in a free-pool call.
|
| 81 |
|
| 82 |
Raises ValueError listing any non-`:free` entries. Returns the
|
| 83 |
deterministically-ordered free pool on success.
|
|
|
|
| 131 |
routing; if `models` is omitted the lone `model` field is used).
|
| 132 |
Reference: https://openrouter.ai/docs/features/model-routing
|
| 133 |
|
| 134 |
+
`free_only=True` activates the cost guard:
|
| 135 |
- rejects any non-`:free` model in `models` (raises ValueError)
|
| 136 |
- reorders the survivors via `order_free_pool` for stable, smaller-
|
| 137 |
first preference. The single-model `self.model` is checked too
|
|
|
|
| 196 |
continue
|
| 197 |
resp.raise_for_status()
|
| 198 |
break
|
| 199 |
+
# Opportunistic between-poll signal from OpenRouter response
|
| 200 |
+
# headers. The authoritative truth (usd_balance)
|
| 201 |
# comes from poll_openrouter_credits() every 10 min; this is a
|
| 202 |
# finer-grained per-call backup that catches per-minute bucket
|
| 203 |
# exhaustion before the next poll tick.
|
|
|
|
| 232 |
# a sane single-model default; callers that want server-side fallback should
|
| 233 |
# pass `models=[...]` to `chat()`.
|
| 234 |
#
|
| 235 |
+
# The free-tier model IDs below are the set that declare `response_format`
|
| 236 |
+
# (and most also `structured_outputs`) in their `supported_parameters`,
|
| 237 |
+
# so JSON mode is preserved:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 238 |
# - nvidia/nemotron-3-super-120b-a12b:free (120B, structured_outputs)
|
| 239 |
# - qwen/qwen3-next-80b-a3b-instruct:free (80B, structured_outputs)
|
| 240 |
# - google/gemma-4-31b-it:free (31B, response_format)
|
|
|
|
| 252 |
model: str = DEFAULT_FREE_BRAIN_MODEL,
|
| 253 |
timeout: float = 60.0,
|
| 254 |
) -> "OpenRouterLLM":
|
| 255 |
+
"""Return a fresh OpenRouterLLM client.
|
| 256 |
|
| 257 |
The returned client is single-use-shaped (one chat() call per turn) and
|
| 258 |
intended to be passed `models=[primary, fallback1, fallback2]` so the
|
|
@@ -29,8 +29,8 @@ from backend.providers.base import TTSProvider
|
|
| 29 |
|
| 30 |
logger = logging.getLogger(__name__)
|
| 31 |
|
| 32 |
-
#
|
| 33 |
-
#
|
| 34 |
# decodes natively. Sarvam Bulbul's text-to-speech API itself does NOT support
|
| 35 |
# codec negotiation in the request body (response is always base64 WAV), so we
|
| 36 |
# transcode locally with pydub (already in requirements.txt; the Dockerfile
|
|
@@ -48,14 +48,11 @@ except Exception: # pragma: no cover — pydub missing in dev shell
|
|
| 48 |
_SUPPORTED_CODECS = {"audio/wav", "audio/mp4", "audio/webm"}
|
| 49 |
|
| 50 |
|
| 51 |
-
#
|
| 52 |
-
#
|
| 53 |
-
#
|
| 54 |
-
#
|
| 55 |
-
#
|
| 56 |
-
# HTTP 429 {"code":"insufficient_quota_error"} → resp.raise_for_status()
|
| 57 |
-
# raised → chat endpoint logged it to turns.jsonl and returned a text-only
|
| 58 |
-
# reply with NO voice and NO explanation ("no voice in reply. wtf?").
|
| 59 |
TTS_ERROR_RATE_LIMIT = "rate_limit"
|
| 60 |
TTS_ERROR_SERVICE = "service_unavailable"
|
| 61 |
TTS_ERROR_NETWORK = "network"
|
|
|
|
| 29 |
|
| 30 |
logger = logging.getLogger(__name__)
|
| 31 |
|
| 32 |
+
# The frontend sends an `X-Preferred-Codec: audio/{wav,mp4,webm}` header
|
| 33 |
+
# so the chat endpoint can return audio in the codec the user's browser
|
| 34 |
# decodes natively. Sarvam Bulbul's text-to-speech API itself does NOT support
|
| 35 |
# codec negotiation in the request body (response is always base64 WAV), so we
|
| 36 |
# transcode locally with pydub (already in requirements.txt; the Dockerfile
|
|
|
|
| 48 |
_SUPPORTED_CODECS = {"audio/wav", "audio/mp4", "audio/webm"}
|
| 49 |
|
| 50 |
|
| 51 |
+
# Voice-OUTPUT error vocabulary. Mirrors the STT closed-enum contract
|
| 52 |
+
# (sarvam_stt.STT_ERROR_*) so the chat endpoint can surface a structured
|
| 53 |
+
# tts_error_code instead of returning audio_base64=None with zero signal
|
| 54 |
+
# to the client (e.g. when Sarvam returns HTTP 429
|
| 55 |
+
# {"code":"insufficient_quota_error"}).
|
|
|
|
|
|
|
|
|
|
| 56 |
TTS_ERROR_RATE_LIMIT = "rate_limit"
|
| 57 |
TTS_ERROR_SERVICE = "service_unavailable"
|
| 58 |
TTS_ERROR_NETWORK = "network"
|
|
@@ -3,8 +3,7 @@
|
|
| 3 |
This module is a SIDECAR to the retrieval path (`rag/retrieve.py` and the
|
| 4 |
single-brain tool layer in `backend/brain_tools.py`). It is deliberately a
|
| 5 |
separate file so retrieval-correctness work stays isolated from the
|
| 6 |
-
turn-handling code.
|
| 7 |
-
`backend/orchestrator.py`, deleted in the 2026-05-15 single-brain rewrite.)
|
| 8 |
|
| 9 |
Public API:
|
| 10 |
apply_profile_filter(chunks, profile) -> list[RetrievedChunk]
|
|
@@ -34,8 +33,8 @@ Public API:
|
|
| 34 |
dedup_by_policy(chunks) -> list[RetrievedChunk]
|
| 35 |
Within a top-K, keep highest-score chunk per policy_id.
|
| 36 |
|
| 37 |
-
All functions are pure / side-effect-free; safe to call from the
|
| 38 |
-
or from rag/retrieve.py without import-cycle risk.
|
| 39 |
"""
|
| 40 |
|
| 41 |
from __future__ import annotations
|
|
@@ -43,7 +42,7 @@ from __future__ import annotations
|
|
| 43 |
import re
|
| 44 |
from typing import Any, Iterable, Optional
|
| 45 |
|
| 46 |
-
#
|
| 47 |
# UIN-primary + product_key invariant the marketplace endpoint uses
|
| 48 |
# (main.py /api/policies/all). Imported (not reimplemented) so the
|
| 49 |
# recommender and the marketplace agree on "the same policy". The module
|
|
@@ -90,7 +89,7 @@ MATERNITY_NAME_RE = re.compile(
|
|
| 90 |
)
|
| 91 |
|
| 92 |
# ---------------------------------------------------------------------------
|
| 93 |
-
#
|
| 94 |
# ---------------------------------------------------------------------------
|
| 95 |
# A plan is a TOP-UP / SUPER-TOP-UP (only useful ALONGSIDE an existing base
|
| 96 |
# policy or above a large out-of-pocket deductible) when ANY of these hold:
|
|
@@ -150,8 +149,7 @@ UIN_RE = re.compile(
|
|
| 150 |
)
|
| 151 |
|
| 152 |
# ---------------------------------------------------------------------------
|
| 153 |
-
#
|
| 154 |
-
# (2026-05-16)
|
| 155 |
# ---------------------------------------------------------------------------
|
| 156 |
# A FIXED-BENEFIT product (hospital daily cash, personal accident, critical
|
| 157 |
# illness, cancer / other defined-benefit) pays a fixed lump-sum / per-day
|
|
@@ -200,13 +198,12 @@ _COMPREHENSIVE_GOAL_TOKENS = (
|
|
| 200 |
)
|
| 201 |
|
| 202 |
# ---------------------------------------------------------------------------
|
| 203 |
-
#
|
| 204 |
# ---------------------------------------------------------------------------
|
| 205 |
-
#
|
| 206 |
-
#
|
| 207 |
-
#
|
| 208 |
-
#
|
| 209 |
-
# grade/rank inversion + the cost-objective lead.
|
| 210 |
|
| 211 |
# Entry-age grace. The catalog's max_entry_age is the regulator-filed entry
|
| 212 |
# age; underwriters allow a small grace band (mirrors PROFILE_AGE_TOLERANCE).
|
|
@@ -348,15 +345,14 @@ def apply_profile_filter(chunks: Iterable[Any], profile: Any) -> list[Any]:
|
|
| 348 |
if not chunks_list:
|
| 349 |
return chunks_list
|
| 350 |
|
| 351 |
-
#
|
| 352 |
-
#
|
| 353 |
-
#
|
| 354 |
-
#
|
| 355 |
-
#
|
| 356 |
-
#
|
| 357 |
-
#
|
| 358 |
-
#
|
| 359 |
-
# age. Falls back to the raw profile age if no resolved signal.
|
| 360 |
_resolved_age = _oldest_insured_age(profile)
|
| 361 |
if _resolved_age is None:
|
| 362 |
_resolved_age = _profile_get(profile, "age")
|
|
@@ -713,13 +709,12 @@ def apply_eligibility_filter(chunks: Iterable[Any], profile: Any) -> list[Any]:
|
|
| 713 |
expenses, so they are a wrong PRIMARY recommendation. They are
|
| 714 |
NEVER dropped for a user who explicitly wants a supplement.
|
| 715 |
|
| 716 |
-
5. ENTRY-AGE GATE
|
| 717 |
-
|
| 718 |
-
|
| 719 |
-
|
| 720 |
-
|
| 721 |
-
|
| 722 |
-
when both the insured age AND the policy fact are present.
|
| 723 |
|
| 724 |
Conservative on missing data: a rule only fires when BOTH the profile
|
| 725 |
signal AND the policy fact are present. A chunk with no enriched facts
|
|
@@ -755,14 +750,14 @@ def apply_eligibility_filter(chunks: Iterable[Any], profile: Any) -> list[Any]:
|
|
| 755 |
if wants_comprehensive and _is_fixed_benefit_chunk(m):
|
| 756 |
continue
|
| 757 |
|
| 758 |
-
# Rule 5
|
| 759 |
-
#
|
| 760 |
-
#
|
| 761 |
-
#
|
| 762 |
-
#
|
| 763 |
-
#
|
| 764 |
-
#
|
| 765 |
-
#
|
| 766 |
if oldest_age is not None:
|
| 767 |
maxe = _as_int(m.get("max_entry_age"))
|
| 768 |
if maxe is not None and maxe + ENTRY_AGE_TOLERANCE < oldest_age:
|
|
@@ -1044,10 +1039,8 @@ def enforce_citation_grounding(chunks: Iterable[Any]) -> list[Any]:
|
|
| 1044 |
INFORMATIONAL only — it is not required for citation grounding because
|
| 1045 |
upstream call sites (e.g. brain_tools.retrieve_policies) build pruned
|
| 1046 |
dicts that intentionally omit it, and the brain cites by policy
|
| 1047 |
-
identity, not by chunk offset.
|
| 1048 |
-
|
| 1049 |
-
offset that the upstream builder never included → every chunk dropped
|
| 1050 |
-
here even though raw retrieval was healthy.
|
| 1051 |
"""
|
| 1052 |
kept: list[Any] = []
|
| 1053 |
for ch in chunks:
|
|
@@ -1071,16 +1064,15 @@ def dedup_by_policy(chunks: Iterable[Any]) -> list[Any]:
|
|
| 1071 |
duplicates to one chunk, keeping the highest-scoring chunk. Preserves
|
| 1072 |
the order of first appearance of each kept product.
|
| 1073 |
|
| 1074 |
-
|
| 1075 |
(policy_identity.canonical_key) — UIN-primary, product_key fallback —
|
| 1076 |
-
NOT the raw policy_id. The
|
| 1077 |
-
|
| 1078 |
-
|
| 1079 |
-
|
| 1080 |
-
|
| 1081 |
-
|
| 1082 |
-
|
| 1083 |
-
marketplace agree on "the same policy".
|
| 1084 |
"""
|
| 1085 |
best: dict[str, Any] = {}
|
| 1086 |
order: list[str] = []
|
|
|
|
| 3 |
This module is a SIDECAR to the retrieval path (`rag/retrieve.py` and the
|
| 4 |
single-brain tool layer in `backend/brain_tools.py`). It is deliberately a
|
| 5 |
separate file so retrieval-correctness work stays isolated from the
|
| 6 |
+
turn-handling code.
|
|
|
|
| 7 |
|
| 8 |
Public API:
|
| 9 |
apply_profile_filter(chunks, profile) -> list[RetrievedChunk]
|
|
|
|
| 33 |
dedup_by_policy(chunks) -> list[RetrievedChunk]
|
| 34 |
Within a top-K, keep highest-score chunk per policy_id.
|
| 35 |
|
| 36 |
+
All functions are pure / side-effect-free; safe to call from the
|
| 37 |
+
single-brain tool layer or from rag/retrieve.py without import-cycle risk.
|
| 38 |
"""
|
| 39 |
|
| 40 |
from __future__ import annotations
|
|
|
|
| 42 |
import re
|
| 43 |
from typing import Any, Iterable, Optional
|
| 44 |
|
| 45 |
+
# Shared canonical-identity / dedup rule. The same
|
| 46 |
# UIN-primary + product_key invariant the marketplace endpoint uses
|
| 47 |
# (main.py /api/policies/all). Imported (not reimplemented) so the
|
| 48 |
# recommender and the marketplace agree on "the same policy". The module
|
|
|
|
| 89 |
)
|
| 90 |
|
| 91 |
# ---------------------------------------------------------------------------
|
| 92 |
+
# Eligibility / profile-fit tunables
|
| 93 |
# ---------------------------------------------------------------------------
|
| 94 |
# A plan is a TOP-UP / SUPER-TOP-UP (only useful ALONGSIDE an existing base
|
| 95 |
# policy or above a large out-of-pocket deductible) when ANY of these hold:
|
|
|
|
| 149 |
)
|
| 150 |
|
| 151 |
# ---------------------------------------------------------------------------
|
| 152 |
+
# Fixed-benefit exclusion for comprehensive-indemnity intent
|
|
|
|
| 153 |
# ---------------------------------------------------------------------------
|
| 154 |
# A FIXED-BENEFIT product (hospital daily cash, personal accident, critical
|
| 155 |
# illness, cancer / other defined-benefit) pays a fixed lump-sum / per-day
|
|
|
|
| 198 |
)
|
| 199 |
|
| 200 |
# ---------------------------------------------------------------------------
|
| 201 |
+
# Unified recommendation-fit gate tunables
|
| 202 |
# ---------------------------------------------------------------------------
|
| 203 |
+
# Recommendation-fit is SYSTEMIC: the cited-card list and the advisory prose
|
| 204 |
+
# are gated by the SAME fitness logic. Two hard gates (entry-age for the
|
| 205 |
+
# INSURED person; required feature = maternity/newborn) plus ranking signals
|
| 206 |
+
# for grade/rank ordering and the cost-objective lead.
|
|
|
|
| 207 |
|
| 208 |
# Entry-age grace. The catalog's max_entry_age is the regulator-filed entry
|
| 209 |
# age; underwriters allow a small grace band (mirrors PROFILE_AGE_TOLERANCE).
|
|
|
|
| 345 |
if not chunks_list:
|
| 346 |
return chunks_list
|
| 347 |
|
| 348 |
+
# The demographic gates (numeric age-range, senior-only, adult-only)
|
| 349 |
+
# reason about the age of the person the policy actually INSURES, not
|
| 350 |
+
# the payer's. When the policy covers parents (~70), using the paying
|
| 351 |
+
# child's age (e.g. 36) would wrongly DROP a senior-citizen plan
|
| 352 |
+
# ("min_entry_age>=60 AND payer<50") and wrongly KEEP a max-entry-65
|
| 353 |
+
# plan. `_oldest_insured_age` returns the eldest parent's age when
|
| 354 |
+
# insuring parents, else the profile's own age. Falls back to the raw
|
| 355 |
+
# profile age if no resolved signal.
|
|
|
|
| 356 |
_resolved_age = _oldest_insured_age(profile)
|
| 357 |
if _resolved_age is None:
|
| 358 |
_resolved_age = _profile_get(profile, "age")
|
|
|
|
| 709 |
expenses, so they are a wrong PRIMARY recommendation. They are
|
| 710 |
NEVER dropped for a user who explicitly wants a supplement.
|
| 711 |
|
| 712 |
+
5. ENTRY-AGE GATE — drop a plan whose max entry age cannot accept
|
| 713 |
+
the OLDEST person the policy must insure. When the profile covers
|
| 714 |
+
parents (~70), the gate uses the parents' age, not the paying
|
| 715 |
+
child's (the demographic pre-filter only sees the payer's age).
|
| 716 |
+
Conservative: fires only when both the insured age AND the policy
|
| 717 |
+
fact are present.
|
|
|
|
| 718 |
|
| 719 |
Conservative on missing data: a rule only fires when BOTH the profile
|
| 720 |
signal AND the policy fact are present. A chunk with no enriched facts
|
|
|
|
| 750 |
if wants_comprehensive and _is_fixed_benefit_chunk(m):
|
| 751 |
continue
|
| 752 |
|
| 753 |
+
# Rule 5 — HARD ELIGIBILITY: a plan whose max entry age cannot
|
| 754 |
+
# accept the oldest insured person is structurally unusable (e.g. a
|
| 755 |
+
# 70yo parent cannot be enrolled in a plan whose max_entry_age is
|
| 756 |
+
# 65). Fires only when BOTH the age and the policy fact are present
|
| 757 |
+
# (conservative on missing data) and only when the profile actually
|
| 758 |
+
# insures someone whose age we know (the demographic
|
| 759 |
+
# apply_profile_filter uses the *payer's* age and never sees the
|
| 760 |
+
# parents' age).
|
| 761 |
if oldest_age is not None:
|
| 762 |
maxe = _as_int(m.get("max_entry_age"))
|
| 763 |
if maxe is not None and maxe + ENTRY_AGE_TOLERANCE < oldest_age:
|
|
|
|
| 1039 |
INFORMATIONAL only — it is not required for citation grounding because
|
| 1040 |
upstream call sites (e.g. brain_tools.retrieve_policies) build pruned
|
| 1041 |
dicts that intentionally omit it, and the brain cites by policy
|
| 1042 |
+
identity, not by chunk offset. Requiring an offset here would drop
|
| 1043 |
+
every chunk built by those pruned-dict call sites.
|
|
|
|
|
|
|
| 1044 |
"""
|
| 1045 |
kept: list[Any] = []
|
| 1046 |
for ch in chunks:
|
|
|
|
| 1064 |
duplicates to one chunk, keeping the highest-scoring chunk. Preserves
|
| 1065 |
the order of first appearance of each kept product.
|
| 1066 |
|
| 1067 |
+
Keying is by the SHARED canonical identity
|
| 1068 |
(policy_identity.canonical_key) — UIN-primary, product_key fallback —
|
| 1069 |
+
NOT the raw policy_id. The same product can appear under different
|
| 1070 |
+
policy_ids: a marketing rename ("my:Optima Secure" vs "my:Optima
|
| 1071 |
+
Secure (older variant)" — same UIN) or two doctype siblings
|
| 1072 |
+
("...__wordings" vs "...__brochure" — same product_key). policy_id-only
|
| 1073 |
+
dedup would let both through; canonical-identity keying collapses them.
|
| 1074 |
+
This reuses the exact rule the marketplace endpoint uses so the
|
| 1075 |
+
recommender and the marketplace agree on "the same policy".
|
|
|
|
| 1076 |
"""
|
| 1077 |
best: dict[str, Any] = {}
|
| 1078 |
order: list[str] = []
|
|
@@ -791,17 +791,14 @@ def _profile_tuned_weights(profile: Optional[dict]) -> dict[str, float]:
|
|
| 791 |
def profile_completeness(profile: Optional[dict]) -> float:
|
| 792 |
"""0.0–1.0 measure of how much we know about the buyer.
|
| 793 |
|
| 794 |
-
|
| 795 |
-
|
| 796 |
-
|
| 797 |
-
parents_age_max which Path B doesn't gather, causing the UI bar to read
|
| 798 |
-
70% while Path B already considered the profile "ready to recommend".
|
| 799 |
|
| 800 |
The 7 slots: name, age, dependents, location_tier, income_band,
|
| 801 |
primary_goal, health_conditions. `name` is the identifier; the other 6
|
| 802 |
are decision-critical for retrieval. Existing-cover and budget-band are
|
| 803 |
-
captured opportunistically
|
| 804 |
-
but are NOT required to recommend.
|
| 805 |
|
| 806 |
Used by the frontend to GATE the personalized scorecard view — until
|
| 807 |
completeness >= 0.6, we show insurer-level metrics (CSR, complaints —
|
|
|
|
| 791 |
def profile_completeness(profile: Optional[dict]) -> float:
|
| 792 |
"""0.0–1.0 measure of how much we know about the buyer.
|
| 793 |
|
| 794 |
+
Aligned with the `_REQUIRED_FOR_READY` 7-slot list (see brain_tools.py
|
| 795 |
+
+ single_brain.py) so this measure agrees with the brain's "ready to
|
| 796 |
+
recommend" gate.
|
|
|
|
|
|
|
| 797 |
|
| 798 |
The 7 slots: name, age, dependents, location_tier, income_band,
|
| 799 |
primary_goal, health_conditions. `name` is the identifier; the other 6
|
| 800 |
are decision-critical for retrieval. Existing-cover and budget-band are
|
| 801 |
+
captured opportunistically but are NOT required to recommend.
|
|
|
|
| 802 |
|
| 803 |
Used by the frontend to GATE the personalized scorecard view — until
|
| 804 |
completeness >= 0.6, we show insurer-level metrics (CSR, complaints —
|
|
@@ -74,39 +74,24 @@ class SessionState:
|
|
| 74 |
# "Conversation turn" column in the admin Recommendation History panel
|
| 75 |
# (previously showed "—" because no caller populated the field).
|
| 76 |
turn_idx: int = 0
|
| 77 |
-
#
|
| 78 |
-
#
|
| 79 |
-
#
|
| 80 |
-
#
|
| 81 |
-
# legacy orchestrator mid-stream. Belt-and-suspenders: main.py stamps
|
| 82 |
-
# this True after the FIRST successful single_brain turn, and
|
| 83 |
-
# subsequent SingleBrainError responses on the same session must NOT
|
| 84 |
-
# fall through to the orchestrator — they emit a graceful retry
|
| 85 |
-
# prompt so the session stays sticky on single_brain.
|
| 86 |
single_brain_sticky: bool = False
|
| 87 |
-
#
|
| 88 |
-
#
|
| 89 |
-
#
|
| 90 |
-
#
|
| 91 |
-
#
|
| 92 |
-
_recall_sniff_done: bool = False
|
| 93 |
-
# Bug #108 + #110 (2026-05-16) — POST-RECAP pricing & family-history
|
| 94 |
-
# bundle re-ask gate (brain_tools.retrieve_policies).
|
| 95 |
-
# pricing_bundle_reasked — one-shot guard. Set True the first time the
|
| 96 |
-
# gate re-asks an unresolved bundle slot so the NEXT recommendation
|
| 97 |
-
# retrieve proceeds even if the user skips (SOFT capture, no hard
|
| 98 |
-
# loop).
|
| 99 |
# pricing_bundle_skipped — set True by single_brain when the user
|
| 100 |
-
# explicitly declines the pricing inputs
|
| 101 |
-
# "you decide", "skip"). Bypasses the re-ask entirely.
|
| 102 |
pricing_bundle_reasked: bool = False
|
| 103 |
pricing_bundle_skipped: bool = False
|
| 104 |
|
| 105 |
def _flush(self) -> None:
|
| 106 |
-
"""No-op
|
| 107 |
-
|
| 108 |
-
callers (orchestrator + fact_find_brain + tests) don't have to change
|
| 109 |
-
their write paths.
|
| 110 |
"""
|
| 111 |
return None
|
| 112 |
|
|
|
|
| 74 |
# "Conversation turn" column in the admin Recommendation History panel
|
| 75 |
# (previously showed "—" because no caller populated the field).
|
| 76 |
turn_idx: int = 0
|
| 77 |
+
# Set True after the first successful single_brain turn; a later
|
| 78 |
+
# SingleBrainError on the same session then emits a graceful retry
|
| 79 |
+
# prompt instead of switching handlers, so the session stays on
|
| 80 |
+
# single_brain.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 81 |
single_brain_sticky: bool = False
|
| 82 |
+
# Post-recap pricing & family-history bundle re-ask gate
|
| 83 |
+
# (brain_tools.retrieve_policies):
|
| 84 |
+
# pricing_bundle_reasked — one-shot guard; set True the first time
|
| 85 |
+
# the gate re-asks an unresolved bundle slot so the next
|
| 86 |
+
# recommendation retrieve proceeds even if the user skips.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 87 |
# pricing_bundle_skipped — set True by single_brain when the user
|
| 88 |
+
# explicitly declines the pricing inputs; bypasses the re-ask.
|
|
|
|
| 89 |
pricing_bundle_reasked: bool = False
|
| 90 |
pricing_bundle_skipped: bool = False
|
| 91 |
|
| 92 |
def _flush(self) -> None:
|
| 93 |
+
"""No-op. Session state lives only in the in-memory dict; the
|
| 94 |
+
method is kept so callers' write paths don't have to change.
|
|
|
|
|
|
|
| 95 |
"""
|
| 96 |
return None
|
| 97 |
|
|
@@ -1,20 +1,19 @@
|
|
| 1 |
-
"""Single-brain conversation handler
|
| 2 |
|
| 3 |
-
One Gemini Flash call per turn
|
| 4 |
-
|
| 5 |
-
iteration whether to:
|
| 6 |
- call `save_profile_field` to persist captured slots,
|
| 7 |
- call `retrieve_policies` to pull policy chunks from Chroma,
|
| 8 |
- call `mark_recommendation` to flag the policies just pitched,
|
| 9 |
- or emit a final text reply.
|
| 10 |
|
| 11 |
-
The loop iterates up to `MAX_ITERATIONS`
|
| 12 |
-
|
| 13 |
-
|
| 14 |
|
| 15 |
-
Wire-up: /api/chat → main.py.chat() →
|
| 16 |
-
On
|
| 17 |
-
|
| 18 |
|
| 19 |
We call the Gemini REST API directly (httpx, like google_gemini_llm.py)
|
| 20 |
rather than using the `google.generativeai` SDK so we don't need to pin
|
|
@@ -57,13 +56,10 @@ GEMINI_BASE_URL = "https://generativelanguage.googleapis.com/v1beta/models"
|
|
| 57 |
PER_CALL_TIMEOUT_SEC = 25.0
|
| 58 |
|
| 59 |
# Max iterations of the tool-call loop. Prevents runaway tool-call cycles
|
| 60 |
-
# where the LLM keeps calling save_profile_field on the same value.
|
| 61 |
-
#
|
| 62 |
-
#
|
| 63 |
-
#
|
| 64 |
-
# exhaust. Coercer fix in brain_tools resolves the primary cause; the
|
| 65 |
-
# extra headroom protects against the next variant where Gemini chains
|
| 66 |
-
# 3-4 saves + 2 retrieves on a long pre-recommendation user turn.
|
| 67 |
MAX_ITERATIONS = 8
|
| 68 |
|
| 69 |
# Transient-error retry policy (2026-05-15 / KI-singlebrain-503).
|
|
@@ -393,12 +389,9 @@ class TurnResult:
|
|
| 393 |
blocked: bool = False
|
| 394 |
profile_updates: dict = field(default_factory=dict)
|
| 395 |
followup_policy_id: Optional[str] = None
|
| 396 |
-
# main.py stamps ChatResponse.returning_user_recalled from this.
|
| 397 |
-
#
|
| 398 |
-
#
|
| 399 |
-
# handle_turn now always leaves this False; explicit returning-user
|
| 400 |
-
# recall is the separate POST /api/profile/recall-by-name endpoint.
|
| 401 |
-
# Field kept for ChatResponse / frontend compatibility.
|
| 402 |
returning_user_recalled: bool = False
|
| 403 |
|
| 404 |
|
|
@@ -627,22 +620,9 @@ def _detect_language(user_text: str) -> str:
|
|
| 627 |
return "en"
|
| 628 |
|
| 629 |
|
| 630 |
-
#
|
| 631 |
-
#
|
| 632 |
-
#
|
| 633 |
-
# `_synthesise_fallback` (the deterministic "ask the next missing slot"
|
| 634 |
-
# generator) were leftover rule-based scaffolding from the pre-Path-B
|
| 635 |
-
# rule/3-brain/judge era. They masked the single LLM: when Gemini returned
|
| 636 |
-
# no text the fallback injected a canned "What's your name?", simulating a
|
| 637 |
-
# fact-find that wasn't happening → an infinite loop. The single LLM
|
| 638 |
-
# (RULE 1 → save_profile_field) is the SOLE fact-find driver now; a genuine
|
| 639 |
-
# empty-LLM turn surfaces an HONEST error (see end of handle_turn), never a
|
| 640 |
-
# fabricated slot-question. Do NOT reintroduce deterministic fact-find.
|
| 641 |
-
|
| 642 |
-
# Honest reply for the rare genuine empty-LLM turn (should be near-zero now
|
| 643 |
-
# that thinkingConfig is set so gemini-2.5-flash actually emits output). We
|
| 644 |
-
# surface a transparent retry ask — NEVER a fabricated slot-question that
|
| 645 |
-
# simulates a fact-find that did not happen.
|
| 646 |
_HONEST_EMPTY_REPLY = (
|
| 647 |
"I'm having trouble generating a response right now — could you "
|
| 648 |
"rephrase that, or try again in a moment?"
|
|
@@ -718,28 +698,15 @@ async def _gemini_call(
|
|
| 718 |
"tools": [{"functionDeclarations": tools}],
|
| 719 |
"toolConfig": {"functionCallingConfig": {"mode": "AUTO"}},
|
| 720 |
"generationConfig": {
|
| 721 |
-
# Z2 fix — Issue 1 (mid-session amnesia). Priya T3 + Vikram T2/T4
|
| 722 |
-
# came back with the "I lost my train of thought" template even
|
| 723 |
-
# though slot capture succeeded. Root cause matches KI-150
|
| 724 |
-
# (fact_find LLM, 420 → 700): when Gemini must emit prose AND a
|
| 725 |
-
# tool-call trailer in the same response, the model hits
|
| 726 |
-
# maxOutputTokens mid-emission, the trailer truncates, and the
|
| 727 |
-
# caller falls through to the defensive reply. Budget breakdown
|
| 728 |
-
# at p95: prose ~600 tok + tool-call JSON ~800 tok + 20% margin
|
| 729 |
-
# ⇒ 1680, rounded up to a safe power-of-two-ish 2048.
|
| 730 |
"temperature": 0.4,
|
|
|
|
|
|
|
|
|
|
| 731 |
"maxOutputTokens": 2048,
|
| 732 |
-
#
|
| 733 |
-
#
|
| 734 |
-
#
|
| 735 |
-
#
|
| 736 |
-
# completion (finishReason=STOP, content has no `parts`, zero
|
| 737 |
-
# candidate tokens) for tool-calling turns — so _synthesise_
|
| 738 |
-
# fallback injects the canned "What's your name?" and the
|
| 739 |
-
# fact-find loops forever (the live core-flow regression).
|
| 740 |
-
# thinkingBudget=0 disables thinking → the model behaves like
|
| 741 |
-
# the fast non-thinking frontier model it should be and emits
|
| 742 |
-
# the save_profile_field tool call correctly. Do NOT remove.
|
| 743 |
"thinkingConfig": {"thinkingBudget": 0},
|
| 744 |
},
|
| 745 |
}
|
|
@@ -1080,25 +1047,16 @@ def _build_recommendation_citations(
|
|
| 1080 |
) -> tuple[list[dict], bool]:
|
| 1081 |
"""Single source of truth for the structured "CITED POLICIES" cards.
|
| 1082 |
|
| 1083 |
-
|
| 1084 |
-
|
| 1085 |
-
LLM's curated shortlist (free text). Cards used to be the raw
|
| 1086 |
-
`retrieve_policies` recall dump deduped by chunk_id and rendered in
|
| 1087 |
-
vector-score order — so the cards listed policies the LLM never named
|
| 1088 |
-
and dropped ones it did.
|
| 1089 |
-
|
| 1090 |
-
This builds the citation set as EXACTLY the policies the assistant
|
| 1091 |
-
actually recommended:
|
| 1092 |
|
| 1093 |
1. If the LLM called `mark_recommendation(policy_ids=[...])`, those
|
| 1094 |
-
ids are the
|
| 1095 |
-
2. Otherwise
|
| 1096 |
-
per KI-254), parse the reply prose: every retrieved policy whose
|
| 1097 |
name appears in `reply_text` is selected.
|
| 1098 |
|
| 1099 |
-
|
| 1100 |
-
|
| 1101 |
-
cards ARE the gated, fit-ranked set:
|
| 1102 |
|
| 1103 |
• CANONICAL DEDUP: `retrieved_chunks_all` is the union of every
|
| 1104 |
retrieve_policies result this turn. Each result is already gated +
|
|
@@ -1521,18 +1479,10 @@ async def handle_turn(
|
|
| 1521 |
# conversation — not a returning user.
|
| 1522 |
_current_turn = int(getattr(session, "turn_idx", 1) or 1)
|
| 1523 |
|
| 1524 |
-
#
|
| 1525 |
-
#
|
| 1526 |
-
#
|
| 1527 |
-
#
|
| 1528 |
-
# rule/3-brain/judge era: it masked the single LLM, discarded the name
|
| 1529 |
-
# the user gave, and produced an infinite fact-find loop. The single
|
| 1530 |
-
# LLM (RULE 1 → save_profile_field) is now the SOLE fact-find driver.
|
| 1531 |
-
# Explicit returning-user recall lives ONLY in the separate
|
| 1532 |
-
# POST /api/profile/recall-by-name endpoint (an explicit user action,
|
| 1533 |
-
# not a deterministic in-chat short-circuit). `_did_recall_this_turn`
|
| 1534 |
-
# is kept as a constant False so the TurnResult.returning_user_recalled
|
| 1535 |
-
# field (read by main.py) stays valid without the removed machinery.
|
| 1536 |
_did_recall_this_turn = False
|
| 1537 |
|
| 1538 |
_has_prior_profile = any(
|
|
@@ -1545,8 +1495,7 @@ async def handle_turn(
|
|
| 1545 |
is_returning_user = (_current_turn == 1) and _has_prior_profile
|
| 1546 |
|
| 1547 |
# GOOGLE_API_KEY gate — asserted just before anything that talks to
|
| 1548 |
-
# Gemini.
|
| 1549 |
-
# used to precede this was removed 2026-05-17.)
|
| 1550 |
api_key = os.environ.get("GOOGLE_API_KEY", "").strip()
|
| 1551 |
if not api_key:
|
| 1552 |
raise SingleBrainError("GOOGLE_API_KEY not set")
|
|
|
|
| 1 |
+
"""Single-brain conversation handler.
|
| 2 |
|
| 3 |
+
One Gemini Flash call per turn with native function-calling. The LLM
|
| 4 |
+
decides on each iteration whether to:
|
|
|
|
| 5 |
- call `save_profile_field` to persist captured slots,
|
| 6 |
- call `retrieve_policies` to pull policy chunks from Chroma,
|
| 7 |
- call `mark_recommendation` to flag the policies just pitched,
|
| 8 |
- or emit a final text reply.
|
| 9 |
|
| 10 |
+
The loop iterates up to `MAX_ITERATIONS` so the LLM can chain multiple
|
| 11 |
+
tool calls in a single user turn before responding. Beyond that cap an
|
| 12 |
+
honest retry message is returned.
|
| 13 |
|
| 14 |
+
Wire-up: /api/chat → main.py.chat() → single_brain.handle_turn(...).
|
| 15 |
+
On a SingleBrainError the caller falls through to nim_fallback so the
|
| 16 |
+
user always gets a reply.
|
| 17 |
|
| 18 |
We call the Gemini REST API directly (httpx, like google_gemini_llm.py)
|
| 19 |
rather than using the `google.generativeai` SDK so we don't need to pin
|
|
|
|
| 56 |
PER_CALL_TIMEOUT_SEC = 25.0
|
| 57 |
|
| 58 |
# Max iterations of the tool-call loop. Prevents runaway tool-call cycles
|
| 59 |
+
# where the LLM keeps calling save_profile_field on the same value. Sized
|
| 60 |
+
# so Gemini can chain a long pre-recommendation turn (several
|
| 61 |
+
# save_profile_field calls + one or two retrieve_policies) within one
|
| 62 |
+
# user turn.
|
|
|
|
|
|
|
|
|
|
| 63 |
MAX_ITERATIONS = 8
|
| 64 |
|
| 65 |
# Transient-error retry policy (2026-05-15 / KI-singlebrain-503).
|
|
|
|
| 389 |
blocked: bool = False
|
| 390 |
profile_updates: dict = field(default_factory=dict)
|
| 391 |
followup_policy_id: Optional[str] = None
|
| 392 |
+
# main.py stamps ChatResponse.returning_user_recalled from this.
|
| 393 |
+
# handle_turn leaves it False; explicit returning-user recall is the
|
| 394 |
+
# separate POST /api/profile/recall-by-name endpoint.
|
|
|
|
|
|
|
|
|
|
| 395 |
returning_user_recalled: bool = False
|
| 396 |
|
| 397 |
|
|
|
|
| 620 |
return "en"
|
| 621 |
|
| 622 |
|
| 623 |
+
# Reply for a turn where the LLM returns no text and no tool calls — a
|
| 624 |
+
# transparent retry ask. The single LLM is the only fact-find driver; we
|
| 625 |
+
# never fabricate a slot-question.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 626 |
_HONEST_EMPTY_REPLY = (
|
| 627 |
"I'm having trouble generating a response right now — could you "
|
| 628 |
"rephrase that, or try again in a moment?"
|
|
|
|
| 698 |
"tools": [{"functionDeclarations": tools}],
|
| 699 |
"toolConfig": {"functionCallingConfig": {"mode": "AUTO"}},
|
| 700 |
"generationConfig": {
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 701 |
"temperature": 0.4,
|
| 702 |
+
# Sized so a turn emitting prose plus a tool-call trailer does
|
| 703 |
+
# not truncate mid-emission (p95 ≈ prose 600 + tool JSON 800 +
|
| 704 |
+
# margin).
|
| 705 |
"maxOutputTokens": 2048,
|
| 706 |
+
# gemini-2.5-flash is a thinking model; thinkingBudget=0
|
| 707 |
+
# disables the internal thinking phase so it emits the tool
|
| 708 |
+
# call / text directly (a non-zero budget can consume the
|
| 709 |
+
# output allowance and return an empty completion).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 710 |
"thinkingConfig": {"thinkingBudget": 0},
|
| 711 |
},
|
| 712 |
}
|
|
|
|
| 1047 |
) -> tuple[list[dict], bool]:
|
| 1048 |
"""Single source of truth for the structured "CITED POLICIES" cards.
|
| 1049 |
|
| 1050 |
+
The cited-card set IS exactly the policies the assistant recommended,
|
| 1051 |
+
gated by the same fitness logic as the prose:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1052 |
|
| 1053 |
1. If the LLM called `mark_recommendation(policy_ids=[...])`, those
|
| 1054 |
+
ids are the selection.
|
| 1055 |
+
2. Otherwise, parse the reply prose: every retrieved policy whose
|
|
|
|
| 1056 |
name appears in `reply_text` is selected.
|
| 1057 |
|
| 1058 |
+
The cited-card list and the advisory prose are gated by the SAME
|
| 1059 |
+
fitness logic, with:
|
|
|
|
| 1060 |
|
| 1061 |
• CANONICAL DEDUP: `retrieved_chunks_all` is the union of every
|
| 1062 |
retrieve_policies result this turn. Each result is already gated +
|
|
|
|
| 1479 |
# conversation — not a returning user.
|
| 1480 |
_current_turn = int(getattr(session, "turn_idx", 1) or 1)
|
| 1481 |
|
| 1482 |
+
# The single LLM (RULE 1 → save_profile_field) is the sole fact-find
|
| 1483 |
+
# driver. Explicit returning-user recall is the separate
|
| 1484 |
+
# POST /api/profile/recall-by-name endpoint, so handle_turn never sets
|
| 1485 |
+
# returning_user_recalled True.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1486 |
_did_recall_this_turn = False
|
| 1487 |
|
| 1488 |
_has_prior_profile = any(
|
|
|
|
| 1495 |
is_returning_user = (_current_turn == 1) and _has_prior_profile
|
| 1496 |
|
| 1497 |
# GOOGLE_API_KEY gate — asserted just before anything that talks to
|
| 1498 |
+
# Gemini.
|
|
|
|
| 1499 |
api_key = os.environ.get("GOOGLE_API_KEY", "").strip()
|
| 1500 |
if not api_key:
|
| 1501 |
raise SingleBrainError("GOOGLE_API_KEY not set")
|
|
@@ -3,19 +3,15 @@ spoken-language text for Sarvam Bulbul TTS.
|
|
| 3 |
|
| 4 |
Why this exists: an unprocessed LLM reply with markdown bold, inline
|
| 5 |
[Source: ...] tags, and acronyms reads like a screenshot when spoken. Users
|
| 6 |
-
hear "asterisk asterisk bold asterisk asterisk A-Y-U-S-H
|
| 7 |
-
|
| 8 |
-
|
| 9 |
-
|
| 10 |
-
|
| 11 |
-
|
| 12 |
-
|
| 13 |
-
|
| 14 |
-
|
| 15 |
-
`strip_cot_preamble` is called from `persona.strip_think_tags` so every
|
| 16 |
-
reply path that already goes through the <think>-tag strip also gets the
|
| 17 |
-
preamble strip — no orchestrator.py changes needed (that file is owned
|
| 18 |
-
by another lane / KI-101).
|
| 19 |
|
| 20 |
The function turns text like:
|
| 21 |
|
|
@@ -73,15 +69,15 @@ ACRONYMS = {
|
|
| 73 |
r"\bpp\.(\d+)": r"page \1",
|
| 74 |
}
|
| 75 |
|
| 76 |
-
#
|
| 77 |
-
# letter-by-letter.
|
| 78 |
-
#
|
| 79 |
-
#
|
| 80 |
#
|
| 81 |
-
#
|
| 82 |
-
#
|
| 83 |
-
# `
|
| 84 |
-
#
|
| 85 |
_MONEY_RANGE_L = re.compile(
|
| 86 |
r"₹?\s*(\d+(?:\.\d+)?)\s*-\s*(\d+(?:\.\d+)?)\s*L\b",
|
| 87 |
re.IGNORECASE,
|
|
@@ -383,13 +379,13 @@ def _truncate_for_voice(text: str, max_words: int = 60) -> str:
|
|
| 383 |
|
| 384 |
|
| 385 |
# ============================================================================
|
| 386 |
-
#
|
| 387 |
# ============================================================================
|
| 388 |
-
#
|
| 389 |
-
#
|
| 390 |
-
# 1.
|
| 391 |
-
# followed by the answer
|
| 392 |
-
#
|
| 393 |
# 2. The faithfulness JUDGE model occasionally returns its own reasoning
|
| 394 |
# instead of a clean rescue reply.
|
| 395 |
# 3. The brain model misunderstands the system prompt and echoes the
|
|
@@ -400,7 +396,7 @@ def _truncate_for_voice(text: str, max_words: int = 60) -> str:
|
|
| 400 |
# lines / first 600 chars), so substantive mid-reply content like
|
| 401 |
# "We have three options: A, B, C" is preserved.
|
| 402 |
|
| 403 |
-
# ---- Sentence-level preamble patterns
|
| 404 |
#
|
| 405 |
# A CoT preamble can appear as:
|
| 406 |
# (a) a full line of its own: "We need to respond carefully.\n<answer>"
|
|
@@ -457,9 +453,9 @@ _BRACKET_INTERNAL = re.compile(
|
|
| 457 |
flags=re.IGNORECASE | re.DOTALL,
|
| 458 |
)
|
| 459 |
|
| 460 |
-
# Stray, unbalanced <think> tags
|
| 461 |
-
#
|
| 462 |
-
#
|
| 463 |
_STRAY_CLOSE_THINK = re.compile(r"^.*?</think>", flags=re.DOTALL | re.IGNORECASE)
|
| 464 |
|
| 465 |
# Maximum scan window for preamble. Beyond this, content is treated as
|
|
@@ -567,15 +563,14 @@ def tts_preprocess(text: str, language: str = "en", max_words: int = 60) -> str:
|
|
| 567 |
"""Public entry — turn an LLM reply into spoken-language text for TTS."""
|
| 568 |
if not text:
|
| 569 |
return ""
|
| 570 |
-
#
|
| 571 |
-
#
|
| 572 |
-
#
|
| 573 |
-
# TTS of a cached reply).
|
| 574 |
cleaned = strip_cot_preamble(text)
|
| 575 |
cleaned = _strip_markdown(cleaned)
|
| 576 |
-
#
|
| 577 |
-
#
|
| 578 |
-
#
|
| 579 |
cleaned = _normalize_money(cleaned)
|
| 580 |
# #106 — AFTER money (lakh/crore already words): expand remaining bare
|
| 581 |
# integers ("29" → "twenty-nine") + neutralise en/em dashes so Sarvam
|
|
|
|
| 3 |
|
| 4 |
Why this exists: an unprocessed LLM reply with markdown bold, inline
|
| 5 |
[Source: ...] tags, and acronyms reads like a screenshot when spoken. Users
|
| 6 |
+
would otherwise hear "asterisk asterisk bold asterisk asterisk A-Y-U-S-H
|
| 7 |
+
pp dot 1 dash 2".
|
| 8 |
+
|
| 9 |
+
This module also exposes `strip_cot_preamble`, the chain-of-thought /
|
| 10 |
+
instruction-echo stripper that runs on TEXT replies (not just TTS).
|
| 11 |
+
Reasoning models can leak internal reasoning into `reply_text` — e.g.
|
| 12 |
+
"We need to respond to user question…", "We must ground every factual
|
| 13 |
+
claim…", "<think>...</think>The answer is X." `strip_cot_preamble`
|
| 14 |
+
removes that preamble.
|
|
|
|
|
|
|
|
|
|
|
|
|
| 15 |
|
| 16 |
The function turns text like:
|
| 17 |
|
|
|
|
| 69 |
r"\bpp\.(\d+)": r"page \1",
|
| 70 |
}
|
| 71 |
|
| 72 |
+
# Money / range shorthand that Sarvam Bulbul would otherwise read
|
| 73 |
+
# letter-by-letter (e.g. "₹25L+" as "two five L plus"). Order in
|
| 74 |
+
# `_normalize_money` matters: handle RANGES first, then PLUS-SUFFIXES,
|
| 75 |
+
# then bare unit-suffixes, then standalone "+".
|
| 76 |
#
|
| 77 |
+
# Also covers `k` (thousand), word-forms (`1 lakh`, `1 crore`), `+`
|
| 78 |
+
# suffix → "above", and bare numerics like `30000` → "30 thousand",
|
| 79 |
+
# `1,00,000` → "1 lakh". `k` is currency-gated to avoid breaking
|
| 80 |
+
# "10K marathon".
|
| 81 |
_MONEY_RANGE_L = re.compile(
|
| 82 |
r"₹?\s*(\d+(?:\.\d+)?)\s*-\s*(\d+(?:\.\d+)?)\s*L\b",
|
| 83 |
re.IGNORECASE,
|
|
|
|
| 379 |
|
| 380 |
|
| 381 |
# ============================================================================
|
| 382 |
+
# Chain-of-thought / instruction-echo strip
|
| 383 |
# ============================================================================
|
| 384 |
+
# LLM replies can leak internal reasoning into user-visible reply_text in
|
| 385 |
+
# three modes:
|
| 386 |
+
# 1. Reasoning models (Qwen3-Next 80B) emit a <think>...</think> block
|
| 387 |
+
# followed by the answer, sometimes with a missing / malformed
|
| 388 |
+
# <think> tag that a plain tag strip would let through.
|
| 389 |
# 2. The faithfulness JUDGE model occasionally returns its own reasoning
|
| 390 |
# instead of a clean rescue reply.
|
| 391 |
# 3. The brain model misunderstands the system prompt and echoes the
|
|
|
|
| 396 |
# lines / first 600 chars), so substantive mid-reply content like
|
| 397 |
# "We have three options: A, B, C" is preserved.
|
| 398 |
|
| 399 |
+
# ---- Sentence-level preamble patterns ----
|
| 400 |
#
|
| 401 |
# A CoT preamble can appear as:
|
| 402 |
# (a) a full line of its own: "We need to respond carefully.\n<answer>"
|
|
|
|
| 453 |
flags=re.IGNORECASE | re.DOTALL,
|
| 454 |
)
|
| 455 |
|
| 456 |
+
# Stray, unbalanced <think> tags (a balanced-tag strip requires both open
|
| 457 |
+
# and close in the same blob). If we see an isolated </think> mid-reply,
|
| 458 |
+
# drop everything before it.
|
| 459 |
_STRAY_CLOSE_THINK = re.compile(r"^.*?</think>", flags=re.DOTALL | re.IGNORECASE)
|
| 460 |
|
| 461 |
# Maximum scan window for preamble. Beyond this, content is treated as
|
|
|
|
| 563 |
"""Public entry — turn an LLM reply into spoken-language text for TTS."""
|
| 564 |
if not text:
|
| 565 |
return ""
|
| 566 |
+
# Defense in depth: run the preamble strip again here in case this is
|
| 567 |
+
# called on a path that didn't already strip (e.g. direct TTS of a
|
| 568 |
+
# cached reply).
|
|
|
|
| 569 |
cleaned = strip_cot_preamble(text)
|
| 570 |
cleaned = _strip_markdown(cleaned)
|
| 571 |
+
# Currency/range shorthand expansion before acronym handling so ₹5L
|
| 572 |
+
# becomes "5 lakhs" instead of getting caught by the bare-L acronym
|
| 573 |
+
# path.
|
| 574 |
cleaned = _normalize_money(cleaned)
|
| 575 |
# #106 — AFTER money (lakh/crore already words): expand remaining bare
|
| 576 |
# integers ("29" → "twenty-nine") + neutralise en/em dashes so Sarvam
|
|
@@ -2,11 +2,9 @@
|
|
| 2 |
|
| 3 |
For each PDF in rag/corpus/:
|
| 4 |
1. Read full text via pdfplumber (already have per-page text from ingest)
|
| 5 |
-
2. Pass to LLM (
|
| 6 |
-
|
| 7 |
-
|
| 8 |
-
2048-token <think> reasoning + output cap was truncating JSON mid-stream
|
| 9 |
-
when used as extraction fallback, causing ~20 extraction failures).
|
| 10 |
3. Self-critique pass — LLM scores per-field confidence vs source text
|
| 11 |
4. Validate via Pydantic
|
| 12 |
5. Upsert into DuckDB `policies` table
|
|
@@ -73,13 +71,11 @@ EXTRACT_SYSTEM = """You extract structured fields from Indian health insurance p
|
|
| 73 |
|
| 74 |
|
| 75 |
def build_extract_prompt(policy_text: str, schema_excerpt: str, policy_id: str) -> str:
|
| 76 |
-
#
|
| 77 |
-
#
|
| 78 |
-
#
|
| 79 |
-
#
|
| 80 |
-
#
|
| 81 |
-
# doesn't yield many structured fields, just policy_exclusions list which
|
| 82 |
-
# is mostly noise.
|
| 83 |
MAX_CHARS = 12_000
|
| 84 |
if len(policy_text) > MAX_CHARS:
|
| 85 |
# Front-bias: schedules, definitions, waiting periods, UIN, sum-insured
|
|
@@ -143,8 +139,8 @@ def load_manifest() -> dict:
|
|
| 143 |
def json_from_llm_text(text: str) -> dict:
|
| 144 |
"""Strip code fences and <think> blocks, extract the first balanced {...} block."""
|
| 145 |
text = text.strip()
|
| 146 |
-
#
|
| 147 |
-
#
|
| 148 |
# truncated mid-thought).
|
| 149 |
text = re.sub(r"<think>.*?</think>", "", text, flags=re.DOTALL)
|
| 150 |
text = re.sub(r"<think>.*", "", text, flags=re.DOTALL)
|
|
@@ -265,10 +261,8 @@ async def extract_one(pdf_path: Path, manifest_entry: dict, llm_primary, llm_fal
|
|
| 265 |
for attempt, llm in enumerate([llm_primary, llm_fallback]):
|
| 266 |
try:
|
| 267 |
# Hard per-attempt timeout so a hung TCP connection in httpx
|
| 268 |
-
# pooling can't stall the whole sweep.
|
| 269 |
-
#
|
| 270 |
-
# so give it 120s ceiling. Sarvam (fallback) has no retries; 60s
|
| 271 |
-
# is plenty.
|
| 272 |
attempt_timeout = 180 if llm is llm_primary else 120
|
| 273 |
res = await asyncio.wait_for(
|
| 274 |
llm.chat(messages=messages, temperature=0.0, max_tokens=2048),
|
|
@@ -328,19 +322,8 @@ async def main():
|
|
| 328 |
if args.limit:
|
| 329 |
pdfs = pdfs[: args.limit]
|
| 330 |
|
| 331 |
-
#
|
| 332 |
-
#
|
| 333 |
-
# 40 req/min with no daily cap. Clean JSON output, no <think> verbosity.
|
| 334 |
-
# Used as primary because of the bigger reasoning budget.
|
| 335 |
-
# - NIM DeepSeek-V4-Flash: 284B/13B MoE, ~3× faster than V4-Pro, same
|
| 336 |
-
# clean JSON discipline. Used as fallback on V4-Pro ReadTimeout —
|
| 337 |
-
# different MoE routing path so a Pro timeout doesn't always reproduce
|
| 338 |
-
# in Flash. Sarvam-M was the old fallback but its <think> tags ate the
|
| 339 |
-
# 2048 output cap before reaching JSON, causing ~20 truncation failures.
|
| 340 |
-
# Three-chain collapse (2026-05-15): the separate FAST_BRAIN_CHAIN /
|
| 341 |
-
# get_fast_brain_llm accessor was removed. One brain chain (get_brain_llm)
|
| 342 |
-
# whose NimChainLLM already does internal multi-candidate fallback;
|
| 343 |
-
# primary/fallback both resolve to it.
|
| 344 |
primary = get_brain_llm()
|
| 345 |
fallback = get_brain_llm()
|
| 346 |
|
|
|
|
| 2 |
|
| 3 |
For each PDF in rag/corpus/:
|
| 4 |
1. Read full text via pdfplumber (already have per-page text from ingest)
|
| 5 |
+
2. Pass to the brain LLM (get_brain_llm) with the Pydantic schema as a
|
| 6 |
+
structured-output target. Sarvam-M is not in the reasoning path (it
|
| 7 |
+
is used for Indic translation only).
|
|
|
|
|
|
|
| 8 |
3. Self-critique pass — LLM scores per-field confidence vs source text
|
| 9 |
4. Validate via Pydantic
|
| 10 |
5. Upsert into DuckDB `policies` table
|
|
|
|
| 71 |
|
| 72 |
|
| 73 |
def build_extract_prompt(policy_text: str, schema_excerpt: str, policy_id: str) -> str:
|
| 74 |
+
# The policy text is capped at 12k chars (~3k tokens): that captures
|
| 75 |
+
# the Schedule + Key Definitions + Benefits front-matter where the
|
| 76 |
+
# structured fields live. The Exclusions section (the back half)
|
| 77 |
+
# doesn't yield many structured fields, just the policy_exclusions
|
| 78 |
+
# list which is mostly noise.
|
|
|
|
|
|
|
| 79 |
MAX_CHARS = 12_000
|
| 80 |
if len(policy_text) > MAX_CHARS:
|
| 81 |
# Front-bias: schedules, definitions, waiting periods, UIN, sum-insured
|
|
|
|
| 139 |
def json_from_llm_text(text: str) -> dict:
|
| 140 |
"""Strip code fences and <think> blocks, extract the first balanced {...} block."""
|
| 141 |
text = text.strip()
|
| 142 |
+
# Reasoning models emit <think>...</think> before the JSON; strip them
|
| 143 |
+
# (handles both closed and unterminated think blocks when output is
|
| 144 |
# truncated mid-thought).
|
| 145 |
text = re.sub(r"<think>.*?</think>", "", text, flags=re.DOTALL)
|
| 146 |
text = re.sub(r"<think>.*", "", text, flags=re.DOTALL)
|
|
|
|
| 261 |
for attempt, llm in enumerate([llm_primary, llm_fallback]):
|
| 262 |
try:
|
| 263 |
# Hard per-attempt timeout so a hung TCP connection in httpx
|
| 264 |
+
# pooling can't stall the whole sweep. The primary gets a larger
|
| 265 |
+
# ceiling than the fallback.
|
|
|
|
|
|
|
| 266 |
attempt_timeout = 180 if llm is llm_primary else 120
|
| 267 |
res = await asyncio.wait_for(
|
| 268 |
llm.chat(messages=messages, temperature=0.0, max_tokens=2048),
|
|
|
|
| 322 |
if args.limit:
|
| 323 |
pdfs = pdfs[: args.limit]
|
| 324 |
|
| 325 |
+
# get_brain_llm() returns a NimChainLLM that already does internal
|
| 326 |
+
# multi-candidate fallback, so primary and fallback both resolve to it.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 327 |
primary = get_brain_llm()
|
| 328 |
fallback = get_brain_llm()
|
| 329 |
|
|
@@ -31,25 +31,19 @@ def _safe_collection_get(
|
|
| 31 |
where: Optional[dict] = None,
|
| 32 |
include: Optional[list[str]] = None,
|
| 33 |
) -> Optional[dict]:
|
| 34 |
-
"""
|
| 35 |
-
|
| 36 |
-
|
| 37 |
-
|
| 38 |
-
|
| 39 |
-
|
| 40 |
-
|
| 41 |
-
sqlite lock contention during HNSW compaction). The pre-KI-107 code
|
| 42 |
-
wrapped these calls in bare `except: pass`, which silently masked the
|
| 43 |
-
failure and let downstream callers index into None — producing the
|
| 44 |
-
confusing "Error finding id" plan-executor message instead of a
|
| 45 |
-
coherent retrieval fallback.
|
| 46 |
|
| 47 |
This helper:
|
| 48 |
- Returns None on ANY exception (caller MUST treat None as "miss").
|
| 49 |
- Returns the raw dict (may have empty lists) on success.
|
| 50 |
-
- Logs at WARNING level so the failure is observable
|
| 51 |
-
|
| 52 |
-
regressions.
|
| 53 |
"""
|
| 54 |
try:
|
| 55 |
kwargs: dict = {}
|
|
@@ -149,7 +143,7 @@ def _build_chunk(cid: str, doc: str, meta: dict, score: float) -> RetrievedChunk
|
|
| 149 |
)
|
| 150 |
|
| 151 |
|
| 152 |
-
#
|
| 153 |
# (query_text, top_k, sorted policy_ids, sorted insurer_slugs). Caps at 256
|
| 154 |
# entries with FIFO eviction so memory stays bounded across long sessions.
|
| 155 |
# Within a single chat session, users frequently rephrase or follow up on the
|
|
@@ -170,8 +164,8 @@ def _cache_key(
|
|
| 170 |
insurer_slugs: Optional[list[str]],
|
| 171 |
session_id: Optional[str] = None,
|
| 172 |
) -> tuple:
|
| 173 |
-
#
|
| 174 |
-
#
|
| 175 |
# supplied the quarantine boost pass prepends that session's uploaded
|
| 176 |
# PDF chunks. Without session_id in the key, a result computed for
|
| 177 |
# session A (carrying A's private uploaded chunks) would be served
|
|
@@ -223,13 +217,12 @@ async def retrieve(
|
|
| 223 |
This ensures the brain sees regulatory ceilings even when policy
|
| 224 |
chunks dominate raw cosine.
|
| 225 |
|
| 226 |
-
|
| 227 |
-
|
| 228 |
-
|
| 229 |
-
quarantine (user-uploaded PDF) lookup but is no longer used for
|
| 230 |
profile boosting.
|
| 231 |
"""
|
| 232 |
-
#
|
| 233 |
# session_id is part of the key so one session's cached result (which
|
| 234 |
# may carry that session's private quarantine chunks) is never served
|
| 235 |
# to another session.
|
|
@@ -238,15 +231,12 @@ async def retrieve(
|
|
| 238 |
if cached is not None:
|
| 239 |
return cached
|
| 240 |
|
| 241 |
-
#
|
| 242 |
-
#
|
| 243 |
-
#
|
| 244 |
-
#
|
| 245 |
-
#
|
| 246 |
-
#
|
| 247 |
-
# queries clearly asking about a structured cap, retrieve more
|
| 248 |
-
# candidates so the table chunk has a higher chance of landing in
|
| 249 |
-
# the LLM's context window.
|
| 250 |
_TABLE_CELL_TRIGGERS = _re.compile(
|
| 251 |
r"\b(room\s*rent|sub[\s\-]?limit|sub[\s\-]?limits|copay|co[\s\-]?pay|"
|
| 252 |
r"cap\s+on|capped\s+at|sum\s+insured\s+limit|"
|
|
@@ -261,13 +251,12 @@ async def retrieve(
|
|
| 261 |
embedder = embedder or VoyageEmbeddings()
|
| 262 |
[query_vec] = await embedder.embed([query], input_type="query")
|
| 263 |
|
| 264 |
-
#
|
| 265 |
-
#
|
| 266 |
-
#
|
| 267 |
-
#
|
| 268 |
-
#
|
| 269 |
-
# profile into the current session
|
| 270 |
-
# B4 retrieving profile chunks from smokeA_1, ki100_ve, smokeB_B2).
|
| 271 |
#
|
| 272 |
# We translate this to Chroma's `where` DSL using $ne (not-equals) on
|
| 273 |
# doc_type. Combined with optional policy_id / insurer_slug filters via
|
|
@@ -284,17 +273,13 @@ async def retrieve(
|
|
| 284 |
|
| 285 |
collection = get_collection()
|
| 286 |
|
| 287 |
-
# Standard retrieval —
|
| 288 |
-
#
|
| 289 |
-
#
|
| 290 |
-
#
|
| 291 |
-
#
|
| 292 |
-
#
|
| 293 |
-
#
|
| 294 |
-
# has the same failure mode. On error we degrade to empty retrieval:
|
| 295 |
-
# the brain still answers (acknowledging it can't access the catalog)
|
| 296 |
-
# instead of the user seeing KI-106's generic "something went wrong"
|
| 297 |
-
# fallback for every message.
|
| 298 |
out: list[RetrievedChunk] = []
|
| 299 |
try:
|
| 300 |
res = collection.query(
|
|
@@ -316,24 +301,20 @@ async def retrieve(
|
|
| 316 |
effective_top_k, where, type(e).__name__, str(e)[:300],
|
| 317 |
)
|
| 318 |
|
| 319 |
-
# Profile boost pass
|
| 320 |
-
#
|
| 321 |
-
#
|
| 322 |
-
#
|
| 323 |
if profile_name_slug:
|
| 324 |
profile_chunk_id = f"profile_{profile_name_slug}"
|
| 325 |
-
#
|
| 326 |
-
#
|
| 327 |
-
#
|
| 328 |
#
|
| 329 |
-
#
|
| 330 |
-
#
|
| 331 |
-
#
|
| 332 |
-
#
|
| 333 |
-
# `except: pass` masked the failure, the orchestrator's plan-executor
|
| 334 |
-
# still saw an AttributeError on the swallowed-None path, and the
|
| 335 |
-
# user got HTTP 500 "Error executing plan: Internal error: Error
|
| 336 |
-
# finding id".
|
| 337 |
prof_res = _safe_collection_get(
|
| 338 |
collection,
|
| 339 |
ids=[profile_chunk_id],
|
|
@@ -355,7 +336,7 @@ async def retrieve(
|
|
| 355 |
out = [profile_chunk] + [c for c in out if c.chunk_id != profile_chunk_id]
|
| 356 |
out = out[:top_k]
|
| 357 |
|
| 358 |
-
# Quarantine boost pass — when the
|
| 359 |
# query the SEPARATE quarantine collection (user-uploaded PDFs) scoped to
|
| 360 |
# this session. Results get a ×1.1 score boost so the user's own uploaded
|
| 361 |
# doc ranks above generic policies when relevant. Quarantine lives in a
|
|
|
|
| 31 |
where: Optional[dict] = None,
|
| 32 |
include: Optional[list[str]] = None,
|
| 33 |
) -> Optional[dict]:
|
| 34 |
+
"""Defensive wrapper around `collection.get(...)`.
|
| 35 |
+
|
| 36 |
+
Chroma's `collection.get(ids=[...])` is documented to return empty
|
| 37 |
+
lists when ids miss, but the Rust-backed client can still raise on
|
| 38 |
+
certain edge cases (legacy schema rows, where-clause + missing-id
|
| 39 |
+
combinations, transient sqlite lock contention during HNSW
|
| 40 |
+
compaction).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 41 |
|
| 42 |
This helper:
|
| 43 |
- Returns None on ANY exception (caller MUST treat None as "miss").
|
| 44 |
- Returns the raw dict (may have empty lists) on success.
|
| 45 |
+
- Logs at WARNING level so the failure is observable instead of
|
| 46 |
+
swallowed silently.
|
|
|
|
| 47 |
"""
|
| 48 |
try:
|
| 49 |
kwargs: dict = {}
|
|
|
|
| 143 |
)
|
| 144 |
|
| 145 |
|
| 146 |
+
# In-process retrieval cache. Keyed by
|
| 147 |
# (query_text, top_k, sorted policy_ids, sorted insurer_slugs). Caps at 256
|
| 148 |
# entries with FIFO eviction so memory stays bounded across long sessions.
|
| 149 |
# Within a single chat session, users frequently rephrase or follow up on the
|
|
|
|
| 164 |
insurer_slugs: Optional[list[str]],
|
| 165 |
session_id: Optional[str] = None,
|
| 166 |
) -> tuple:
|
| 167 |
+
# session_id MUST be part of the cache key (quarantine isolation).
|
| 168 |
+
# The result set is session-dependent: when a session_id is
|
| 169 |
# supplied the quarantine boost pass prepends that session's uploaded
|
| 170 |
# PDF chunks. Without session_id in the key, a result computed for
|
| 171 |
# session A (carrying A's private uploaded chunks) would be served
|
|
|
|
| 217 |
This ensures the brain sees regulatory ceilings even when policy
|
| 218 |
chunks dominate raw cosine.
|
| 219 |
|
| 220 |
+
Profile chunks are keyed by `profile_name_slug` (canonical user name);
|
| 221 |
+
only named users have a profile chunk to boost. `session_id` is used
|
| 222 |
+
for the per-session quarantine (user-uploaded PDF) lookup, not for
|
|
|
|
| 223 |
profile boosting.
|
| 224 |
"""
|
| 225 |
+
# Short-circuit identical-query re-asks via the LRU cache.
|
| 226 |
# session_id is part of the key so one session's cached result (which
|
| 227 |
# may carry that session's private quarantine chunks) is never served
|
| 228 |
# to another session.
|
|
|
|
| 231 |
if cached is not None:
|
| 232 |
return cached
|
| 233 |
|
| 234 |
+
# Wider net for table-cell questions. The policy table holding the
|
| 235 |
+
# cap structure (room-rent / sub-limit / room category) is one chunk
|
| 236 |
+
# in Chroma and at top_k=5 it can lose to other relevant prose
|
| 237 |
+
# chunks. For queries clearly asking about a structured cap, retrieve
|
| 238 |
+
# more candidates so the table chunk has a higher chance of landing
|
| 239 |
+
# in the LLM's context window.
|
|
|
|
|
|
|
|
|
|
| 240 |
_TABLE_CELL_TRIGGERS = _re.compile(
|
| 241 |
r"\b(room\s*rent|sub[\s\-]?limit|sub[\s\-]?limits|copay|co[\s\-]?pay|"
|
| 242 |
r"cap\s+on|capped\s+at|sum\s+insured\s+limit|"
|
|
|
|
| 251 |
embedder = embedder or VoyageEmbeddings()
|
| 252 |
[query_vec] = await embedder.embed([query], input_type="query")
|
| 253 |
|
| 254 |
+
# Privacy invariant: the main retrieval pass MUST NEVER return chunks
|
| 255 |
+
# of doc_type='profile' because those chunks are scoped to a specific
|
| 256 |
+
# user. Profile chunks are exclusively surfaced via the explicit
|
| 257 |
+
# per-session collection.get(ids=[f"profile_{session_id}"]) lookup
|
| 258 |
+
# below; allowing them through the cosine pass would leak another
|
| 259 |
+
# user's profile into the current session.
|
|
|
|
| 260 |
#
|
| 261 |
# We translate this to Chroma's `where` DSL using $ne (not-equals) on
|
| 262 |
# doc_type. Combined with optional policy_id / insurer_slug filters via
|
|
|
|
| 273 |
|
| 274 |
collection = get_collection()
|
| 275 |
|
| 276 |
+
# Standard retrieval — collection.query() is wrapped with the same
|
| 277 |
+
# defensive pattern as _safe_collection_get because .query() can raise
|
| 278 |
+
# `chromadb.errors.InternalError: Error executing plan` when an HNSW
|
| 279 |
+
# index entry points to a missing doc. On error we degrade to empty
|
| 280 |
+
# retrieval: the brain still answers (acknowledging it can't access
|
| 281 |
+
# the catalog) instead of the user seeing a generic "something went
|
| 282 |
+
# wrong" fallback for every message.
|
|
|
|
|
|
|
|
|
|
|
|
|
| 283 |
out: list[RetrievedChunk] = []
|
| 284 |
try:
|
| 285 |
res = collection.query(
|
|
|
|
| 301 |
effective_top_k, where, type(e).__name__, str(e)[:300],
|
| 302 |
)
|
| 303 |
|
| 304 |
+
# Profile boost pass. Profile chunks are keyed by `profile_name_slug`
|
| 305 |
+
# (canonical user name), not session_id. The caller passes the slug
|
| 306 |
+
# only when the live session has a known name; anonymous chats skip
|
| 307 |
+
# this branch entirely.
|
| 308 |
if profile_name_slug:
|
| 309 |
profile_chunk_id = f"profile_{profile_name_slug}"
|
| 310 |
+
# Filter by BOTH id AND name_slug metadata so any ID collision
|
| 311 |
+
# (or migration-era chunk written under a shared id) can't leak
|
| 312 |
+
# the wrong profile into this user's context.
|
| 313 |
#
|
| 314 |
+
# Route through _safe_collection_get because first-time named
|
| 315 |
+
# users (no profile saved yet) and certain transient Chroma
|
| 316 |
+
# sqlite states cause collection.get(ids=[missing_id]) to raise
|
| 317 |
+
# instead of returning empty lists.
|
|
|
|
|
|
|
|
|
|
|
|
|
| 318 |
prof_res = _safe_collection_get(
|
| 319 |
collection,
|
| 320 |
ids=[profile_chunk_id],
|
|
|
|
| 336 |
out = [profile_chunk] + [c for c in out if c.chunk_id != profile_chunk_id]
|
| 337 |
out = out[:top_k]
|
| 338 |
|
| 339 |
+
# Quarantine boost pass — when the caller passes a session_id, also
|
| 340 |
# query the SEPARATE quarantine collection (user-uploaded PDFs) scoped to
|
| 341 |
# this session. Results get a ×1.1 score boost so the user's own uploaded
|
| 342 |
# doc ranks above generic policies when relevant. Quarantine lives in a
|
|
@@ -30,9 +30,9 @@ ROOT = Path(__file__).resolve().parent.parent
|
|
| 30 |
RESULTS_JSON = ROOT / "eval" / "chunk_sweep_results.json"
|
| 31 |
RESULTS_MD = ROOT / "kb" / "calculations" / "chunk_sweep_results.md"
|
| 32 |
|
| 33 |
-
# Sweep grid.
|
| 34 |
-
#
|
| 35 |
-
#
|
| 36 |
GRID = [
|
| 37 |
(400, 60),
|
| 38 |
(600, 100),
|
|
|
|
| 30 |
RESULTS_JSON = ROOT / "eval" / "chunk_sweep_results.json"
|
| 31 |
RESULTS_MD = ROOT / "kb" / "calculations" / "chunk_sweep_results.md"
|
| 32 |
|
| 33 |
+
# Sweep grid. Spans a ~6x range so chunk-boundary effects on retrieval
|
| 34 |
+
# are visible (a narrow grid converges to identical scores because
|
| 35 |
+
# retrieval is insensitive within a small band).
|
| 36 |
GRID = [
|
| 37 |
(400, 60),
|
| 38 |
(600, 100),
|
|
@@ -1,10 +1,9 @@
|
|
| 1 |
"""Programmatically set HF Space secrets via the official HF API.
|
| 2 |
|
| 3 |
No browser needed. Reads keys from local .env and pushes them as Space
|
| 4 |
-
secrets so the deployed app can authenticate.
|
| 5 |
-
|
| 6 |
-
|
| 7 |
-
to prevent confusion (they are no longer referenced by the code).
|
| 8 |
|
| 9 |
Run:
|
| 10 |
python tools/set_hf_secrets.py
|
|
@@ -23,15 +22,15 @@ ROOT = Path(__file__).resolve().parent.parent
|
|
| 23 |
load_dotenv(ROOT / ".env")
|
| 24 |
|
| 25 |
REPO_ID = "rohitsar567/InsuranceBot"
|
| 26 |
-
# Active secrets read by the running code
|
| 27 |
-
# - SARVAM / NIM = the live providers
|
| 28 |
-
# - VOYAGE
|
| 29 |
-
# - OPENROUTER / GROQ
|
| 30 |
-
# - ADMIN_*
|
| 31 |
SECRETS_TO_SET = [
|
| 32 |
"SARVAM_API_KEY", "VOYAGE_API_KEY", "NVIDIA_NIM_API_KEY",
|
| 33 |
"OPENROUTER_API_KEY", "GROQ_API_KEY",
|
| 34 |
-
"GOOGLE_API_KEY", #
|
| 35 |
"ADMIN_IP_ALLOWLIST", "ADMIN_PASSWORD",
|
| 36 |
]
|
| 37 |
# Truly retired providers — delete from Space to prevent confusion.
|
|
|
|
| 1 |
"""Programmatically set HF Space secrets via the official HF API.
|
| 2 |
|
| 3 |
No browser needed. Reads keys from local .env and pushes them as Space
|
| 4 |
+
secrets so the deployed app can authenticate. Only the keys in
|
| 5 |
+
SECRETS_TO_SET below are pushed; any other provider keys are deleted
|
| 6 |
+
from the Space to prevent confusion.
|
|
|
|
| 7 |
|
| 8 |
Run:
|
| 9 |
python tools/set_hf_secrets.py
|
|
|
|
| 22 |
load_dotenv(ROOT / ".env")
|
| 23 |
|
| 24 |
REPO_ID = "rohitsar567/InsuranceBot"
|
| 25 |
+
# Active secrets read by the running code:
|
| 26 |
+
# - SARVAM / NIM / GOOGLE = the live providers
|
| 27 |
+
# - VOYAGE = kept for back-compat with extracted artifacts
|
| 28 |
+
# - OPENROUTER / GROQ = optional cross-provider fallbacks
|
| 29 |
+
# - ADMIN_* = control-panel gate
|
| 30 |
SECRETS_TO_SET = [
|
| 31 |
"SARVAM_API_KEY", "VOYAGE_API_KEY", "NVIDIA_NIM_API_KEY",
|
| 32 |
"OPENROUTER_API_KEY", "GROQ_API_KEY",
|
| 33 |
+
"GOOGLE_API_KEY", # Gemini Flash via Google AI Studio
|
| 34 |
"ADMIN_IP_ALLOWLIST", "ADMIN_PASSWORD",
|
| 35 |
]
|
| 36 |
# Truly retired providers — delete from Space to prevent confusion.
|
|
@@ -1,16 +1,16 @@
|
|
| 1 |
-
"""One-shot uploader to Hugging Face Space
|
| 2 |
|
| 3 |
Pushes the code-only Space (minus build artifacts, .venv, node_modules, etc.) to
|
| 4 |
-
huggingface.co/spaces/rohitsar567/InsuranceBot.
|
| 5 |
-
|
| 6 |
dataset rohitsar567/insurance-bot-data — the Space's Dockerfile pulls it via
|
| 7 |
huggingface_hub.snapshot_download at image build time.
|
| 8 |
|
| 9 |
-
|
| 10 |
-
|
| 11 |
-
DELETES any remote file
|
| 12 |
-
(renamed/deleted brochures) from lingering on the Space and being served
|
| 13 |
-
the Docker image after a rebuild.
|
| 14 |
|
| 15 |
Run:
|
| 16 |
python tools/upload_to_hf.py # full upload + sync delete
|
|
@@ -57,10 +57,9 @@ IGNORE = [
|
|
| 57 |
"rag/corpus/**",
|
| 58 |
"rag/vectors/**",
|
| 59 |
"rag/extracted/**",
|
| 60 |
-
#
|
| 61 |
-
# Space
|
| 62 |
-
#
|
| 63 |
-
# post_lfs_batch_info). The data path is HF Dataset, NOT the Space.
|
| 64 |
"rag/_hf_dataset_backup/**",
|
| 65 |
"**/_hf_dataset_backup/**",
|
| 66 |
"rag/policies.duckdb", # LFS-tracked, comes from dataset at build
|
|
|
|
| 1 |
+
"""One-shot code-only uploader to the Hugging Face Space.
|
| 2 |
|
| 3 |
Pushes the code-only Space (minus build artifacts, .venv, node_modules, etc.) to
|
| 4 |
+
huggingface.co/spaces/rohitsar567/InsuranceBot. The large data (corpus
|
| 5 |
+
PDFs + Chroma vectors + extracted JSONs) lives in the companion HF
|
| 6 |
dataset rohitsar567/insurance-bot-data — the Space's Dockerfile pulls it via
|
| 7 |
huggingface_hub.snapshot_download at image build time.
|
| 8 |
|
| 9 |
+
Operates in SYNC mode: after `upload_folder` (add/replace), walks the
|
| 10 |
+
local tree, lists the remote repo, and issues a second commit that
|
| 11 |
+
DELETES any remote file not present locally. Prevents ghost files
|
| 12 |
+
(renamed/deleted brochures) from lingering on the Space and being served
|
| 13 |
+
by the Docker image after a rebuild.
|
| 14 |
|
| 15 |
Run:
|
| 16 |
python tools/upload_to_hf.py # full upload + sync delete
|
|
|
|
| 57 |
"rag/corpus/**",
|
| 58 |
"rag/vectors/**",
|
| 59 |
"rag/extracted/**",
|
| 60 |
+
# Exclude the backup dir + LFS pointers: pushing them would blow the
|
| 61 |
+
# Space's 1 GB LFS storage quota. The data path is the HF Dataset,
|
| 62 |
+
# NOT the Space.
|
|
|
|
| 63 |
"rag/_hf_dataset_backup/**",
|
| 64 |
"**/_hf_dataset_backup/**",
|
| 65 |
"rag/policies.duckdb", # LFS-tracked, comes from dataset at build
|
|
@@ -2,8 +2,8 @@
|
|
| 2 |
|
| 3 |
Run AFTER `.venv/bin/python -m rag.ingest` on the developer Mac. The Space's
|
| 4 |
Dockerfile pulls rag/vectors/** at build time so the deployed container has a
|
| 5 |
-
ready-to-serve index
|
| 6 |
-
|
| 7 |
"""
|
| 8 |
from __future__ import annotations
|
| 9 |
import os
|
|
|
|
| 2 |
|
| 3 |
Run AFTER `.venv/bin/python -m rag.ingest` on the developer Mac. The Space's
|
| 4 |
Dockerfile pulls rag/vectors/** at build time so the deployed container has a
|
| 5 |
+
ready-to-serve index; the Space does not auto-ingest on boot (see
|
| 6 |
+
entrypoint.sh).
|
| 7 |
"""
|
| 8 |
from __future__ import annotations
|
| 9 |
import os
|