Spaces:
Sleeping
fix: KI-033 — move fact-find normalizer + profile extractor to NIM chain
Browse filesBoth modules were calling NvidiaNimLLM(model="meta/llama-3.3-70b-instruct")
as a hardcoded single-model client. When that specific NIM pool got
rate-limited (the 40 req/min cap is shared across all models on the same
key), the call raised, our except-Exception swallowed it, and the caller
got None — which:
• In fact_find_normalizer: caused the bot to mark the answer "ambiguous"
and reask. Two consecutive reasks → give up + move on with no slot
captured. This is the cascade pattern that produced D-005 (only 12/100
personas got age captured even though every persona was canonically
asked for age on turn 1).
• In profile_extractor: returned {} → no free-form profile updates
applied → recommendation flow gets a stale or empty profile.
Fix: replace single-model client with NimChainLLM(FAST_BRAIN_CHAIN, ...).
FAST_BRAIN_CHAIN already includes Qwen 80B, Nemotron Nano 30B, GPT-OSS,
Qwen 122B, DeepSeek V4-Flash, and Groq Llama-3.3-70B — so when NIM's
Llama pool is saturated, we fall through to Qwen or Groq automatically.
Per-link 10s, total budget 15s — bounded so a single ingest call can't
queue forever.
Combined with KI-025's NIM↔Groq primary rotation, the fact-find pipeline
now has TWO independent providers' rate quotas backing it up.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
@@ -338,13 +338,20 @@ Examples for guidance:
|
|
| 338 |
|
| 339 |
async def _llm_normalize(question_id: str, raw_text: str, schema: dict) -> Any:
|
| 340 |
from backend.providers.base import ChatMessage
|
| 341 |
-
from backend.providers.nvidia_nim_llm import
|
| 342 |
|
| 343 |
sys_msg = _LLM_SYSTEM_TEMPLATE.format(qid=question_id, schema=json.dumps(schema))
|
| 344 |
user_msg = f'User said: "{raw_text[:600]}"\n\nReturn the JSON value.'
|
| 345 |
|
| 346 |
try:
|
| 347 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 348 |
result = await llm.chat(
|
| 349 |
messages=[
|
| 350 |
ChatMessage(role="system", content=sys_msg),
|
|
|
|
| 338 |
|
| 339 |
async def _llm_normalize(question_id: str, raw_text: str, schema: dict) -> Any:
|
| 340 |
from backend.providers.base import ChatMessage
|
| 341 |
+
from backend.providers.nvidia_nim_llm import FAST_BRAIN_CHAIN, NimChainLLM
|
| 342 |
|
| 343 |
sys_msg = _LLM_SYSTEM_TEMPLATE.format(qid=question_id, schema=json.dumps(schema))
|
| 344 |
user_msg = f'User said: "{raw_text[:600]}"\n\nReturn the JSON value.'
|
| 345 |
|
| 346 |
try:
|
| 347 |
+
# KI-033 (2026-05-14) — was hardcoded NvidiaNimLLM(meta/llama-3.3-70b);
|
| 348 |
+
# moved to fast-brain chain so when that one NIM pool rate-limits we
|
| 349 |
+
# fall through to Qwen/Nemotron/Groq instead of silently returning None
|
| 350 |
+
# (which made valid Indian-accented answers like "twenty-five" appear
|
| 351 |
+
# to fail the verifier, then trip the 2-reask cap, then move on with
|
| 352 |
+
# no profile captured — the D-005 cascade).
|
| 353 |
+
llm = NimChainLLM(chain=FAST_BRAIN_CHAIN, timeout=10.0,
|
| 354 |
+
role="fact_find_normalizer", total_budget_s=15.0)
|
| 355 |
result = await llm.chat(
|
| 356 |
messages=[
|
| 357 |
ChatMessage(role="system", content=sys_msg),
|
|
@@ -72,8 +72,8 @@ async def extract_profile_updates(
|
|
| 72 |
if not user_text or not user_text.strip():
|
| 73 |
return {}
|
| 74 |
|
| 75 |
-
from backend.providers.nvidia_nim_llm import NvidiaNimLLM
|
| 76 |
from backend.providers.base import ChatMessage
|
|
|
|
| 77 |
|
| 78 |
summary_parts = []
|
| 79 |
for k, v in current_profile.__dict__.items():
|
|
@@ -94,7 +94,10 @@ async def extract_profile_updates(
|
|
| 94 |
]
|
| 95 |
|
| 96 |
try:
|
| 97 |
-
|
|
|
|
|
|
|
|
|
|
| 98 |
result = await llm.chat(messages=messages, temperature=0.0, max_tokens=300)
|
| 99 |
raw = (result.text or "").strip()
|
| 100 |
except Exception as e:
|
|
|
|
| 72 |
if not user_text or not user_text.strip():
|
| 73 |
return {}
|
| 74 |
|
|
|
|
| 75 |
from backend.providers.base import ChatMessage
|
| 76 |
+
from backend.providers.nvidia_nim_llm import FAST_BRAIN_CHAIN, NimChainLLM
|
| 77 |
|
| 78 |
summary_parts = []
|
| 79 |
for k, v in current_profile.__dict__.items():
|
|
|
|
| 94 |
]
|
| 95 |
|
| 96 |
try:
|
| 97 |
+
# KI-033 — was hardcoded NvidiaNimLLM(llama-3.3-70b); moved to chain so
|
| 98 |
+
# NIM pool failures fall through to Groq instead of silently returning {}.
|
| 99 |
+
llm = NimChainLLM(chain=FAST_BRAIN_CHAIN, timeout=10.0,
|
| 100 |
+
role="profile_extractor", total_budget_s=15.0)
|
| 101 |
result = await llm.chat(messages=messages, temperature=0.0, max_tokens=300)
|
| 102 |
raw = (result.text or "").strip()
|
| 103 |
except Exception as e:
|