rohitsar567 Claude Opus 4.7 (1M context) commited on
Commit
9a977de
·
1 Parent(s): 6f495c1

fix: KI-033 — move fact-find normalizer + profile extractor to NIM chain

Browse files

Both modules were calling NvidiaNimLLM(model="meta/llama-3.3-70b-instruct")
as a hardcoded single-model client. When that specific NIM pool got
rate-limited (the 40 req/min cap is shared across all models on the same
key), the call raised, our except-Exception swallowed it, and the caller
got None — which:

• In fact_find_normalizer: caused the bot to mark the answer "ambiguous"
and reask. Two consecutive reasks → give up + move on with no slot
captured. This is the cascade pattern that produced D-005 (only 12/100
personas got age captured even though every persona was canonically
asked for age on turn 1).
• In profile_extractor: returned {} → no free-form profile updates
applied → recommendation flow gets a stale or empty profile.

Fix: replace single-model client with NimChainLLM(FAST_BRAIN_CHAIN, ...).
FAST_BRAIN_CHAIN already includes Qwen 80B, Nemotron Nano 30B, GPT-OSS,
Qwen 122B, DeepSeek V4-Flash, and Groq Llama-3.3-70B — so when NIM's
Llama pool is saturated, we fall through to Qwen or Groq automatically.
Per-link 10s, total budget 15s — bounded so a single ingest call can't
queue forever.

Combined with KI-025's NIM↔Groq primary rotation, the fact-find pipeline
now has TWO independent providers' rate quotas backing it up.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

backend/fact_find_normalizer.py CHANGED
@@ -338,13 +338,20 @@ Examples for guidance:
338
 
339
  async def _llm_normalize(question_id: str, raw_text: str, schema: dict) -> Any:
340
  from backend.providers.base import ChatMessage
341
- from backend.providers.nvidia_nim_llm import NvidiaNimLLM
342
 
343
  sys_msg = _LLM_SYSTEM_TEMPLATE.format(qid=question_id, schema=json.dumps(schema))
344
  user_msg = f'User said: "{raw_text[:600]}"\n\nReturn the JSON value.'
345
 
346
  try:
347
- llm = NvidiaNimLLM(model="meta/llama-3.3-70b-instruct")
 
 
 
 
 
 
 
348
  result = await llm.chat(
349
  messages=[
350
  ChatMessage(role="system", content=sys_msg),
 
338
 
339
  async def _llm_normalize(question_id: str, raw_text: str, schema: dict) -> Any:
340
  from backend.providers.base import ChatMessage
341
+ from backend.providers.nvidia_nim_llm import FAST_BRAIN_CHAIN, NimChainLLM
342
 
343
  sys_msg = _LLM_SYSTEM_TEMPLATE.format(qid=question_id, schema=json.dumps(schema))
344
  user_msg = f'User said: "{raw_text[:600]}"\n\nReturn the JSON value.'
345
 
346
  try:
347
+ # KI-033 (2026-05-14) — was hardcoded NvidiaNimLLM(meta/llama-3.3-70b);
348
+ # moved to fast-brain chain so when that one NIM pool rate-limits we
349
+ # fall through to Qwen/Nemotron/Groq instead of silently returning None
350
+ # (which made valid Indian-accented answers like "twenty-five" appear
351
+ # to fail the verifier, then trip the 2-reask cap, then move on with
352
+ # no profile captured — the D-005 cascade).
353
+ llm = NimChainLLM(chain=FAST_BRAIN_CHAIN, timeout=10.0,
354
+ role="fact_find_normalizer", total_budget_s=15.0)
355
  result = await llm.chat(
356
  messages=[
357
  ChatMessage(role="system", content=sys_msg),
backend/profile_extractor.py CHANGED
@@ -72,8 +72,8 @@ async def extract_profile_updates(
72
  if not user_text or not user_text.strip():
73
  return {}
74
 
75
- from backend.providers.nvidia_nim_llm import NvidiaNimLLM
76
  from backend.providers.base import ChatMessage
 
77
 
78
  summary_parts = []
79
  for k, v in current_profile.__dict__.items():
@@ -94,7 +94,10 @@ async def extract_profile_updates(
94
  ]
95
 
96
  try:
97
- llm = NvidiaNimLLM(model="meta/llama-3.3-70b-instruct")
 
 
 
98
  result = await llm.chat(messages=messages, temperature=0.0, max_tokens=300)
99
  raw = (result.text or "").strip()
100
  except Exception as e:
 
72
  if not user_text or not user_text.strip():
73
  return {}
74
 
 
75
  from backend.providers.base import ChatMessage
76
+ from backend.providers.nvidia_nim_llm import FAST_BRAIN_CHAIN, NimChainLLM
77
 
78
  summary_parts = []
79
  for k, v in current_profile.__dict__.items():
 
94
  ]
95
 
96
  try:
97
+ # KI-033 — was hardcoded NvidiaNimLLM(llama-3.3-70b); moved to chain so
98
+ # NIM pool failures fall through to Groq instead of silently returning {}.
99
+ llm = NimChainLLM(chain=FAST_BRAIN_CHAIN, timeout=10.0,
100
+ role="profile_extractor", total_budget_s=15.0)
101
  result = await llm.chat(messages=messages, temperature=0.0, max_tokens=300)
102
  raw = (result.text or "").strip()
103
  except Exception as e: