rohitsar567 Claude Opus 4.7 (1M context) commited on
Commit
87ee522
·
1 Parent(s): 078ff45

feat(llm-chain): KI-079 — Groq earlier in FAST_BRAIN_CHAIN + escalate to BRAIN_CHAIN on timeout before canonical fallback

Browse files

Live 10-turn probe (commit 078ff45) showed 7/10 fact-find turns timing
out at exactly 26.6s with _fallback_reason="timeout" — ALL failures are
real timeouts, not parse errors. Root cause: FAST_BRAIN_CHAIN's first 5
candidates were all NIM-hosted; under NIM per-key concurrency limits
they queued together and the chain burned its 22s total_budget_s inside
NIM before reaching Groq Llama-3.3 70B at the chain bottom.

FIX 1 (nvidia_nim_llm.py)
- Move groq:llama-3.3-70b-versatile from FAST_BRAIN_CHAIN[5] (last) to
FAST_BRAIN_CHAIN[1] (right after Nemotron primary). All other
candidates kept. Nemotron remains primary (KI-035 latency win).
- New worst-case time-to-cross-provider on a NIM hang: ~6-7s (per-link
6s timeout × 1 link) vs the prior ~22s.

FIX 2 (fact_find_brain.py)
- Add _TIMEOUT_S_ESCALATION = 15.0 alongside existing _TIMEOUT_S = 25.0.
- On asyncio.TimeoutError from FAST_BRAIN_CHAIN, escalate ONCE to
get_brain_llm() (BRAIN_CHAIN — Qwen 80B primary + 8 fallbacks incl.
OpenRouter + Groq) inside a shorter 15s budget before falling to the
canonical-question fallback. New _fallback_reason values:
"timeout_after_escalation" and "llm_error_after_escalation" — the
KI-078 telemetry stamp already wires these into brain_used.
- Worst-case end-to-end wall time before canonical: 25 + 15 = 40s, but
realistic escalation-success path lands in 3-8s because the heavy
brain hits a different NIM pool primary (Qwen 80B) and falls through
to OpenRouter / Groq on its own.

NOT TOUCHED
- _TIMEOUT_S = 25.0 (kept; KI-079 is additive)
- KI-074 greedy multi-slot canonical fallback
- frontend/**
- CLAUDE.md, 80-audit/ENTERPRISE_AUDIT.md (separate sync commit)

VERIFICATION
- python3 -m py_compile both files: OK
- Inline test: FactFindOutcome accepts both new _fallback_reason values
- Inline test: FAST_BRAIN_CHAIN[0:3] = ['nvidia/nemotron-3-nano-30b-a3b',
'groq:llama-3.3-70b-versatile', 'qwen/qwen3-next-80b-a3b-instruct']
- pytest tests/test_routing_regression.py -x -q: 15 passed, 13 subtests

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

backend/fact_find_brain.py CHANGED
@@ -59,11 +59,17 @@ class FactFindOutcome:
59
  # bailed and `_canonical_fallback` was used, this stamps WHY so the
60
  # orchestrator can append it to `brain_used` and admin telemetry can
61
  # measure the fallback-reason mix. One of:
62
- # "timeout" — asyncio.wait_for(_TIMEOUT_S) expired
63
- # "llm_error" — chain raised (non-timeout) before returning
64
  # "no_trailer" — reply had no <FF>...</FF> JSON block, or it failed parse
65
  # "empty_reply" — trailer stripped to an empty user-facing reply
66
- # None when the brain succeeded.
 
 
 
 
 
 
67
  _fallback_reason: Optional[str] = None
68
 
69
 
@@ -293,7 +299,15 @@ def _bump_brain_history(session, slot_driving: Optional[str]) -> int:
293
  # after a Space rebuild; the 12s wait_for was killing the brain BEFORE
294
  # the cross-provider fallback links (Groq, OpenRouter) ever got tried.
295
  # 25s gives NIM cold-start headroom + leaves room for one chain fallback.
296
- _TIMEOUT_S = 25.0
 
 
 
 
 
 
 
 
297
 
298
 
299
  async def drive_fact_find(
@@ -345,19 +359,47 @@ async def drive_fact_find(
345
  messages.append(ChatMessage(role=role, content=content))
346
  messages.append(ChatMessage(role="user", content=user_text or ""))
347
 
348
- # Hard 12-second timeout. The fast-brain chain already has its own budget
349
- # but we wrap with asyncio.wait_for as a belt-and-braces stop.
350
- llm = get_fast_brain_llm()
 
351
  try:
352
  result = await asyncio.wait_for(
353
- llm.chat(messages=messages, temperature=0.6, max_tokens=420),
354
  timeout=_TIMEOUT_S,
355
  )
356
  except asyncio.TimeoutError:
 
 
 
 
357
  logging.warning(
358
- "fact_find_brain timeout (session=%s, %.1fs)", session_id, time.time() - t0
 
359
  )
360
- return _canonical_fallback(session, user_text, reason="timeout")
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
361
  except Exception as e:
362
  logging.warning(
363
  "fact_find_brain LLM call failed (session=%s): %s: %s",
 
59
  # bailed and `_canonical_fallback` was used, this stamps WHY so the
60
  # orchestrator can append it to `brain_used` and admin telemetry can
61
  # measure the fallback-reason mix. One of:
62
+ # "timeout" — asyncio.wait_for(_TIMEOUT_S) expired on FAST_BRAIN_CHAIN
63
+ # "llm_error" — FAST_BRAIN_CHAIN raised (non-timeout) before returning
64
  # "no_trailer" — reply had no <FF>...</FF> JSON block, or it failed parse
65
  # "empty_reply" — trailer stripped to an empty user-facing reply
66
+ # KI-079 (2026-05-15) added two escalation-path reasons. When FAST
67
+ # chain times out we re-try once against BRAIN_CHAIN (heavier; Qwen 80B
68
+ # primary, more cross-provider fallbacks) inside a shorter budget. The
69
+ # _fallback_reason then captures the escalation outcome:
70
+ # "timeout_after_escalation" — both FAST + BRAIN chains timed out
71
+ # "llm_error_after_escalation" — BRAIN chain raised on the retry
72
+ # None when the brain succeeded (either FAST primary OR BRAIN escalation).
73
  _fallback_reason: Optional[str] = None
74
 
75
 
 
299
  # after a Space rebuild; the 12s wait_for was killing the brain BEFORE
300
  # the cross-provider fallback links (Groq, OpenRouter) ever got tried.
301
  # 25s gives NIM cold-start headroom + leaves room for one chain fallback.
302
+ _TIMEOUT_S = 25.0 # FAST_BRAIN_CHAIN primary attempt budget
303
+ # KI-079 (2026-05-15) — if FAST exhausted, try BRAIN_CHAIN (Qwen 80B primary
304
+ # + more cross-provider fallbacks incl. OpenRouter + Groq) with a SHORTER
305
+ # budget so the user doesn't wait 25+15=40s on a fully-dead network. Total
306
+ # worst-case latency before canonical fallback: 25 + 15 = 40s, but the FAST
307
+ # chain only hits 25s when NIM is wedged — the BRAIN escalation then has a
308
+ # Qwen primary on a different NIM pool + OpenRouter + Groq, so realistic
309
+ # escalation success cases land in 3-8s.
310
+ _TIMEOUT_S_ESCALATION = 15.0
311
 
312
 
313
  async def drive_fact_find(
 
359
  messages.append(ChatMessage(role=role, content=content))
360
  messages.append(ChatMessage(role="user", content=user_text or ""))
361
 
362
+ # Hard 25-second timeout on FAST_BRAIN_CHAIN. The fast-brain chain already
363
+ # has its own per-link + total-chain budget but we wrap with asyncio.wait_for
364
+ # as a belt-and-braces stop.
365
+ llm_fast = get_fast_brain_llm()
366
  try:
367
  result = await asyncio.wait_for(
368
+ llm_fast.chat(messages=messages, temperature=0.6, max_tokens=420),
369
  timeout=_TIMEOUT_S,
370
  )
371
  except asyncio.TimeoutError:
372
+ # KI-079 (2026-05-15) — fast brain timed out. Before falling to the
373
+ # canonical-question fallback, escalate ONCE to BRAIN_CHAIN (Qwen 80B
374
+ # primary, more cross-provider fallbacks) with a shorter budget so the
375
+ # user doesn't wait 25+15=40s on a dead network.
376
  logging.warning(
377
+ "KI-079: fast brain timeout (session=%s, %.1fs) escalating to heavy brain",
378
+ session_id, time.time() - t0,
379
  )
380
+ from backend.providers.nvidia_nim_llm import get_brain_llm
381
+ llm_heavy = get_brain_llm()
382
+ try:
383
+ result = await asyncio.wait_for(
384
+ llm_heavy.chat(messages=messages, temperature=0.6, max_tokens=420),
385
+ timeout=_TIMEOUT_S_ESCALATION,
386
+ )
387
+ except asyncio.TimeoutError:
388
+ logging.warning(
389
+ "KI-079: heavy brain ALSO timed out (session=%s, total=%.1fs)",
390
+ session_id, time.time() - t0,
391
+ )
392
+ return _canonical_fallback(
393
+ session, user_text, reason="timeout_after_escalation"
394
+ )
395
+ except Exception as e2:
396
+ logging.warning(
397
+ "KI-079: heavy brain escalation failed (session=%s): %s: %s",
398
+ session_id, type(e2).__name__, str(e2)[:200],
399
+ )
400
+ return _canonical_fallback(
401
+ session, user_text, reason="llm_error_after_escalation"
402
+ )
403
  except Exception as e:
404
  logging.warning(
405
  "fact_find_brain LLM call failed (session=%s): %s: %s",
backend/providers/nvidia_nim_llm.py CHANGED
@@ -226,14 +226,25 @@ FAST_BRAIN_CHAIN = [
226
  # TTFT, not capability. Nemotron Nano 30B hits ~1.6s; Qwen 80B is
227
  # ~2-3s. Moved Nemotron to primary; Qwen 80B stays as next fallback so
228
  # if Nemotron's NIM pool degrades we still get quality.
 
 
 
 
 
 
 
 
 
 
229
  "nvidia/nemotron-3-nano-30b-a3b", # ~1.6s TTFT (Reddit bench), NIM
 
 
 
 
230
  "qwen/qwen3-next-80b-a3b-instruct", # ~2-3s, NIM
231
  "openai/gpt-oss-120b",
232
  "qwen/qwen3.5-122b-a10b",
233
  "deepseek-ai/deepseek-v4-flash",
234
- # CROSS-PROVIDER FALLBACK — Groq Llama-3.3 70B (LPU, lowest TTFT of all
235
- # free-tier options; OK for a fast-brain call when NIM is down).
236
- "groq:llama-3.3-70b-versatile",
237
  ]
238
 
239
  # Judge chain — non-Qwen, non-DeepSeek (different family from brain primary)
 
226
  # TTFT, not capability. Nemotron Nano 30B hits ~1.6s; Qwen 80B is
227
  # ~2-3s. Moved Nemotron to primary; Qwen 80B stays as next fallback so
228
  # if Nemotron's NIM pool degrades we still get quality.
229
+ #
230
+ # KI-079 (2026-05-15) — moved Groq Llama-3.3 70B from chain bottom to
231
+ # candidate #2 (right after Nemotron primary). Live 10-turn probe
232
+ # (commit 078ff45) showed 7/10 fact-find turns timing out at 26.6s with
233
+ # _fallback_reason="timeout" — ALL the slow links were NIM-hosted, so
234
+ # NIM per-key concurrency had them queueing together and the chain
235
+ # burned its 22s total_budget_s inside NIM before ever reaching Groq.
236
+ # With Groq as #2, a Nemotron hang (~6s per-link timeout) falls
237
+ # through to Groq's LPU (~0.3s TTFT) in ~6-7s total — well inside the
238
+ # 22s chain budget AND the 25s wait_for cap.
239
  "nvidia/nemotron-3-nano-30b-a3b", # ~1.6s TTFT (Reddit bench), NIM
240
+ # CROSS-PROVIDER FALLBACK #1 — Groq Llama-3.3 70B (LPU, lowest TTFT of
241
+ # all free-tier options). Promoted to #2 in KI-079 so a single NIM
242
+ # degradation falls through to a non-NIM provider in ~6s, not 22s.
243
+ "groq:llama-3.3-70b-versatile",
244
  "qwen/qwen3-next-80b-a3b-instruct", # ~2-3s, NIM
245
  "openai/gpt-oss-120b",
246
  "qwen/qwen3.5-122b-a10b",
247
  "deepseek-ai/deepseek-v4-flash",
 
 
 
248
  ]
249
 
250
  # Judge chain — non-Qwen, non-DeepSeek (different family from brain primary)