Spaces:
Sleeping
feat(llm-chain): KI-079 — Groq earlier in FAST_BRAIN_CHAIN + escalate to BRAIN_CHAIN on timeout before canonical fallback
Browse filesLive 10-turn probe (commit 078ff45) showed 7/10 fact-find turns timing
out at exactly 26.6s with _fallback_reason="timeout" — ALL failures are
real timeouts, not parse errors. Root cause: FAST_BRAIN_CHAIN's first 5
candidates were all NIM-hosted; under NIM per-key concurrency limits
they queued together and the chain burned its 22s total_budget_s inside
NIM before reaching Groq Llama-3.3 70B at the chain bottom.
FIX 1 (nvidia_nim_llm.py)
- Move groq:llama-3.3-70b-versatile from FAST_BRAIN_CHAIN[5] (last) to
FAST_BRAIN_CHAIN[1] (right after Nemotron primary). All other
candidates kept. Nemotron remains primary (KI-035 latency win).
- New worst-case time-to-cross-provider on a NIM hang: ~6-7s (per-link
6s timeout × 1 link) vs the prior ~22s.
FIX 2 (fact_find_brain.py)
- Add _TIMEOUT_S_ESCALATION = 15.0 alongside existing _TIMEOUT_S = 25.0.
- On asyncio.TimeoutError from FAST_BRAIN_CHAIN, escalate ONCE to
get_brain_llm() (BRAIN_CHAIN — Qwen 80B primary + 8 fallbacks incl.
OpenRouter + Groq) inside a shorter 15s budget before falling to the
canonical-question fallback. New _fallback_reason values:
"timeout_after_escalation" and "llm_error_after_escalation" — the
KI-078 telemetry stamp already wires these into brain_used.
- Worst-case end-to-end wall time before canonical: 25 + 15 = 40s, but
realistic escalation-success path lands in 3-8s because the heavy
brain hits a different NIM pool primary (Qwen 80B) and falls through
to OpenRouter / Groq on its own.
NOT TOUCHED
- _TIMEOUT_S = 25.0 (kept; KI-079 is additive)
- KI-074 greedy multi-slot canonical fallback
- frontend/**
- CLAUDE.md, 80-audit/ENTERPRISE_AUDIT.md (separate sync commit)
VERIFICATION
- python3 -m py_compile both files: OK
- Inline test: FactFindOutcome accepts both new _fallback_reason values
- Inline test: FAST_BRAIN_CHAIN[0:3] = ['nvidia/nemotron-3-nano-30b-a3b',
'groq:llama-3.3-70b-versatile', 'qwen/qwen3-next-80b-a3b-instruct']
- pytest tests/test_routing_regression.py -x -q: 15 passed, 13 subtests
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- backend/fact_find_brain.py +52 -10
- backend/providers/nvidia_nim_llm.py +14 -3
|
@@ -59,11 +59,17 @@ class FactFindOutcome:
|
|
| 59 |
# bailed and `_canonical_fallback` was used, this stamps WHY so the
|
| 60 |
# orchestrator can append it to `brain_used` and admin telemetry can
|
| 61 |
# measure the fallback-reason mix. One of:
|
| 62 |
-
# "timeout" — asyncio.wait_for(_TIMEOUT_S) expired
|
| 63 |
-
# "llm_error" —
|
| 64 |
# "no_trailer" — reply had no <FF>...</FF> JSON block, or it failed parse
|
| 65 |
# "empty_reply" — trailer stripped to an empty user-facing reply
|
| 66 |
-
#
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 67 |
_fallback_reason: Optional[str] = None
|
| 68 |
|
| 69 |
|
|
@@ -293,7 +299,15 @@ def _bump_brain_history(session, slot_driving: Optional[str]) -> int:
|
|
| 293 |
# after a Space rebuild; the 12s wait_for was killing the brain BEFORE
|
| 294 |
# the cross-provider fallback links (Groq, OpenRouter) ever got tried.
|
| 295 |
# 25s gives NIM cold-start headroom + leaves room for one chain fallback.
|
| 296 |
-
_TIMEOUT_S = 25.0
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 297 |
|
| 298 |
|
| 299 |
async def drive_fact_find(
|
|
@@ -345,19 +359,47 @@ async def drive_fact_find(
|
|
| 345 |
messages.append(ChatMessage(role=role, content=content))
|
| 346 |
messages.append(ChatMessage(role="user", content=user_text or ""))
|
| 347 |
|
| 348 |
-
# Hard
|
| 349 |
-
# but we wrap with asyncio.wait_for
|
| 350 |
-
|
|
|
|
| 351 |
try:
|
| 352 |
result = await asyncio.wait_for(
|
| 353 |
-
|
| 354 |
timeout=_TIMEOUT_S,
|
| 355 |
)
|
| 356 |
except asyncio.TimeoutError:
|
|
|
|
|
|
|
|
|
|
|
|
|
| 357 |
logging.warning(
|
| 358 |
-
"
|
|
|
|
| 359 |
)
|
| 360 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 361 |
except Exception as e:
|
| 362 |
logging.warning(
|
| 363 |
"fact_find_brain LLM call failed (session=%s): %s: %s",
|
|
|
|
| 59 |
# bailed and `_canonical_fallback` was used, this stamps WHY so the
|
| 60 |
# orchestrator can append it to `brain_used` and admin telemetry can
|
| 61 |
# measure the fallback-reason mix. One of:
|
| 62 |
+
# "timeout" — asyncio.wait_for(_TIMEOUT_S) expired on FAST_BRAIN_CHAIN
|
| 63 |
+
# "llm_error" — FAST_BRAIN_CHAIN raised (non-timeout) before returning
|
| 64 |
# "no_trailer" — reply had no <FF>...</FF> JSON block, or it failed parse
|
| 65 |
# "empty_reply" — trailer stripped to an empty user-facing reply
|
| 66 |
+
# KI-079 (2026-05-15) — added two escalation-path reasons. When FAST
|
| 67 |
+
# chain times out we re-try once against BRAIN_CHAIN (heavier; Qwen 80B
|
| 68 |
+
# primary, more cross-provider fallbacks) inside a shorter budget. The
|
| 69 |
+
# _fallback_reason then captures the escalation outcome:
|
| 70 |
+
# "timeout_after_escalation" — both FAST + BRAIN chains timed out
|
| 71 |
+
# "llm_error_after_escalation" — BRAIN chain raised on the retry
|
| 72 |
+
# None when the brain succeeded (either FAST primary OR BRAIN escalation).
|
| 73 |
_fallback_reason: Optional[str] = None
|
| 74 |
|
| 75 |
|
|
|
|
| 299 |
# after a Space rebuild; the 12s wait_for was killing the brain BEFORE
|
| 300 |
# the cross-provider fallback links (Groq, OpenRouter) ever got tried.
|
| 301 |
# 25s gives NIM cold-start headroom + leaves room for one chain fallback.
|
| 302 |
+
_TIMEOUT_S = 25.0 # FAST_BRAIN_CHAIN primary attempt budget
|
| 303 |
+
# KI-079 (2026-05-15) — if FAST exhausted, try BRAIN_CHAIN (Qwen 80B primary
|
| 304 |
+
# + more cross-provider fallbacks incl. OpenRouter + Groq) with a SHORTER
|
| 305 |
+
# budget so the user doesn't wait 25+15=40s on a fully-dead network. Total
|
| 306 |
+
# worst-case latency before canonical fallback: 25 + 15 = 40s, but the FAST
|
| 307 |
+
# chain only hits 25s when NIM is wedged — the BRAIN escalation then has a
|
| 308 |
+
# Qwen primary on a different NIM pool + OpenRouter + Groq, so realistic
|
| 309 |
+
# escalation success cases land in 3-8s.
|
| 310 |
+
_TIMEOUT_S_ESCALATION = 15.0
|
| 311 |
|
| 312 |
|
| 313 |
async def drive_fact_find(
|
|
|
|
| 359 |
messages.append(ChatMessage(role=role, content=content))
|
| 360 |
messages.append(ChatMessage(role="user", content=user_text or ""))
|
| 361 |
|
| 362 |
+
# Hard 25-second timeout on FAST_BRAIN_CHAIN. The fast-brain chain already
|
| 363 |
+
# has its own per-link + total-chain budget but we wrap with asyncio.wait_for
|
| 364 |
+
# as a belt-and-braces stop.
|
| 365 |
+
llm_fast = get_fast_brain_llm()
|
| 366 |
try:
|
| 367 |
result = await asyncio.wait_for(
|
| 368 |
+
llm_fast.chat(messages=messages, temperature=0.6, max_tokens=420),
|
| 369 |
timeout=_TIMEOUT_S,
|
| 370 |
)
|
| 371 |
except asyncio.TimeoutError:
|
| 372 |
+
# KI-079 (2026-05-15) — fast brain timed out. Before falling to the
|
| 373 |
+
# canonical-question fallback, escalate ONCE to BRAIN_CHAIN (Qwen 80B
|
| 374 |
+
# primary, more cross-provider fallbacks) with a shorter budget so the
|
| 375 |
+
# user doesn't wait 25+15=40s on a dead network.
|
| 376 |
logging.warning(
|
| 377 |
+
"KI-079: fast brain timeout (session=%s, %.1fs) → escalating to heavy brain",
|
| 378 |
+
session_id, time.time() - t0,
|
| 379 |
)
|
| 380 |
+
from backend.providers.nvidia_nim_llm import get_brain_llm
|
| 381 |
+
llm_heavy = get_brain_llm()
|
| 382 |
+
try:
|
| 383 |
+
result = await asyncio.wait_for(
|
| 384 |
+
llm_heavy.chat(messages=messages, temperature=0.6, max_tokens=420),
|
| 385 |
+
timeout=_TIMEOUT_S_ESCALATION,
|
| 386 |
+
)
|
| 387 |
+
except asyncio.TimeoutError:
|
| 388 |
+
logging.warning(
|
| 389 |
+
"KI-079: heavy brain ALSO timed out (session=%s, total=%.1fs)",
|
| 390 |
+
session_id, time.time() - t0,
|
| 391 |
+
)
|
| 392 |
+
return _canonical_fallback(
|
| 393 |
+
session, user_text, reason="timeout_after_escalation"
|
| 394 |
+
)
|
| 395 |
+
except Exception as e2:
|
| 396 |
+
logging.warning(
|
| 397 |
+
"KI-079: heavy brain escalation failed (session=%s): %s: %s",
|
| 398 |
+
session_id, type(e2).__name__, str(e2)[:200],
|
| 399 |
+
)
|
| 400 |
+
return _canonical_fallback(
|
| 401 |
+
session, user_text, reason="llm_error_after_escalation"
|
| 402 |
+
)
|
| 403 |
except Exception as e:
|
| 404 |
logging.warning(
|
| 405 |
"fact_find_brain LLM call failed (session=%s): %s: %s",
|
|
@@ -226,14 +226,25 @@ FAST_BRAIN_CHAIN = [
|
|
| 226 |
# TTFT, not capability. Nemotron Nano 30B hits ~1.6s; Qwen 80B is
|
| 227 |
# ~2-3s. Moved Nemotron to primary; Qwen 80B stays as next fallback so
|
| 228 |
# if Nemotron's NIM pool degrades we still get quality.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 229 |
"nvidia/nemotron-3-nano-30b-a3b", # ~1.6s TTFT (Reddit bench), NIM
|
|
|
|
|
|
|
|
|
|
|
|
|
| 230 |
"qwen/qwen3-next-80b-a3b-instruct", # ~2-3s, NIM
|
| 231 |
"openai/gpt-oss-120b",
|
| 232 |
"qwen/qwen3.5-122b-a10b",
|
| 233 |
"deepseek-ai/deepseek-v4-flash",
|
| 234 |
-
# CROSS-PROVIDER FALLBACK — Groq Llama-3.3 70B (LPU, lowest TTFT of all
|
| 235 |
-
# free-tier options; OK for a fast-brain call when NIM is down).
|
| 236 |
-
"groq:llama-3.3-70b-versatile",
|
| 237 |
]
|
| 238 |
|
| 239 |
# Judge chain — non-Qwen, non-DeepSeek (different family from brain primary)
|
|
|
|
| 226 |
# TTFT, not capability. Nemotron Nano 30B hits ~1.6s; Qwen 80B is
|
| 227 |
# ~2-3s. Moved Nemotron to primary; Qwen 80B stays as next fallback so
|
| 228 |
# if Nemotron's NIM pool degrades we still get quality.
|
| 229 |
+
#
|
| 230 |
+
# KI-079 (2026-05-15) — moved Groq Llama-3.3 70B from chain bottom to
|
| 231 |
+
# candidate #2 (right after Nemotron primary). Live 10-turn probe
|
| 232 |
+
# (commit 078ff45) showed 7/10 fact-find turns timing out at 26.6s with
|
| 233 |
+
# _fallback_reason="timeout" — ALL the slow links were NIM-hosted, so
|
| 234 |
+
# NIM per-key concurrency had them queueing together and the chain
|
| 235 |
+
# burned its 22s total_budget_s inside NIM before ever reaching Groq.
|
| 236 |
+
# With Groq as #2, a Nemotron hang (~6s per-link timeout) falls
|
| 237 |
+
# through to Groq's LPU (~0.3s TTFT) in ~6-7s total — well inside the
|
| 238 |
+
# 22s chain budget AND the 25s wait_for cap.
|
| 239 |
"nvidia/nemotron-3-nano-30b-a3b", # ~1.6s TTFT (Reddit bench), NIM
|
| 240 |
+
# CROSS-PROVIDER FALLBACK #1 — Groq Llama-3.3 70B (LPU, lowest TTFT of
|
| 241 |
+
# all free-tier options). Promoted to #2 in KI-079 so a single NIM
|
| 242 |
+
# degradation falls through to a non-NIM provider in ~6s, not 22s.
|
| 243 |
+
"groq:llama-3.3-70b-versatile",
|
| 244 |
"qwen/qwen3-next-80b-a3b-instruct", # ~2-3s, NIM
|
| 245 |
"openai/gpt-oss-120b",
|
| 246 |
"qwen/qwen3.5-122b-a10b",
|
| 247 |
"deepseek-ai/deepseek-v4-flash",
|
|
|
|
|
|
|
|
|
|
| 248 |
]
|
| 249 |
|
| 250 |
# Judge chain — non-Qwen, non-DeepSeek (different family from brain primary)
|