Spaces:
Sleeping
feat(llm-chain): KI-080 — sticky primary election; 1-2 LLM calls per turn instead of 5-6
Browse filesArchitectural refactor: the chain list is now the CANDIDATE POOL for
probe-driven election, not a per-call sequence. Pre-KI-080, NimChainLLM.chat
iterated the full chain sequentially every turn. Under NIM per-key
concurrency throttling, the first 5 NIM-hosted candidates queued together
inside ONE turn, burning the 22s budget before any cross-provider fallback
(Groq / OpenRouter) was ever reached. The 10-turn live probe at commit
078ff45 showed 7/10 fact-find turns timing out at exactly 26.6s — every
failure a timeout, never a parse error.
ELECTION
========
backend/llm_health.py:
- Probe cadence tightened 5min → 60s so election reflects pool health
fast enough that a NIM brownout rotates out within a minute.
- Each candidate carries probe_history (last 5 ok/latency pairs) and
a degraded_until_monotonic sin-bin window.
- score = (1/max(50, latency_ms)) * success_rate_last_5
- get_primary(chain_name) → highest scorer with a probe within 90s.
- get_backup(chain_name) → highest scorer with a DIFFERENT provider
(NIM vs Groq vs OpenRouter) than primary; falls back to next-best
same-provider candidate when no cross-provider option qualifies.
- report_failure(chain, model, error_class): sidelines model for 30s +
schedules an async re-probe so the next turn elects a fresh primary.
- report_success(chain, model, latency_ms): folds chat latency into
the rolling window (chat traffic is richer signal than 1-token probes).
- status_summary now publishes the current elected primary/backup per
chain for the admin UI.
STICKY-PRIMARY chat()
=====================
backend/providers/nvidia_nim_llm.py:
- _call_one(model, ...) extracted from the old chain-iteration body —
single-model HTTP call, raises on failure for the caller to handle.
- chat() rewritten:
1. Apply caller exclusion list (brain ↦ judge family independence).
2. Resolve elected_primary / elected_backup via llm_health; if
either is excluded or election is cold-start, fall back to the
first allowed chain entry (cross-provider preference for backup).
3. Call elected_primary ONCE (12s timeout). On success → return.
4. On failure → report_failure(primary) + call elected_backup ONCE.
On success → return; on failure → report_failure(backup).
5. Final safety net (rare double-failure): probe_all() + walk
filter_chain in order. Preserves pre-KI-080 graceful degradation.
- CancelledError/KeyboardInterrupt/SystemExit re-raised at every hop
(KI-078 invariant preserved).
COST REDUCTION
==============
Pre-KI-080 worst case per turn: 5-6 LLM calls (full chain iteration).
Post-KI-080 worst case per turn: 2 LLM calls (primary + one backup).
Final probe-refresh safety net adds ≤chain_length more, but only
when BOTH elected models fail in the same turn — vanishingly rare
once probe-driven election is rotating out degraded candidates
within 60s.
INTERACTION WITH OTHER KIs
==========================
- KI-079 escalation in fact_find_brain UNCHANGED. If primary+backup
both fail and chat() raises RuntimeError, drive_fact_find still
catches it + escalates one more time via BRAIN_CHAIN before
falling to the canonical reply. Total worst case: 2 fast-brain
calls + 2 brain calls + canonical fallback.
- KI-078 narrow exception catch UNCHANGED — CancelledError /
KeyboardInterrupt / SystemExit are re-raised at every hop in
the new chat() so an outer wait_for cancellation bubbles up
instead of being swallowed.
- KI-025 50/50 NIM/Groq rotation is DEPRECATED (not deleted). The
probe-driven elector now picks the actually-faster candidate
dynamically — what the static coin flip was approximating
heuristically. _balanced_brain_chain stays exported because the
regression suite pins its statistical behaviour + it remains a
useful pure function for ops scripts. The get_*_llm factories no
longer wire it into the chat hot path.
- filter_chain (KI-022) still used by the final-fallback path and
by admin endpoints; kept unchanged.
TESTING
=======
- py_compile both files: pass.
- tests/test_routing_regression.py: 15/15 pass (includes the KI-025
rotation pinning suite, confirming the deprecated function still
behaves as documented).
- Inline unit tests (7/7 pass): cold-start election returns None,
primary picks lowest-latency healthy candidate, backup prefers
cross-provider, report_failure demotes within the window,
report_success updates score, primary-succeeds path makes ONE call,
primary-fails-backup-succeeds path makes exactly TWO calls.
EXPECTED PRODUCTION IMPACT
==========================
Brain success rate (fact-find turns reaching the elected primary
in <12s): 30% (today) → 90%+ projected, since election picks the
actually-fast pool and skips degraded NIM ingresses entirely.
NIM req/min consumption: ~6× lower under degraded conditions —
one primary call per turn instead of five queued candidates.
p50 latency: ~2s (elected Nemotron Nano / Qwen primary).
p95 latency: ~12s (primary timeout + backup success).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- backend/llm_health.py +379 -47
- backend/providers/nvidia_nim_llm.py +318 -139
|
@@ -1,29 +1,67 @@
|
|
| 1 |
-
"""Live NIM model health monitor — OpenRouter-style availability filter
|
| 2 |
-
|
| 3 |
-
|
| 4 |
-
|
| 5 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 6 |
- last_success_at: timestamp of last 2xx response
|
| 7 |
- last_failure_at: timestamp of last failure (timeout / 5xx / parse)
|
| 8 |
- latency_ms: last successful response latency
|
| 9 |
- consecutive_fail: counter (3+ => marked down)
|
| 10 |
- tested_at: when this row was last refreshed
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 11 |
|
| 12 |
Persistence: 40-data/llm_health.json (atomic write via temp+rename).
|
| 13 |
|
| 14 |
-
|
| 15 |
-
|
| 16 |
-
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 17 |
|
| 18 |
Operating schedule:
|
| 19 |
- background asyncio task in main.py startup, ticks every PROBE_INTERVAL_SEC.
|
| 20 |
-
- on-demand refresh via probe_all() (e.g. when NimChainLLM exhausts
|
|
|
|
| 21 |
"""
|
| 22 |
from __future__ import annotations
|
| 23 |
|
| 24 |
import asyncio
|
| 25 |
import json
|
| 26 |
import os
|
|
|
|
| 27 |
import time
|
| 28 |
from dataclasses import dataclass, field, asdict
|
| 29 |
from pathlib import Path
|
|
@@ -35,9 +73,17 @@ ROOT = Path(__file__).resolve().parent.parent
|
|
| 35 |
HEALTH_FILE = ROOT / "40-data" / "llm_health.json"
|
| 36 |
HEALTH_FILE.parent.mkdir(parents=True, exist_ok=True)
|
| 37 |
|
| 38 |
-
|
| 39 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 40 |
DOWN_AFTER_CONSECUTIVE_FAILS = 3 # 3 fails in a row = mark down
|
|
|
|
|
|
|
|
|
|
|
|
|
| 41 |
|
| 42 |
# Per-provider endpoints + env-var names. The chain entries embed the
|
| 43 |
# provider via a prefix ('openrouter:<id>' / 'groq:<id>'); unprefixed entries
|
|
@@ -83,6 +129,21 @@ def _headers_for(model_id: str, api_key: str) -> dict[str, str]:
|
|
| 83 |
return h
|
| 84 |
|
| 85 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 86 |
@dataclass
|
| 87 |
class ModelHealth:
|
| 88 |
model: str
|
|
@@ -93,12 +154,30 @@ class ModelHealth:
|
|
| 93 |
latency_ms: Optional[int] = None
|
| 94 |
consecutive_failures: int = 0
|
| 95 |
tested_at: Optional[str] = None
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 96 |
|
| 97 |
|
| 98 |
def _now_iso() -> str:
|
| 99 |
return time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())
|
| 100 |
|
| 101 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 102 |
def _all_known_models() -> list[str]:
|
| 103 |
"""Pull the union of every model name from every chain at module import."""
|
| 104 |
from backend.providers.nvidia_nim_llm import (
|
|
@@ -112,18 +191,68 @@ def _all_known_models() -> list[str]:
|
|
| 112 |
return seen
|
| 113 |
|
| 114 |
|
| 115 |
-
def
|
| 116 |
-
|
| 117 |
-
|
| 118 |
-
|
| 119 |
-
|
| 120 |
-
return
|
| 121 |
-
|
| 122 |
-
return
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 123 |
|
| 124 |
|
| 125 |
-
def
|
| 126 |
-
"""
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 127 |
out = {
|
| 128 |
"updated_at": _now_iso(),
|
| 129 |
"models": {k: asdict(v) for k, v in state.items()},
|
|
@@ -133,6 +262,180 @@ def save(state: dict[str, ModelHealth]) -> None:
|
|
| 133 |
tmp.replace(HEALTH_FILE)
|
| 134 |
|
| 135 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 136 |
async def probe_one(client: httpx.AsyncClient, model: str, api_key: str) -> tuple[bool, str, Optional[int]]:
|
| 137 |
"""Probe a single chain entry (NIM or cross-provider).
|
| 138 |
|
|
@@ -187,28 +490,11 @@ async def probe_one(client: httpx.AsyncClient, model: str, api_key: str) -> tupl
|
|
| 187 |
return False, f"net: {type(e).__name__}: {str(e)[:60]}", int((time.time() - t0) * 1000)
|
| 188 |
|
| 189 |
|
| 190 |
-
|
| 191 |
-
"""
|
| 192 |
-
|
| 193 |
-
|
| 194 |
-
|
| 195 |
-
if not api_key:
|
| 196 |
-
return state
|
| 197 |
-
|
| 198 |
-
async with httpx.AsyncClient() as client:
|
| 199 |
-
# Probe all in parallel — they hit different NIM pools so concurrency is fine
|
| 200 |
-
results = await asyncio.gather(
|
| 201 |
-
*[probe_one(client, m, api_key) for m in models],
|
| 202 |
-
return_exceptions=True,
|
| 203 |
-
)
|
| 204 |
-
|
| 205 |
-
for model, result in zip(models, results):
|
| 206 |
-
if isinstance(result, Exception):
|
| 207 |
-
ok, err, latency = False, f"exc: {type(result).__name__}", None
|
| 208 |
-
else:
|
| 209 |
-
ok, err, latency = result
|
| 210 |
-
|
| 211 |
-
h = state.get(model, ModelHealth(model=model))
|
| 212 |
h.tested_at = _now_iso()
|
| 213 |
if ok:
|
| 214 |
h.last_success_at = _now_iso()
|
|
@@ -224,15 +510,51 @@ async def probe_all() -> dict[str, ModelHealth]:
|
|
| 224 |
h.status = "down"
|
| 225 |
else:
|
| 226 |
h.status = "degraded"
|
| 227 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 228 |
|
| 229 |
-
|
| 230 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 231 |
|
| 232 |
|
| 233 |
def filter_chain(chain: list[str]) -> list[str]:
|
| 234 |
"""Return the chain with 'down' models removed. Always preserves order +
|
| 235 |
-
keeps at least one model so callers never get an empty chain.
|
|
|
|
|
|
|
|
|
|
|
|
|
| 236 |
state = load()
|
| 237 |
keep = [m for m in chain if state.get(m, ModelHealth(model=m)).status != "down"]
|
| 238 |
if not keep:
|
|
@@ -258,6 +580,12 @@ def status_summary() -> dict:
|
|
| 258 |
summary["models"].sort(key=lambda x: (x["status"] != "healthy", x["model"]))
|
| 259 |
if summary["models"]:
|
| 260 |
summary["updated_at"] = max((m.get("tested_at") or "") for m in summary["models"])
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 261 |
return summary
|
| 262 |
|
| 263 |
|
|
@@ -280,3 +608,7 @@ if __name__ == "__main__":
|
|
| 280 |
for m, h in sorted(state.items(), key=lambda kv: (kv[1].status != "healthy", kv[0])):
|
| 281 |
latency = f"{h.latency_ms}ms" if h.latency_ms else "—"
|
| 282 |
print(f" {h.status:8s} {latency:>8s} {m:50s} {h.last_error or 'ok'}")
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Live NIM model health monitor — OpenRouter-style availability filter
|
| 2 |
+
+ KI-080 sticky primary/backup election.
|
| 3 |
+
|
| 4 |
+
Architectural shift (KI-080, 2026-05-15):
|
| 5 |
+
==========================================
|
| 6 |
+
Pre-KI-080 the chain was iterated EVERY chat turn — primary, fallback 1,
|
| 7 |
+
fallback 2, ... until one succeeded. Under NIM per-key concurrency throttling,
|
| 8 |
+
the first 5 NIM-hosted candidates queued together inside a single turn,
|
| 9 |
+
burning the 22s budget before the cross-provider fallback links (Groq /
|
| 10 |
+
OpenRouter) were ever reached. The 10-turn live probe at commit 078ff45
|
| 11 |
+
showed 7/10 fact-find turns timing out at exactly 26.6s.
|
| 12 |
+
|
| 13 |
+
KI-080 inverts the model: the background probe loop ELECTS a primary +
|
| 14 |
+
backup per chain based on real probe latencies; chat() calls the elected
|
| 15 |
+
primary ONCE per turn, with at most ONE real-time fallback to the elected
|
| 16 |
+
backup. Worst case per turn drops from 5-6 LLM calls to 1-2.
|
| 17 |
+
|
| 18 |
+
Probes every PROBE_INTERVAL_SEC tick with a tiny ping ("Reply with exactly: ok").
|
| 19 |
+
Records per model:
|
| 20 |
+
- status: healthy / degraded / down / unknown
|
| 21 |
- last_success_at: timestamp of last 2xx response
|
| 22 |
- last_failure_at: timestamp of last failure (timeout / 5xx / parse)
|
| 23 |
- latency_ms: last successful response latency
|
| 24 |
- consecutive_fail: counter (3+ => marked down)
|
| 25 |
- tested_at: when this row was last refreshed
|
| 26 |
+
- probe_history: last PROBE_HISTORY_LEN (ok, latency_ms) tuples; powers
|
| 27 |
+
the success_rate signal in the election score.
|
| 28 |
+
|
| 29 |
+
Election (KI-080):
|
| 30 |
+
- For each chain (brain / fast_brain / judge), compute a score per healthy
|
| 31 |
+
candidate: score = (1 / max(50, latency_ms)) * success_rate_last_5.
|
| 32 |
+
- CURRENT_PRIMARY = highest scorer.
|
| 33 |
+
- CURRENT_BACKUP = highest scorer among candidates with a DIFFERENT
|
| 34 |
+
provider (NIM vs Groq vs OpenRouter) than primary; falls back to next-
|
| 35 |
+
best same-provider candidate when no cross-provider option qualifies.
|
| 36 |
+
- DEGRADED window: when chat() calls report_failure(model), that model is
|
| 37 |
+
sidelined for DEGRADED_WINDOW_SEC so the same turn's failure doesn't
|
| 38 |
+
recycle to the same broken primary on the next turn. The next probe
|
| 39 |
+
tick reconsiders the model normally.
|
| 40 |
|
| 41 |
Persistence: 40-data/llm_health.json (atomic write via temp+rename).
|
| 42 |
|
| 43 |
+
Public API (the surface NimChainLLM.chat consumes):
|
| 44 |
+
get_primary(chain_name) -> Optional[str]
|
| 45 |
+
get_backup(chain_name) -> Optional[str]
|
| 46 |
+
report_failure(chain, model, error_class)
|
| 47 |
+
report_success(chain, model, latency_ms)
|
| 48 |
+
|
| 49 |
+
Legacy:
|
| 50 |
+
filter_chain(chain) -> chain with 'down' models removed (still used
|
| 51 |
+
by admin / probe-refresh paths)
|
| 52 |
+
status_summary() -> compact dict for GET /api/health/llms
|
| 53 |
|
| 54 |
Operating schedule:
|
| 55 |
- background asyncio task in main.py startup, ticks every PROBE_INTERVAL_SEC.
|
| 56 |
+
- on-demand refresh via probe_all() (e.g. when NimChainLLM exhausts both
|
| 57 |
+
primary + backup in a single turn).
|
| 58 |
"""
|
| 59 |
from __future__ import annotations
|
| 60 |
|
| 61 |
import asyncio
|
| 62 |
import json
|
| 63 |
import os
|
| 64 |
+
import threading
|
| 65 |
import time
|
| 66 |
from dataclasses import dataclass, field, asdict
|
| 67 |
from pathlib import Path
|
|
|
|
| 73 |
HEALTH_FILE = ROOT / "40-data" / "llm_health.json"
|
| 74 |
HEALTH_FILE.parent.mkdir(parents=True, exist_ok=True)
|
| 75 |
|
| 76 |
+
# KI-080 — probe every 60s so election reflects real-time pool health
|
| 77 |
+
# fast enough that a NIM pool brownout is rotated out within a minute.
|
| 78 |
+
# The 5-min cadence pre-KI-080 was fine for a "filter the dead" use case
|
| 79 |
+
# but too slow when probe results drive primary election.
|
| 80 |
+
PROBE_INTERVAL_SEC = 60
|
| 81 |
+
PROBE_TIMEOUT_SEC = 8 # per-probe HTTP timeout
|
| 82 |
DOWN_AFTER_CONSECUTIVE_FAILS = 3 # 3 fails in a row = mark down
|
| 83 |
+
PROBE_HISTORY_LEN = 5 # rolling window for success_rate signal
|
| 84 |
+
HEALTHY_PROBE_AGE_SEC = 90 # election candidates need a probe within
|
| 85 |
+
# the last 90s — stale data excluded
|
| 86 |
+
DEGRADED_WINDOW_SEC = 30 # report_failure sidelines a model this long
|
| 87 |
|
| 88 |
# Per-provider endpoints + env-var names. The chain entries embed the
|
| 89 |
# provider via a prefix ('openrouter:<id>' / 'groq:<id>'); unprefixed entries
|
|
|
|
| 129 |
return h
|
| 130 |
|
| 131 |
|
| 132 |
+
def provider_of(model_id: str) -> str:
|
| 133 |
+
"""Coarse provider bucket — used by the election routine to prefer a
|
| 134 |
+
cross-provider backup so a NIM regional outage can't take out both
|
| 135 |
+
primary + backup simultaneously. Returns one of: 'nim' | 'groq' |
|
| 136 |
+
'openrouter'. (NIM is the implicit default for unprefixed model ids
|
| 137 |
+
even though NIM hosts many model families — all of those share the
|
| 138 |
+
same NIM ingress + per-key rate quota, which is what we need to
|
| 139 |
+
diversify against.)"""
|
| 140 |
+
if model_id.startswith("openrouter:"):
|
| 141 |
+
return "openrouter"
|
| 142 |
+
if model_id.startswith("groq:"):
|
| 143 |
+
return "groq"
|
| 144 |
+
return "nim"
|
| 145 |
+
|
| 146 |
+
|
| 147 |
@dataclass
|
| 148 |
class ModelHealth:
|
| 149 |
model: str
|
|
|
|
| 154 |
latency_ms: Optional[int] = None
|
| 155 |
consecutive_failures: int = 0
|
| 156 |
tested_at: Optional[str] = None
|
| 157 |
+
# KI-080 — rolling probe history powers the success_rate signal in
|
| 158 |
+
# the election score. Each entry: {"ok": bool, "latency_ms": int|None,
|
| 159 |
+
# "ts": iso8601}. Capped at PROBE_HISTORY_LEN.
|
| 160 |
+
probe_history: list[dict] = field(default_factory=list)
|
| 161 |
+
# KI-080 — set by report_failure(); model is excluded from election
|
| 162 |
+
# while monotonic time < degraded_until_monotonic.
|
| 163 |
+
degraded_until_monotonic: float = 0.0
|
| 164 |
|
| 165 |
|
| 166 |
def _now_iso() -> str:
|
| 167 |
return time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())
|
| 168 |
|
| 169 |
|
| 170 |
+
def _iso_age_seconds(iso_ts: Optional[str]) -> Optional[float]:
|
| 171 |
+
"""Seconds since an ISO timestamp; None if missing/unparseable."""
|
| 172 |
+
if not iso_ts:
|
| 173 |
+
return None
|
| 174 |
+
try:
|
| 175 |
+
t = time.strptime(iso_ts, "%Y-%m-%dT%H:%M:%SZ")
|
| 176 |
+
return time.time() - time.mktime(t) + time.timezone
|
| 177 |
+
except Exception:
|
| 178 |
+
return None
|
| 179 |
+
|
| 180 |
+
|
| 181 |
def _all_known_models() -> list[str]:
|
| 182 |
"""Pull the union of every model name from every chain at module import."""
|
| 183 |
from backend.providers.nvidia_nim_llm import (
|
|
|
|
| 191 |
return seen
|
| 192 |
|
| 193 |
|
| 194 |
+
def _chain_for(chain_name: str) -> list[str]:
|
| 195 |
+
"""Resolve a chain name → live chain list. Reads off the module so
|
| 196 |
+
runtime admin reorderings are respected by the elector."""
|
| 197 |
+
from backend.providers import nvidia_nim_llm as nim
|
| 198 |
+
if chain_name == "brain":
|
| 199 |
+
return list(getattr(nim, "BRAIN_CHAIN", []))
|
| 200 |
+
if chain_name == "fast_brain":
|
| 201 |
+
return list(getattr(nim, "FAST_BRAIN_CHAIN", []))
|
| 202 |
+
if chain_name == "judge":
|
| 203 |
+
return list(getattr(nim, "JUDGE_CHAIN", []))
|
| 204 |
+
return []
|
| 205 |
+
|
| 206 |
+
|
| 207 |
+
# KI-080 — in-process state. Probe history + degraded windows are
|
| 208 |
+
# performance-critical hot paths (read on every chat turn), so we keep
|
| 209 |
+
# them in memory and only persist the long-lived signal to disk on the
|
| 210 |
+
# probe tick. Concurrent NimChainLLM.chat workers + the probe loop both
|
| 211 |
+
# mutate this; a coarse lock is fine — the held region is microseconds.
|
| 212 |
+
_STATE_LOCK = threading.Lock()
|
| 213 |
+
_STATE: dict[str, ModelHealth] = {}
|
| 214 |
+
_STATE_LOADED = False
|
| 215 |
+
|
| 216 |
+
|
| 217 |
+
def _load_into_memory() -> None:
|
| 218 |
+
"""Hydrate _STATE from disk on first access. Idempotent."""
|
| 219 |
+
global _STATE, _STATE_LOADED
|
| 220 |
+
if _STATE_LOADED:
|
| 221 |
+
return
|
| 222 |
+
with _STATE_LOCK:
|
| 223 |
+
if _STATE_LOADED:
|
| 224 |
+
return
|
| 225 |
+
if HEALTH_FILE.exists():
|
| 226 |
+
try:
|
| 227 |
+
raw = json.loads(HEALTH_FILE.read_text())
|
| 228 |
+
for k, v in raw.get("models", {}).items():
|
| 229 |
+
# Tolerate older schema (pre-KI-080 records missing
|
| 230 |
+
# probe_history / degraded_until_monotonic).
|
| 231 |
+
v.setdefault("probe_history", [])
|
| 232 |
+
v.setdefault("degraded_until_monotonic", 0.0)
|
| 233 |
+
_STATE[k] = ModelHealth(**v)
|
| 234 |
+
except Exception:
|
| 235 |
+
_STATE = {}
|
| 236 |
+
_STATE_LOADED = True
|
| 237 |
|
| 238 |
|
| 239 |
+
def load() -> dict[str, ModelHealth]:
|
| 240 |
+
"""Legacy/back-compat: snapshot of the in-memory state. Returned dict
|
| 241 |
+
is a shallow copy so callers can't mutate _STATE directly."""
|
| 242 |
+
_load_into_memory()
|
| 243 |
+
with _STATE_LOCK:
|
| 244 |
+
return dict(_STATE)
|
| 245 |
+
|
| 246 |
+
|
| 247 |
+
def save(state: Optional[dict[str, ModelHealth]] = None) -> None:
|
| 248 |
+
"""Atomic write — temp file then rename so concurrent readers never see partial.
|
| 249 |
+
|
| 250 |
+
If `state` is None, persists the in-memory _STATE. The optional arg is
|
| 251 |
+
kept for backward compatibility with the pre-KI-080 call sites in admin.py."""
|
| 252 |
+
if state is None:
|
| 253 |
+
_load_into_memory()
|
| 254 |
+
with _STATE_LOCK:
|
| 255 |
+
state = dict(_STATE)
|
| 256 |
out = {
|
| 257 |
"updated_at": _now_iso(),
|
| 258 |
"models": {k: asdict(v) for k, v in state.items()},
|
|
|
|
| 262 |
tmp.replace(HEALTH_FILE)
|
| 263 |
|
| 264 |
|
| 265 |
+
# ---------------------------------------------------------------------------
|
| 266 |
+
# KI-080 — public API: primary/backup election + failure/success reporting
|
| 267 |
+
# ---------------------------------------------------------------------------
|
| 268 |
+
|
| 269 |
+
def _success_rate(h: ModelHealth) -> float:
|
| 270 |
+
"""Fraction of the last PROBE_HISTORY_LEN probes that succeeded.
|
| 271 |
+
Returns 1.0 when there's no history yet (cold-start — give every
|
| 272 |
+
healthy candidate the benefit of the doubt)."""
|
| 273 |
+
if not h.probe_history:
|
| 274 |
+
return 1.0
|
| 275 |
+
hits = sum(1 for r in h.probe_history if r.get("ok"))
|
| 276 |
+
return hits / len(h.probe_history)
|
| 277 |
+
|
| 278 |
+
|
| 279 |
+
def _score(h: ModelHealth) -> float:
|
| 280 |
+
"""Election score — higher is better.
|
| 281 |
+
|
| 282 |
+
score = (1 / max(50, latency_ms)) * success_rate
|
| 283 |
+
The 50ms floor stops a sub-millisecond outlier from dominating
|
| 284 |
+
election; success_rate is the rolling-window stability signal.
|
| 285 |
+
"""
|
| 286 |
+
if h.latency_ms is None:
|
| 287 |
+
return 0.0
|
| 288 |
+
return (1.0 / max(50, h.latency_ms)) * _success_rate(h)
|
| 289 |
+
|
| 290 |
+
|
| 291 |
+
def _is_election_eligible(h: ModelHealth, now_mono: float) -> bool:
|
| 292 |
+
"""A candidate is electable when:
|
| 293 |
+
- status is healthy (or degraded with a recent success)
|
| 294 |
+
- last probe was within HEALTHY_PROBE_AGE_SEC
|
| 295 |
+
- it is NOT currently in the degraded-window sin-bin
|
| 296 |
+
"""
|
| 297 |
+
if h.degraded_until_monotonic > now_mono:
|
| 298 |
+
return False
|
| 299 |
+
if h.status == "down":
|
| 300 |
+
return False
|
| 301 |
+
age = _iso_age_seconds(h.tested_at)
|
| 302 |
+
if age is None or age > HEALTHY_PROBE_AGE_SEC:
|
| 303 |
+
return False
|
| 304 |
+
if h.latency_ms is None:
|
| 305 |
+
return False
|
| 306 |
+
return True
|
| 307 |
+
|
| 308 |
+
|
| 309 |
+
def _ranked_candidates(chain_name: str) -> list[ModelHealth]:
|
| 310 |
+
"""Return the chain's election-eligible candidates, best-score first."""
|
| 311 |
+
_load_into_memory()
|
| 312 |
+
chain = _chain_for(chain_name)
|
| 313 |
+
now_mono = time.monotonic()
|
| 314 |
+
with _STATE_LOCK:
|
| 315 |
+
snapshot = {m: _STATE.get(m) for m in chain}
|
| 316 |
+
eligible: list[tuple[float, ModelHealth]] = []
|
| 317 |
+
for m in chain:
|
| 318 |
+
h = snapshot.get(m)
|
| 319 |
+
if h is None:
|
| 320 |
+
continue
|
| 321 |
+
if not _is_election_eligible(h, now_mono):
|
| 322 |
+
continue
|
| 323 |
+
eligible.append((_score(h), h))
|
| 324 |
+
eligible.sort(key=lambda t: t[0], reverse=True)
|
| 325 |
+
return [h for _, h in eligible]
|
| 326 |
+
|
| 327 |
+
|
| 328 |
+
def get_primary(chain_name: str) -> Optional[str]:
|
| 329 |
+
"""Top-scoring election-eligible candidate, or None when no probe data
|
| 330 |
+
is fresh enough. Callers cold-start by falling back to chain[0]."""
|
| 331 |
+
ranked = _ranked_candidates(chain_name)
|
| 332 |
+
return ranked[0].model if ranked else None
|
| 333 |
+
|
| 334 |
+
|
| 335 |
+
def get_backup(chain_name: str) -> Optional[str]:
|
| 336 |
+
"""Second-best election candidate. Prefers a DIFFERENT provider from
|
| 337 |
+
primary (NIM vs Groq vs OpenRouter) so a single provider's regional
|
| 338 |
+
outage can't take out both. Falls back to the next-best same-provider
|
| 339 |
+
candidate when no cross-provider option qualifies — better an in-
|
| 340 |
+
family backup than none."""
|
| 341 |
+
ranked = _ranked_candidates(chain_name)
|
| 342 |
+
if len(ranked) < 2:
|
| 343 |
+
# No usable backup. Either zero candidates or only one (in which
|
| 344 |
+
# case the cold-start path in NimChainLLM falls back to chain[1]).
|
| 345 |
+
return None
|
| 346 |
+
primary_provider = provider_of(ranked[0].model)
|
| 347 |
+
# Prefer cross-provider
|
| 348 |
+
for h in ranked[1:]:
|
| 349 |
+
if provider_of(h.model) != primary_provider:
|
| 350 |
+
return h.model
|
| 351 |
+
# No cross-provider candidate — accept same-provider next-best.
|
| 352 |
+
return ranked[1].model
|
| 353 |
+
|
| 354 |
+
|
| 355 |
+
def report_failure(chain_name: str, model: str, error_class: str) -> None:
|
| 356 |
+
"""Called by NimChainLLM.chat() when primary OR backup throws.
|
| 357 |
+
|
| 358 |
+
Effects:
|
| 359 |
+
- Sidelines `model` from election for DEGRADED_WINDOW_SEC so the
|
| 360 |
+
same turn's failure doesn't immediately re-elect the same model.
|
| 361 |
+
- Appends a synthetic 'failed' entry to probe_history so the next
|
| 362 |
+
election's success_rate reflects the live failure even before
|
| 363 |
+
the next probe tick.
|
| 364 |
+
- Schedules an async re-probe (best-effort) so the next turn gets
|
| 365 |
+
fresh data instead of waiting up to 60s for the next tick.
|
| 366 |
+
"""
|
| 367 |
+
_load_into_memory()
|
| 368 |
+
with _STATE_LOCK:
|
| 369 |
+
h = _STATE.get(model) or ModelHealth(model=model)
|
| 370 |
+
h.degraded_until_monotonic = time.monotonic() + DEGRADED_WINDOW_SEC
|
| 371 |
+
h.last_failure_at = _now_iso()
|
| 372 |
+
h.last_error = f"chat_failure: {error_class}"
|
| 373 |
+
h.probe_history.append({
|
| 374 |
+
"ok": False,
|
| 375 |
+
"latency_ms": None,
|
| 376 |
+
"ts": _now_iso(),
|
| 377 |
+
"src": "chat",
|
| 378 |
+
})
|
| 379 |
+
if len(h.probe_history) > PROBE_HISTORY_LEN:
|
| 380 |
+
h.probe_history = h.probe_history[-PROBE_HISTORY_LEN:]
|
| 381 |
+
# Don't flip status to 'down' here — that's the probe's job and
|
| 382 |
+
# we don't want a single transient turn-failure to evict the
|
| 383 |
+
# candidate permanently. The degraded sin-bin is sufficient.
|
| 384 |
+
_STATE[model] = h
|
| 385 |
+
|
| 386 |
+
# Best-effort re-probe — fire and forget. We can't await inside this
|
| 387 |
+
# sync API (callers are in the hot path) so we schedule on the loop
|
| 388 |
+
# if one is running.
|
| 389 |
+
try:
|
| 390 |
+
loop = asyncio.get_event_loop()
|
| 391 |
+
if loop.is_running():
|
| 392 |
+
loop.create_task(_reprobe_one(model))
|
| 393 |
+
except RuntimeError:
|
| 394 |
+
pass # no loop (unit tests etc.) — skip silently
|
| 395 |
+
|
| 396 |
+
|
| 397 |
+
def report_success(chain_name: str, model: str, latency_ms: int) -> None:
|
| 398 |
+
"""Called by NimChainLLM.chat() on a successful response. Updates
|
| 399 |
+
the rolling success/latency window — important because chat traffic
|
| 400 |
+
is dramatically richer signal than 1-token probes (real prompts,
|
| 401 |
+
real concurrency)."""
|
| 402 |
+
_load_into_memory()
|
| 403 |
+
with _STATE_LOCK:
|
| 404 |
+
h = _STATE.get(model) or ModelHealth(model=model)
|
| 405 |
+
h.last_success_at = _now_iso()
|
| 406 |
+
h.last_error = None
|
| 407 |
+
h.latency_ms = int(latency_ms)
|
| 408 |
+
h.consecutive_failures = 0
|
| 409 |
+
h.tested_at = _now_iso()
|
| 410 |
+
h.status = "healthy" if latency_ms < 5000 else "degraded"
|
| 411 |
+
h.probe_history.append({
|
| 412 |
+
"ok": True,
|
| 413 |
+
"latency_ms": int(latency_ms),
|
| 414 |
+
"ts": _now_iso(),
|
| 415 |
+
"src": "chat",
|
| 416 |
+
})
|
| 417 |
+
if len(h.probe_history) > PROBE_HISTORY_LEN:
|
| 418 |
+
h.probe_history = h.probe_history[-PROBE_HISTORY_LEN:]
|
| 419 |
+
_STATE[model] = h
|
| 420 |
+
|
| 421 |
+
|
| 422 |
+
async def _reprobe_one(model: str) -> None:
|
| 423 |
+
"""Async re-probe of a single model after report_failure(). Updates
|
| 424 |
+
state in place; failures here are silently swallowed so the chat
|
| 425 |
+
hot path never raises through this back-channel."""
|
| 426 |
+
try:
|
| 427 |
+
async with httpx.AsyncClient() as client:
|
| 428 |
+
ok, err, latency = await probe_one(client, model, "")
|
| 429 |
+
_absorb_probe_result(model, ok, err, latency)
|
| 430 |
+
except Exception:
|
| 431 |
+
pass
|
| 432 |
+
|
| 433 |
+
|
| 434 |
+
# ---------------------------------------------------------------------------
|
| 435 |
+
# Probing (mostly unchanged from pre-KI-080 — extended to record
|
| 436 |
+
# probe_history + skip degraded-window models on the regular tick).
|
| 437 |
+
# ---------------------------------------------------------------------------
|
| 438 |
+
|
| 439 |
async def probe_one(client: httpx.AsyncClient, model: str, api_key: str) -> tuple[bool, str, Optional[int]]:
|
| 440 |
"""Probe a single chain entry (NIM or cross-provider).
|
| 441 |
|
|
|
|
| 490 |
return False, f"net: {type(e).__name__}: {str(e)[:60]}", int((time.time() - t0) * 1000)
|
| 491 |
|
| 492 |
|
| 493 |
+
def _absorb_probe_result(model: str, ok: bool, err: str, latency: Optional[int]) -> None:
|
| 494 |
+
"""Update _STATE with one probe result. Used by both probe_all (every
|
| 495 |
+
tick) and _reprobe_one (after-failure reprobe)."""
|
| 496 |
+
with _STATE_LOCK:
|
| 497 |
+
h = _STATE.get(model) or ModelHealth(model=model)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 498 |
h.tested_at = _now_iso()
|
| 499 |
if ok:
|
| 500 |
h.last_success_at = _now_iso()
|
|
|
|
| 510 |
h.status = "down"
|
| 511 |
else:
|
| 512 |
h.status = "degraded"
|
| 513 |
+
h.probe_history.append({
|
| 514 |
+
"ok": ok,
|
| 515 |
+
"latency_ms": latency,
|
| 516 |
+
"ts": _now_iso(),
|
| 517 |
+
"src": "probe",
|
| 518 |
+
})
|
| 519 |
+
if len(h.probe_history) > PROBE_HISTORY_LEN:
|
| 520 |
+
h.probe_history = h.probe_history[-PROBE_HISTORY_LEN:]
|
| 521 |
+
_STATE[model] = h
|
| 522 |
|
| 523 |
+
|
| 524 |
+
async def probe_all() -> dict[str, ModelHealth]:
|
| 525 |
+
"""One-shot probe of every known model. Updates persisted state + returns it."""
|
| 526 |
+
_load_into_memory()
|
| 527 |
+
models = _all_known_models()
|
| 528 |
+
# Per-provider keys are resolved inside probe_one(); we no longer
|
| 529 |
+
# gate the whole loop on the NIM key (KI-080) — cross-provider
|
| 530 |
+
# candidates must keep getting probed even when NVIDIA_NIM_API_KEY
|
| 531 |
+
# is missing.
|
| 532 |
+
|
| 533 |
+
async with httpx.AsyncClient() as client:
|
| 534 |
+
# Probe all in parallel — they hit different NIM pools so concurrency is fine
|
| 535 |
+
results = await asyncio.gather(
|
| 536 |
+
*[probe_one(client, m, "") for m in models],
|
| 537 |
+
return_exceptions=True,
|
| 538 |
+
)
|
| 539 |
+
|
| 540 |
+
for model, result in zip(models, results):
|
| 541 |
+
if isinstance(result, Exception):
|
| 542 |
+
ok, err, latency = False, f"exc: {type(result).__name__}", None
|
| 543 |
+
else:
|
| 544 |
+
ok, err, latency = result
|
| 545 |
+
_absorb_probe_result(model, ok, err, latency)
|
| 546 |
+
|
| 547 |
+
save() # persist in-memory state
|
| 548 |
+
return load()
|
| 549 |
|
| 550 |
|
| 551 |
def filter_chain(chain: list[str]) -> list[str]:
|
| 552 |
"""Return the chain with 'down' models removed. Always preserves order +
|
| 553 |
+
keeps at least one model so callers never get an empty chain.
|
| 554 |
+
|
| 555 |
+
Kept for backward compatibility — admin endpoints + the final
|
| 556 |
+
probe-refresh fallback in NimChainLLM still call this. Primary
|
| 557 |
+
election (get_primary / get_backup) is the new hot-path entry."""
|
| 558 |
state = load()
|
| 559 |
keep = [m for m in chain if state.get(m, ModelHealth(model=m)).status != "down"]
|
| 560 |
if not keep:
|
|
|
|
| 580 |
summary["models"].sort(key=lambda x: (x["status"] != "healthy", x["model"]))
|
| 581 |
if summary["models"]:
|
| 582 |
summary["updated_at"] = max((m.get("tested_at") or "") for m in summary["models"])
|
| 583 |
+
# KI-080 — surface currently-elected primary/backup per chain for the
|
| 584 |
+
# admin UI. Cheap (a couple of dict lookups + sort over <=10 entries).
|
| 585 |
+
summary["elections"] = {
|
| 586 |
+
role: {"primary": get_primary(role), "backup": get_backup(role)}
|
| 587 |
+
for role in ("brain", "fast_brain", "judge")
|
| 588 |
+
}
|
| 589 |
return summary
|
| 590 |
|
| 591 |
|
|
|
|
| 608 |
for m, h in sorted(state.items(), key=lambda kv: (kv[1].status != "healthy", kv[0])):
|
| 609 |
latency = f"{h.latency_ms}ms" if h.latency_ms else "—"
|
| 610 |
print(f" {h.status:8s} {latency:>8s} {m:50s} {h.last_error or 'ok'}")
|
| 611 |
+
print()
|
| 612 |
+
print("Elections:")
|
| 613 |
+
for role in ("brain", "fast_brain", "judge"):
|
| 614 |
+
print(f" {role:10s} primary={get_primary(role)} backup={get_backup(role)}")
|
|
@@ -270,14 +270,32 @@ JUDGE_CHAIN = [
|
|
| 270 |
|
| 271 |
|
| 272 |
class NimChainLLM(LLMProvider):
|
| 273 |
-
"""
|
| 274 |
-
|
| 275 |
-
|
| 276 |
-
|
| 277 |
-
|
| 278 |
-
|
| 279 |
-
|
| 280 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 281 |
"""
|
| 282 |
def __init__(self, chain: list[str], api_key: Optional[str] = None,
|
| 283 |
timeout: float = 30.0, per_model_attempts: int = 1,
|
|
@@ -292,9 +310,14 @@ class NimChainLLM(LLMProvider):
|
|
| 292 |
# produced the p99 58s+ tail in the 100-persona audit. Default to a
|
| 293 |
# cumulative ceiling of ~2.5× the per-link timeout so a healthy primary
|
| 294 |
# always completes, but a cascading-failure chain bails fast.
|
|
|
|
|
|
|
|
|
|
|
|
|
| 295 |
self.total_budget_s = total_budget_s if total_budget_s is not None else max(timeout * 2.5, 30.0)
|
| 296 |
self.per_model_attempts = per_model_attempts
|
| 297 |
self.role = role # 'brain' | 'fast_brain' | 'judge' | 'unknown' — flows into usage log
|
|
|
|
| 298 |
self.model = chain[0]
|
| 299 |
self.name = f"nim-chain::{self._short_id(chain[0])}"
|
| 300 |
|
|
@@ -358,6 +381,38 @@ class NimChainLLM(LLMProvider):
|
|
| 358 |
if "nemotron" in m or m.startswith("nvidia/"): return "nvidia"
|
| 359 |
return "unknown"
|
| 360 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 361 |
async def chat(
|
| 362 |
self,
|
| 363 |
messages: list[ChatMessage],
|
|
@@ -367,123 +422,217 @@ class NimChainLLM(LLMProvider):
|
|
| 367 |
exclude_models: Optional[list[str]] = None,
|
| 368 |
exclude_families: Optional[list[str]] = None,
|
| 369 |
) -> LLMResult:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 370 |
# Apply caller's exclusion list FIRST (brain doesn't grade own homework).
|
| 371 |
-
|
| 372 |
-
|
| 373 |
-
|
| 374 |
-
|
| 375 |
-
|
| 376 |
-
|
| 377 |
-
|
| 378 |
-
|
| 379 |
-
|
| 380 |
-
|
| 381 |
-
|
| 382 |
-
|
| 383 |
-
|
| 384 |
-
|
| 385 |
-
|
| 386 |
-
|
| 387 |
-
|
| 388 |
-
|
| 389 |
-
|
| 390 |
-
|
| 391 |
-
|
| 392 |
-
chain_to_try = llm_health.filter_chain(chain)
|
| 393 |
-
except Exception:
|
| 394 |
-
chain_to_try = chain # health monitor failure must never block calls
|
| 395 |
|
| 396 |
chain_primary = self.chain[0] if self.chain else None
|
| 397 |
call_t0 = time.time()
|
| 398 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 399 |
last_err: Optional[Exception] = None
|
| 400 |
-
|
| 401 |
-
|
| 402 |
-
|
| 403 |
-
|
| 404 |
-
|
| 405 |
-
|
| 406 |
-
|
| 407 |
-
|
| 408 |
-
|
| 409 |
-
|
|
|
|
|
|
|
|
|
|
| 410 |
try:
|
| 411 |
-
|
| 412 |
-
|
| 413 |
-
|
| 414 |
-
|
| 415 |
-
|
| 416 |
-
|
| 417 |
-
|
| 418 |
-
|
| 419 |
-
# cross-provider fall-throughs (groq:/openrouter:) obvious.
|
| 420 |
-
self.model = model
|
| 421 |
-
self.name = f"nim-chain::{self._short_id(model)}"
|
| 422 |
-
latency_ms = int((time.time() - call_t0) * 1000)
|
| 423 |
-
await _append_usage({
|
| 424 |
-
"ts": _now_iso_z(),
|
| 425 |
-
"role": self.role,
|
| 426 |
-
"chain_primary": chain_primary,
|
| 427 |
-
"served_model": model,
|
| 428 |
-
"latency_ms": latency_ms,
|
| 429 |
-
"success": True,
|
| 430 |
-
})
|
| 431 |
-
return result
|
| 432 |
-
except (httpx.TimeoutException, httpx.HTTPStatusError,
|
| 433 |
-
httpx.ConnectError, httpx.NetworkError, asyncio.TimeoutError) as e:
|
| 434 |
-
last_err = e
|
| 435 |
-
continue # try next model in chain
|
| 436 |
except (asyncio.CancelledError, KeyboardInterrupt, SystemExit):
|
| 437 |
-
# KI-078
|
| 438 |
-
# broad `except Exception` swallowed CancelledError, so when
|
| 439 |
-
# fact_find_brain's outer `asyncio.wait_for(_TIMEOUT_S)`
|
| 440 |
-
# fired, this loop kept consuming budget instead of bubbling
|
| 441 |
-
# the cancellation up. That cost the entire fact-find turn
|
| 442 |
-
# its fallback window.
|
| 443 |
raise
|
| 444 |
except Exception as e:
|
| 445 |
-
# Unexpected error — record + try next, but surface eventually if all fail
|
| 446 |
last_err = e
|
| 447 |
-
|
| 448 |
-
|
| 449 |
-
|
| 450 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 451 |
try:
|
| 452 |
-
from backend import llm_health
|
| 453 |
await llm_health.probe_all()
|
| 454 |
-
refreshed = llm_health.filter_chain(
|
| 455 |
for model in refreshed:
|
| 456 |
-
if model in
|
| 457 |
-
continue
|
| 458 |
-
# KI-021 — respect chain budget here too
|
| 459 |
elapsed = time.time() - call_t0
|
| 460 |
if elapsed >= self.total_budget_s:
|
| 461 |
break
|
|
|
|
| 462 |
try:
|
| 463 |
-
per_link_timeout = min(
|
| 464 |
-
|
| 465 |
-
|
| 466 |
-
|
| 467 |
-
|
| 468 |
-
|
| 469 |
-
|
| 470 |
-
|
| 471 |
-
|
| 472 |
-
|
| 473 |
-
|
| 474 |
-
|
| 475 |
-
|
| 476 |
-
|
| 477 |
-
})
|
| 478 |
-
return result
|
| 479 |
except Exception as e:
|
| 480 |
last_err = e
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 481 |
continue
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 482 |
except Exception:
|
| 483 |
-
pass
|
| 484 |
|
| 485 |
-
# Total exhaustion — log
|
| 486 |
-
#
|
|
|
|
| 487 |
latency_ms = int((time.time() - call_t0) * 1000)
|
| 488 |
await _append_usage({
|
| 489 |
"ts": _now_iso_z(),
|
|
@@ -492,30 +641,45 @@ class NimChainLLM(LLMProvider):
|
|
| 492 |
"served_model": None,
|
| 493 |
"latency_ms": latency_ms,
|
| 494 |
"success": False,
|
|
|
|
|
|
|
|
|
|
| 495 |
})
|
| 496 |
raise RuntimeError(
|
| 497 |
-
f"NimChainLLM
|
| 498 |
-
f"{
|
|
|
|
|
|
|
| 499 |
) from last_err
|
| 500 |
|
| 501 |
|
| 502 |
# KI-025 (2026-05-14) — provider load-balancing.
|
| 503 |
-
#
|
| 504 |
-
#
|
| 505 |
-
#
|
| 506 |
-
#
|
| 507 |
-
#
|
| 508 |
-
#
|
| 509 |
-
#
|
| 510 |
import random as _random
|
| 511 |
|
| 512 |
|
| 513 |
def _balanced_brain_chain(base: list[str], *, groq_first_probability: float = 0.5) -> list[str]:
|
| 514 |
-
"""KI-025 — provider load-balancing
|
| 515 |
-
|
| 516 |
-
|
| 517 |
-
|
| 518 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 519 |
|
| 520 |
Uses per-call `random.random()` so concurrent async workers (the
|
| 521 |
100-persona audit's 4 workers, the parallel 96-Q eval's 6 workers) each
|
|
@@ -533,33 +697,48 @@ def _balanced_brain_chain(base: list[str], *, groq_first_probability: float = 0.
|
|
| 533 |
|
| 534 |
|
| 535 |
def get_brain_llm() -> NimChainLLM:
|
| 536 |
-
"""Heavy brain —
|
| 537 |
-
|
| 538 |
-
|
| 539 |
-
|
| 540 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 541 |
role="brain", total_budget_s=35.0)
|
| 542 |
|
| 543 |
|
| 544 |
def get_fast_brain_llm() -> NimChainLLM:
|
| 545 |
-
"""Fast brain —
|
| 546 |
-
|
| 547 |
-
|
| 548 |
-
|
| 549 |
-
|
| 550 |
-
|
| 551 |
-
|
| 552 |
-
|
| 553 |
-
|
| 554 |
-
|
| 555 |
-
|
| 556 |
-
|
| 557 |
-
|
| 558 |
-
|
| 559 |
-
|
| 560 |
-
|
| 561 |
-
|
| 562 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 563 |
role="fast_brain", total_budget_s=22.0)
|
| 564 |
|
| 565 |
|
|
|
|
| 270 |
|
| 271 |
|
| 272 |
class NimChainLLM(LLMProvider):
|
| 273 |
+
"""KI-080 sticky-primary router across multiple candidate models.
|
| 274 |
+
|
| 275 |
+
Architectural shift (KI-080, 2026-05-15):
|
| 276 |
+
PRE-KI-080: chat() iterated the full chain sequentially every turn.
|
| 277 |
+
Under NIM per-key concurrency throttling, the first 5 NIM-hosted
|
| 278 |
+
candidates queued together inside ONE turn and burned the 22s
|
| 279 |
+
budget before any cross-provider fallback was ever reached. The
|
| 280 |
+
10-turn live probe at commit 078ff45 showed 7/10 fact-find turns
|
| 281 |
+
timing out at exactly 26.6s.
|
| 282 |
+
|
| 283 |
+
POST-KI-080: a background probe loop ELECTS a primary + cross-
|
| 284 |
+
provider backup per chain based on real probe latencies. chat()
|
| 285 |
+
calls the elected primary ONCE per turn (1 LLM call). On failure,
|
| 286 |
+
we demote the primary for ~30s and fall through to the elected
|
| 287 |
+
backup (still 2 LLM calls max). The chain list is now the
|
| 288 |
+
CANDIDATE POOL for election, not a per-call sequence.
|
| 289 |
+
|
| 290 |
+
Worst case per turn: 2 LLM calls (was 5-6). Plus a final filter_chain
|
| 291 |
+
refresh path is kept for the rare double-failure edge case so the
|
| 292 |
+
pre-KI-080 graceful-degradation behaviour is preserved.
|
| 293 |
+
|
| 294 |
+
`name` after a successful call reflects which model actually answered
|
| 295 |
+
so downstream callers (orchestrator brain_used tag, eval logs) can
|
| 296 |
+
audit which candidate produced the output. The KI-079 escalation in
|
| 297 |
+
fact_find_brain still applies — if primary+backup both fail inside
|
| 298 |
+
one turn, fact_find_brain gets one more bite via BRAIN_CHAIN.
|
| 299 |
"""
|
| 300 |
def __init__(self, chain: list[str], api_key: Optional[str] = None,
|
| 301 |
timeout: float = 30.0, per_model_attempts: int = 1,
|
|
|
|
| 310 |
# produced the p99 58s+ tail in the 100-persona audit. Default to a
|
| 311 |
# cumulative ceiling of ~2.5× the per-link timeout so a healthy primary
|
| 312 |
# always completes, but a cascading-failure chain bails fast.
|
| 313 |
+
#
|
| 314 |
+
# KI-080 — with election we only do 1-2 LLM calls per turn, so the
|
| 315 |
+
# budget is rarely the binding constraint anymore. Kept for the
|
| 316 |
+
# final filter_chain-refresh fallback path + cold-start edge cases.
|
| 317 |
self.total_budget_s = total_budget_s if total_budget_s is not None else max(timeout * 2.5, 30.0)
|
| 318 |
self.per_model_attempts = per_model_attempts
|
| 319 |
self.role = role # 'brain' | 'fast_brain' | 'judge' | 'unknown' — flows into usage log
|
| 320 |
+
self._chain_name = role if role in ("brain", "fast_brain", "judge") else "unknown"
|
| 321 |
self.model = chain[0]
|
| 322 |
self.name = f"nim-chain::{self._short_id(chain[0])}"
|
| 323 |
|
|
|
|
| 381 |
if "nemotron" in m or m.startswith("nvidia/"): return "nvidia"
|
| 382 |
return "unknown"
|
| 383 |
|
| 384 |
+
# KI-080 — per-call timeout used in the sticky-primary path. Each elected
|
| 385 |
+
# candidate gets at most this long; with at most 2 calls per turn this
|
| 386 |
+
# cleanly fits inside any caller's outer wait_for cap (25s for fact-find,
|
| 387 |
+
# higher for brain/judge).
|
| 388 |
+
_ELECTED_CALL_TIMEOUT_S = 12.0
|
| 389 |
+
|
| 390 |
+
async def _call_one(
|
| 391 |
+
self,
|
| 392 |
+
model: str,
|
| 393 |
+
*,
|
| 394 |
+
messages: list[ChatMessage],
|
| 395 |
+
temperature: float,
|
| 396 |
+
max_tokens: int,
|
| 397 |
+
response_format: Optional[dict],
|
| 398 |
+
timeout: float,
|
| 399 |
+
) -> LLMResult:
|
| 400 |
+
"""Single-model HTTP call. Extracted from the old chain-iteration
|
| 401 |
+
loop (KI-080) so chat() can call a SPECIFIC candidate ONCE without
|
| 402 |
+
the iterate-the-chain envelope.
|
| 403 |
+
|
| 404 |
+
Raises whatever the underlying provider raises (TimeoutException /
|
| 405 |
+
HTTPStatusError / network error). Caller is responsible for failure
|
| 406 |
+
handling (report_failure + fall through to backup).
|
| 407 |
+
"""
|
| 408 |
+
worker = self._get_worker_for(model, timeout)
|
| 409 |
+
return await worker.chat(
|
| 410 |
+
messages=messages,
|
| 411 |
+
temperature=temperature,
|
| 412 |
+
max_tokens=max_tokens,
|
| 413 |
+
response_format=response_format,
|
| 414 |
+
)
|
| 415 |
+
|
| 416 |
async def chat(
|
| 417 |
self,
|
| 418 |
messages: list[ChatMessage],
|
|
|
|
| 422 |
exclude_models: Optional[list[str]] = None,
|
| 423 |
exclude_families: Optional[list[str]] = None,
|
| 424 |
) -> LLMResult:
|
| 425 |
+
"""KI-080 — sticky primary election. Call ONE elected candidate per
|
| 426 |
+
turn, with at most ONE real-time fallback to the elected backup if
|
| 427 |
+
primary fails. The chain list is the candidate POOL for election —
|
| 428 |
+
NOT a per-call sequence.
|
| 429 |
+
|
| 430 |
+
Cold-start path (no probe data yet): use self.chain[0] / [1].
|
| 431 |
+
Brain/judge family-exclusion (e.g. brain doesn't grade own
|
| 432 |
+
homework): apply exclusions to both elected primary + backup; if
|
| 433 |
+
either is excluded, re-elect from the filtered pool by scanning
|
| 434 |
+
the chain in order.
|
| 435 |
+
Final fallback (both elected models fail): trigger a synchronous
|
| 436 |
+
probe refresh + try whatever filter_chain now offers, walking the
|
| 437 |
+
chain in order. This path is the pre-KI-080 graceful-degradation
|
| 438 |
+
safety net for the double-failure edge case.
|
| 439 |
+
"""
|
| 440 |
+
from backend import llm_health
|
| 441 |
+
|
| 442 |
# Apply caller's exclusion list FIRST (brain doesn't grade own homework).
|
| 443 |
+
excl_m = set(exclude_models or [])
|
| 444 |
+
excl_f = set(exclude_families or [])
|
| 445 |
+
def _allowed(m: Optional[str]) -> bool:
|
| 446 |
+
if not m:
|
| 447 |
+
return False
|
| 448 |
+
if m in excl_m:
|
| 449 |
+
return False
|
| 450 |
+
if self._family_of(m) in excl_f:
|
| 451 |
+
return False
|
| 452 |
+
return True
|
| 453 |
+
|
| 454 |
+
# Filter the chain by exclusions (kept for the final-fallback path
|
| 455 |
+
# + cold-start primary/backup election).
|
| 456 |
+
allowed_chain = [m for m in self.chain if _allowed(m)]
|
| 457 |
+
if not allowed_chain:
|
| 458 |
+
# Every candidate excluded — relax family constraint, keep exact
|
| 459 |
+
# model constraint. Better to use a same-family model than to
|
| 460 |
+
# fail the request entirely.
|
| 461 |
+
allowed_chain = [m for m in self.chain if m not in excl_m]
|
| 462 |
+
if not allowed_chain:
|
| 463 |
+
raise RuntimeError("NimChainLLM: every chain candidate is excluded.")
|
|
|
|
|
|
|
|
|
|
| 464 |
|
| 465 |
chain_primary = self.chain[0] if self.chain else None
|
| 466 |
call_t0 = time.time()
|
| 467 |
|
| 468 |
+
# --- Election (KI-080) ----------------------------------------------
|
| 469 |
+
elected_primary: Optional[str] = None
|
| 470 |
+
elected_backup: Optional[str] = None
|
| 471 |
+
try:
|
| 472 |
+
primary_candidate = llm_health.get_primary(self._chain_name)
|
| 473 |
+
backup_candidate = llm_health.get_backup(self._chain_name)
|
| 474 |
+
except Exception:
|
| 475 |
+
primary_candidate, backup_candidate = None, None
|
| 476 |
+
|
| 477 |
+
if primary_candidate and _allowed(primary_candidate):
|
| 478 |
+
elected_primary = primary_candidate
|
| 479 |
+
else:
|
| 480 |
+
# Cold-start / excluded election → first allowed chain entry.
|
| 481 |
+
elected_primary = allowed_chain[0]
|
| 482 |
+
|
| 483 |
+
if backup_candidate and _allowed(backup_candidate) and backup_candidate != elected_primary:
|
| 484 |
+
elected_backup = backup_candidate
|
| 485 |
+
else:
|
| 486 |
+
# Cold-start / excluded election → first allowed chain entry
|
| 487 |
+
# that isn't the elected primary. Prefer a different provider.
|
| 488 |
+
for m in allowed_chain:
|
| 489 |
+
if m != elected_primary and llm_health.provider_of(m) != llm_health.provider_of(elected_primary):
|
| 490 |
+
elected_backup = m
|
| 491 |
+
break
|
| 492 |
+
if elected_backup is None:
|
| 493 |
+
for m in allowed_chain:
|
| 494 |
+
if m != elected_primary:
|
| 495 |
+
elected_backup = m
|
| 496 |
+
break
|
| 497 |
+
|
| 498 |
+
tried: list[str] = []
|
| 499 |
last_err: Optional[Exception] = None
|
| 500 |
+
|
| 501 |
+
async def _try(model: str) -> Optional[LLMResult]:
|
| 502 |
+
"""Attempt one elected candidate. On success: stamp self.model
|
| 503 |
+
+ self.name, report_success, log usage, return result. On
|
| 504 |
+
failure: report_failure, mutate last_err, return None.
|
| 505 |
+
|
| 506 |
+
CancelledError / KeyboardInterrupt / SystemExit are re-raised
|
| 507 |
+
(KI-078) so an outer wait_for cancellation bubbles up
|
| 508 |
+
instead of getting swallowed by the fallback path.
|
| 509 |
+
"""
|
| 510 |
+
nonlocal last_err
|
| 511 |
+
tried.append(model)
|
| 512 |
+
attempt_t0 = time.time()
|
| 513 |
try:
|
| 514 |
+
result = await self._call_one(
|
| 515 |
+
model,
|
| 516 |
+
messages=messages,
|
| 517 |
+
temperature=temperature,
|
| 518 |
+
max_tokens=max_tokens,
|
| 519 |
+
response_format=response_format,
|
| 520 |
+
timeout=self._ELECTED_CALL_TIMEOUT_S,
|
| 521 |
+
)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 522 |
except (asyncio.CancelledError, KeyboardInterrupt, SystemExit):
|
| 523 |
+
# KI-078 — propagate cancellation immediately.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 524 |
raise
|
| 525 |
except Exception as e:
|
|
|
|
| 526 |
last_err = e
|
| 527 |
+
try:
|
| 528 |
+
llm_health.report_failure(
|
| 529 |
+
self._chain_name, model, type(e).__name__
|
| 530 |
+
)
|
| 531 |
+
except Exception:
|
| 532 |
+
pass
|
| 533 |
+
return None
|
| 534 |
+
|
| 535 |
+
# Success — stamp + log.
|
| 536 |
+
latency_ms = int((time.time() - attempt_t0) * 1000)
|
| 537 |
+
try:
|
| 538 |
+
llm_health.report_success(self._chain_name, model, latency_ms)
|
| 539 |
+
except Exception:
|
| 540 |
+
pass
|
| 541 |
+
self.model = model
|
| 542 |
+
self.name = f"nim-chain::{self._short_id(model)}"
|
| 543 |
+
total_ms = int((time.time() - call_t0) * 1000)
|
| 544 |
+
await _append_usage({
|
| 545 |
+
"ts": _now_iso_z(),
|
| 546 |
+
"role": self.role,
|
| 547 |
+
"chain_primary": chain_primary,
|
| 548 |
+
"served_model": model,
|
| 549 |
+
"latency_ms": total_ms,
|
| 550 |
+
"success": True,
|
| 551 |
+
"elected_primary": elected_primary,
|
| 552 |
+
"elected_backup": elected_backup,
|
| 553 |
+
})
|
| 554 |
+
return result
|
| 555 |
+
|
| 556 |
+
# --- Primary attempt -------------------------------------------------
|
| 557 |
+
res = await _try(elected_primary)
|
| 558 |
+
if res is not None:
|
| 559 |
+
return res
|
| 560 |
+
|
| 561 |
+
# --- Backup attempt (KI-080 single real-time fallback) ---------------
|
| 562 |
+
if elected_backup and elected_backup != elected_primary:
|
| 563 |
+
res = await _try(elected_backup)
|
| 564 |
+
if res is not None:
|
| 565 |
+
return res
|
| 566 |
+
|
| 567 |
+
# --- Both elected failed — final safety net --------------------------
|
| 568 |
+
# Trigger ONE synchronous probe refresh; whatever the refreshed
|
| 569 |
+
# filter_chain offers, walk it in order and try anything we haven't
|
| 570 |
+
# touched this turn. This is the pre-KI-080 graceful-degradation
|
| 571 |
+
# behaviour preserved for the (rare) double-failure case.
|
| 572 |
try:
|
|
|
|
| 573 |
await llm_health.probe_all()
|
| 574 |
+
refreshed = llm_health.filter_chain(allowed_chain)
|
| 575 |
for model in refreshed:
|
| 576 |
+
if model in tried:
|
| 577 |
+
continue
|
|
|
|
| 578 |
elapsed = time.time() - call_t0
|
| 579 |
if elapsed >= self.total_budget_s:
|
| 580 |
break
|
| 581 |
+
attempt_t0 = time.time()
|
| 582 |
try:
|
| 583 |
+
per_link_timeout = min(
|
| 584 |
+
self._ELECTED_CALL_TIMEOUT_S,
|
| 585 |
+
max(2.0, self.total_budget_s - elapsed),
|
| 586 |
+
)
|
| 587 |
+
result = await self._call_one(
|
| 588 |
+
model,
|
| 589 |
+
messages=messages,
|
| 590 |
+
temperature=temperature,
|
| 591 |
+
max_tokens=max_tokens,
|
| 592 |
+
response_format=response_format,
|
| 593 |
+
timeout=per_link_timeout,
|
| 594 |
+
)
|
| 595 |
+
except (asyncio.CancelledError, KeyboardInterrupt, SystemExit):
|
| 596 |
+
raise
|
|
|
|
|
|
|
| 597 |
except Exception as e:
|
| 598 |
last_err = e
|
| 599 |
+
try:
|
| 600 |
+
llm_health.report_failure(
|
| 601 |
+
self._chain_name, model, type(e).__name__
|
| 602 |
+
)
|
| 603 |
+
except Exception:
|
| 604 |
+
pass
|
| 605 |
continue
|
| 606 |
+
|
| 607 |
+
try:
|
| 608 |
+
llm_health.report_success(
|
| 609 |
+
self._chain_name, model, int((time.time() - attempt_t0) * 1000)
|
| 610 |
+
)
|
| 611 |
+
except Exception:
|
| 612 |
+
pass
|
| 613 |
+
self.model = model
|
| 614 |
+
self.name = f"nim-chain::{self._short_id(model)}"
|
| 615 |
+
total_ms = int((time.time() - call_t0) * 1000)
|
| 616 |
+
await _append_usage({
|
| 617 |
+
"ts": _now_iso_z(),
|
| 618 |
+
"role": self.role,
|
| 619 |
+
"chain_primary": chain_primary,
|
| 620 |
+
"served_model": model,
|
| 621 |
+
"latency_ms": total_ms,
|
| 622 |
+
"success": True,
|
| 623 |
+
"elected_primary": elected_primary,
|
| 624 |
+
"elected_backup": elected_backup,
|
| 625 |
+
"fallback_phase": "post_reprobe",
|
| 626 |
+
})
|
| 627 |
+
return result
|
| 628 |
+
except (asyncio.CancelledError, KeyboardInterrupt, SystemExit):
|
| 629 |
+
raise
|
| 630 |
except Exception:
|
| 631 |
+
pass # probe refresh failure must never block the outer raise
|
| 632 |
|
| 633 |
+
# Total exhaustion — log + raise. fact_find_brain.drive_fact_find
|
| 634 |
+
# (KI-079) catches the resulting RuntimeError + escalates one more
|
| 635 |
+
# time to BRAIN_CHAIN before falling to the canonical reply.
|
| 636 |
latency_ms = int((time.time() - call_t0) * 1000)
|
| 637 |
await _append_usage({
|
| 638 |
"ts": _now_iso_z(),
|
|
|
|
| 641 |
"served_model": None,
|
| 642 |
"latency_ms": latency_ms,
|
| 643 |
"success": False,
|
| 644 |
+
"elected_primary": elected_primary,
|
| 645 |
+
"elected_backup": elected_backup,
|
| 646 |
+
"tried": tried,
|
| 647 |
})
|
| 648 |
raise RuntimeError(
|
| 649 |
+
f"NimChainLLM ({self._chain_name}) elected primary={elected_primary} "
|
| 650 |
+
f"and backup={elected_backup} both failed; tried={tried}. "
|
| 651 |
+
f"Last error: {type(last_err).__name__ if last_err else 'None'}: "
|
| 652 |
+
f"{str(last_err)[:120] if last_err else ''}"
|
| 653 |
) from last_err
|
| 654 |
|
| 655 |
|
| 656 |
# KI-025 (2026-05-14) — provider load-balancing.
|
| 657 |
+
# DEPRECATED 2026-05-15 by KI-080: primary election supersedes the 50/50
|
| 658 |
+
# rotation. The probe loop now picks the actually-faster candidate
|
| 659 |
+
# DYNAMICALLY (real probe latencies, not a coin flip), and the elected
|
| 660 |
+
# backup is chosen with explicit cross-provider preference — both signals
|
| 661 |
+
# the rotation was approximating heuristically. Kept around (not deleted)
|
| 662 |
+
# because the regression suite still pins its statistical behaviour. The
|
| 663 |
+
# get_*_llm factories no longer call it.
|
| 664 |
import random as _random
|
| 665 |
|
| 666 |
|
| 667 |
def _balanced_brain_chain(base: list[str], *, groq_first_probability: float = 0.5) -> list[str]:
|
| 668 |
+
"""KI-025 — provider load-balancing (DEPRECATED 2026-05-15 by KI-080).
|
| 669 |
+
|
| 670 |
+
With `groq_first_probability` (default 50%), hoist the Groq Llama entry
|
| 671 |
+
to the head of the chain so it serves as the primary instead of the
|
| 672 |
+
NIM Qwen entry. The remaining candidates stay in their existing
|
| 673 |
+
fallback order — Groq calls that fail (rare; LPU is very reliable)
|
| 674 |
+
still get the full NIM fallback chain.
|
| 675 |
+
|
| 676 |
+
SUPERSESSION NOTE (KI-080): the elector in `backend.llm_health` now
|
| 677 |
+
picks the actually-faster candidate dynamically from background probe
|
| 678 |
+
data, so this static-coin-flip rotation is no longer wired into the
|
| 679 |
+
chat hot path. Kept exported because:
|
| 680 |
+
(a) regression tests still pin its statistical behaviour, and
|
| 681 |
+
(b) it remains a useful pure-function for ops / sim / overrides
|
| 682 |
+
(e.g. wanting to force a non-elected order in a debug script).
|
| 683 |
|
| 684 |
Uses per-call `random.random()` so concurrent async workers (the
|
| 685 |
100-persona audit's 4 workers, the parallel 96-Q eval's 6 workers) each
|
|
|
|
| 697 |
|
| 698 |
|
| 699 |
def get_brain_llm() -> NimChainLLM:
|
| 700 |
+
"""Heavy brain — KI-080 sticky-primary election over BRAIN_CHAIN.
|
| 701 |
+
|
| 702 |
+
Pre-KI-080: per-call _balanced_brain_chain rotation between NIM Qwen
|
| 703 |
+
and Groq Llama (KI-025 50/50 heuristic) → 5-6 LLM calls per turn
|
| 704 |
+
under degraded conditions.
|
| 705 |
+
|
| 706 |
+
Post-KI-080: the background probe loop in backend.llm_health elects
|
| 707 |
+
the actually-fastest candidate dynamically (probe-driven, not coin-
|
| 708 |
+
flipped) and a cross-provider backup. chat() calls 1-2 candidates max
|
| 709 |
+
per turn. The rotation is preserved as a deprecated pure-function in
|
| 710 |
+
case overrides need it, but the factory no longer wires it in.
|
| 711 |
+
|
| 712 |
+
KI-021 — per-link 12s (KI-080 ELECTED_CALL_TIMEOUT_S), total chain
|
| 713 |
+
budget 35s (only binding for the final filter_chain-refresh safety net).
|
| 714 |
+
"""
|
| 715 |
+
return NimChainLLM(chain=BRAIN_CHAIN, timeout=20.0,
|
| 716 |
role="brain", total_budget_s=35.0)
|
| 717 |
|
| 718 |
|
| 719 |
def get_fast_brain_llm() -> NimChainLLM:
|
| 720 |
+
"""Fast brain — KI-080 sticky-primary election over FAST_BRAIN_CHAIN.
|
| 721 |
+
|
| 722 |
+
Pre-KI-080 (KI-025 + KI-078 + KI-079): per-call rotation between NIM
|
| 723 |
+
Qwen and Groq Llama, 6s per-link timeout, 22s total chain budget,
|
| 724 |
+
Groq promoted to chain position #2 so a single NIM degradation could
|
| 725 |
+
fall through fast enough. Even so, the 10-turn live probe at commit
|
| 726 |
+
078ff45 showed 7/10 fact-find turns hitting the wait_for cap at 26.6s
|
| 727 |
+
because every chain-iteration burned the full budget exploring 5+
|
| 728 |
+
queued NIM candidates.
|
| 729 |
+
|
| 730 |
+
Post-KI-080: probe-driven election picks ONE primary per chain (the
|
| 731 |
+
fastest-responding healthy candidate by rolling probe data) and ONE
|
| 732 |
+
cross-provider backup. chat() invokes the primary ONCE with a 12s
|
| 733 |
+
timeout, falls to the backup ONCE on failure, and only walks the
|
| 734 |
+
filter_chain fallback as a final safety net. The KI-079 escalation in
|
| 735 |
+
fact_find_brain still runs if primary+backup both fail, providing one
|
| 736 |
+
more bite via BRAIN_CHAIN.
|
| 737 |
+
|
| 738 |
+
KI-025 supersession: the 50/50 rotation is no longer wired in — see
|
| 739 |
+
the docstring on `_balanced_brain_chain` for the reasoning.
|
| 740 |
+
"""
|
| 741 |
+
return NimChainLLM(chain=FAST_BRAIN_CHAIN, timeout=6.0,
|
| 742 |
role="fast_brain", total_budget_s=22.0)
|
| 743 |
|
| 744 |
|