rohitsar567 Claude Opus 4.7 (1M context) commited on
Commit
6159c54
·
1 Parent(s): 87ee522

feat(llm-chain): KI-080 — sticky primary election; 1-2 LLM calls per turn instead of 5-6

Browse files

Architectural refactor: the chain list is now the CANDIDATE POOL for
probe-driven election, not a per-call sequence. Pre-KI-080, NimChainLLM.chat
iterated the full chain sequentially every turn. Under NIM per-key
concurrency throttling, the first 5 NIM-hosted candidates queued together
inside ONE turn, burning the 22s budget before any cross-provider fallback
(Groq / OpenRouter) was ever reached. The 10-turn live probe at commit
078ff45 showed 7/10 fact-find turns timing out at exactly 26.6s — every
failure a timeout, never a parse error.

ELECTION
========
backend/llm_health.py:
- Probe cadence tightened 5min → 60s so election reflects pool health
fast enough that a NIM brownout rotates out within a minute.
- Each candidate carries probe_history (last 5 ok/latency pairs) and
a degraded_until_monotonic sin-bin window.
- score = (1/max(50, latency_ms)) * success_rate_last_5
- get_primary(chain_name) → highest scorer with a probe within 90s.
- get_backup(chain_name) → highest scorer with a DIFFERENT provider
(NIM vs Groq vs OpenRouter) than primary; falls back to next-best
same-provider candidate when no cross-provider option qualifies.
- report_failure(chain, model, error_class): sidelines model for 30s +
schedules an async re-probe so the next turn elects a fresh primary.
- report_success(chain, model, latency_ms): folds chat latency into
the rolling window (chat traffic is richer signal than 1-token probes).
- status_summary now publishes the current elected primary/backup per
chain for the admin UI.

STICKY-PRIMARY chat()
=====================
backend/providers/nvidia_nim_llm.py:
- _call_one(model, ...) extracted from the old chain-iteration body —
single-model HTTP call, raises on failure for the caller to handle.
- chat() rewritten:
1. Apply caller exclusion list (brain ↦ judge family independence).
2. Resolve elected_primary / elected_backup via llm_health; if
either is excluded or election is cold-start, fall back to the
first allowed chain entry (cross-provider preference for backup).
3. Call elected_primary ONCE (12s timeout). On success → return.
4. On failure → report_failure(primary) + call elected_backup ONCE.
On success → return; on failure → report_failure(backup).
5. Final safety net (rare double-failure): probe_all() + walk
filter_chain in order. Preserves pre-KI-080 graceful degradation.
- CancelledError/KeyboardInterrupt/SystemExit re-raised at every hop
(KI-078 invariant preserved).

COST REDUCTION
==============
Pre-KI-080 worst case per turn: 5-6 LLM calls (full chain iteration).
Post-KI-080 worst case per turn: 2 LLM calls (primary + one backup).
Final probe-refresh safety net adds ≤chain_length more, but only
when BOTH elected models fail in the same turn — vanishingly rare
once probe-driven election is rotating out degraded candidates
within 60s.

INTERACTION WITH OTHER KIs
==========================
- KI-079 escalation in fact_find_brain UNCHANGED. If primary+backup
both fail and chat() raises RuntimeError, drive_fact_find still
catches it + escalates one more time via BRAIN_CHAIN before
falling to the canonical reply. Total worst case: 2 fast-brain
calls + 2 brain calls + canonical fallback.
- KI-078 narrow exception catch UNCHANGED — CancelledError /
KeyboardInterrupt / SystemExit are re-raised at every hop in
the new chat() so an outer wait_for cancellation bubbles up
instead of being swallowed.
- KI-025 50/50 NIM/Groq rotation is DEPRECATED (not deleted). The
probe-driven elector now picks the actually-faster candidate
dynamically — what the static coin flip was approximating
heuristically. _balanced_brain_chain stays exported because the
regression suite pins its statistical behaviour + it remains a
useful pure function for ops scripts. The get_*_llm factories no
longer wire it into the chat hot path.
- filter_chain (KI-022) still used by the final-fallback path and
by admin endpoints; kept unchanged.

TESTING
=======
- py_compile both files: pass.
- tests/test_routing_regression.py: 15/15 pass (includes the KI-025
rotation pinning suite, confirming the deprecated function still
behaves as documented).
- Inline unit tests (7/7 pass): cold-start election returns None,
primary picks lowest-latency healthy candidate, backup prefers
cross-provider, report_failure demotes within the window,
report_success updates score, primary-succeeds path makes ONE call,
primary-fails-backup-succeeds path makes exactly TWO calls.

EXPECTED PRODUCTION IMPACT
==========================
Brain success rate (fact-find turns reaching the elected primary
in <12s): 30% (today) → 90%+ projected, since election picks the
actually-fast pool and skips degraded NIM ingresses entirely.
NIM req/min consumption: ~6× lower under degraded conditions —
one primary call per turn instead of five queued candidates.
p50 latency: ~2s (elected Nemotron Nano / Qwen primary).
p95 latency: ~12s (primary timeout + backup success).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

backend/llm_health.py CHANGED
@@ -1,29 +1,67 @@
1
- """Live NIM model health monitor — OpenRouter-style availability filter.
2
-
3
- Probes every model in BRAIN_CHAIN + FAST_BRAIN_CHAIN + JUDGE_CHAIN periodically
4
- with a tiny ping ("reply 'ok'"). Records:
5
- - status: healthy / degraded / down
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
6
  - last_success_at: timestamp of last 2xx response
7
  - last_failure_at: timestamp of last failure (timeout / 5xx / parse)
8
  - latency_ms: last successful response latency
9
  - consecutive_fail: counter (3+ => marked down)
10
  - tested_at: when this row was last refreshed
 
 
 
 
 
 
 
 
 
 
 
 
 
 
11
 
12
  Persistence: 40-data/llm_health.json (atomic write via temp+rename).
13
 
14
- Consumers:
15
- - NimChainLLM.chat() filters the chain to status != 'down' before iterating.
16
- - GET /api/health/llms returns the current state for the frontend.
 
 
 
 
 
 
 
17
 
18
  Operating schedule:
19
  - background asyncio task in main.py startup, ticks every PROBE_INTERVAL_SEC.
20
- - on-demand refresh via probe_all() (e.g. when NimChainLLM exhausts the chain).
 
21
  """
22
  from __future__ import annotations
23
 
24
  import asyncio
25
  import json
26
  import os
 
27
  import time
28
  from dataclasses import dataclass, field, asdict
29
  from pathlib import Path
@@ -35,9 +73,17 @@ ROOT = Path(__file__).resolve().parent.parent
35
  HEALTH_FILE = ROOT / "40-data" / "llm_health.json"
36
  HEALTH_FILE.parent.mkdir(parents=True, exist_ok=True)
37
 
38
- PROBE_INTERVAL_SEC = 300 # ping each model every 5 min
39
- PROBE_TIMEOUT_SEC = 12 # per-probe HTTP timeout
 
 
 
 
40
  DOWN_AFTER_CONSECUTIVE_FAILS = 3 # 3 fails in a row = mark down
 
 
 
 
41
 
42
  # Per-provider endpoints + env-var names. The chain entries embed the
43
  # provider via a prefix ('openrouter:<id>' / 'groq:<id>'); unprefixed entries
@@ -83,6 +129,21 @@ def _headers_for(model_id: str, api_key: str) -> dict[str, str]:
83
  return h
84
 
85
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
86
  @dataclass
87
  class ModelHealth:
88
  model: str
@@ -93,12 +154,30 @@ class ModelHealth:
93
  latency_ms: Optional[int] = None
94
  consecutive_failures: int = 0
95
  tested_at: Optional[str] = None
 
 
 
 
 
 
 
96
 
97
 
98
  def _now_iso() -> str:
99
  return time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())
100
 
101
 
 
 
 
 
 
 
 
 
 
 
 
102
  def _all_known_models() -> list[str]:
103
  """Pull the union of every model name from every chain at module import."""
104
  from backend.providers.nvidia_nim_llm import (
@@ -112,18 +191,68 @@ def _all_known_models() -> list[str]:
112
  return seen
113
 
114
 
115
- def load() -> dict[str, ModelHealth]:
116
- if not HEALTH_FILE.exists():
117
- return {}
118
- try:
119
- raw = json.loads(HEALTH_FILE.read_text())
120
- return {k: ModelHealth(**v) for k, v in raw.get("models", {}).items()}
121
- except Exception:
122
- return {}
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
123
 
124
 
125
- def save(state: dict[str, ModelHealth]) -> None:
126
- """Atomic write temp file then rename so concurrent readers never see partial."""
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
127
  out = {
128
  "updated_at": _now_iso(),
129
  "models": {k: asdict(v) for k, v in state.items()},
@@ -133,6 +262,180 @@ def save(state: dict[str, ModelHealth]) -> None:
133
  tmp.replace(HEALTH_FILE)
134
 
135
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
136
  async def probe_one(client: httpx.AsyncClient, model: str, api_key: str) -> tuple[bool, str, Optional[int]]:
137
  """Probe a single chain entry (NIM or cross-provider).
138
 
@@ -187,28 +490,11 @@ async def probe_one(client: httpx.AsyncClient, model: str, api_key: str) -> tupl
187
  return False, f"net: {type(e).__name__}: {str(e)[:60]}", int((time.time() - t0) * 1000)
188
 
189
 
190
- async def probe_all() -> dict[str, ModelHealth]:
191
- """One-shot probe of every known model. Updates persisted state + returns it."""
192
- state = load()
193
- models = _all_known_models()
194
- api_key = os.environ.get("NVIDIA_NIM_API_KEY", "")
195
- if not api_key:
196
- return state
197
-
198
- async with httpx.AsyncClient() as client:
199
- # Probe all in parallel — they hit different NIM pools so concurrency is fine
200
- results = await asyncio.gather(
201
- *[probe_one(client, m, api_key) for m in models],
202
- return_exceptions=True,
203
- )
204
-
205
- for model, result in zip(models, results):
206
- if isinstance(result, Exception):
207
- ok, err, latency = False, f"exc: {type(result).__name__}", None
208
- else:
209
- ok, err, latency = result
210
-
211
- h = state.get(model, ModelHealth(model=model))
212
  h.tested_at = _now_iso()
213
  if ok:
214
  h.last_success_at = _now_iso()
@@ -224,15 +510,51 @@ async def probe_all() -> dict[str, ModelHealth]:
224
  h.status = "down"
225
  else:
226
  h.status = "degraded"
227
- state[model] = h
 
 
 
 
 
 
 
 
228
 
229
- save(state)
230
- return state
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
231
 
232
 
233
  def filter_chain(chain: list[str]) -> list[str]:
234
  """Return the chain with 'down' models removed. Always preserves order +
235
- keeps at least one model so callers never get an empty chain."""
 
 
 
 
236
  state = load()
237
  keep = [m for m in chain if state.get(m, ModelHealth(model=m)).status != "down"]
238
  if not keep:
@@ -258,6 +580,12 @@ def status_summary() -> dict:
258
  summary["models"].sort(key=lambda x: (x["status"] != "healthy", x["model"]))
259
  if summary["models"]:
260
  summary["updated_at"] = max((m.get("tested_at") or "") for m in summary["models"])
 
 
 
 
 
 
261
  return summary
262
 
263
 
@@ -280,3 +608,7 @@ if __name__ == "__main__":
280
  for m, h in sorted(state.items(), key=lambda kv: (kv[1].status != "healthy", kv[0])):
281
  latency = f"{h.latency_ms}ms" if h.latency_ms else "—"
282
  print(f" {h.status:8s} {latency:>8s} {m:50s} {h.last_error or 'ok'}")
 
 
 
 
 
1
+ """Live NIM model health monitor — OpenRouter-style availability filter
2
+ + KI-080 sticky primary/backup election.
3
+
4
+ Architectural shift (KI-080, 2026-05-15):
5
+ ==========================================
6
+ Pre-KI-080 the chain was iterated EVERY chat turn — primary, fallback 1,
7
+ fallback 2, ... until one succeeded. Under NIM per-key concurrency throttling,
8
+ the first 5 NIM-hosted candidates queued together inside a single turn,
9
+ burning the 22s budget before the cross-provider fallback links (Groq /
10
+ OpenRouter) were ever reached. The 10-turn live probe at commit 078ff45
11
+ showed 7/10 fact-find turns timing out at exactly 26.6s.
12
+
13
+ KI-080 inverts the model: the background probe loop ELECTS a primary +
14
+ backup per chain based on real probe latencies; chat() calls the elected
15
+ primary ONCE per turn, with at most ONE real-time fallback to the elected
16
+ backup. Worst case per turn drops from 5-6 LLM calls to 1-2.
17
+
18
+ Probes every PROBE_INTERVAL_SEC tick with a tiny ping ("Reply with exactly: ok").
19
+ Records per model:
20
+ - status: healthy / degraded / down / unknown
21
  - last_success_at: timestamp of last 2xx response
22
  - last_failure_at: timestamp of last failure (timeout / 5xx / parse)
23
  - latency_ms: last successful response latency
24
  - consecutive_fail: counter (3+ => marked down)
25
  - tested_at: when this row was last refreshed
26
+ - probe_history: last PROBE_HISTORY_LEN (ok, latency_ms) tuples; powers
27
+ the success_rate signal in the election score.
28
+
29
+ Election (KI-080):
30
+ - For each chain (brain / fast_brain / judge), compute a score per healthy
31
+ candidate: score = (1 / max(50, latency_ms)) * success_rate_last_5.
32
+ - CURRENT_PRIMARY = highest scorer.
33
+ - CURRENT_BACKUP = highest scorer among candidates with a DIFFERENT
34
+ provider (NIM vs Groq vs OpenRouter) than primary; falls back to next-
35
+ best same-provider candidate when no cross-provider option qualifies.
36
+ - DEGRADED window: when chat() calls report_failure(model), that model is
37
+ sidelined for DEGRADED_WINDOW_SEC so the same turn's failure doesn't
38
+ recycle to the same broken primary on the next turn. The next probe
39
+ tick reconsiders the model normally.
40
 
41
  Persistence: 40-data/llm_health.json (atomic write via temp+rename).
42
 
43
+ Public API (the surface NimChainLLM.chat consumes):
44
+ get_primary(chain_name) -> Optional[str]
45
+ get_backup(chain_name) -> Optional[str]
46
+ report_failure(chain, model, error_class)
47
+ report_success(chain, model, latency_ms)
48
+
49
+ Legacy:
50
+ filter_chain(chain) -> chain with 'down' models removed (still used
51
+ by admin / probe-refresh paths)
52
+ status_summary() -> compact dict for GET /api/health/llms
53
 
54
  Operating schedule:
55
  - background asyncio task in main.py startup, ticks every PROBE_INTERVAL_SEC.
56
+ - on-demand refresh via probe_all() (e.g. when NimChainLLM exhausts both
57
+ primary + backup in a single turn).
58
  """
59
  from __future__ import annotations
60
 
61
  import asyncio
62
  import json
63
  import os
64
+ import threading
65
  import time
66
  from dataclasses import dataclass, field, asdict
67
  from pathlib import Path
 
73
  HEALTH_FILE = ROOT / "40-data" / "llm_health.json"
74
  HEALTH_FILE.parent.mkdir(parents=True, exist_ok=True)
75
 
76
+ # KI-080 probe every 60s so election reflects real-time pool health
77
+ # fast enough that a NIM pool brownout is rotated out within a minute.
78
+ # The 5-min cadence pre-KI-080 was fine for a "filter the dead" use case
79
+ # but too slow when probe results drive primary election.
80
+ PROBE_INTERVAL_SEC = 60
81
+ PROBE_TIMEOUT_SEC = 8 # per-probe HTTP timeout
82
  DOWN_AFTER_CONSECUTIVE_FAILS = 3 # 3 fails in a row = mark down
83
+ PROBE_HISTORY_LEN = 5 # rolling window for success_rate signal
84
+ HEALTHY_PROBE_AGE_SEC = 90 # election candidates need a probe within
85
+ # the last 90s — stale data excluded
86
+ DEGRADED_WINDOW_SEC = 30 # report_failure sidelines a model this long
87
 
88
  # Per-provider endpoints + env-var names. The chain entries embed the
89
  # provider via a prefix ('openrouter:<id>' / 'groq:<id>'); unprefixed entries
 
129
  return h
130
 
131
 
132
+ def provider_of(model_id: str) -> str:
133
+ """Coarse provider bucket — used by the election routine to prefer a
134
+ cross-provider backup so a NIM regional outage can't take out both
135
+ primary + backup simultaneously. Returns one of: 'nim' | 'groq' |
136
+ 'openrouter'. (NIM is the implicit default for unprefixed model ids
137
+ even though NIM hosts many model families — all of those share the
138
+ same NIM ingress + per-key rate quota, which is what we need to
139
+ diversify against.)"""
140
+ if model_id.startswith("openrouter:"):
141
+ return "openrouter"
142
+ if model_id.startswith("groq:"):
143
+ return "groq"
144
+ return "nim"
145
+
146
+
147
  @dataclass
148
  class ModelHealth:
149
  model: str
 
154
  latency_ms: Optional[int] = None
155
  consecutive_failures: int = 0
156
  tested_at: Optional[str] = None
157
+ # KI-080 — rolling probe history powers the success_rate signal in
158
+ # the election score. Each entry: {"ok": bool, "latency_ms": int|None,
159
+ # "ts": iso8601}. Capped at PROBE_HISTORY_LEN.
160
+ probe_history: list[dict] = field(default_factory=list)
161
+ # KI-080 — set by report_failure(); model is excluded from election
162
+ # while monotonic time < degraded_until_monotonic.
163
+ degraded_until_monotonic: float = 0.0
164
 
165
 
166
  def _now_iso() -> str:
167
  return time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())
168
 
169
 
170
+ def _iso_age_seconds(iso_ts: Optional[str]) -> Optional[float]:
171
+ """Seconds since an ISO timestamp; None if missing/unparseable."""
172
+ if not iso_ts:
173
+ return None
174
+ try:
175
+ t = time.strptime(iso_ts, "%Y-%m-%dT%H:%M:%SZ")
176
+ return time.time() - time.mktime(t) + time.timezone
177
+ except Exception:
178
+ return None
179
+
180
+
181
  def _all_known_models() -> list[str]:
182
  """Pull the union of every model name from every chain at module import."""
183
  from backend.providers.nvidia_nim_llm import (
 
191
  return seen
192
 
193
 
194
+ def _chain_for(chain_name: str) -> list[str]:
195
+ """Resolve a chain name → live chain list. Reads off the module so
196
+ runtime admin reorderings are respected by the elector."""
197
+ from backend.providers import nvidia_nim_llm as nim
198
+ if chain_name == "brain":
199
+ return list(getattr(nim, "BRAIN_CHAIN", []))
200
+ if chain_name == "fast_brain":
201
+ return list(getattr(nim, "FAST_BRAIN_CHAIN", []))
202
+ if chain_name == "judge":
203
+ return list(getattr(nim, "JUDGE_CHAIN", []))
204
+ return []
205
+
206
+
207
+ # KI-080 — in-process state. Probe history + degraded windows are
208
+ # performance-critical hot paths (read on every chat turn), so we keep
209
+ # them in memory and only persist the long-lived signal to disk on the
210
+ # probe tick. Concurrent NimChainLLM.chat workers + the probe loop both
211
+ # mutate this; a coarse lock is fine — the held region is microseconds.
212
+ _STATE_LOCK = threading.Lock()
213
+ _STATE: dict[str, ModelHealth] = {}
214
+ _STATE_LOADED = False
215
+
216
+
217
+ def _load_into_memory() -> None:
218
+ """Hydrate _STATE from disk on first access. Idempotent."""
219
+ global _STATE, _STATE_LOADED
220
+ if _STATE_LOADED:
221
+ return
222
+ with _STATE_LOCK:
223
+ if _STATE_LOADED:
224
+ return
225
+ if HEALTH_FILE.exists():
226
+ try:
227
+ raw = json.loads(HEALTH_FILE.read_text())
228
+ for k, v in raw.get("models", {}).items():
229
+ # Tolerate older schema (pre-KI-080 records missing
230
+ # probe_history / degraded_until_monotonic).
231
+ v.setdefault("probe_history", [])
232
+ v.setdefault("degraded_until_monotonic", 0.0)
233
+ _STATE[k] = ModelHealth(**v)
234
+ except Exception:
235
+ _STATE = {}
236
+ _STATE_LOADED = True
237
 
238
 
239
+ def load() -> dict[str, ModelHealth]:
240
+ """Legacy/back-compat: snapshot of the in-memory state. Returned dict
241
+ is a shallow copy so callers can't mutate _STATE directly."""
242
+ _load_into_memory()
243
+ with _STATE_LOCK:
244
+ return dict(_STATE)
245
+
246
+
247
+ def save(state: Optional[dict[str, ModelHealth]] = None) -> None:
248
+ """Atomic write — temp file then rename so concurrent readers never see partial.
249
+
250
+ If `state` is None, persists the in-memory _STATE. The optional arg is
251
+ kept for backward compatibility with the pre-KI-080 call sites in admin.py."""
252
+ if state is None:
253
+ _load_into_memory()
254
+ with _STATE_LOCK:
255
+ state = dict(_STATE)
256
  out = {
257
  "updated_at": _now_iso(),
258
  "models": {k: asdict(v) for k, v in state.items()},
 
262
  tmp.replace(HEALTH_FILE)
263
 
264
 
265
+ # ---------------------------------------------------------------------------
266
+ # KI-080 — public API: primary/backup election + failure/success reporting
267
+ # ---------------------------------------------------------------------------
268
+
269
+ def _success_rate(h: ModelHealth) -> float:
270
+ """Fraction of the last PROBE_HISTORY_LEN probes that succeeded.
271
+ Returns 1.0 when there's no history yet (cold-start — give every
272
+ healthy candidate the benefit of the doubt)."""
273
+ if not h.probe_history:
274
+ return 1.0
275
+ hits = sum(1 for r in h.probe_history if r.get("ok"))
276
+ return hits / len(h.probe_history)
277
+
278
+
279
+ def _score(h: ModelHealth) -> float:
280
+ """Election score — higher is better.
281
+
282
+ score = (1 / max(50, latency_ms)) * success_rate
283
+ The 50ms floor stops a sub-millisecond outlier from dominating
284
+ election; success_rate is the rolling-window stability signal.
285
+ """
286
+ if h.latency_ms is None:
287
+ return 0.0
288
+ return (1.0 / max(50, h.latency_ms)) * _success_rate(h)
289
+
290
+
291
+ def _is_election_eligible(h: ModelHealth, now_mono: float) -> bool:
292
+ """A candidate is electable when:
293
+ - status is healthy (or degraded with a recent success)
294
+ - last probe was within HEALTHY_PROBE_AGE_SEC
295
+ - it is NOT currently in the degraded-window sin-bin
296
+ """
297
+ if h.degraded_until_monotonic > now_mono:
298
+ return False
299
+ if h.status == "down":
300
+ return False
301
+ age = _iso_age_seconds(h.tested_at)
302
+ if age is None or age > HEALTHY_PROBE_AGE_SEC:
303
+ return False
304
+ if h.latency_ms is None:
305
+ return False
306
+ return True
307
+
308
+
309
+ def _ranked_candidates(chain_name: str) -> list[ModelHealth]:
310
+ """Return the chain's election-eligible candidates, best-score first."""
311
+ _load_into_memory()
312
+ chain = _chain_for(chain_name)
313
+ now_mono = time.monotonic()
314
+ with _STATE_LOCK:
315
+ snapshot = {m: _STATE.get(m) for m in chain}
316
+ eligible: list[tuple[float, ModelHealth]] = []
317
+ for m in chain:
318
+ h = snapshot.get(m)
319
+ if h is None:
320
+ continue
321
+ if not _is_election_eligible(h, now_mono):
322
+ continue
323
+ eligible.append((_score(h), h))
324
+ eligible.sort(key=lambda t: t[0], reverse=True)
325
+ return [h for _, h in eligible]
326
+
327
+
328
+ def get_primary(chain_name: str) -> Optional[str]:
329
+ """Top-scoring election-eligible candidate, or None when no probe data
330
+ is fresh enough. Callers cold-start by falling back to chain[0]."""
331
+ ranked = _ranked_candidates(chain_name)
332
+ return ranked[0].model if ranked else None
333
+
334
+
335
+ def get_backup(chain_name: str) -> Optional[str]:
336
+ """Second-best election candidate. Prefers a DIFFERENT provider from
337
+ primary (NIM vs Groq vs OpenRouter) so a single provider's regional
338
+ outage can't take out both. Falls back to the next-best same-provider
339
+ candidate when no cross-provider option qualifies — better an in-
340
+ family backup than none."""
341
+ ranked = _ranked_candidates(chain_name)
342
+ if len(ranked) < 2:
343
+ # No usable backup. Either zero candidates or only one (in which
344
+ # case the cold-start path in NimChainLLM falls back to chain[1]).
345
+ return None
346
+ primary_provider = provider_of(ranked[0].model)
347
+ # Prefer cross-provider
348
+ for h in ranked[1:]:
349
+ if provider_of(h.model) != primary_provider:
350
+ return h.model
351
+ # No cross-provider candidate — accept same-provider next-best.
352
+ return ranked[1].model
353
+
354
+
355
+ def report_failure(chain_name: str, model: str, error_class: str) -> None:
356
+ """Called by NimChainLLM.chat() when primary OR backup throws.
357
+
358
+ Effects:
359
+ - Sidelines `model` from election for DEGRADED_WINDOW_SEC so the
360
+ same turn's failure doesn't immediately re-elect the same model.
361
+ - Appends a synthetic 'failed' entry to probe_history so the next
362
+ election's success_rate reflects the live failure even before
363
+ the next probe tick.
364
+ - Schedules an async re-probe (best-effort) so the next turn gets
365
+ fresh data instead of waiting up to 60s for the next tick.
366
+ """
367
+ _load_into_memory()
368
+ with _STATE_LOCK:
369
+ h = _STATE.get(model) or ModelHealth(model=model)
370
+ h.degraded_until_monotonic = time.monotonic() + DEGRADED_WINDOW_SEC
371
+ h.last_failure_at = _now_iso()
372
+ h.last_error = f"chat_failure: {error_class}"
373
+ h.probe_history.append({
374
+ "ok": False,
375
+ "latency_ms": None,
376
+ "ts": _now_iso(),
377
+ "src": "chat",
378
+ })
379
+ if len(h.probe_history) > PROBE_HISTORY_LEN:
380
+ h.probe_history = h.probe_history[-PROBE_HISTORY_LEN:]
381
+ # Don't flip status to 'down' here — that's the probe's job and
382
+ # we don't want a single transient turn-failure to evict the
383
+ # candidate permanently. The degraded sin-bin is sufficient.
384
+ _STATE[model] = h
385
+
386
+ # Best-effort re-probe — fire and forget. We can't await inside this
387
+ # sync API (callers are in the hot path) so we schedule on the loop
388
+ # if one is running.
389
+ try:
390
+ loop = asyncio.get_event_loop()
391
+ if loop.is_running():
392
+ loop.create_task(_reprobe_one(model))
393
+ except RuntimeError:
394
+ pass # no loop (unit tests etc.) — skip silently
395
+
396
+
397
+ def report_success(chain_name: str, model: str, latency_ms: int) -> None:
398
+ """Called by NimChainLLM.chat() on a successful response. Updates
399
+ the rolling success/latency window — important because chat traffic
400
+ is dramatically richer signal than 1-token probes (real prompts,
401
+ real concurrency)."""
402
+ _load_into_memory()
403
+ with _STATE_LOCK:
404
+ h = _STATE.get(model) or ModelHealth(model=model)
405
+ h.last_success_at = _now_iso()
406
+ h.last_error = None
407
+ h.latency_ms = int(latency_ms)
408
+ h.consecutive_failures = 0
409
+ h.tested_at = _now_iso()
410
+ h.status = "healthy" if latency_ms < 5000 else "degraded"
411
+ h.probe_history.append({
412
+ "ok": True,
413
+ "latency_ms": int(latency_ms),
414
+ "ts": _now_iso(),
415
+ "src": "chat",
416
+ })
417
+ if len(h.probe_history) > PROBE_HISTORY_LEN:
418
+ h.probe_history = h.probe_history[-PROBE_HISTORY_LEN:]
419
+ _STATE[model] = h
420
+
421
+
422
+ async def _reprobe_one(model: str) -> None:
423
+ """Async re-probe of a single model after report_failure(). Updates
424
+ state in place; failures here are silently swallowed so the chat
425
+ hot path never raises through this back-channel."""
426
+ try:
427
+ async with httpx.AsyncClient() as client:
428
+ ok, err, latency = await probe_one(client, model, "")
429
+ _absorb_probe_result(model, ok, err, latency)
430
+ except Exception:
431
+ pass
432
+
433
+
434
+ # ---------------------------------------------------------------------------
435
+ # Probing (mostly unchanged from pre-KI-080 — extended to record
436
+ # probe_history + skip degraded-window models on the regular tick).
437
+ # ---------------------------------------------------------------------------
438
+
439
  async def probe_one(client: httpx.AsyncClient, model: str, api_key: str) -> tuple[bool, str, Optional[int]]:
440
  """Probe a single chain entry (NIM or cross-provider).
441
 
 
490
  return False, f"net: {type(e).__name__}: {str(e)[:60]}", int((time.time() - t0) * 1000)
491
 
492
 
493
+ def _absorb_probe_result(model: str, ok: bool, err: str, latency: Optional[int]) -> None:
494
+ """Update _STATE with one probe result. Used by both probe_all (every
495
+ tick) and _reprobe_one (after-failure reprobe)."""
496
+ with _STATE_LOCK:
497
+ h = _STATE.get(model) or ModelHealth(model=model)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
498
  h.tested_at = _now_iso()
499
  if ok:
500
  h.last_success_at = _now_iso()
 
510
  h.status = "down"
511
  else:
512
  h.status = "degraded"
513
+ h.probe_history.append({
514
+ "ok": ok,
515
+ "latency_ms": latency,
516
+ "ts": _now_iso(),
517
+ "src": "probe",
518
+ })
519
+ if len(h.probe_history) > PROBE_HISTORY_LEN:
520
+ h.probe_history = h.probe_history[-PROBE_HISTORY_LEN:]
521
+ _STATE[model] = h
522
 
523
+
524
+ async def probe_all() -> dict[str, ModelHealth]:
525
+ """One-shot probe of every known model. Updates persisted state + returns it."""
526
+ _load_into_memory()
527
+ models = _all_known_models()
528
+ # Per-provider keys are resolved inside probe_one(); we no longer
529
+ # gate the whole loop on the NIM key (KI-080) — cross-provider
530
+ # candidates must keep getting probed even when NVIDIA_NIM_API_KEY
531
+ # is missing.
532
+
533
+ async with httpx.AsyncClient() as client:
534
+ # Probe all in parallel — they hit different NIM pools so concurrency is fine
535
+ results = await asyncio.gather(
536
+ *[probe_one(client, m, "") for m in models],
537
+ return_exceptions=True,
538
+ )
539
+
540
+ for model, result in zip(models, results):
541
+ if isinstance(result, Exception):
542
+ ok, err, latency = False, f"exc: {type(result).__name__}", None
543
+ else:
544
+ ok, err, latency = result
545
+ _absorb_probe_result(model, ok, err, latency)
546
+
547
+ save() # persist in-memory state
548
+ return load()
549
 
550
 
551
  def filter_chain(chain: list[str]) -> list[str]:
552
  """Return the chain with 'down' models removed. Always preserves order +
553
+ keeps at least one model so callers never get an empty chain.
554
+
555
+ Kept for backward compatibility — admin endpoints + the final
556
+ probe-refresh fallback in NimChainLLM still call this. Primary
557
+ election (get_primary / get_backup) is the new hot-path entry."""
558
  state = load()
559
  keep = [m for m in chain if state.get(m, ModelHealth(model=m)).status != "down"]
560
  if not keep:
 
580
  summary["models"].sort(key=lambda x: (x["status"] != "healthy", x["model"]))
581
  if summary["models"]:
582
  summary["updated_at"] = max((m.get("tested_at") or "") for m in summary["models"])
583
+ # KI-080 — surface currently-elected primary/backup per chain for the
584
+ # admin UI. Cheap (a couple of dict lookups + sort over <=10 entries).
585
+ summary["elections"] = {
586
+ role: {"primary": get_primary(role), "backup": get_backup(role)}
587
+ for role in ("brain", "fast_brain", "judge")
588
+ }
589
  return summary
590
 
591
 
 
608
  for m, h in sorted(state.items(), key=lambda kv: (kv[1].status != "healthy", kv[0])):
609
  latency = f"{h.latency_ms}ms" if h.latency_ms else "—"
610
  print(f" {h.status:8s} {latency:>8s} {m:50s} {h.last_error or 'ok'}")
611
+ print()
612
+ print("Elections:")
613
+ for role in ("brain", "fast_brain", "judge"):
614
+ print(f" {role:10s} primary={get_primary(role)} backup={get_backup(role)}")
backend/providers/nvidia_nim_llm.py CHANGED
@@ -270,14 +270,32 @@ JUDGE_CHAIN = [
270
 
271
 
272
  class NimChainLLM(LLMProvider):
273
- """OpenRouter-style fallback router across multiple NIM models.
274
-
275
- Tries each model in `chain` in order; on TimeoutException / 5xx / network
276
- error, advances to the next. Surfaces the first success transparently.
277
-
278
- `name` after a successful call reflects which model actually answered, so
279
- downstream callers (orchestrator brain_used tag, eval logs) can audit
280
- which model in the chain produced the output.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
281
  """
282
  def __init__(self, chain: list[str], api_key: Optional[str] = None,
283
  timeout: float = 30.0, per_model_attempts: int = 1,
@@ -292,9 +310,14 @@ class NimChainLLM(LLMProvider):
292
  # produced the p99 58s+ tail in the 100-persona audit. Default to a
293
  # cumulative ceiling of ~2.5× the per-link timeout so a healthy primary
294
  # always completes, but a cascading-failure chain bails fast.
 
 
 
 
295
  self.total_budget_s = total_budget_s if total_budget_s is not None else max(timeout * 2.5, 30.0)
296
  self.per_model_attempts = per_model_attempts
297
  self.role = role # 'brain' | 'fast_brain' | 'judge' | 'unknown' — flows into usage log
 
298
  self.model = chain[0]
299
  self.name = f"nim-chain::{self._short_id(chain[0])}"
300
 
@@ -358,6 +381,38 @@ class NimChainLLM(LLMProvider):
358
  if "nemotron" in m or m.startswith("nvidia/"): return "nvidia"
359
  return "unknown"
360
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
361
  async def chat(
362
  self,
363
  messages: list[ChatMessage],
@@ -367,123 +422,217 @@ class NimChainLLM(LLMProvider):
367
  exclude_models: Optional[list[str]] = None,
368
  exclude_families: Optional[list[str]] = None,
369
  ) -> LLMResult:
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
370
  # Apply caller's exclusion list FIRST (brain doesn't grade own homework).
371
- # exclude_models: skip exact model IDs (handles same-model collision)
372
- # exclude_families: skip everything in those families (handles weight-sharing siblings)
373
- chain = self.chain
374
- if exclude_models or exclude_families:
375
- excl_m = set(exclude_models or [])
376
- excl_f = set(exclude_families or [])
377
- chain = [m for m in chain
378
- if m not in excl_m and self._family_of(m) not in excl_f]
379
- if not chain:
380
- # Every candidate excluded — relax family constraint, keep exact model
381
- # constraint (the strict one). Better to use a same-family model than
382
- # to fail the request entirely.
383
- chain = [m for m in self.chain if m not in excl_m]
384
-
385
- # Filter out models known-down (from background probe loop). If all
386
- # models are down, filter_chain returns the full chain unchanged so
387
- # we still try the infrastructure may have recovered between probes.
388
- # Use the post-exclusion `chain` (NOT self.chain) so brain↔judge
389
- # independence is preserved even after health filtering.
390
- try:
391
- from backend import llm_health
392
- chain_to_try = llm_health.filter_chain(chain)
393
- except Exception:
394
- chain_to_try = chain # health monitor failure must never block calls
395
 
396
  chain_primary = self.chain[0] if self.chain else None
397
  call_t0 = time.time()
398
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
399
  last_err: Optional[Exception] = None
400
- for model in chain_to_try:
401
- # KI-021 bail out if the cumulative chain budget is gone so a single
402
- # turn can never wedge for minutes through a long fallback cascade.
403
- elapsed = time.time() - call_t0
404
- if elapsed >= self.total_budget_s:
405
- last_err = TimeoutError(
406
- f"chain budget exhausted ({elapsed:.1f}s {self.total_budget_s:.1f}s) "
407
- f"after trying {[m for m in chain_to_try if chain_to_try.index(m) < chain_to_try.index(model)]}"
408
- )
409
- break
 
 
 
410
  try:
411
- # Cap per-link timeout to remaining chain budget so the final
412
- # link can't single-handedly blow past the ceiling.
413
- per_link_timeout = min(self.timeout, max(2.0, self.total_budget_s - elapsed))
414
- worker = self._get_worker_for(model, per_link_timeout)
415
- result = await worker.chat(messages=messages, temperature=temperature,
416
- max_tokens=max_tokens, response_format=response_format)
417
- # Successful response — record which model answered + return.
418
- # `self.name` preserves any provider prefix so audit logs make
419
- # cross-provider fall-throughs (groq:/openrouter:) obvious.
420
- self.model = model
421
- self.name = f"nim-chain::{self._short_id(model)}"
422
- latency_ms = int((time.time() - call_t0) * 1000)
423
- await _append_usage({
424
- "ts": _now_iso_z(),
425
- "role": self.role,
426
- "chain_primary": chain_primary,
427
- "served_model": model,
428
- "latency_ms": latency_ms,
429
- "success": True,
430
- })
431
- return result
432
- except (httpx.TimeoutException, httpx.HTTPStatusError,
433
- httpx.ConnectError, httpx.NetworkError, asyncio.TimeoutError) as e:
434
- last_err = e
435
- continue # try next model in chain
436
  except (asyncio.CancelledError, KeyboardInterrupt, SystemExit):
437
- # KI-078 (2026-05-15) must re-raise these. The previous
438
- # broad `except Exception` swallowed CancelledError, so when
439
- # fact_find_brain's outer `asyncio.wait_for(_TIMEOUT_S)`
440
- # fired, this loop kept consuming budget instead of bubbling
441
- # the cancellation up. That cost the entire fact-find turn
442
- # its fallback window.
443
  raise
444
  except Exception as e:
445
- # Unexpected error — record + try next, but surface eventually if all fail
446
  last_err = e
447
- continue
448
- # All models in (filtered) chain failed. Trigger one synchronous
449
- # probe refresh — maybe a transient outage just recovered. If the
450
- # refreshed state opens up models we previously skipped, try them.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
451
  try:
452
- from backend import llm_health
453
  await llm_health.probe_all()
454
- refreshed = llm_health.filter_chain(self.chain)
455
  for model in refreshed:
456
- if model in chain_to_try:
457
- continue # already tried this turn
458
- # KI-021 — respect chain budget here too
459
  elapsed = time.time() - call_t0
460
  if elapsed >= self.total_budget_s:
461
  break
 
462
  try:
463
- per_link_timeout = min(self.timeout, max(2.0, self.total_budget_s - elapsed))
464
- worker = self._get_worker_for(model, per_link_timeout)
465
- result = await worker.chat(messages=messages, temperature=temperature,
466
- max_tokens=max_tokens, response_format=response_format)
467
- self.model = model
468
- self.name = f"nim-chain::{self._short_id(model)}"
469
- latency_ms = int((time.time() - call_t0) * 1000)
470
- await _append_usage({
471
- "ts": _now_iso_z(),
472
- "role": self.role,
473
- "chain_primary": chain_primary,
474
- "served_model": model,
475
- "latency_ms": latency_ms,
476
- "success": True,
477
- })
478
- return result
479
  except Exception as e:
480
  last_err = e
 
 
 
 
 
 
481
  continue
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
482
  except Exception:
483
- pass
484
 
485
- # Total exhaustion — log a single failure record so the admin panel
486
- # surfaces "judge has 12% failure rate over last 24h" type signal.
 
487
  latency_ms = int((time.time() - call_t0) * 1000)
488
  await _append_usage({
489
  "ts": _now_iso_z(),
@@ -492,30 +641,45 @@ class NimChainLLM(LLMProvider):
492
  "served_model": None,
493
  "latency_ms": latency_ms,
494
  "success": False,
 
 
 
495
  })
496
  raise RuntimeError(
497
- f"NimChainLLM exhausted all {len(self.chain)} candidates. Last error: "
498
- f"{type(last_err).__name__}: {str(last_err)[:120]}"
 
 
499
  ) from last_err
500
 
501
 
502
  # KI-025 (2026-05-14) — provider load-balancing.
503
- # The NIM rate cap (40 req/min) is the single biggest throughput bottleneck.
504
- # Groq has independent rate quota and a Llama-3.3-70B LPU primary that's
505
- # already in BRAIN_CHAIN as a cross-provider fallback. By rotating which
506
- # provider serves as the *primary* for each call (50/50 between NIM Qwen and
507
- # Groq Llama), we spread load across two independent quotas — effectively
508
- # doubling sustained brain throughput. On rotation-to-Groq, the chain still
509
- # falls back to NIM if Groq fails, so reliability is unchanged.
510
  import random as _random
511
 
512
 
513
  def _balanced_brain_chain(base: list[str], *, groq_first_probability: float = 0.5) -> list[str]:
514
- """KI-025 — provider load-balancing. With `groq_first_probability` (default
515
- 50%), hoist the Groq Llama entry to the head of the chain so it serves
516
- as the primary instead of the NIM Qwen entry. The remaining candidates
517
- stay in their existing fallback order Groq calls that fail (rare; LPU
518
- is very reliable) still get the full NIM fallback chain.
 
 
 
 
 
 
 
 
 
 
519
 
520
  Uses per-call `random.random()` so concurrent async workers (the
521
  100-persona audit's 4 workers, the parallel 96-Q eval's 6 workers) each
@@ -533,33 +697,48 @@ def _balanced_brain_chain(base: list[str], *, groq_first_probability: float = 0.
533
 
534
 
535
  def get_brain_llm() -> NimChainLLM:
536
- """Heavy brain — multi-model NIM chain with automatic fallback.
537
- Primary: Qwen 3-Next 80B (50% of calls) or Groq Llama-3.3-70B (50% of calls).
538
- See BRAIN_CHAIN for fallback order. KI-021 per-link 20s, total chain
539
- budget 35s. KI-025 load-balanced across NIM/Groq for brain throughput."""
540
- return NimChainLLM(chain=_balanced_brain_chain(BRAIN_CHAIN), timeout=20.0,
 
 
 
 
 
 
 
 
 
 
 
541
  role="brain", total_budget_s=35.0)
542
 
543
 
544
  def get_fast_brain_llm() -> NimChainLLM:
545
- """Fast brain — multi-model NIM chain optimized for low TTFT.
546
- Primary: Qwen 3-Next 80B (50%) or Groq Llama-3.3-70B (50%). See
547
- FAST_BRAIN_CHAIN for fallback order.
548
-
549
- KI-078 (2026-05-15) per-link timeout tightened 12s 6s. With a 22s
550
- total chain budget and a 12s per-link, only 1 candidate could complete
551
- before the budget expired i.e. the fallback chain was effectively
552
- dead. Groq LPU + NIM Nemotron Nano + NIM Qwen all hit TTFT in <2s on
553
- healthy paths, so 6s catches a degraded link fast and lets the chain
554
- try 3-4 candidates inside the 22s budget. The cold-start case (NIM
555
- pool warm-up taking 10-15s) is still handled by the outer
556
- fact_find_brain `_TIMEOUT_S=25s` wait_for + Groq's cross-provider link
557
- which has no NIM cold-start exposure.
558
-
559
- KI-025 provider-balanced for fast-brain throughput; Groq LPU's
560
- sub-1s TTFT is actually faster than NIM Qwen on average, so this
561
- rotation is a strict win for fact-find / QA latency."""
562
- return NimChainLLM(chain=_balanced_brain_chain(FAST_BRAIN_CHAIN), timeout=6.0,
 
 
 
 
563
  role="fast_brain", total_budget_s=22.0)
564
 
565
 
 
270
 
271
 
272
  class NimChainLLM(LLMProvider):
273
+ """KI-080 sticky-primary router across multiple candidate models.
274
+
275
+ Architectural shift (KI-080, 2026-05-15):
276
+ PRE-KI-080: chat() iterated the full chain sequentially every turn.
277
+ Under NIM per-key concurrency throttling, the first 5 NIM-hosted
278
+ candidates queued together inside ONE turn and burned the 22s
279
+ budget before any cross-provider fallback was ever reached. The
280
+ 10-turn live probe at commit 078ff45 showed 7/10 fact-find turns
281
+ timing out at exactly 26.6s.
282
+
283
+ POST-KI-080: a background probe loop ELECTS a primary + cross-
284
+ provider backup per chain based on real probe latencies. chat()
285
+ calls the elected primary ONCE per turn (1 LLM call). On failure,
286
+ we demote the primary for ~30s and fall through to the elected
287
+ backup (still 2 LLM calls max). The chain list is now the
288
+ CANDIDATE POOL for election, not a per-call sequence.
289
+
290
+ Worst case per turn: 2 LLM calls (was 5-6). Plus a final filter_chain
291
+ refresh path is kept for the rare double-failure edge case so the
292
+ pre-KI-080 graceful-degradation behaviour is preserved.
293
+
294
+ `name` after a successful call reflects which model actually answered
295
+ so downstream callers (orchestrator brain_used tag, eval logs) can
296
+ audit which candidate produced the output. The KI-079 escalation in
297
+ fact_find_brain still applies — if primary+backup both fail inside
298
+ one turn, fact_find_brain gets one more bite via BRAIN_CHAIN.
299
  """
300
  def __init__(self, chain: list[str], api_key: Optional[str] = None,
301
  timeout: float = 30.0, per_model_attempts: int = 1,
 
310
  # produced the p99 58s+ tail in the 100-persona audit. Default to a
311
  # cumulative ceiling of ~2.5× the per-link timeout so a healthy primary
312
  # always completes, but a cascading-failure chain bails fast.
313
+ #
314
+ # KI-080 — with election we only do 1-2 LLM calls per turn, so the
315
+ # budget is rarely the binding constraint anymore. Kept for the
316
+ # final filter_chain-refresh fallback path + cold-start edge cases.
317
  self.total_budget_s = total_budget_s if total_budget_s is not None else max(timeout * 2.5, 30.0)
318
  self.per_model_attempts = per_model_attempts
319
  self.role = role # 'brain' | 'fast_brain' | 'judge' | 'unknown' — flows into usage log
320
+ self._chain_name = role if role in ("brain", "fast_brain", "judge") else "unknown"
321
  self.model = chain[0]
322
  self.name = f"nim-chain::{self._short_id(chain[0])}"
323
 
 
381
  if "nemotron" in m or m.startswith("nvidia/"): return "nvidia"
382
  return "unknown"
383
 
384
+ # KI-080 — per-call timeout used in the sticky-primary path. Each elected
385
+ # candidate gets at most this long; with at most 2 calls per turn this
386
+ # cleanly fits inside any caller's outer wait_for cap (25s for fact-find,
387
+ # higher for brain/judge).
388
+ _ELECTED_CALL_TIMEOUT_S = 12.0
389
+
390
+ async def _call_one(
391
+ self,
392
+ model: str,
393
+ *,
394
+ messages: list[ChatMessage],
395
+ temperature: float,
396
+ max_tokens: int,
397
+ response_format: Optional[dict],
398
+ timeout: float,
399
+ ) -> LLMResult:
400
+ """Single-model HTTP call. Extracted from the old chain-iteration
401
+ loop (KI-080) so chat() can call a SPECIFIC candidate ONCE without
402
+ the iterate-the-chain envelope.
403
+
404
+ Raises whatever the underlying provider raises (TimeoutException /
405
+ HTTPStatusError / network error). Caller is responsible for failure
406
+ handling (report_failure + fall through to backup).
407
+ """
408
+ worker = self._get_worker_for(model, timeout)
409
+ return await worker.chat(
410
+ messages=messages,
411
+ temperature=temperature,
412
+ max_tokens=max_tokens,
413
+ response_format=response_format,
414
+ )
415
+
416
  async def chat(
417
  self,
418
  messages: list[ChatMessage],
 
422
  exclude_models: Optional[list[str]] = None,
423
  exclude_families: Optional[list[str]] = None,
424
  ) -> LLMResult:
425
+ """KI-080 — sticky primary election. Call ONE elected candidate per
426
+ turn, with at most ONE real-time fallback to the elected backup if
427
+ primary fails. The chain list is the candidate POOL for election —
428
+ NOT a per-call sequence.
429
+
430
+ Cold-start path (no probe data yet): use self.chain[0] / [1].
431
+ Brain/judge family-exclusion (e.g. brain doesn't grade own
432
+ homework): apply exclusions to both elected primary + backup; if
433
+ either is excluded, re-elect from the filtered pool by scanning
434
+ the chain in order.
435
+ Final fallback (both elected models fail): trigger a synchronous
436
+ probe refresh + try whatever filter_chain now offers, walking the
437
+ chain in order. This path is the pre-KI-080 graceful-degradation
438
+ safety net for the double-failure edge case.
439
+ """
440
+ from backend import llm_health
441
+
442
  # Apply caller's exclusion list FIRST (brain doesn't grade own homework).
443
+ excl_m = set(exclude_models or [])
444
+ excl_f = set(exclude_families or [])
445
+ def _allowed(m: Optional[str]) -> bool:
446
+ if not m:
447
+ return False
448
+ if m in excl_m:
449
+ return False
450
+ if self._family_of(m) in excl_f:
451
+ return False
452
+ return True
453
+
454
+ # Filter the chain by exclusions (kept for the final-fallback path
455
+ # + cold-start primary/backup election).
456
+ allowed_chain = [m for m in self.chain if _allowed(m)]
457
+ if not allowed_chain:
458
+ # Every candidate excluded relax family constraint, keep exact
459
+ # model constraint. Better to use a same-family model than to
460
+ # fail the request entirely.
461
+ allowed_chain = [m for m in self.chain if m not in excl_m]
462
+ if not allowed_chain:
463
+ raise RuntimeError("NimChainLLM: every chain candidate is excluded.")
 
 
 
464
 
465
  chain_primary = self.chain[0] if self.chain else None
466
  call_t0 = time.time()
467
 
468
+ # --- Election (KI-080) ----------------------------------------------
469
+ elected_primary: Optional[str] = None
470
+ elected_backup: Optional[str] = None
471
+ try:
472
+ primary_candidate = llm_health.get_primary(self._chain_name)
473
+ backup_candidate = llm_health.get_backup(self._chain_name)
474
+ except Exception:
475
+ primary_candidate, backup_candidate = None, None
476
+
477
+ if primary_candidate and _allowed(primary_candidate):
478
+ elected_primary = primary_candidate
479
+ else:
480
+ # Cold-start / excluded election → first allowed chain entry.
481
+ elected_primary = allowed_chain[0]
482
+
483
+ if backup_candidate and _allowed(backup_candidate) and backup_candidate != elected_primary:
484
+ elected_backup = backup_candidate
485
+ else:
486
+ # Cold-start / excluded election → first allowed chain entry
487
+ # that isn't the elected primary. Prefer a different provider.
488
+ for m in allowed_chain:
489
+ if m != elected_primary and llm_health.provider_of(m) != llm_health.provider_of(elected_primary):
490
+ elected_backup = m
491
+ break
492
+ if elected_backup is None:
493
+ for m in allowed_chain:
494
+ if m != elected_primary:
495
+ elected_backup = m
496
+ break
497
+
498
+ tried: list[str] = []
499
  last_err: Optional[Exception] = None
500
+
501
+ async def _try(model: str) -> Optional[LLMResult]:
502
+ """Attempt one elected candidate. On success: stamp self.model
503
+ + self.name, report_success, log usage, return result. On
504
+ failure: report_failure, mutate last_err, return None.
505
+
506
+ CancelledError / KeyboardInterrupt / SystemExit are re-raised
507
+ (KI-078) so an outer wait_for cancellation bubbles up
508
+ instead of getting swallowed by the fallback path.
509
+ """
510
+ nonlocal last_err
511
+ tried.append(model)
512
+ attempt_t0 = time.time()
513
  try:
514
+ result = await self._call_one(
515
+ model,
516
+ messages=messages,
517
+ temperature=temperature,
518
+ max_tokens=max_tokens,
519
+ response_format=response_format,
520
+ timeout=self._ELECTED_CALL_TIMEOUT_S,
521
+ )
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
522
  except (asyncio.CancelledError, KeyboardInterrupt, SystemExit):
523
+ # KI-078 — propagate cancellation immediately.
 
 
 
 
 
524
  raise
525
  except Exception as e:
 
526
  last_err = e
527
+ try:
528
+ llm_health.report_failure(
529
+ self._chain_name, model, type(e).__name__
530
+ )
531
+ except Exception:
532
+ pass
533
+ return None
534
+
535
+ # Success — stamp + log.
536
+ latency_ms = int((time.time() - attempt_t0) * 1000)
537
+ try:
538
+ llm_health.report_success(self._chain_name, model, latency_ms)
539
+ except Exception:
540
+ pass
541
+ self.model = model
542
+ self.name = f"nim-chain::{self._short_id(model)}"
543
+ total_ms = int((time.time() - call_t0) * 1000)
544
+ await _append_usage({
545
+ "ts": _now_iso_z(),
546
+ "role": self.role,
547
+ "chain_primary": chain_primary,
548
+ "served_model": model,
549
+ "latency_ms": total_ms,
550
+ "success": True,
551
+ "elected_primary": elected_primary,
552
+ "elected_backup": elected_backup,
553
+ })
554
+ return result
555
+
556
+ # --- Primary attempt -------------------------------------------------
557
+ res = await _try(elected_primary)
558
+ if res is not None:
559
+ return res
560
+
561
+ # --- Backup attempt (KI-080 single real-time fallback) ---------------
562
+ if elected_backup and elected_backup != elected_primary:
563
+ res = await _try(elected_backup)
564
+ if res is not None:
565
+ return res
566
+
567
+ # --- Both elected failed — final safety net --------------------------
568
+ # Trigger ONE synchronous probe refresh; whatever the refreshed
569
+ # filter_chain offers, walk it in order and try anything we haven't
570
+ # touched this turn. This is the pre-KI-080 graceful-degradation
571
+ # behaviour preserved for the (rare) double-failure case.
572
  try:
 
573
  await llm_health.probe_all()
574
+ refreshed = llm_health.filter_chain(allowed_chain)
575
  for model in refreshed:
576
+ if model in tried:
577
+ continue
 
578
  elapsed = time.time() - call_t0
579
  if elapsed >= self.total_budget_s:
580
  break
581
+ attempt_t0 = time.time()
582
  try:
583
+ per_link_timeout = min(
584
+ self._ELECTED_CALL_TIMEOUT_S,
585
+ max(2.0, self.total_budget_s - elapsed),
586
+ )
587
+ result = await self._call_one(
588
+ model,
589
+ messages=messages,
590
+ temperature=temperature,
591
+ max_tokens=max_tokens,
592
+ response_format=response_format,
593
+ timeout=per_link_timeout,
594
+ )
595
+ except (asyncio.CancelledError, KeyboardInterrupt, SystemExit):
596
+ raise
 
 
597
  except Exception as e:
598
  last_err = e
599
+ try:
600
+ llm_health.report_failure(
601
+ self._chain_name, model, type(e).__name__
602
+ )
603
+ except Exception:
604
+ pass
605
  continue
606
+
607
+ try:
608
+ llm_health.report_success(
609
+ self._chain_name, model, int((time.time() - attempt_t0) * 1000)
610
+ )
611
+ except Exception:
612
+ pass
613
+ self.model = model
614
+ self.name = f"nim-chain::{self._short_id(model)}"
615
+ total_ms = int((time.time() - call_t0) * 1000)
616
+ await _append_usage({
617
+ "ts": _now_iso_z(),
618
+ "role": self.role,
619
+ "chain_primary": chain_primary,
620
+ "served_model": model,
621
+ "latency_ms": total_ms,
622
+ "success": True,
623
+ "elected_primary": elected_primary,
624
+ "elected_backup": elected_backup,
625
+ "fallback_phase": "post_reprobe",
626
+ })
627
+ return result
628
+ except (asyncio.CancelledError, KeyboardInterrupt, SystemExit):
629
+ raise
630
  except Exception:
631
+ pass # probe refresh failure must never block the outer raise
632
 
633
+ # Total exhaustion — log + raise. fact_find_brain.drive_fact_find
634
+ # (KI-079) catches the resulting RuntimeError + escalates one more
635
+ # time to BRAIN_CHAIN before falling to the canonical reply.
636
  latency_ms = int((time.time() - call_t0) * 1000)
637
  await _append_usage({
638
  "ts": _now_iso_z(),
 
641
  "served_model": None,
642
  "latency_ms": latency_ms,
643
  "success": False,
644
+ "elected_primary": elected_primary,
645
+ "elected_backup": elected_backup,
646
+ "tried": tried,
647
  })
648
  raise RuntimeError(
649
+ f"NimChainLLM ({self._chain_name}) elected primary={elected_primary} "
650
+ f"and backup={elected_backup} both failed; tried={tried}. "
651
+ f"Last error: {type(last_err).__name__ if last_err else 'None'}: "
652
+ f"{str(last_err)[:120] if last_err else ''}"
653
  ) from last_err
654
 
655
 
656
  # KI-025 (2026-05-14) — provider load-balancing.
657
+ # DEPRECATED 2026-05-15 by KI-080: primary election supersedes the 50/50
658
+ # rotation. The probe loop now picks the actually-faster candidate
659
+ # DYNAMICALLY (real probe latencies, not a coin flip), and the elected
660
+ # backup is chosen with explicit cross-provider preference both signals
661
+ # the rotation was approximating heuristically. Kept around (not deleted)
662
+ # because the regression suite still pins its statistical behaviour. The
663
+ # get_*_llm factories no longer call it.
664
  import random as _random
665
 
666
 
667
  def _balanced_brain_chain(base: list[str], *, groq_first_probability: float = 0.5) -> list[str]:
668
+ """KI-025 — provider load-balancing (DEPRECATED 2026-05-15 by KI-080).
669
+
670
+ With `groq_first_probability` (default 50%), hoist the Groq Llama entry
671
+ to the head of the chain so it serves as the primary instead of the
672
+ NIM Qwen entry. The remaining candidates stay in their existing
673
+ fallback order — Groq calls that fail (rare; LPU is very reliable)
674
+ still get the full NIM fallback chain.
675
+
676
+ SUPERSESSION NOTE (KI-080): the elector in `backend.llm_health` now
677
+ picks the actually-faster candidate dynamically from background probe
678
+ data, so this static-coin-flip rotation is no longer wired into the
679
+ chat hot path. Kept exported because:
680
+ (a) regression tests still pin its statistical behaviour, and
681
+ (b) it remains a useful pure-function for ops / sim / overrides
682
+ (e.g. wanting to force a non-elected order in a debug script).
683
 
684
  Uses per-call `random.random()` so concurrent async workers (the
685
  100-persona audit's 4 workers, the parallel 96-Q eval's 6 workers) each
 
697
 
698
 
699
  def get_brain_llm() -> NimChainLLM:
700
+ """Heavy brain — KI-080 sticky-primary election over BRAIN_CHAIN.
701
+
702
+ Pre-KI-080: per-call _balanced_brain_chain rotation between NIM Qwen
703
+ and Groq Llama (KI-025 50/50 heuristic) 5-6 LLM calls per turn
704
+ under degraded conditions.
705
+
706
+ Post-KI-080: the background probe loop in backend.llm_health elects
707
+ the actually-fastest candidate dynamically (probe-driven, not coin-
708
+ flipped) and a cross-provider backup. chat() calls 1-2 candidates max
709
+ per turn. The rotation is preserved as a deprecated pure-function in
710
+ case overrides need it, but the factory no longer wires it in.
711
+
712
+ KI-021 — per-link 12s (KI-080 ELECTED_CALL_TIMEOUT_S), total chain
713
+ budget 35s (only binding for the final filter_chain-refresh safety net).
714
+ """
715
+ return NimChainLLM(chain=BRAIN_CHAIN, timeout=20.0,
716
  role="brain", total_budget_s=35.0)
717
 
718
 
719
  def get_fast_brain_llm() -> NimChainLLM:
720
+ """Fast brain — KI-080 sticky-primary election over FAST_BRAIN_CHAIN.
721
+
722
+ Pre-KI-080 (KI-025 + KI-078 + KI-079): per-call rotation between NIM
723
+ Qwen and Groq Llama, 6s per-link timeout, 22s total chain budget,
724
+ Groq promoted to chain position #2 so a single NIM degradation could
725
+ fall through fast enough. Even so, the 10-turn live probe at commit
726
+ 078ff45 showed 7/10 fact-find turns hitting the wait_for cap at 26.6s
727
+ because every chain-iteration burned the full budget exploring 5+
728
+ queued NIM candidates.
729
+
730
+ Post-KI-080: probe-driven election picks ONE primary per chain (the
731
+ fastest-responding healthy candidate by rolling probe data) and ONE
732
+ cross-provider backup. chat() invokes the primary ONCE with a 12s
733
+ timeout, falls to the backup ONCE on failure, and only walks the
734
+ filter_chain fallback as a final safety net. The KI-079 escalation in
735
+ fact_find_brain still runs if primary+backup both fail, providing one
736
+ more bite via BRAIN_CHAIN.
737
+
738
+ KI-025 supersession: the 50/50 rotation is no longer wired in — see
739
+ the docstring on `_balanced_brain_chain` for the reasoning.
740
+ """
741
+ return NimChainLLM(chain=FAST_BRAIN_CHAIN, timeout=6.0,
742
  role="fast_brain", total_budget_s=22.0)
743
 
744