feat(llm-chain): KI-088 — cap NIM outbound concurrency at 2 + serialise probe + drop inner retry

Live probe at 6a47549 measured only 2/10 brain turns succeeding despite
KI-080 election + KI-085 credit gating + KI-087 NIM-first + KI-079
escalation all live. The chain architecture is correct; the binding
constraint is NIM's free-tier per-key concurrency (~3-5 slots).

Root cause: 5 sources of NIM traffic stack on the same key with no
global throttle.

1. Probe loop (every 300s) fired ~6 candidates in parallel via
asyncio.gather → 6-slot burst.
2. Admin tab polls /api/admin/llm-health every 30s.
3. Per-user chat turn: 1-2 NIM calls.
4. N concurrent users → N simultaneous calls.
5. Inner 4-attempt 429/5xx retry held one slot for up to ~15s on
exponential backoff sleeps.

6+ in-flight → queue → 15-25s response times → bot's 12s outer cap
fires → user sees fallback. 20% brain success rate measured.

Three coordinated fixes:

FIX 1 — Module-level asyncio.Semaphore(2) at backend/providers/
nvidia_nim_llm.py shared across every NvidiaNimLLM instance. Wraps
ONLY the httpx.post round-trip, not election or response parsing,
so we cap network concurrency without serialising the reasoning
pipeline. Verified inline: 5 concurrent chat() tasks → peak
in-flight = 2.

FIX 2 — Serial probe in backend/llm_health.py: replaced
asyncio.gather(*probe_one(m) for m in models) with an explicit
for-loop. With KI-088's semaphore in place, parallel gather would
still cap at 2 in-flight but add no benefit over serial; serial is
simpler and naturally yields control to user traffic between
candidates. 6 NIM probes × ~2s = ~12s, well under the 300s cadence.

FIX 3 — Dropped the inner 4-attempt exponential backoff retry in
NvidiaNimLLM.chat(). Single httpx.post per call. KI-080's elector
already demotes a 429/5xx primary + falls through to the elected
cross-provider backup in the same turn, then KI-079 escalates via
BRAIN_CHAIN. Per-call retry was a pre-KI-080 vestige that only
re-queued against the same slot-starved pool.

FIX 4 — Comment marker on /api/admin/llm-health confirming it
reads cached llm_health.load() snapshot only; future contributors
must not add live probing to the admin path.

Expected impact: brain success rate climbs from ~20% → 80%+ even under
heavy traffic because user calls never wait past NIM's idle response
time (~4s). Worst-case per-turn NIM concurrency = 2 (ours) + whatever
NIM accepts from the rest of the world; we no longer self-saturate.

Testing:
- python -m py_compile on all three modified files: OK.
- Inline semaphore test (5 parallel chat() tasks, fake httpx.post
instrumented to track concurrent entrants): peak in-flight = 2.
- tests/test_routing_regression.py: 15/15 pass, 13 subtests pass.
(test_credits_election failure on main is pre-existing, unrelated.)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

backend/admin.py

@@ -787,11 +787,18 @@ async def admin_llm_health(
     """
     _check_admin(request, x_admin_password)
 
+    # KI-088: admin endpoint must never trigger probes — read cached state only.
+    # Live probing from the admin tab (polled every 30s by the frontend) would
+    # stack 6+ NIM candidates onto the same per-key concurrency budget and
+    # starve user chat traffic. All data below comes from llm_health.load()
+    # (in-memory snapshot persisted by the background_probe_loop) and the
+    # llm_usage.jsonl append-only log — both are read-only and trigger zero
+    # outbound LLM calls.
     import time as _time
     now_mono = _time.monotonic()
 
     chains = _chain_names_map()
-    state = llm_health.load()  # {model -> ModelHealth}
+    state = llm_health.load()  # {model -> ModelHealth}  (cached snapshot only)
 
     # Section A: per-chain election + credit banner.
     chains_block = [

backend/llm_health.py

@@ -984,12 +984,32 @@ async def probe_all() -> dict[str, ModelHealth]:
     # candidates must keep getting probed even when NVIDIA_NIM_API_KEY
     # is missing.
 
+    # KI-088 (2026-05-15) — Probe candidates serially, not in parallel.
+    #
+    # Pre-KI-088: `asyncio.gather(*probe_one(m) for m in models)` fired
+    # all candidates simultaneously. Even though they hit different NIM
+    # *model pools*, they all share the same per-API-key concurrency
+    # quota (~3-5 slots free-tier). A 6-candidate parallel burst every
+    # 300s would queue inside NIM and steal slots from in-flight user
+    # turns — exactly the saturation pattern the global outbound
+    # semaphore in nvidia_nim_llm.py is sized to prevent.
+    #
+    # With KI-088's semaphore in place, parallel gather would still
+    # be hard-capped at 2 in-flight but introduce no benefit over a
+    # serial loop. Serial is simpler, easier to reason about, and
+    # naturally yields control to user-traffic between candidates.
+    #
+    # Cost: 6 NIM candidates × ~2s healthy probe = ~12s, well under
+    # the 300s probe cadence. No sleep between candidates — the
+    # outbound semaphore handles pacing.
     async with httpx.AsyncClient() as client:
-        # Probe all in parallel — they hit different NIM pools so concurrency is fine
-        results = await asyncio.gather(
-            *[probe_one(client, m, "") for m in models],
-            return_exceptions=True,
-        )
+        results = []
+        for m in models:
+            try:
+                result = await probe_one(client, m, "")
+            except Exception as exc:
+                result = exc
+            results.append(result)
 
     for model, result in zip(models, results):
         if isinstance(result, Exception):

backend/providers/nvidia_nim_llm.py

@@ -79,6 +79,31 @@ async def _append_usage(record: dict) -> None:
         pass
 
 
+# KI-088 (2026-05-15) — Global outbound NIM concurrency cap.
+#
+# Live probe at commit 6a47549 measured 20% brain-turn success despite
+# KI-080 election + KI-085 credit gating + KI-087 NIM-first + KI-079
+# escalation all live. The architecture is correct; the binding constraint
+# is NIM's free-tier per-key concurrency (~3-5 slots).
+#
+# We have 5 sources of NIM traffic that all stack on the same key with no
+# global throttle: probe loop (6-slot parallel burst every 300s), admin
+# tab polling (every 30s), per-user chat turn (1-2 calls), concurrent
+# users (N × 1-2), and the inner 4-attempt retry loop (holds a slot up
+# to 15s on 429/5xx backoff).
+#
+# 6+ in-flight → queue → 15-25s response times → bot's 12s outer cap
+# fires → user sees fallback.
+#
+# This module-level semaphore is shared across ALL NvidiaNimLLM instances
+# in the process. It wraps ONLY the actual httpx.post — not election,
+# response parsing, or usage logging — so it serialises the NIM round-
+# trip without serialising the whole reasoning pipeline. With cap=2,
+# our own in-flight count never exceeds NIM's idle concurrency budget,
+# so probes + admin + users + retries cannot starve each other.
+_NIM_OUTBOUND_SEMAPHORE = asyncio.Semaphore(2)
+
+
 NVIDIA_NIM_BASE_URL = "https://integrate.api.nvidia.com/v1"
 # 2026-05-14 brain swap (D-022): NIM's DeepSeek-V4 + Meta Llama inference pools
 # are repeatedly timing out (15-120s on chat completions, no response). Qwen
@@ -155,21 +180,32 @@ class NvidiaNimLLM(LLMProvider):
             write=2.0,
             pool=2.0,
         )
+        # KI-088 (2026-05-15) — drop the inner 4-attempt exponential
+        # backoff retry. The previous loop held a NIM concurrency slot
+        # for up to ~15s on 429/5xx while sleeping between attempts,
+        # which directly contributed to the queueing that the outer
+        # semaphore is now sized to prevent.
+        #
+        # Retry was a pre-KI-080 vestige that predated the chain
+        # election architecture. With KI-080 in place, NimChainLLM
+        # already handles 429/5xx failover by:
+        #   (1) catching the exception in _try(),
+        #   (2) calling report_failure() so the elector demotes the
+        #       slot-blocked candidate for ~30s, and
+        #   (3) falling through to the elected backup (cross-provider
+        #       by preference) within the same turn.
+        # KI-079 then provides one more bite via BRAIN_CHAIN if both
+        # elected models fail. Per-call retries inside this method
+        # would only re-queue against the same slot-starved pool,
+        # multiplying the queueing problem the semaphore fixes.
+        #
+        # The semaphore wraps ONLY the HTTP round-trip — not response
+        # parsing or usage logging — so we cap concurrent network
+        # traffic without serialising the rest of the pipeline.
         async with httpx.AsyncClient(timeout=client_timeout) as client:
-            attempts = 4
-            delay = 1.0
-            for attempt in range(attempts):
+            async with _NIM_OUTBOUND_SEMAPHORE:
                 resp = await client.post(url, headers=headers, json=body)
-                if resp.status_code == 429 or (500 <= resp.status_code < 600):
-                    if attempt == attempts - 1:
-                        resp.raise_for_status()
-                    ra = resp.headers.get("Retry-After")
-                    wait = float(ra) if ra and ra.replace(".", "").isdigit() else delay
-                    await asyncio.sleep(wait)
-                    delay *= 2
-                    continue
-                resp.raise_for_status()
-                break
+            resp.raise_for_status()
             payload = resp.json()
 
         choice = payload["choices"][0]