fix(admin): KI-122 — clear LLM availability + in-use indicators

The admin LLM Chain panel showed a contradictory state — a chain's
elected primary marked HEALTHY 100% while a red banner declared "all
candidates credit-exhausted". The KI-116 reset-window fix mirrored
`is_credit_eligible` but still left a hole: when one chain member had
credits_reset_at=None (e.g. OpenRouter's prepaid wallet always stamps
None) AND credits_remaining<=low_water, while peer NIM candidates had
no credit signal yet (cold start), the banner loop registered
any_signal=True on the gated member and never broke out via the
permissive NIM peer. Banner fired despite the elected NIM primary
being perfectly callable.

Backend (`backend/admin.py::_chain_summary`):
- New helpers `_candidate_available_for_calls` + `_candidate_credit_
exhausted_strict` produce per-member booleans the wire payload
surfaces directly.
- New strict banner rule: `chain_credit_exhausted` is true ONLY when
ZERO members are available_for_calls AND at least one is strictly
credit-exhausted (credits<=low_water AND a future reset_at). Members
with reset_at=None are treated as permissive — wallet=$0 with healthy
probes is the free-tier OpenRouter signal, not a real outage.
- Wire payload adds `current_primary`, `current_primary_available`,
`chain_members[]` (one row per chain entry with availability +
credit_exhausted + is_current_primary), and `last_probe_at` +
`last_probe_age_seconds` for the operator's staleness signal.

Frontend (`frontend/public/admin/llm-control.html`):
- Top of each chain card now reads "Currently in use: <model> ● IN USE"
in a green callout so the operator doesn't have to infer the live
model from the PRIMARY detail row.
- Full chain member list rendered with per-row pills (● IN USE / ●
AVAILABLE / ● GATED / ● UNAVAILABLE) so every wired-up model is
visible at a glance with its current callability.
- Chain head shows "probed Ns ago" so probe-loop staleness is obvious.
- Banner remains backend-gated by the strict rule — no more false
positives when an exhausted Groq backup co-exists with a healthy
NIM primary.

Tests:
- Adds TestStrictChainCreditExhausted covering: healthy NIM + exhausted
Groq (banner stays down), all reset_at=None members (banner stays
down), all in-window strictly exhausted (banner fires), and the new
wire fields are populated.
- All 39 existing tests still pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

backend/admin.py

@@ -687,49 +687,131 @@ def _candidate_snapshot(model: str, health, chain_membership: list[str],
     }
 
 
+def _candidate_available_for_calls(h, now_mono: float) -> bool:
+    """KI-122 — Is this candidate AVAILABLE RIGHT NOW for real chat traffic?
+
+    This is the operator-facing definition of "live and usable":
+      - probe says healthy or degraded (NOT 'down' / 'unknown')
+      - has not been sin-binned by report_failure() in the last
+        DEGRADED_WINDOW_SEC / DEGRADE_DURATION_LONG_S window
+      - credits gate (`_has_credits`) passes — either no signal, OR
+        credits_reset_at has elapsed (stale snapshot), OR credits_remaining
+        is above the candidate's low_water.
+
+    Returns False for None / unknown candidates."""
+    if h is None:
+        return False
+    if h.status in ("down", "unknown"):
+        return False
+    if h.degraded_until_monotonic and h.degraded_until_monotonic > now_mono:
+        return False
+    if not llm_health._has_credits(h, now_mono):
+        return False
+    return True
+
+
+def _candidate_credit_exhausted_strict(h, now_mono: float) -> bool:
+    """KI-122 — STRICT credit-exhausted rule for a single candidate.
+
+    True ONLY when ALL three hold:
+      (1) credits_remaining is NOT None (we have a real signal)
+      (2) credits_remaining <= credits_low_water
+      (3) credits_reset_at is set AND in the FUTURE (we're inside an
+          active gating window — stale or absent reset means the
+          snapshot is no longer authoritative).
+
+    This is intentionally stricter than `not _has_credits()` because the
+    banner is louder than the elector. The elector falls through cheaply
+    on a single bad candidate; the banner falsely scaring the operator
+    when one quota-exhausted backup co-exists with a perfectly healthy
+    primary is the bug we're fixing here."""
+    if h is None:
+        return False
+    if h.credits_remaining is None:
+        return False
+    if h.credits_remaining > (h.credits_low_water or 0.0):
+        return False
+    if h.credits_reset_at is None:
+        # No scheduled reset (e.g. OpenRouter usd_balance — prepaid wallet).
+        # Could still be a real "wallet empty" signal, BUT we won't flag the
+        # chain banner on it alone because OpenRouter free-tier accounts
+        # report $0 even when calls succeed. The probe / chat success
+        # signal is the authoritative truth — fall through to the probe
+        # status check in `_candidate_available_for_calls` instead.
+        return False
+    if now_mono >= h.credits_reset_at:
+        # Reset window has elapsed — snapshot is stale, treat as permissive.
+        return False
+    return True
+
+
 def _chain_summary(role: str, chains: dict[str, list[str]],
                    state: dict, now_mono: float) -> dict:
     """Per-chain block for Section A. Includes elected primary/backup +
-    `all_credit_exhausted` so the frontend can render a banner when every
-    candidate in the chain is gated out by credits/quota."""
+    `chain_credit_exhausted` so the frontend can render a banner when every
+    candidate in the chain is genuinely unusable."""
     chain = chains.get(role) or []
     primary = llm_health.get_primary(role)
     backup = llm_health.get_backup(role)
 
-    # all_credit_exhausted: every chain member with a non-None credit signal
-    # is at-or-below its low-water mark. Chains with no signal at all are
-    # NOT flagged exhausted (cold-start should be permissive — election will
-    # try them and surface a real failure if any).
+    # KI-122 (2026-05-15) — STRICT chain_credit_exhausted rule.
     #
-    # KI-116 (2026-05-15) — mirror the `is_credit_eligible` reset-window logic.
-    # When credits_reset_at has elapsed, the LAST observed credits_remaining
-    # is stale (e.g., NIM 40-RPM cap from a spike 10 minutes ago says
-    # `remaining=0` but the 60s window has long since cycled). The elector
-    # treats elapsed-reset as permissive; the banner must match or the UI
-    # contradicts itself ("HEALTHY 100% success" + "credit-exhausted" at the
-    # same time, which user reported on the live admin panel).
-    any_signal = False
-    all_exhausted = True
+    # Earlier rules (KI-085, KI-116) tried to skip cold-start candidates
+    # and elapsed-reset snapshots but still left a hole: when one chain
+    # member had a None reset_at + low credits (typical for OpenRouter's
+    # usd_balance) while sibling NIM candidates had `credits_remaining=None`
+    # (no signal yet), the loop registered `any_signal=True` on the bad
+    # OpenRouter row but never broke out via a fresh NIM peer — so the
+    # banner fired despite the elected NIM primary being HEALTHY · 100%.
+    #
+    # New rule: banner fires ONLY when EVERY chain member is BOTH
+    #   (a) not available_for_calls (down / sin-binned / credit-gated), AND
+    #   (b) at least one of those failures is a credit-exhaustion signal
+    #       (otherwise it's a "chain entirely down" situation, which
+    #        deserves a different banner — handled by the elected_primary
+    #        == None path in the frontend).
+    members_info: list[dict] = []
+    any_available = False
+    any_credit_exhausted = False
+    last_probe_at_iso: Optional[str] = None
     for m in chain:
         h = state.get(m)
-        if h is None or h.credits_remaining is None:
-            continue
-        # If the credit signal's reset window has elapsed, skip — the snapshot
-        # is stale and the elector will try this candidate next call.
-        if h.credits_reset_at is not None and now_mono >= h.credits_reset_at:
-            all_exhausted = False
-            break
-        any_signal = True
-        if h.credits_remaining > (h.credits_low_water or 0.0):
-            all_exhausted = False
-            break
-    chain_credit_exhausted = bool(any_signal and all_exhausted)
+        available = _candidate_available_for_calls(h, now_mono)
+        credit_exhausted = _candidate_credit_exhausted_strict(h, now_mono)
+        if available:
+            any_available = True
+        if credit_exhausted:
+            any_credit_exhausted = True
+        # Track the most-recent probe timestamp across the chain.
+        ts = getattr(h, "tested_at", None) if h is not None else None
+        if ts and (last_probe_at_iso is None or ts > last_probe_at_iso):
+            last_probe_at_iso = ts
+        members_info.append({
+            "model":               m,
+            "available_for_calls": available,
+            "credit_exhausted":    credit_exhausted,
+            "is_current_primary":  bool(primary and m == primary),
+            "is_current_backup":   bool(backup and m == backup),
+        })
+    # Banner fires only when: zero available members AND at least one
+    # member is strictly credit-exhausted. If everyone is just 'down'
+    # (probe failures) the banner is the wrong message — the operator
+    # needs the "no eligible candidate" state, which the frontend already
+    # renders via `elected_primary == None`.
+    chain_credit_exhausted = bool((not any_available) and any_credit_exhausted)
 
     return {
         "role":            role,
         "chain":           chain,
+        "chain_members":   members_info,
         "elected_primary": primary,
         "elected_backup":  backup,
+        # KI-122 — explicit "currently in use" name + a boolean the
+        # frontend uses to render the "● IN USE" pill without re-deriving.
+        "current_primary": primary,
+        "current_primary_available": _candidate_available_for_calls(
+            state.get(primary) if primary else None, now_mono,
+        ),
         "primary_snapshot": _candidate_snapshot(
             primary, state.get(primary), [role], now_mono,
         ) if primary else None,
@@ -737,6 +819,10 @@ def _chain_summary(role: str, chains: dict[str, list[str]],
             backup, state.get(backup), [role], now_mono,
         ) if backup else None,
         "chain_credit_exhausted": chain_credit_exhausted,
+        # KI-122 — operator-facing staleness signal. ISO string + a
+        # convenience seconds-ago value so the UI can render "Probed Ns ago".
+        "last_probe_at":         last_probe_at_iso,
+        "last_probe_age_seconds": _probe_age_seconds(last_probe_at_iso),
     }
 
 

frontend/public/admin/llm-control.html

@@ -406,6 +406,63 @@
       font-size: 12px;
     }
 
+    /* KI-122 — "Currently in use" header + per-member availability pills */
+    .llm-health-chain .in-use-line {
+      display: flex;
+      align-items: center;
+      gap: 8px;
+      flex-wrap: wrap;
+      padding: 8px 10px;
+      margin-bottom: 12px;
+      background: rgba(63, 185, 80, 0.06);
+      border: 1px solid rgba(63, 185, 80, 0.25);
+      border-radius: 6px;
+    }
+    .llm-health-chain .in-use-line .role-label {
+      margin-bottom: 0;
+      color: var(--text);
+    }
+    .llm-health-chain .in-use-line .model-name {
+      font-family: ui-monospace, SFMono-Regular, "SF Mono", Menlo, Monaco, monospace;
+      font-size: 12px;
+      word-break: break-all;
+    }
+    .llm-health-chain .chain-members-list {
+      display: flex;
+      flex-direction: column;
+      gap: 4px;
+      margin: 6px 0 12px;
+      padding: 8px 10px;
+      background: #050709;
+      border: 1px solid var(--border);
+      border-radius: 6px;
+    }
+    .llm-health-chain .chain-member-row {
+      display: flex;
+      align-items: center;
+      gap: 8px;
+      font-size: 11px;
+    }
+    .llm-health-chain .chain-member-row .model-name {
+      font-family: ui-monospace, SFMono-Regular, "SF Mono", Menlo, Monaco, monospace;
+      word-break: break-all;
+    }
+    .avail-pill {
+      display: inline-block;
+      padding: 2px 8px;
+      border-radius: 999px;
+      font-size: 10px;
+      font-weight: 700;
+      letter-spacing: 0.04em;
+      border: 1px solid transparent;
+      white-space: nowrap;
+    }
+    .avail-pill.pill-inuse     { background: rgba(63, 185, 80, 0.20);  color: var(--green);  border-color: rgba(63, 185, 80, 0.55); }
+    .avail-pill.pill-available { background: rgba(63, 185, 80, 0.10);  color: var(--green);  border-color: rgba(63, 185, 80, 0.35); }
+    .avail-pill.pill-gated     { background: rgba(139, 149, 164, 0.15);color: var(--muted);  border-color: rgba(139, 149, 164, 0.40); }
+    .avail-pill.pill-degraded  { background: rgba(210, 153, 34, 0.15); color: var(--yellow); border-color: rgba(210, 153, 34, 0.45); }
+    .avail-pill.pill-down      { background: rgba(248, 81, 73, 0.15);  color: var(--red);    border-color: rgba(248, 81, 73, 0.45); }
+
     /* Status badges — three colors: ✓ green / ⚠ amber / ✗ red */
     .health-badge {
       display: inline-block;
@@ -1439,13 +1496,70 @@
         var badge = createEl('span', { className: 'role-badge', text: (ROLE_LABELS[chainBlock.role] || chainBlock.role).toUpperCase() });
         badge.style.background = ROLE_COLORS[chainBlock.role] || '#444';
         head.appendChild(badge);
-        var title = createEl('span', { text: 'Election' });
+        var title = createEl('span', { text: 'Chain status' });
         title.style.fontWeight = '600';
         title.style.fontSize = '13px';
         head.appendChild(title);
+        // KI-122 — staleness signal at the chain head so the operator can
+        // see whether they're looking at fresh probe data or stale state.
+        if (chainBlock.last_probe_age_seconds != null) {
+          var stale = createEl('span', {
+            className: 'small muted',
+            text: '· probed ' + fmtDurationShort(chainBlock.last_probe_age_seconds) + ' ago',
+          });
+          stale.style.marginLeft = 'auto';
+          head.appendChild(stale);
+        }
         card.appendChild(head);
 
-        // Render PRIMARY + BACKUP rows.
+        // KI-122 — top-of-card "Currently in use" line so the operator
+        // doesn't have to infer the live model from the PRIMARY row.
+        var inUse = createEl('div', { className: 'in-use-line' });
+        var inUseLabel = createEl('span', { className: 'role-label', text: 'Currently in use:' });
+        inUse.appendChild(inUseLabel);
+        if (chainBlock.current_primary) {
+          var pillCls = chainBlock.current_primary_available ? 'pill-inuse' : 'pill-degraded';
+          var pillText = chainBlock.current_primary_available ? '● IN USE' : '● ELECTED BUT GATED';
+          var pill = createEl('span', { className: 'avail-pill ' + pillCls, text: pillText });
+          inUse.appendChild(pill);
+          var nm = createEl('span', { className: 'model-name', text: chainBlock.current_primary });
+          inUse.appendChild(nm);
+        } else {
+          var none = createEl('span', { className: 'avail-pill pill-down', text: '● NONE ELECTED' });
+          inUse.appendChild(none);
+        }
+        card.appendChild(inUse);
+
+        // KI-122 — full chain member list with per-model availability pills,
+        // so the operator can see at a glance every model that's wired up
+        // and which ones are actually callable RIGHT NOW.
+        var members = chainBlock.chain_members || [];
+        if (members.length) {
+          var listLabel = createEl('div', { className: 'role-label', text: 'Chain members (' + members.length + ')' });
+          listLabel.style.marginTop = '10px';
+          card.appendChild(listLabel);
+          var list = createEl('div', { className: 'chain-members-list' });
+          members.forEach(function (mi) {
+            var row = createEl('div', { className: 'chain-member-row' });
+            var p;
+            if (mi.is_current_primary) {
+              p = createEl('span', { className: 'avail-pill pill-inuse', text: '● IN USE' });
+            } else if (mi.available_for_calls) {
+              p = createEl('span', { className: 'avail-pill pill-available', text: '● AVAILABLE' });
+            } else if (mi.credit_exhausted) {
+              p = createEl('span', { className: 'avail-pill pill-gated', text: '● GATED' });
+            } else {
+              p = createEl('span', { className: 'avail-pill pill-down', text: '● UNAVAILABLE' });
+            }
+            row.appendChild(p);
+            row.appendChild(createEl('span', { className: 'model-name', text: mi.model }));
+            list.appendChild(row);
+          });
+          card.appendChild(list);
+        }
+
+        // Render PRIMARY + BACKUP detail rows (kept for the per-model
+        // latency/success/credit telemetry the operator drills into).
         ['primary', 'backup'].forEach(function (role) {
           var snap = chainBlock[role + '_snapshot'];
           var name = chainBlock['elected_' + role];
@@ -1453,8 +1567,8 @@
           var label = createEl('div', { className: 'role-label', text: role.toUpperCase() });
           block.appendChild(label);
           if (!name || !snap) {
-            var none = createEl('div', { className: 'none-elected', text: 'none elected (no eligible candidate)' });
-            block.appendChild(none);
+            var noneEl = createEl('div', { className: 'none-elected', text: 'none elected (no eligible candidate)' });
+            block.appendChild(noneEl);
             card.appendChild(block);
             return;
           }
@@ -1487,6 +1601,11 @@
           card.appendChild(block);
         });
 
+        // KI-122 — banner is now backend-gated by the STRICT exhausted rule
+        // (`chain_credit_exhausted` true only when zero members are
+        // available_for_calls AND at least one is strictly credit-exhausted).
+        // This prevents the historic "HEALTHY 100% + credit-exhausted banner"
+        // contradiction the operator reported.
         if (chainBlock.chain_credit_exhausted) {
           var banner = createEl('div', { className: 'credit-banner' });
           banner.textContent = 'All candidates for ' + (ROLE_LABELS[chainBlock.role] || chainBlock.role).toUpperCase() +

tests/test_credits_election.py

@@ -245,5 +245,138 @@ class TestElectionCreditGate(unittest.TestCase):
                         "Cold-start (None credits) must NOT gate out a healthy candidate.")
 
 
+class TestStrictChainCreditExhausted(unittest.TestCase):
+    """KI-122 (2026-05-15) — strict banner rule.
+
+    Banner fires ONLY when zero chain members are available_for_calls AND at
+    least one has a credit-exhaustion signal that's strictly in-window
+    (credits_remaining<=low_water AND credits_reset_at is in the future).
+
+    Pre-KI-122 holes the new rule must close:
+      H1) one healthy NIM primary (credits=None) + one credit-low Groq backup
+          → banner USED to fire because `any_signal=True` on Groq + skip on
+            NIM's None signal. Must NOT fire.
+      H2) every chain member has reset_at=None (e.g. OpenRouter wallet) and
+          credits below water → banner USED to fire. Must NOT fire by itself
+          (probe success is the authoritative truth; wallet-zero with
+          successful probes means free-tier models).
+      H3) every member is genuinely gated (low + future reset) AND not
+          available → banner SHOULD fire.
+    """
+
+    def setUp(self) -> None:
+        _fresh_state()
+        # Import locally so test discovery doesn't load admin until needed.
+        from backend.admin import _chain_summary
+        self._chain_summary = _chain_summary
+
+    def _build(self, chain_models, model_states):
+        """Inject states + run _chain_summary against a mocked chain."""
+        for m, h in model_states.items():
+            llm_health._STATE[m] = h
+        now_mono = time.monotonic()
+        with mock.patch.object(
+            llm_health, "_chain_for", return_value=list(chain_models),
+        ):
+            return self._chain_summary(
+                "brain", {"brain": list(chain_models)}, llm_health._STATE, now_mono,
+            )
+
+    def test_healthy_nim_plus_exhausted_groq_does_not_fire_banner(self) -> None:
+        """H1 — NIM primary HEALTHY/None-credits + Groq backup credit-low.
+        Banner must STAY DOWN because NIM is available_for_calls."""
+        nim = "qwen/qwen3-next-80b-a3b-instruct"
+        groq = "groq:llama-3.3-70b-versatile"
+        nh = _healthy_now(nim)
+        nh.latency_ms = 300
+        # NIM has NO credit signal (None) — cold start permissive.
+        gh = _healthy_now(groq)
+        gh.latency_ms = 100
+        gh.credits_remaining = 100.0
+        gh.credits_unit = "tokens_day"
+        gh.credits_low_water = GROQ_TOKENS_LOW_WATER
+        gh.credits_reset_at = time.monotonic() + 3600.0  # future reset
+        gh.credits_observed_at = time.monotonic()
+        block = self._build([nim, groq], {nim: nh, groq: gh})
+        self.assertFalse(
+            block["chain_credit_exhausted"],
+            "Banner must NOT fire when a healthy cold-start NIM primary "
+            "shares a chain with one credit-exhausted Groq backup.",
+        )
+        # Sanity: NIM should be in_use, Groq should be gated.
+        names = {mi["model"]: mi for mi in block["chain_members"]}
+        self.assertTrue(names[nim]["available_for_calls"])
+        self.assertTrue(names[nim]["is_current_primary"])
+        self.assertFalse(names[groq]["available_for_calls"])
+        self.assertTrue(names[groq]["credit_exhausted"])
+
+    def test_credits_reset_at_none_does_not_fire_banner(self) -> None:
+        """H2 — every member has reset_at=None + credits below water (typical
+        OpenRouter usd_balance free-tier shape). Banner must NOT fire on the
+        reset_at=None signal alone; the strict rule requires an in-window reset."""
+        m1 = "openrouter:openai/gpt-oss-120b"
+        m2 = "openrouter:meta-llama/llama-3.3-70b-instruct:free"
+        h1 = _healthy_now(m1)
+        h1.latency_ms = 800
+        h1.credits_remaining = 0.0
+        h1.credits_unit = "usd_balance"
+        h1.credits_low_water = 0.05
+        h1.credits_reset_at = None  # OpenRouter wallet: no scheduled reset
+        h1.credits_observed_at = time.monotonic()
+        h2 = _healthy_now(m2)
+        h2.latency_ms = 900
+        h2.credits_remaining = 0.0
+        h2.credits_unit = "usd_balance"
+        h2.credits_low_water = 0.05
+        h2.credits_reset_at = None
+        h2.credits_observed_at = time.monotonic()
+        block = self._build([m1, m2], {m1: h1, m2: h2})
+        self.assertFalse(
+            block["chain_credit_exhausted"],
+            "Banner must NOT fire when credits_reset_at is None on every "
+            "member — usd_balance=$0 with healthy probes is a free-tier "
+            "signal, not a real outage.",
+        )
+
+    def test_all_in_window_exhausted_fires_banner(self) -> None:
+        """H3 — every chain member has credits<=low_water AND a future reset
+        AND is consequently NOT available_for_calls. Banner SHOULD fire."""
+        m1 = "groq:llama-3.3-70b-versatile"
+        m2 = "groq:meta-llama/llama-4-scout-17b-16e-instruct"
+        future = time.monotonic() + 3600.0
+        h1 = _healthy_now(m1)
+        h1.latency_ms = 100
+        h1.credits_remaining = 100.0
+        h1.credits_unit = "tokens_day"
+        h1.credits_low_water = GROQ_TOKENS_LOW_WATER
+        h1.credits_reset_at = future
+        h1.credits_observed_at = time.monotonic()
+        h2 = _healthy_now(m2)
+        h2.latency_ms = 110
+        h2.credits_remaining = 50.0
+        h2.credits_unit = "tokens_day"
+        h2.credits_low_water = GROQ_TOKENS_LOW_WATER
+        h2.credits_reset_at = future
+        h2.credits_observed_at = time.monotonic()
+        block = self._build([m1, m2], {m1: h1, m2: h2})
+        self.assertTrue(
+            block["chain_credit_exhausted"],
+            "Banner SHOULD fire when every chain member is strictly "
+            "credit-exhausted with an in-window future reset.",
+        )
+
+    def test_chain_block_carries_current_primary_and_last_probe(self) -> None:
+        """KI-122 — verify the new wire fields are populated."""
+        nim = "qwen/qwen3-next-80b-a3b-instruct"
+        nh = _healthy_now(nim)
+        nh.latency_ms = 250
+        block = self._build([nim], {nim: nh})
+        self.assertEqual(block["current_primary"], nim)
+        self.assertTrue(block["current_primary_available"])
+        self.assertIsNotNone(block["last_probe_at"])
+        self.assertIsNotNone(block["last_probe_age_seconds"])
+        self.assertGreaterEqual(block["last_probe_age_seconds"], 0.0)
+
+
 if __name__ == "__main__":
     unittest.main(verbosity=2)