rohitsar567 Claude Opus 4.7 (1M context) commited on
Commit
2689ab0
·
1 Parent(s): f6901d8

fix(admin): KI-267 — Health column + consolidated LLM Health header

Browse files

C2 — Health column on both admin tables:
- backend/llm_health.py:
* ModelHealth.last_status_code: Optional[int] field (HTTP code from probe/chat
failures) + back-compat default in _load_into_memory.
* New helpers _classify_error_reason() + _extract_status_code() (regex with
digit-boundary so Status429 / http_429 / HTTPStatusError:503 all match).
* _absorb_probe_result, report_failure, report_success stamp last_status_code
(cleared on success).
* status_summary now emits effective_status + last_status_code +
health_reason per row. Categories: "rate limit (429)" / "out of credits" /
"network issue" / "service unavailable ({code})" / "auth error ({code})" /
"stale" / first 40 chars fallback.
- backend/admin.py: _candidate_snapshot exposes the new fields.
- frontend/public/admin/llm-control.html:
* NEW renderHealthCell(snap) + buildCandidatesMap(payload) helpers.
* "Currently In Use" + "All Eligible Models" tables get a "Health" column.
* Renders green dot + "Live" or red dot + "Off — {reason}". Tier-0 Gemini
rows synth a healthy snapshot since the NIM probe loop doesn't cover it
(Gemini health comes from chat-path report_failure).
* Reused existing .health-dot.ok / .health-dot.bad CSS — no new styles.

C1 — Consolidated LLM Health header into one line:
- New header row: bold "LLM Health" + muted meta "Last refresh: Xm Ys ago ·
Next in: Xm Ys" + right-aligned "Refresh now" button.
- Dropped legacy #llm-health-snapshot-ts + #llm-health-stale-badge spans
(FRESH/STALE pill removed — timing line conveys it).
- renderUpdatedLabel rewritten for the new ago/next spans.
- tickFooters badge block removed.
- "Probing…" / "auto-probe" strings replaced with "Refreshing…" / "next
refresh". Backend endpoint /api/admin/probe unchanged.
- "Refresh" button label → "Refresh now"; reset string updated.

KI-258b — Countdown never shows "due now":
- renderUpdatedLabel now projects forward to the next interval boundary:
elapsed = (now - probeAt)
intoNext = ((elapsed % PROBE_INTERVAL_MS) + PROBE_INTERVAL_MS) % PROBE_INTERVAL_MS
untilNext = PROBE_INTERVAL_MS - intoNext
So an overdue auto-probe rolls over and the countdown always shows real
minutes:seconds the user can act on.

Verification:
- python -m py_compile clean on llm_health.py + admin.py.
- node --check on the inline <script> block: OK.
- grep llm-health-stale-badge / auto-probe: 0 hits.
- Classifier unit tests: 11/11 categories correct.
- Render simulation: 10 synthetic snapshots produce expected dot+label+tooltip.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

backend/admin.py CHANGED
@@ -678,10 +678,13 @@ def _candidate_snapshot(model: str, health, chain_membership: list[str],
678
  "provider": llm_health.provider_of(model),
679
  "chain_membership": chain_membership,
680
  "status": "unknown",
 
681
  "latency_ms": None,
682
  "success_rate": None,
683
  "probe_age_seconds": None,
684
  "last_error": None,
 
 
685
  "credits_remaining": None,
686
  "credits_unit": None,
687
  "credits_low_water": None,
@@ -692,15 +695,29 @@ def _candidate_snapshot(model: str, health, chain_membership: list[str],
692
  deg_for = None
693
  if deg_until and deg_until > now_mono:
694
  deg_for = round(deg_until - now_mono, 1)
 
 
 
 
 
 
 
 
 
 
 
695
  return {
696
  "model": model,
697
  "provider": llm_health.provider_of(model),
698
  "chain_membership": chain_membership,
699
  "status": health.status,
 
700
  "latency_ms": health.latency_ms,
701
  "success_rate": _success_rate_for(health),
702
  "probe_age_seconds": _probe_age_seconds(health.tested_at),
703
  "last_error": health.last_error,
 
 
704
  "credits_remaining": health.credits_remaining,
705
  "credits_unit": health.credits_unit,
706
  "credits_low_water": health.credits_low_water,
 
678
  "provider": llm_health.provider_of(model),
679
  "chain_membership": chain_membership,
680
  "status": "unknown",
681
+ "effective_status": "unknown",
682
  "latency_ms": None,
683
  "success_rate": None,
684
  "probe_age_seconds": None,
685
  "last_error": None,
686
+ "last_status_code": None,
687
+ "health_reason": None,
688
  "credits_remaining": None,
689
  "credits_unit": None,
690
  "credits_low_water": None,
 
695
  deg_for = None
696
  if deg_until and deg_until > now_mono:
697
  deg_for = round(deg_until - now_mono, 1)
698
+ # KI-202 — operator-facing reason for the admin Health column. None when
699
+ # the row is healthy (renders "Live" only) or has no error signal yet.
700
+ eff = llm_health.effective_status(health)
701
+ if eff == "stale":
702
+ health_reason = "stale"
703
+ elif eff == "healthy":
704
+ health_reason = None
705
+ else:
706
+ health_reason = llm_health._classify_error_reason(
707
+ health.last_error, health.last_status_code,
708
+ )
709
  return {
710
  "model": model,
711
  "provider": llm_health.provider_of(model),
712
  "chain_membership": chain_membership,
713
  "status": health.status,
714
+ "effective_status": eff,
715
  "latency_ms": health.latency_ms,
716
  "success_rate": _success_rate_for(health),
717
  "probe_age_seconds": _probe_age_seconds(health.tested_at),
718
  "last_error": health.last_error,
719
+ "last_status_code": health.last_status_code,
720
+ "health_reason": health_reason,
721
  "credits_remaining": health.credits_remaining,
722
  "credits_unit": health.credits_unit,
723
  "credits_low_water": health.credits_low_water,
backend/llm_health.py CHANGED
@@ -194,6 +194,94 @@ def _headers_for(model_id: str, api_key: str) -> dict[str, str]:
194
  return h
195
 
196
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
197
  def provider_of(model_id: str) -> str:
198
  """Coarse provider bucket — used by the election routine to prefer a
199
  cross-provider backup so a NIM regional outage can't take out both
@@ -221,6 +309,13 @@ class ModelHealth:
221
  last_success_at: Optional[str] = None
222
  last_failure_at: Optional[str] = None
223
  last_error: Optional[str] = None
 
 
 
 
 
 
 
224
  latency_ms: Optional[int] = None
225
  consecutive_failures: int = 0
226
  tested_at: Optional[str] = None
@@ -312,6 +407,7 @@ def _load_into_memory() -> None:
312
  # records missing the five credits_* fields).
313
  v.setdefault("probe_history", [])
314
  v.setdefault("degraded_until_monotonic", 0.0)
 
315
  v.setdefault("credits_remaining", None)
316
  v.setdefault("credits_unit", None)
317
  v.setdefault("credits_reset_at", None)
@@ -642,6 +738,11 @@ def report_failure(chain_name: str, model: str, error_class: str) -> None:
642
  h.degraded_until_monotonic = time.monotonic() + degrade_for_s
643
  h.last_failure_at = _now_iso()
644
  h.last_error = f"chat_failure: {error_class}"
 
 
 
 
 
645
  h.probe_history.append({
646
  "ok": False,
647
  "latency_ms": None,
@@ -676,6 +777,7 @@ def report_success(chain_name: str, model: str, latency_ms: int) -> None:
676
  h = _STATE.get(model) or ModelHealth(model=model)
677
  h.last_success_at = _now_iso()
678
  h.last_error = None
 
679
  h.latency_ms = int(latency_ms)
680
  h.consecutive_failures = 0
681
  h.tested_at = _now_iso()
@@ -1059,12 +1161,17 @@ def _absorb_probe_result(model: str, ok: bool, err: str, latency: Optional[int])
1059
  if ok:
1060
  h.last_success_at = _now_iso()
1061
  h.last_error = None
 
1062
  h.latency_ms = latency
1063
  h.consecutive_failures = 0
1064
  h.status = "healthy" if (latency or 0) < 5000 else "degraded"
1065
  else:
1066
  h.last_failure_at = _now_iso()
1067
  h.last_error = err
 
 
 
 
1068
  h.consecutive_failures += 1
1069
  if h.consecutive_failures >= DOWN_AFTER_CONSECUTIVE_FAILS:
1070
  h.status = "down"
@@ -1152,14 +1259,26 @@ def status_summary() -> dict:
1152
  # rows whose stored 'healthy' verdict is older than STALE_AGE_SEC.
1153
  eff = effective_status(h)
1154
  summary["by_status"][eff] = summary["by_status"].get(eff, 0) + 1
 
 
 
 
 
 
 
 
 
1155
  summary["models"].append({
1156
  "model": m,
1157
  "status": eff,
 
1158
  "stored_status": h.status, # preserved for debug / drift detection
1159
  "latency_ms": h.latency_ms,
1160
  "last_success_at": h.last_success_at,
1161
  "last_failure_at": h.last_failure_at,
1162
  "last_error": h.last_error,
 
 
1163
  "tested_at": h.tested_at,
1164
  # KI-085 — surface credits state for the admin UI.
1165
  "credits_remaining": h.credits_remaining,
 
194
  return h
195
 
196
 
197
+ # KI-202 (2026-05-15) — categorize a `last_error` string + optional HTTP
198
+ # status code into the short operator-facing reason rendered by the admin
199
+ # Health columns. Centralised here so backend and any other consumer agree
200
+ # on the same vocabulary ("network issue" / "rate limit (429)" / etc.).
201
+ # Use lookbehind/lookahead on non-digit so we still match e.g. "Status429",
202
+ # "http_429", "HTTPStatusError:503" — \b word-boundary alone treats "_" as
203
+ # a word char so "http_429" wouldn't match.
204
+ _RATE_LIMIT_RE = re.compile(r"(?<!\d)429(?!\d)|rate[_\s-]?limit", re.IGNORECASE)
205
+ _AUTH_RE = re.compile(r"(?<!\d)40[13](?!\d)|unauthor|forbidden", re.IGNORECASE)
206
+ _QUOTA_RE = re.compile(r"(?<!\d)402(?!\d)|quota|out[\s_-]?of[\s_-]?credit|insufficient[_\s]?funds|payment[_\s]?required", re.IGNORECASE)
207
+ _TIMEOUT_RE = re.compile(r"timeout|timed[_\s-]?out|connect(ion)?[_\s-]?(refused|reset|error)|network|dns|name[_\s-]?or[_\s-]?service", re.IGNORECASE)
208
+ _5XX_RE = re.compile(r"(?<!\d)5\d{2}(?!\d)|service[_\s-]?unavailable|bad[_\s-]?gateway|gateway[_\s-]?timeout|internal[_\s-]?server", re.IGNORECASE)
209
+
210
+
211
+ def _classify_error_reason(last_error: Optional[str],
212
+ last_status_code: Optional[int] = None) -> Optional[str]:
213
+ """Map a stored `last_error` string (+ optional HTTP code) into a short
214
+ operator-facing reason for the admin Health column. Returns None when
215
+ there is no error signal at all (caller renders just "Live" or "Off —
216
+ unknown" depending on status).
217
+
218
+ Categories (matches the display contract in the admin spec):
219
+ - "network issue" — timeout / connection refused / DNS
220
+ - "out of credits" — HTTP 402 / quota / insufficient funds
221
+ - "rate limit (429)" — HTTP 429 / rate-limit keywords
222
+ - "service unavailable (5xx)"— HTTP 5xx
223
+ - "auth error (4xx)" — HTTP 401 / 403
224
+ - "stale" — explicitly surfaced by effective_status()
225
+ - first-40-chars fallback — anything else, truncated so the cell
226
+ doesn't blow up the table width.
227
+ """
228
+ if not last_error:
229
+ return None
230
+
231
+ # Prefer the explicit status code when present — it's unambiguous.
232
+ if last_status_code is not None:
233
+ code = int(last_status_code)
234
+ if code == 429:
235
+ return "rate limit (429)"
236
+ if code == 402:
237
+ return "out of credits"
238
+ if code in (401, 403):
239
+ return f"auth error ({code})"
240
+ if 500 <= code < 600:
241
+ return f"service unavailable ({code})"
242
+
243
+ text = str(last_error)
244
+
245
+ if _RATE_LIMIT_RE.search(text):
246
+ return "rate limit (429)"
247
+ if _QUOTA_RE.search(text):
248
+ return "out of credits"
249
+ if _AUTH_RE.search(text):
250
+ # Try to surface the code if it's embedded in the string.
251
+ m = re.search(r"(?<!\d)(40[13])(?!\d)", text)
252
+ return f"auth error ({m.group(1)})" if m else "auth error"
253
+ if _TIMEOUT_RE.search(text):
254
+ return "network issue"
255
+ if _5XX_RE.search(text):
256
+ m = re.search(r"(?<!\d)(5\d{2})(?!\d)", text)
257
+ return f"service unavailable ({m.group(1)})" if m else "service unavailable"
258
+
259
+ # Final fallback — truncate so the table cell stays readable.
260
+ snippet = text.strip().splitlines()[0] if text.strip() else ""
261
+ return snippet[:40] if snippet else None
262
+
263
+
264
+ def _extract_status_code(err: str) -> Optional[int]:
265
+ """Best-effort scrape of an HTTP status code out of an error string —
266
+ used by the probe path to backfill `last_status_code` from the existing
267
+ `http_{code}` / `HTTPStatusError:{code}` tags without changing every
268
+ probe-result producer. Returns None when no 3-digit HTTP code is found."""
269
+ if not err:
270
+ return None
271
+ # Surround-by-non-digit boundary so we still catch "http_429" / "Status429"
272
+ # where \b would fail because '_' is a word char.
273
+ m = re.search(r"(?<!\d)(\d{3})(?!\d)", err)
274
+ if not m:
275
+ return None
276
+ try:
277
+ code = int(m.group(1))
278
+ except ValueError:
279
+ return None
280
+ if 100 <= code < 600:
281
+ return code
282
+ return None
283
+
284
+
285
  def provider_of(model_id: str) -> str:
286
  """Coarse provider bucket — used by the election routine to prefer a
287
  cross-provider backup so a NIM regional outage can't take out both
 
309
  last_success_at: Optional[str] = None
310
  last_failure_at: Optional[str] = None
311
  last_error: Optional[str] = None
312
+ # KI-202 (2026-05-15) — explicit HTTP status code from the most recent
313
+ # failed probe / chat call. `last_error` is a free-form string ("timeout",
314
+ # "http_429", "net: TimeoutException: ..."); `last_status_code` is the
315
+ # parsed integer (or None when the failure wasn't HTTP-shaped). Surfaced
316
+ # in admin Health columns so the operator can see "Off — rate limit (429)"
317
+ # vs "Off — service unavailable (503)" at a glance.
318
+ last_status_code: Optional[int] = None
319
  latency_ms: Optional[int] = None
320
  consecutive_failures: int = 0
321
  tested_at: Optional[str] = None
 
407
  # records missing the five credits_* fields).
408
  v.setdefault("probe_history", [])
409
  v.setdefault("degraded_until_monotonic", 0.0)
410
+ v.setdefault("last_status_code", None)
411
  v.setdefault("credits_remaining", None)
412
  v.setdefault("credits_unit", None)
413
  v.setdefault("credits_reset_at", None)
 
738
  h.degraded_until_monotonic = time.monotonic() + degrade_for_s
739
  h.last_failure_at = _now_iso()
740
  h.last_error = f"chat_failure: {error_class}"
741
+ # KI-202 — `error_class` from NimChainLLM._classify_error is
742
+ # `Status429`, `HTTPStatusError:503`, `TimeoutException`, etc.
743
+ # Pull the trailing 3-digit code out so the admin Health column
744
+ # can render a code-specific label.
745
+ h.last_status_code = _extract_status_code(error_class)
746
  h.probe_history.append({
747
  "ok": False,
748
  "latency_ms": None,
 
777
  h = _STATE.get(model) or ModelHealth(model=model)
778
  h.last_success_at = _now_iso()
779
  h.last_error = None
780
+ h.last_status_code = None
781
  h.latency_ms = int(latency_ms)
782
  h.consecutive_failures = 0
783
  h.tested_at = _now_iso()
 
1161
  if ok:
1162
  h.last_success_at = _now_iso()
1163
  h.last_error = None
1164
+ h.last_status_code = None
1165
  h.latency_ms = latency
1166
  h.consecutive_failures = 0
1167
  h.status = "healthy" if (latency or 0) < 5000 else "degraded"
1168
  else:
1169
  h.last_failure_at = _now_iso()
1170
  h.last_error = err
1171
+ # KI-202 — backfill last_status_code from the err tag so the
1172
+ # admin Health column can render code-specific reasons (429 /
1173
+ # 503 / etc.) without changing every producer.
1174
+ h.last_status_code = _extract_status_code(err)
1175
  h.consecutive_failures += 1
1176
  if h.consecutive_failures >= DOWN_AFTER_CONSECUTIVE_FAILS:
1177
  h.status = "down"
 
1259
  # rows whose stored 'healthy' verdict is older than STALE_AGE_SEC.
1260
  eff = effective_status(h)
1261
  summary["by_status"][eff] = summary["by_status"].get(eff, 0) + 1
1262
+ # KI-202 — operator-facing reason string for the admin Health
1263
+ # column. None when there is no error signal (caller renders just
1264
+ # "Live" or "Off — stale"/"Off — unknown" based on `effective_status`).
1265
+ if eff == "stale":
1266
+ health_reason = "stale"
1267
+ elif eff in ("healthy",):
1268
+ health_reason = None
1269
+ else:
1270
+ health_reason = _classify_error_reason(h.last_error, h.last_status_code)
1271
  summary["models"].append({
1272
  "model": m,
1273
  "status": eff,
1274
+ "effective_status": eff,
1275
  "stored_status": h.status, # preserved for debug / drift detection
1276
  "latency_ms": h.latency_ms,
1277
  "last_success_at": h.last_success_at,
1278
  "last_failure_at": h.last_failure_at,
1279
  "last_error": h.last_error,
1280
+ "last_status_code": h.last_status_code,
1281
+ "health_reason": health_reason,
1282
  "tested_at": h.tested_at,
1283
  # KI-085 — surface credits state for the admin UI.
1284
  "credits_remaining": h.credits_remaining,
frontend/public/admin/llm-control.html CHANGED
@@ -585,6 +585,9 @@
585
  gap: 6px;
586
  font-variant-numeric: tabular-nums;
587
  }
 
 
 
588
  .stale-badge {
589
  display: inline-block;
590
  padding: 1px 7px;
@@ -610,6 +613,26 @@
610
  border-color: rgba(248, 81, 73, 0.40);
611
  }
612
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
613
  /* A5 — Persona drift panel + recommendation history panel */
614
  .collapsible {
615
  border: 1px solid var(--border);
@@ -795,8 +818,15 @@
795
  may live inside this section: the h2 header (with snapshot timestamp
796
  span), the refresh button, and the chains container that holds the
797
  two generated tables (Currently In Use + All Eligible Models). -->
798
- <h2>LLM health <span class="small muted" id="llm-health-snapshot-ts"></span> <span id="llm-health-stale-badge"></span></h2>
799
- <button id="btn-refresh-llm-health">Refresh</button>
 
 
 
 
 
 
 
800
  <div id="llm-health-chains" class="llm-simple-tables"></div>
801
  <div id="llm-health-footer" class="table-footer"></div>
802
  </section>
@@ -1409,22 +1439,27 @@
1409
  }
1410
 
1411
  function renderUpdatedLabel() {
1412
- var el = $('last-updated');
1413
- if (!el) return;
1414
- // Only show the LLM-Chain probe staleness when that tab is active —
1415
- // it's irrelevant on Profiles / Performance.
 
1416
  if (STATE.activeTab !== 'chain') {
1417
- el.textContent = '';
 
1418
  return;
1419
  }
1420
  var probeAt = probeTimestampMs();
1421
- if (!probeAt) { el.textContent = ''; return; }
1422
  var now = Date.now();
1423
- var since = formatMinSec(now - probeAt);
1424
- var nextAt = probeAt + PROBE_INTERVAL_MS;
1425
- var untilNext = nextAt > now ? formatMinSec(nextAt - now) : 'due now';
1426
- var wallClock = new Date(probeAt).toLocaleTimeString();
1427
- el.textContent = 'Updated ' + wallClock + ' · ' + since + ' ago · next auto-probe in ' + untilNext;
 
 
 
1428
  }
1429
 
1430
  setInterval(renderUpdatedLabel, 1000);
@@ -1739,6 +1774,62 @@
1739
  return td;
1740
  }
1741
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1742
  // KI-200 — accepts the full llm-health payload so we can read
1743
  // `gemini_available` (when backend exposes it). Caller signature
1744
  // updated below in renderLlmHealth().
@@ -1746,6 +1837,8 @@
1746
  var chains = (healthPayload && healthPayload.chains) || [];
1747
  var chainsByRole = {};
1748
  chains.forEach(function (c) { if (c && c.role) chainsByRole[c.role] = c; });
 
 
1749
 
1750
  // ---- Table 1: Currently In Use ----
1751
  var liveBlock = createEl('div', { className: 'simple-table-block' });
@@ -1753,9 +1846,10 @@
1753
  var liveTable = createEl('table');
1754
  var liveThead = createEl('thead');
1755
  var liveHr = createEl('tr');
1756
- ['Use', 'Current Model', 'Provider', 'Tier'].forEach(function (h, i) {
 
1757
  var th = createEl('th', { text: h });
1758
- if (i >= 2) th.style.textAlign = 'center';
1759
  liveHr.appendChild(th);
1760
  });
1761
  liveThead.appendChild(liveHr);
@@ -1789,6 +1883,19 @@
1789
  tierCell.textContent = live.tier;
1790
  tr.appendChild(tierCell);
1791
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1792
  liveTbody.appendChild(tr);
1793
  });
1794
  liveTable.appendChild(liveTbody);
@@ -1801,10 +1908,13 @@
1801
  var availTable = createEl('table');
1802
  var availThead = createEl('thead');
1803
  var availHr = createEl('tr');
1804
- var availCols = ['Model', 'Provider', 'Tier'].concat(SIMPLE_USE_ORDER.map(function (r) { return SIMPLE_USE_LABELS[r]; }));
 
 
 
1805
  availCols.forEach(function (h, i) {
1806
  var th = createEl('th', { text: h });
1807
- if (i >= 1) th.style.textAlign = 'center';
1808
  availHr.appendChild(th);
1809
  });
1810
  availThead.appendChild(availHr);
@@ -1834,6 +1944,13 @@
1834
  tr.appendChild(renderRoleLabelCell(entry.roles[role]));
1835
  });
1836
 
 
 
 
 
 
 
 
1837
  availTbody.appendChild(tr);
1838
  });
1839
  availTable.appendChild(availTbody);
@@ -1959,15 +2076,15 @@
1959
 
1960
  function renderLlmHealth() {
1961
  var chainsHost = $('llm-health-chains');
1962
- var snapEl = $('llm-health-snapshot-ts');
1963
  if (!chainsHost) return;
1964
 
 
 
1965
  if (!STATE.llmHealth) {
1966
  clearChildren(chainsHost);
1967
  chainsHost.appendChild(createEl('div', { className: 'empty-state', text: 'Loading…' }));
1968
  clearChildren($('llm-health-candidates'));
1969
  clearChildren($('llm-health-recent'));
1970
- if (snapEl) snapEl.textContent = '';
1971
  return;
1972
  }
1973
  if (STATE.llmHealth.__notDeployed) {
@@ -1975,7 +2092,6 @@
1975
  chainsHost.appendChild(buildPendingMessage('/api/admin/llm-health'));
1976
  clearChildren($('llm-health-candidates'));
1977
  clearChildren($('llm-health-recent'));
1978
- if (snapEl) snapEl.textContent = '';
1979
  return;
1980
  }
1981
 
@@ -1985,12 +2101,6 @@
1985
 
1986
  renderLlmHealthCandidates(STATE.llmHealth.candidates || []);
1987
  renderLlmHealthRecent(STATE.llmHealth.recent_turns || []);
1988
-
1989
- if (snapEl) {
1990
- snapEl.textContent = STATE.llmHealth.snapshot_ts
1991
- ? '· updated ' + fmtIST(STATE.llmHealth.snapshot_ts)
1992
- : '';
1993
- }
1994
  }
1995
 
1996
  // Auto-poll every 30s while the LLM Chain tab is the active tab.
@@ -2757,21 +2867,8 @@
2757
  }
2758
  if (!probeAt) probeAt = STATE.llmHealthLoadedAt;
2759
  setFooter('llm-health-footer', probeAt);
2760
- // LLM-health stale badge sits next to the section header — render
2761
- // separately so it's visible even when the body is empty.
2762
- var badgeHost = $('llm-health-stale-badge');
2763
- if (badgeHost) {
2764
- clearChildren(badgeHost);
2765
- if (probeAt) {
2766
- var ageMs = Date.now() - probeAt;
2767
- var stale = ageMs > STALE_THRESHOLD_MS;
2768
- var badge = createEl('span', {
2769
- className: 'stale-badge ' + (stale ? '' : 'fresh'),
2770
- text: stale ? 'STALE' : 'FRESH'
2771
- });
2772
- badgeHost.appendChild(badge);
2773
- }
2774
- }
2775
  }
2776
  setInterval(tickFooters, 1000);
2777
 
@@ -2977,13 +3074,13 @@
2977
  llmHealthBtn.onclick = function () {
2978
  var btn = this;
2979
  btn.disabled = true;
2980
- btn.textContent = 'Probing…';
2981
  apiPost('/api/admin/probe', null)
2982
  .catch(function () { /* tolerate probe failure; still re-fetch cached state */ })
2983
  .then(function () { return fetchLlmHealth(); })
2984
- .then(function () { renderLlmHealth(); toast('LLM health refreshed (fresh probe)', 'success'); })
2985
  .catch(handleFetchErr)
2986
- .then(function () { btn.disabled = false; btn.textContent = 'Refresh'; });
2987
  };
2988
  }
2989
  }
 
585
  gap: 6px;
586
  font-variant-numeric: tabular-nums;
587
  }
588
+ /* LEGACY (KI-208) — .stale-badge rules retained but no longer referenced by
589
+ the LLM Health header (consolidated into a single-line refresh-meta row).
590
+ Still used by per-table footers via setFooter(). Safe to keep. */
591
  .stale-badge {
592
  display: inline-block;
593
  padding: 1px 7px;
 
613
  border-color: rgba(248, 81, 73, 0.40);
614
  }
615
 
616
+ /* KI-208 — consolidated LLM Health header row: heading + last refresh +
617
+ next refresh + refresh button on one flex line. */
618
+ .llm-health-header-row {
619
+ display: flex;
620
+ align-items: center;
621
+ gap: 12px;
622
+ flex-wrap: wrap;
623
+ margin-bottom: 8px;
624
+ }
625
+ .llm-health-header-row h2 { margin: 0; }
626
+ .llm-refresh-meta {
627
+ color: var(--muted, #888);
628
+ font-size: 12px;
629
+ display: inline-flex;
630
+ gap: 6px;
631
+ align-items: center;
632
+ }
633
+ .llm-refresh-meta .dot-sep { opacity: 0.5; }
634
+ #btn-refresh-llm-health { margin-left: auto; }
635
+
636
  /* A5 — Persona drift panel + recommendation history panel */
637
  .collapsible {
638
  border: 1px solid var(--border);
 
818
  may live inside this section: the h2 header (with snapshot timestamp
819
  span), the refresh button, and the chains container that holds the
820
  two generated tables (Currently In Use + All Eligible Models). -->
821
+ <div class="llm-health-header-row">
822
+ <h2>LLM Health</h2>
823
+ <span class="llm-refresh-meta">
824
+ <span>Last refresh: <span id="llm-last-refresh-ago">--</span></span>
825
+ <span class="dot-sep">·</span>
826
+ <span>Next in: <span id="llm-next-refresh-in">--</span></span>
827
+ </span>
828
+ <button id="btn-refresh-llm-health">Refresh now</button>
829
+ </div>
830
  <div id="llm-health-chains" class="llm-simple-tables"></div>
831
  <div id="llm-health-footer" class="table-footer"></div>
832
  </section>
 
1439
  }
1440
 
1441
  function renderUpdatedLabel() {
1442
+ var ago = $('llm-last-refresh-ago');
1443
+ var next = $('llm-next-refresh-in');
1444
+ if (!ago || !next) return;
1445
+ // Only render the LLM-Chain refresh timing when that tab is active —
1446
+ // irrelevant on Profiles / Performance.
1447
  if (STATE.activeTab !== 'chain') {
1448
+ ago.textContent = '--';
1449
+ next.textContent = '--';
1450
  return;
1451
  }
1452
  var probeAt = probeTimestampMs();
1453
+ if (!probeAt) { ago.textContent = '--'; next.textContent = '--'; return; }
1454
  var now = Date.now();
1455
+ ago.textContent = formatMinSec(now - probeAt) + ' ago';
1456
+ // KI-258b never display "due now". If the auto-probe is overdue,
1457
+ // project forward to the NEXT interval boundary so the countdown
1458
+ // always shows a real minutes:seconds value the user can act on.
1459
+ var elapsed = now - probeAt;
1460
+ var intoNext = ((elapsed % PROBE_INTERVAL_MS) + PROBE_INTERVAL_MS) % PROBE_INTERVAL_MS;
1461
+ var untilNext = PROBE_INTERVAL_MS - intoNext;
1462
+ next.textContent = formatMinSec(untilNext);
1463
  }
1464
 
1465
  setInterval(renderUpdatedLabel, 1000);
 
1774
  return td;
1775
  }
1776
 
1777
+ // KI-202 (2026-05-15) — operator-facing Health cell renderer. Reads
1778
+ // the per-model snapshot built from `healthPayload.candidates` (a
1779
+ // {model -> snapshot} map populated by renderLlmSimpleTables before
1780
+ // each row). Returns a <td> element so the caller can append directly.
1781
+ //
1782
+ // Display contract (matches admin spec):
1783
+ // Live — green dot + "Live" (effective_status='healthy')
1784
+ // Off — [reason] — red dot + "Off — reason" (everything else)
1785
+ // Reasons come from backend `health_reason`; fallback "unknown" when
1786
+ // the field is absent. The full `last_error` (when present) is
1787
+ // surfaced via a tooltip on hover.
1788
+ function renderHealthCell(snap) {
1789
+ var td = createEl('td', { className: 'health-cell' });
1790
+ td.style.whiteSpace = 'nowrap';
1791
+ var dot = createEl('span', { className: 'health-dot' });
1792
+ dot.style.marginRight = '6px';
1793
+ if (!snap) {
1794
+ // No snapshot for this model — render "Off — unknown" rather
1795
+ // than blank so the operator sees the gap.
1796
+ dot.className = 'health-dot bad';
1797
+ td.appendChild(dot);
1798
+ td.appendChild(document.createTextNode('Off — unknown'));
1799
+ return td;
1800
+ }
1801
+ var eff = (snap.effective_status || snap.status || 'unknown').toLowerCase();
1802
+ var isLive = (eff === 'healthy');
1803
+ dot.className = 'health-dot ' + (isLive ? 'ok' : 'bad');
1804
+ td.appendChild(dot);
1805
+ if (isLive) {
1806
+ td.appendChild(document.createTextNode('Live'));
1807
+ } else {
1808
+ var reason = snap.health_reason;
1809
+ if (!reason) {
1810
+ if (eff === 'stale') reason = 'stale';
1811
+ else if (eff === 'unknown') reason = 'unknown';
1812
+ else reason = 'unknown';
1813
+ }
1814
+ td.appendChild(document.createTextNode('Off — ' + reason));
1815
+ }
1816
+ if (snap.last_error) {
1817
+ td.title = String(snap.last_error);
1818
+ }
1819
+ return td;
1820
+ }
1821
+
1822
+ // Build {model -> candidate snapshot} from healthPayload.candidates
1823
+ // so renderHealthCell() can look up by model id in O(1).
1824
+ function buildCandidatesMap(healthPayload) {
1825
+ var out = {};
1826
+ var cands = (healthPayload && healthPayload.candidates) || [];
1827
+ cands.forEach(function (c) {
1828
+ if (c && c.model) out[c.model] = c;
1829
+ });
1830
+ return out;
1831
+ }
1832
+
1833
  // KI-200 — accepts the full llm-health payload so we can read
1834
  // `gemini_available` (when backend exposes it). Caller signature
1835
  // updated below in renderLlmHealth().
 
1837
  var chains = (healthPayload && healthPayload.chains) || [];
1838
  var chainsByRole = {};
1839
  chains.forEach(function (c) { if (c && c.role) chainsByRole[c.role] = c; });
1840
+ // KI-202 — model -> candidate snapshot map, used by renderHealthCell().
1841
+ var candByModel = buildCandidatesMap(healthPayload);
1842
 
1843
  // ---- Table 1: Currently In Use ----
1844
  var liveBlock = createEl('div', { className: 'simple-table-block' });
 
1846
  var liveTable = createEl('table');
1847
  var liveThead = createEl('thead');
1848
  var liveHr = createEl('tr');
1849
+ // KI-202 Health column appended at end (5 cols total).
1850
+ ['Use', 'Current Model', 'Provider', 'Tier', 'Health'].forEach(function (h, i) {
1851
  var th = createEl('th', { text: h });
1852
+ if (i >= 2 && i <= 3) th.style.textAlign = 'center';
1853
  liveHr.appendChild(th);
1854
  });
1855
  liveThead.appendChild(liveHr);
 
1883
  tierCell.textContent = live.tier;
1884
  tr.appendChild(tierCell);
1885
 
1886
+ // KI-202 — HEALTH cell. Tier-0 (Gemini) currently has no
1887
+ // health snapshot in `candidates` (probe loop only covers NIM/
1888
+ // OpenRouter chain entries), so fall back to a "Live" pill when
1889
+ // gemini_available flag is set / defaulted true. For Tier-1 NIM
1890
+ // rows we look up by model id in the candidates map.
1891
+ var healthSnap = candByModel[live.model];
1892
+ if (!healthSnap && live.isTier0) {
1893
+ // Synthesize a "live" snapshot — Gemini availability is assumed
1894
+ // until backend exposes a per-Gemini health row.
1895
+ healthSnap = { effective_status: 'healthy' };
1896
+ }
1897
+ tr.appendChild(renderHealthCell(healthSnap));
1898
+
1899
  liveTbody.appendChild(tr);
1900
  });
1901
  liveTable.appendChild(liveTbody);
 
1908
  var availTable = createEl('table');
1909
  var availThead = createEl('thead');
1910
  var availHr = createEl('tr');
1911
+ // KI-202 Health column appended at end.
1912
+ var availCols = ['Model', 'Provider', 'Tier']
1913
+ .concat(SIMPLE_USE_ORDER.map(function (r) { return SIMPLE_USE_LABELS[r]; }))
1914
+ .concat(['Health']);
1915
  availCols.forEach(function (h, i) {
1916
  var th = createEl('th', { text: h });
1917
+ if (i >= 1 && i < availCols.length - 1) th.style.textAlign = 'center';
1918
  availHr.appendChild(th);
1919
  });
1920
  availThead.appendChild(availHr);
 
1944
  tr.appendChild(renderRoleLabelCell(entry.roles[role]));
1945
  });
1946
 
1947
+ // KI-202 — HEALTH cell. Same Tier-0 fallback as Table 1.
1948
+ var entryHealthSnap = candByModel[entry.model];
1949
+ if (!entryHealthSnap && entry.tier === 'T0') {
1950
+ entryHealthSnap = { effective_status: 'healthy' };
1951
+ }
1952
+ tr.appendChild(renderHealthCell(entryHealthSnap));
1953
+
1954
  availTbody.appendChild(tr);
1955
  });
1956
  availTable.appendChild(availTbody);
 
2076
 
2077
  function renderLlmHealth() {
2078
  var chainsHost = $('llm-health-chains');
 
2079
  if (!chainsHost) return;
2080
 
2081
+ // KI-208 — snapshot timestamp moved out of the header into the
2082
+ // consolidated "Last refresh / Next in" row driven by renderUpdatedLabel.
2083
  if (!STATE.llmHealth) {
2084
  clearChildren(chainsHost);
2085
  chainsHost.appendChild(createEl('div', { className: 'empty-state', text: 'Loading…' }));
2086
  clearChildren($('llm-health-candidates'));
2087
  clearChildren($('llm-health-recent'));
 
2088
  return;
2089
  }
2090
  if (STATE.llmHealth.__notDeployed) {
 
2092
  chainsHost.appendChild(buildPendingMessage('/api/admin/llm-health'));
2093
  clearChildren($('llm-health-candidates'));
2094
  clearChildren($('llm-health-recent'));
 
2095
  return;
2096
  }
2097
 
 
2101
 
2102
  renderLlmHealthCandidates(STATE.llmHealth.candidates || []);
2103
  renderLlmHealthRecent(STATE.llmHealth.recent_turns || []);
 
 
 
 
 
 
2104
  }
2105
 
2106
  // Auto-poll every 30s while the LLM Chain tab is the active tab.
 
2867
  }
2868
  if (!probeAt) probeAt = STATE.llmHealthLoadedAt;
2869
  setFooter('llm-health-footer', probeAt);
2870
+ // KI-208 FRESH/STALE header badge removed; the consolidated header
2871
+ // row's "Last refresh / Next in" timing conveys freshness instead.
 
 
 
 
 
 
 
 
 
 
 
 
 
2872
  }
2873
  setInterval(tickFooters, 1000);
2874
 
 
3074
  llmHealthBtn.onclick = function () {
3075
  var btn = this;
3076
  btn.disabled = true;
3077
+ btn.textContent = 'Refreshing…';
3078
  apiPost('/api/admin/probe', null)
3079
  .catch(function () { /* tolerate probe failure; still re-fetch cached state */ })
3080
  .then(function () { return fetchLlmHealth(); })
3081
+ .then(function () { renderLlmHealth(); toast('LLM health refreshed', 'success'); })
3082
  .catch(handleFetchErr)
3083
+ .then(function () { btn.disabled = false; btn.textContent = 'Refresh now'; });
3084
  };
3085
  }
3086
  }