Spaces:
Sleeping
feat(llm-chain): KI-087 — NIM-first election preference; non-NIM only when NIM pool empty
Browse filesUser architectural decision: NIM is the strategic free provider (ADR-019:
single-key, $0 cost, 110+ models, no daily token cap). Groq has a hard
100K tokens/day free-tier cap that we burned through today. OpenRouter
charges a real USD balance. Both should serve as **emergency fallback
only**, not as primary just because their probe latency is lower.
Pre-KI-087: election scored by `latency × success_rate`. Groq's 161ms
LPU TTFT consistently beat NIM's 500ms-1s, so every probe round elected
Groq as PRIMARY across all 3 chains. Result: every chat call hit Groq
first, ate Groq's daily TPD inside 50 turns, then started returning 429s.
KI-087: election now prefers ANY eligible NIM candidate over ALL non-NIM
candidates. Within the NIM pool, the standard latency × success_rate
score still picks the fastest healthy NIM model. Only when the NIM pool
is empty (every NIM model is down, degraded, or quota-exhausted via
KI-085 credit gating) does election fall through to Groq/OpenRouter as
primary.
BACKUP rule is unchanged in spirit: cross-provider against PRIMARY. So
when PRIMARY is NIM (typical), BACKUP is the best non-NIM candidate
(Groq / OpenRouter / Groq Llama-3.3) — the cross-provider safety net
survives KI-087.
Expected production impact:
- Brain success rate climbs from 30% → 80-95%. NIM under modest load
(post-KI-084 probe cuts) consistently answers in 4-5s, comfortably
under the 12s elected-call timeout.
- Groq daily TPD stops getting burned by election (probes no longer
use Groq except when NIM is wholly unavailable). Groq stays
viable as a TRUE emergency fallback for an actual NIM outage.
- OpenRouter USD balance preserved — no spurious paid calls when
NIM has free capacity.
Together with KI-080 (sticky election), KI-084 (probe overhead cut +
429 demote), KI-085 (proactive credit gating), KI-087 is the LLM-chain
architecture's final shape.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- backend/admin.py +247 -0
- backend/llm_health.py +43 -10
- frontend/public/admin/llm-control.html +462 -1
|
@@ -584,3 +584,250 @@ async def admin_performance(
|
|
| 584 |
"usage_24h": _read_usage_24h(),
|
| 585 |
"snapshot_ts": datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"),
|
| 586 |
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 584 |
"usage_24h": _read_usage_24h(),
|
| 585 |
"snapshot_ts": datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"),
|
| 586 |
}
|
| 587 |
+
|
| 588 |
+
|
| 589 |
+
# ---------------------------------------------------------------------------
|
| 590 |
+
# /api/admin/llm-health — KI-086 LLM Health & Credits snapshot
|
| 591 |
+
#
|
| 592 |
+
# Surfaces KI-080..KI-085 telemetry on the existing admin "LLM Chain" tab so
|
| 593 |
+
# the operator can see at a glance:
|
| 594 |
+
# - per-chain elected PRIMARY + BACKUP (KI-080)
|
| 595 |
+
# - each candidate's latest probe latency + success rate (KI-080)
|
| 596 |
+
# - credits remaining + unit + reset deadline (KI-085)
|
| 597 |
+
# - degraded-until window when a candidate was demoted on 429 (KI-084)
|
| 598 |
+
# - per-turn served-model distribution from the last N llm_usage.jsonl rows
|
| 599 |
+
#
|
| 600 |
+
# Response shape — three top-level keys (chains / candidates / recent_turns) +
|
| 601 |
+
# a snapshot_ts. All durations on the wire are seconds-from-now (relative,
|
| 602 |
+
# never absolute monotonic) so the frontend doesn't need to know the server's
|
| 603 |
+
# monotonic clock origin.
|
| 604 |
+
# ---------------------------------------------------------------------------
|
| 605 |
+
|
| 606 |
+
|
| 607 |
+
# Map of the on-wire chain role name → human-friendly label. The roles
|
| 608 |
+
# themselves match llm_health.get_primary() input strings exactly.
|
| 609 |
+
_LLM_HEALTH_CHAIN_ROLES = ("brain", "fast_brain", "judge")
|
| 610 |
+
|
| 611 |
+
|
| 612 |
+
def _chain_names_map() -> dict[str, list[str]]:
|
| 613 |
+
"""Live (post-admin-override) chain config — read off the module so admin
|
| 614 |
+
reorders applied earlier in the same process are reflected immediately."""
|
| 615 |
+
from backend.providers import nvidia_nim_llm as nim
|
| 616 |
+
return {
|
| 617 |
+
"brain": list(getattr(nim, "BRAIN_CHAIN", [])),
|
| 618 |
+
"fast_brain": list(getattr(nim, "FAST_BRAIN_CHAIN", [])),
|
| 619 |
+
"judge": list(getattr(nim, "JUDGE_CHAIN", [])),
|
| 620 |
+
}
|
| 621 |
+
|
| 622 |
+
|
| 623 |
+
def _seconds_until_monotonic(deadline: Optional[float]) -> Optional[float]:
|
| 624 |
+
"""Convert a monotonic-time deadline (as stamped in ModelHealth) into a
|
| 625 |
+
seconds-from-now value the frontend can render as an ETA. Returns None
|
| 626 |
+
when the deadline is missing OR already in the past; the caller decides
|
| 627 |
+
how to render `None` vs `0`."""
|
| 628 |
+
if deadline is None:
|
| 629 |
+
return None
|
| 630 |
+
import time as _time
|
| 631 |
+
rem = deadline - _time.monotonic()
|
| 632 |
+
if rem <= 0:
|
| 633 |
+
return 0.0
|
| 634 |
+
return round(rem, 1)
|
| 635 |
+
|
| 636 |
+
|
| 637 |
+
def _probe_age_seconds(iso_ts: Optional[str]) -> Optional[float]:
|
| 638 |
+
"""Wall-clock seconds since a probe iso8601 timestamp. Cheap wrapper
|
| 639 |
+
around llm_health._iso_age_seconds for the wire payload."""
|
| 640 |
+
age = llm_health._iso_age_seconds(iso_ts)
|
| 641 |
+
if age is None:
|
| 642 |
+
return None
|
| 643 |
+
return max(0.0, round(age, 1))
|
| 644 |
+
|
| 645 |
+
|
| 646 |
+
def _success_rate_for(h) -> Optional[float]:
|
| 647 |
+
"""Last-N probes success rate as a float fraction. Returns None when no
|
| 648 |
+
probe history yet."""
|
| 649 |
+
hist = getattr(h, "probe_history", None) or []
|
| 650 |
+
if not hist:
|
| 651 |
+
return None
|
| 652 |
+
hits = sum(1 for r in hist if r.get("ok"))
|
| 653 |
+
return round(hits / len(hist), 4)
|
| 654 |
+
|
| 655 |
+
|
| 656 |
+
def _candidate_snapshot(model: str, health, chain_membership: list[str],
|
| 657 |
+
now_mono: float) -> dict:
|
| 658 |
+
"""Per-candidate row for Section B. Always returns a dict — even when
|
| 659 |
+
the model has never been probed yet (status='unknown', everything else
|
| 660 |
+
None) — so the frontend table doesn't have to handle missing rows."""
|
| 661 |
+
if health is None:
|
| 662 |
+
return {
|
| 663 |
+
"model": model,
|
| 664 |
+
"provider": llm_health.provider_of(model),
|
| 665 |
+
"chain_membership": chain_membership,
|
| 666 |
+
"status": "unknown",
|
| 667 |
+
"latency_ms": None,
|
| 668 |
+
"success_rate": None,
|
| 669 |
+
"probe_age_seconds": None,
|
| 670 |
+
"last_error": None,
|
| 671 |
+
"credits_remaining": None,
|
| 672 |
+
"credits_unit": None,
|
| 673 |
+
"credits_low_water": None,
|
| 674 |
+
"credits_reset_in_seconds": None,
|
| 675 |
+
"degraded_for_seconds": None,
|
| 676 |
+
}
|
| 677 |
+
deg_until = getattr(health, "degraded_until_monotonic", 0.0) or 0.0
|
| 678 |
+
deg_for = None
|
| 679 |
+
if deg_until and deg_until > now_mono:
|
| 680 |
+
deg_for = round(deg_until - now_mono, 1)
|
| 681 |
+
return {
|
| 682 |
+
"model": model,
|
| 683 |
+
"provider": llm_health.provider_of(model),
|
| 684 |
+
"chain_membership": chain_membership,
|
| 685 |
+
"status": health.status,
|
| 686 |
+
"latency_ms": health.latency_ms,
|
| 687 |
+
"success_rate": _success_rate_for(health),
|
| 688 |
+
"probe_age_seconds": _probe_age_seconds(health.tested_at),
|
| 689 |
+
"last_error": health.last_error,
|
| 690 |
+
"credits_remaining": health.credits_remaining,
|
| 691 |
+
"credits_unit": health.credits_unit,
|
| 692 |
+
"credits_low_water": health.credits_low_water,
|
| 693 |
+
"credits_reset_in_seconds": _seconds_until_monotonic(health.credits_reset_at),
|
| 694 |
+
"degraded_for_seconds": deg_for,
|
| 695 |
+
}
|
| 696 |
+
|
| 697 |
+
|
| 698 |
+
def _chain_summary(role: str, chains: dict[str, list[str]],
|
| 699 |
+
state: dict, now_mono: float) -> dict:
|
| 700 |
+
"""Per-chain block for Section A. Includes elected primary/backup +
|
| 701 |
+
`all_credit_exhausted` so the frontend can render a banner when every
|
| 702 |
+
candidate in the chain is gated out by credits/quota."""
|
| 703 |
+
chain = chains.get(role) or []
|
| 704 |
+
primary = llm_health.get_primary(role)
|
| 705 |
+
backup = llm_health.get_backup(role)
|
| 706 |
+
|
| 707 |
+
# all_credit_exhausted: every chain member with a non-None credit signal
|
| 708 |
+
# is at-or-below its low-water mark. Chains with no signal at all are
|
| 709 |
+
# NOT flagged exhausted (cold-start should be permissive — election will
|
| 710 |
+
# try them and surface a real failure if any).
|
| 711 |
+
any_signal = False
|
| 712 |
+
all_exhausted = True
|
| 713 |
+
for m in chain:
|
| 714 |
+
h = state.get(m)
|
| 715 |
+
if h is None or h.credits_remaining is None:
|
| 716 |
+
continue
|
| 717 |
+
any_signal = True
|
| 718 |
+
if h.credits_remaining > (h.credits_low_water or 0.0):
|
| 719 |
+
all_exhausted = False
|
| 720 |
+
break
|
| 721 |
+
chain_credit_exhausted = bool(any_signal and all_exhausted)
|
| 722 |
+
|
| 723 |
+
return {
|
| 724 |
+
"role": role,
|
| 725 |
+
"chain": chain,
|
| 726 |
+
"elected_primary": primary,
|
| 727 |
+
"elected_backup": backup,
|
| 728 |
+
"primary_snapshot": _candidate_snapshot(
|
| 729 |
+
primary, state.get(primary), [role], now_mono,
|
| 730 |
+
) if primary else None,
|
| 731 |
+
"backup_snapshot": _candidate_snapshot(
|
| 732 |
+
backup, state.get(backup), [role], now_mono,
|
| 733 |
+
) if backup else None,
|
| 734 |
+
"chain_credit_exhausted": chain_credit_exhausted,
|
| 735 |
+
}
|
| 736 |
+
|
| 737 |
+
|
| 738 |
+
def _recent_turns(n: int = 20) -> list[dict]:
|
| 739 |
+
"""Section C — last N completed turns from 40-data/llm_usage.jsonl,
|
| 740 |
+
most-recent-first. We keep the row shape close to the raw log entries
|
| 741 |
+
so the frontend can render new fields if the producer adds them.
|
| 742 |
+
|
| 743 |
+
Fields surfaced (all optional — older rows pre-KI-080 won't carry
|
| 744 |
+
elected_primary/backup):
|
| 745 |
+
ts / role / chain_primary / elected_primary / elected_backup /
|
| 746 |
+
served_model / latency_ms / success / fallback_reason
|
| 747 |
+
"""
|
| 748 |
+
usage_path = _REPO_ROOT / "40-data" / "llm_usage.jsonl"
|
| 749 |
+
# Re-use the existing tail-reader; cap at N so we don't pay for the
|
| 750 |
+
# full 1000-row tail when the panel only renders 20.
|
| 751 |
+
rows = _tail_jsonl(usage_path, max(n, 20))
|
| 752 |
+
if not rows:
|
| 753 |
+
return []
|
| 754 |
+
out: list[dict] = []
|
| 755 |
+
# tail_jsonl returns oldest-first; reverse to newest-first.
|
| 756 |
+
for r in reversed(rows[-n:]):
|
| 757 |
+
out.append({
|
| 758 |
+
"ts": r.get("ts"),
|
| 759 |
+
"role": r.get("role"),
|
| 760 |
+
"chain_primary": r.get("chain_primary"),
|
| 761 |
+
"elected_primary": r.get("elected_primary"),
|
| 762 |
+
"elected_backup": r.get("elected_backup"),
|
| 763 |
+
"served_model": r.get("served_model"),
|
| 764 |
+
"latency_ms": r.get("latency_ms"),
|
| 765 |
+
"success": r.get("success"),
|
| 766 |
+
"fallback_reason": r.get("fallback_reason"),
|
| 767 |
+
})
|
| 768 |
+
return out
|
| 769 |
+
|
| 770 |
+
|
| 771 |
+
@router.get("/api/admin/llm-health")
|
| 772 |
+
async def admin_llm_health(
|
| 773 |
+
request: Request,
|
| 774 |
+
x_admin_password: Optional[str] = Header(default=None, alias="X-Admin-Password"),
|
| 775 |
+
):
|
| 776 |
+
"""KI-086 — composite LLM health + credits snapshot for the LLM Chain tab.
|
| 777 |
+
|
| 778 |
+
Returns three keys:
|
| 779 |
+
chains — Section A: one entry per chain (brain/fast_brain/judge)
|
| 780 |
+
with elected primary + backup + their snapshots +
|
| 781 |
+
chain_credit_exhausted banner flag.
|
| 782 |
+
candidates — Section B: one row per known candidate across all chains,
|
| 783 |
+
with chain_membership listing every chain it appears in,
|
| 784 |
+
latency / success / credits / degraded-window state.
|
| 785 |
+
recent_turns — Section C: last 20 served turns from llm_usage.jsonl.
|
| 786 |
+
Plus snapshot_ts so the UI can show "updated <wallclock>".
|
| 787 |
+
"""
|
| 788 |
+
_check_admin(request, x_admin_password)
|
| 789 |
+
|
| 790 |
+
import time as _time
|
| 791 |
+
now_mono = _time.monotonic()
|
| 792 |
+
|
| 793 |
+
chains = _chain_names_map()
|
| 794 |
+
state = llm_health.load() # {model -> ModelHealth}
|
| 795 |
+
|
| 796 |
+
# Section A: per-chain election + credit banner.
|
| 797 |
+
chains_block = [
|
| 798 |
+
_chain_summary(role, chains, state, now_mono)
|
| 799 |
+
for role in _LLM_HEALTH_CHAIN_ROLES
|
| 800 |
+
]
|
| 801 |
+
|
| 802 |
+
# Section B: every known candidate × chain membership (deduped).
|
| 803 |
+
# Membership is the list of chain roles a model belongs to.
|
| 804 |
+
membership: dict[str, list[str]] = {}
|
| 805 |
+
for role, models in chains.items():
|
| 806 |
+
for m in models:
|
| 807 |
+
membership.setdefault(m, []).append(role)
|
| 808 |
+
# Also include any candidate present in state but not currently in any
|
| 809 |
+
# chain (admin reorder may have just removed it) so the operator can
|
| 810 |
+
# still see its last probe + credits.
|
| 811 |
+
for m in state.keys():
|
| 812 |
+
membership.setdefault(m, [])
|
| 813 |
+
|
| 814 |
+
candidates_block = [
|
| 815 |
+
_candidate_snapshot(m, state.get(m), membership[m], now_mono)
|
| 816 |
+
for m in membership.keys()
|
| 817 |
+
]
|
| 818 |
+
# Sort: degraded first (red), then non-healthy (amber), healthy last,
|
| 819 |
+
# then by model name. Helps the operator see problems at the top.
|
| 820 |
+
_status_rank = {"down": 0, "degraded": 1, "unknown": 2, "healthy": 3}
|
| 821 |
+
candidates_block.sort(
|
| 822 |
+
key=lambda c: (_status_rank.get(c["status"], 9), c["model"])
|
| 823 |
+
)
|
| 824 |
+
|
| 825 |
+
# Section C: last 20 turns.
|
| 826 |
+
recent = _recent_turns(20)
|
| 827 |
+
|
| 828 |
+
return {
|
| 829 |
+
"chains": chains_block,
|
| 830 |
+
"candidates": candidates_block,
|
| 831 |
+
"recent_turns": recent,
|
| 832 |
+
"snapshot_ts": datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"),
|
| 833 |
+
}
|
|
@@ -421,29 +421,62 @@ def _ranked_candidates(chain_name: str) -> list[ModelHealth]:
|
|
| 421 |
|
| 422 |
def get_primary(chain_name: str) -> Optional[str]:
|
| 423 |
"""Top-scoring election-eligible candidate, or None when no probe data
|
| 424 |
-
is fresh enough.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 425 |
ranked = _ranked_candidates(chain_name)
|
| 426 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 427 |
|
| 428 |
|
| 429 |
def get_backup(chain_name: str) -> Optional[str]:
|
| 430 |
"""Second-best election candidate. Prefers a DIFFERENT provider from
|
| 431 |
-
primary
|
| 432 |
-
|
| 433 |
-
|
| 434 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 435 |
ranked = _ranked_candidates(chain_name)
|
| 436 |
if len(ranked) < 2:
|
| 437 |
# No usable backup. Either zero candidates or only one (in which
|
| 438 |
# case the cold-start path in NimChainLLM falls back to chain[1]).
|
| 439 |
return None
|
| 440 |
-
|
| 441 |
-
|
| 442 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 443 |
if provider_of(h.model) != primary_provider:
|
| 444 |
return h.model
|
| 445 |
# No cross-provider candidate — accept same-provider next-best.
|
| 446 |
-
|
|
|
|
|
|
|
|
|
|
| 447 |
|
| 448 |
|
| 449 |
def _is_rate_limit_error(error_class: str) -> bool:
|
|
|
|
| 421 |
|
| 422 |
def get_primary(chain_name: str) -> Optional[str]:
|
| 423 |
"""Top-scoring election-eligible candidate, or None when no probe data
|
| 424 |
+
is fresh enough.
|
| 425 |
+
|
| 426 |
+
KI-087 (2026-05-15) — **NIM-first preference.** NIM is the strategic
|
| 427 |
+
free provider (ADR-019, single-key, $0 cost, 110+ models, no daily
|
| 428 |
+
cap). Groq has a hard 100K tokens/day free-tier cap and OpenRouter
|
| 429 |
+
charges a real USD balance. Both should serve as EMERGENCY fallback
|
| 430 |
+
only, not as primary just because their probe latency is lower.
|
| 431 |
+
Election therefore prefers ANY eligible NIM candidate over ALL
|
| 432 |
+
non-NIM candidates. Only when the NIM pool is empty (every NIM
|
| 433 |
+
model is down, throttled, or out of credits) does election fall
|
| 434 |
+
through to Groq/OpenRouter as primary. Within the NIM pool, the
|
| 435 |
+
standard latency × success_rate score still picks the fastest
|
| 436 |
+
healthy NIM model.
|
| 437 |
+
"""
|
| 438 |
ranked = _ranked_candidates(chain_name)
|
| 439 |
+
if not ranked:
|
| 440 |
+
return None
|
| 441 |
+
# KI-087: NIM-first within the eligible set.
|
| 442 |
+
nim_pool = [h for h in ranked if provider_of(h.model) == "nim"]
|
| 443 |
+
if nim_pool:
|
| 444 |
+
return nim_pool[0].model
|
| 445 |
+
# No eligible NIM candidate — fall through to cross-provider primary.
|
| 446 |
+
return ranked[0].model
|
| 447 |
|
| 448 |
|
| 449 |
def get_backup(chain_name: str) -> Optional[str]:
|
| 450 |
"""Second-best election candidate. Prefers a DIFFERENT provider from
|
| 451 |
+
primary so a single provider's regional outage can't take out both.
|
| 452 |
+
|
| 453 |
+
KI-087 (2026-05-15) — when PRIMARY is NIM (the typical case under
|
| 454 |
+
NIM-first preference), BACKUP is the best NON-NIM candidate
|
| 455 |
+
(Groq / OpenRouter) — kept as the cross-provider safety net. When
|
| 456 |
+
NIM is wholly unavailable and PRIMARY is non-NIM, BACKUP is the
|
| 457 |
+
next-best non-NIM candidate or, failing that, the next-best
|
| 458 |
+
same-provider candidate.
|
| 459 |
+
"""
|
| 460 |
ranked = _ranked_candidates(chain_name)
|
| 461 |
if len(ranked) < 2:
|
| 462 |
# No usable backup. Either zero candidates or only one (in which
|
| 463 |
# case the cold-start path in NimChainLLM falls back to chain[1]).
|
| 464 |
return None
|
| 465 |
+
primary = get_primary(chain_name)
|
| 466 |
+
if primary is None:
|
| 467 |
+
return None
|
| 468 |
+
primary_provider = provider_of(primary)
|
| 469 |
+
# Prefer cross-provider against the PRIMARY's provider.
|
| 470 |
+
for h in ranked:
|
| 471 |
+
if h.model == primary:
|
| 472 |
+
continue
|
| 473 |
if provider_of(h.model) != primary_provider:
|
| 474 |
return h.model
|
| 475 |
# No cross-provider candidate — accept same-provider next-best.
|
| 476 |
+
for h in ranked:
|
| 477 |
+
if h.model != primary:
|
| 478 |
+
return h.model
|
| 479 |
+
return None
|
| 480 |
|
| 481 |
|
| 482 |
def _is_rate_limit_error(error_class: str) -> bool:
|
|
@@ -347,6 +347,114 @@
|
|
| 347 |
.chain-table .icon-btn { padding: 2px 7px; font-size: 12px; line-height: 1.2; }
|
| 348 |
.chain-table tbody tr.is-primary td { background: rgba(63, 185, 80, 0.05); }
|
| 349 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 350 |
.unverified-flag {
|
| 351 |
display: inline-block;
|
| 352 |
margin-left: 6px;
|
|
@@ -554,6 +662,24 @@
|
|
| 554 |
|
| 555 |
<!-- Tab 3: LLM Chain (existing functionality, preserved) -->
|
| 556 |
<section id="tab-chain" class="tabpane" role="tabpanel">
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 557 |
<!-- Health -->
|
| 558 |
<div class="card">
|
| 559 |
<div class="row-between" style="margin-bottom: 12px;">
|
|
@@ -597,6 +723,10 @@
|
|
| 597 |
health: null, // { models: [...], counters: {...} }
|
| 598 |
chains: null, // { brain: [...], fast_brain: [...], judge: [...] }
|
| 599 |
usage: null, // { brain: {...}, fast_brain: {...}, judge: {...} }
|
|
|
|
|
|
|
|
|
|
|
|
|
| 600 |
lastUpdatedAt: null,
|
| 601 |
chainLoaded: false,
|
| 602 |
// Profiles tab
|
|
@@ -1222,15 +1352,323 @@
|
|
| 1222 |
}
|
| 1223 |
|
| 1224 |
function refreshChain() {
|
| 1225 |
-
return Promise.all([fetchHealth(), fetchChains(), fetchUsage()]).then(function () {
|
| 1226 |
renderCounters();
|
| 1227 |
renderHealthTable();
|
| 1228 |
renderChains();
|
|
|
|
| 1229 |
setLastUpdated();
|
| 1230 |
STATE.chainLoaded = true;
|
| 1231 |
});
|
| 1232 |
}
|
| 1233 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1234 |
function handleFetchErr(err) {
|
| 1235 |
if (err && err.code === 404) {
|
| 1236 |
try { localStorage.removeItem(STORAGE_KEY); } catch (_) {}
|
|
@@ -1753,12 +2191,19 @@
|
|
| 1753 |
// Lazy-load this tab's data
|
| 1754 |
if (tab === 'profiles') {
|
| 1755 |
loadProfiles(false);
|
|
|
|
| 1756 |
} else if (tab === 'performance') {
|
| 1757 |
loadPerformance(false);
|
|
|
|
| 1758 |
} else if (tab === 'chain') {
|
| 1759 |
if (!STATE.chainLoaded) {
|
| 1760 |
refreshChain().catch(handleFetchErr);
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1761 |
}
|
|
|
|
| 1762 |
}
|
| 1763 |
}
|
| 1764 |
|
|
@@ -1828,11 +2273,13 @@
|
|
| 1828 |
STATE.health = null;
|
| 1829 |
STATE.chains = null;
|
| 1830 |
STATE.usage = null;
|
|
|
|
| 1831 |
STATE.profiles = null;
|
| 1832 |
STATE.profilesLoaded = false;
|
| 1833 |
STATE.performance = null;
|
| 1834 |
STATE.performanceLoaded = false;
|
| 1835 |
STATE.chainLoaded = false;
|
|
|
|
| 1836 |
try { localStorage.removeItem(STORAGE_KEY); } catch (_) {}
|
| 1837 |
showGate();
|
| 1838 |
toast('Locked', 'info');
|
|
@@ -1885,6 +2332,20 @@
|
|
| 1885 |
.catch(handleFetchErr)
|
| 1886 |
.then(function () { btn.disabled = false; btn.textContent = 'Force fresh probe (slow)'; });
|
| 1887 |
};
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1888 |
}
|
| 1889 |
|
| 1890 |
// ----- Boot -----
|
|
|
|
| 347 |
.chain-table .icon-btn { padding: 2px 7px; font-size: 12px; line-height: 1.2; }
|
| 348 |
.chain-table tbody tr.is-primary td { background: rgba(63, 185, 80, 0.05); }
|
| 349 |
|
| 350 |
+
/* KI-086: LLM Health & Credits section */
|
| 351 |
+
.llm-health-chains {
|
| 352 |
+
display: grid;
|
| 353 |
+
grid-template-columns: repeat(auto-fit, minmax(360px, 1fr));
|
| 354 |
+
gap: 12px;
|
| 355 |
+
}
|
| 356 |
+
.llm-health-chain {
|
| 357 |
+
padding: 12px;
|
| 358 |
+
background: #0b0d12;
|
| 359 |
+
border: 1px solid var(--border);
|
| 360 |
+
border-radius: 8px;
|
| 361 |
+
}
|
| 362 |
+
.llm-health-chain .chain-head {
|
| 363 |
+
display: flex; align-items: center; gap: 8px;
|
| 364 |
+
margin-bottom: 10px;
|
| 365 |
+
}
|
| 366 |
+
.llm-health-chain .chain-head .role-badge { margin-right: 4px; }
|
| 367 |
+
.llm-health-chain .role-block {
|
| 368 |
+
margin-bottom: 8px;
|
| 369 |
+
padding-bottom: 8px;
|
| 370 |
+
border-bottom: 1px dashed var(--border);
|
| 371 |
+
}
|
| 372 |
+
.llm-health-chain .role-block:last-of-type { border-bottom: none; }
|
| 373 |
+
.llm-health-chain .role-block .role-label {
|
| 374 |
+
font-size: 10px;
|
| 375 |
+
text-transform: uppercase;
|
| 376 |
+
letter-spacing: 0.08em;
|
| 377 |
+
color: var(--muted);
|
| 378 |
+
margin-bottom: 4px;
|
| 379 |
+
}
|
| 380 |
+
.llm-health-chain .role-block .model-name {
|
| 381 |
+
font-family: ui-monospace, SFMono-Regular, "SF Mono", Menlo, Monaco, monospace;
|
| 382 |
+
font-size: 12px;
|
| 383 |
+
word-break: break-all;
|
| 384 |
+
}
|
| 385 |
+
.llm-health-chain .role-block .stats-line {
|
| 386 |
+
margin-top: 4px;
|
| 387 |
+
font-size: 11px;
|
| 388 |
+
color: var(--muted);
|
| 389 |
+
}
|
| 390 |
+
.llm-health-chain .role-block .stats-line .sep {
|
| 391 |
+
margin: 0 6px;
|
| 392 |
+
color: #3a4250;
|
| 393 |
+
}
|
| 394 |
+
.llm-health-chain .credit-banner {
|
| 395 |
+
margin-top: 8px;
|
| 396 |
+
padding: 8px 10px;
|
| 397 |
+
background: rgba(248, 81, 73, 0.12);
|
| 398 |
+
border: 1px solid rgba(248, 81, 73, 0.45);
|
| 399 |
+
border-radius: 6px;
|
| 400 |
+
font-size: 12px;
|
| 401 |
+
color: var(--red);
|
| 402 |
+
}
|
| 403 |
+
.llm-health-chain .none-elected {
|
| 404 |
+
color: var(--yellow);
|
| 405 |
+
font-style: italic;
|
| 406 |
+
font-size: 12px;
|
| 407 |
+
}
|
| 408 |
+
|
| 409 |
+
/* Status badges — three colors: ✓ green / ⚠ amber / ✗ red */
|
| 410 |
+
.health-badge {
|
| 411 |
+
display: inline-block;
|
| 412 |
+
padding: 2px 8px;
|
| 413 |
+
border-radius: 4px;
|
| 414 |
+
font-size: 11px;
|
| 415 |
+
font-weight: 600;
|
| 416 |
+
letter-spacing: 0.04em;
|
| 417 |
+
vertical-align: middle;
|
| 418 |
+
font-family: ui-monospace, SFMono-Regular, "SF Mono", Menlo, Monaco, monospace;
|
| 419 |
+
}
|
| 420 |
+
.health-badge.ok { background: rgba(63, 185, 80, 0.15); color: var(--green); border: 1px solid rgba(63, 185, 80, 0.4); }
|
| 421 |
+
.health-badge.warn { background: rgba(210, 153, 34, 0.15); color: var(--yellow); border: 1px solid rgba(210, 153, 34, 0.4); }
|
| 422 |
+
.health-badge.bad { background: rgba(248, 81, 73, 0.15); color: var(--red); border: 1px solid rgba(248, 81, 73, 0.4); }
|
| 423 |
+
.health-badge.unknown { background: rgba(139, 149, 164, 0.15);color: var(--muted); border: 1px solid rgba(139, 149, 164, 0.4); }
|
| 424 |
+
|
| 425 |
+
/* Candidate health grid */
|
| 426 |
+
.llm-health-candidates {
|
| 427 |
+
overflow-x: auto;
|
| 428 |
+
}
|
| 429 |
+
.llm-health-candidates table { font-size: 12px; }
|
| 430 |
+
.llm-health-candidates td.mono,
|
| 431 |
+
.llm-health-candidates td.model-cell {
|
| 432 |
+
font-family: ui-monospace, SFMono-Regular, "SF Mono", Menlo, Monaco, monospace;
|
| 433 |
+
word-break: break-all;
|
| 434 |
+
max-width: 280px;
|
| 435 |
+
}
|
| 436 |
+
.llm-health-candidates td.num { text-align: right; font-variant-numeric: tabular-nums; }
|
| 437 |
+
.llm-health-candidates td.chain-mem {
|
| 438 |
+
font-size: 11px;
|
| 439 |
+
color: var(--muted);
|
| 440 |
+
}
|
| 441 |
+
.llm-health-candidates td.credits {
|
| 442 |
+
font-family: ui-monospace, SFMono-Regular, "SF Mono", Menlo, Monaco, monospace;
|
| 443 |
+
font-size: 11px;
|
| 444 |
+
}
|
| 445 |
+
|
| 446 |
+
/* Recent turns table */
|
| 447 |
+
.llm-health-recent table { font-size: 12px; }
|
| 448 |
+
.llm-health-recent td { vertical-align: middle; }
|
| 449 |
+
.llm-health-recent td.mono {
|
| 450 |
+
font-family: ui-monospace, SFMono-Regular, "SF Mono", Menlo, Monaco, monospace;
|
| 451 |
+
font-size: 11px;
|
| 452 |
+
word-break: break-all;
|
| 453 |
+
max-width: 240px;
|
| 454 |
+
}
|
| 455 |
+
.llm-health-recent td.num { text-align: right; font-variant-numeric: tabular-nums; }
|
| 456 |
+
.llm-health-recent .served-diff { color: var(--yellow); }
|
| 457 |
+
|
| 458 |
.unverified-flag {
|
| 459 |
display: inline-block;
|
| 460 |
margin-left: 6px;
|
|
|
|
| 662 |
|
| 663 |
<!-- Tab 3: LLM Chain (existing functionality, preserved) -->
|
| 664 |
<section id="tab-chain" class="tabpane" role="tabpanel">
|
| 665 |
+
<!-- KI-086: LLM Health & Credits snapshot (KI-080..KI-085 telemetry) -->
|
| 666 |
+
<div class="card" id="llm-health-card">
|
| 667 |
+
<div class="row-between" style="margin-bottom: 12px;">
|
| 668 |
+
<h2>LLM health & credits <span class="small muted" id="llm-health-snapshot-ts"></span></h2>
|
| 669 |
+
<div class="actions">
|
| 670 |
+
<button id="btn-refresh-llm-health">Refresh</button>
|
| 671 |
+
</div>
|
| 672 |
+
</div>
|
| 673 |
+
<!-- Section A: per-chain election cards -->
|
| 674 |
+
<div id="llm-health-chains" class="llm-health-chains"></div>
|
| 675 |
+
<!-- Section B: candidate health grid -->
|
| 676 |
+
<h3 style="margin-top: 18px;">Candidate health grid</h3>
|
| 677 |
+
<div id="llm-health-candidates" class="llm-health-candidates"></div>
|
| 678 |
+
<!-- Section C: recent turns -->
|
| 679 |
+
<h3 style="margin-top: 18px;">Recent turns (last 20)</h3>
|
| 680 |
+
<div id="llm-health-recent" class="llm-health-recent"></div>
|
| 681 |
+
</div>
|
| 682 |
+
|
| 683 |
<!-- Health -->
|
| 684 |
<div class="card">
|
| 685 |
<div class="row-between" style="margin-bottom: 12px;">
|
|
|
|
| 723 |
health: null, // { models: [...], counters: {...} }
|
| 724 |
chains: null, // { brain: [...], fast_brain: [...], judge: [...] }
|
| 725 |
usage: null, // { brain: {...}, fast_brain: {...}, judge: {...} }
|
| 726 |
+
// KI-086 — LLM Health & Credits snapshot
|
| 727 |
+
// Shape: { chains: [...], candidates: [...], recent_turns: [...], snapshot_ts }
|
| 728 |
+
llmHealth: null,
|
| 729 |
+
llmHealthPollTimer: null,
|
| 730 |
lastUpdatedAt: null,
|
| 731 |
chainLoaded: false,
|
| 732 |
// Profiles tab
|
|
|
|
| 1352 |
}
|
| 1353 |
|
| 1354 |
function refreshChain() {
|
| 1355 |
+
return Promise.all([fetchHealth(), fetchChains(), fetchUsage(), fetchLlmHealth()]).then(function () {
|
| 1356 |
renderCounters();
|
| 1357 |
renderHealthTable();
|
| 1358 |
renderChains();
|
| 1359 |
+
renderLlmHealth();
|
| 1360 |
setLastUpdated();
|
| 1361 |
STATE.chainLoaded = true;
|
| 1362 |
});
|
| 1363 |
}
|
| 1364 |
|
| 1365 |
+
// ----- KI-086: LLM Health & Credits -----
|
| 1366 |
+
// Fetcher — tolerant of 404 (endpoint not deployed yet); leaves
|
| 1367 |
+
// STATE.llmHealth null so renderer falls back to pending-message.
|
| 1368 |
+
function fetchLlmHealth() {
|
| 1369 |
+
return apiGetTolerant('/api/admin/llm-health').then(function (data) {
|
| 1370 |
+
STATE.llmHealth = data;
|
| 1371 |
+
}).catch(function (err) {
|
| 1372 |
+
if (err && err.notDeployed) {
|
| 1373 |
+
STATE.llmHealth = { __notDeployed: true };
|
| 1374 |
+
} else {
|
| 1375 |
+
STATE.llmHealth = null;
|
| 1376 |
+
}
|
| 1377 |
+
});
|
| 1378 |
+
}
|
| 1379 |
+
|
| 1380 |
+
function fmtCredits(remaining, unit, lowWater) {
|
| 1381 |
+
if (remaining == null || unit == null) return '—';
|
| 1382 |
+
var n = Number(remaining);
|
| 1383 |
+
if (isNaN(n)) return '—';
|
| 1384 |
+
var rounded;
|
| 1385 |
+
if (n >= 1000) rounded = (n / 1000).toFixed(1) + 'K';
|
| 1386 |
+
else if (n >= 1) rounded = n.toFixed(0);
|
| 1387 |
+
else rounded = n.toFixed(3);
|
| 1388 |
+
var unitLabel;
|
| 1389 |
+
switch (unit) {
|
| 1390 |
+
case 'tokens_day': unitLabel = 'tokens/day'; break;
|
| 1391 |
+
case 'tokens_min': unitLabel = 'tokens/min'; break;
|
| 1392 |
+
case 'usd_balance': unitLabel = 'USD'; break;
|
| 1393 |
+
case 'requests_min': unitLabel = 'req/min'; break;
|
| 1394 |
+
default: unitLabel = String(unit);
|
| 1395 |
+
}
|
| 1396 |
+
var s = rounded + ' ' + unitLabel;
|
| 1397 |
+
if (lowWater != null && Number(lowWater) > 0) {
|
| 1398 |
+
s += ' (water ' + Number(lowWater).toFixed(unit === 'usd_balance' ? 2 : 0) + ')';
|
| 1399 |
+
}
|
| 1400 |
+
return s;
|
| 1401 |
+
}
|
| 1402 |
+
|
| 1403 |
+
function fmtDurationShort(s) {
|
| 1404 |
+
if (s == null) return '—';
|
| 1405 |
+
var n = Number(s);
|
| 1406 |
+
if (!isFinite(n) || n <= 0) return '0s';
|
| 1407 |
+
if (n < 60) return Math.round(n) + 's';
|
| 1408 |
+
if (n < 3600) return Math.floor(n / 60) + 'm ' + Math.round(n % 60) + 's';
|
| 1409 |
+
var h = Math.floor(n / 3600);
|
| 1410 |
+
var rem = n - h * 3600;
|
| 1411 |
+
var m = Math.floor(rem / 60);
|
| 1412 |
+
return h + 'h ' + m + 'm';
|
| 1413 |
+
}
|
| 1414 |
+
|
| 1415 |
+
// Pick badge severity for the candidate state.
|
| 1416 |
+
// ✓ ok — healthy + no degraded window
|
| 1417 |
+
// ⚠ warn — degraded OR rate-limited (credits at/below water) OR credits exhausting OR degraded_for_seconds>0
|
| 1418 |
+
// ✗ bad — down OR no probe within 600s (status='unknown' AND probe age >0)
|
| 1419 |
+
function candidateBadge(row) {
|
| 1420 |
+
var st = (row.status || 'unknown').toLowerCase();
|
| 1421 |
+
var degFor = row.degraded_for_seconds;
|
| 1422 |
+
var credsRem = row.credits_remaining;
|
| 1423 |
+
var credsLow = row.credits_low_water;
|
| 1424 |
+
if (st === 'down') return { cls: 'bad', symbol: '✗', label: 'DOWN' };
|
| 1425 |
+
if (degFor != null && degFor > 0) {
|
| 1426 |
+
return { cls: 'warn', symbol: '⚠', label: 'DEGRADED ' + fmtDurationShort(degFor) };
|
| 1427 |
+
}
|
| 1428 |
+
if (credsRem != null && credsLow != null && Number(credsRem) <= Number(credsLow)) {
|
| 1429 |
+
return { cls: 'warn', symbol: '⚠', label: 'RATE-LIMITED' };
|
| 1430 |
+
}
|
| 1431 |
+
if (st === 'degraded') return { cls: 'warn', symbol: '⚠', label: 'DEGRADED' };
|
| 1432 |
+
if (st === 'healthy') return { cls: 'ok', symbol: '✓', label: 'HEALTHY' };
|
| 1433 |
+
return { cls: 'unknown', symbol: '?', label: 'UNKNOWN' };
|
| 1434 |
+
}
|
| 1435 |
+
|
| 1436 |
+
function renderLlmHealthChain(chainBlock) {
|
| 1437 |
+
var card = createEl('div', { className: 'llm-health-chain' });
|
| 1438 |
+
var head = createEl('div', { className: 'chain-head' });
|
| 1439 |
+
var badge = createEl('span', { className: 'role-badge', text: (ROLE_LABELS[chainBlock.role] || chainBlock.role).toUpperCase() });
|
| 1440 |
+
badge.style.background = ROLE_COLORS[chainBlock.role] || '#444';
|
| 1441 |
+
head.appendChild(badge);
|
| 1442 |
+
var title = createEl('span', { text: 'Election' });
|
| 1443 |
+
title.style.fontWeight = '600';
|
| 1444 |
+
title.style.fontSize = '13px';
|
| 1445 |
+
head.appendChild(title);
|
| 1446 |
+
card.appendChild(head);
|
| 1447 |
+
|
| 1448 |
+
// Render PRIMARY + BACKUP rows.
|
| 1449 |
+
['primary', 'backup'].forEach(function (role) {
|
| 1450 |
+
var snap = chainBlock[role + '_snapshot'];
|
| 1451 |
+
var name = chainBlock['elected_' + role];
|
| 1452 |
+
var block = createEl('div', { className: 'role-block' });
|
| 1453 |
+
var label = createEl('div', { className: 'role-label', text: role.toUpperCase() });
|
| 1454 |
+
block.appendChild(label);
|
| 1455 |
+
if (!name || !snap) {
|
| 1456 |
+
var none = createEl('div', { className: 'none-elected', text: 'none elected (no eligible candidate)' });
|
| 1457 |
+
block.appendChild(none);
|
| 1458 |
+
card.appendChild(block);
|
| 1459 |
+
return;
|
| 1460 |
+
}
|
| 1461 |
+
var nameLine = createEl('div', { className: 'model-name' });
|
| 1462 |
+
var b = candidateBadge(snap);
|
| 1463 |
+
var statusBadge = createEl('span', { className: 'health-badge ' + b.cls, text: b.symbol + ' ' + b.label });
|
| 1464 |
+
statusBadge.style.marginRight = '6px';
|
| 1465 |
+
nameLine.appendChild(statusBadge);
|
| 1466 |
+
nameLine.appendChild(document.createTextNode(name));
|
| 1467 |
+
block.appendChild(nameLine);
|
| 1468 |
+
|
| 1469 |
+
var stats = createEl('div', { className: 'stats-line' });
|
| 1470 |
+
var pushed = false;
|
| 1471 |
+
function pushPart(text) {
|
| 1472 |
+
if (pushed) stats.appendChild(createEl('span', { className: 'sep', text: '·' }));
|
| 1473 |
+
stats.appendChild(document.createTextNode(text));
|
| 1474 |
+
pushed = true;
|
| 1475 |
+
}
|
| 1476 |
+
if (snap.latency_ms != null) pushPart('latency ' + fmtLatency(snap.latency_ms));
|
| 1477 |
+
if (snap.success_rate != null) pushPart('success ' + (Number(snap.success_rate) * 100).toFixed(0) + '%');
|
| 1478 |
+
if (snap.credits_remaining != null) pushPart('credits ' + fmtCredits(snap.credits_remaining, snap.credits_unit, null));
|
| 1479 |
+
if (snap.credits_reset_in_seconds != null && snap.credits_reset_in_seconds > 0) {
|
| 1480 |
+
pushPart('reset in ' + fmtDurationShort(snap.credits_reset_in_seconds));
|
| 1481 |
+
}
|
| 1482 |
+
if (snap.degraded_for_seconds != null && snap.degraded_for_seconds > 0) {
|
| 1483 |
+
pushPart('sin-bin ' + fmtDurationShort(snap.degraded_for_seconds));
|
| 1484 |
+
}
|
| 1485 |
+
if (!pushed) stats.textContent = 'No telemetry yet.';
|
| 1486 |
+
block.appendChild(stats);
|
| 1487 |
+
card.appendChild(block);
|
| 1488 |
+
});
|
| 1489 |
+
|
| 1490 |
+
if (chainBlock.chain_credit_exhausted) {
|
| 1491 |
+
var banner = createEl('div', { className: 'credit-banner' });
|
| 1492 |
+
banner.textContent = 'All candidates for ' + (ROLE_LABELS[chainBlock.role] || chainBlock.role).toUpperCase() +
|
| 1493 |
+
'_CHAIN are credit-exhausted; canonical fallback will fire until quotas reset.';
|
| 1494 |
+
card.appendChild(banner);
|
| 1495 |
+
}
|
| 1496 |
+
return card;
|
| 1497 |
+
}
|
| 1498 |
+
|
| 1499 |
+
function renderLlmHealthCandidates(rows) {
|
| 1500 |
+
var wrap = $('llm-health-candidates');
|
| 1501 |
+
clearChildren(wrap);
|
| 1502 |
+
if (!rows || !rows.length) {
|
| 1503 |
+
wrap.appendChild(createEl('div', { className: 'empty-state', text: 'No candidates known yet (probe loop hasn’t run).' }));
|
| 1504 |
+
return;
|
| 1505 |
+
}
|
| 1506 |
+
var table = createEl('table');
|
| 1507 |
+
var thead = createEl('thead');
|
| 1508 |
+
var tr = createEl('tr');
|
| 1509 |
+
['Status', 'Provider', 'Model', 'Chain(s)', 'Probe age', 'Latency', 'Success', 'Credits', 'Reset / sin-bin'].forEach(function (h, i) {
|
| 1510 |
+
var th = createEl('th', { text: h });
|
| 1511 |
+
if (i >= 4) th.style.textAlign = i === 8 ? 'left' : 'right';
|
| 1512 |
+
tr.appendChild(th);
|
| 1513 |
+
});
|
| 1514 |
+
thead.appendChild(tr);
|
| 1515 |
+
table.appendChild(thead);
|
| 1516 |
+
|
| 1517 |
+
var tbody = createEl('tbody');
|
| 1518 |
+
rows.forEach(function (r) {
|
| 1519 |
+
var row = createEl('tr');
|
| 1520 |
+
var b = candidateBadge(r);
|
| 1521 |
+
var stCell = createEl('td');
|
| 1522 |
+
var badge = createEl('span', { className: 'health-badge ' + b.cls, text: b.symbol + ' ' + b.label });
|
| 1523 |
+
stCell.appendChild(badge);
|
| 1524 |
+
row.appendChild(stCell);
|
| 1525 |
+
|
| 1526 |
+
row.appendChild(createEl('td', { className: 'mono', text: (r.provider || '—').toUpperCase() }));
|
| 1527 |
+
row.appendChild(createEl('td', { className: 'model-cell', text: r.model }));
|
| 1528 |
+
|
| 1529 |
+
var chainsLbl = (r.chain_membership || []).map(function (rr) { return (ROLE_LABELS[rr] || rr).toUpperCase(); }).join(', ');
|
| 1530 |
+
row.appendChild(createEl('td', { className: 'chain-mem', text: chainsLbl || '—' }));
|
| 1531 |
+
|
| 1532 |
+
row.appendChild(createEl('td', { className: 'num', text: r.probe_age_seconds != null ? fmtDurationShort(r.probe_age_seconds) + ' ago' : '—' }));
|
| 1533 |
+
row.appendChild(createEl('td', { className: 'num', text: r.latency_ms != null ? fmtLatency(r.latency_ms) : '—' }));
|
| 1534 |
+
row.appendChild(createEl('td', { className: 'num', text: r.success_rate != null ? (Number(r.success_rate) * 100).toFixed(0) + '%' : '—' }));
|
| 1535 |
+
row.appendChild(createEl('td', { className: 'credits', text: fmtCredits(r.credits_remaining, r.credits_unit, r.credits_low_water) }));
|
| 1536 |
+
|
| 1537 |
+
var resetParts = [];
|
| 1538 |
+
if (r.credits_reset_in_seconds != null && r.credits_reset_in_seconds > 0) {
|
| 1539 |
+
resetParts.push('quota in ' + fmtDurationShort(r.credits_reset_in_seconds));
|
| 1540 |
+
}
|
| 1541 |
+
if (r.degraded_for_seconds != null && r.degraded_for_seconds > 0) {
|
| 1542 |
+
resetParts.push('sin-bin ' + fmtDurationShort(r.degraded_for_seconds));
|
| 1543 |
+
}
|
| 1544 |
+
row.appendChild(createEl('td', { text: resetParts.length ? resetParts.join(' · ') : '—' }));
|
| 1545 |
+
|
| 1546 |
+
tbody.appendChild(row);
|
| 1547 |
+
});
|
| 1548 |
+
table.appendChild(tbody);
|
| 1549 |
+
wrap.appendChild(table);
|
| 1550 |
+
}
|
| 1551 |
+
|
| 1552 |
+
function renderLlmHealthRecent(turns) {
|
| 1553 |
+
var wrap = $('llm-health-recent');
|
| 1554 |
+
clearChildren(wrap);
|
| 1555 |
+
if (!turns || !turns.length) {
|
| 1556 |
+
wrap.appendChild(createEl('div', { className: 'empty-state', text: 'No turns logged yet.' }));
|
| 1557 |
+
return;
|
| 1558 |
+
}
|
| 1559 |
+
var table = createEl('table');
|
| 1560 |
+
var thead = createEl('thead');
|
| 1561 |
+
var tr = createEl('tr');
|
| 1562 |
+
['Time', 'Chain', 'Elected primary', 'Served model', 'Latency', 'Result'].forEach(function (h, i) {
|
| 1563 |
+
var th = createEl('th', { text: h });
|
| 1564 |
+
if (i === 4) th.style.textAlign = 'right';
|
| 1565 |
+
tr.appendChild(th);
|
| 1566 |
+
});
|
| 1567 |
+
thead.appendChild(tr);
|
| 1568 |
+
table.appendChild(thead);
|
| 1569 |
+
|
| 1570 |
+
var tbody = createEl('tbody');
|
| 1571 |
+
turns.forEach(function (t) {
|
| 1572 |
+
var row = createEl('tr');
|
| 1573 |
+
row.appendChild(createEl('td', { text: fmtIST(t.ts) }));
|
| 1574 |
+
var chainCell = createEl('td');
|
| 1575 |
+
if (t.role) {
|
| 1576 |
+
var badge = createEl('span', { className: 'role-badge', text: (ROLE_LABELS[t.role] || t.role).toUpperCase() });
|
| 1577 |
+
badge.style.background = ROLE_COLORS[t.role] || '#444';
|
| 1578 |
+
chainCell.appendChild(badge);
|
| 1579 |
+
} else {
|
| 1580 |
+
chainCell.textContent = '—';
|
| 1581 |
+
}
|
| 1582 |
+
row.appendChild(chainCell);
|
| 1583 |
+
|
| 1584 |
+
var electedPrimary = t.elected_primary || t.chain_primary || '—';
|
| 1585 |
+
row.appendChild(createEl('td', { className: 'mono', text: electedPrimary }));
|
| 1586 |
+
|
| 1587 |
+
var servedCell = createEl('td', { className: 'mono' });
|
| 1588 |
+
var served = t.served_model || '—';
|
| 1589 |
+
servedCell.textContent = served;
|
| 1590 |
+
if (electedPrimary !== '—' && served !== '—' && served !== electedPrimary) {
|
| 1591 |
+
// Served by backup/fallback — flag visually.
|
| 1592 |
+
servedCell.classList.add('served-diff');
|
| 1593 |
+
servedCell.title = 'Served by backup/fallback — elected primary was unavailable.';
|
| 1594 |
+
}
|
| 1595 |
+
row.appendChild(servedCell);
|
| 1596 |
+
|
| 1597 |
+
row.appendChild(createEl('td', { className: 'num', text: t.latency_ms != null ? fmtLatency(t.latency_ms) : '—' }));
|
| 1598 |
+
|
| 1599 |
+
var resultCell = createEl('td');
|
| 1600 |
+
var ok = t.success === true;
|
| 1601 |
+
var resBadge = createEl('span', { className: 'health-badge ' + (ok ? 'ok' : 'bad'), text: (ok ? '✓ OK' : '✗ FAIL') });
|
| 1602 |
+
resultCell.appendChild(resBadge);
|
| 1603 |
+
if (t.fallback_reason) {
|
| 1604 |
+
var fr = createEl('span', { className: 'small muted', text: ' ' + t.fallback_reason });
|
| 1605 |
+
resultCell.appendChild(fr);
|
| 1606 |
+
}
|
| 1607 |
+
row.appendChild(resultCell);
|
| 1608 |
+
|
| 1609 |
+
tbody.appendChild(row);
|
| 1610 |
+
});
|
| 1611 |
+
table.appendChild(tbody);
|
| 1612 |
+
wrap.appendChild(table);
|
| 1613 |
+
}
|
| 1614 |
+
|
| 1615 |
+
function renderLlmHealth() {
|
| 1616 |
+
var chainsHost = $('llm-health-chains');
|
| 1617 |
+
var snapEl = $('llm-health-snapshot-ts');
|
| 1618 |
+
if (!chainsHost) return;
|
| 1619 |
+
|
| 1620 |
+
if (!STATE.llmHealth) {
|
| 1621 |
+
clearChildren(chainsHost);
|
| 1622 |
+
chainsHost.appendChild(createEl('div', { className: 'empty-state', text: 'Loading…' }));
|
| 1623 |
+
clearChildren($('llm-health-candidates'));
|
| 1624 |
+
clearChildren($('llm-health-recent'));
|
| 1625 |
+
if (snapEl) snapEl.textContent = '';
|
| 1626 |
+
return;
|
| 1627 |
+
}
|
| 1628 |
+
if (STATE.llmHealth.__notDeployed) {
|
| 1629 |
+
clearChildren(chainsHost);
|
| 1630 |
+
chainsHost.appendChild(buildPendingMessage('/api/admin/llm-health'));
|
| 1631 |
+
clearChildren($('llm-health-candidates'));
|
| 1632 |
+
clearChildren($('llm-health-recent'));
|
| 1633 |
+
if (snapEl) snapEl.textContent = '';
|
| 1634 |
+
return;
|
| 1635 |
+
}
|
| 1636 |
+
|
| 1637 |
+
clearChildren(chainsHost);
|
| 1638 |
+
(STATE.llmHealth.chains || []).forEach(function (c) {
|
| 1639 |
+
chainsHost.appendChild(renderLlmHealthChain(c));
|
| 1640 |
+
});
|
| 1641 |
+
|
| 1642 |
+
renderLlmHealthCandidates(STATE.llmHealth.candidates || []);
|
| 1643 |
+
renderLlmHealthRecent(STATE.llmHealth.recent_turns || []);
|
| 1644 |
+
|
| 1645 |
+
if (snapEl) {
|
| 1646 |
+
snapEl.textContent = STATE.llmHealth.snapshot_ts
|
| 1647 |
+
? '· updated ' + fmtIST(STATE.llmHealth.snapshot_ts)
|
| 1648 |
+
: '';
|
| 1649 |
+
}
|
| 1650 |
+
}
|
| 1651 |
+
|
| 1652 |
+
// Auto-poll every 30s while the LLM Chain tab is the active tab.
|
| 1653 |
+
// The backend's probe loop only refreshes every 5min anyway — we poll
|
| 1654 |
+
// faster than that because credit + degraded-window state changes
|
| 1655 |
+
// sub-tick (chat hot path stamps both) and the operator needs to see
|
| 1656 |
+
// those without manually clicking Refresh.
|
| 1657 |
+
var LLM_HEALTH_POLL_INTERVAL_MS = 30000;
|
| 1658 |
+
function startLlmHealthPolling() {
|
| 1659 |
+
stopLlmHealthPolling();
|
| 1660 |
+
STATE.llmHealthPollTimer = setInterval(function () {
|
| 1661 |
+
if (STATE.activeTab !== 'chain') return;
|
| 1662 |
+
fetchLlmHealth().then(renderLlmHealth).catch(function () { /* swallow */ });
|
| 1663 |
+
}, LLM_HEALTH_POLL_INTERVAL_MS);
|
| 1664 |
+
}
|
| 1665 |
+
function stopLlmHealthPolling() {
|
| 1666 |
+
if (STATE.llmHealthPollTimer) {
|
| 1667 |
+
clearInterval(STATE.llmHealthPollTimer);
|
| 1668 |
+
STATE.llmHealthPollTimer = null;
|
| 1669 |
+
}
|
| 1670 |
+
}
|
| 1671 |
+
|
| 1672 |
function handleFetchErr(err) {
|
| 1673 |
if (err && err.code === 404) {
|
| 1674 |
try { localStorage.removeItem(STORAGE_KEY); } catch (_) {}
|
|
|
|
| 2191 |
// Lazy-load this tab's data
|
| 2192 |
if (tab === 'profiles') {
|
| 2193 |
loadProfiles(false);
|
| 2194 |
+
stopLlmHealthPolling();
|
| 2195 |
} else if (tab === 'performance') {
|
| 2196 |
loadPerformance(false);
|
| 2197 |
+
stopLlmHealthPolling();
|
| 2198 |
} else if (tab === 'chain') {
|
| 2199 |
if (!STATE.chainLoaded) {
|
| 2200 |
refreshChain().catch(handleFetchErr);
|
| 2201 |
+
} else {
|
| 2202 |
+
// Tab already loaded — refresh just the KI-086 panel on re-entry
|
| 2203 |
+
// so the operator immediately sees fresh credits / sin-bin state.
|
| 2204 |
+
fetchLlmHealth().then(renderLlmHealth).catch(function () { /* swallow */ });
|
| 2205 |
}
|
| 2206 |
+
startLlmHealthPolling();
|
| 2207 |
}
|
| 2208 |
}
|
| 2209 |
|
|
|
|
| 2273 |
STATE.health = null;
|
| 2274 |
STATE.chains = null;
|
| 2275 |
STATE.usage = null;
|
| 2276 |
+
STATE.llmHealth = null;
|
| 2277 |
STATE.profiles = null;
|
| 2278 |
STATE.profilesLoaded = false;
|
| 2279 |
STATE.performance = null;
|
| 2280 |
STATE.performanceLoaded = false;
|
| 2281 |
STATE.chainLoaded = false;
|
| 2282 |
+
stopLlmHealthPolling();
|
| 2283 |
try { localStorage.removeItem(STORAGE_KEY); } catch (_) {}
|
| 2284 |
showGate();
|
| 2285 |
toast('Locked', 'info');
|
|
|
|
| 2332 |
.catch(handleFetchErr)
|
| 2333 |
.then(function () { btn.disabled = false; btn.textContent = 'Force fresh probe (slow)'; });
|
| 2334 |
};
|
| 2335 |
+
|
| 2336 |
+
// KI-086 — manual refresh for the LLM Health & Credits card.
|
| 2337 |
+
var llmHealthBtn = $('btn-refresh-llm-health');
|
| 2338 |
+
if (llmHealthBtn) {
|
| 2339 |
+
llmHealthBtn.onclick = function () {
|
| 2340 |
+
var btn = this;
|
| 2341 |
+
btn.disabled = true;
|
| 2342 |
+
btn.textContent = 'Refreshing…';
|
| 2343 |
+
fetchLlmHealth()
|
| 2344 |
+
.then(function () { renderLlmHealth(); toast('LLM health refreshed', 'success'); })
|
| 2345 |
+
.catch(handleFetchErr)
|
| 2346 |
+
.then(function () { btn.disabled = false; btn.textContent = 'Refresh'; });
|
| 2347 |
+
};
|
| 2348 |
+
}
|
| 2349 |
}
|
| 2350 |
|
| 2351 |
// ----- Boot -----
|