docs: ADR-032 LLM chain architecture reference + KI-080..KI-087 doc sync

Adds new architecture reference ADR consolidating KI-079..KI-087:
- 70-docs/60-decisions/ADR-032-llm-chain-architecture.md (NEW, 435 lines)
- TL;DR + ASCII data flow diagram
- Component reference (every ModelHealth field)
- Election algorithm pseudo-code + score formula
- Probe loop cadence + OpenRouter credits polling
- Per-provider credit signal table (Groq headers / OR endpoint / NIM rate-meter)
- Per-phase httpx timeout justification
- Failure classification table
- KI-079 escalation path
- Telemetry surface (brain_used variants)
- Performance characteristics + operational runbook

Updated docs:
- CLAUDE.md "LLM stack" section: KI-080..KI-087 full coverage
+ NIM-first preference (KI-087), credit gating (KI-085), per-phase
timeouts (KI-084), admin tab (KI-086), HF Space secrets (KI-081)
- 80-audit/ENTERPRISE_AUDIT.md: 5 new rows (KI-081, KI-084, KI-085,
KI-086, KI-087) appended to "Fixes shipped today" table
- backend/providers/README.md: new "Credit-aware election (KI-085)"
section after Chain budgets — per-provider signal sources,
election predicate, cold-start behaviour, link to ADR-032

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

70-docs/60-decisions/ADR-032-llm-chain-architecture.md

@@ -0,0 +1,435 @@
+# ADR-032 — LLM Chain Architecture Reference
+
+**Status:** Accepted — 2026-05-15
+**Type:** Architecture reference (not a decision ADR)
+**Owner:** Rohit Saraf
+**Consolidates:** [ADR-019](ADR-019-nim-single-provider-consolidation.md), [ADR-026](ADR-026-provider-load-balancing.md) (superseded), [ADR-030](ADR-030-llm-driven-fact-find.md), [ADR-031](ADR-031-sticky-primary-election.md)
+**Related KIs:** KI-079 (`87ee522`), KI-080 (`6159c54`), KI-081 (HF Space env), KI-084 (`119e0fd`), KI-085 (`8fc7979`)
+
+> This is **not a decision ADR.** No alternatives or trade-offs are weighed here.
+> ADR-032 is the single readable spec for how the LLM chain works in production
+> after the KI-080 → KI-085 sweep. New decisions still ship as their own ADRs;
+> this file is updated when the spec shifts.
+
+## 1. TL;DR
+
+Every LLM role (`brain` / `fast_brain` / `judge`) is a **candidate pool**, not a
+hardcoded model. A background probe loop in `backend/llm_health.py` scores every
+candidate every 300s and elects a sticky PRIMARY + provider-diverse BACKUP per
+chain. `NimChainLLM.chat()` calls the elected PRIMARY exactly once per turn with
+explicit `httpx` per-phase timeouts (`connect=2s, read=12s, write=2s, pool=2s`);
+on real-time failure it falls to the elected BACKUP once. Election is gated by
+**liveness AND credits** — each provider's credit signal (Groq response
+headers / OpenRouter `/api/v1/credits` endpoint / NIM local rate-meter)
+proactively excludes quota-exhausted candidates BEFORE the user hits a 429,
+while a reactive 1-hour demotion absorbs any 429 that slips through. Result:
+per-turn LLM call count is 1 (happy path), 2 (PRIMARY failover), or 3 (KI-079
+fast→heavy escalation), with `_canonical_fallback` (KI-072 / KI-074 greedy
+slot capture) as the always-available last bite.
+
+## 2. Data flow
+
+```
+User chat turn
+    ↓
+FastAPI /api/chat  →  backend/orchestrator.py
+    ↓
+fact_find_brain.drive_fact_find()
+    ↓
+asyncio.wait_for(_TIMEOUT_S=25s) wrapping NimChainLLM(FAST_BRAIN_CHAIN).chat()
+    ↓
+┌─────────────────────────────────────────────────────────┐
+│  NimChainLLM.chat()  — KI-080 election + KI-084/085     │
+│  ┌───────────────────────────────────────────────────┐  │
+│  │ Election  (backend/llm_health.py)                 │  │
+│  │   PRIMARY  = highest score in chain               │  │
+│  │             AND is_alive (probe < 600s)           │  │
+│  │             AND has_credits > low_water (KI-085)  │  │
+│  │             AND NOT in 1h demote window (KI-084)  │  │
+│  │   BACKUP   = next-best, cross-provider preferred  │  │
+│  │             (same eligibility predicate)          │  │
+│  │   score    = (1/max(50, latency_ms)) * success    │  │
+│  └───────────────────────────────────────────────────┘  │
+│  ┌───────────────────────────────────────────────────┐  │
+│  │ _call_one(PRIMARY)  — KI-084 per-phase timeouts   │  │
+│  │   httpx.Timeout(connect=2, read=12, write=2,      │  │
+│  │                 pool=2)                           │  │
+│  │   12s read-deadline = NimChainLLM._ELECTED_CALL_  │  │
+│  │   TIMEOUT_S; outer wait_for is the hard ceiling.  │  │
+│  └───────────────────────────────────────────────────┘  │
+│       │                                                 │
+│       ├── success → report_success(latency_ms)          │
+│       │            → record_nim_call() (KI-085)         │
+│       │            → return LLMResult                   │
+│       │                                                 │
+│       └── failure → _classify_error(e)                  │
+│             │                                           │
+│             ├── Status429 / "RateLimit"                 │
+│             │     → report_failure → demote 1h (KI-084) │
+│             │                                           │
+│             ├── HTTPStatusError:5xx / ReadTimeout /     │
+│             │   TimeoutException / net errors           │
+│             │     → report_failure → demote 30s         │
+│             │                                           │
+│             └── (credit state already updated from      │
+│                 response headers in groq/openrouter     │
+│                 client; KI-085)                         │
+│       ↓                                                 │
+│  _call_one(BACKUP)  — same shape, same timeouts         │
+│       ↓ fail → trigger probe_all() refresh + walk       │
+│              filter_chain order (final safety net,      │
+│              budget-clipped per remaining total_budget)│
+│       ↓ fail → raise RuntimeError                       │
+└─────────────────────────────────────────────────────────┘
+    ↓ outer wait_for fires (25s) OR RuntimeError raised
+    ↓
+fact_find_brain.drive_fact_find() catches asyncio.TimeoutError
+    ↓
+[KI-079] escalation:
+    asyncio.wait_for(_TIMEOUT_S_ESCALATION=15s)
+        wrapping NimChainLLM(BRAIN_CHAIN).chat()
+    ↓ same election + per-phase + credit gating, heavier pool
+    ↓ success → reply prose to user, brain_used = fact_find_brain::continue
+    ↓ fail → _canonical_fallback(session, user_text, reason="…")
+            (KI-072 / KI-074 greedy slot capture; never wedges)
+    ↓
+user-facing reply
+    brain_used = fact_find_brain::continue
+               | fact_find_brain::complete
+               | fact_find_brain::fallback:<reason>
+```
+
+## 3. Component reference — `llm_health._STATE`
+
+Per-model state is a `ModelHealth` dataclass keyed by chain entry (model id with
+optional `groq:` / `openrouter:` prefix). Fields the elector reads on every
+call:
+
+| Field | Type | Producer | Meaning |
+|---|---|---|---|
+| `status` | `'healthy' / 'degraded' / 'down' / 'unknown'` | probe loop (`_absorb_probe_result`) | Coarse health bucket; `'down'` ⇒ election-ineligible. |
+| `latency_ms` | `int / None` | probe + `report_success` | Last observed latency; `None` ⇒ ineligible. Floor of 50ms in scoring. |
+| `last_success_at` | ISO8601 | probe + chat | Wall-clock of last 2xx. |
+| `last_failure_at` | ISO8601 | probe + chat | Wall-clock of last failure. |
+| `tested_at` | ISO8601 | probe loop | Drives the `HEALTHY_PROBE_AGE_SEC = 600s` freshness gate. |
+| `probe_history` | `list[dict]`, cap 5 | probe + chat | `[{ok, latency_ms, ts, src}]`. Powers `success_rate`. |
+| `degraded_until_monotonic` | `float` (monotonic time) | `report_failure` | Sin-bin deadline; 30s transient, 3600s rate-limit (KI-084). |
+| `credits_remaining` | `float / None` | KI-085 trackers | Tokens / USD / req-slots remaining. `None` ⇒ cold-start permissive. |
+| `credits_unit` | `'tokens_day' / 'usd_balance' / 'requests_min'` | KI-085 trackers | Semantic of the number above. |
+| `credits_reset_at` | `float / None` (monotonic) | KI-085 trackers | When the quota resets; past-now ⇒ signal treated as stale (permissive). |
+| `credits_observed_at` | `float` (monotonic) | KI-085 trackers | Snapshot timestamp; surfaced in admin tab. |
+| `credits_low_water` | `float` | KI-085 constants | Gate threshold. `credits_remaining > credits_low_water` ⇒ electable. |
+
+Persisted snapshot: `40-data/llm_health.json` (atomic write).
+
+## 4. Election algorithm
+
+`backend/llm_health.py::_ranked_candidates(chain_name)` returns
+election-eligible candidates sorted by score descending. Election runs every
+`chat()` call (microsecond hot path; in-memory `_STATE` under a coarse lock).
+
+```python
+def is_electable(h, now_mono):
+    if h.degraded_until_monotonic > now_mono: return False    # KI-084 sin-bin
+    if h.status == "down":                    return False
+    if age_of(h.tested_at) > 600s:            return False    # HEALTHY_PROBE_AGE_SEC
+    if h.latency_ms is None:                  return False
+    if not has_credits(h, now_mono):          return False    # KI-085
+    return True
+
+def has_credits(h, now_mono):
+    if h.credits_reset_at is not None and now_mono >= h.credits_reset_at:
+        return True                                # quota already reset
+    if h.credits_remaining is None:                # cold-start permissive
+        return True
+    return h.credits_remaining > h.credits_low_water
+
+def score(h):
+    return (1 / max(50, h.latency_ms)) * success_rate(h.probe_history)
+
+def get_primary(chain): return ranked[0].model
+def get_backup(chain):
+    primary_provider = provider_of(ranked[0])
+    for h in ranked[1:]:
+        if provider_of(h) != primary_provider: return h.model   # provider-diverse
+    return ranked[1].model                                      # graceful degradation
+```
+
+**Score formula.** `score = (1 / max(50, latency_ms)) * success_rate`. The 50 ms
+floor prevents a sub-millisecond outlier from dominating election; the rolling
+`success_rate` over the last 5 probes is the stability signal. Both factors
+matter — a very fast model that flakes 1-in-3 calls scores lower than a stable
+model 2× slower.
+
+**Provider-diverse BACKUP.** Mandatory, not advisory. A NIM-PRIMARY whose
+underlying pool is throttled MUST fall to a non-NIM BACKUP, otherwise the second
+call queues in the same throttle window. Iterate ranked candidates and return
+the first whose `provider_of()` differs from PRIMARY's. If only one provider has
+live candidates (regional outage), BACKUP gracefully degrades to the next-best
+same-provider candidate.
+
+**Cold-start fallback.** Before the first probe completes (process restart,
+HF Space rebuild), `get_primary` / `get_backup` return `None`. `NimChainLLM.chat()`
+catches that case and uses `chain[0]` as PRIMARY and `chain[1]` (preferring a
+different provider) as BACKUP. The probe loop runs immediately on startup so
+cold-start lasts at most a few seconds.
+
+**Family exclusion.** Brain ↔ judge family diversity (Qwen brain ↔ Mistral judge)
+is enforced by the caller via `exclude_families=[...]` on
+`NimChainLLM.chat()`. The election then filters election-eligible candidates by
+`_family_of()` before scoring. Families: `qwen`, `mistral`, `meta`, `openai`,
+`deepseek`, `moonshot`, `minimax`, `nvidia`. A NIM-hosted GPT-OSS 120B and an
+OpenRouter-hosted GPT-OSS 120B share the `openai` family and are NOT pickable
+as brain ↔ judge pair.
+
+## 5. Probe loop
+
+`backend/llm_health.py::background_probe_loop` ticks every
+`PROBE_INTERVAL_SEC = 300s` (KI-084 — was 60s; raised because the prior cadence
+burned ~30-50K tokens/day on Groq's free-tier and self-tripped the 100K daily
+TPD cap).
+
+Each tick:
+1. `probe_all()` — parallel `httpx.post` to every chain entry with the prompt
+   `"Reply with exactly: ok"`, `max_tokens=1` (KI-084 — was 5; cuts probe-driven
+   token spend ~50× since we never read the body content), `timeout=8s`.
+2. Status flip rules: 200 + non-empty content ⇒ `healthy` (or `degraded` if
+   latency > 5000ms); 3+ consecutive failures ⇒ `down`.
+3. Append `(ok, latency_ms, ts, src='probe')` to `probe_history` (capped at 5).
+4. `save()` atomic-write of `40-data/llm_health.json`.
+
+Every `OPENROUTER_CREDITS_POLL_EVERY_N_TICKS = 2` ticks (i.e. 600s / 10min):
+- `poll_openrouter_credits()` issues `GET https://openrouter.ai/api/v1/credits`,
+  parses `{total_credits, total_usage}`, and stamps every `openrouter:`-prefixed
+  candidate with `credits_unit="usd_balance"`, `credits_remaining = total_credits - total_usage`,
+  `credits_low_water = $0.05`.
+
+Initial OpenRouter credits poll fires immediately on startup so the elector has
+a non-`None` USD balance before the first chat call.
+
+## 6. Credit signal sources
+
+| Provider | Signal source | Header / endpoint | Stored unit | Low-water | Producer |
+|---|---|---|---|---|---|
+| **Groq** | Response headers on every successful chat | `x-ratelimit-remaining-tokens-day` (preferred — daily TPD is what bit us in KI-084) + `x-ratelimit-reset-tokens-day` for reset deadline | `tokens_day` | `5000` tokens (≈ one ~2K-input / ~400-output fact-find round-trip with margin) | `groq_llm.py::chat` → `llm_health.update_credits_from_groq` |
+| **OpenRouter** | Dedicated account endpoint | `GET /api/v1/credits` → `{total_credits, total_usage}` (account-level USD balance) + opportunistic `x-ratelimit-remaining` from response headers as between-poll fallback | `usd_balance` (authoritative) / `requests_min` (header fallback) | `$0.05` USD (5¢ safety margin — free-models charge $0 but the account-level signal still tells us if prepaid credits are gone) | `llm_health.poll_openrouter_credits` (10-min) + `update_credits_from_openrouter_headers` (per-call) |
+| **NIM** | No clean header — local rate-meter | `_NIM_CALL_TIMES[model]` deque of monotonic timestamps over a 60s window | `requests_min` | `5.0` request slots; gate at `cap - headroom = 40 - 5 = 35` in-window calls | `nvidia_nim_llm.py::chat` → `llm_health.record_nim_call` |
+
+For Groq specifically, daily TPD is the dominant signal — the minute-window
+header is noisy and KI-084's 1h sin-bin already covers minute-window blips. We
+deliberately ignore `x-ratelimit-remaining-tokens-min` to keep the elector
+stable.
+
+## 7. Per-phase httpx timeouts (KI-084)
+
+`backend/providers/nvidia_nim_llm.py::NvidiaNimLLM.chat` uses an explicit
+`httpx.Timeout` rather than the scalar `timeout=self.timeout`:
+
+```python
+client_timeout = httpx.Timeout(
+    connect=2.0,    # TCP handshake must finish in 2s
+    read=self.timeout,   # 12s for elected calls; 6s for legacy fast-brain calls
+    write=2.0,      # request-body upload deadline
+    pool=2.0,       # connection-pool checkout deadline
+)
+```
+
+**Why each value.**
+
+- `connect=2.0` — TCP handshake to `integrate.api.nvidia.com` is sub-100ms in
+  steady state; anything past 2s means the ingress is down and we want the
+  candidate demoted, not the chat call hanging.
+- `read=self.timeout` (12.0 in the KI-080 elected path) — the wall-clock budget
+  for the upstream to produce a complete response. Matches
+  `NimChainLLM._ELECTED_CALL_TIMEOUT_S` so a stuck NIM pool can't burn the
+  outer `wait_for` ceiling.
+- `write=2.0` — our request bodies are <10 KB; 2s is generous.
+- `pool=2.0` — if every NIM HTTP/2 connection is in use and we can't even check
+  one out within 2s, fail fast so BACKUP gets called.
+
+**Why per-phase, not scalar.** Pre-KI-084, `httpx` collapsed a scalar timeout to
+a single read deadline. A stuck NIM pool could occupy the TCP connection past
+the outer `asyncio.wait_for` cancellation — the BACKUP started but PRIMARY's
+socket was still held, leaking a NIM concurrency slot. Explicit per-phase
+deadlines guarantee the TCP connection itself releases independently.
+
+## 8. Failure classification
+
+`_classify_error(e)` in `nvidia_nim_llm.py` maps a raised exception to a stable
+string consumed by `llm_health.report_failure`. The string drives the sin-bin
+duration.
+
+| Error class string | Source | Demote duration | Rationale |
+|---|---|---|---|
+| `Status429` | `HTTPStatusError.response.status_code == 429` | **3600s (1h)** — `DEGRADE_DURATION_LONG_S` | Free-tier daily quotas don't reset in 30s. KI-084. |
+| `HTTPStatusError:503` / `:502` / `:500` | non-429 HTTP errors | 30s — `DEGRADED_WINDOW_SEC` | Upstream brownouts typically clear inside a minute. |
+| `ReadTimeout` / `TimeoutException` | `httpx` per-phase or scalar timeout | 30s | TCP/upstream stall; next probe re-tests. |
+| `ConnectError` / `ConnectTimeout` | DNS / TLS / TCP failures | 30s | Network blip; recover quickly. |
+| Any other `Exception` class name | parse failures, unexpected payload shapes | 30s | Defensive same-window. |
+
+The rate-limit detector
+(`_is_rate_limit_error`) matches `"429"`, `"ratelimit"`, or `"rate_limit"` (case
+insensitive). It deliberately does NOT match bare `"HTTPStatusError"` so a 503
+falls to the short window, not the 1h window.
+
+Side effect on every `report_failure`: a synthetic
+`{"ok": False, "src": "chat"}` entry is appended to `probe_history` so the next
+election's `success_rate` reflects the live failure before the next probe tick.
+An async re-probe of the failed model is scheduled best-effort so a quota that
+happened to reset early is picked up immediately.
+
+## 9. Escalation path (KI-079)
+
+`backend/fact_find_brain.py::drive_fact_find` wraps the FAST_BRAIN call in
+`asyncio.wait_for(_TIMEOUT_S=25s)`. On `asyncio.TimeoutError`:
+
+1. **Heavy-brain retry.** Log `KI-079: fast brain timeout …` and call
+   `get_brain_llm()` (BRAIN_CHAIN) wrapped in
+   `asyncio.wait_for(_TIMEOUT_S_ESCALATION=15s)`. The heavy chain uses a
+   different election (Qwen 80B primary in steady state), different fallback
+   ladder, and reaches OpenRouter + Groq earlier in the candidate pool —
+   realistic escalation success cases land in 3-8s.
+2. **Canonical fallback.** If heavy also times out, returns
+   `_canonical_fallback(session, user_text, reason="timeout_after_escalation")`
+   which:
+   - Greedily applies `_normalize_for_slot` to every unfilled slot in priority
+     order (age → dependents → income_band → existing_cover → primary_goal →
+     location → parents_age → budget → name), with slot-specific trigger guards
+     to prevent cross-contamination (KI-072 / KI-074).
+   - Picks the next still-empty slot and returns the canonical question.
+   - Fact-find never wedges. A fully-dead network still walks the user through
+     fact-find via canonical questions.
+
+Total worst-case wall-clock before canonical fallback: 25s (FAST) + 15s
+(BRAIN escalation) = **40s**. The 25s FAST cap only fires when NIM is wedged AND
+no cross-provider election candidate is electable, which is rare with KI-080 +
+KI-085 in place.
+
+## 10. Telemetry surface
+
+`TurnResult.brain_used` is the single string downstream consumers (admin
+analytics, eval harness) read to attribute outcomes. Emitted variants:
+
+| brain_used | Meaning |
+|---|---|
+| `fact_find_brain::continue` | LLM brain succeeded, fact-find still in progress |
+| `fact_find_brain::complete` | LLM brain succeeded, fact-find now complete |
+| `fact_find_brain::fallback:timeout` | FAST_BRAIN_CHAIN exhausted, escalation flag NOT yet applied (pre-KI-079 leftover; should rarely appear) |
+| `fact_find_brain::fallback:timeout_after_escalation` | FAST timed out AND heavy-brain (BRAIN_CHAIN) also timed out — canonical fallback fired |
+| `fact_find_brain::fallback:llm_error_after_escalation` | FAST timed out, heavy-brain raised (non-timeout exception) — canonical fallback fired |
+| `fact_find_brain::fallback:llm_error` | FAST raised a non-timeout exception (HTTP / parse / etc.) |
+| `fact_find_brain::fallback:no_trailer` | Brain replied but the `<FF>{...}</FF>` block was missing or malformed |
+| `fact_find_brain::fallback:empty_reply` | Brain replied with only a `<FF>` block — no prose |
+
+In addition, every successful `NimChainLLM.chat` writes a JSONL record to
+`40-data/llm_usage.jsonl` with `{role, chain_primary, served_model,
+elected_primary, elected_backup, latency_ms, success, [fallback_phase]}` so the
+admin tab can audit which candidate served each turn. The admin
+`status_summary()` surface now also returns
+`elections: {brain: {primary, backup}, fast_brain: {…}, judge: {…}}` and the
+per-model `credits_remaining` / `credits_unit` / `credits_low_water` from
+KI-085 so operators can see why a candidate is gated out.
+
+## 11. Performance characteristics
+
+**Per-turn LLM call count** (under normal conditions with at least one healthy
+candidate per provider):
+
+| Scenario | Calls | Wall-clock (steady state) |
+|---|---|---|
+| Happy path: elected PRIMARY succeeds | **1** | 2-6s (depends on which provider wins election; Groq LPU ~1s, NIM Qwen ~2-3s, NIM Nemotron ~1.6s) |
+| PRIMARY fails real-time → BACKUP succeeds | **2** | 4-12s (PRIMARY's 12s read deadline + BACKUP latency) |
+| Both fail → KI-079 escalation succeeds on heavy chain | **3** | 18-25s (FAST budget + 3-8s heavy escalation) |
+| Total exhaustion → canonical fallback | (heavy chain attempted) | up to 40s + canonical reply |
+
+**Probe-driven token spend** (KI-084 cadence). With 25 candidates × 1
+token per probe × `300s` cadence = ~7K probe tokens/day on Groq's free-tier TPD
+(well inside the 100K cap). Pre-KI-084 (60s cadence, 5 tokens per probe) was
+~150K/day → tripped the cap on probe traffic alone.
+
+**Expected p50 / p95 latency** in steady state with elected primaries (Groq
+Llama-3.3-70B fast-brain / NIM Qwen 80B brain / Mistral Large 3 judge):
+
+| Role | p50 | p95 |
+|---|---|---|
+| fast_brain (fact-find turn) | ~2.0s | ~6s (one BACKUP failover) |
+| brain (synthesis / comparison) | ~3.5s | ~10s |
+| judge (faithfulness Gate 4) | ~4.5s | ~12s |
+
+## 12. Operational runbook
+
+### Provider exhausts 0 credits (Groq daily TPD hits 100,000/100,000)
+
+Sequence is fully automatic — no operator action required:
+
+1. **Reactive** — next user chat turn that elects Groq raises `Status429` from
+   the upstream. `_classify_error` returns `"Status429"`.
+   `llm_health.report_failure` sets `degraded_until_monotonic = now + 3600s`
+   and appends a failed `probe_history` entry.
+2. **Election re-runs** on the next call. `is_electable(groq)` returns False
+   (sin-bin). The elector picks the next-ranked non-Groq candidate (typically
+   NIM Qwen 80B for fast-brain).
+3. **User-visible behaviour.** First post-exhaustion turn pays ONE failover
+   (PRIMARY=Groq raises 429 ⇒ BACKUP=NIM Qwen answers). Total wall-clock: 12s
+   read-deadline + NIM Qwen latency ≈ 14-15s. User sees a real reply, not a
+   canonical fallback. Every subsequent turn for the next hour is a single
+   1-call NIM Qwen response (~2-3s).
+4. **Proactive (KI-085) — what should have happened instead.** If
+   `update_credits_from_groq` had received an `x-ratelimit-remaining-tokens-day`
+   header showing < 5000 tokens on the previous successful call, election
+   would have excluded Groq BEFORE the 429-producing call. The user would
+   never have seen the 14-15s failover turn. KI-085 closes the one-turn
+   reactive gap that KI-084 alone leaves.
+5. **Recovery.** At `T + 1h` the sin-bin expires; `is_electable(groq)`
+   becomes True again. If the daily quota actually reset by then, the next
+   probe (or the next chat turn) re-stamps `credits_remaining` from headers
+   and Groq re-enters election. If the daily quota did NOT reset, the next
+   chat turn re-raises 429, sin-bin extends another hour.
+
+### Live diagnosis worked example (2026-05-15)
+
+Production hit `Status429` on Groq with the response headers showing
+`x-ratelimit-remaining-tokens-day: 546` and `total used: 99,454 / 100,000`. Pre-
+KI-085: every user turn for the next hour paid the 14-15s reactive failover
+because the elector had no proactive signal. Post-KI-085: the *previous*
+successful Groq call stamped `credits_remaining = 99454 - <last_call_tokens>`,
+the next call saw `credits_remaining < 5000` and election excluded Groq before
+the 429-producing call. Fast-brain primary flipped to NIM Qwen 80B; user saw
+2-3s natural-LLM replies throughout the quota-exhausted window.
+
+### When manual intervention IS needed
+
+- **Full NIM regional outage** (all NIM-hosted candidates returning 5xx for
+  >15 min). Probe loop marks every NIM candidate as `down` after 3 consecutive
+  fails. Cross-provider candidates (OpenRouter, Groq) keep serving. No
+  operator action; if both Groq and OpenRouter are also degraded, the canonical
+  fallback path takes over and the user still gets a coherent (if scripted)
+  reply. Page Rohit only if `_canonical_fallback` reason `:no_trailer` /
+  `:empty_reply` rates spike — that means the brain *is* responding but
+  malformed, which the chain logic can't auto-heal.
+- **API key rotation.** `.env` (local) / HF Space environment secrets
+  (production) hold `NVIDIA_NIM_API_KEY`, `GROQ_API_KEY`, `OPENROUTER_API_KEY`.
+  KI-081 pushed the Groq + OpenRouter keys to the HF Space environment so the
+  chain works in production without code redeploys. Rotating any key requires:
+  (a) update local `.env`, (b) update HF Space secrets via Settings → Repository
+  secrets, (c) bounce the Space (`huggingface_hub.HfApi().restart_space()` or
+  manual restart).
+- **OpenRouter wallet refill.** `poll_openrouter_credits()` will pick up the
+  new balance within 10 min. To accelerate, restart the Space.
+
+## 13. Files touched (across KI-079 / KI-080 / KI-084 / KI-085)
+
+- `backend/providers/nvidia_nim_llm.py` — `NimChainLLM.chat` election rewrite
+  (KI-080), `_classify_error` for Status429 (KI-084), per-phase httpx timeouts
+  (KI-084), `record_nim_call` hook on every NIM success (KI-085).
+- `backend/llm_health.py` — KI-080 election + 60s probe (now 300s in KI-084),
+  KI-084 1h rate-limit demote + `DEGRADE_DURATION_LONG_S` + probe cadence /
+  max_tokens reduction, KI-085 `update_credits_from_groq` /
+  `update_credits_from_openrouter_headers` / `poll_openrouter_credits` /
+  `record_nim_call` + the `_has_credits` election predicate.
+- `backend/providers/groq_llm.py` — `update_credits_from_groq` call after
+  successful HTTP (KI-085).
+- `backend/providers/openrouter_llm.py` — `update_credits_from_openrouter_headers`
+  call after successful HTTP (KI-085).
+- `backend/fact_find_brain.py` — `_TIMEOUT_S_ESCALATION = 15.0` + heavy-brain
+  retry path on FAST timeout (KI-079).

80-audit/ENTERPRISE_AUDIT.md

@@ -268,6 +268,11 @@ The full eval was sending **every** QA question to `needs_finder`. Sample bot an
 | KI-078 | `078ff45` | LLM chain hardening: per-link timeout 12s → 6s so chain can try 3-4 candidates inside `total_budget_s=22s` instead of 1. Narrowed `except Exception` to re-raise `CancelledError`/`KeyboardInterrupt`/`SystemExit` so `asyncio.wait_for` actually bubbles. New `_fallback_reason` stamped on `FactFindOutcome` and surfaced as `fact_find_brain::fallback:timeout` / `:no_trailer` / `:empty_reply` / `:llm_error` in `TurnResult.brain_used` for production telemetry. |
 | KI-079 | `87ee522` | Two-layer chain hardening for fact-find. (1) FAST_BRAIN_CHAIN reorder: Groq Llama-3.3-70B promoted from position #5 to #2 (right after Nemotron primary) so cross-provider fallback is reached in ~6-7s of budget instead of ~20s. (2) Heavy-chain escalation: when `drive_fact_find()` raises `TimeoutError` on fast-brain, orchestrator retries once on `BRAIN_CHAIN` (Qwen 80B primary, 35s budget) before falling to `_canonical_fallback`. Adds `fallback:timeout_after_escalation` / `:llm_error_after_escalation` to telemetry vocabulary. Now serves as the last-bite safety net once KI-080's primary election is in place. |
 | **KI-080** | **`6159c54`** | **Sticky primary election for LLM chains ([ADR-031](../70-docs/60-decisions/ADR-031-sticky-primary-election.md)).** `NimChainLLM.chat()` refactored from "iterate every chain candidate sequentially per call" to "call the probe-elected PRIMARY once; on real-time failure, call the cross-provider BACKUP once and re-trigger the probe." `backend/llm_health.py` runs a 60s background probe loop that scores every candidate on (latency × success rate) and elects a sticky PRIMARY + provider-diverse BACKUP for each of `BRAIN_CHAIN` / `FAST_BRAIN_CHAIN` / `JUDGE_CHAIN`. **Per-turn LLM call count drops from 5-6 (sustained NIM degradation, every candidate queued + timed out) to 1 (most cases) or 2 (primary fails real-time → backup + re-probe).** ADR-026's `_balanced_brain_chain` (KI-025 50/50 NIM ↔ Groq rotation) is deprecated — the probe-driven election dynamically picks the actually-faster candidate instead of a fixed 50/50 coin. Code retained as a bypassed branch behind a feature flag for one-release rollback. Cold-start fallback (no probes complete yet) uses `chain[0]` as the initial primary. KI-079 escalation still applies as the last bite if both primary AND backup fail in the same turn. Inline tests: 7/7 OK (cold-start, lowest-latency primary, cross-provider backup, demote-on-failure, score-update, primary-success makes 1 call, primary-fail/backup-success makes exactly 2 calls). `tests/test_routing_regression.py` → 15/15 pass. |
+| KI-081 | (no commit — HF Space env secrets) | Pushed `GROQ_API_KEY` + `OPENROUTER_API_KEY` to the HF Space repository secrets so the KI-080 cross-provider election candidates actually have working keys in production. Pre-KI-081 only `NVIDIA_NIM_API_KEY` was set on the Space; the elector would mark every Groq + OpenRouter candidate as `no_api_key` and election degraded to NIM-only candidates — defeating the cross-provider BACKUP invariant. |
+| KI-084 | `119e0fd` | **LLM chain telemetry hardening + free-tier guards.** Four changes in one commit. (1) Probe cadence `PROBE_INTERVAL_SEC` raised 60s → **300s** — the prior cadence burned ~30-50K probe tokens/day on Groq alone, self-tripping Groq's 100K/day TPD free-tier cap. (2) `PROBE_MAX_TOKENS` cut 5 → **1** — same 200 envelope, ~50× less token spend per probe. (3) Explicit per-phase `httpx.Timeout(connect=2, read=self.timeout, write=2, pool=2)` on every chat call — previously `timeout=self.timeout` collapsed to a single read deadline so a stuck NIM pool could occupy the TCP connection past `asyncio.wait_for` cancellation, leaking NIM concurrency slots. (4) New `_classify_error` surfaces HTTP status codes explicitly (`Status429` vs `HTTPStatusError:503`); rate-limit failures get a **1-hour** sin-bin (`DEGRADE_DURATION_LONG_S = 3600s`) instead of the 30s transient window — free-tier daily quotas don't reset in 30 seconds. |
+| KI-085 | `8fc7979` | **Proactive credit tracking — closes the reactive-only gap KI-084 leaves.** KI-084 demotes a candidate for 1h AFTER a 429 hits, costing one user-facing failover turn per dead quota. KI-085 promotes `llm_health` from liveness-only to liveness-AND-credits so election excludes quota-exhausted candidates BEFORE the user gets stuck behind a 429. Three signal sources: (1) Groq response headers `x-ratelimit-remaining-tokens-day` + `x-ratelimit-reset-tokens-day` (low-water 5K tokens); (2) OpenRouter `/api/v1/credits` polled every 10 min from probe loop, plus per-call header fallback (low-water $0.05); (3) NIM local 60s rate-meter, gate at 35-of-40 req/min (headroom 5). Election adds `_has_credits(h, now_mono)` to eligibility predicate. Admin `status_summary` extended with `credits_remaining` / `credits_unit` / `credits_low_water` per model. 11/11 inline tests pass + routing_regression 15/15. |
+| KI-086 | `d90f8c0` (bundled with KI-087) | **Admin "LLM Health & Credits" tab.** New `GET /api/admin/llm-health` endpoint returns `{chains, candidates, recent_turns, snapshot_ts}` JSON: per-chain elected PRIMARY + BACKUP with snapshots, per-candidate health grid with credits + degraded-until, last 20 turn outcomes from `40-data/llm_usage.jsonl`. Same `_check_admin` IP-allowlist + password gate as other admin endpoints. Frontend extends the existing "LLM Chain" tab in `frontend/public/admin/llm-control.html` with three sections: (A) per-chain election cards, (B) candidate health table, (C) recent turns table. Auto-polls every 30s while tab is active. Operator now sees at-a-glance which LLM is in use where, why a candidate is gated out, and how the election state evolves. |
+| **KI-087** | **`d90f8c0`** | **NIM-first election preference.** Pre-KI-087 election scored purely by `latency × success_rate`, which consistently favoured Groq's 161ms LPU TTFT over NIM's 500ms-1s — so every probe round elected Groq as PRIMARY across all 3 chains. Result: every chat call hit Groq first, burned Groq's 100K daily TPD inside 50 turns, then started returning 429s. KI-087 changes election so it prefers ANY eligible NIM candidate over ALL non-NIM candidates. Within the NIM pool the standard score still picks the fastest healthy NIM model. Only when the NIM pool is empty does election fall through to Groq / OpenRouter as PRIMARY. BACKUP rule unchanged in spirit: cross-provider against PRIMARY. Rationale: NIM is the strategic free provider (ADR-019, no daily cap, 110+ models, single-key, $0); Groq has 100K daily TPD; OpenRouter charges real USD. Both should serve as emergency fallback only. |
 | D-001 | (multi) | ChromaDB HNSW bloat 3-layer prevention (ADR-029) |
 | D-002 | (LaunchAgent edit) | Three silently-failing LaunchAgent scripts fixed |
 | D-009 | `bcb7079` | Removed `tmp_*.py` debug files from repo root |

CLAUDE.md

@@ -18,18 +18,23 @@ This file is read by Claude Code (and any compatible AI tool) at the start of a
 - **Hands-free was removed entirely** in KI-027. Anything in the codebase still referring to it is stale.
 - **Bot TTS plays via the in-DOM `<audio>` element** inside `Message` (autoplay-on-mount via ref'd `useEffect`). Never use `new Audio(url).play()` — those detached instances are invisible to `document.querySelectorAll("audio").pause()` in the barge-in handler.
 
-## LLM stack (ADR-019 + ADR-026 → ADR-031 supersession) — KI-080
-
-Every LLM role is a `NimChainLLM` fallback chain, NOT a hardcoded single model. Chains preserve brain↔judge family diversity (Qwen brain ↔ Mistral judge) so failovers can't accidentally produce circular grading.
-
-- **Probe-driven sticky primary election (KI-080, [ADR-031](70-docs/60-decisions/ADR-031-sticky-primary-election.md)).** All three chains (`BRAIN_CHAIN`, `FAST_BRAIN_CHAIN`, `JUDGE_CHAIN`) elect a sticky PRIMARY + provider-diverse BACKUP from a background probe. `backend/llm_health.py` runs a 60s probe loop that scores every candidate on (latency × success rate) and writes the current election to process state. `NimChainLLM.chat()` no longer iterates every candidate per call — it calls PRIMARY once; on real-time failure it falls to BACKUP (cross-provider by construction) and triggers an immediate probe refresh. **Per-turn LLM call count: 1 (most cases) or 2 (primary fails real-time → backup + re-probe).** Pre-KI-080 worst case under sustained NIM concurrency throttling was 5-6 NIM calls per turn, all queued and timing out.
-- **KI-025's 50/50 NIM ↔ Groq rotation ([ADR-026](70-docs/60-decisions/ADR-026-provider-load-balancing.md)) is deprecated.** `_balanced_brain_chain` is retained in `backend/providers/nvidia_nim_llm.py` behind a feature flag for one-release rollback but is bypassed by default; the probe-driven election picks the actually-faster candidate dynamically instead of a fixed coin flip.
-- **Cold-start fallback.** Before the first probe completes (process restart, HF Space rebuild), `chain[0]` is the initial primary and `chain[1]` is the initial backup; the probe overwrites both within 60s.
-- **Brain / fast-brain / judge primaries in steady state** are currently **NIM Qwen 80B** (heavy brain), **NIM Nemotron Nano 30B** (fast brain, ~1.6s TTFT), **Mistral Large 3 675B** (judge — different family from brain → non-circular grading). These are the typical probe winners but are no longer hardcoded — the elected primary follows live latency × success scores.
-- **KI-079 escalation as last bite.** If both PRIMARY and BACKUP fail in a single fact-find turn, orchestrator retries once on `BRAIN_CHAIN` (heavy brain, 35s budget) before falling to `_canonical_fallback`. Final guardrail; fires after probe + backup are exhausted, not before.
-- **STT/TTS/Translator** = Sarvam (Saarika v2.5 / Bulbul v2 / Sarvam-M).
-- **Embeddings** = local BGE-small-en-v1.5 (`backend/providers/local_embeddings.py`). Voyage is configured in `.env` for ingest if needed but not on the hot path.
-- **Chain budgets:** brain 20s × 35s total, fast-brain 12s × 22s total, judge 30s × 75s total. Per-link timeout is dynamically clipped to remaining budget. With KI-080, only PRIMARY + BACKUP consume the budget in the common case — leaves headroom for KI-079 escalation.
+## LLM stack (ADR-019 + ADR-026 → ADR-031 + ADR-032) — KI-080 → KI-087
+
+Every LLM role is a `NimChainLLM` candidate pool, NOT a hardcoded single model. End-to-end spec: [ADR-032](70-docs/60-decisions/ADR-032-llm-chain-architecture.md). Chains preserve brain ↔ judge family diversity (Qwen brain ↔ Mistral judge) so failovers can't accidentally produce circular grading.
+
+- **Probe-driven sticky primary election (KI-080, [ADR-031](70-docs/60-decisions/ADR-031-sticky-primary-election.md)).** All three chains (`BRAIN_CHAIN`, `FAST_BRAIN_CHAIN`, `JUDGE_CHAIN`) elect a sticky PRIMARY + provider-diverse BACKUP from a background probe. `backend/llm_health.py` scores every candidate on `(1 / max(50, latency_ms)) * success_rate` and writes the current election to process state. `NimChainLLM.chat()` calls PRIMARY once; on real-time failure it falls to BACKUP (cross-provider by construction) and triggers an immediate probe refresh. **Per-turn LLM call count: 1 (most cases) or 2 (PRIMARY fails real-time → BACKUP).** Pre-KI-080 worst case was 5-6 NIM calls per turn, all queued and timing out.
+- **NIM-first election preference (KI-087, `d90f8c0`).** Election prefers ANY eligible NIM candidate over ALL non-NIM candidates. Within the NIM pool, score still picks the fastest healthy NIM model. Only when the NIM pool is empty (every NIM model down, throttled, or quota-exhausted) does election fall through to Groq / OpenRouter as PRIMARY. Rationale: NIM is the strategic free provider ($0, no daily cap, 110+ models); Groq has 100K tokens/day; OpenRouter charges real USD. Both serve as emergency fallback only.
+- **Probe cadence + per-phase timeouts (KI-084, `119e0fd`).** Probe loop ticks at `PROBE_INTERVAL_SEC = 300s` (was 60s ��� raised so probe-driven token spend stays inside Groq's 100K/day free-tier cap). Probe `max_tokens` cut `5 → 1`. Every chat call uses explicit `httpx.Timeout(connect=2, read=12, write=2, pool=2)` so a stuck NIM pool releases its TCP socket independently of the outer `asyncio.wait_for`, preventing NIM concurrency-slot leaks across PRIMARY → BACKUP. Rate-limit failures (HTTP 429 / `RateLimit` body) get a **1h sin-bin** (`DEGRADE_DURATION_LONG_S = 3600s`) instead of the 30s transient window — free-tier daily quotas don't reset in 30 seconds.
+- **Proactive credit gating (KI-085, `8fc7979`).** Election is gated by `is_alive AND has_credits` so quota-exhausted candidates are excluded BEFORE the user hits a 429. Signal sources: Groq response headers (`x-ratelimit-remaining-tokens-day`, low-water 5,000 tokens); OpenRouter `GET /api/v1/credits` polled every 10 min (low-water $0.05); NIM local 60-second rate-meter (gate at 35-of-40 req/min, headroom 5). Closes the one-turn reactive gap KI-084 alone leaves.
+- **HF Space secrets (KI-081, no commit; HF Space env secrets push).** `GROQ_API_KEY` + `OPENROUTER_API_KEY` pushed to the Space repository secrets so KI-080 cross-provider election candidates have working keys in production. Pre-KI-081 only `NVIDIA_NIM_API_KEY` was set on the Space; the elector marked Groq + OR as `no_api_key` and election degraded to NIM-only candidates.
+- **Admin telemetry (KI-086, `d90f8c0`).** `GET /api/admin/llm-health` returns `{chains, candidates, recent_turns, snapshot_ts}` with per-chain elected primary/backup, per-candidate health + credits + degraded-until, and last 20 turn outcomes. Surfaced in the admin "LLM Chain" tab with auto-refresh every 30s.
+- **KI-025's 50/50 NIM ↔ Groq rotation ([ADR-026](70-docs/60-decisions/ADR-026-provider-load-balancing.md)) is deprecated** — `_balanced_brain_chain` retained behind a feature flag for one-release rollback; the probe-driven election picks the actually-faster candidate dynamically.
+- **Cold-start fallback.** Before the first probe completes (process restart, HF Space rebuild), `chain[0]` is the initial primary and `chain[1]` (preferring a different provider) is the initial backup. The probe loop runs immediately on startup; OpenRouter credits are polled on startup so the elector has a non-None USD balance before the first chat call.
+- **Brain / fast-brain / judge primaries in steady state** are typically **NIM Qwen 3-Next 80B** (heavy brain, KI-087 NIM-first), **NIM Nemotron Nano 30B** (fast brain), and **Mistral Large 3 675B** (judge). Not hardcoded — the elected primary follows live `latency × success_rate × credits_available` with NIM-first preference.
+- **KI-079 escalation as last bite (`87ee522`).** If both PRIMARY and BACKUP fail in a single fact-find turn, orchestrator retries once on `BRAIN_CHAIN` (heavy brain, `_TIMEOUT_S_ESCALATION = 15s`, 35s chain budget) before falling to `_canonical_fallback` (KI-072 / KI-074 greedy slot capture). Worst-case wall-clock before canonical: 25s FAST + 15s heavy = 40s.
+- **Chain budgets:** brain 20s × 35s total, fast-brain 12s × 22s total, judge 30s × 75s total. With KI-080 only PRIMARY + BACKUP consume budget in the common case — leaves headroom for KI-079 escalation. KI-084 per-phase httpx timeouts are nested inside these budgets.
+- **STT/TTS/Translator** = Sarvam (Saarika v2.5 / Bulbul v2 / Sarvam-M). **Embeddings** = local BGE-small-en-v1.5.
+- **Provider keys.** `NVIDIA_NIM_API_KEY` + `GROQ_API_KEY` + `OPENROUTER_API_KEY` required in `.env` (local) and HF Space environment (production — KI-081).
 
 ## Fact-find loop (ADR-030, supersedes ADR-027) — KI-070
 

backend/providers/README.md

@@ -31,7 +31,37 @@ Every external model is fronted by a small typed client here. The orchestrator a
 | Fast brain | 12 | 22 | `nvidia_nim_llm.py::get_fast_brain_llm` |
 | Judge | 30 | 75 | `nvidia_nim_llm.py::get_judge_llm` |
 
-Per-link timeout is dynamically clipped to remaining budget.
+Per-link timeout is dynamically clipped to remaining budget. KI-084 explicit per-phase `httpx.Timeout(connect=2, read=<per-link>, write=2, pool=2)` is nested inside.
+
+## Credit-aware election (KI-085)
+
+Beyond liveness, election in `backend/llm_health.py` is gated on `credits_remaining > credits_low_water` per candidate so quota-exhausted providers are excluded BEFORE producing a user-facing 429. KI-087 further prefers NIM as primary; Groq/OpenRouter serve as emergency fallback.
+
+### Per-provider signal sources
+
+| Provider | Producer | Signal | Unit | Low-water |
+|---|---|---|---|---|
+| **Groq** | `update_credits_from_groq` from `groq_llm.py::chat` | `x-ratelimit-remaining-tokens-day` header + `x-ratelimit-reset-tokens-day` for reset | `tokens_day` | **5,000** (one fact-find round-trip ~2.4K + margin) |
+| **OpenRouter** | `poll_openrouter_credits` (every 10 min) + `update_credits_from_openrouter_headers` per-call fallback | `GET /api/v1/credits` → `{total_credits, total_usage}` | `usd_balance` | **$0.05** USD |
+| **NIM** | `record_nim_call` from `NimChainLLM._try` | Local 60s deque of monotonic timestamps per model | `requests_min` (remaining in current 60s window) | **5.0** slots (gate at 35/40 req/min) |
+
+### Election predicate
+
+In `_is_election_eligible(h, now_mono)`:
+
+```python
+if h.credits_reset_at is not None and now_mono >= h.credits_reset_at:
+    return True                                # quota already reset
+if h.credits_remaining is None:                # cold-start permissive
+    return True
+return h.credits_remaining > h.credits_low_water
+```
+
+### Cold-start
+
+`credits_remaining = None` is permissive — fresh process restarts don't grind to a halt before the first call has stamped credit state. OpenRouter poll fires on startup so the USD balance is non-None within seconds.
+
+See [ADR-032](../../70-docs/60-decisions/ADR-032-llm-chain-architecture.md) §6 for the full table and §12 for the operational runbook.
 
 ## Related