docs: KI-088 + KI-089 + KI-090 — audit rows + CLAUDE.md sync (semaphore / test-fix / lenient parser)

80-audit/ENTERPRISE_AUDIT.md

@@ -273,6 +273,9 @@ The full eval was sending **every** QA question to `needs_finder`. Sample bot an
 | KI-085 | `8fc7979` | **Proactive credit tracking — closes the reactive-only gap KI-084 leaves.** KI-084 demotes a candidate for 1h AFTER a 429 hits, costing one user-facing failover turn per dead quota. KI-085 promotes `llm_health` from liveness-only to liveness-AND-credits so election excludes quota-exhausted candidates BEFORE the user gets stuck behind a 429. Three signal sources: (1) Groq response headers `x-ratelimit-remaining-tokens-day` + `x-ratelimit-reset-tokens-day` (low-water 5K tokens); (2) OpenRouter `/api/v1/credits` polled every 10 min from probe loop, plus per-call header fallback (low-water $0.05); (3) NIM local 60s rate-meter, gate at 35-of-40 req/min (headroom 5). Election adds `_has_credits(h, now_mono)` to eligibility predicate. Admin `status_summary` extended with `credits_remaining` / `credits_unit` / `credits_low_water` per model. 11/11 inline tests pass + routing_regression 15/15. |
 | KI-086 | `d90f8c0` (bundled with KI-087) | **Admin "LLM Health & Credits" tab.** New `GET /api/admin/llm-health` endpoint returns `{chains, candidates, recent_turns, snapshot_ts}` JSON: per-chain elected PRIMARY + BACKUP with snapshots, per-candidate health grid with credits + degraded-until, last 20 turn outcomes from `40-data/llm_usage.jsonl`. Same `_check_admin` IP-allowlist + password gate as other admin endpoints. Frontend extends the existing "LLM Chain" tab in `frontend/public/admin/llm-control.html` with three sections: (A) per-chain election cards, (B) candidate health table, (C) recent turns table. Auto-polls every 30s while tab is active. Operator now sees at-a-glance which LLM is in use where, why a candidate is gated out, and how the election state evolves. |
 | **KI-087** | **`d90f8c0`** | **NIM-first election preference.** Pre-KI-087 election scored purely by `latency × success_rate`, which consistently favoured Groq's 161ms LPU TTFT over NIM's 500ms-1s — so every probe round elected Groq as PRIMARY across all 3 chains. Result: every chat call hit Groq first, burned Groq's 100K daily TPD inside 50 turns, then started returning 429s. KI-087 changes election so it prefers ANY eligible NIM candidate over ALL non-NIM candidates. Within the NIM pool the standard score still picks the fastest healthy NIM model. Only when the NIM pool is empty does election fall through to Groq / OpenRouter as PRIMARY. BACKUP rule unchanged in spirit: cross-provider against PRIMARY. Rationale: NIM is the strategic free provider (ADR-019, no daily cap, 110+ models, single-key, $0); Groq has 100K daily TPD; OpenRouter charges real USD. Both should serve as emergency fallback only. |
+| **KI-088** | **`14ee008`** | **NIM concurrency semaphore + serial probe + dropped inner retry.** Pre-KI-088 the process could fire 6+ concurrent NIM HTTP calls (probe burst `asyncio.gather` across 6 candidates + admin pollers + per-user turns), self-saturating the NIM endpoint and producing `timeout_after_escalation` failures at 41s wall-clock — the NIM endpoint serialises internally so every overlap added pure queueing latency. Three changes in one commit. (1) Module-level `asyncio.Semaphore(2)` at `backend/nvidia_nim_llm.py:104` wraps every `httpx.post` to `integrate.api.nvidia.com` so the entire process never has more than 2 NIM requests in flight simultaneously, regardless of source — probe loop, admin polls, and per-user turns all serialise through the same semaphore. (2) `backend/llm_health.py::probe_all` changed from `asyncio.gather(...)` to a serial `for m in models:` loop so the 6-NIM probe burst becomes a 1-slot trickle over ~12s instead of contending with live user turns. (3) The 4-attempt exponential-backoff inner retry inside `NvidiaNimLLM.chat()` was deleted — KI-080 sticky-primary election + KI-079 heavy-chain escalation already handle failover at the right layer, so the inner retry only amplified the self-saturation. Live verify post-deploy: failure mode flipped from `fallback:timeout_after_escalation` at 41s → `fallback:no_trailer` at 4-10s. NIM concurrency bottleneck closed; new parser-side bottleneck surfaced and is addressed in KI-090. |
+| **KI-089** | **`8a87526`** | **Credits-election test fix + paired NIM-empty test.** `test_groq_above_water_picked_in_election` had been failing on `main` since KI-087 landed: it asserted Groq wins election on raw latency (161ms LPU TTFT), but KI-087 inverted election to prefer any eligible NIM candidate over all non-NIM candidates regardless of latency. Replaced with two paired tests that pin KI-087's invariant explicitly. (1) `test_nim_preferred_over_faster_groq_when_eligible` — when an eligible NIM candidate exists, election picks it as PRIMARY even though Groq is measurably faster. (2) `test_groq_picked_when_nim_pool_empty` — when every NIM candidate is dead / throttled / `no_credits`, election correctly falls through to Groq as PRIMARY. Together the pair pin both halves of the KI-087 contract (NIM-first AND fallthrough-when-NIM-empty) so a future regression that breaks either half fails loudly. Full inline test count: credits-election 12/12 pass, routing-regression 15/15 pass. |
+| **KI-090** | **`11cf4b3`** | **Lenient FF-block parser.** Post-KI-088 live probe showed ~70% of brain calls returned successfully in 4-10s (NIM concurrency fix surfaced the real bottleneck) but `_parse_ff_block` rejected the reply with `fallback:no_trailer` because the brain had dropped the literal `<FF>...</FF>` tags around its JSON tail. Real LLMs under load (Qwen 3-Next 80B, Nemotron Nano 30B, Groq Llama-3.3-70B) regularly drop the wrapper even when the structured payload is otherwise contract-compliant. New `_parse_ff_block` tries three strategies in order: (a) strict `<FF>{...}</FF>` (the original contract); (b) fenced ```` ```json {...} ``` ```` (common LLM habit); (c) bare `{...}` JSON object at the end of the reply. Each candidate must `json.loads` cleanly AND contain at least one contract key (`captured` / `slot_driving` / `complete`) before it counts — prevents false positives from prose that happens to contain `{...}`. `_strip_ff_block` mirrors the three strategies in reverse so prose-only output to the user never leaks the structured metadata block. Inline tests: 7/7 parse tests + 7/7 strip tests pass. Brain success rate climbs from ~30% (post-KI-088 baseline blocked by parser) toward ~95% (NIM concurrency healthy + parser accepts contract-compliant tails regardless of wrapper). |
 | D-001 | (multi) | ChromaDB HNSW bloat 3-layer prevention (ADR-029) |
 | D-002 | (LaunchAgent edit) | Three silently-failing LaunchAgent scripts fixed |
 | D-009 | `bcb7079` | Removed `tmp_*.py` debug files from repo root |

CLAUDE.md

@@ -32,6 +32,8 @@ Every LLM role is a `NimChainLLM` candidate pool, NOT a hardcoded single model.
 - **Cold-start fallback.** Before the first probe completes (process restart, HF Space rebuild), `chain[0]` is the initial primary and `chain[1]` (preferring a different provider) is the initial backup. The probe loop runs immediately on startup; OpenRouter credits are polled on startup so the elector has a non-None USD balance before the first chat call.
 - **Brain / fast-brain / judge primaries in steady state** are typically **NIM Qwen 3-Next 80B** (heavy brain, KI-087 NIM-first), **NIM Nemotron Nano 30B** (fast brain), and **Mistral Large 3 675B** (judge). Not hardcoded — the elected primary follows live `latency × success_rate × credits_available` with NIM-first preference.
 - **KI-079 escalation as last bite (`87ee522`).** If both PRIMARY and BACKUP fail in a single fact-find turn, orchestrator retries once on `BRAIN_CHAIN` (heavy brain, `_TIMEOUT_S_ESCALATION = 15s`, 35s chain budget) before falling to `_canonical_fallback` (KI-072 / KI-074 greedy slot capture). Worst-case wall-clock before canonical: 25s FAST + 15s heavy = 40s.
+- **NIM concurrency semaphore + serial probe (KI-088, `14ee008`).** Module-level `asyncio.Semaphore(2)` wraps every NIM HTTP call so our process never has >2 NIM requests in flight simultaneously, regardless of source (probe loop + admin polls + per-user turns all serialise through the same semaphore). Probe loop changed parallel→serial so the 6-NIM probe burst becomes a 1-slot trickle over ~12s. Inner 4-attempt exponential-backoff retry deleted from `NvidiaNimLLM.chat()` — KI-080 election + KI-079 escalation now handle failover. Result: latency-based failures (41s timeouts under self-saturation) dropped to zero; replaced by a parser-side bottleneck (KI-090).
+- **Lenient FF-block parser (KI-090, `11cf4b3`).** Real LLMs (Qwen, Nemotron under load, Groq Llama-3.3) sometimes drop the literal `<FF>...</FF>` tags around their JSON tail. Pre-KI-090 those replies fell to `fallback:no_trailer` even though the brain had produced a perfectly valid structured response. Now `_parse_ff_block` tries strict → fenced ```` ```json``` ```` → bare-JSON-tail, each candidate validated by presence of a contract key (`captured` / `slot_driving` / `complete`). `_strip_ff_block` mirrors the strategies so prose-only output never leaks structured metadata.
 - **Chain budgets:** brain 20s × 35s total, fast-brain 12s × 22s total, judge 30s × 75s total. With KI-080 only PRIMARY + BACKUP consume budget in the common case — leaves headroom for KI-079 escalation. KI-084 per-phase httpx timeouts are nested inside these budgets.
 - **STT/TTS/Translator** = Sarvam (Saarika v2.5 / Bulbul v2 / Sarvam-M). **Embeddings** = local BGE-small-en-v1.5.
 - **Provider keys.** `NVIDIA_NIM_API_KEY` + `GROQ_API_KEY` + `OPENROUTER_API_KEY` required in `.env` (local) and HF Space environment (production — KI-081).