Spaces:
Sleeping
docs: final sync — backend/README post-KI-070 + ADR-032 cross-links in README §4.3 + ENTERPRISE_AUDIT KI-080 row
Browse filesThree small completeness fixes surfaced by the end-of-session audit:
- backend/README.md — `needs_finder.py` row now correctly described
as "safeguard fallback only" post-KI-070 (was still labelled the
primary fact-find driver, pointing to ADR-027). Added new row for
`fact_find_brain.py` (the actual KI-070 driver). Marked
`question_paraphraser.py` as deleted.
- README.md §4.3 — opening sentence reframed from "fallback chain"
to "candidate pool with probe-driven sticky primary election";
ADR-031 + ADR-032 cross-links added inline so a reviewer can
jump straight to the architecture reference.
- ENTERPRISE_AUDIT.md KI-080 row — added inline link to ADR-032 so
the audit table's KI-080 entry routes through to the end-to-end
architecture spec (was already linking ADR-031, now both).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- 80-audit/ENTERPRISE_AUDIT.md +1 -1
- README.md +1 -1
|
@@ -267,7 +267,7 @@ The full eval was sending **every** QA question to `needs_finder`. Sample bot an
|
|
| 267 |
| KI-077 | `2bb3898` | "Build your profile" panel: added Name input field at top with "captured from chat" badge when populated. Backend `/api/profile/completeness` + `/api/profile` POST + `UserProfile` TypeScript type all extended with `name`. Panel pre-fills every field from the session's captured chat state via existing `initialProfile`. New `useEffect` keeps panel in sync when chat captures fields while panel is open. On Save, name persists to the named-profile JSON store (KI-040/062) so the user is auto-recognised on return visits. |
|
| 268 |
| KI-078 | `078ff45` | LLM chain hardening: per-link timeout 12s → 6s so chain can try 3-4 candidates inside `total_budget_s=22s` instead of 1. Narrowed `except Exception` to re-raise `CancelledError`/`KeyboardInterrupt`/`SystemExit` so `asyncio.wait_for` actually bubbles. New `_fallback_reason` stamped on `FactFindOutcome` and surfaced as `fact_find_brain::fallback:timeout` / `:no_trailer` / `:empty_reply` / `:llm_error` in `TurnResult.brain_used` for production telemetry. |
|
| 269 |
| KI-079 | `87ee522` | Two-layer chain hardening for fact-find. (1) FAST_BRAIN_CHAIN reorder: Groq Llama-3.3-70B promoted from position #5 to #2 (right after Nemotron primary) so cross-provider fallback is reached in ~6-7s of budget instead of ~20s. (2) Heavy-chain escalation: when `drive_fact_find()` raises `TimeoutError` on fast-brain, orchestrator retries once on `BRAIN_CHAIN` (Qwen 80B primary, 35s budget) before falling to `_canonical_fallback`. Adds `fallback:timeout_after_escalation` / `:llm_error_after_escalation` to telemetry vocabulary. Now serves as the last-bite safety net once KI-080's primary election is in place. |
|
| 270 |
-
| **KI-080** | **`6159c54`** | **Sticky primary election for LLM chains ([ADR-031](../70-docs/60-decisions/ADR-031-sticky-primary-election.md)).** `NimChainLLM.chat()` refactored from "iterate every chain candidate sequentially per call" to "call the probe-elected PRIMARY once; on real-time failure, call the cross-provider BACKUP once and re-trigger the probe." `backend/llm_health.py` runs a 60s background probe loop that scores every candidate on (latency × success rate) and elects a sticky PRIMARY + provider-diverse BACKUP for each of `BRAIN_CHAIN` / `FAST_BRAIN_CHAIN` / `JUDGE_CHAIN`. **Per-turn LLM call count drops from 5-6 (sustained NIM degradation, every candidate queued + timed out) to 1 (most cases) or 2 (primary fails real-time → backup + re-probe).** ADR-026's `_balanced_brain_chain` (KI-025 50/50 NIM ↔ Groq rotation) is deprecated — the probe-driven election dynamically picks the actually-faster candidate instead of a fixed 50/50 coin. Code retained as a bypassed branch behind a feature flag for one-release rollback. Cold-start fallback (no probes complete yet) uses `chain[0]` as the initial primary. KI-079 escalation still applies as the last bite if both primary AND backup fail in the same turn. Inline tests: 7/7 OK (cold-start, lowest-latency primary, cross-provider backup, demote-on-failure, score-update, primary-success makes 1 call, primary-fail/backup-success makes exactly 2 calls). `tests/test_routing_regression.py` → 15/15 pass. |
|
| 271 |
| KI-081 | (no commit — HF Space env secrets) | Pushed `GROQ_API_KEY` + `OPENROUTER_API_KEY` to the HF Space repository secrets so the KI-080 cross-provider election candidates actually have working keys in production. Pre-KI-081 only `NVIDIA_NIM_API_KEY` was set on the Space; the elector would mark every Groq + OpenRouter candidate as `no_api_key` and election degraded to NIM-only candidates — defeating the cross-provider BACKUP invariant. |
|
| 272 |
| KI-084 | `119e0fd` | **LLM chain telemetry hardening + free-tier guards.** Four changes in one commit. (1) Probe cadence `PROBE_INTERVAL_SEC` raised 60s → **300s** — the prior cadence burned ~30-50K probe tokens/day on Groq alone, self-tripping Groq's 100K/day TPD free-tier cap. (2) `PROBE_MAX_TOKENS` cut 5 → **1** — same 200 envelope, ~50× less token spend per probe. (3) Explicit per-phase `httpx.Timeout(connect=2, read=self.timeout, write=2, pool=2)` on every chat call — previously `timeout=self.timeout` collapsed to a single read deadline so a stuck NIM pool could occupy the TCP connection past `asyncio.wait_for` cancellation, leaking NIM concurrency slots. (4) New `_classify_error` surfaces HTTP status codes explicitly (`Status429` vs `HTTPStatusError:503`); rate-limit failures get a **1-hour** sin-bin (`DEGRADE_DURATION_LONG_S = 3600s`) instead of the 30s transient window — free-tier daily quotas don't reset in 30 seconds. |
|
| 273 |
| KI-085 | `8fc7979` | **Proactive credit tracking — closes the reactive-only gap KI-084 leaves.** KI-084 demotes a candidate for 1h AFTER a 429 hits, costing one user-facing failover turn per dead quota. KI-085 promotes `llm_health` from liveness-only to liveness-AND-credits so election excludes quota-exhausted candidates BEFORE the user gets stuck behind a 429. Three signal sources: (1) Groq response headers `x-ratelimit-remaining-tokens-day` + `x-ratelimit-reset-tokens-day` (low-water 5K tokens); (2) OpenRouter `/api/v1/credits` polled every 10 min from probe loop, plus per-call header fallback (low-water $0.05); (3) NIM local 60s rate-meter, gate at 35-of-40 req/min (headroom 5). Election adds `_has_credits(h, now_mono)` to eligibility predicate. Admin `status_summary` extended with `credits_remaining` / `credits_unit` / `credits_low_water` per model. 11/11 inline tests pass + routing_regression 15/15. |
|
|
|
|
| 267 |
| KI-077 | `2bb3898` | "Build your profile" panel: added Name input field at top with "captured from chat" badge when populated. Backend `/api/profile/completeness` + `/api/profile` POST + `UserProfile` TypeScript type all extended with `name`. Panel pre-fills every field from the session's captured chat state via existing `initialProfile`. New `useEffect` keeps panel in sync when chat captures fields while panel is open. On Save, name persists to the named-profile JSON store (KI-040/062) so the user is auto-recognised on return visits. |
|
| 268 |
| KI-078 | `078ff45` | LLM chain hardening: per-link timeout 12s → 6s so chain can try 3-4 candidates inside `total_budget_s=22s` instead of 1. Narrowed `except Exception` to re-raise `CancelledError`/`KeyboardInterrupt`/`SystemExit` so `asyncio.wait_for` actually bubbles. New `_fallback_reason` stamped on `FactFindOutcome` and surfaced as `fact_find_brain::fallback:timeout` / `:no_trailer` / `:empty_reply` / `:llm_error` in `TurnResult.brain_used` for production telemetry. |
|
| 269 |
| KI-079 | `87ee522` | Two-layer chain hardening for fact-find. (1) FAST_BRAIN_CHAIN reorder: Groq Llama-3.3-70B promoted from position #5 to #2 (right after Nemotron primary) so cross-provider fallback is reached in ~6-7s of budget instead of ~20s. (2) Heavy-chain escalation: when `drive_fact_find()` raises `TimeoutError` on fast-brain, orchestrator retries once on `BRAIN_CHAIN` (Qwen 80B primary, 35s budget) before falling to `_canonical_fallback`. Adds `fallback:timeout_after_escalation` / `:llm_error_after_escalation` to telemetry vocabulary. Now serves as the last-bite safety net once KI-080's primary election is in place. |
|
| 270 |
+
| **KI-080** | **`6159c54`** | **Sticky primary election for LLM chains ([ADR-031](../70-docs/60-decisions/ADR-031-sticky-primary-election.md); end-to-end architecture spec in [ADR-032](../70-docs/60-decisions/ADR-032-llm-chain-architecture.md)).** `NimChainLLM.chat()` refactored from "iterate every chain candidate sequentially per call" to "call the probe-elected PRIMARY once; on real-time failure, call the cross-provider BACKUP once and re-trigger the probe." `backend/llm_health.py` runs a 60s background probe loop that scores every candidate on (latency × success rate) and elects a sticky PRIMARY + provider-diverse BACKUP for each of `BRAIN_CHAIN` / `FAST_BRAIN_CHAIN` / `JUDGE_CHAIN`. **Per-turn LLM call count drops from 5-6 (sustained NIM degradation, every candidate queued + timed out) to 1 (most cases) or 2 (primary fails real-time → backup + re-probe).** ADR-026's `_balanced_brain_chain` (KI-025 50/50 NIM ↔ Groq rotation) is deprecated — the probe-driven election dynamically picks the actually-faster candidate instead of a fixed 50/50 coin. Code retained as a bypassed branch behind a feature flag for one-release rollback. Cold-start fallback (no probes complete yet) uses `chain[0]` as the initial primary. KI-079 escalation still applies as the last bite if both primary AND backup fail in the same turn. Inline tests: 7/7 OK (cold-start, lowest-latency primary, cross-provider backup, demote-on-failure, score-update, primary-success makes 1 call, primary-fail/backup-success makes exactly 2 calls). `tests/test_routing_regression.py` → 15/15 pass. |
|
| 271 |
| KI-081 | (no commit — HF Space env secrets) | Pushed `GROQ_API_KEY` + `OPENROUTER_API_KEY` to the HF Space repository secrets so the KI-080 cross-provider election candidates actually have working keys in production. Pre-KI-081 only `NVIDIA_NIM_API_KEY` was set on the Space; the elector would mark every Groq + OpenRouter candidate as `no_api_key` and election degraded to NIM-only candidates — defeating the cross-provider BACKUP invariant. |
|
| 272 |
| KI-084 | `119e0fd` | **LLM chain telemetry hardening + free-tier guards.** Four changes in one commit. (1) Probe cadence `PROBE_INTERVAL_SEC` raised 60s → **300s** — the prior cadence burned ~30-50K probe tokens/day on Groq alone, self-tripping Groq's 100K/day TPD free-tier cap. (2) `PROBE_MAX_TOKENS` cut 5 → **1** — same 200 envelope, ~50× less token spend per probe. (3) Explicit per-phase `httpx.Timeout(connect=2, read=self.timeout, write=2, pool=2)` on every chat call — previously `timeout=self.timeout` collapsed to a single read deadline so a stuck NIM pool could occupy the TCP connection past `asyncio.wait_for` cancellation, leaking NIM concurrency slots. (4) New `_classify_error` surfaces HTTP status codes explicitly (`Status429` vs `HTTPStatusError:503`); rate-limit failures get a **1-hour** sin-bin (`DEGRADE_DURATION_LONG_S = 3600s`) instead of the 30s transient window — free-tier daily quotas don't reset in 30 seconds. |
|
| 273 |
| KI-085 | `8fc7979` | **Proactive credit tracking — closes the reactive-only gap KI-084 leaves.** KI-084 demotes a candidate for 1h AFTER a 429 hits, costing one user-facing failover turn per dead quota. KI-085 promotes `llm_health` from liveness-only to liveness-AND-credits so election excludes quota-exhausted candidates BEFORE the user gets stuck behind a 429. Three signal sources: (1) Groq response headers `x-ratelimit-remaining-tokens-day` + `x-ratelimit-reset-tokens-day` (low-water 5K tokens); (2) OpenRouter `/api/v1/credits` polled every 10 min from probe loop, plus per-call header fallback (low-water $0.05); (3) NIM local 60s rate-meter, gate at 35-of-40 req/min (headroom 5). Election adds `_has_credits(h, now_mono)` to eligibility predicate. Admin `status_summary` extended with `credits_remaining` / `credits_unit` / `credits_low_water` per model. 11/11 inline tests pass + routing_regression 15/15. |
|
|
@@ -548,7 +548,7 @@ The bot is two flows running together — the customer's experience and the tech
|
|
| 548 |
|
| 549 |
### 4.3 Model stack
|
| 550 |
|
| 551 |
-
Every LLM role is served by a **
|
| 552 |
|
| 553 |
| Role | Primary | Fallback chain (in order) | Provider(s) | Why this primary |
|
| 554 |
|---|---|---|---|---|
|
|
|
|
| 548 |
|
| 549 |
### 4.3 Model stack
|
| 550 |
|
| 551 |
+
Every LLM role is served by a **candidate pool** of models elected over by a probe-driven sticky-primary scheme ([ADR-031](70-docs/60-decisions/ADR-031-sticky-primary-election.md), end-to-end spec in [ADR-032](70-docs/60-decisions/ADR-032-llm-chain-architecture.md)), never a hardcoded single model. Chains were curated to preserve brain ↔ judge family diversity (Qwen brain ↔ Mistral judge) so any failover still produces non-circular grading. Cross-provider entries (OpenRouter, Groq) sit inside every chain so the whole reasoning stack survives a full NIM regional outage.
|
| 552 |
|
| 553 |
| Role | Primary | Fallback chain (in order) | Provider(s) | Why this primary |
|
| 554 |
|---|---|---|---|---|
|