Spaces:
Sleeping
fix(orchestrator): KI-091 — skip profile_extractor + faithfulness judge on fact-find turns (drop dependent LLM calls on saturated chains)
Browse filesLive admin /api/admin/llm-health showed BRAIN_CHAIN + JUDGE_CHAIN
credit_exhausted=TRUE while FAST_BRAIN_CHAIN credit_exhausted=FALSE. The
fact_find_brain runs on FAST_BRAIN_CHAIN and was succeeding, but the
orchestrator's same-turn profile_extractor (BRAIN_CHAIN) and faithfulness
judge (JUDGE_CHAIN) calls were hanging on the saturated chains —
profile_extractor telemetry showed 10822ms / 15728ms / 16139ms / 20393ms
on consecutive turns; a single judge call ran for 733760ms (12 minutes!)
before completing.
Turn-1 in the user screenshot succeeded because both dependent calls
happened to come in under budget; turns 2-3 fell back to the canonical
GRAPH[k] prompt because the 25s outer wait_for tripped on the dependent
calls — fact_find_brain itself was never the killer.
Two gates added in backend/orchestrator.py:
Gate 1 — extract_profile_updates:
_in_fact_find_now = bool(in_fact_find_continuation) or intent == "fact_find"
_run_extractor = (
not _skip_extract
and bool(getattr(session, "free_form_session", False))
and not _in_fact_find_now
)
Only runs when session.free_form_session=True AND not in a fact-find
continuation. Safe because fact_find_brain's <FF>{"captured":...}
trailer already extracts every slot upstream (early-return block).
Gate 2 — check_faithfulness:
_skip_judge = (intent == "fact_find") or bool(in_fact_find_continuation)
if _skip_judge:
verdict = FaithfulnessVerdict(passed=True, reasons=["ki091_skip_on_fact_find"])
else:
verdict = await check_faithfulness(...)
Fact-find prose doesn't cite policy text — no factual claim to grade.
Expected impact: 20-25s shaved off per-turn latency on fact-find turns
when chains are saturated; 2-3s saved even on healthy chains. Removes
the dependence on BRAIN_CHAIN + JUDGE_CHAIN for the fact-find UX path.
Tests:
tests/test_routing_regression.py 15 passed
tests/test_credits_election.py 12 passed
DO NOT TOUCH (per KI-091 brief):
- backend/fact_find_brain.py
- backend/llm_health.py
- backend/providers/*.py
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- backend/orchestrator.py +46 -9
|
@@ -515,12 +515,35 @@ async def handle_turn(
|
|
| 515 |
# request per question without contributing to grading. Skipping it cuts
|
| 516 |
# ~25% off eval wall time at zero quality cost. Production runs (env unset)
|
| 517 |
# behaviour is identical to before.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 518 |
profile_updates_applied: dict = {}
|
| 519 |
import os as _os_pe
|
| 520 |
_skip_extract = _os_pe.environ.get("INSURANCE_BOT_SKIP_PROFILE_EXTRACTOR") == "1"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 521 |
try:
|
| 522 |
-
if
|
| 523 |
-
extracted = None # eval mode — bypass LLM extractor
|
| 524 |
else:
|
| 525 |
from backend.profile_extractor import extract_profile_updates
|
| 526 |
extracted = await extract_profile_updates(user_text, session.profile)
|
|
@@ -618,13 +641,27 @@ async def handle_turn(
|
|
| 618 |
# 5. FAITHFULNESS GATE — every reply runs through 4-gate verification.
|
| 619 |
# If any gate fails, replace the reply with a safe refusal. The original
|
| 620 |
# blocked reply is logged to logs/hallucinations.jsonl for audit.
|
| 621 |
-
|
| 622 |
-
|
| 623 |
-
|
| 624 |
-
|
| 625 |
-
|
| 626 |
-
|
| 627 |
-
)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 628 |
|
| 629 |
# 5a. CROSS-CHECK RETRY — if faithfulness blocked AND the failure isn't
|
| 630 |
# Gate 1 (no evidence at all), retry with a DIFFERENT-ARCHITECTURE NIM
|
|
|
|
| 515 |
# request per question without contributing to grading. Skipping it cuts
|
| 516 |
# ~25% off eval wall time at zero quality cost. Production runs (env unset)
|
| 517 |
# behaviour is identical to before.
|
| 518 |
+
#
|
| 519 |
+
# KI-091 (2026-05-15) — fact-find-turn skip. Live admin /llm-health showed
|
| 520 |
+
# BRAIN_CHAIN + JUDGE_CHAIN credit_exhausted while FAST_BRAIN_CHAIN was
|
| 521 |
+
# healthy. The fact_find_brain (FAST_BRAIN_CHAIN) succeeded but per-turn
|
| 522 |
+
# telemetry showed profile_extractor calls hitting 10-20s on the saturated
|
| 523 |
+
# heavy brain, which combined with the 25s _TIMEOUT_S in the fact-find
|
| 524 |
+
# path was tripping outer wait_for and emitting canonical fallback.
|
| 525 |
+
# Skipping the extractor on fact-find turns is safe: fact_find_brain's
|
| 526 |
+
# <FF>{"captured": ...} trailer already extracts every slot it needs and
|
| 527 |
+
# populated session.profile upstream (see the early-return block above).
|
| 528 |
+
# Gate: only run extractor when we're in free-form AND not in a fact-find
|
| 529 |
+
# continuation.
|
| 530 |
profile_updates_applied: dict = {}
|
| 531 |
import os as _os_pe
|
| 532 |
_skip_extract = _os_pe.environ.get("INSURANCE_BOT_SKIP_PROFILE_EXTRACTOR") == "1"
|
| 533 |
+
# KI-091 — fact-find branch already returned at line ~499. Anything past
|
| 534 |
+
# this point is free-form OR an explicit-question turn. The flag pair
|
| 535 |
+
# below is the belt-and-braces gate: only call the extractor when we're
|
| 536 |
+
# genuinely in free-form chat (session.free_form_session=True) and not
|
| 537 |
+
# mid-fact-find-continuation. Eval mode (env var) still wins.
|
| 538 |
+
_in_fact_find_now = bool(in_fact_find_continuation) or intent == "fact_find"
|
| 539 |
+
_run_extractor = (
|
| 540 |
+
not _skip_extract
|
| 541 |
+
and bool(getattr(session, "free_form_session", False))
|
| 542 |
+
and not _in_fact_find_now
|
| 543 |
+
)
|
| 544 |
try:
|
| 545 |
+
if not _run_extractor:
|
| 546 |
+
extracted = None # eval mode OR fact-find turn — bypass LLM extractor
|
| 547 |
else:
|
| 548 |
from backend.profile_extractor import extract_profile_updates
|
| 549 |
extracted = await extract_profile_updates(user_text, session.profile)
|
|
|
|
| 641 |
# 5. FAITHFULNESS GATE — every reply runs through 4-gate verification.
|
| 642 |
# If any gate fails, replace the reply with a safe refusal. The original
|
| 643 |
# blocked reply is logged to logs/hallucinations.jsonl for audit.
|
| 644 |
+
#
|
| 645 |
+
# KI-091 (2026-05-15) — skip on fact-find turns. The judge LLM runs on
|
| 646 |
+
# JUDGE_CHAIN, which the admin /api/admin/llm-health endpoint just showed
|
| 647 |
+
# as credit_exhausted=TRUE. Fact-find replies don't cite policy text and
|
| 648 |
+
# carry no factual claim about coverage/exclusions/waiting-periods — the
|
| 649 |
+
# judge has nothing to grade. Running it on a saturated chain hung one
|
| 650 |
+
# production call for 12 minutes (733760ms). Fact-find prose is generated
|
| 651 |
+
# by fact_find_brain which has its own slot-extraction validation; the
|
| 652 |
+
# judge gate is additive overhead with negative production value here.
|
| 653 |
+
# Gate: skip if intent=='fact_find' OR we're mid-fact-find continuation.
|
| 654 |
+
_skip_judge = (intent == "fact_find") or bool(in_fact_find_continuation)
|
| 655 |
+
if _skip_judge:
|
| 656 |
+
verdict = FaithfulnessVerdict(passed=True, reasons=["ki091_skip_on_fact_find"])
|
| 657 |
+
else:
|
| 658 |
+
verdict: FaithfulnessVerdict = await check_faithfulness(
|
| 659 |
+
reply=reply,
|
| 660 |
+
chunks=chunks,
|
| 661 |
+
user_text=user_text,
|
| 662 |
+
run_llm_judge=True,
|
| 663 |
+
brain_model_used=brain_model_actual,
|
| 664 |
+
)
|
| 665 |
|
| 666 |
# 5a. CROSS-CHECK RETRY — if faithfulness blocked AND the failure isn't
|
| 667 |
# Gate 1 (no evidence at all), retry with a DIFFERENT-ARCHITECTURE NIM
|