rohitsar567 Claude Opus 4.7 (1M context) commited on
Commit
9813994
·
1 Parent(s): f287dc6

fix(orchestrator): KI-091 — skip profile_extractor + faithfulness judge on fact-find turns (drop dependent LLM calls on saturated chains)

Browse files

Live admin /api/admin/llm-health showed BRAIN_CHAIN + JUDGE_CHAIN
credit_exhausted=TRUE while FAST_BRAIN_CHAIN credit_exhausted=FALSE. The
fact_find_brain runs on FAST_BRAIN_CHAIN and was succeeding, but the
orchestrator's same-turn profile_extractor (BRAIN_CHAIN) and faithfulness
judge (JUDGE_CHAIN) calls were hanging on the saturated chains —
profile_extractor telemetry showed 10822ms / 15728ms / 16139ms / 20393ms
on consecutive turns; a single judge call ran for 733760ms (12 minutes!)
before completing.

Turn-1 in the user screenshot succeeded because both dependent calls
happened to come in under budget; turns 2-3 fell back to the canonical
GRAPH[k] prompt because the 25s outer wait_for tripped on the dependent
calls — fact_find_brain itself was never the killer.

Two gates added in backend/orchestrator.py:

Gate 1 — extract_profile_updates:
_in_fact_find_now = bool(in_fact_find_continuation) or intent == "fact_find"
_run_extractor = (
not _skip_extract
and bool(getattr(session, "free_form_session", False))
and not _in_fact_find_now
)
Only runs when session.free_form_session=True AND not in a fact-find
continuation. Safe because fact_find_brain's <FF>{"captured":...}
trailer already extracts every slot upstream (early-return block).

Gate 2 — check_faithfulness:
_skip_judge = (intent == "fact_find") or bool(in_fact_find_continuation)
if _skip_judge:
verdict = FaithfulnessVerdict(passed=True, reasons=["ki091_skip_on_fact_find"])
else:
verdict = await check_faithfulness(...)
Fact-find prose doesn't cite policy text — no factual claim to grade.

Expected impact: 20-25s shaved off per-turn latency on fact-find turns
when chains are saturated; 2-3s saved even on healthy chains. Removes
the dependence on BRAIN_CHAIN + JUDGE_CHAIN for the fact-find UX path.

Tests:
tests/test_routing_regression.py 15 passed
tests/test_credits_election.py 12 passed

DO NOT TOUCH (per KI-091 brief):
- backend/fact_find_brain.py
- backend/llm_health.py
- backend/providers/*.py

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Files changed (1) hide show
  1. backend/orchestrator.py +46 -9
backend/orchestrator.py CHANGED
@@ -515,12 +515,35 @@ async def handle_turn(
515
  # request per question without contributing to grading. Skipping it cuts
516
  # ~25% off eval wall time at zero quality cost. Production runs (env unset)
517
  # behaviour is identical to before.
 
 
 
 
 
 
 
 
 
 
 
 
518
  profile_updates_applied: dict = {}
519
  import os as _os_pe
520
  _skip_extract = _os_pe.environ.get("INSURANCE_BOT_SKIP_PROFILE_EXTRACTOR") == "1"
 
 
 
 
 
 
 
 
 
 
 
521
  try:
522
- if _skip_extract:
523
- extracted = None # eval mode — bypass LLM extractor entirely
524
  else:
525
  from backend.profile_extractor import extract_profile_updates
526
  extracted = await extract_profile_updates(user_text, session.profile)
@@ -618,13 +641,27 @@ async def handle_turn(
618
  # 5. FAITHFULNESS GATE — every reply runs through 4-gate verification.
619
  # If any gate fails, replace the reply with a safe refusal. The original
620
  # blocked reply is logged to logs/hallucinations.jsonl for audit.
621
- verdict: FaithfulnessVerdict = await check_faithfulness(
622
- reply=reply,
623
- chunks=chunks,
624
- user_text=user_text,
625
- run_llm_judge=True,
626
- brain_model_used=brain_model_actual,
627
- )
 
 
 
 
 
 
 
 
 
 
 
 
 
 
628
 
629
  # 5a. CROSS-CHECK RETRY — if faithfulness blocked AND the failure isn't
630
  # Gate 1 (no evidence at all), retry with a DIFFERENT-ARCHITECTURE NIM
 
515
  # request per question without contributing to grading. Skipping it cuts
516
  # ~25% off eval wall time at zero quality cost. Production runs (env unset)
517
  # behaviour is identical to before.
518
+ #
519
+ # KI-091 (2026-05-15) — fact-find-turn skip. Live admin /llm-health showed
520
+ # BRAIN_CHAIN + JUDGE_CHAIN credit_exhausted while FAST_BRAIN_CHAIN was
521
+ # healthy. The fact_find_brain (FAST_BRAIN_CHAIN) succeeded but per-turn
522
+ # telemetry showed profile_extractor calls hitting 10-20s on the saturated
523
+ # heavy brain, which combined with the 25s _TIMEOUT_S in the fact-find
524
+ # path was tripping outer wait_for and emitting canonical fallback.
525
+ # Skipping the extractor on fact-find turns is safe: fact_find_brain's
526
+ # <FF>{"captured": ...} trailer already extracts every slot it needs and
527
+ # populated session.profile upstream (see the early-return block above).
528
+ # Gate: only run extractor when we're in free-form AND not in a fact-find
529
+ # continuation.
530
  profile_updates_applied: dict = {}
531
  import os as _os_pe
532
  _skip_extract = _os_pe.environ.get("INSURANCE_BOT_SKIP_PROFILE_EXTRACTOR") == "1"
533
+ # KI-091 — fact-find branch already returned at line ~499. Anything past
534
+ # this point is free-form OR an explicit-question turn. The flag pair
535
+ # below is the belt-and-braces gate: only call the extractor when we're
536
+ # genuinely in free-form chat (session.free_form_session=True) and not
537
+ # mid-fact-find-continuation. Eval mode (env var) still wins.
538
+ _in_fact_find_now = bool(in_fact_find_continuation) or intent == "fact_find"
539
+ _run_extractor = (
540
+ not _skip_extract
541
+ and bool(getattr(session, "free_form_session", False))
542
+ and not _in_fact_find_now
543
+ )
544
  try:
545
+ if not _run_extractor:
546
+ extracted = None # eval mode OR fact-find turn — bypass LLM extractor
547
  else:
548
  from backend.profile_extractor import extract_profile_updates
549
  extracted = await extract_profile_updates(user_text, session.profile)
 
641
  # 5. FAITHFULNESS GATE — every reply runs through 4-gate verification.
642
  # If any gate fails, replace the reply with a safe refusal. The original
643
  # blocked reply is logged to logs/hallucinations.jsonl for audit.
644
+ #
645
+ # KI-091 (2026-05-15) — skip on fact-find turns. The judge LLM runs on
646
+ # JUDGE_CHAIN, which the admin /api/admin/llm-health endpoint just showed
647
+ # as credit_exhausted=TRUE. Fact-find replies don't cite policy text and
648
+ # carry no factual claim about coverage/exclusions/waiting-periods — the
649
+ # judge has nothing to grade. Running it on a saturated chain hung one
650
+ # production call for 12 minutes (733760ms). Fact-find prose is generated
651
+ # by fact_find_brain which has its own slot-extraction validation; the
652
+ # judge gate is additive overhead with negative production value here.
653
+ # Gate: skip if intent=='fact_find' OR we're mid-fact-find continuation.
654
+ _skip_judge = (intent == "fact_find") or bool(in_fact_find_continuation)
655
+ if _skip_judge:
656
+ verdict = FaithfulnessVerdict(passed=True, reasons=["ki091_skip_on_fact_find"])
657
+ else:
658
+ verdict: FaithfulnessVerdict = await check_faithfulness(
659
+ reply=reply,
660
+ chunks=chunks,
661
+ user_text=user_text,
662
+ run_llm_judge=True,
663
+ brain_model_used=brain_model_actual,
664
+ )
665
 
666
  # 5a. CROSS-CHECK RETRY — if faithfulness blocked AND the failure isn't
667
  # Gate 1 (no evidence at all), retry with a DIFFERENT-ARCHITECTURE NIM