rohitsar567 Claude Opus 4.7 (1M context) commited on
Commit
3f462fa
·
1 Parent(s): 1b6d24d

fix(recommend+citations): KI-171 + KI-172 — recommendation flow unblocked

Browse files

KI-171 — Skip faithfulness judge on recommendation queries.

The faithfulness judge is designed for POLICY-FACT lookups ("does
Policy X cover Y?") not for GENERATIVE RECOMMENDATIONS ("which policy
fits me?"). After fact_find completed and the user asked "show me
top 3 policies", the judge rejected the synthesized recommendation
with "I'd rather not answer that without stronger evidence in the
policy documents I have. Could you rephrase, or narrow your question
to a specific policy?" — blocking the entire recommendation flow.

Added query-shape detector. _is_recommendation true when user_text
contains: recommend / suggest / best polic / top N / show me polic /
fits me / right for me / policies for me / which polic / good polic /
what polic / your suggestion / your recommendation. When true, skip
faithfulness with reason=ki171_skip_on_recommendation.

This mirrors the existing KI-091 skip-on-fact_find pattern.

KI-172 — Filter profile chunks from user-facing citations.

"User profile (rohit)" was appearing in the chat's CITED POLICIES
card. The profile chunk is correctly retrieved for LLM context (so
the bot knows the user's age/dependents/income) but is NOT a policy
citation and shouldn't display as one.

Filter `chunks` where `insurer_slug == 'profile'` (case-insensitive)
before building the citations list. The chunk still flows into the
brain prompt for context — just not into the user-visible citations.

VERIFICATION:
py_compile backend/orchestrator.py — clean.

DOCS CASCADE: still held (stash @0).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Files changed (1) hide show
  1. backend/orchestrator.py +25 -3
backend/orchestrator.py CHANGED
@@ -937,9 +937,25 @@ async def handle_turn(
937
  # by fact_find_brain which has its own slot-extraction validation; the
938
  # judge gate is additive overhead with negative production value here.
939
  # Gate: skip if intent=='fact_find' OR we're mid-fact-find continuation.
940
- _skip_judge = (intent == "fact_find") or bool(in_fact_find_continuation)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
941
  if _skip_judge:
942
- verdict = FaithfulnessVerdict(passed=True, reasons=["ki091_skip_on_fact_find"])
 
943
  else:
944
  verdict: FaithfulnessVerdict = await check_faithfulness(
945
  reply=reply,
@@ -999,7 +1015,12 @@ async def handle_turn(
999
  blocked = True
1000
  reply = verdict.suggested_reply or "I don't have grounded evidence for that. Could you rephrase?"
1001
 
1002
- # 6. Citations (derived from retrieved chunks)
 
 
 
 
 
1003
  citations = [
1004
  {
1005
  "policy_id": c.policy_id,
@@ -1011,6 +1032,7 @@ async def handle_turn(
1011
  "score": round(c.score, 3),
1012
  }
1013
  for c in chunks
 
1014
  ]
1015
 
1016
  # 7. INDIC CASCADE — translate the English reply back into Hinglish/Hindi,
 
937
  # by fact_find_brain which has its own slot-extraction validation; the
938
  # judge gate is additive overhead with negative production value here.
939
  # Gate: skip if intent=='fact_find' OR we're mid-fact-find continuation.
940
+ # KI-171 (2026-05-15) also skip faithfulness on RECOMMENDATION queries.
941
+ # The judge is designed for policy-fact lookups ("does Policy X cover Y?"),
942
+ # not for generative synthesis ("which policy fits me?"). Recommendation
943
+ # answers stitch together evidence from many policies plus the user's
944
+ # profile — the judge can't grade that fairly and currently blocks valid
945
+ # recommendation flows after fact_find completes. Detect via query shape.
946
+ _user_text_lc = (user_text or "").lower()
947
+ _is_recommendation = any(
948
+ kw in _user_text_lc for kw in (
949
+ "recommend", "suggest", "best polic", "top 3", "top three",
950
+ "top 5", "top five", "show me polic", "show me what", "fits me",
951
+ "right for me", "policies for me", "which polic", "good polic",
952
+ "what polic", "your suggestion", "your recommendation",
953
+ )
954
+ )
955
+ _skip_judge = (intent == "fact_find") or bool(in_fact_find_continuation) or _is_recommendation
956
  if _skip_judge:
957
+ skip_reason = "ki171_skip_on_recommendation" if _is_recommendation else "ki091_skip_on_fact_find"
958
+ verdict = FaithfulnessVerdict(passed=True, reasons=[skip_reason])
959
  else:
960
  verdict: FaithfulnessVerdict = await check_faithfulness(
961
  reply=reply,
 
1015
  blocked = True
1016
  reply = verdict.suggested_reply or "I don't have grounded evidence for that. Could you rephrase?"
1017
 
1018
+ # 6. Citations (derived from retrieved chunks).
1019
+ # KI-172 (2026-05-15) — exclude profile chunks from user-facing citations.
1020
+ # The profile chunk is retrieved alongside policy chunks so the LLM has
1021
+ # context about the user (age/dependents/income/etc.), but it is NOT a
1022
+ # "cited policy" and should not appear in the chat's "CITED POLICIES"
1023
+ # card. Filter by insurer_slug=='profile' (set by profile_rag.upsert).
1024
  citations = [
1025
  {
1026
  "policy_id": c.policy_id,
 
1032
  "score": round(c.score, 3),
1033
  }
1034
  for c in chunks
1035
+ if (c.insurer_slug or "").lower() != "profile"
1036
  ]
1037
 
1038
  # 7. INDIC CASCADE — translate the English reply back into Hinglish/Hindi,