Spaces:
Sleeping
fix(recommend+citations): KI-171 + KI-172 — recommendation flow unblocked
Browse filesKI-171 — Skip faithfulness judge on recommendation queries.
The faithfulness judge is designed for POLICY-FACT lookups ("does
Policy X cover Y?") not for GENERATIVE RECOMMENDATIONS ("which policy
fits me?"). After fact_find completed and the user asked "show me
top 3 policies", the judge rejected the synthesized recommendation
with "I'd rather not answer that without stronger evidence in the
policy documents I have. Could you rephrase, or narrow your question
to a specific policy?" — blocking the entire recommendation flow.
Added query-shape detector. _is_recommendation true when user_text
contains: recommend / suggest / best polic / top N / show me polic /
fits me / right for me / policies for me / which polic / good polic /
what polic / your suggestion / your recommendation. When true, skip
faithfulness with reason=ki171_skip_on_recommendation.
This mirrors the existing KI-091 skip-on-fact_find pattern.
KI-172 — Filter profile chunks from user-facing citations.
"User profile (rohit)" was appearing in the chat's CITED POLICIES
card. The profile chunk is correctly retrieved for LLM context (so
the bot knows the user's age/dependents/income) but is NOT a policy
citation and shouldn't display as one.
Filter `chunks` where `insurer_slug == 'profile'` (case-insensitive)
before building the citations list. The chunk still flows into the
brain prompt for context — just not into the user-visible citations.
VERIFICATION:
py_compile backend/orchestrator.py — clean.
DOCS CASCADE: still held (stash @0).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- backend/orchestrator.py +25 -3
|
@@ -937,9 +937,25 @@ async def handle_turn(
|
|
| 937 |
# by fact_find_brain which has its own slot-extraction validation; the
|
| 938 |
# judge gate is additive overhead with negative production value here.
|
| 939 |
# Gate: skip if intent=='fact_find' OR we're mid-fact-find continuation.
|
| 940 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 941 |
if _skip_judge:
|
| 942 |
-
|
|
|
|
| 943 |
else:
|
| 944 |
verdict: FaithfulnessVerdict = await check_faithfulness(
|
| 945 |
reply=reply,
|
|
@@ -999,7 +1015,12 @@ async def handle_turn(
|
|
| 999 |
blocked = True
|
| 1000 |
reply = verdict.suggested_reply or "I don't have grounded evidence for that. Could you rephrase?"
|
| 1001 |
|
| 1002 |
-
# 6. Citations (derived from retrieved chunks)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1003 |
citations = [
|
| 1004 |
{
|
| 1005 |
"policy_id": c.policy_id,
|
|
@@ -1011,6 +1032,7 @@ async def handle_turn(
|
|
| 1011 |
"score": round(c.score, 3),
|
| 1012 |
}
|
| 1013 |
for c in chunks
|
|
|
|
| 1014 |
]
|
| 1015 |
|
| 1016 |
# 7. INDIC CASCADE — translate the English reply back into Hinglish/Hindi,
|
|
|
|
| 937 |
# by fact_find_brain which has its own slot-extraction validation; the
|
| 938 |
# judge gate is additive overhead with negative production value here.
|
| 939 |
# Gate: skip if intent=='fact_find' OR we're mid-fact-find continuation.
|
| 940 |
+
# KI-171 (2026-05-15) — also skip faithfulness on RECOMMENDATION queries.
|
| 941 |
+
# The judge is designed for policy-fact lookups ("does Policy X cover Y?"),
|
| 942 |
+
# not for generative synthesis ("which policy fits me?"). Recommendation
|
| 943 |
+
# answers stitch together evidence from many policies plus the user's
|
| 944 |
+
# profile — the judge can't grade that fairly and currently blocks valid
|
| 945 |
+
# recommendation flows after fact_find completes. Detect via query shape.
|
| 946 |
+
_user_text_lc = (user_text or "").lower()
|
| 947 |
+
_is_recommendation = any(
|
| 948 |
+
kw in _user_text_lc for kw in (
|
| 949 |
+
"recommend", "suggest", "best polic", "top 3", "top three",
|
| 950 |
+
"top 5", "top five", "show me polic", "show me what", "fits me",
|
| 951 |
+
"right for me", "policies for me", "which polic", "good polic",
|
| 952 |
+
"what polic", "your suggestion", "your recommendation",
|
| 953 |
+
)
|
| 954 |
+
)
|
| 955 |
+
_skip_judge = (intent == "fact_find") or bool(in_fact_find_continuation) or _is_recommendation
|
| 956 |
if _skip_judge:
|
| 957 |
+
skip_reason = "ki171_skip_on_recommendation" if _is_recommendation else "ki091_skip_on_fact_find"
|
| 958 |
+
verdict = FaithfulnessVerdict(passed=True, reasons=[skip_reason])
|
| 959 |
else:
|
| 960 |
verdict: FaithfulnessVerdict = await check_faithfulness(
|
| 961 |
reply=reply,
|
|
|
|
| 1015 |
blocked = True
|
| 1016 |
reply = verdict.suggested_reply or "I don't have grounded evidence for that. Could you rephrase?"
|
| 1017 |
|
| 1018 |
+
# 6. Citations (derived from retrieved chunks).
|
| 1019 |
+
# KI-172 (2026-05-15) — exclude profile chunks from user-facing citations.
|
| 1020 |
+
# The profile chunk is retrieved alongside policy chunks so the LLM has
|
| 1021 |
+
# context about the user (age/dependents/income/etc.), but it is NOT a
|
| 1022 |
+
# "cited policy" and should not appear in the chat's "CITED POLICIES"
|
| 1023 |
+
# card. Filter by insurer_slug=='profile' (set by profile_rag.upsert).
|
| 1024 |
citations = [
|
| 1025 |
{
|
| 1026 |
"policy_id": c.policy_id,
|
|
|
|
| 1032 |
"score": round(c.score, 3),
|
| 1033 |
}
|
| 1034 |
for c in chunks
|
| 1035 |
+
if (c.insurer_slug or "").lower() != "profile"
|
| 1036 |
]
|
| 1037 |
|
| 1038 |
# 7. INDIC CASCADE — translate the English reply back into Hinglish/Hindi,
|