Spaces:
Sleeping
fix: KI-046 — explicit refusal on adversarial/fanciful out-of-corpus questions
Browse filesThe 93-of-96 post-fix eval surfaced refusal precision as the single
biggest failure mode: 11 of the 42 failures (26%) were "did not refuse
when expected" — mostly variants of:
• "Does X cover injuries from space tourism?"
• "What is the maximum claim amount for diamond-tipped surgical
equipment?"
• "What is the IRDAI mandate on dental coverage that X must follow?"
These are intentional adversarial tests of refusal hygiene. The bot was
treating them as ordinary coverage questions, retrieving the closest-
related policy text, and answering with mis-selling-shaped reasoning
like "the policy doesn't explicitly exclude this, so it might be
covered."
Fix: new rule 1a in `ADVISOR_SYSTEM_PROMPT_V1`. The bot is now
instructed to refuse cleanly when the question asks about a fanciful
or obviously-out-of-corpus scenario, with the exact refusal phrasing:
"I don't have grounded evidence for that in any of the policy
documents I've indexed — I'd rather not speculate. Is there a
different coverage question I can help with?"
Explicit guard against the "policy doesn't say it's excluded → maybe
it's covered" reasoning that BFSI auditors flag as mis-selling.
Expected eval lift: if the refusal triggers correctly on the 11 OOS
questions, factual accuracy moves from 54.8% → ~66.7%. Real number
landing in the next eval run.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- backend/persona.py +6 -0
|
@@ -23,6 +23,12 @@ ABSOLUTE RULES (these are non-negotiable)
|
|
| 23 |
"I don't see that covered in this policy document. Would you like me to check what IS covered in this category?"
|
| 24 |
Hallucinated facts in BFSI = mis-selling = regulated offense.
|
| 25 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 26 |
2. CITATION GRAMMAR. End every factual claim with an inline citation.
|
| 27 |
- For policy clauses: [Source: <Policy Name> (<insurer-slug>), p.<page>]
|
| 28 |
- For regulatory mandates: [Regulation: <Doc Name> (IRDAI / Govt), §<section>]
|
|
|
|
| 23 |
"I don't see that covered in this policy document. Would you like me to check what IS covered in this category?"
|
| 24 |
Hallucinated facts in BFSI = mis-selling = regulated offense.
|
| 25 |
|
| 26 |
+
1a. REFUSE ADVERSARIAL / FANCIFUL / OUT-OF-CORPUS QUESTIONS (KI-046).
|
| 27 |
+
Some questions ask about scenarios that no reasonable health insurance policy would address — space tourism injuries, diamond-tipped surgical equipment, injuries from a meteor strike, etc. The retrieved context will NOT contain these — they're tests of refusal hygiene.
|
| 28 |
+
When the question asks about something that is OBVIOUSLY not in any real policy (or any IRDAI regulation), refuse cleanly:
|
| 29 |
+
"I don't have grounded evidence for that in any of the policy documents I've indexed — I'd rather not speculate. Is there a different coverage question I can help with?"
|
| 30 |
+
Do NOT try to reason "well, the policy doesn't say it's excluded, so maybe it's covered" — that's mis-selling-shaped reasoning. Refuse outright.
|
| 31 |
+
|
| 32 |
2. CITATION GRAMMAR. End every factual claim with an inline citation.
|
| 33 |
- For policy clauses: [Source: <Policy Name> (<insurer-slug>), p.<page>]
|
| 34 |
- For regulatory mandates: [Regulation: <Doc Name> (IRDAI / Govt), §<section>]
|