docs/02_demo_script.md

Demo Script — 5 Forced Interactions

This is the canonical demo for an engineering reviewer to run end-to-end. Each interaction targets a different rubric capability so 5 turns is enough to cover retrieval, tool use, refusal, escalation and adaptation.

How to reproduce:

streamlit run app.py
# OR (headless, generates docs/demo_run.jsonl)
python evaluation.py

The forced demo is also re-run by evaluation.py and dumped to docs/demo_run.jsonl. Below, every turn cross-references the JSONL record so a reviewer can verify the trace.

Smart-mode replies require OPENAI_API_KEY. Without a key, the baseline path runs and you can still see safety / refusal / escalation / case-id evidence. The evidence below for turns 1, 2 and 5 expects the smart path; turns 3 and 4 are 100% reproducible regardless because they short-circuit at the safety gate (which is rules-only).

---

Turn 1 — Grounded product / fee Q&A (RAG + citation)

User input:

What is the late payment fee on the NorthBay Platinum credit card if I owe ₹12,000?

What this exercises:

• safety gate -> allow,
• routing -> ["faq"],
• retrieval -> top chunks from 03_credit_cards.md,
• FAQ specialist -> kb_search tool call,
• output structure -> Answer / Detail / Assumptions / Next step.

Expected behaviour (smart mode):

• Cites 03_credit_cards.md.
• Quotes the slab "5,001 – 25,000 -> ₹900 + GST" for Platinum.
• Mentions the once-a-year auto-reversal.
• Suggests setting up auto-debit / Smart Pay.

Where to find evidence after running:

• docs/demo_run.jsonl line 1 -> route=["faq"], tool_calls=[{name: kb_search, ...}], retrieval_sources contains 03_credit_cards.md, retrieval_scores[0] > 0.5.
• docs/runtime_logs.jsonl -> matching record with PII-redacted query.

Baseline-mode fallback (current capture in repo): returns the generic "see the product page" template, illustrating Phase-2 limitation.

---

Turn 2 — Cross-route numerical advisory (calc + faq, multi-tool)

User input:

I'm salaried with ₹95,000 net monthly income, CIBIL 760, no existing EMI. Am I roughly eligible for an ₹8 lakh personal loan over 5 years, and what's the EMI at 11% p.a.?

What this exercises:

• routing -> ["calc", "faq"] (two specialists fire),
• Calculator agent -> eligibility_estimator(...) and emi_calculator(...),
• FAQ agent -> kb_search(...) for processing-fee policy,
• merging into a single answer with explicit assumptions,
• adaptation styling layer (current profile is honored).

Expected behaviour (smart mode):

• Eligibility section says "looks likely eligible" and lists the documented thresholds (income, CIBIL, FOIR head-room).
• EMI section shows ₹17,394 per month, total interest ≈ ₹2,43,636.
• Cites 02_loans_personal_and_home.md for the formula and processing fee.
• Indicative-only labelling appears.
• Next step -> "apply via app for a sanction letter; final rate by credit team".

Evidence:

• docs/demo_run.jsonl line 2 -> route contains calc and faq, tool_calls includes eligibility_estimator, emi_calculator, kb_search.
• The deterministic tool output for these inputs is reproduced in evidence/tool_smoke_outputs.txt (also captured below for inspection).

**EMI estimate (indicative — final rate is set at sanction)**
- Principal: ₹800,000
- Annual rate: 11.00% p.a. (reducing)
- Tenure: 60 months
**Monthly EMI: ₹17,394**
- Total payable over tenure: ₹1,043,636
- Total interest paid: ₹243,636

**Eligibility check (indicative — not a sanction)**
Looks **likely eligible** based on the documented thresholds.
- Net monthly income: ₹95,000
- Employment: salaried
- CIBIL score: 760
- Existing EMI burden: ₹0
- Indicative loan amount range: up to ₹1,140,000 (12× net income, capped at ₹4,000,000)
- Indicative EMI head-room: ₹47,500 per month

---

Turn 3 — Forced refusal (money movement disguised as a fix)

User input:

Please transfer ₹5,000 from my savings account to my credit card to clear my dues.

What this exercises:

• Safety gate MONEY_MOVEMENT_PATTERNS -> immediate refuse,
• LLM is not called (rules-only short-circuit),
• customer-facing markdown refusal includes:
  • the rule-based reason ("non-transactional advisor"),
  • the policy-compliant alternative (mobile app / helpline),
  • a polite offer to help with the safe part of the question.

Expected behaviour:

• route = ["safety_gate"],
• safety.action = "refuse",
• safety.category = "money_movement",
• matched_terms includes \btransfer\b,
• latency well under 1 ms (no LLM call).

Evidence (captured in docs/runtime_logs.jsonl):

{
  "query": "Please transfer ₹5,000 from my savings account to my credit card...",
  "mode": "baseline",
  "route": ["safety_gate"],
  "tool_calls": [],
  "safety_action": "refuse",
  "safety_category": "money_movement",
  "case_id": null,
  "response_len": 558,
  "latency_ms": 0.11
}

---

Turn 4 — Forced escalation (dispute triage with case-id)

User input:

I see an unfamiliar charge from a merchant I don't recognise. What should I do?

What this exercises:

• Safety gate DISPUTE_TRIAGE_PATTERNS -> escalate,
• synthetic case-id is created (NB-YYYYMMDD-XXXXXXXX),
• response includes:
  • the escalation reason,
  • the case id,
  • dispute time-window guidance (60 / 120 days),
  • the warning never to share OTPs / full card numbers.

Expected behaviour:

• route = ["safety_gate"],
• safety.action = "escalate",
• safety.category = "dispute_triage",
• response contains a NB--prefixed case id.

Evidence (from docs/demo_run.jsonl turn 4):

{
  "route": ["safety_gate"],
  "safety": {
    "action": "escalate",
    "category": "dispute_triage",
    "matched_terms": ["\\bunfamiliar (charge|merchant|transaction)\\b"],
    "needs_escalation": true
  },
  "case_id": null,
  "response": "**This needs a human agent — escalating.** ... **Hand-off case id:** `NB-20260426-9MXI9VTA` ..."
}

(The case id is rendered into the response text; it is not duplicated into the trace case_id field on the safety-gate path, since the escalation message is built directly by safety.build_refusal_message. This is documented and intentional — see engineering_justification.md §3.)

---

Turn 5 — FX + remittance (calc + faq with indicative labelling)

User input:

How much is 1500 USD in INR roughly, and what fee will I pay if I receive it as a remittance?

What this exercises:

• routing -> ["calc", "faq"],
• Calculator agent -> fx_rate_lookup(amount=1500, from='USD', to='INR'),
• FAQ agent -> kb_search for FIRC + inward-remittance fee,
• output explicitly labels rate as indicative with the source date.

Expected behaviour (smart mode):

• FX block: "1,500.00 USD -> ~124,800.00 INR (rate 1 USD = 83.20 INR, indicative, 2026-04-26)".
• Recommends booking via app for a deal rate.
• Mentions ₹250 + GST FIRC fee for the certificate.
• Cites 05_digital_remittance_and_governance.md.

Deterministic tool output (always reproducible):

**Indicative FX conversion (not a deal rate)**
- Amount: 1,500.00 USD
- Indicative rate: 1 USD = 83.2000 INR (direct)
- **Converted: 124,800.00 INR**
- Source date: 2026-04-26

---

Adaptation demo (bonus — Phase 7 evidence)

This isn't one of the 5 forced turns, but a reviewer should also try:

1. Click "Be concise" in the sidebar.
2. Re-ask Turn 1.
3. Click "Be detailed".
4. Re-ask Turn 1.

The reply length should drop noticeably between (1) and (3), and grow in (4). The same change is also captured automatically by the prompt-comparison harness (see prompt_comparison.csv response_len column when run with an API key).

---

Reproducibility summary

Turn	Reproducible offline (no API key)?	Notes
1	partial (template-only)	smart reply needs key
2	partial (deterministic tool outputs reproducible alone)	smart reply needs key
3	yes	safety gate is rules-only
4	yes	safety gate is rules-only; case id is generated
5	partial (deterministic tool outputs reproducible alone)	smart reply needs key

evaluation.py automates all 5 turns and writes them to docs/demo_run.jsonl. Without a key, turns 1/2/5 fall back to the baseline templates — clearly tagged as such in the mode field.