RESEARCH PREVIEW · AGB

AirNova Guardrail Bot

A support chatbot that drafts like a real LLM, then checks every refund, discount, or waiver it promises against an immutable policy database before the customer ever sees it — the exact safeguard missing from the chatbot that got Air Canada sued.

Launch live demo Read the code

HEADLINE RESULT

Guardrails blocked 6 of 10 synthetic support scenarios — including the exact retroactive-bereavement-refund claim from the real lawsuit — and replaced each one with the verified policy text instead of the LLM's fabricated promise.

6 / 10

draft responses blocked as unauthorized commitments

2 / 10

commitments verified & passed through, citing the matching policy ID

unauthorized claims ever reached the mock customer

policy topics covered by the synthetic authorized-policy database

METHOD

LLM draft, then a rule it can't talk its way around

Retrieval runs over deliberately messy support content — the same kind of loosely worded page that caused the real failure — so the draft step is realistically prone to overreach.

Retrieve

TF-IDF search over a synthetic, intentionally messy support knowledge base (forum posts, marketing blurbs, vague FAQs).

Draft

A generator (mock or real Claude) writes a customer response from that retrieved context.

Classify

A regex commitment classifier flags any refund, discount, waiver, or dollar/percent figure in the draft.

Verify

Every flagged claim is cross-checked against a hardcoded, immutable policy JSON — never editable by the LLM.

Decide

Authorized claims pass through with a policy citation. Unauthorized ones are blocked and replaced with the real policy text.

FINDINGS

What the synthetic test runs showed

The bereavement-refund scenario — modeled directly on the real Air Canada case — was correctly blocked: the draft claimed a retroactive full refund, but the authorized policy only allows a 5% pre-booking discount.

Legitimate commitments still get through: an airline-caused cancellation correctly triggers a 100% refund citation against policy POL-002, with no unnecessary blocking.

The verifier caught failure modes beyond simple refunds — inflated dollar amounts ($300 voucher vs. an authorized $200), and fee waivers with no basis in policy.