eval/results.md

Eval Results — 2026-05-14T10:56:52Z

Headline

Metric	Value
Questions run	5
Factual accuracy	60.0%
Citation accuracy	100.0%
Refusal precision	0.0%
Blocked by faithfulness	0
Elapsed	62.3 s

By question type

Type	Accuracy
waiting_period	100.0%
sub_limit	100.0%
coverage_scope	0.0%
exclusions_oos	0.0%

By brain (router winners)

Brain	Accuracy
nim-chain	60.0%

Sample misses (up to 15)

id	question	bot_answer	reason
aditya-birla__activ-assure-diamond__wordings::ayush_coverage	Does Activ Assure cover AYUSH (Ayurveda, Yoga, Unani, Siddha	Yes, Activ Assure covers AYUSH treatments under in-patient hospitalization, incl	regex: no_overlap (gold_words=[])
aditya-birla__activ-assure-diamond__wordings::REFUSE::exclus	Does Activ Assure cover injuries from space tourism?	Activ Assure does not cover injuries from space tourism. The policy excludes par	did not refuse when expected

---

Grader: Groq Llama-3.3-70B-versatile (different model family from Sarvam-M to avoid circular eval). Full per-question results: eval/results.json.