Open source epistemic reasoning + cost + hallucination evaluation

#1
by vigneshwar234 - opened

Hi ๐Ÿ‘‹

Epistemic reasoning evaluation is a fascinating and underexplored area. A model's ability to know what it doesn't know directly correlates with its hallucination rate โ€” and I built a framework that measures both sides of this.

LLM Evaluation Framework tracks:

โ†’ ๐Ÿ” Hallucination Rate โ€” measures overconfident wrong outputs (anti-epistemic behavior)
โ†’ ๐Ÿง  Reasoning Quality โ€” chain-of-thought depth, which supports epistemic transparency
โ†’ ๐ŸŽฏ Accuracy โ€” task completion
โ†’ ๐Ÿ’ฐ Cost per 1K tokens
โ†’ โšก Latency

The combination of epistemic reasoning score + hallucination rate + reasoning quality gives a much richer picture of model calibration.

Live demo: https://huggingface.co/spaces/vigneshwar234/llm-eval-demo
GitHub: https://github.com/vignesh2027/LLM-Evaluation-Framework

Sign up or log in to comment