Open source multi-metric evaluation: accuracy + cost + hallucination
Hi AMA-Bench team 👋
Multi-benchmark comparison is exactly the right approach. I built an open source framework that goes beyond accuracy to give you 5 metrics per model in one run:
→ 🎯 Accuracy — 4-strategy cascade scorer (exact, normalized, MC, fuzzy)
→ 💰 Cost per 1K tokens — the metric missing from most leaderboards
→ ⚡ Latency p50/p95/p99 — full percentile breakdown
→ 🔍 Hallucination Rate — 0.0-1.0 score, runs locally
→ 🧠 Reasoning Quality — chain-of-thought depth 1-10
One CLI command. Any LiteLLM-compatible model. Full JSON + PDF report.
Live demo: https://huggingface.co/spaces/vigneshwar234/llm-eval-demo
GitHub: https://github.com/vignesh2027/LLM-Evaluation-Framework
Would love to discuss combining our approaches for comprehensive model comparison!