Spaces:
Running
Open source 5-metric evaluation framework — add cost + hallucination to any leaderboard
Hi OpenEvals team 👋
Aggregating every leaderboard is such valuable work. One dimension consistently missing from all the leaderboards you're aggregating is cost per token — the metric teams use for final deployment decisions.
I built an open source LLM Evaluation Framework that fills that gap:
→ 💰 Cost per 1K tokens — 15+ models, real token-count pricing
→ ⚡ Latency p50/p95/p99 — another missing dimension in most leaderboards
→ 🔍 Hallucination Rate — runs locally, adds no API cost
→ 🎯 Accuracy — compatible with standard MC benchmark format
→ 🧠 Reasoning Quality — CoT depth scoring
Could be interesting to include cost/latency data alongside the aggregated accuracy scores.
Live demo: https://huggingface.co/spaces/vigneshwar234/llm-eval-demo
GitHub: https://github.com/vignesh2027/LLM-Evaluation-Framework
Open source, 71 tests, free forever!