Spaces:
Running on CPU Upgrade
Complementary text-LLM evaluation: cost + latency + hallucination tracking
Hi OpenCompass team 👋
Your VLM leaderboard is the reference for vision-language benchmarking. For teams who also need to evaluate the text-only LLM backbone of their VLMs, I built a complementary tool.
LLM Evaluation Framework covers the text evaluation side with 5 simultaneous metrics:
→ 🎯 Accuracy on MMLU + TruthfulQA benchmarks
→ ⚡ Latency — full percentile breakdown (p50/p75/p90/p95/p99)
→ 💰 Cost — precise per-1K-token pricing for every major provider
→ 🔍 Hallucination Rate — runs locally, zero additional cost
→ 🧠 Reasoning Quality — structured chain-of-thought scoring
One CLI command. Any LiteLLM-compatible model. Full PDF + JSON report.
Live demo: https://huggingface.co/spaces/vigneshwar234/llm-eval-demo
GitHub: https://github.com/vignesh2027/LLM-Evaluation-Framework
Open source, free forever. Would love feedback from the OpenCompass community on benchmark design!