Multi-metric evaluation tool for Southeast Asian LLM selection
Hi SeaLLMs community 👋
Excellent work benchmarking LLMs for Southeast Asian languages. For teams in the SEA region, API cost and latency from regional data centers differ significantly from US benchmarks — making cost-aware evaluation critical.
I built an open source LLM Evaluation Framework that helps:
→ 💰 Cost per 1K tokens — compare regional vs frontier model pricing
→ ⚡ Latency p95 — real-world latency from SEA can differ 2-3x from US benchmarks
→ 🔍 Hallucination Rate — especially critical for multilingual outputs
→ 🎯 Accuracy — MMLU + TruthfulQA + custom benchmark support for regional tasks
→ 🧠 Reasoning Quality — CoT depth scoring
Extensible benchmark system supports adding Bahasa, Thai, Vietnamese, Filipino evaluation tasks.
Live demo: https://huggingface.co/spaces/vigneshwar234/llm-eval-demo
GitHub: https://github.com/vignesh2027/LLM-Evaluation-Framework
Would love to discuss extending the benchmark system for SEA languages!