Cost-per-token benchmarking tool to complement quantized model accuracy tracking
#7
by vigneshwar234 - opened
Hi Intel + community 👋
Quantized models are all about the accuracy-vs-efficiency tradeoff. I built an open source framework that makes that tradeoff measurable:
LLM Evaluation Framework tracks:
- 💰 Cost per 1K tokens — exact pricing across 15+ models including quantized variants
- 🎯 Accuracy — so you can plot accuracy vs cost directly
- ⚡ Latency p95 — real-world latency, not just throughput
- 🔍 Hallucination Rate — quantized models sometimes trade accuracy for hallucination rate differently than full-precision
The goal: give teams a single report that answers "is this quantized model good enough for my use case at my budget?"
Live demo: https://huggingface.co/spaces/vigneshwar234/llm-eval-demo
GitHub: https://github.com/vignesh2027/LLM-Evaluation-Framework
Open source, 71 tests, full CI/CD. Feedback welcome!