Inference cost evaluation to complement training cost estimates
#1
by vigneshwar234 - opened
Hi!
Your training cost calculator is super useful for planning. The other side of the cost equation is inference cost — and I built an open source framework that measures this precisely.
LLM Evaluation Framework measures inference cost:
- Cost per 1K tokens (from real token counts, not estimates) across 15+ models
- Latency p50/p95/p99 (for SLA planning alongside cost)
- Accuracy (to compute cost-per-correct-answer)
- Hallucination Rate (to compute cost-per-trustworthy-answer)
- Reasoning Quality
Training cost calculator + inference cost evaluator = complete LLM economics picture.
Live demo (no API key): https://huggingface.co/spaces/vigneshwar234/llm-eval-demo
GitHub: https://github.com/vignesh2027/LLM-Evaluation-Framework
Open source. Free forever!