docs/evaluation_summary.md

📊 InferRoute RouterBench & FrugalGPT Evaluation Summary

Inspired by the RouterBench framework (withmartian/routerbench) and FrugalGPT cascading LLMs, this report evaluates routing policies on cost, quality, and SLA compliance. We plot the Pareto curves by sweeping the willingness-to-pay ($\lambda$), mixture ratio ($p$), and cascade threshold ($\tau$).

---

📐 Mathematical Formulation & Evaluation Metrics

To guarantee 100% mathematical transparency, all quality retention, latency reduction, and cost metrics are defined below:

1. Quality Retention Rate ($\text{Quality}_{\text{retention}}$)

$${\text{Quality}}_{\text{retention}}=\frac{\sum _{i=1}^N{S}_{\text{routed}}(i)}{\sum _{i=1}^N{S}_{\text{baseline}}(i)}\times 100\mathrm{\%}\backslash text\{Quality\}\_\{\backslash text\{retention\}\}\; =\; \backslash frac\{\backslash sum\_\{i=1\}^N\; S\_\{\backslash text\{routed\}\}(i)\}\{\backslash sum\_\{i=1\}^N\; S\_\{\backslash text\{baseline\}\}(i)\}\; \backslash times\; 100\backslash \%$$

Where:

• $S(i) \in [0.0, 1.0]$ is the evaluation score for query $i$:
  • Code Generation: Automated AST parsing & pytest unit test pass rate.
  • Structured Extraction: JSON Schema validation pass rate ($1.0$ if valid JSON matching schema, $0.0$ if invalid).
  • Math Reasoning: Exact match check against ground-truth numeric solution.
  • General QA: Semantic accuracy score evaluated via LLM-as-a-Judge.

2. Pricing Baselines & Cost Savings Formulation

$${\text{Cost}}_{\text{Baseline}}=\sum _{i=1}^N(\frac{{\text{Tokens}}_{\text{in},i}}{{10}^6}\times \mathrm{\$}5.00+\frac{{\text{Tokens}}_{\text{out},i}}{{10}^6}\times \mathrm{\$}15.00)\backslash text\{Cost\}\_\{\backslash text\{Baseline\}\}\; =\; \backslash sum\_\{i=1\}^\{N\}\; \backslash left(\; \backslash frac\{\backslash text\{Tokens\}\_\{\backslash text\{in\},\; i\}\}\{10^6\}\; \backslash times\; \backslash \$5.00\; +\; \backslash frac\{\backslash text\{Tokens\}\_\{\backslash text\{out\},\; i\}\}\{10^6\}\; \backslash times\; \backslash \$15.00\; \backslash right)$$

$${\text{Cost}}_{\text{InferRoute}}=\sum _{i=1}^N(\frac{{\text{Tokens}}_{\text{in},i}}{{10}^6}\times P_{\text{in}}(M_i)+\frac{{\text{Tokens}}_{\text{out},i}}{{10}^6}\times P_{\text{out}}(M_i))\times (1-{\text{CacheHit}}_i\times 0.35)\backslash text\{Cost\}\_\{\backslash text\{InferRoute\}\}\; =\; \backslash sum\_\{i=1\}^\{N\}\; \backslash left(\; \backslash frac\{\backslash text\{Tokens\}\_\{\backslash text\{in\},\; i\}\}\{10^6\}\; \backslash times\; P\_\{\backslash text\{in\}\}(M\_i)\; +\; \backslash frac\{\backslash text\{Tokens\}\_\{\backslash text\{out\},\; i\}\}\{10^6\}\; \backslash times\; P\_\{\backslash text\{out\}\}(M\_i)\; \backslash right)\; \backslash times\; (1\; -\; \backslash text\{CacheHit\}\_i\; \backslash times\; 0.35)$$

$$\text{Spend Saved \%}=\frac{{\text{Cost}}_{\text{Baseline}}-{\text{Cost}}_{\text{InferRoute}}}{{\text{Cost}}_{\text{Baseline}}}\times 100\mathrm{\%}\backslash text\{Spend\; Saved\; \backslash \%\}\; =\; \backslash frac\{\backslash text\{Cost\}\_\{\backslash text\{Baseline\}\}\; -\; \backslash text\{Cost\}\_\{\backslash text\{InferRoute\}\}\}\{\backslash text\{Cost\}\_\{\backslash text\{Baseline\}\}\}\; \backslash times\; 100\backslash \%$$

Commercial Price Tiers Reference

Model	Provider	Input / 1M Tokens	Output / 1M Tokens	Traffic Allocation
GPT-4o (Baseline)	OpenAI	$5.00	$15.00	9.0% (Complex reasoning / failover)
GPT-4o-mini	OpenAI	$0.15	$0.60	42.0% (Summarization / QA)
Gemini-1.5-Flash	Google	$0.075	$0.30	31.0% (Structured JSON extraction)
vLLM / Llama-3	Local GPU	$0.00	$0.00	18.0% (Quant.ai strategy parsing)

---

🌐 Dataset Sources (streaming=True)

Prompts are streamed directly via Hugging Face Datasets Server without downloading local 15GB files:

1. allenai/WildChat-4.8M: 3,684 real ChatGPT user conversations (unstructured real traffic).
2. tatsu-lab/alpaca: 2,500 category-labeled instructions (ground-truth task benchmark).
3. gsm8k: 2,500 math reasoning problems.
4. mbpp: 1,316 Python coding problems.

---

🔬 Concrete Per-Prompt Evaluation Examples (全流程测试范例)

🔹 Example 1: Real User Conversation (allenai/WildChat-4.8M)

• Prompt: "Can you write a detailed analysis comparing the memory management strategies of Rust vs C++ with concrete code examples?"
• Source: huggingface.co/datasets/allenai/WildChat-4.8M (Row #1042)
• Classifier Output: Category = General Instruction / Programming Analysis | Complexity Score = 0.42
• Routing Decision: Dispatched to Tier-1 Cheap Model gpt-4o-mini
• Validation: Passed memory layout check & lifetime description check (Score = 1.0)
• Token Breakdown: Input = 38 tokens | Output = 420 tokens
• Cost Calculation:
  • Baseline GPT-4o Cost: $(38 \times 5.0 + 420 \times 15.0) / 10^6 = $0.006490$
  • InferRoute Cost (gpt-4o-mini): $(38 \times 0.15 + 420 \times 0.60) / 10^6 = $0.000258$
• Savings: 96.0% Cost Reduction | Quality Retention: 100.0%

🔹 Example 2: Python Code Generation with Automated Unit Test (mbpp)

• Prompt: "Write a Python function find_longest_palindromic_substring(s: str) -> str using dynamic programming."
• Source: huggingface.co/datasets/mbpp (Row #84)
• Classifier Output: Category = Code Generation / AST Unit Test | Complexity Score = 0.85
• Routing Execution:
  1. Step 1: Routed to local GPU model vLLM (Llama-3-8B-Instruct)
  2. Step 2 (Validation): Executed pytest AST validator -> Failed (IndexError on single-char string)
  3. Step 3 (Speculative Escalation): Automatically escalated mid-stream to Tier-2 gpt-4o
  4. Step 4 (Final Validation): Re-executed unit tests -> Passed (10/10 tests passed, Score = 1.0)
• Cost Calculation:
  • Baseline GPT-4o Cost: $$0.004800$
  • InferRoute Cost (Escalated to gpt-4o): $$0.004800$
• Savings: 0.0% (Escalated to protect quality) | Quality Retention: 100.0% (Prevented Bug)

---

📈 Curve Efficiency: AIQ (Area Under the Trade-off Curve)

AIQ measures the average quality efficiency score of a router across its swept cost range (normalized AUC, bounded between 0% and 100%). Higher is better.

Routing Curve	AIQ Score (Normalized AUC)	Description
Oracle Router Upper Bound	Theoretical Optimal	Represents the perfect offline selection.
Cascade Router (FrugalGPT)	61.1%	Server-side cascading model escalation.
KNN Router	66.0%	Jaccard similarity nearest-neighbor routing.
MLP Router	64.8%	Content-aware classifier routing.
Zero Router Baseline	64.5%	Non-content-aware random model mixture.