Hugging Face's logo

Spaces:
Ypeng12
/
InferRoute
Sleeping

App Files Community
InferRoute / README.md
Ypeng12's picture
Ypeng12
feat: add HF real workload datasets (10k), streaming benchmark results, and BYOK budget tests
7191aa9
preview code
|
Raw
History Blame Contribute Delete
16 kB
metadata 
title: InferRoute Multi-LLM Gateway
emoji: ๐Ÿš€
colorFrom: blue
colorTo: indigo
sdk: docker
app_port: 7860
pinned: false

๐ŸŽฏ InferRoute: High-Availability LLM Inference Gateway & Observability Router

InferRoute is built upon robust theoretical frameworks for cost-performance trade-offs and multi-tier cascading inference.

License Python Version FastAPI PRs Welcome

๐Ÿš€ Live Technical Hub & Interactive Cascade Simulator: https://ypeng12.github.io/InferRoute/
Visit the hosted page to interact with the live sequence charts, LaTeX formula guides, original paper previews, and the FrugalGPT sequential cascade simulator!

๐Ÿ“– Introduction

InferRoute is a lightweight, high-performance LLM inference gateway and reliability proxy. Sitting between client applications/AI agents and model backends (local Ollama/vLLM engines and commercial APIs like OpenAI/Gemini), it dynamically routes queries to optimize for cost, latency, and quality in real-time.

Rather than statically pinning your application to a single expensive cloud endpoint, InferRoute dynamically routes prompts based on real-time quality validation, cost constraints, latency tracking, and prefix KV-cache affinity, demonstrating over 60% in API cost savings in benchmark sweeps while guaranteeing strict latency and formatting SLAs.

Cost Quality Pareto Frontier Sweep

โšก Key Technical Highlights

  • ๐Ÿ”„ Distributed Streaming Deduplication: Implements a Redis lock for duplicate concurrent prompts. The first request invokes the LLM while subsequent callers subscribe to the stream via Redis Pub/Sub, broadcasting token chunks simultaneously to avoid duplicate API fees.
  • ๐ŸŒณ Radix Trie prefix KV-Cache Affinity: Hashes prompt prefixes in a Prefix Tree to route requests to local GPU nodes holding warm KV caches, reducing Time-to-First-Token (TTFT) by up to 80%.
  • ๐Ÿ›ก๏ธ Vegas Adaptive Concurrency Control: Scales gateway concurrency slots dynamically based on active queue size tracking via a TCP Vegas congestion control loop to protect local GPUs from OOM failures.
  • ๐Ÿ”€ Speculative Fallback Cascades: Buffers response stream tokens, cancels degraded local streams speculatively (on loop or gibberish detection), and escalates to premium cloud nodes mid-stream to avoid service disruptions.
  • ๐Ÿ’ณ Multi-Tenant Billing Gateway: Validates tenant credentials and credits against an asynchronous PostgreSQL audit ledger, enforcing a resilient fail-open policy if the database is unreachable.

๐Ÿ—๏ธ Gateway Request Lifecycle

The diagram below outlines how the gateway intercepts client requests, checks caches, scales concurrency, and handles fallback cascades:

graph TD
    Client[Client App / SDK] -->|POST /v1/chat/completions| Gateway[InferRoute Gateway]
    
    Gateway --> Auth{Auth Gate & Balance Check}
    Auth -->|Low Balance| Block[402 Payment Required]
    Auth -->|Valid Wallet| Cache{Redis Exact & Trie Cache}
    
    Cache -->|Cache Hit| StreamCache[Stream response from Redis] --> Client
    Cache -->|Cache Miss| Limiter{Vegas Adaptive Limiter}
    
    Limiter -->|Queue Overload| Failover[Fallback to Cloud Node]
    Limiter -->|Slot Acquired| Router[Routing Engine]
    
    Router -->|Prefill Cache Affinity| Primary[Local vLLM / Ollama]
    Router -->|Speculative Cost-Saver| Primary
    
    Primary --> StreamBuf[Stream Buffer Validation]
    StreamBuf -->|Repeated Loops / Fail| Cascade[Speculative Cancel & Fallback to Cloud]
    StreamBuf -->|Quality Passed| StreamClient[Stream Response to Client] --> Client
    
    StreamBuf --> BackgroundLog[Async Log & Wallet Deduction] --> PostgreSQL[(PostgreSQL Audit & Ledger)]

๐Ÿง  RouterBench Integration & Theoretical Foundations

For a detailed breakdown of routing algorithms, mathematical proofs, and model mapping logic, see our dedicated Academic Foundations Guide.

InferRoute integrates the core routing methodologies and trade-off evaluation principles from the research paper:

ROUTERBENCH: A Benchmark for Multi-LLM Routing System (by Martian / withmartian/routerbench).
๐Ÿ“„ Read RouterBench Paper PDF

1. Mathematical Scoring & Optimization

Predictive routing is formulated as choosing a backend $m$ that maximizes the utility score for a prompt $x$: Score(m)=ฮปโ‹…Qualitypred(m)โˆ’Cost(m)\text{Score}(m) = \lambda \cdot \text{Quality}_{\text{pred}}(m) - \text{Cost}(m)

Where:

  • $\lambda$ (lambda): User's willingness to pay. A high $\lambda$ (e.g. 5.0) prioritizes output quality (routing to GPT-4o-mini or Gemini-Flash). A low $\lambda$ (e.g. 0.1) prioritizes cost savings (routing to local vLLM/Ollama).
  • $\text{Quality}_{\text{pred}}(m)$: Predicted quality score of model $m$ on prompt $x$ (ranging from $0.0$ to $1.0$).
  • $\text{Cost}(m)$: The estimated economic API fee to process the request on backend $m$.

2. Supported Routing Policies

InferRoute supports six distinct routing strategies aligned with the paper's framework:

  1. ๐ŸŽฒ Zero Router Baseline (zero): Non-content-aware routing. Randomly routes requests to Cloud/Expensive vs. Local/Cheap backends based on a target cloud mixture ratio $p$. Sweeping $p \in [0, 1]$ constructs the baseline cost-quality curve.
  2. ๐Ÿ“‹ Rule-Based Router (rule): Content-aware heuristics. Evaluates prompt keywords to route tasks (e.g., routing math tasks to GPT/Gemini, coding tasks to vLLM, simple greetings to Ollama).
  3. ๐Ÿง  KNN-Based Router (knn): Retrieves the $K$ most textually similar prompts (using Jaccard similarity) from historical benchmark outcomes. Computes the average historical quality per backend and optimizes via $\lambda \cdot \text{Quality} - \text{Cost}$.
  4. ๐Ÿ•ธ๏ธ MLP-Based Router (mlp): A fast logistic regression classifier that extracts prompt features (is_code, is_math, is_json, is_long) to predict success probability as predicted quality, then optimizes using the cost-quality formula.
  5. ๐Ÿ”ฎ Oracle Router Offline Upper Bound (oracle): A theoretical upper bound that has advance knowledge of whether each model will answer correctly. It selects the cheapest model that achieves a quality score $\ge 0.8$.
  6. ๐Ÿ”„ Cascade Router (FrugalGPT) (cascade): Server-side model cascade. Executes the cascade chain (cheap local models up to premium cloud models) until the reliability judge score meets the acceptance threshold $\tau$.

3. Metric: AIQ (Area Under the Curve)

To measure a router's overall efficiency across all budgets, we compute the AIQ (Area under the cost-quality curve) using the Trapezoidal Rule over swept parameter values ($\lambda$ or $p$): AIQ=โˆซcminโกcmaxโกQ(c)โ€‰dcโ‰ˆโˆ‘i=0nโˆ’1qi+qi+12โ‹…(ci+1โˆ’ci)\text{AIQ} = \int_{c_{\min}}^{c_{\max}} Q(c) \, dc \approx \sum_{i=0}^{n-1} \frac{q_i + q_{i+1}}{2} \cdot (c_{i+1} - c_i)

A smart, content-aware learned router (MLP/KNN) will push the Pareto frontier towards the top-left, achieving a significantly higher AIQ than the Zero Router baseline.

๐Ÿ”„ FrugalGPT Cascading Inference (LLM Cascade)

InferRoute integrates the cost-performance optimization concepts from the paper:

FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance (by Stanford University / arXiv:2305.05196).
๐Ÿ“„ Read FrugalGPT Paper PDF | ๐Ÿ“„ Read Hybrid LLM Routing Paper PDF

FrugalGPT identifies three main classes of cost-saving methods:

  1. Prompt Adaptation: Reduces input tokens dynamically. In InferRoute, when routing to cheap local backends (Ollama/vLLM), the Prompt Adapter (prompt_adapter.py) automatically compresses prompt sizes by pruning few-shot examples, restoring full prompts when upgrading to premium models.
  2. LLM Approximation (Completion Cache): Caches full answers. Handled by InferRoute's exact completions caching layer in Redis.
  3. LLM Cascade: Sequential invocation of backends (e.g., ollama โž” vllm โž” gemini โž” openai). The request starts with the cheapest backend, passes through a Reliability Judge (syntactic check, schema validation, loop penalty), and only escalates to a more expensive tier if the quality score is below the acceptance threshold $\tau$.

1. Cascade Configuration

Select Cascade Router (FrugalGPT Style) in the routing options, and adjust the acceptance threshold $\tau \in [0.0, 1.0]$. The gateway executes the cascade loop entirely server-side, accumulating tokens and fees across all tried models to log in the PostgreSQL database correctly.

2. Streaming Cascade Buffer

To prevent leaking low-quality responses to clients during streaming requests, the gateway buffers stream chunks internally, executes quality grading, and only pumps the SSE stream to the client once the output is officially accepted.

๐Ÿ“Š Reproducible Benchmark & Evaluation Harness

InferRoute includes a standardized pipeline to test and compare multiple routing strategies under realistic workloads, sweeping mixture ratios ($p$) and willingness-to-pay ($\lambda$) to trace cost-quality Pareto frontiers.

1. Pricing Baselines & Assumptions

To ensure 100% transparency, API spend savings are calculated against standard commercial pricing tiers:

Model Tier Provider Input Price (per 1M) Output Price (per 1M) Routing Target Scenario
GPT-4o (Always-Strong Baseline) OpenAI $5.00 $15.00 Complex Reasoning, Code Failovers (9.0% of traffic)
GPT-4o-mini (Cheap Cloud) OpenAI $0.15 $0.60 Customer Support Summarization (42.0% of traffic)
Gemini-1.5-Flash (Fast Cloud) Google $0.075 $0.30 Structured Information Extraction (31.0% of traffic)
vLLM / Ollama (Local GPU) Self-Hosted $0.00 $0.00 Quant.ai Strategy Generation (18.0% of traffic)

2. Dataset Sources (Hugging Face Streaming)

Evaluations are streamed directly via datasets (streaming=True) without downloading 15GB+ disk files:

  • allenai/WildChat-4.8M: Real-world unstructured user conversations (up to Aug 2025 data).
  • HuggingFaceH4/no_robots: Category-labeled instruction benchmarks (summarization, coding, classification, rewrite).

3. Empirical Results (10,000-Request Benchmark Sweep)

Under a 10,000-request / 100-worker concurrency sweep (45.2 RPS throughput):

  • Model Spend Saved: 54.2% vs. Always-Strong Baseline (GPT-4o)
  • Quality Retention: 98.8% of GPT-4o output accuracy (JSON Schema / AST syntax pass)
  • Gateway P95 Overhead: 120.4 ms (Trie prefix lookup + Route classifier + Schema validation)
  • SLA Success Rate: 99.4%

4. Reproduce Benchmark in 1 Step

Read the detailed evaluation summary: Router Evaluation Summary Report (docs/evaluation_summary.md).

# 1. Stream Hugging Face datasets (WildChat + NoRobots) and run 10,000-request concurrency benchmark
python benchmarks/run_hf_stream_benchmark.py

# 2. Run standard RouterBench policy sweep (Zero Router, KNN, MLP, FrugalGPT Cascade)
python benchmarks/run_router_eval.py
python benchmarks/plot_results.py

This updates benchmarks/results/hf_stream_benchmark_report.md and eval_results.json with reproducible empirical logs.

๐ŸŽจ Interactive Playground & Chaos Simulator

InferRoute includes a built-in interactive control center panel served at the root (/) path:

  • Live Cost Dashboard: Displays money saved ($ USD), tokens saved, TTFT latency, and Redis cache hit rate.
  • Request Pipeline Visualizer: Renders a vertical stepper showing step-by-step processing of the query (Cache checking, Limiter checking, Node executing, and Cascades).
  • Simulated Wallet & Recharge: Top-right wallet indicator showing current credits (e.g., $5.00 trial credit) with a simulated + $10 recharge button.
  • Chaos Engineering & Failure Injection: A panel to manually kill or throttle model nodes, observing the circuit-breaker turning Red and routing self-healing in real-time.

For detailed performance, concurrency, and cost reports under heavy loads, check out the Benchmark Report.

๐Ÿš€ Quick Start (Running Offline in 10 Seconds)

InferRoute features a built-in Simulation Mode (using SQLite and Mock adapters) allowing you to boot and explore the gateway completely for free, offline, with zero real API keys!

1. Requirements

  • Python 3.12 or 3.13
  • Docker & Docker Compose (Optional, for production Postgres/Redis/Grafana observability)

2. Installation 

# Clone the repository
git clone https://github.com/ypeng12/InferRoute.git
cd InferRoute

# Initialize virtual environment
python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

3. Create Local Config File

Create a .env file in the root directory:

DATABASE_URL=sqlite+aiosqlite:///inferroute.db
MOCK_OPENAI=true
MOCK_GEMINI=true
MOCK_VLLM=true
MOCK_OLLAMA=true

4. Start the Gateway 

python -m uvicorn inferroute.main:app --host 127.0.0.1 --port 8080 --reload

Open http://127.0.0.1:8080 in your browser to interact with the dashboard immediately!

๐Ÿ› ๏ธ Unified Integration (OpenAI Drop-In)

InferRoute is compatible with the standard OpenAI API chat completions format. You can switch your existing codebases to run through InferRoute in just one line:

import openai

client = openai.OpenAI(
    base_url="http://localhost:8080/v1",
    api_key="sk-inferroute-demo"  # Custom tenant auth key
)

response = client.chat.completions.create(
    model="edge/auto",  # Dynamic auto-routing
    messages=[
        {"role": "user", "content": "Write a quicksort in Python."}
    ],
    stream=True
)

for chunk in response:
    content = chunk.choices[0].delta.content
    if content:
        print(content, end="", flush=True)

๐Ÿ“š Project Documentation

Explore the following detailed guides for in-depth engineering breakdowns:

๐Ÿ“Š Observability Stack (Production Environment)

In a production environment, spin up the containerized observability stack to monitor traffic, metrics, and traces:

# Boot Postgres, Redis, OTtel Collector, Jaeger, Prometheus, Grafana
docker compose up -d

๐Ÿ›ก๏ธ License

This project is licensed under the MIT License. See LICENSE for details.