implementation_plan.md

Implementation Plan: Complexity-Aware Multi-Tier Routing & Cost/Resource Optimization

This plan outlines the design and integration of an enterprise-grade Complexity-Aware Multi-Tier Routing Engine into the LiteLLM Proxy server. It classifies incoming prompts dynamically into Low, Medium, and High complexity levels, and routes them to optimal physical LLM backends to maximize response quality while minimizing credit expenditure and system resource load.

---

User Review Required

Dynamic Model Catalog Upgrades & Local Backup Placement To support high-fidelity reasoning for High Complexity queries (like coding, algorithm design, or multi-step logic), we will add high-parameter 70B models to our catalog using your existing API keys:

1. Groq: Add groq/llama-3.3-70b-versatile ($0.70/M tokens, 128k context).
2. Together AI: Add together_ai/meta-llama/Llama-3.3-70B-Instruct-Turbo ($0.90/M tokens, 128k context).

Local Backup Placement: Per your request, the local ollama/llama3.1 model (cost: $0.00/M tokens) will be placed inside the backup-cluster (previously under its own local cluster). This allows the backup cluster to dynamically load-balance or failover between premium Together AI and zero-cost local Ollama!

Backward Compatibility Guard All existing virtual model names (such as primary-cluster, backup-cluster, and local-fallback-cluster) and general fallback chains will remain fully supported. The complexity-aware router will intercept requests to these endpoints and intelligently route them to the most optimal underlying physical endpoint matching the prompt's characteristics.

---

Proposed Changes

1. Configuration & Models

[MODIFY] config.yaml

• Add explicit complexity_tier metadata properties to existing models.
• Move local Ollama (ollama/llama3.1) into backup-cluster.
• Add new High-Complexity premium endpoints to the cluster lists:
  • Groq Llama 3.3 70B (groq/llama-3.3-70b-versatile) under primary-cluster (tier: high, cost: 0.70, tpr: 131072).
  • Together AI Llama 3.3 70B (together_ai/meta-llama/Llama-3.3-70B-Instruct-Turbo) under backup-cluster (tier: high, cost: 0.90, tpr: 131072).

2. Configuration Parser

[MODIFY] proxy/config.py

• Extend ModelEndpointConfig Pydantic model with a new field: complexity_tier: Optional[str] = Field(None, description="Complexity tier for this endpoint: low, medium, or high").
• Update load_config parser to read complexity_tier from litellm_params.
• Add an intelligent fallback to default the tier based on pricing and context if not explicitly set in YAML:
  • cost_per_million >= 0.50 -> high
  • cost_per_million >= 0.04 -> medium
  • cost_per_million < 0.04 -> low

3. Routing Engine

[MODIFY] proxy/router.py

• Prompt Complexity Classifier: Implement classify_prompt_complexity(self, messages: List[Dict[str, str]], required_context: int) -> str to assess prompts based on:
  • Required Context Length: Any query requiring >8K context is auto-classified as high.
  • Semantic Indicators: Scans user input for coding keywords (python, sql, refactor, debug), mathematical/logical keywords (solve, prove, theorem, logic), or deep analysis tasks (optimize, architecture, step-by-step).
• Complexity-Aware Selection: Refactor execute_chat_completion to score available endpoints using a multi-objective utility function:
  • Tier Match: Strongly rewards endpoints aligning with the classified complexity of the prompt.
  • Cost Optimization: Prefers nodes with the lowest cost_per_million.
  • Resource Optimization: Prefers nodes with the lowest RPM/TPM usage to spread system load.
• Ensure context-window overflows dynamically trigger high-capacity models regardless of semantic complexity.

4. Integration Verification

[MODIFY] test_proxy.py

• Add a new comprehensive unit test: test_7_complexity_routing:
  • Submit a conversational greeting ("Hello! How is it going?") and verify it routes to a low complexity node (Cerebras or Ollama).
  • Submit a complex python debugging prompt ("Optimize this SQL query and write a python refactoring function...") and verify it routes to a high complexity node (Groq 70B or Together 70B, or standard large backup if sandbox).

---

Verification Plan

Automated Tests

• Run python3 test_proxy.py to ensure all 7 tests pass successfully with 100% success.

Manual Sandbox Verification

• Launch the FastAPI server: python3 main.py.
• Run mock sandbox queries from the console or Streamlit dashboard on Port 8502.
• Observe logs in the Telemetry dashboard confirming the dynamic complexity assessments and selected cost-optimized targets.