backend/docs/MULTI_LLM_ROUTING_SYSTEM.md

Multi-LLM Routing System for AegisLM

Overview

The Multi-LLM Routing System provides a reliable, zero-cost LLM system with automatic failover and role-based routing. It integrates three free-tier providers (Groq, Mistral, HuggingFace) to ensure high availability and cost-effective operation.

Architecture

backend/ai/llm_router/
├── llm_router.py          # Core routing engine with failover logic
├── base_provider.py       # Abstract base classes and data models
├── integration.py          # Drop-in replacement for existing LLM calls
├── __init__.py            # Package exports
└── providers/             # Provider implementations
    ├── __init__.py
    ├── groq_client.py      # Groq API integration
    ├── mistral_client.py    # Mistral API integration
    └── huggingface_client.py # HuggingFace API integration

Features

🔥 Core Capabilities

• Zero-Cost Operation: Uses only free-tier APIs from Groq, Mistral, and HuggingFace
• Automatic Failover: Intelligent provider switching when failures occur
• Role-Based Routing: Task-specific provider selection for optimal performance
• Health Monitoring: Continuous provider health checking and statistics
• Cost Control: Built-in rate limiting and usage tracking
• Timeout Management: Configurable timeouts and retry logic
• Comprehensive Logging: Detailed logging of all routing operations

🛡️ Provider Support

Groq Provider

• Models: llama-3.1-8b-instant, llama-3.1-70b-instant
• Priority: 1 (Highest)
• Rate Limit: 30 RPM
• Features: Fast inference, high-quality responses

Mistral Provider

• Models: mistral-tiny, mistral-small, mistral-medium
• Priority: 2 (Medium)
• Rate Limit: 20 RPM
• Features: Balanced performance, good for judge tasks

HuggingFace Provider

• Models: google/flan-t5-base, microsoft/DialoGPT-medium
• Priority: 3 (Lowest - Fallback)
• Rate Limit: 10 RPM
• Features: Wide model variety, reliable fallback

🧠 Task-Based Routing Logic

The system uses intelligent routing based on task types:

# Generation Tasks
IF task == "generation":
    Groq → Mistral → HuggingFace

# Judge Tasks  
IF task == "judge":
    Mistral → Groq → HuggingFace

# Analysis Tasks
IF task == "analysis":
    Groq → HuggingFace → Mistral

# Summarization Tasks
IF task == "summarization":
    Mistral → Groq → HuggingFace

# Translation Tasks
IF task == "translation":
    Groq → Mistral → HuggingFace

Configuration

Environment Variables

# Core Routing Settings
ENABLE_LLM_ROUTING=false                    # Enable/disable routing system
DEFAULT_LLM_PROVIDER=groq                 # Default provider
ENABLE_LLM_FALLBACK=true                    # Enable automatic fallback
LLM_REQUEST_TIMEOUT_SECONDS=60                 # Request timeout
MAX_LLM_FALLBACK_ATTEMPTS=3                # Max fallback attempts

# Provider Configuration
GROQ_API_KEY=your-groq-api-key-here
GROQ_MODEL=llama-3.1-8b-instant
GROQ_PRIORITY=1
GROQ_RATE_LIMIT_RPM=30

MISTRAL_API_KEY=your-mistral-api-key-here
MISTRAL_MODEL=mistral-tiny
MISTRAL_PRIORITY=2
MISTRAL_RATE_LIMIT_RPM=20

HUGGINGFACE_API_KEY=your-hf-api-key-here
HUGGINGFACE_MODEL=google/flan-t5-base
HUGGINGFACE_PRIORITY=3
HUGGINGFACE_RATE_LIMIT_RPM=10

# Task-Based Routing
GENERATION_PROVIDERS=groq,mistral,huggingface
JUDGE_PROVIDERS=mistral,groq,huggingface
ANALYSIS_PROVIDERS=groq,huggingface,mistral
SUMMARIZATION_PROVIDERS=mistral,groq,huggingface
TRANSLATION_PROVIDERS=groq,mistral,huggingface

# Performance Settings
ENABLE_ROUTER_LOGGING=true
ROUTER_LOG_LEVEL=INFO
ENABLE_ROUTER_METRICS=true
ENABLE_HEALTH_CHECKS=true
HEALTH_CHECK_INTERVAL_SECONDS=300

Usage

Basic Usage

from ai.llm_router import create_llm_router, TaskType

# Create router with default configuration
router = create_llm_router()

# Generate response with automatic routing
result = await router.generate(
    prompt="What is the capital of France?",
    task_type=TaskType.GENERATION
)

print(f"Response: {result.response.content}")
print(f"Provider used: {result.provider_used}")
print(f"Fallback chain: {result.fallback_chain}")

Integration with Existing Code

# Replace existing LLM calls with routed version
from ai.llm_router.integration import generate_with_routing, judge_with_routing

# Instead of: response = await llm.generate(prompt)
response = await generate_with_routing(prompt, task_type="generation")

# Instead of: evaluation = await judge.evaluate(prompt, response)
evaluation = await judge_with_routing(prompt, original_response=response)

Advanced Configuration

from ai.llm_router import RouterConfig, create_llm_router

# Custom routing configuration
config = RouterConfig(
    enable_routing=True,
    default_provider="mistral",
    timeout_seconds=45,
    max_fallback_attempts=5,
    enable_logging=True,
    log_level="DEBUG"
)

router = create_llm_router(config)

Error Handling

Automatic Failover

The system implements sophisticated failover logic:

1. Provider Selection: Choose provider based on task type and availability
2. Health Checking: Verify provider availability before attempting request
3. Retry Logic: Exponential backoff for failed providers
4. Fallback Chain: Automatic progression through provider list
5. Error Recovery: Graceful degradation when all providers fail

Error Types Handled

• API Failures: HTTP errors, invalid responses
• Timeout Errors: Request timeouts, slow providers
• Rate Limiting: RPM limits exceeded
• Network Issues: Connection problems, DNS failures
• Configuration Errors: Invalid API keys, model not found

Monitoring and Statistics

Routing Statistics

# Get comprehensive routing statistics
stats = router.get_routing_statistics()

print(f"Total requests: {stats['total_requests']}")
print(f"Provider usage: {stats['provider_usage']}")
print(f"Fallback events: {stats['fallback_events']}")
print(f"Error count: {len(stats['errors'])}")

Health Monitoring

# Check health of all providers
health_status = await router.health_check()

for provider, status in health_status.items():
    print(f"{provider}: {status.value}")

Performance Metrics

• Request Success Rate: Percentage of successful requests
• Provider Performance: Response times per provider
• Fallback Usage: Frequency of fallback events
• Error Distribution: Types and frequency of errors
• Cost Tracking: Token usage and API calls

Testing

Running Tests

# Run comprehensive test suite
python test_simple_llm_router.py

# Run full integration tests
python tests/test_llm_router.py

Test Coverage

• ✅ Router initialization and configuration
• ✅ Provider chain selection logic
• ✅ Automatic failover mechanisms
• ✅ Error handling and recovery
• ✅ Health monitoring and statistics
• ✅ Provider factory functions
• ✅ Integration layer compatibility
• ✅ Configuration management
• ✅ Task-based routing rules

Performance Optimization

Caching

• Provider Caching: Cache provider responses for identical requests
• Configuration Caching: Cache provider configurations
• Health Result Caching: Cache health check results

Rate Limiting

• Per-Provider Limits: Respect individual provider RPM limits
• Global Rate Limiting: System-wide request throttling
• Adaptive Throttling: Dynamic adjustment based on provider performance

Connection Management

• Connection Pooling: Reuse HTTP connections for efficiency
• Session Management: Persistent sessions where supported
• Timeout Optimization: Adaptive timeouts based on provider performance

Security Considerations

API Key Management

• Environment Variables: API keys stored securely in environment
• No Hardcoded Keys: Never commit API keys to version control
• Key Rotation: Support for API key rotation without restart
• Access Logging: Log API access for security monitoring

Data Privacy

• Minimal Data Transfer: Only send necessary prompt/response data
• No PII Logging: Personal information not logged
• Configurable Retention: Control data retention policies
• Secure Transmission: HTTPS for all API communications

Troubleshooting

Common Issues

Provider Not Available

# Check provider configuration
echo $GROQ_API_KEY
echo $MISTRAL_API_KEY
echo $HUGGINGFACE_API_KEY

# Check provider status
python -c "
from ai.llm_router import create_llm_router
router = create_llm_router()
health = asyncio.run(router.health_check())
print(health)
"

High Fallback Rate

# Check rate limits
python -c "
from ai.llm_router import create_llm_router
router = create_llm_router()
stats = router.get_routing_statistics()
print(f'Fallback rate: {stats[\"fallback_events\"] / stats[\"total_requests\"]:.2%}')
"

Performance Issues

# Enable debug logging
export ROUTER_LOG_LEVEL=DEBUG

# Check provider response times
python -c "
from ai.llm_router import create_llm_router
router = create_llm_router()
stats = router.get_routing_statistics()
print(stats)
"

Best Practices

Configuration

1. Start with Routing Disabled: Test individual providers first
2. Use Appropriate Models: Choose models based on task requirements
3. Set Realistic Timeouts: Balance between reliability and responsiveness
4. Monitor Usage: Track provider performance and costs
5. Plan Fallback Strategy: Configure provider chains wisely

Integration

1. Gradual Rollout: Enable routing for subset of users first
2. Monitor Performance: Track success rates and response times
3. A/B Testing: Compare routing vs. single provider performance
4. Fallback Testing: Regularly test fallback mechanisms
5. Documentation: Keep integration documentation updated

Operations

1. Health Monitoring: Set up automated health checks
2. Alerting: Configure alerts for high failure rates
3. Log Analysis: Regularly review routing logs
4. Performance Tuning: Adjust timeouts and retry logic
5. Capacity Planning: Scale based on usage patterns

Future Enhancements

Planned Features

• Model Auto-Selection: Automatic model selection based on prompt analysis
• Performance-Based Routing: Route based on historical performance data
• Load Balancing: Distribute requests across multiple provider instances
• Advanced Caching: Semantic similarity-based caching
• Cost Optimization: Dynamic provider selection based on cost/performance

Integration Roadmap

1. Phase 1: Core routing system (✅ Complete)
2. Phase 2: Advanced monitoring and analytics
3. Phase 3: Performance optimization and auto-tuning
4. Phase 4: Machine learning-based routing decisions

---

🎯 System Status: PRODUCTION READY

The Multi-LLM Routing System is fully implemented and tested with:

• ✅ Zero-Cost Operation: Free-tier providers only
• ✅ High Availability: Automatic failover and health monitoring
• ✅ Intelligent Routing: Task-based provider selection
• ✅ Comprehensive Error Handling: Graceful degradation and recovery
• ✅ Performance Monitoring: Detailed statistics and metrics
• ✅ Easy Integration: Drop-in replacement for existing LLM calls
• ✅ Production Configuration: Environment-based configuration management
• ✅ Security Best Practices: Secure API key management
• ✅ Extensible Architecture: Easy addition of new providers

The system provides enterprise-grade reliability while maintaining zero operational costs.