# Multi-LLM Routing System for AegisLM

## Overview

The Multi-LLM Routing System provides a reliable, zero-cost LLM system with automatic failover and role-based routing. It integrates three free-tier providers (Groq, Mistral, HuggingFace) to ensure high availability and cost-effective operation.

## Architecture

```
backend/ai/llm_router/
├── llm_router.py          # Core routing engine with failover logic
├── base_provider.py       # Abstract base classes and data models
├── integration.py          # Drop-in replacement for existing LLM calls
├── __init__.py            # Package exports
└── providers/             # Provider implementations
    ├── __init__.py
    ├── groq_client.py      # Groq API integration
    ├── mistral_client.py    # Mistral API integration
    └── huggingface_client.py # HuggingFace API integration
```

## Features

### 🔥 Core Capabilities

- **Zero-Cost Operation**: Uses only free-tier APIs from Groq, Mistral, and HuggingFace
- **Automatic Failover**: Intelligent provider switching when failures occur
- **Role-Based Routing**: Task-specific provider selection for optimal performance
- **Health Monitoring**: Continuous provider health checking and statistics
- **Cost Control**: Built-in rate limiting and usage tracking
- **Timeout Management**: Configurable timeouts and retry logic
- **Comprehensive Logging**: Detailed logging of all routing operations

### 🛡️ Provider Support

#### **Groq Provider**
- **Models**: llama-3.1-8b-instant, llama-3.1-70b-instant
- **Priority**: 1 (Highest)
- **Rate Limit**: 30 RPM
- **Features**: Fast inference, high-quality responses

#### **Mistral Provider**
- **Models**: mistral-tiny, mistral-small, mistral-medium
- **Priority**: 2 (Medium)
- **Rate Limit**: 20 RPM
- **Features**: Balanced performance, good for judge tasks

#### **HuggingFace Provider**
- **Models**: google/flan-t5-base, microsoft/DialoGPT-medium
- **Priority**: 3 (Lowest - Fallback)
- **Rate Limit**: 10 RPM
- **Features**: Wide model variety, reliable fallback

### 🧠 Task-Based Routing Logic

The system uses intelligent routing based on task types:

```python
# Generation Tasks
IF task == "generation":
    Groq → Mistral → HuggingFace

# Judge Tasks  
IF task == "judge":
    Mistral → Groq → HuggingFace

# Analysis Tasks
IF task == "analysis":
    Groq → HuggingFace → Mistral

# Summarization Tasks
IF task == "summarization":
    Mistral → Groq → HuggingFace

# Translation Tasks
IF task == "translation":
    Groq → Mistral → HuggingFace
```

## Configuration

### Environment Variables

```bash
# Core Routing Settings
ENABLE_LLM_ROUTING=false                    # Enable/disable routing system
DEFAULT_LLM_PROVIDER=groq                 # Default provider
ENABLE_LLM_FALLBACK=true                    # Enable automatic fallback
LLM_REQUEST_TIMEOUT_SECONDS=60                 # Request timeout
MAX_LLM_FALLBACK_ATTEMPTS=3                # Max fallback attempts

# Provider Configuration
GROQ_API_KEY=your-groq-api-key-here
GROQ_MODEL=llama-3.1-8b-instant
GROQ_PRIORITY=1
GROQ_RATE_LIMIT_RPM=30

MISTRAL_API_KEY=your-mistral-api-key-here
MISTRAL_MODEL=mistral-tiny
MISTRAL_PRIORITY=2
MISTRAL_RATE_LIMIT_RPM=20

HUGGINGFACE_API_KEY=your-hf-api-key-here
HUGGINGFACE_MODEL=google/flan-t5-base
HUGGINGFACE_PRIORITY=3
HUGGINGFACE_RATE_LIMIT_RPM=10

# Task-Based Routing
GENERATION_PROVIDERS=groq,mistral,huggingface
JUDGE_PROVIDERS=mistral,groq,huggingface
ANALYSIS_PROVIDERS=groq,huggingface,mistral
SUMMARIZATION_PROVIDERS=mistral,groq,huggingface
TRANSLATION_PROVIDERS=groq,mistral,huggingface

# Performance Settings
ENABLE_ROUTER_LOGGING=true
ROUTER_LOG_LEVEL=INFO
ENABLE_ROUTER_METRICS=true
ENABLE_HEALTH_CHECKS=true
HEALTH_CHECK_INTERVAL_SECONDS=300
```

## Usage

### Basic Usage

```python
from ai.llm_router import create_llm_router, TaskType

# Create router with default configuration
router = create_llm_router()

# Generate response with automatic routing
result = await router.generate(
    prompt="What is the capital of France?",
    task_type=TaskType.GENERATION
)

print(f"Response: {result.response.content}")
print(f"Provider used: {result.provider_used}")
print(f"Fallback chain: {result.fallback_chain}")
```

### Integration with Existing Code

```python
# Replace existing LLM calls with routed version
from ai.llm_router.integration import generate_with_routing, judge_with_routing

# Instead of: response = await llm.generate(prompt)
response = await generate_with_routing(prompt, task_type="generation")

# Instead of: evaluation = await judge.evaluate(prompt, response)
evaluation = await judge_with_routing(prompt, original_response=response)
```

### Advanced Configuration

```python
from ai.llm_router import RouterConfig, create_llm_router

# Custom routing configuration
config = RouterConfig(
    enable_routing=True,
    default_provider="mistral",
    timeout_seconds=45,
    max_fallback_attempts=5,
    enable_logging=True,
    log_level="DEBUG"
)

router = create_llm_router(config)
```

## Error Handling

### Automatic Failover

The system implements sophisticated failover logic:

1. **Provider Selection**: Choose provider based on task type and availability
2. **Health Checking**: Verify provider availability before attempting request
3. **Retry Logic**: Exponential backoff for failed providers
4. **Fallback Chain**: Automatic progression through provider list
5. **Error Recovery**: Graceful degradation when all providers fail

### Error Types Handled

- **API Failures**: HTTP errors, invalid responses
- **Timeout Errors**: Request timeouts, slow providers
- **Rate Limiting**: RPM limits exceeded
- **Network Issues**: Connection problems, DNS failures
- **Configuration Errors**: Invalid API keys, model not found

## Monitoring and Statistics

### Routing Statistics

```python
# Get comprehensive routing statistics
stats = router.get_routing_statistics()

print(f"Total requests: {stats['total_requests']}")
print(f"Provider usage: {stats['provider_usage']}")
print(f"Fallback events: {stats['fallback_events']}")
print(f"Error count: {len(stats['errors'])}")
```

### Health Monitoring

```python
# Check health of all providers
health_status = await router.health_check()

for provider, status in health_status.items():
    print(f"{provider}: {status.value}")
```

### Performance Metrics

- **Request Success Rate**: Percentage of successful requests
- **Provider Performance**: Response times per provider
- **Fallback Usage**: Frequency of fallback events
- **Error Distribution**: Types and frequency of errors
- **Cost Tracking**: Token usage and API calls

## Testing

### Running Tests

```bash
# Run comprehensive test suite
python test_simple_llm_router.py

# Run full integration tests
python tests/test_llm_router.py
```

### Test Coverage

- ✅ Router initialization and configuration
- ✅ Provider chain selection logic
- ✅ Automatic failover mechanisms
- ✅ Error handling and recovery
- ✅ Health monitoring and statistics
- ✅ Provider factory functions
- ✅ Integration layer compatibility
- ✅ Configuration management
- ✅ Task-based routing rules

## Performance Optimization

### Caching

- **Provider Caching**: Cache provider responses for identical requests
- **Configuration Caching**: Cache provider configurations
- **Health Result Caching**: Cache health check results

### Rate Limiting

- **Per-Provider Limits**: Respect individual provider RPM limits
- **Global Rate Limiting**: System-wide request throttling
- **Adaptive Throttling**: Dynamic adjustment based on provider performance

### Connection Management

- **Connection Pooling**: Reuse HTTP connections for efficiency
- **Session Management**: Persistent sessions where supported
- **Timeout Optimization**: Adaptive timeouts based on provider performance

## Security Considerations

### API Key Management

- **Environment Variables**: API keys stored securely in environment
- **No Hardcoded Keys**: Never commit API keys to version control
- **Key Rotation**: Support for API key rotation without restart
- **Access Logging**: Log API access for security monitoring

### Data Privacy

- **Minimal Data Transfer**: Only send necessary prompt/response data
- **No PII Logging**: Personal information not logged
- **Configurable Retention**: Control data retention policies
- **Secure Transmission**: HTTPS for all API communications

## Troubleshooting

### Common Issues

#### Provider Not Available
```bash
# Check provider configuration
echo $GROQ_API_KEY
echo $MISTRAL_API_KEY
echo $HUGGINGFACE_API_KEY

# Check provider status
python -c "
from ai.llm_router import create_llm_router
router = create_llm_router()
health = asyncio.run(router.health_check())
print(health)
"
```

#### High Fallback Rate
```bash
# Check rate limits
python -c "
from ai.llm_router import create_llm_router
router = create_llm_router()
stats = router.get_routing_statistics()
print(f'Fallback rate: {stats[\"fallback_events\"] / stats[\"total_requests\"]:.2%}')
"
```

#### Performance Issues
```bash
# Enable debug logging
export ROUTER_LOG_LEVEL=DEBUG

# Check provider response times
python -c "
from ai.llm_router import create_llm_router
router = create_llm_router()
stats = router.get_routing_statistics()
print(stats)
"
```

## Best Practices

### Configuration

1. **Start with Routing Disabled**: Test individual providers first
2. **Use Appropriate Models**: Choose models based on task requirements
3. **Set Realistic Timeouts**: Balance between reliability and responsiveness
4. **Monitor Usage**: Track provider performance and costs
5. **Plan Fallback Strategy**: Configure provider chains wisely

### Integration

1. **Gradual Rollout**: Enable routing for subset of users first
2. **Monitor Performance**: Track success rates and response times
3. **A/B Testing**: Compare routing vs. single provider performance
4. **Fallback Testing**: Regularly test fallback mechanisms
5. **Documentation**: Keep integration documentation updated

### Operations

1. **Health Monitoring**: Set up automated health checks
2. **Alerting**: Configure alerts for high failure rates
3. **Log Analysis**: Regularly review routing logs
4. **Performance Tuning**: Adjust timeouts and retry logic
5. **Capacity Planning**: Scale based on usage patterns

## Future Enhancements

### Planned Features

- **Model Auto-Selection**: Automatic model selection based on prompt analysis
- **Performance-Based Routing**: Route based on historical performance data
- **Load Balancing**: Distribute requests across multiple provider instances
- **Advanced Caching**: Semantic similarity-based caching
- **Cost Optimization**: Dynamic provider selection based on cost/performance

### Integration Roadmap

1. **Phase 1**: Core routing system (✅ Complete)
2. **Phase 2**: Advanced monitoring and analytics
3. **Phase 3**: Performance optimization and auto-tuning
4. **Phase 4**: Machine learning-based routing decisions

---

## 🎯 **System Status: PRODUCTION READY**

The Multi-LLM Routing System is fully implemented and tested with:

- ✅ **Zero-Cost Operation**: Free-tier providers only
- ✅ **High Availability**: Automatic failover and health monitoring
- ✅ **Intelligent Routing**: Task-based provider selection
- ✅ **Comprehensive Error Handling**: Graceful degradation and recovery
- ✅ **Performance Monitoring**: Detailed statistics and metrics
- ✅ **Easy Integration**: Drop-in replacement for existing LLM calls
- ✅ **Production Configuration**: Environment-based configuration management
- ✅ **Security Best Practices**: Secure API key management
- ✅ **Extensible Architecture**: Easy addition of new providers

**The system provides enterprise-grade reliability while maintaining zero operational costs.**