src/README_API_UTILS.md

# MorphGuard API Utilities

The `api_utils.py` module provides a sophisticated and resilient API client system with intelligent fallbacks for the MorphGuard application. This system is designed to handle network instability, server errors, and other common issues that can affect API communication.

## Key Features

- **Smart Request Queuing**: Prioritize critical requests to ensure they are processed first
- **Comprehensive Error Handling**: Detailed error diagnostics and classification
- **Intelligent Caching**: Stale-while-revalidate pattern for improved performance and resilience
- **Circuit Breaker Pattern**: Prevent cascading failures by failing fast when services are down
- **Extended Metrics and Monitoring**: Track performance and reliability metrics
- **Context-Aware Retry Strategies**: Intelligent backoff and jitter for optimal retry behavior

## Core Components

### 1. CircuitBreaker

Implements the circuit breaker pattern with three states:
- **CLOSED**: Normal operation, requests are sent normally
- **OPEN**: Circuit is tripped, all requests fail fast
- **HALF-OPEN**: Testing if the service has recovered

```python
# Example usage
cb = CircuitBreaker(failure_threshold=5, recovery_timeout=30.0)

if cb.allow_request():
    try:
        # Make request
        result = make_api_call()
        cb.record_success()
        return result
    except Exception as e:
        cb.record_failure()
        raise
else:
    # Circuit is open, use fallback or fail fast
    return fallback_data
```

### 2. StaleWhileRevalidateCache

Enhanced cache with stale-while-revalidate strategy:
- Returns stale data immediately while asynchronously fetching fresh data
- Reduces perceived latency and improves availability
- Supports custom TTL for different cache entries

```python
# Example usage
cache = StaleWhileRevalidateCache(cache_dir=".cache", stale_timeout=300.0)

# Define refresh callback
def refresh_data():
    return fetch_fresh_data_from_api()

# Get data from cache, refresh in background if stale
data, is_fresh = cache.get("cache_key", refresh_callback=refresh_data)

# Use data immediately, even if stale
process_data(data)
```

### 3. APIUtils

The main API client with comprehensive resilience features:
- Configurable timeout, retries, and caching
- Circuit breaker pattern integration
- Priority-based request handling
- Detailed metrics and telemetry

```python
# Example usage
api = get_api_utils({
    "base_url": "https://api.example.com",
    "timeout": 30.0,
    "retries": 3,
    "cache_enabled": True
})

# Make a GET request with fallback
try:
    result = api.get("/users", {"limit": 10}, {
        "fallback": {"users": []},
        "priority": "high"
    })
    process_users(result)
except Exception as e:
    handle_error(e)
```

### 4. Decorator Functions

Utility decorators for applying resilience patterns to functions:

#### @retry

Automatically retry functions on failure:

```python
@retry(retries=3, retry_delay=1.0, fallback=default_value)
def fetch_user_data(user_id):
    # This function will be retried up to 3 times if it fails
    return api_call(f"/users/{user_id}")
```

#### @circuit_breaker

Apply circuit breaker pattern to functions:

```python
@circuit_breaker(failure_threshold=5, recovery_timeout=30.0, fallback=empty_list)
def search_products(query):
    # This function will fail fast if it fails 5 times in a row
    return api_call("/search", {"q": query})
```

#### @cacheable

Cache function results with TTL:

```python
@cacheable(ttl=60.0)  # Cache for 1 minute
def get_product_categories():
    # Results will be cached for 60 seconds
    return api_call("/categories")
```

## Usage Example

Here's a complete example of using the API utilities:

```python
from src.api_utils import get_api_utils, retry, circuit_breaker, cacheable

# Configure API utilities
api = get_api_utils({
    "base_url": "https://api.example.com",
    "timeout": 30.0,
    "retries": 3,
    "cache_enabled": True
})

# Make a simple GET request
users = api.get("/users")

# POST request with error handling
try:
    new_user = api.post("/users", {
        "name": "John Doe",
        "email": "john@example.com"
    }, {
        "critical": True,  # High priority
        "timeout": 60.0    # Longer timeout
    })
    print(f"Created user: {new_user}")
except Exception as e:
    print(f"Failed to create user: {e}")

# Function with retry decorator
@retry(retries=3, retry_delay=1.0)
def fetch_user_data(user_id):
    return api.get(f"/users/{user_id}")

# Function with circuit breaker
@circuit_breaker(failure_threshold=5, recovery_timeout=30.0)
def search_products(query):
    return api.get("/search", {"q": query})

# Function with caching
@cacheable(ttl=60.0)
def get_product_categories():
    return api.get("/categories")

# Get API metrics
metrics = api.get_metrics()
print(f"API metrics: {metrics}")
```

## Configuration Options

The API utilities can be configured with the following options:

| Option | Description | Default |
|--------|-------------|---------|
| `base_url` | Base URL for the API | `"http://localhost:5000/api"` |
| `timeout` | Request timeout in seconds | `30.0` |
| `retries` | Number of retries for failed requests | `3` |
| `retry_delay` | Base delay between retries in seconds | `1.0` |
| `retry_status_codes` | HTTP status codes to retry | `[408, 429, 500, 502, 503, 504]` |
| `cache_enabled` | Whether to enable caching | `True` |
| `cache_dir` | Directory to store cache files | `".mg_api_cache"` |
| `max_cache_entries` | Maximum number of in-memory cache entries | `100` |
| `stale_timeout` | Time in seconds before data is considered stale | `300.0` |
| `revalidation_timeout` | Maximum age for using stale data | `600.0` |
| `circuit_breaker_enabled` | Whether to enable circuit breaker | `True` |
| `failure_threshold` | Number of failures before circuit opens | `5` |
| `recovery_timeout` | Time in seconds before circuit half-opens | `30.0` |
| `priority_queue_enabled` | Whether to enable priority-based request queue | `True` |
| `max_queue_size` | Maximum size of request queue | `100` |
| `debug` | Whether to enable debug logging | `False` |

## Error Handling

The API utilities integrate with MorphGuard's error handling system to provide detailed error information:

```python
try:
    result = api.get("/users/123")
except APIError as e:
    if e.code == ErrorCode.NOT_FOUND:
        print("User not found")
    elif e.code == ErrorCode.UNAUTHORIZED:
        print("Authentication required")
    else:
        print(f"API error: {e.message}")
except NetworkError as e:
    print(f"Network error: {e.message}")
except Exception as e:
    print(f"Unexpected error: {e}")
```

## Metrics and Monitoring

The API utilities track various metrics for monitoring:

```python
metrics = api.get_metrics()
print(f"Total requests: {metrics['total_requests']}")
print(f"Successful requests: {metrics['successful_requests']}")
print(f"Failed requests: {metrics['failed_requests']}")
print(f"Cached responses: {metrics['cached_responses']}")
print(f"Stale responses: {metrics['stale_responses']}")
print(f"Retries: {metrics['retries']}")
print(f"Circuit breaks: {metrics['circuit_breaks']}")
print(f"Average response time: {metrics['total_time'] / max(1, metrics['total_requests']):.2f}s")
```

## Testing

The API utilities include comprehensive unit tests to ensure reliability:

```bash
# Run all tests
python -m unittest tests.test_api_utils

# Run specific test class
python -m unittest tests.test_api_utils.TestCircuitBreaker

# Run specific test method
python -m unittest tests.test_api_utils.TestAPIUtils.test_circuit_breaker
```