docs/API_KEY_MANAGER_GUIDE.md

API Key Manager Guide

Overview

The API Key Manager provides automatic failover and rotation across multiple API keys for the same service. If one API key fails due to rate limiting or errors, the system automatically switches to a backup key.

Features

✅ Automatic Failover - Switches to backup keys when one fails
✅ Load Balancing - Rotates through keys using round-robin
✅ Failure Tracking - Blocks keys after consecutive failures
✅ Auto-Recovery - Unblocks keys after cooldown period
✅ Success Rate Monitoring - Tracks performance of each key
✅ Thread-Safe - Can be used in multi-threaded environments

Configuration

Setting Up Multiple API Keys

You can configure multiple API keys for each service using environment variables:

Method 1: Numbered Keys (Recommended)

# Gemini API Keys
export GEMINI_API_KEY_1="your-first-gemini-key"
export GEMINI_API_KEY_2="your-second-gemini-key"
export GEMINI_API_KEY_3="your-third-gemini-key"

# NVIDIA API Keys
export NVIDIA_API_KEY_1="your-first-nvidia-key"
export NVIDIA_API_KEY_2="your-second-nvidia-key"

# OpenRouter API Keys (for Nova)
export OPENROUTER_API_KEY_1="your-first-openrouter-key"
export OPENROUTER_API_KEY_2="your-second-openrouter-key"

Method 2: Single Key (Backward Compatible)

export GEMINI_API_KEY="your-gemini-key"
export NVIDIA_API_KEY="your-nvidia-key"
export OPENROUTER_API_KEY="your-openrouter-key"

Method 3: Mixed (Both Work Together)

# These will all be combined into the pool
export GEMINI_API_KEY="key-1"
export GEMINI_API_KEY_1="key-2"
export GEMINI_API_KEY_2="key-3"
# Result: 3 keys total (duplicates are automatically removed)

Supported Services

Service	Environment Variable Pattern	Used For
nvidia	NVIDIA_API_KEY or NVIDIA_API_KEY_1, NVIDIA_API_KEY_2, etc.	OCR processing
gemini	GEMINI_API_KEY, GOOGLE_API_KEY, or numbered variants	Question classification, Q&A extraction
openrouter	OPENROUTER_API_KEY or OPENROUTER_API_KEY_1, etc.	Amazon Nova classification

How It Works

1. Key Rotation

Keys are automatically rotated using round-robin:

Request 1 → Key 1
Request 2 → Key 2  
Request 3 → Key 3
Request 4 → Key 1 (back to start)

2. Failure Handling

When a key fails:

• Failure count is incremented
• After 3 consecutive failures, the key is blocked for 5 minutes
• System automatically switches to next available key
• After cooldown period, key is automatically unblocked

3. Success Tracking

When a key succeeds:

• Success count is incremented
• Failure count is reset to 0
• Key is marked as available
• System rotates to next key for load balancing

Usage in Code

Automatic (Already Integrated)

The API Key Manager is already integrated into:

• ✅ gemini_classifier.py - Gemini question classification
• ✅ nova_classifier.py - Nova question classification
• ✅ processing.py - NVIDIA OCR API

No code changes needed! Just set up multiple API keys and the system handles the rest.

Manual Usage (Advanced)

If you need to add API key management to other modules:

from api_key_manager import get_api_key_manager

# Get the manager instance
manager = get_api_key_manager()

# Get an API key
api_key, key_index = manager.get_key('gemini')

if api_key:
    try:
        # Make your API call
        response = make_api_call(api_key)
        
        # Mark as successful
        manager.mark_success('gemini', key_index)
        
    except Exception as e:
        # Mark as failed (will block after 3 failures)
        manager.mark_failure('gemini', key_index)
else:
    print("No API keys available!")

Monitoring

Get Service Status

from api_key_manager import get_api_key_manager

manager = get_api_key_manager()

# Get status for one service
status = manager.get_service_status('gemini')
print(f"Available keys: {status['available_keys']}/{status['total_keys']}")
print(f"Blocked keys: {status['blocked_keys']}")

# Get status for all services
all_status = manager.get_all_services_status()
for service, info in all_status.items():
    print(f"{service}: {info['available_keys']}/{info['total_keys']} keys available")

Example Output

{
  "service": "gemini",
  "available": true,
  "total_keys": 3,
  "available_keys": 2,
  "blocked_keys": 1,
  "keys": [
    {
      "index": 0,
      "is_available": true,
      "is_blocked": false,
      "failure_count": 0,
      "total_requests": 15,
      "success_rate": 100.0,
      "blocked_until": null
    },
    {
      "index": 1,
      "is_available": true,
      "is_blocked": false,
      "failure_count": 0,
      "total_requests": 12,
      "success_rate": 100.0,
      "blocked_until": null
    },
    {
      "index": 2,
      "is_available": false,
      "is_blocked": true,
      "failure_count": 3,
      "total_requests": 8,
      "success_rate": 62.5,
      "blocked_until": "2025-12-08T04:30:00.000000"
    }
  ]
}

Configuration Options

Block Duration

By default, keys are blocked for 5 minutes after 3 failures. You can customize this:

# Block for 10 minutes instead
manager.mark_failure('gemini', key_index, block_duration_minutes=10)

Failure Threshold

The failure threshold is currently hardcoded to 3 consecutive failures. This is defined in api_key_manager.py in the mark_failure() method:

if self.failure_count >= 3:
    self.is_blocked = True

Troubleshooting

Problem: "No API keys available"

Cause: All keys are blocked or no keys are configured.

Solution:

1. Check environment variables are set correctly
2. Wait for cooldown period (5 minutes)
3. Manually reset the service:

   manager.reset_service('gemini')
   

Problem: Keys getting blocked frequently

Cause: Rate limiting or invalid API keys.

Solution:

1. Check API key validity
2. Verify rate limits with your API provider
3. Add more API keys to distribute load
4. Increase block duration to avoid rapid retries

Problem: Not using multiple keys even though they're configured

Cause: Check if keys are being loaded correctly.

Solution:

manager = get_api_key_manager()
status = manager.get_service_status('gemini')
print(f"Total keys loaded: {status['total_keys']}")

Best Practices

1. Use at least 2-3 keys per service for better reliability
2. Monitor success rates to identify problematic keys
3. Stagger API requests to avoid hitting rate limits
4. Keep backup keys from different accounts if possible
5. Test keys periodically to ensure they're still valid

Logging

The API Key Manager logs important events:

INFO: Loaded API keys: NVIDIA=2, Gemini=3, OpenRouter=2
INFO: Registered 3 API key(s) for service: gemini
DEBUG: Using API key 1/3 for gemini
DEBUG: API key 1 for gemini marked as successful
WARNING: API key 2 for gemini marked as failed
WARNING: API key for gemini blocked until 2025-12-08 04:30:00 after 3 failures
INFO: API key for gemini unblocked after cooldown period

Architecture

┌─────────────────────────────────────────────────────────────┐
│                    API Key Manager                           │
├─────────────────────────────────────────────────────────────┤
│                                                               │
│  Service: nvidia        Service: gemini    Service: openrouter│
│  ┌──────────────┐      ┌──────────────┐   ┌──────────────┐  │
│  │ Key 1 ✓      │      │ Key 1 ✓      │   │ Key 1 ✓      │  │
│  │ Key 2 ✓      │      │ Key 2 ✓      │   │ Key 2 ✓      │  │
│  └──────────────┘      │ Key 3 ✗      │   └──────────────┘  │
│                        │ (blocked)    │                      │
│                        └──────────────┘                      │
│                                                               │
│  Features:                                                    │
│  • Round-robin rotation                                      │
│  • Automatic failover                                        │
│  • Failure tracking                                          │
│  • Auto-recovery after cooldown                              │
│                                                               │
└─────────────────────────────────────────────────────────────┘
         │              │              │
         ▼              ▼              ▼
   processing.py   gemini_classifier.py   nova_classifier.py
   (NVIDIA OCR)    (Gemini AI)           (Amazon Nova)

Future Enhancements

Potential improvements for the API Key Manager:

• Web dashboard for monitoring key status
• Configurable failure threshold per service
• Exponential backoff for blocked keys
• API key health checks
• Cost tracking per key
• Rate limit detection and adaptive throttling
• Database persistence for key statistics
• Email alerts when all keys are blocked
• Integration with settings page for user-visible status