Spaces:

elmerzole
/

llm-api-proxy

Paused

App Files Files Community

llm-api-proxy / DOCUMENTATION.md

Mirrowel

Merge remote-tracking branch 'origin/feature/filtering-tool' into dev

cfa8697 about 2 months ago

preview code

raw

history blame contribute delete

70.9 kB

Technical Documentation: Universal LLM API Proxy & Resilience Library

This document provides a detailed technical explanation of the project's architecture, internal components, and data flows. It is intended for developers who want to understand how the system achieves high availability and resilience.

1. Architecture Overview

The project is a monorepo containing two primary components:

The Proxy Application (proxy_app): This is the user-facing component. It's a FastAPI application that acts as a universal gateway. It uses litellm to translate requests to various provider formats and includes:
- Batch Manager: Optimizes high-volume embedding requests.
- Detailed Logger: Provides per-request file logging for debugging.
- OpenAI-Compatible Endpoints: /v1/chat/completions, /v1/embeddings, etc.
- Model Filter GUI: Visual interface for configuring model ignore/whitelist rules per provider (see Section 6).
The Resilience Library (rotator_library): This is the core engine that provides high availability. It is consumed by the proxy app to manage a pool of API keys, handle errors gracefully, and ensure requests are completed successfully even when individual keys or provider endpoints face issues.

This architecture cleanly separates the API interface from the resilience logic, making the library a portable and powerful tool for any application needing robust API key management.

2. `rotator_library` - The Resilience Engine

This library is the heart of the project, containing all the logic for managing a pool of API keys, tracking their usage, and handling provider interactions to ensure application resilience.

2.1. `client.py` - The `RotatingClient`

The RotatingClient is the central class that orchestrates all operations. It is designed as a long-lived, async-native object.

Initialization

The client is initialized with your provider API keys, retry settings, and a new global_timeout.

client = RotatingClient(
    api_keys=api_keys,
    oauth_credentials=oauth_credentials,
    max_retries=2,
    usage_file_path="key_usage.json",
    configure_logging=True,
    global_timeout=30,
    abort_on_callback_error=True,
    litellm_provider_params={},
    ignore_models={},
    whitelist_models={},
    enable_request_logging=False,
    max_concurrent_requests_per_key={}
)

api_keys (Optional[Dict[str, List[str]]], default: None): A dictionary mapping provider names to a list of API keys.
oauth_credentials (Optional[Dict[str, List[str]]], default: None): A dictionary mapping provider names to a list of file paths to OAuth credential JSON files.
max_retries (int, default: 2): The number of times to retry a request with the same key if a transient server error occurs.
usage_file_path (str, default: "key_usage.json"): The path to the JSON file where usage statistics are persisted.
configure_logging (bool, default: True): If True, configures the library's logger to propagate logs to the root logger.
global_timeout (int, default: 30): A hard time limit (in seconds) for the entire request lifecycle.
abort_on_callback_error (bool, default: True): If True, any exception raised by pre_request_callback will abort the request.
litellm_provider_params (Optional[Dict[str, Any]], default: None): Extra parameters to pass to litellm for specific providers.
ignore_models (Optional[Dict[str, List[str]]], default: None): Blacklist of models to exclude (supports wildcards).
whitelist_models (Optional[Dict[str, List[str]]], default: None): Whitelist of models to always include, overriding ignore_models.
enable_request_logging (bool, default: False): If True, enables detailed per-request file logging.
max_concurrent_requests_per_key (Optional[Dict[str, int]], default: None): Max concurrent requests allowed for a single API key per provider.
rotation_tolerance (float, default: 3.0): Controls the credential rotation strategy. See Section 2.2 for details.

Core Responsibilities

Lifecycle Management: Manages a shared httpx.AsyncClient for all non-blocking HTTP requests.
Key Management: Interfacing with the UsageManager to acquire and release API keys based on load and health.
Plugin System: Dynamically loading and using provider-specific plugins from the providers/ directory.
Execution Logic: Executing API calls via litellm with a robust, deadline-driven retry and key selection strategy.
Streaming Safety: Providing a safe, stateful wrapper (_safe_streaming_wrapper) for handling streaming responses, buffering incomplete JSON chunks, and detecting mid-stream errors.
Model Filtering: Filtering available models using configurable whitelists and blacklists.
Request Sanitization: Automatically cleaning invalid parameters (like dimensions for non-OpenAI models) via request_sanitizer.py.

Model Filtering Logic

The RotatingClient provides fine-grained control over which models are exposed via the /v1/models endpoint. This is handled by the get_available_models method.

The logic applies in the following order:

Whitelist Check: If a provider has a whitelist defined (WHITELIST_MODELS_<PROVIDER>), any model on that list will always be available, even if it matches a blacklist pattern. This acts as a definitive override.
Blacklist Check: For any model not on the whitelist, the client checks the blacklist (IGNORE_MODELS_<PROVIDER>). If the model matches a blacklist pattern (supports wildcards like *-preview), it is excluded.
Default: If a model is on neither list, it is included.

Request Lifecycle: A Deadline-Driven Approach

The request lifecycle has been designed around a single, authoritative time budget to ensure predictable performance:

Deadline Establishment: The moment acompletion or aembedding is called, a deadline is calculated: time.time() + self.global_timeout. This deadline is the absolute point in time by which the entire operation must complete.
Deadline-Aware Key Selection: The main loop checks this deadline before every key acquisition attempt. If the deadline is exceeded, the request fails immediately.
Deadline-Aware Key Acquisition: The UsageManager itself takes this deadline. It will only wait for a key (if all are busy) until the deadline is reached.
Deadline-Aware Retries: If a transient error occurs (like a 500 or 429), the client calculates the backoff time. If waiting would push the total time past the deadline, the wait is skipped, and the client immediately rotates to the next key.

Streaming Resilience

The _safe_streaming_wrapper is a critical component for stability. It:

Buffers Fragments: Reads raw chunks from the stream and buffers them until a valid JSON object can be parsed. This handles providers that may split JSON tokens across network packets.
Error Interception: Detects if a chunk contains an API error (like a quota limit) instead of content, and raises a specific StreamedAPIError.
Quota Handling: If a specific "quota exceeded" error is detected mid-stream multiple times, it can terminate the stream gracefully to prevent infinite retry loops on oversized inputs.

2.2. `usage_manager.py` - Stateful Concurrency & Usage Management

This class is the stateful core of the library, managing concurrency, usage tracking, cooldowns, and quota resets.

Key Concepts

Async-Native & Lazy-Loaded: Fully asynchronous, using aiofiles for non-blocking file I/O. Usage data is loaded only when needed.
Fine-Grained Locking: Each API key has its own asyncio.Lock and asyncio.Condition. This allows for highly granular control.
Multiple Reset Modes: Supports three reset strategies:
- per_model: Each model has independent usage window with authoritative quota_reset_ts (from provider errors)
- credential: One window per credential with custom duration (e.g., 5 hours, 7 days)
- daily: Legacy daily reset at daily_reset_time_utc
Model Quota Groups: Models can be grouped to share quota limits. When one model in a group hits quota, all receive the same reset timestamp.

Tiered Key Acquisition Strategy

The acquire_key method uses a sophisticated strategy to balance load:

Filtering: Keys currently on cooldown (global or model-specific) are excluded.
Rotation Mode: Determines credential selection strategy:
- Balanced Mode (default): Credentials sorted by usage count - least-used first for even distribution
- Sequential Mode: Credentials sorted by usage count descending - most-used first to maintain sticky behavior until exhausted
Tiering: Valid keys are split into two tiers:
- Tier 1 (Ideal): Keys that are completely idle (0 concurrent requests).
- Tier 2 (Acceptable): Keys that are busy but still under their configured MAX_CONCURRENT_REQUESTS_PER_KEY_<PROVIDER> limit for the requested model. This allows a single key to be used multiple times for the same model, maximizing throughput.
Selection Strategy (configurable via rotation_tolerance):
- Deterministic (tolerance=0.0): Within each tier, keys are sorted by daily usage count and the least-used key is always selected. This provides perfect load balance but predictable patterns.
- Weighted Random (tolerance>0, default): Keys are selected randomly with weights biased toward less-used ones:
  - Formula: weight = (max_usage - credential_usage) + tolerance + 1
  - tolerance=2.0 (recommended): Balanced randomness - credentials within 2 uses of the maximum can still be selected with reasonable probability
  - tolerance=5.0+: High randomness - even heavily-used credentials have significant probability
  - Security Benefit: Unpredictable selection patterns make rate limit detection and fingerprinting harder
  - Load Balance: Lower-usage credentials still preferred, maintaining reasonable distribution
Concurrency Limits: Checks against max_concurrent limits (with priority multipliers applied) to prevent overloading a single key.
Priority Groups: When credential prioritization is enabled, higher-tier credentials (lower priority numbers) are tried first before moving to lower tiers.

Failure Handling & Cooldowns

Escalating Backoff: When a failure occurs, the key gets a temporary cooldown for that specific model. Consecutive failures increase this time (10s -> 30s -> 60s -> 120s).
Key-Level Lockouts: If a key accumulates failures across multiple distinct models (3+), it is assumed to be dead/revoked and placed on a global 5-minute lockout.
Authentication Errors: Immediate 5-minute global lockout.
Quota Exhausted Errors: When a provider returns a quota exhausted error with an authoritative reset timestamp:
- The quota_reset_ts is extracted from the error response (via provider's parse_quota_error() method)
- Applied to the affected model (and all models in its quota group if defined)
- Cooldown preserved even during daily/window resets until the actual quota reset time
- Logs show the exact reset time in local timezone with ISO format

2.3. `batch_manager.py` - Efficient Request Aggregation

The EmbeddingBatcher class optimizes high-throughput embedding workloads.

Mechanism: It uses an asyncio.Queue to collect incoming requests.
Triggers: A batch is dispatched when either:
1. The queue size reaches batch_size (default: 64).
2. A time window (timeout, default: 0.1s) elapses since the first request in the batch.
Efficiency: This reduces dozens of HTTP calls to a single API request, significantly reducing overhead and rate limit usage.

2.4. `background_refresher.py` - Automated Token Maintenance & Provider Jobs

The BackgroundRefresher manages background tasks for the proxy, including OAuth token refresh and provider-specific periodic jobs.

OAuth Token Refresh

Periodic Checks: It runs a background task that wakes up at a configurable interval (default: 600 seconds/10 minutes via OAUTH_REFRESH_INTERVAL).
Proactive Refresh: It iterates through all loaded OAuth credentials and calls their proactively_refresh method to ensure tokens are valid before they are needed.

Provider-Specific Background Jobs

Providers can define their own background jobs that run on independent schedules:

Independent Timers: Each provider's job runs on its own interval, separate from the OAuth refresh cycle.
Configuration: Providers implement get_background_job_config() to define their job settings.
Execution: Providers implement run_background_job() to execute the periodic task.

Provider Job Configuration:

def get_background_job_config(self) -> Optional[Dict[str, Any]]:
    """Return configuration for provider-specific background job."""
    return {
        "interval": 300,      # seconds between runs
        "name": "quota_refresh",  # for logging
        "run_on_start": True,  # whether to run immediately at startup
    }

async def run_background_job(
    self,
    usage_manager: "UsageManager",
    credentials: List[str],
) -> None:
    """Execute the provider's periodic background job."""
    # Provider-specific logic here
    pass

Current Provider Jobs:

Provider	Job Name	Default Interval	Purpose
Antigravity	`quota_baseline_refresh`	300s (5 min)	Fetches quota status from API to update remaining quota estimates

2.6. Credential Management Architecture

The CredentialManager class (credential_manager.py) centralizes the lifecycle of all API credentials. It adheres to a "Local First" philosophy.

2.6.1. Automated Discovery & Preparation

On startup (unless SKIP_OAUTH_INIT_CHECK=true), the manager performs a comprehensive sweep:

System-Wide Scan: Searches for OAuth credential files in standard locations:
- ~/.gemini/ → All *.json files (typically credentials.json)
- ~/.qwen/ → All *.json files (typically oauth_creds.json)
- ~/.iflow/ → All *. json files
Local Import: Valid credentials are copied (not moved) to the project's oauth_creds/ directory with standardized names:
- gemini_cli_oauth_1.json, gemini_cli_oauth_2.json, etc.
- qwen_code_oauth_1.json, qwen_code_oauth_2.json, etc.
- iflow_oauth_1.json, iflow_oauth_2.json, etc.
Intelligent Deduplication:
- The manager inspects each credential file for a _proxy_metadata field containing the user's email or ID
- If this field doesn't exist, it's added during import using provider-specific APIs (e.g., fetching Google account email for Gemini)
- Duplicate accounts (same email/ID) are detected and skipped with a warning log
- Prevents the same account from being added multiple times, even if the files are in different locations
Isolation: The project's credentials in oauth_creds/ are completely isolated from system-wide credentials, preventing cross-contamination

2.6.2. Credential Loading & Stateless Operation

The manager supports loading credentials from two sources, with a clear priority:

Priority 1: Local Files (oauth_creds/ directory)

Standard .json files are loaded first
Naming convention: {provider}_oauth_{number}.json
Example: oauth_creds/gemini_cli_oauth_1.json

Priority 2: Environment Variables (Stateless Deployment)

If no local files are found, the manager checks for provider-specific environment variables
This is the key to "Stateless Deployment" for platforms like Railway, Render, Heroku

Gemini CLI Environment Variables:

GEMINI_CLI_ACCESS_TOKEN
GEMINI_CLI_REFRESH_TOKEN
GEMINI_CLI_E XPIRY_DATE
GEMINI_CLI_EMAIL
GEMINI_CLI_PROJECT_ID (optional)
GEMINI_CLI_CLIENT_ID (optional)

Qwen Code Environment Variables:

QWEN_CODE_ACCESS_TOKEN
QWEN_CODE_REFRESH_TOKEN
QWEN_CODE_EXPIRY_DATE
QWEN_CODE_EMAIL

iFlow Environment Variables:

IFLOW_ACCESS_TOKEN
IFLOW_REFRESH_TOKEN
IFLOW_EXPIRY_DATE
IFLOW_EMAIL
IFLOW_API_KEY

How it works:

If the manager finds (e.g.) GEMINI_CLI_ACCESS_TOKEN, it constructs an in-memory credential object that mimics the file structure
The credential behaves exactly like a file-based credential (automatic refresh, expiry detection, etc.)
No physical files are created or needed on the host system
Perfect for ephemeral containers or read-only filesystems

2.6.3. Credential Tool Integration

The credential_tool.py provides a user-friendly CLI interface to the CredentialManager:

Key Functions:

OAuth Setup: Wraps provider-specific AuthBase classes (GeminiAuthBase, QwenAuthBase, IFlowAuthBase) to handle interactive login flows
Credential Export: Reads local .json files and generates .env format output for stateless deployment
API Key Management: Adds or updates PROVIDER_API_KEY_N entries in the .env file

2.7. Request Sanitizer (`request_sanitizer.py`)

The sanitize_request_payload function ensures requests are compatible with each provider's specific requirements:

Parameter Cleaning Logic:

dimensions Parameter:
- Only supported by OpenAI's text-embedding-3-small and text-embedding-3-large models
- Automatically removed for all other models to prevent 400 Bad Request errors
thinking Parameter (Gemini-specific):
- Format: {"type": "enabled", "budget_tokens": -1}
- Only valid for gemini/gemini-2.5-pro and gemini/gemini-2.5-flash
- Removed for all other models

Provider-Specific Tool Schema Cleaning:

Implemented in individual provider classes (QwenCodeProvider, IFlowProvider):

Recursively removes unsupported properties from tool function schemas:
- strict: OpenAI-specific, causes validation errors on Qwen/iFlow
- additionalProperties: Same issue
Prevents 400 Bad Request errors when using complex tool definitions
Applied automatically before sending requests to the provider

2.8. Error Classification (`error_handler.py`)

The ClassifiedError class wraps all exceptions from litellm and categorizes them for intelligent handling:

Error Types:

class ErrorType(Enum):
    RATE_LIMIT = "rate_limit"           # 429 errors, temporary backoff needed
    AUTHENTICATION = "authentication"    # 401/403, invalid/revoked key
    SERVER_ERROR = "server_error"       # 500/502/503, provider infrastructure issues
    QUOTA = "quota"                      # Daily/monthly quota exceeded
    CONTEXT_LENGTH = "context_length"    # Input too long for model
    CONTENT_FILTER = "content_filter"    # Request blocked by safety filters
    NOT_FOUND = "not_found"              # Model/endpoint doesn't exist
    TIMEOUT = "timeout"                  # Request took too long
    UNKNOWN = "unknown"                  # Unclassified error

Classification Logic:

Status Code Analysis: Primary classification method
- 401/403 → AUTHENTICATION
- 429 → RATE_LIMIT
- 400 with "context_length" or "tokens" → CONTEXT_LENGTH
- 400 with "quota" → QUOTA
- 500/502/503 → SERVER_ERROR
Special Exception Types:
- EmptyResponseError → SERVER_ERROR (status 503, rotatable)
- TransientQuotaError → SERVER_ERROR (status 503, rotatable - bare 429 without retry info)
Message Analysis: Fallback for ambiguous errors
- Searches for keywords like "quota exceeded", "rate limit", "invalid api key"
Provider-Specific Overrides: Some providers use non-standard error formats

Usage in Client:

AUTHENTICATION → Immediate 5-minute global lockout
RATE_LIMIT/QUOTA → Escalating per-model cooldown
SERVER_ERROR → Retry with same key (up to max_retries), then rotate
CONTEXT_LENGTH/CONTENT_FILTER → Immediate failure (user needs to fix request)

2.9. Cooldown Management (`cooldown_manager.py`)

The CooldownManager handles IP or account-level rate limiting that affects all keys for a provider:

Purpose:

Some providers (like NVIDIA NIM) have rate limits tied to account/IP rather than API key
When a 429 error occurs, ALL keys for that provider must be paused

Key Methods:

is_cooling_down(provider: str) -> bool:
- Checks if a provider is currently in a global cooldown period
- Returns True if the current time is still within the cooldown window
start_cooldown(provider: str, duration: int):
- Initiates or extends a cooldown for a provider
- Duration is typically 60-120 seconds for 429 errors
get_cooldown_remaining(provider: str) -> float:
- Returns remaining cooldown time in seconds
- Used for logging and diagnostics

Integration with UsageManager:

When a key fails with RATE_LIMIT error type, the client checks if it's likely an IP-level limit
If so, CooldownManager.start_cooldown() is called for the entire provider
All subsequent acquire_key() calls for that provider will wait until the cooldown expires

2.10. Credential Prioritization System (`client.py` & `usage_manager.py`)

The library now includes an intelligent credential prioritization system that automatically detects credential tiers and ensures optimal credential selection for each request.

Key Concepts:

Provider-Level Priorities: Providers can implement get_credential_priority() to return a priority level (1=highest, 10=lowest) for each credential
Model-Level Requirements: Providers can implement get_model_tier_requirement() to specify minimum priority required for specific models
Automatic Filtering: The client automatically filters out incompatible credentials before making requests
Priority-Aware Selection: The UsageManager prioritizes higher-tier credentials (lower numbers) within the same priority group

Implementation Example (Gemini CLI):

def get_credential_priority(self, credential: str) -> Optional[int]:
    """Returns priority based on Gemini tier."""
    tier = self.project_tier_cache.get(credential)
    if not tier:
        return None  # Not yet discovered
    
    # Paid tiers get highest priority
    if tier not in ['free-tier', 'legacy-tier', 'unknown']:
        return 1
    
    # Free tier gets lower priority
    if tier == 'free-tier':
        return 2
    
    return 10

def get_model_tier_requirement(self, model: str) -> Optional[int]:
    """Returns minimum priority required for model."""
    if model.startswith("gemini-3-"):
        return 1  # Only paid tier (priority 1) credentials
    
    return None  # All other models have no restrictions

Provider Support:

The following providers implement credential prioritization:

Gemini CLI: Paid tier (priority 1), Free tier (priority 2), Legacy/Unknown (priority 10). Gemini 3 models require paid tier.
Antigravity: Same priority system as Gemini CLI. No model-tier restrictions (all models work on all tiers). Paid tier resets every 5 hours, free tier resets weekly.

Usage Manager Integration:

The acquire_key() method has been enhanced to:

Group credentials by priority level
Try highest priority group first (priority 1, then 2, etc.)
Within each group, use existing tier1/tier2 logic (idle keys first, then busy keys)
Load balance within priority groups by usage count
Only move to next priority if all higher-priority credentials are exhausted

Benefits:

Ensures paid-tier credentials are always used for premium models
Prevents failed requests due to tier restrictions
Optimal cost distribution (free tier used when possible, paid when required)
Graceful fallback if primary credentials are unavailable

2.11. Provider Cache System (`providers/provider_cache.py`)

A modular, shared caching system for providers to persist conversation state across requests.

Architecture:

Dual-TTL Design: Short-lived memory cache (default: 1 hour) + longer-lived disk persistence (default: 24 hours)
Background Persistence: Batched disk writes every 60 seconds (configurable)
Automatic Cleanup: Background task removes expired entries from memory cache

2.15. Antigravity Quota Tracker (`providers/utilities/antigravity_quota_tracker.py`)

A mixin class providing quota tracking functionality for the Antigravity provider. This enables accurate remaining quota estimation based on API-fetched baselines and local request counting.

Core Concepts

Quota Baseline Tracking:

Periodically fetches quota status from the Antigravity fetchAvailableModels API
Stores the remaining fraction as a baseline in UsageManager
Tracks requests since baseline to estimate current remaining quota
Syncs local request count with API's authoritative values

Quota Cost Constants: Based on empirical testing (see docs/ANTIGRAVITY_QUOTA_REPORT.md), quota costs are known per model and tier:

Tier	Model Group	Cost per Request	Requests per 100%
standard-tier	Claude/GPT-OSS	0.40%	250
standard-tier	Gemini 3 Pro	0.25%	400
standard-tier	Gemini 2.5 Flash	0.0333%	~3000
free-tier	Claude/GPT-OSS	1.333%	75
free-tier	Gemini 3 Pro	0.40%	250

Model Name Mappings: Some user-facing model names don't exist directly in the API response:

claude-opus-4-5 → claude-opus-4-5-thinking (Opus only exists as thinking variant)
gemini-3-pro-preview → gemini-3-pro-high (preview maps to high by default)

Key Methods

fetch_quota_from_api(credential_path): Fetches current quota status from the Antigravity API. Returns remaining fraction and reset times for all models.

estimate_remaining_quota(credential_path, model, model_data, tier): Estimates remaining quota based on baseline + request tracking. Returns confidence level (high/medium/low) based on baseline age.

refresh_active_quota_baselines(credentials, usage_data): Only refreshes baselines for credentials that have been used recently (within the refresh interval).

discover_quota_costs(credential_path, models_to_test): Manual utility to discover quota costs by making test requests and measuring before/after quota. Saves learned costs to cache/antigravity/learned_quota_costs.json.

Integration with Background Jobs

The Antigravity provider defines a background job for quota baseline refresh:

def get_background_job_config(self) -> Optional[Dict[str, Any]]:
    return {
        "interval": 300,  # 5 minutes (configurable via ANTIGRAVITY_QUOTA_REFRESH_INTERVAL)
        "name": "quota_baseline_refresh",
        "run_on_start": True,
    }

This job:

Identifies credentials used since the last refresh
Fetches current quota from the API for those credentials
Updates baselines in UsageManager for accurate estimation

Data Storage

Quota baselines are stored in UsageManager's per-model data:

{
  "credential_path": {
    "models": {
      "antigravity/claude-sonnet-4-5": {
        "request_count": 15,
        "baseline_remaining_fraction": 0.94,
        "baseline_fetched_at": 1734567890.0,
        "requests_at_baseline": 15,
        "quota_max_requests": 250,
        "quota_display": "15/250"
      }
    }
  }
}

2.16. TransientQuotaError (`error_handler.py`)

A new error type for handling bare 429 responses without retry timing information.

When Raised:

Provider returns HTTP 429 status code
Response doesn't contain retry timing info (no quotaResetTimeStamp or retryDelay)
After internal retry attempts are exhausted

Behavior:

Classified as server_error (status 503) rather than quota exhaustion
Causes credential rotation to try the next credential
Does NOT trigger long-term quota cooldowns

Implementation in Antigravity:

# Non-streaming and streaming both retry bare 429s
for attempt in range(EMPTY_RESPONSE_MAX_ATTEMPTS):
    try:
        result = await self._handle_request(...)
    except httpx.HTTPStatusError as e:
        if e.response.status_code == 429:
            quota_info = self.parse_quota_error(e)
            if quota_info is None:
                # Bare 429 - retry like empty response
                if attempt < EMPTY_RESPONSE_MAX_ATTEMPTS - 1:
                    await asyncio.sleep(EMPTY_RESPONSE_RETRY_DELAY)
                    continue
                else:
                    raise TransientQuotaError(provider, model, message)
            # Has retry info - real quota exhaustion
            raise

Rationale: Some 429 responses are transient rate limits rather than true quota exhaustion. These occur when the API is temporarily overloaded but the credential still has quota available. Retrying internally before rotating credentials provides better resilience.

3.5. Antigravity (`antigravity_provider.py`)

The most sophisticated provider implementation, supporting Google's internal Antigravity API for Gemini 3 and Claude models (including Claude Opus 4.5, Anthropic's most powerful model).

Architecture

Unified Streaming/Non-Streaming: Single code path handles both response types with optimal transformations
Thought Signature Caching: Server-side caching of encrypted signatures for multi-turn Gemini 3 conversations
Model-Specific Logic: Automatic configuration based on model type (Gemini 3, Claude Sonnet, Claude Opus)
Credential Prioritization: Automatic tier detection with paid credentials prioritized over free (paid tier resets every 5 hours, free tier resets weekly)
Sequential Rotation Mode: Default rotation mode is sequential (use credentials until exhausted) to maximize thought signature cache hits
Per-Model Quota Tracking: Each model tracks independent usage windows with authoritative reset timestamps from quota errors
Quota Groups: Models that share quota limits are grouped together (Claude/GPT-OSS share quota, Gemini 3 Pro variants share quota, Gemini 2.5 Flash variants share quota)
Priority Multipliers: Paid tier credentials get higher concurrency limits (Priority 1: 5x, Priority 2: 3x, Priority 3+: 2x in sequential mode)
Quota Baseline Tracking: Background job fetches quota status from API to provide accurate remaining quota estimates
TransientQuotaError Handling: Bare 429 responses (without retry info) are retried internally before credential rotation

Model Support

Gemini 3 Pro:

Uses thinkingLevel parameter (string: "low" or "high")
Tool Hallucination Prevention:
- Automatic system instruction injection explaining custom tool schema rules
- Parameter signature injection into tool descriptions (e.g., "STRICT PARAMETERS: files (ARRAY_OF_OBJECTS[path: string REQUIRED, ...])")
- Namespace prefix for tool names (gemini3_ prefix) to avoid training data conflicts
- Malformed JSON auto-correction (handles extra trailing braces)
ThoughtSignature Management:
- Caching signatures from responses for reuse in follow-up messages
- Automatic injection into functionCalls for multi-turn conversations
- Fallback to bypass value if signature unavailable
Parallel Tool Usage Instruction: Configurable instruction injection to encourage parallel tool calls (disabled by default for Gemini 3)

Gemini 2.5 Flash:

Uses -thinking variant when reasoning_effort is provided
Shares quota with gemini-2.5-flash-thinking and gemini-2.5-flash-lite variants
Parallel tool usage instruction configurable

Gemini 2.5 Flash Lite:

Configurable thinking budget, no name change required
Shares quota with Flash variants

Claude Opus 4.5:

Anthropic's most powerful model, now available via Antigravity proxy
Always uses thinking variant - claude-opus-4-5-thinking is the only available variant (non-thinking version doesn't exist)
Uses thinkingBudget parameter for extended thinking control (-1 for auto, 0 to disable, or specific token count)
Full support for tool use with schema cleaning
Same thinking preservation and sanitization features as Sonnet
Increased default max output tokens to 64000 to accommodate thinking output

Claude Sonnet 4.5:

Proxied through Antigravity API
Supports both thinking and non-thinking modes:
- With reasoning_effort: Uses claude-sonnet-4-5-thinking variant with thinkingBudget
- Without reasoning_effort: Uses standard claude-sonnet-4-5 variant
Thinking Preservation: Caches thinking content using composite keys (tool_call_id + text_hash)
Schema Cleaning: Removes unsupported properties ($schema, additionalProperties, const → enum)
Parallel Tool Usage Instruction: Automatic instruction injection to encourage parallel tool calls (enabled by default for Claude)

GPT-OSS 120B Medium:

OpenAI-compatible model available via Antigravity
Shares quota with Claude models (Claude/GPT-OSS quota group)

Base URL Fallback

Automatic fallback chain for resilience:

daily-cloudcode-pa.sandbox.googleapis.com (primary sandbox)
autopush-cloudcode-pa.sandbox.googleapis.com (fallback sandbox)
cloudcode-pa.googleapis.com (production fallback)

Message Transformation

OpenAI → Gemini Format:

System messages → systemInstruction with parts array
Multi-part content (text + images) → inlineData format
Tool calls → functionCall with args and id
Tool responses → functionResponse with name and response
ThoughtSignatures preserved/injected as needed

Tool Response Grouping:

Converts linear format (call, response, call, response) to grouped format
Groups all function calls in one model message
Groups all responses in one user message
Required for Antigravity API compatibility

Configuration (Environment Variables)

# Cache control
ANTIGRAVITY_SIGNATURE_CACHE_TTL=3600  # Memory cache TTL
ANTIGRAVITY_SIGNATURE_DISK_TTL=86400  # Disk cache TTL
ANTIGRAVITY_ENABLE_SIGNATURE_CACHE=true

# Feature flags
ANTIGRAVITY_PRESERVE_THOUGHT_SIGNATURES=true  # Include signatures in client responses
ANTIGRAVITY_ENABLE_DYNAMIC_MODELS=false  # Use API model discovery
ANTIGRAVITY_GEMINI3_TOOL_FIX=true  # Enable Gemini 3 hallucination prevention
ANTIGRAVITY_CLAUDE_THINKING_SANITIZATION=true  # Enable Claude thinking mode auto-correction

# Gemini 3 tool fix customization
ANTIGRAVITY_GEMINI3_TOOL_PREFIX="gemini3_"  # Namespace prefix
ANTIGRAVITY_GEMINI3_DESCRIPTION_PROMPT="\n\nSTRICT PARAMETERS: {params}."
ANTIGRAVITY_GEMINI3_SYSTEM_INSTRUCTION="..."  # Full system prompt

# Parallel tool usage instruction
ANTIGRAVITY_PARALLEL_TOOL_INSTRUCTION_CLAUDE=true  # Inject parallel tool instruction for Claude (default: true)
ANTIGRAVITY_PARALLEL_TOOL_INSTRUCTION_GEMINI3=false  # Inject parallel tool instruction for Gemini 3 (default: false)
ANTIGRAVITY_PARALLEL_TOOL_INSTRUCTION="..."  # Custom instruction text

# Quota tracking
ANTIGRAVITY_QUOTA_REFRESH_INTERVAL=300  # Background quota refresh interval in seconds (default: 300 = 5 min)

Claude Extended Thinking Sanitization

The provider now includes robust automatic sanitization for Claude's extended thinking mode, handling all common error scenarios with conversation history.

Problem: Claude's extended thinking API requires strict consistency in thinking blocks:

If thinking is enabled, the final assistant turn must start with a thinking block
If thinking is disabled, no thinking blocks can be present in the final turn
Tool use loops are part of a single "assistant turn"
You cannot toggle thinking mode mid-turn (this is invalid per Claude API)

Scenarios Handled:

Scenario	Action
Tool loop WITH thinking + thinking enabled	Preserve thinking, continue normally
Tool loop WITHOUT thinking + thinking enabled	Inject synthetic closure to start fresh turn with thinking
Thinking disabled	Strip all thinking blocks
Normal conversation (no tool loop)	Strip old thinking, new response adds thinking naturally
Function call ID mismatch	Three-tier recovery: ID match → name match → fallback
Missing tool responses	Automatic placeholder injection
Compacted/cached conversations	Recover thinking from cache post-transformation

Key Implementation Details:

The _sanitize_thinking_for_claude() method now:

Operates on Gemini-format messages (parts[] with "thought": true markers)
Detects tool results as user messages with functionResponse parts
Uses _analyze_turn_state() to classify conversation state on Gemini format
Recovers thinking from cache when client strips reasoning_content
When enabling thinking in a tool loop started without thinking:
- Injects synthetic assistant message to close the previous turn
- Allows Claude to start fresh turn with thinking capability

Function Call Response Grouping:

The enhanced pairing system ensures conversation history integrity:

Problem: Client/proxy may mutate response IDs or lose responses during context processing

Solution:
1. Try direct ID match (tool_call_id == response.id)
2. If no match, try function name match (tool.name == response.name)
3. If still no match, use order-based fallback (nth tool → nth response)
4. Repair "unknown_function" responses with correct names
5. Create placeholders for completely missing responses

Configuration:

ANTIGRAVITY_CLAUDE_THINKING_SANITIZATION=true  # Enable/disable auto-correction (default: true)

Note: These fixes ensure Claude thinking mode works seamlessly with tool use, model switching, context compression, and cached conversations. No manual intervention required.

File Logging

Optional transaction logging for debugging:

Enabled via enable_request_logging parameter
Creates logs/antigravity_logs/TIMESTAMP_MODEL_UUID/ directory per request
Logs: request_payload.json, response_stream.log, final_response.json, error.log

Atomic Disk Writes: Uses temp-file-and-move pattern to prevent corruption

Key Methods:

store(key, value): Synchronously queues value for storage (schedules async write)
retrieve(key): Synchronously retrieves from memory, optionally schedules disk fallback
store_async(key, value): Awaitable storage for guaranteed persistence
retrieve_async(key): Awaitable retrieval with disk fallback

Use Cases:

Gemini 3 ThoughtSignatures: Caching tool call signatures for multi-turn conversations
Claude Thinking: Preserving thinking content for consistency across conversation turns
Any Transient State: Generic key-value storage for provider-specific needs

Configuration (Environment Variables):

# Cache control (prefix can be customized per cache instance)
PROVIDER_CACHE_ENABLE=true
PROVIDER_CACHE_WRITE_INTERVAL=60  # seconds between disk writes
PROVIDER_CACHE_CLEANUP_INTERVAL=1800  # 30 min between cleanups

# Gemini 3 specific
GEMINI_CLI_SIGNATURE_CACHE_ENABLE=true
GEMINI_CLI_SIGNATURE_CACHE_TTL=3600  # 1 hour memory TTL
GEMINI_CLI_SIGNATURE_DISK_TTL=86400  # 24 hours disk TTL

File Structure:

cache/
├── gemini_cli/
│   └── gemini3_signatures.json
└── antigravity/
    ├── gemini3_signatures.json
    └── claude_thinking.json

2.13. Sequential Rotation & Per-Model Quota Tracking

A comprehensive credential rotation and quota management system introduced in PR #31.

Rotation Modes

Two rotation strategies are available per provider:

Balanced Mode (Default):

Distributes load evenly across all credentials
Least-used credentials selected first
Best for providers with per-minute rate limits
Prevents any single credential from being overused

Sequential Mode:

Uses one credential until it's exhausted (429 quota error)
Switches to next credential only after current one fails
Most-used credentials selected first (sticky behavior)
Best for providers with daily/weekly quotas
Maximizes cache hit rates (e.g., Antigravity thought signatures)
Default for Antigravity provider

Configuration:

# Set per provider
ROTATION_MODE_GEMINI=sequential
ROTATION_MODE_OPENAI=balanced
ROTATION_MODE_ANTIGRAVITY=balanced  # Override default

Per-Model Quota Tracking

Instead of tracking usage at the credential level, the system now supports granular per-model tracking:

Data Structure (when mode="per_model"):

{
  "credential_id": {
    "models": {
      "gemini-2.5-pro": {
        "window_start_ts": 1733678400.0,
        "quota_reset_ts": 1733696400.0,
        "success_count": 15,
        "prompt_tokens": 5000,
        "completion_tokens": 1000,
        "approx_cost": 0.05,
        "window_started": "2025-12-08 14:00:00 +0100",
        "quota_resets": "2025-12-08 19:00:00 +0100"
      }
    },
    "global": {...},
    "model_cooldowns": {...}
  }
}

Key Features:

Each model tracks its own usage window independently
window_start_ts: When the current quota period started
quota_reset_ts: Authoritative reset time from provider error response
Human-readable timestamps added for debugging
Supports custom window durations (5h, 7d, etc.)

Provider-Specific Quota Parsing

Providers can implement parse_quota_error() to extract precise reset times from error responses:

@staticmethod
def parse_quota_error(error, error_body) -> Optional[Dict]:
    """Extract quota reset timestamp from provider error.
    
    Returns:
        {
            'quota_reset_timestamp': 1733696400.0,  # Unix timestamp
            'retry_after': 18000  # Seconds until reset
        }
    """

Google RPC Format (Antigravity, Gemini CLI):

Parses RetryInfo and ErrorInfo from error details
Handles duration strings: "143h4m52.73s" or "515092.73s"
Extracts quotaResetTimeStamp and converts to Unix timestamp
Falls back to quotaResetDelay if timestamp not available

Example Error Response:

{
  "error": {
    "code": 429,
    "message": "Quota exceeded",
    "details": [{
      "@type": "type.googleapis.com/google.rpc.RetryInfo",
      "retryDelay": "143h4m52.73s"
    }, {
      "@type": "type.googleapis.com/google.rpc.ErrorInfo",
      "metadata": {
        "quotaResetTimeStamp": "2025-12-08T19:00:00Z"
      }
    }]
  }
}

Model Quota Groups

Models that share the same quota limits can be grouped:

Configuration:

# Models in a group share quota/cooldown timing
QUOTA_GROUPS_ANTIGRAVITY_CLAUDE="claude-sonnet-4-5,claude-sonnet-4-5-thinking,claude-opus-4-5,claude-opus-4-5-thinking,gpt-oss-120b-medium"
QUOTA_GROUPS_ANTIGRAVITY_GEMINI_3_PRO="gemini-3-pro-high,gemini-3-pro-low,gemini-3-pro-preview"
QUOTA_GROUPS_ANTIGRAVITY_GEMINI_2_5_FLASH="gemini-2.5-flash,gemini-2.5-flash-thinking,gemini-2.5-flash-lite"

# To disable a default group:
QUOTA_GROUPS_ANTIGRAVITY_CLAUDE=""

Default Quota Groups (Antigravity):

Group Name	Models	Shared Quota
`claude`	claude-sonnet-4-5, claude-sonnet-4-5-thinking, claude-opus-4-5, claude-opus-4-5-thinking, gpt-oss-120b-medium	Yes (Claude and GPT-OSS share quota)
`gemini-3-pro`	gemini-3-pro-high, gemini-3-pro-low, gemini-3-pro-preview	Yes
`gemini-2.5-flash`	gemini-2.5-flash, gemini-2.5-flash-thinking, gemini-2.5-flash-lite	Yes

Behavior:

When one model hits quota, all models in the group receive the same quota_reset_ts
Group resets only when ALL models' quotas have reset
Preserves unexpired cooldowns during other resets

Provider Implementation:

class AntigravityProvider(ProviderInterface):
    model_quota_groups = {
        # Claude and GPT-OSS share the same quota pool
        "claude": [
            "claude-sonnet-4-5",
            "claude-sonnet-4-5-thinking",
            "claude-opus-4-5",
            "claude-opus-4-5-thinking",
            "gpt-oss-120b-medium",
        ],
        # Gemini 3 Pro variants share quota
        "gemini-3-pro": [
            "gemini-3-pro-high",
            "gemini-3-pro-low",
            "gemini-3-pro-preview",
        ],
        # Gemini 2.5 Flash variants share quota
        "gemini-2.5-flash": [
            "gemini-2.5-flash",
            "gemini-2.5-flash-thinking",
            "gemini-2.5-flash-lite",
        ],
    }

Priority-Based Concurrency Multipliers

Credentials can be assigned to priority tiers with configurable concurrency limits:

Configuration:

# Universal multipliers (all modes)
CONCURRENCY_MULTIPLIER_ANTIGRAVITY_PRIORITY_1=10
CONCURRENCY_MULTIPLIER_ANTIGRAVITY_PRIORITY_2=3

# Mode-specific overrides
CONCURRENCY_MULTIPLIER_ANTIGRAVITY_PRIORITY_2_BALANCED=1  # Lower in balanced mode

How it works:

effective_concurrent_limit = MAX_CONCURRENT_REQUESTS_PER_KEY * tier_multiplier

Provider Defaults (Antigravity):

Priority 1 (paid ultra): 5x multiplier
Priority 2 (standard paid): 3x multiplier
Priority 3+ (free): 2x (sequential mode) or 1x (balanced mode)

Benefits:

Paid credentials handle more load without manual configuration
Different concurrency for different rotation modes
Automatic tier detection based on credential properties

Reset Window Configuration

Providers can specify custom reset windows per priority tier:

class AntigravityProvider(ProviderInterface):
    usage_reset_configs = {
        frozenset([1, 2]): UsageResetConfigDef(
            mode="per_model",
            window_hours=5,  # 5-hour rolling window for paid tiers
            field_name="5h_window"
        ),
        frozenset([3, 4, 5]): UsageResetConfigDef(
            mode="per_model",
            window_hours=168,  # 7-day window for free tier
            field_name="7d_window"
        )
    }

Supported Modes:

per_model: Independent window per model with authoritative reset times
credential: Single window per credential (legacy)
daily: Daily reset at configured UTC hour (legacy)

Usage Flow

Request arrives for model X with credential Y
Check rotation mode: Sequential or balanced?
Select credential:
- Filter by priority tier requirements
- Apply concurrency multiplier for effective limit
- Sort by rotation mode strategy
Check quota:
- Load model's usage data
- Check if within window (window_start_ts to quota_reset_ts)
- Check model quota groups for combined usage
Execute request
On success: Increment model usage count
On quota error:
- Parse error for quota_reset_ts
- Apply to model (and quota group)
- Credential remains on cooldown until reset time
On window expiration:
- Archive model data to global stats
- Start fresh window with new window_start_ts
- Preserve unexpired quota cooldowns

2.12. Google OAuth Base (`providers/google_oauth_base.py`)

A refactored, reusable OAuth2 base class that eliminates code duplication across Google-based providers.

Refactoring Benefits:

Single Source of Truth: All OAuth logic centralized in one class
Easy Provider Addition: New providers only need to override constants
Consistent Behavior: Token refresh, expiry handling, and validation work identically across providers
Maintainability: OAuth bugs fixed once apply to all inheriting providers

Provider Implementation:

class AntigravityAuthBase(GoogleOAuthBase):
    # Required overrides
    CLIENT_ID = "antigravity-client-id"
    CLIENT_SECRET = "antigravity-secret"
    OAUTH_SCOPES = [
        "https://www.googleapis.com/auth/cloud-platform",
        "https://www.googleapis.com/auth/cclog",  # Antigravity-specific
        "https://www.googleapis.com/auth/experimentsandconfigs",
    ]
    ENV_PREFIX = "ANTIGRAVITY"  # Used for env var loading
    
    # Optional overrides (defaults provided)
    CALLBACK_PORT = 51121
    CALLBACK_PATH = "/oauthcallback"

Inherited Features:

Automatic token refresh with exponential backoff
Invalid grant re-authentication flow
Stateless deployment support (env var loading)
Atomic credential file writes
Headless environment detection
Sequential refresh queue processing

OAuth Callback Port Configuration

Each OAuth provider uses a local callback server during authentication. The callback port can be customized via environment variables to avoid conflicts with other services.

Default Ports:

Provider	Default Port	Environment Variable
Gemini CLI	8085	`GEMINI_CLI_OAUTH_PORT`
Antigravity	51121	`ANTIGRAVITY_OAUTH_PORT`
iFlow	11451	`IFLOW_OAUTH_PORT`

Configuration Methods:

Via TUI Settings Menu:
- Main Menu → 4. View Provider & Advanced Settings → 1. Launch Settings Tool
- Select the provider (Gemini CLI, Antigravity, or iFlow)
- Modify the *_OAUTH_PORT setting
- Use "Reset to Default" to restore the original port

Via .env file:

# Custom OAuth callback ports (optional)
GEMINI_CLI_OAUTH_PORT=8085
ANTIGRAVITY_OAUTH_PORT=51121
IFLOW_OAUTH_PORT=11451

When to Change Ports:

If the default port conflicts with another service on your system
If running multiple proxy instances on the same machine
If firewall rules require specific port ranges

Note: Port changes take effect on the next OAuth authentication attempt. Existing tokens are not affected.

2.14. HTTP Timeout Configuration (`timeout_config.py`)

Centralized timeout configuration for all HTTP requests to LLM providers.

Purpose

The TimeoutConfig class provides fine-grained control over HTTP timeouts for streaming and non-streaming LLM requests. This addresses the common issue of proxy hangs when upstream providers stall during connection establishment or response generation.

Timeout Types Explained

Timeout	Description
connect	Maximum time to establish a TCP/TLS connection to the upstream server
read	Maximum time to wait between receiving data chunks (resets on each chunk for streaming)
write	Maximum time to wait while sending the request body
pool	Maximum time to wait for a connection from the connection pool

Default Values

Setting	Streaming	Non-Streaming	Rationale
connect	30s	30s	Fast fail if server is unreachable
read	180s (3 min)	600s (10 min)	Streaming expects periodic chunks; non-streaming may wait for full generation
write	30s	30s	Request bodies are typically small
pool	60s	60s	Reasonable wait for connection pool

Environment Variable Overrides

All timeout values can be customized via environment variables:

# Connection establishment timeout (seconds)
TIMEOUT_CONNECT=30

# Request body send timeout (seconds)
TIMEOUT_WRITE=30

# Connection pool acquisition timeout (seconds)
TIMEOUT_POOL=60

# Read timeout between chunks for streaming requests (seconds)
# If no data arrives for this duration, the connection is considered stalled
TIMEOUT_READ_STREAMING=180

# Read timeout for non-streaming responses (seconds)
# Longer to accommodate models that take time to generate full responses
TIMEOUT_READ_NON_STREAMING=600

Streaming vs Non-Streaming Behavior

Streaming Requests (TimeoutConfig.streaming()):

Uses shorter read timeout (default 3 minutes)
Timer resets every time a chunk arrives
If no data for 3 minutes → connection considered dead → failover to next credential
Appropriate for chat completions where tokens should arrive periodically

Non-Streaming Requests (TimeoutConfig.non_streaming()):

Uses longer read timeout (default 10 minutes)
Server may take significant time to generate the complete response before sending anything
Complex reasoning tasks or large outputs may legitimately take several minutes
Only used by Antigravity provider's _handle_non_streaming() method

Provider Usage

The following providers use TimeoutConfig:

Provider	Method	Timeout Type
`antigravity_provider.py`	`_handle_non_streaming()`	`non_streaming()`
`antigravity_provider.py`	`_handle_streaming()`	`streaming()`
`gemini_cli_provider.py`	`acompletion()`	`streaming()`
`iflow_provider.py`	`acompletion()`	`streaming()`
`qwen_code_provider.py`	`acompletion()`	`streaming()`

Note: iFlow, Qwen Code, and Gemini CLI providers always use streaming internally (even for non-streaming requests), aggregating chunks into a complete response. Only Antigravity has a true non-streaming path.

Tuning Recommendations

Use Case	Recommendation
Long thinking tasks	Increase `TIMEOUT_READ_STREAMING` to 300-360s
Unstable network	Increase `TIMEOUT_CONNECT` to 60s
High concurrency	Increase `TIMEOUT_POOL` if seeing pool exhaustion
Large context/output	Increase `TIMEOUT_READ_NON_STREAMING` to 900s+

Example Configuration

# For environments with complex reasoning tasks
TIMEOUT_READ_STREAMING=300
TIMEOUT_READ_NON_STREAMING=900

# For unstable network conditions
TIMEOUT_CONNECT=60
TIMEOUT_POOL=120

3. Provider Specific Implementations

The library handles provider idiosyncrasies through specialized "Provider" classes in src/rotator_library/providers/.

3.1. Gemini CLI (`gemini_cli_provider.py`)

The GeminiCliProvider is the most complex implementation, mimicking the Google Cloud Code extension.

New in PR #31:

Quota Parsing: Implements parse_quota_error() using Google RPC format parser
Tier Configuration: Defines tier_priorities and usage_reset_configs for automatic priority resolution
Balanced Rotation: Defaults to balanced mode (unlike Antigravity which uses sequential)
Priority Multipliers: Same as Antigravity (P1: 5x, P2: 3x, others: 1x)

Authentication (`gemini_auth_base.py`)

Device Flow: Uses a standard OAuth 2.0 flow. The credential_tool spins up a local web server (default: localhost:8085, configurable via GEMINI_CLI_OAUTH_PORT) to capture the callback from Google's auth page.
Token Lifecycle: * Proactive Refresh: Tokens are refreshed 5 minutes before expiry. * Atomic Writes: Credential files are updated using a temp-file-and-move strategy to prevent corruption during writes. * Revocation Handling: If a 400 or 401 occurs during refresh, the token is marked as revoked, preventing infinite retry loops.

Project ID Discovery (Zero-Config)

The provider employs a sophisticated, cached discovery mechanism to find a valid Google Cloud Project ID:

Configuration: Checks GEMINI_CLI_PROJECT_ID first.
Code Assist API: Tries CODE_ASSIST_ENDPOINT:loadCodeAssist. This returns the project associated with the Cloud Code extension.
Onboarding Flow: If step 2 fails, it triggers the onboardUser endpoint. This initiates a Long-Running Operation (LRO) that automatically provisions a free-tier Google Cloud Project for the user. The proxy polls this operation for up to 5 minutes until completion.
Resource Manager: As a final fallback, it lists all active projects via the Cloud Resource Manager API and selects the first one.

Rate Limit Handling

Internal Endpoints: Uses https://cloudcode-pa.googleapis.com/v1internal, which typically has higher quotas than the public API.
Smart Fallback: If gemini-2.5-pro hits a rate limit (429), the provider transparently retries the request using gemini-2.5-pro-preview-06-05. This fallback chain is configurable in code.

3.2. Qwen Code (`qwen_code_provider.py`)

Dual Auth: Supports both standard API keys (direct) and OAuth (via QwenAuthBase).
Device Flow: Implements the OAuth Device Authorization Grant (RFC 8628). It displays a code to the user and polls the token endpoint until the user authorizes the device in their browser.
Dummy Tool Injection: To work around a Qwen API bug where streams hang if tools is empty but tool_choice logic is present, the provider injects a benign do_not_call_me tool.
Schema Cleaning: Recursively removes strict and additionalProperties from tool schemas, as Qwen's validation is stricter than OpenAI's.
Reasoning Parsing: Detects <think> tags in the raw stream and redirects their content to a separate reasoning_content field in the delta, mimicking the OpenAI o1 format.

3.3. iFlow (`iflow_provider.py`)

Hybrid Auth: Uses a custom OAuth flow (Authorization Code) to obtain an access_token. However, the actual API calls use a separate apiKey that is retrieved from the user's profile (/api/oauth/getUserInfo) using the access token.
Callback Server: The auth flow spins up a local server (default: port 11451, configurable via IFLOW_OAUTH_PORT) to capture the redirect.
Token Management: Automatically refreshes the OAuth token and re-fetches the API key if needed.
Schema Cleaning: Similar to Qwen, it aggressively sanitizes tool schemas to prevent 400 errors.
Dedicated Logging: Implements _IFlowFileLogger to capture raw chunks for debugging proprietary API behaviors.

3.4. Google Gemini (`gemini_provider.py`)

Thinking Parameter: Automatically handles the thinking parameter transformation required for Gemini 2.5 models (thinking -> gemini-2.5-pro reasoning parameter).
Safety Settings: Ensures default safety settings (blocking nothing) are applied if not provided, preventing over-sensitive refusals.

4. Logging & Debugging

`detailed_logger.py`

To facilitate robust debugging, the proxy includes a comprehensive transaction logging system.

Unique IDs: Every request generates a UUID.
Directory Structure: Logs are stored in logs/detailed_logs/YYYYMMDD_HHMMSS_{uuid}/.
Artifacts:
- request.json: The exact payload sent to the proxy.
- final_response.json: The complete reassembled response.
- streaming_chunks.jsonl: A line-by-line log of every SSE chunk received from the provider.
- metadata.json: Performance metrics (duration, token usage, model used).

This level of detail allows developers to trace exactly why a request failed or why a specific key was rotated.

5. Runtime Resilience

The proxy is engineered to maintain high availability even in the face of runtime filesystem disruptions. This "Runtime Resilience" capability ensures that the service continues to process API requests even if data files or directories are deleted while the application is running.

5.1. Centralized Resilient I/O (`resilient_io.py`)

All file operations are centralized in a single utility module that provides consistent error handling, graceful degradation, and automatic retry with shutdown flush:

`BufferedWriteRegistry` (Singleton)

Global registry for buffered writes with periodic retry and shutdown flush. Ensures critical data is saved even if disk writes fail temporarily:

Per-file buffering: Each file path has its own pending write (latest data always wins)
Periodic retries: Background thread retries failed writes every 30 seconds
Shutdown flush: atexit hook ensures final write attempt on app exit (Ctrl+C)
Thread-safe: Safe for concurrent access from multiple threads

# Get the singleton instance
registry = BufferedWriteRegistry.get_instance()

# Check pending writes (for monitoring)
pending_count = registry.get_pending_count()
pending_files = registry.get_pending_paths()

# Manual flush (optional - atexit handles this automatically)
results = registry.flush_all()  # Returns {path: success_bool}

# Manual shutdown (if needed before atexit)
results = registry.shutdown()

`ResilientStateWriter`

For stateful files that must persist (usage stats):

Memory-first: Always updates in-memory state before attempting disk write
Atomic writes: Uses tempfile + move pattern to prevent corruption
Automatic retry with backoff: If disk fails, waits retry_interval seconds before trying again
Shutdown integration: Registers with BufferedWriteRegistry on failure for final flush
Health monitoring: Exposes is_healthy property for monitoring

writer = ResilientStateWriter("data.json", logger, retry_interval=30.0)
writer.write({"key": "value"})  # Always succeeds (memory update)
if not writer.is_healthy:
    logger.warning("Disk writes failing, data in memory only")
# On next write() call after retry_interval, disk write is attempted again
# On app exit (Ctrl+C), BufferedWriteRegistry attempts final save

`safe_write_json()`

For JSON writes with configurable options (credentials, cache):

Parameter	Default	Description
`path`	required	File path to write to
`data`	required	JSON-serializable data
`logger`	required	Logger for warnings
`atomic`	`True`	Use atomic write pattern (tempfile + move)
`indent`	`2`	JSON indentation level
`ensure_ascii`	`True`	Escape non-ASCII characters
`secure_permissions`	`False`	Set file permissions to 0o600
`buffer_on_failure`	`False`	Register with BufferedWriteRegistry on failure

When buffer_on_failure=True:

Failed writes are registered with BufferedWriteRegistry
Data is retried every 30 seconds in background
On app exit, final write attempt is made automatically
Success unregisters the pending write

# For critical data (auth tokens) - use buffer_on_failure
safe_write_json(path, creds, logger, secure_permissions=True, buffer_on_failure=True)

# For non-critical data (logs) - no buffering needed
safe_write_json(path, data, logger)

`safe_log_write()`

For log files where occasional loss is acceptable:

Fire-and-forget pattern
Creates parent directories if needed
Returns True/False, never raises
No buffering - logs are dropped on failure

`safe_mkdir()`

For directory creation with error handling.

5.2. Resilience Hierarchy

The system follows a strict hierarchy of survival:

Core API Handling (Level 1): The Python runtime keeps all necessary code in memory. Deleting source code files while the proxy is running will not crash active requests.
Credential Management (Level 2): OAuth tokens are cached in memory first. If credential files are deleted, the proxy continues using cached tokens. If a token refresh succeeds but the file cannot be written, the new token is buffered for retry and saved on shutdown.
Usage Tracking (Level 3): Usage statistics (key_usage.json) are maintained in memory via ResilientStateWriter. If the file is deleted, the system tracks usage internally and attempts to recreate the file on the next save interval. Pending writes are flushed on shutdown.
Provider Cache (Level 4): The provider cache tracks disk health and continues operating in memory-only mode if disk writes fail. Has its own shutdown mechanism.
Logging (Level 5): Logging is treated as non-critical. If the logs/ directory is removed, the system attempts to recreate it. If creation fails, logging degrades gracefully without interrupting the request flow. No buffering or retry.

5.3. Component Integration

Component	Utility Used	Behavior on Disk Failure	Shutdown Flush
`UsageManager`	`ResilientStateWriter`	Continues in memory, retries after 30s	Yes (via registry)
`GoogleOAuthBase`	`safe_write_json(buffer_on_failure=True)`	Memory cache preserved, buffered for retry	Yes (via registry)
`QwenAuthBase`	`safe_write_json(buffer_on_failure=True)`	Memory cache preserved, buffered for retry	Yes (via registry)
`IFlowAuthBase`	`safe_write_json(buffer_on_failure=True)`	Memory cache preserved, buffered for retry	Yes (via registry)
`ProviderCache`	`safe_write_json` + own shutdown	Retries via own background loop	Yes (own mechanism)
`DetailedLogger`	`safe_write_json`	Logs dropped, no crash	No
`failure_logger`	Python `logging.RotatingFileHandler`	Falls back to NullHandler	No

5.4. Shutdown Behavior

When the application exits (including Ctrl+C):

atexit handler fires: BufferedWriteRegistry._atexit_handler() is called
Pending writes counted: Registry checks how many files have pending writes
Flush attempted: Each pending file gets a final write attempt
Results logged:
- Success: "Shutdown flush: all N write(s) succeeded"
- Partial: "Shutdown flush: X succeeded, Y failed" with failed file names

Console output example:

INFO:rotator_library.resilient_io:Flushing 2 pending write(s) on shutdown...
INFO:rotator_library.resilient_io:Shutdown flush: all 2 write(s) succeeded

5.5. "Develop While Running"

This architecture supports a robust development workflow:

Log Cleanup: You can safely run rm -rf logs/ while the proxy is serving traffic. The system will recreate the directory structure on the next request.
Config Reset: Deleting key_usage.json resets the persistence layer, but the running instance preserves its current in-memory counts for load balancing consistency.
File Recovery: If you delete a critical file, the system attempts directory auto-recreation before every write operation.
Safe Exit: Ctrl+C triggers graceful shutdown with final data flush attempt.

5.6. Graceful Degradation & Data Loss

While functionality is preserved, persistence may be compromised during filesystem failures:

Logs: If disk writes fail, detailed request logs may be lost (no buffering).
Usage Stats: Buffered in memory and flushed on shutdown. Data loss only if shutdown flush also fails.
Credentials: Buffered in memory and flushed on shutdown. Re-authentication only needed if shutdown flush fails.
Cache: Provider cache entries may need to be regenerated after restart if its own shutdown mechanism fails.

5.7. Monitoring Disk Health

Components expose health information for monitoring:

# BufferedWriteRegistry
registry = BufferedWriteRegistry.get_instance()
pending = registry.get_pending_count()  # Number of files with pending writes
files = registry.get_pending_paths()    # List of pending file names

# UsageManager
writer = usage_manager._state_writer
health = writer.get_health_info()
# Returns: {"healthy": True, "failure_count": 0, "last_success": 1234567890.0, ...}

# ProviderCache
stats = cache.get_stats()
# Includes: {"disk_available": True, "disk_errors": 0, ...}

6. Model Filter GUI

The Model Filter GUI (model_filter_gui.py) provides a visual interface for configuring model ignore and whitelist rules per provider. It replaces the need to manually edit IGNORE_MODELS_* and WHITELIST_MODELS_* environment variables.

6.1. Overview

Purpose: Visually manage which models are exposed via the /v1/models endpoint for each provider.

Launch:

python -c "from src.proxy_app.model_filter_gui import run_model_filter_gui; run_model_filter_gui()"

Or via the launcher TUI if integrated.

6.2. Features

Core Functionality

Provider Selection: Dropdown to switch between available providers with automatic model fetching
Ignore Rules: Pattern-based rules (supports wildcards like *-preview, gpt-4*) to exclude models
Whitelist Rules: Pattern-based rules to explicitly include models, overriding ignore rules
Real-time Preview: Typing in rule input fields highlights affected models before committing
Rule-Model Linking: Click a model to highlight the affecting rule; click a rule to highlight all affected models
Persistence: Rules saved to .env file in standard IGNORE_MODELS_<PROVIDER> and WHITELIST_MODELS_<PROVIDER> format

Dual-Pane Model View

The interface displays two synchronized lists:

Left Pane	Right Pane
All fetched models (plain text)	Same models with color-coded status
Shows total count	Shows available/ignored count
Scrolls in sync with right pane	Color indicates affecting rule

Color Coding:

Green: Model is available (no rule affects it, or whitelisted)
Red/Orange tones: Model is ignored (color matches the specific ignore rule)
Blue/Teal tones: Model is explicitly whitelisted (color matches the whitelist rule)

Rule Management

Comma-separated input: Add multiple rules at once (e.g., *-preview, *-beta, gpt-3.5*)
Wildcard support: * matches any characters (e.g., gemini-*-preview)
Affected count: Each rule shows how many models it affects
Tooltips: Hover over a rule to see the list of affected models
Instant delete: Click the × button to remove a rule immediately

6.3. Keyboard Shortcuts

Shortcut	Action
`Ctrl+S`	Save changes to `.env`
`Ctrl+R`	Refresh models from provider
`Ctrl+F`	Focus search field
`F1`	Show help dialog
`Escape`	Clear search / Clear highlights

6.4. Context Menu

Right-click on any model to access:

Add to Ignore List: Creates an ignore rule for the exact model name
Add to Whitelist: Creates a whitelist rule for the exact model name
View Affecting Rule: Highlights the rule that affects this model
Copy Model Name: Copies the full model ID to clipboard

6.5. Integration with Proxy

The GUI modifies the same environment variables that the RotatingClient reads:

GUI saves rules → Updates .env file
Proxy reads on startup → Loads IGNORE_MODELS_* and WHITELIST_MODELS_*
Proxy applies rules → get_available_models() filters based on rules

Note: The proxy must be restarted to pick up rule changes made via the GUI (or use the Launcher TUI's reload functionality if available).

Technical Documentation: Universal LLM API Proxy & Resilience Library

1. Architecture Overview

2. rotator_library - The Resilience Engine

2.1. client.py - The RotatingClient

Initialization

Core Responsibilities

Model Filtering Logic

Request Lifecycle: A Deadline-Driven Approach

Streaming Resilience

2.2. usage_manager.py - Stateful Concurrency & Usage Management

Key Concepts

Tiered Key Acquisition Strategy

Failure Handling & Cooldowns

2.3. batch_manager.py - Efficient Request Aggregation

2.4. background_refresher.py - Automated Token Maintenance & Provider Jobs

OAuth Token Refresh

Provider-Specific Background Jobs

2.6. Credential Management Architecture

2.6.1. Automated Discovery & Preparation

2.6.2. Credential Loading & Stateless Operation

2.6.3. Credential Tool Integration

2.7. Request Sanitizer (request_sanitizer.py)

2.8. Error Classification (error_handler.py)

2.9. Cooldown Management (cooldown_manager.py)

2.10. Credential Prioritization System (client.py & usage_manager.py)

2.11. Provider Cache System (providers/provider_cache.py)

2.15. Antigravity Quota Tracker (providers/utilities/antigravity_quota_tracker.py)

Core Concepts

Key Methods

Integration with Background Jobs

Data Storage

2.16. TransientQuotaError (error_handler.py)

3.5. Antigravity (antigravity_provider.py)

Architecture

Model Support

Base URL Fallback

Message Transformation

Configuration (Environment Variables)

Claude Extended Thinking Sanitization

File Logging

2.13. Sequential Rotation & Per-Model Quota Tracking

Rotation Modes

Per-Model Quota Tracking

Provider-Specific Quota Parsing

Model Quota Groups

Priority-Based Concurrency Multipliers

Reset Window Configuration

Usage Flow

2.12. Google OAuth Base (providers/google_oauth_base.py)

OAuth Callback Port Configuration

2.14. HTTP Timeout Configuration (timeout_config.py)

Purpose

Timeout Types Explained

Default Values

Environment Variable Overrides

Streaming vs Non-Streaming Behavior

Provider Usage

Tuning Recommendations

Example Configuration

3. Provider Specific Implementations

3.1. Gemini CLI (gemini_cli_provider.py)

Authentication (gemini_auth_base.py)

Project ID Discovery (Zero-Config)

Rate Limit Handling

3.2. Qwen Code (qwen_code_provider.py)

3.3. iFlow (iflow_provider.py)

3.4. Google Gemini (gemini_provider.py)

4. Logging & Debugging

detailed_logger.py

5. Runtime Resilience

5.1. Centralized Resilient I/O (resilient_io.py)

BufferedWriteRegistry (Singleton)

ResilientStateWriter

safe_write_json()

safe_log_write()

safe_mkdir()

5.2. Resilience Hierarchy

5.3. Component Integration

5.4. Shutdown Behavior

5.5. "Develop While Running"

2. `rotator_library` - The Resilience Engine

2.1. `client.py` - The `RotatingClient`

2.2. `usage_manager.py` - Stateful Concurrency & Usage Management

2.3. `batch_manager.py` - Efficient Request Aggregation

2.4. `background_refresher.py` - Automated Token Maintenance & Provider Jobs

2.7. Request Sanitizer (`request_sanitizer.py`)

2.8. Error Classification (`error_handler.py`)

2.9. Cooldown Management (`cooldown_manager.py`)

2.10. Credential Prioritization System (`client.py` & `usage_manager.py`)

2.11. Provider Cache System (`providers/provider_cache.py`)

2.15. Antigravity Quota Tracker (`providers/utilities/antigravity_quota_tracker.py`)

2.16. TransientQuotaError (`error_handler.py`)

3.5. Antigravity (`antigravity_provider.py`)

2.12. Google OAuth Base (`providers/google_oauth_base.py`)

2.14. HTTP Timeout Configuration (`timeout_config.py`)

3.1. Gemini CLI (`gemini_cli_provider.py`)

Authentication (`gemini_auth_base.py`)

3.2. Qwen Code (`qwen_code_provider.py`)

3.3. iFlow (`iflow_provider.py`)

3.4. Google Gemini (`gemini_provider.py`)

`detailed_logger.py`

5.1. Centralized Resilient I/O (`resilient_io.py`)

`BufferedWriteRegistry` (Singleton)

`ResilientStateWriter`

`safe_write_json()`

`safe_log_write()`

`safe_mkdir()`