up/PRD.md

Redis Caching Integration - Product Requirements Document

Executive Summary

This PRD outlines the integration of Redis caching into the CLI Proxy API to enable horizontal scaling, improve performance, and add distributed state management capabilities.

Background

Current State

The CLI Proxy API currently uses:

• In-memory caching: Signature cache, Codex prompt cache, OAuth sessions
• File-based storage: Authentication tokens, configuration
• Single-instance limitations: No shared state across multiple server instances
• No persistence: Usage statistics and cache data lost on restart

Problems

1. Cannot scale horizontally: Multiple instances don't share state
2. Data loss on restart: Usage statistics and cache data are ephemeral
3. No distributed rate limiting: Each instance tracks limits independently
4. Session management issues: OAuth sessions not shared across instances
5. Performance bottlenecks: Repeated expensive operations (token counting, model lookups)

Goals

Primary Goals

1. Enable Horizontal Scaling: Allow multiple server instances to share state via Redis
2. Improve Performance: Cache expensive operations (API responses, token counts, model registry)
3. Add Persistence: Preserve usage statistics and cache data across restarts
4. Distributed Rate Limiting: Implement per-API-key and per-model rate limiting

Secondary Goals

1. Backward Compatibility: Maintain existing file-based storage as fallback
2. Optional Redis: System should work without Redis (degraded mode)
3. Monitoring: Add Redis health checks and metrics
4. Configuration: Simple Redis configuration via config.yaml

Target Users

1. Self-hosted users: Running multiple instances for high availability
2. Enterprise users: Need distributed rate limiting and usage tracking
3. Cloud deployments: Kubernetes/Docker environments with multiple replicas
4. Development teams: Need persistent cache during development

Requirements

Functional Requirements

FR-1: Redis Connection Management

• Support Redis standalone, Sentinel, and Cluster modes
• Connection pooling with configurable pool size
• Automatic reconnection on connection loss
• Health check endpoint for Redis status

FR-2: Distributed Caching Layer

• Cache-Aside Pattern: Application checks cache first, loads from source on miss
• Write-Through Pattern: Update cache and source simultaneously for critical data
• TTL Management: Configurable expiration for different cache types
• Namespace Support: Prefix keys by instance/environment

FR-3: Authentication Token Caching

• Cache OAuth tokens with automatic refresh
• Share tokens across instances
• Fallback to file-based storage if Redis unavailable
• Secure storage with encryption support

FR-4: Model Registry Caching

• Cache model availability across instances
• Real-time updates when models added/removed
• TTL: 5 minutes (configurable)
• Pub/Sub for instant registry updates

FR-5: Usage Statistics Persistence

• Persist aggregated statistics to Redis
• Support statistics export/import
• Hourly/daily aggregation with Redis sorted sets
• Retention policy (e.g., 30 days)

FR-6: Response Caching

• Cache identical API requests
• Configurable TTL per model/endpoint
• Cache key based on: model, messages hash, parameters
• Respect streaming vs non-streaming
• Optional feature (disabled by default)

FR-7: Rate Limiting

• Distributed rate limiting per API key
• Per-model rate limiting
• Sliding window algorithm using Redis sorted sets
• Configurable limits: requests/minute, requests/hour, requests/day
• Return proper 429 responses with Retry-After header

FR-8: OAuth Session Management

• Store OAuth sessions in Redis
• Share sessions across instances
• TTL: 10 minutes (existing behavior)
• Automatic cleanup of expired sessions

FR-9: Signature Cache (Thinking Blocks)

• Migrate existing in-memory signature cache to Redis
• TTL: 3 hours (existing behavior)
• Support for Claude thinking block signatures

FR-10: Codex Prompt Cache

• Migrate existing Codex cache to Redis
• TTL: 1 hour (existing behavior)
• Share prompt cache IDs across instances

Non-Functional Requirements

NFR-1: Performance

• Redis operations should add < 5ms latency
• Connection pool should handle 1000+ concurrent requests
• Cache hit rate > 80% for frequently accessed data

NFR-2: Reliability

• Graceful degradation when Redis unavailable
• Automatic failover in Sentinel/Cluster mode
• No data loss for critical operations

NFR-3: Security

• Support Redis AUTH password
• TLS/SSL connection support
• Encryption for sensitive cached data
• No credentials in cache keys

NFR-4: Scalability

• Support 10+ server instances
• Handle 10,000+ requests/second aggregate
• Redis memory usage < 2GB for typical workload

NFR-5: Maintainability

• Clear logging for Redis operations
• Metrics for cache hit/miss rates
• Configuration validation on startup
• Migration path from in-memory to Redis

Architecture

High-Level Design

┌─────────────────────────────────────────────────────────┐
│                   Load Balancer                         │
└─────────────────────────────────────────────────────────┘
                          │
        ┌─────────────────┼─────────────────┐
        │                 │                 │
┌───────▼──────┐  ┌──────▼───────┐  ┌──────▼───────┐
│  Instance 1  │  │  Instance 2  │  │  Instance 3  │
│              │  │              │  │              │
│ ┌──────────┐ │  │ ┌──────────┐ │  │ ┌──────────┐ │
│ │ App Code │ │  │ │ App Code │ │  │ │ App Code │ │
│ └────┬─────┘ │  │ └────┬─────┘ │  │ └────┬─────┘ │
│      │       │  │      │       │  │      │       │
│ ┌────▼─────┐ │  │ ┌────▼─────┐ │  │ ┌────▼─────┐ │
│ │  Cache   │ │  │ │  Cache   │ │  │ │  Cache   │ │
│ │  Layer   │ │  │ │  Layer   │ │  │ │  Layer   │ │
│ └────┬─────┘ │  │ └────┬─────┘ │  │ └────┬─────┘ │
└──────┼───────┘  └──────┼───────┘  └──────┼───────┘
       │                 │                 │
       └─────────────────┼─────────────────┘
                         │
                ┌────────▼────────┐
                │  Redis Cluster  │
                │                 │
                │  ┌───────────┐  │
                │  │  Cache    │  │
                │  │  Storage  │  │
                │  └───────────┘  │
                │  ┌───────────┐  │
                │  │  Pub/Sub  │  │
                │  └───────────┘  │
                └─────────────────┘

Cache Layer Structure

// Proposed package structure
internal/cache/
├── redis/
│   ├── client.go          // Redis client wrapper
│   ├── config.go          // Configuration
│   ├── health.go          // Health checks
│   └── pool.go            // Connection pooling
├── store/
│   ├── interface.go       // Cache store interface
│   ├── redis_store.go     // Redis implementation
│   ├── memory_store.go    // In-memory fallback
│   └── hybrid_store.go    // Redis + memory hybrid
├── auth_cache.go          // Authentication caching
├── model_cache.go         // Model registry caching
├── response_cache.go      // Response caching
├── rate_limiter.go        // Distributed rate limiting
├── session_cache.go       // OAuth session caching
├── signature_cache.go     // Thinking signature caching (migrate existing)
└── usage_cache.go         // Usage statistics caching

Configuration Schema

# config.yaml additions
redis:
  # Enable Redis caching (default: false)
  enabled: true
  
  # Redis connection
  host: "localhost"
  port: 6379
  password: ""
  db: 0
  
  # TLS configuration
  tls:
    enabled: false
    cert: ""
    key: ""
    ca: ""
  
  # Connection pool
  pool:
    max-idle: 10
    max-active: 100
    idle-timeout: 300s
    max-conn-lifetime: 3600s
  
  # Cluster/Sentinel mode
  mode: "standalone"  # standalone, sentinel, cluster
  sentinel:
    master-name: "mymaster"
    addresses:
      - "sentinel1:26379"
      - "sentinel2:26379"
  cluster:
    addresses:
      - "node1:6379"
      - "node2:6379"
      - "node3:6379"
  
  # Key prefix for namespacing
  key-prefix: "cliproxy:"
  
  # Feature-specific settings
  caching:
    # Authentication tokens
    auth-tokens:
      enabled: true
      ttl: 3600s
    
    # Model registry
    model-registry:
      enabled: true
      ttl: 300s
    
    # Response caching
    responses:
      enabled: false  # Disabled by default
      ttl: 60s
      max-size: 1000  # Max cached responses
    
    # OAuth sessions
    oauth-sessions:
      enabled: true
      ttl: 600s
    
    # Signature cache (thinking blocks)
    signatures:
      enabled: true
      ttl: 10800s  # 3 hours
    
    # Codex prompt cache
    codex-cache:
      enabled: true
      ttl: 3600s
    
    # Usage statistics
    usage-stats:
      enabled: true
      retention-days: 30
  
  # Rate limiting
  rate-limiting:
    enabled: false  # Disabled by default
    rules:
      - api-key: "default"
        requests-per-minute: 60
        requests-per-hour: 1000
        requests-per-day: 10000
      - model: "claude-opus-4"
        requests-per-minute: 10
        requests-per-hour: 100

Cache Strategies by Component

1. Authentication Tokens (Write-Through)

• Pattern: Write-Through + Cache-Aside
• TTL: 1 hour (or token expiry)
• Key: cliproxy:auth:{provider}:{auth_id}
• Fallback: File-based storage
• Invalidation: On token refresh or logout

2. Model Registry (Cache-Aside + Pub/Sub)

• Pattern: Cache-Aside with Pub/Sub updates
• TTL: 5 minutes
• Key: cliproxy:models:{provider}:{client_id}
• Pub/Sub: cliproxy:events:models
• Invalidation: On model registration/unregistration

3. Response Caching (Cache-Aside)

• Pattern: Cache-Aside (optional feature)
• TTL: 60 seconds (configurable)
• Key: cliproxy:response:{hash(model+messages+params)}
• Conditions: Only cache successful, non-streaming responses
• Invalidation: TTL expiry only

4. Rate Limiting (Sliding Window)

• Pattern: Sliding Window with Sorted Sets
• Key: cliproxy:ratelimit:{api_key}:{window}
• Windows: minute, hour, day
• Algorithm: ZREMRANGEBYSCORE + ZADD + ZCARD
• Cleanup: Automatic via TTL

5. OAuth Sessions (Cache-Aside)

• Pattern: Cache-Aside
• TTL: 10 minutes
• Key: cliproxy:oauth:session:{state}
• Invalidation: On completion or TTL expiry

6. Usage Statistics (Write-Behind)

• Pattern: Write-Behind (async aggregation)
• Key: cliproxy:usage:{api_key}:{date}:{hour}
• Storage: Sorted Sets for time-series data
• Aggregation: Hourly background job
• Retention: 30 days

Implementation Phases

Phase 1: Foundation (Week 1)

• Redis client wrapper and connection management
• Configuration schema and validation
• Health check endpoints
• Basic cache interface and Redis store implementation

Phase 2: Core Caching (Week 2)

• Migrate signature cache to Redis
• Migrate Codex cache to Redis
• OAuth session caching
• Model registry caching with Pub/Sub

Phase 3: Advanced Features (Week 3)

• Authentication token caching with fallback
• Usage statistics persistence
• Response caching (optional)
• Monitoring and metrics

Phase 4: Rate Limiting (Week 4)

• Distributed rate limiting implementation
• Per-API-key and per-model limits
• Admin API for rate limit management
• Testing and optimization

Success Metrics

Performance Metrics

• Cache hit rate > 80% for model registry
• Cache hit rate > 60% for responses (if enabled)
• Redis operation latency < 5ms (p95)
• Overall request latency increase < 10ms

Reliability Metrics

• Zero data loss for authentication tokens
• 99.9% uptime with Redis failover
• Graceful degradation when Redis unavailable

Scalability Metrics

• Support 10+ instances without performance degradation
• Handle 10,000+ requests/second aggregate
• Redis memory usage < 2GB for typical workload

Risks and Mitigations

Risk 1: Redis Unavailability

• Impact: High - System cannot share state
• Mitigation: Fallback to in-memory/file-based storage, health checks, automatic reconnection

Risk 2: Cache Stampede

• Impact: Medium - Multiple instances fetch same data simultaneously
• Mitigation: Use locking mechanisms, staggered TTLs, cache warming

Risk 3: Memory Exhaustion

• Impact: High - Redis runs out of memory
• Mitigation: Eviction policies (LRU), memory limits, monitoring, TTL enforcement

Risk 4: Data Inconsistency

• Impact: Medium - Cache and source out of sync
• Mitigation: Proper invalidation strategies, short TTLs for critical data, Pub/Sub updates

Risk 5: Security Concerns

• Impact: High - Sensitive data exposed in cache
• Mitigation: Encryption, Redis AUTH, TLS, no credentials in keys, secure key generation

Dependencies

External Dependencies

• Redis: v6.0+ (for ACL support) or v7.0+ (recommended)
• Go Redis Client: github.com/redis/go-redis/v9
• Existing: All current dependencies remain

Internal Dependencies

• Modify: internal/cache/signature_cache.go
• Modify: internal/runtime/executor/cache_helpers.go
• Modify: internal/api/handlers/management/oauth_sessions.go
• Modify: internal/usage/logger_plugin.go
• Modify: sdk/cliproxy/auth/conductor.go
• Modify: internal/config/config.go

Open Questions

1. Should response caching be enabled by default or opt-in?
2. What should be the default rate limits per API key?
3. Should we support Redis Cluster from day one or add later?
4. How to handle cache warming on startup?
5. Should we add a cache admin API for manual invalidation?

Appendix

A. Redis Key Naming Convention

cliproxy:{component}:{identifier}:{sub-key}

Examples:
- cliproxy:auth:claude:abc123
- cliproxy:models:gemini:client-1
- cliproxy:response:hash-xyz
- cliproxy:ratelimit:api-key-1:minute
- cliproxy:oauth:session:state-xyz
- cliproxy:usage:api-1:2026-02-10:14

B. Cache TTL Guidelines

• Authentication tokens: 1 hour or token expiry
• Model registry: 5 minutes
• Responses: 60 seconds (if enabled)
• OAuth sessions: 10 minutes
• Signatures: 3 hours
• Codex cache: 1 hour
• Rate limit windows: 1 day (for daily limits)
• Usage stats: 30 days

C. Redis Memory Estimation

For 1000 concurrent users:

• Auth tokens: ~1MB (1KB per token)
• Model registry: ~500KB
• OAuth sessions: ~100KB
• Signatures: ~5MB
• Responses (1000 cached): ~50MB
• Rate limiting: ~10MB
• Usage stats (30 days): ~100MB
• Total: ~170MB + overhead = ~250MB

For 10,000 concurrent users: ~2.5GB