up/PRD.md

# Redis Caching Integration - Product Requirements Document

## Executive Summary

This PRD outlines the integration of Redis caching into the CLI Proxy API to enable horizontal scaling, improve performance, and add distributed state management capabilities.

## Background

### Current State
The CLI Proxy API currently uses:
- **In-memory caching**: Signature cache, Codex prompt cache, OAuth sessions
- **File-based storage**: Authentication tokens, configuration
- **Single-instance limitations**: No shared state across multiple server instances
- **No persistence**: Usage statistics and cache data lost on restart

### Problems
1. **Cannot scale horizontally**: Multiple instances don't share state
2. **Data loss on restart**: Usage statistics and cache data are ephemeral
3. **No distributed rate limiting**: Each instance tracks limits independently
4. **Session management issues**: OAuth sessions not shared across instances
5. **Performance bottlenecks**: Repeated expensive operations (token counting, model lookups)

## Goals

### Primary Goals
1. **Enable Horizontal Scaling**: Allow multiple server instances to share state via Redis
2. **Improve Performance**: Cache expensive operations (API responses, token counts, model registry)
3. **Add Persistence**: Preserve usage statistics and cache data across restarts
4. **Distributed Rate Limiting**: Implement per-API-key and per-model rate limiting

### Secondary Goals
1. **Backward Compatibility**: Maintain existing file-based storage as fallback
2. **Optional Redis**: System should work without Redis (degraded mode)
3. **Monitoring**: Add Redis health checks and metrics
4. **Configuration**: Simple Redis configuration via config.yaml

## Target Users

1. **Self-hosted users**: Running multiple instances for high availability
2. **Enterprise users**: Need distributed rate limiting and usage tracking
3. **Cloud deployments**: Kubernetes/Docker environments with multiple replicas
4. **Development teams**: Need persistent cache during development

## Requirements

### Functional Requirements

#### FR-1: Redis Connection Management
- Support Redis standalone, Sentinel, and Cluster modes
- Connection pooling with configurable pool size
- Automatic reconnection on connection loss
- Health check endpoint for Redis status

#### FR-2: Distributed Caching Layer
- **Cache-Aside Pattern**: Application checks cache first, loads from source on miss
- **Write-Through Pattern**: Update cache and source simultaneously for critical data
- **TTL Management**: Configurable expiration for different cache types
- **Namespace Support**: Prefix keys by instance/environment

#### FR-3: Authentication Token Caching
- Cache OAuth tokens with automatic refresh
- Share tokens across instances
- Fallback to file-based storage if Redis unavailable
- Secure storage with encryption support

#### FR-4: Model Registry Caching
- Cache model availability across instances
- Real-time updates when models added/removed
- TTL: 5 minutes (configurable)
- Pub/Sub for instant registry updates

#### FR-5: Usage Statistics Persistence
- Persist aggregated statistics to Redis
- Support statistics export/import
- Hourly/daily aggregation with Redis sorted sets
- Retention policy (e.g., 30 days)

#### FR-6: Response Caching
- Cache identical API requests
- Configurable TTL per model/endpoint
- Cache key based on: model, messages hash, parameters
- Respect streaming vs non-streaming
- Optional feature (disabled by default)

#### FR-7: Rate Limiting
- Distributed rate limiting per API key
- Per-model rate limiting
- Sliding window algorithm using Redis sorted sets
- Configurable limits: requests/minute, requests/hour, requests/day
- Return proper 429 responses with Retry-After header

#### FR-8: OAuth Session Management
- Store OAuth sessions in Redis
- Share sessions across instances
- TTL: 10 minutes (existing behavior)
- Automatic cleanup of expired sessions

#### FR-9: Signature Cache (Thinking Blocks)
- Migrate existing in-memory signature cache to Redis
- TTL: 3 hours (existing behavior)
- Support for Claude thinking block signatures

#### FR-10: Codex Prompt Cache
- Migrate existing Codex cache to Redis
- TTL: 1 hour (existing behavior)
- Share prompt cache IDs across instances

### Non-Functional Requirements

#### NFR-1: Performance
- Redis operations should add < 5ms latency
- Connection pool should handle 1000+ concurrent requests
- Cache hit rate > 80% for frequently accessed data

#### NFR-2: Reliability
- Graceful degradation when Redis unavailable
- Automatic failover in Sentinel/Cluster mode
- No data loss for critical operations

#### NFR-3: Security
- Support Redis AUTH password
- TLS/SSL connection support
- Encryption for sensitive cached data
- No credentials in cache keys

#### NFR-4: Scalability
- Support 10+ server instances
- Handle 10,000+ requests/second aggregate
- Redis memory usage < 2GB for typical workload

#### NFR-5: Maintainability
- Clear logging for Redis operations
- Metrics for cache hit/miss rates
- Configuration validation on startup
- Migration path from in-memory to Redis

## Architecture

### High-Level Design

```
┌─────────────────────────────────────────────────────────┐
│                   Load Balancer                         │
└─────────────────────────────────────────────────────────┘
                          │
        ┌─────────────────┼─────────────────┐
        │                 │                 │
┌───────▼──────┐  ┌──────▼───────┐  ┌──────▼───────┐
│  Instance 1  │  │  Instance 2  │  │  Instance 3  │
│              │  │              │  │              │
│ ┌──────────┐ │  │ ┌──────────┐ │  │ ┌──────────┐ │
│ │ App Code │ │  │ │ App Code │ │  │ │ App Code │ │
│ └────┬─────┘ │  │ └────┬─────┘ │  │ └────┬─────┘ │
│      │       │  │      │       │  │      │       │
│ ┌────▼─────┐ │  │ ┌────▼─────┐ │  │ ┌────▼─────┐ │
│ │  Cache   │ │  │ │  Cache   │ │  │ │  Cache   │ │
│ │  Layer   │ │  │ │  Layer   │ │  │ │  Layer   │ │
│ └────┬─────┘ │  │ └────┬─────┘ │  │ └────┬─────┘ │
└──────┼───────┘  └──────┼───────┘  └──────┼───────┘
       │                 │                 │
       └─────────────────┼─────────────────┘
                         │
                ┌────────▼────────┐
                │  Redis Cluster  │
                │                 │
                │  ┌───────────┐  │
                │  │  Cache    │  │
                │  │  Storage  │  │
                │  └───────────┘  │
                │  ┌───────────┐  │
                │  │  Pub/Sub  │  │
                │  └───────────┘  │
                └─────────────────┘
```

### Cache Layer Structure

```go
// Proposed package structure
internal/cache/
├── redis/
│   ├── client.go          // Redis client wrapper
│   ├── config.go          // Configuration
│   ├── health.go          // Health checks
│   └── pool.go            // Connection pooling
├── store/
│   ├── interface.go       // Cache store interface
│   ├── redis_store.go     // Redis implementation
│   ├── memory_store.go    // In-memory fallback
│   └── hybrid_store.go    // Redis + memory hybrid
├── auth_cache.go          // Authentication caching
├── model_cache.go         // Model registry caching
├── response_cache.go      // Response caching
├── rate_limiter.go        // Distributed rate limiting
├── session_cache.go       // OAuth session caching
├── signature_cache.go     // Thinking signature caching (migrate existing)
└── usage_cache.go         // Usage statistics caching
```

### Configuration Schema

```yaml
# config.yaml additions
redis:
  # Enable Redis caching (default: false)
  enabled: true
  
  # Redis connection
  host: "localhost"
  port: 6379
  password: ""
  db: 0
  
  # TLS configuration
  tls:
    enabled: false
    cert: ""
    key: ""
    ca: ""
  
  # Connection pool
  pool:
    max-idle: 10
    max-active: 100
    idle-timeout: 300s
    max-conn-lifetime: 3600s
  
  # Cluster/Sentinel mode
  mode: "standalone"  # standalone, sentinel, cluster
  sentinel:
    master-name: "mymaster"
    addresses:
      - "sentinel1:26379"
      - "sentinel2:26379"
  cluster:
    addresses:
      - "node1:6379"
      - "node2:6379"
      - "node3:6379"
  
  # Key prefix for namespacing
  key-prefix: "cliproxy:"
  
  # Feature-specific settings
  caching:
    # Authentication tokens
    auth-tokens:
      enabled: true
      ttl: 3600s
    
    # Model registry
    model-registry:
      enabled: true
      ttl: 300s
    
    # Response caching
    responses:
      enabled: false  # Disabled by default
      ttl: 60s
      max-size: 1000  # Max cached responses
    
    # OAuth sessions
    oauth-sessions:
      enabled: true
      ttl: 600s
    
    # Signature cache (thinking blocks)
    signatures:
      enabled: true
      ttl: 10800s  # 3 hours
    
    # Codex prompt cache
    codex-cache:
      enabled: true
      ttl: 3600s
    
    # Usage statistics
    usage-stats:
      enabled: true
      retention-days: 30
  
  # Rate limiting
  rate-limiting:
    enabled: false  # Disabled by default
    rules:
      - api-key: "default"
        requests-per-minute: 60
        requests-per-hour: 1000
        requests-per-day: 10000
      - model: "claude-opus-4"
        requests-per-minute: 10
        requests-per-hour: 100
```

## Cache Strategies by Component

### 1. Authentication Tokens (Write-Through)
- **Pattern**: Write-Through + Cache-Aside
- **TTL**: 1 hour (or token expiry)
- **Key**: `cliproxy:auth:{provider}:{auth_id}`
- **Fallback**: File-based storage
- **Invalidation**: On token refresh or logout

### 2. Model Registry (Cache-Aside + Pub/Sub)
- **Pattern**: Cache-Aside with Pub/Sub updates
- **TTL**: 5 minutes
- **Key**: `cliproxy:models:{provider}:{client_id}`
- **Pub/Sub**: `cliproxy:events:models`
- **Invalidation**: On model registration/unregistration

### 3. Response Caching (Cache-Aside)
- **Pattern**: Cache-Aside (optional feature)
- **TTL**: 60 seconds (configurable)
- **Key**: `cliproxy:response:{hash(model+messages+params)}`
- **Conditions**: Only cache successful, non-streaming responses
- **Invalidation**: TTL expiry only

### 4. Rate Limiting (Sliding Window)
- **Pattern**: Sliding Window with Sorted Sets
- **Key**: `cliproxy:ratelimit:{api_key}:{window}`
- **Windows**: minute, hour, day
- **Algorithm**: ZREMRANGEBYSCORE + ZADD + ZCARD
- **Cleanup**: Automatic via TTL

### 5. OAuth Sessions (Cache-Aside)
- **Pattern**: Cache-Aside
- **TTL**: 10 minutes
- **Key**: `cliproxy:oauth:session:{state}`
- **Invalidation**: On completion or TTL expiry

### 6. Usage Statistics (Write-Behind)
- **Pattern**: Write-Behind (async aggregation)
- **Key**: `cliproxy:usage:{api_key}:{date}:{hour}`
- **Storage**: Sorted Sets for time-series data
- **Aggregation**: Hourly background job
- **Retention**: 30 days

## Implementation Phases

### Phase 1: Foundation (Week 1)
- Redis client wrapper and connection management
- Configuration schema and validation
- Health check endpoints
- Basic cache interface and Redis store implementation

### Phase 2: Core Caching (Week 2)
- Migrate signature cache to Redis
- Migrate Codex cache to Redis
- OAuth session caching
- Model registry caching with Pub/Sub

### Phase 3: Advanced Features (Week 3)
- Authentication token caching with fallback
- Usage statistics persistence
- Response caching (optional)
- Monitoring and metrics

### Phase 4: Rate Limiting (Week 4)
- Distributed rate limiting implementation
- Per-API-key and per-model limits
- Admin API for rate limit management
- Testing and optimization

## Success Metrics

### Performance Metrics
- Cache hit rate > 80% for model registry
- Cache hit rate > 60% for responses (if enabled)
- Redis operation latency < 5ms (p95)
- Overall request latency increase < 10ms

### Reliability Metrics
- Zero data loss for authentication tokens
- 99.9% uptime with Redis failover
- Graceful degradation when Redis unavailable

### Scalability Metrics
- Support 10+ instances without performance degradation
- Handle 10,000+ requests/second aggregate
- Redis memory usage < 2GB for typical workload

## Risks and Mitigations

### Risk 1: Redis Unavailability
- **Impact**: High - System cannot share state
- **Mitigation**: Fallback to in-memory/file-based storage, health checks, automatic reconnection

### Risk 2: Cache Stampede
- **Impact**: Medium - Multiple instances fetch same data simultaneously
- **Mitigation**: Use locking mechanisms, staggered TTLs, cache warming

### Risk 3: Memory Exhaustion
- **Impact**: High - Redis runs out of memory
- **Mitigation**: Eviction policies (LRU), memory limits, monitoring, TTL enforcement

### Risk 4: Data Inconsistency
- **Impact**: Medium - Cache and source out of sync
- **Mitigation**: Proper invalidation strategies, short TTLs for critical data, Pub/Sub updates

### Risk 5: Security Concerns
- **Impact**: High - Sensitive data exposed in cache
- **Mitigation**: Encryption, Redis AUTH, TLS, no credentials in keys, secure key generation

## Dependencies

### External Dependencies
- **Redis**: v6.0+ (for ACL support) or v7.0+ (recommended)
- **Go Redis Client**: github.com/redis/go-redis/v9
- **Existing**: All current dependencies remain

### Internal Dependencies
- Modify: `internal/cache/signature_cache.go`
- Modify: `internal/runtime/executor/cache_helpers.go`
- Modify: `internal/api/handlers/management/oauth_sessions.go`
- Modify: `internal/usage/logger_plugin.go`
- Modify: `sdk/cliproxy/auth/conductor.go`
- Modify: `internal/config/config.go`

## Open Questions

1. Should response caching be enabled by default or opt-in?
2. What should be the default rate limits per API key?
3. Should we support Redis Cluster from day one or add later?
4. How to handle cache warming on startup?
5. Should we add a cache admin API for manual invalidation?

## Appendix

### A. Redis Key Naming Convention
```
cliproxy:{component}:{identifier}:{sub-key}

Examples:
- cliproxy:auth:claude:abc123
- cliproxy:models:gemini:client-1
- cliproxy:response:hash-xyz
- cliproxy:ratelimit:api-key-1:minute
- cliproxy:oauth:session:state-xyz
- cliproxy:usage:api-1:2026-02-10:14
```

### B. Cache TTL Guidelines
- **Authentication tokens**: 1 hour or token expiry
- **Model registry**: 5 minutes
- **Responses**: 60 seconds (if enabled)
- **OAuth sessions**: 10 minutes
- **Signatures**: 3 hours
- **Codex cache**: 1 hour
- **Rate limit windows**: 1 day (for daily limits)
- **Usage stats**: 30 days

### C. Redis Memory Estimation
For 1000 concurrent users:
- Auth tokens: ~1MB (1KB per token)
- Model registry: ~500KB
- OAuth sessions: ~100KB
- Signatures: ~5MB
- Responses (1000 cached): ~50MB
- Rate limiting: ~10MB
- Usage stats (30 days): ~100MB
- **Total**: ~170MB + overhead = ~250MB

For 10,000 concurrent users: ~2.5GB