plans/performance-agentic-roadmap.md

Performance & Agentic Roadmap for CLIProxyAPI

This roadmap focuses on optimizing the server for high-performance single-user usage (low latency, high throughput) and enhancing agentic capabilities (tool use, reasoning, debugging).

Phase 1: Core Performance Optimization

1.1 HTTP Client Reuse (Critical)

Problem: Currently, handlers create a new http.Client for every request. This disables TCP connection pooling (Keep-Alive), causing unnecessary TLS handshakes and increasing latency. Solution:

• Create a global *http.Client in main.go with optimized transport settings.
• Inject this client into all handlers.

// Recommended Transport Settings
t := &http.Transport{
    MaxIdleConns:        100,
    MaxIdleConnsPerHost: 20,
    IdleConnTimeout:     90 * time.Second,
    DisableCompression:  true, // Proxy should often pass through raw bytes
}

1.2 Memory Optimization (sync.Pool)

Problem: JSON translation involves allocating new byte slices for every request/response body. Solution:

• Implement sync.Pool for byte buffers used in the translator package.
• Reuse buffers for reading request bodies and constructing responses.

1.3 Asynchronous Logging

Problem: Logging might be blocking the request path. Solution:

• Ensure logrus or the custom logger is writing asynchronously or to a buffered channel to avoid I/O blocking on the main request thread.

Phase 2: Agentic Capabilities & Tooling

2.1 Unified Tool Abstraction

Current State: Tool translation is handled point-to-point (e.g., Gemini->OpenAI). Goal: Create a standardized ToolDefinition struct in sdk that acts as an intermediate representation (IR). Benefit:

• Easier to add new providers (Ollama, DeepSeek, etc.).
• Write tools once, run on any provider.

2.2 "Agentic Trace" Debugging

Goal: When using CLI tools (like Cline/RooCode), it's hard to see why a tool call failed. Solution:

• Add a generic X-Agent-Trace-ID header.
• Create a specific "Trace" log level that captures the exact JSON sent to and from the upstream provider for tool calls.
• Expose a simple /v1/trace/{id} endpoint to view the "thought process" and tool outputs.

2.3 Enhanced "Thinking" Support

Goal: Maximize the reasoning capabilities of models like Claude 3.7 and OpenAI o1. Actions:

• Ensure internal/thinking supports "Budget" parameters for all providers (currently seems focused on specific ones).
• Add support for "Thought Blocks" parsing in the stream to separate "reasoning" from "final answer" for clients that don't support it natively.

Phase 3: Architecture & Maintainability

3.1 Refactor main.go

Problem: The entry point is too complex (God Function). Solution:

• Extract server initialization into internal/bootstrap.
• Move configuration loading to internal/config/loader.go.

3.2 Security Hardening (Architectural)

Action:

• Externalize all hardcoded OAuth secrets to config.yaml.
• Implement a simple "Allowlist" for the Management API's APICall to prevent SSRF, even in a private network (defense in depth).

Implementation Priority

1. Refactor HTTP Client (Highest Impact/Effort ratio).
2. Refactor main.go (Makes future changes easier).
3. Agentic Trace Logging (High value for debugging "smart" agents).
4. Buffer Pools (Micro-optimization, do last).