plans/performance-agentic-roadmap.md

# Performance & Agentic Roadmap for CLIProxyAPI

This roadmap focuses on optimizing the server for high-performance single-user usage (low latency, high throughput) and enhancing agentic capabilities (tool use, reasoning, debugging).

## Phase 1: Core Performance Optimization

### 1.1 HTTP Client Reuse (Critical)
**Problem:** Currently, handlers create a new `http.Client` for every request. This disables TCP connection pooling (Keep-Alive), causing unnecessary TLS handshakes and increasing latency.
**Solution:**
- Create a global `*http.Client` in `main.go` with optimized transport settings.
- Inject this client into all handlers.

```go
// Recommended Transport Settings
t := &http.Transport{
    MaxIdleConns:        100,
    MaxIdleConnsPerHost: 20,
    IdleConnTimeout:     90 * time.Second,
    DisableCompression:  true, // Proxy should often pass through raw bytes
}
```

### 1.2 Memory Optimization (sync.Pool)
**Problem:** JSON translation involves allocating new byte slices for every request/response body.
**Solution:**
- Implement `sync.Pool` for byte buffers used in the `translator` package.
- Reuse buffers for reading request bodies and constructing responses.

### 1.3 Asynchronous Logging
**Problem:** Logging might be blocking the request path.
**Solution:**
- Ensure `logrus` or the custom logger is writing asynchronously or to a buffered channel to avoid I/O blocking on the main request thread.

## Phase 2: Agentic Capabilities & Tooling

### 2.1 Unified Tool Abstraction
**Current State:** Tool translation is handled point-to-point (e.g., Gemini->OpenAI).
**Goal:** Create a standardized `ToolDefinition` struct in `sdk` that acts as an intermediate representation (IR).
**Benefit:**
- Easier to add new providers (Ollama, DeepSeek, etc.).
- Write tools once, run on any provider.

### 2.2 "Agentic Trace" Debugging
**Goal:** When using CLI tools (like Cline/RooCode), it's hard to see *why* a tool call failed.
**Solution:**
- Add a generic `X-Agent-Trace-ID` header.
- Create a specific "Trace" log level that captures the *exact* JSON sent to and from the upstream provider for tool calls.
- Expose a simple `/v1/trace/{id}` endpoint to view the "thought process" and tool outputs.

### 2.3 Enhanced "Thinking" Support
**Goal:** Maximize the reasoning capabilities of models like Claude 3.7 and OpenAI o1.
**Actions:**
- Ensure `internal/thinking` supports "Budget" parameters for all providers (currently seems focused on specific ones).
- Add support for "Thought Blocks" parsing in the stream to separate "reasoning" from "final answer" for clients that don't support it natively.

## Phase 3: Architecture & Maintainability

### 3.1 Refactor `main.go`
**Problem:** The entry point is too complex (`God Function`).
**Solution:**
- Extract server initialization into `internal/bootstrap`.
- Move configuration loading to `internal/config/loader.go`.

### 3.2 Security Hardening (Architectural)
**Action:**
- Externalize all hardcoded OAuth secrets to `config.yaml`.
- Implement a simple "Allowlist" for the Management API's `APICall` to prevent SSRF, even in a private network (defense in depth).

## Implementation Priority

1.  **Refactor HTTP Client** (Highest Impact/Effort ratio).
2.  **Refactor `main.go`** (Makes future changes easier).
3.  **Agentic Trace Logging** (High value for debugging "smart" agents).
4.  **Buffer Pools** (Micro-optimization, do last).