nlproxy/docs/llm.md

# NLProxy LLM Client Module Reference

This document describes `llm/client.py`.

## Purpose

The LLM client module provides a unified interface for multiple providers, implements request retry and concurrency control, and normalizes provider responses for the NLProxy pipeline.

## Core Concepts

### Provider Abstraction

- `LLMProvider` enumerates supported providers:
  - `gemini`
  - `claude`
  - `openai`
  - `deepseek`
  - `qwen`
  - `kimi`
  - `openrouter`
  - `custom`

### Request Model

- `LLMRequest`
  - `prompt: str`
  - `provider: LLMProvider`
  - `model: str`
  - `max_tokens: int`
  - `temperature: float`
  - `top_p: float`
  - `top_k: int`
  - `stop_sequences: Optional[List[str]]`
  - `metadata: Optional[Dict[str, Any]]`

### Response Model

- `LLMResponse`
  - Contains provider text, token counts, latency, cost, and metadata.

## Provider Infrastructure

### `RetryConfig`

Controls retry behavior:

- `max_attempts`
- `base_delay`
- `max_delay`
- `exponential_base`
- `jitter`
- `retryable_exceptions`

### `TimeoutConfig`

Defines connection and read timeouts for all HTTP-based providers.

### `RateLimitConfig`

Implements token-bucket rate limiting for request pacing.

### `TokenBucket`

Used internally by `BaseLLMClient` to prevent burst overload.

### `CircuitBreaker`

Used to mark unhealthy providers and fail fast when repeated errors occur.

## Base Client

### `BaseLLMClient`

#### Responsibilities

- Manages provider-specific HTTP connections.
- Applies concurrency limits with `asyncio.Semaphore`.
- Applies rate limiting and retry logic.
- Tracks metrics such as request count, errors, and latency.
- Exposes `generate()` and `generate_stream()`.

#### HTTP Client Reuse

- `_ensure_http_client(base_url, headers) -> httpx.AsyncClient`
- Ensures a single `httpx.AsyncClient` instance is reused per provider instance.
- Prevents socket exhaustion and reduces connection setup overhead.

#### Async Safety

- All operations are async and reentrant.
- Concurrency is controlled per provider using semaphores.
- `close()` shuts down the HTTP client cleanly.

## Provider Implementations

### `GeminiClient`

- Uses `google-genai` async client.
- Implements `_generate_internal()` and `_generate_stream_internal()`.
- Uses Gemini-specific token counting where available.

### `ClaudeClient`

- Sends requests to Anthropic Claude.
- Uses `/v1/messages` endpoint.
- Implements stream parsing for `content_block_delta` chunks.

### `OpenAIClient`

- Connects to `https://api.openai.com/v1`.
- Uses `/chat/completions` for standard and streaming responses.

### `GenericAPIClient`

- Supports OpenAI-compatible providers such as DeepSeek, Qwen, Kimi, OpenRouter, and custom targets.
- Requires `base_url` for `CUSTOM` provider.

## Factory and Orchestration

### `LLMClientFactory`

- `create(provider, model, api_key, base_url, retry_config, timeout_config, rate_limit_config, circuit_breaker, max_concurrent_requests)`
- `get_or_create(provider, **kwargs)` returns singleton clients keyed by provider and model.
- `close_all()` cleanly closes all managed provider clients.

### `LLMOrchestrator`

- Balances traffic across default and fallback providers.
- Supports load balancing across provider list.
- Implements provider fallback when the primary provider fails.
- Exposes `generate()`, `generate_stream()`, `health_check_all()`, `get_metrics()`, and `close()`.

## Utilities

- `validate_prompt(prompt, max_length=100000)` sanitizes input text.
- `estimate_cost(input_tokens, output_tokens, provider, model)` calculates estimated cost using pricing data.

## Dependencies

- `httpx`
- `pydantic`
- Optional: `google-genai`, `tiktoken`

## Performance and Scalability

- Request handling is async, suitable for FastAPI.
- Concurrency limits and backoff reduce overload on provider APIs.
- Shared HTTP clients reduce connection churn.
- Token counting is O(L) where L = text length.

## Edge Cases

- Missing provider credentials raise explicit configuration errors.
- Circuit breaker prevents repeated calls to failing providers.
- `generate_stream()` preserves provider streaming semantics while protecting against retries during stream consumption.