CODEBASE_GUIDE.md

Project Codebase Guide

This document serves as a guide and index for the project codebase, designed to help developers and AI agents quickly understand its structure, components, and how to contribute.

Table of Contents

1. Project Overview
2. Directory Structure
3. Core Components
   • Configuration (src/config.ts)
   • Server Entry Point (src/server.ts)
   • Proxy Layer (src/proxy/)
   • User Management (src/user/)
   • Admin Interface (src/admin/)
   • Shared Utilities (src/shared/)
4. Proxy Functionality
   • Routing (src/proxy/routes.ts)
   • Supported Models & Providers
   • Middleware (src/proxy/middleware/)
   • Adding New Models
   • Adding New APIs/Providers
5. Model Management
   • Model Family Definitions
   • Adding OpenAI Models
   • Model Mapping & Routing
   • Service Information
   • Step-by-Step Guide for Adding a New Model
   • Model Patterns and Versioning
   • Response Format Handling
6. Key Management
   • Key Pool System
   • Provider-Specific Key Management
   • Key Rotation and Health Checks
7. Data Management
   • Database (src/shared/database/)
   • File Storage (src/shared/file-storage/)
8. Authentication & Authorization
9. Logging & Monitoring
10. Deployment
11. Contributing

Project Overview

This project provides a proxy layer for various Large Language Models (LLMs) and potentially other AI APIs. It aims to offer a unified interface, manage API keys securely, handle rate limiting, usage tracking, and potentially add features like response caching or prompt modification.

Directory Structure

.
├── .env.example          # Example environment variables
├── .gitattributes        # Git attributes
├── .gitignore            # Git ignore rules
├── .husky/               # Git hooks
├── .prettierrc           # Code formatting rules
├── CODEBASE_GUIDE.md     # This file
├── README.md             # Project README
├── data/                 # Data files (e.g., SQLite DB)
├── docker/               # Docker configuration
├── docs/                 # Documentation files
├── http-client.env.json  # HTTP client environment
├── package-lock.json     # NPM lock file
├── package.json          # Project dependencies and scripts
├── patches/              # Patches for dependencies
├── public/               # Static assets served by the web server
├── render.yaml           # Render deployment configuration
├── scripts/              # Utility scripts
├── src/                  # Source code
│   ├── admin/            # Admin interface logic
│   ├── config.ts         # Application configuration
│   ├── info-page.ts      # Logic for the info page
│   ├── logger.ts         # Logging setup
│   ├── proxy/            # Core proxy logic for different providers
│   ├── server.ts         # Express server setup and main entry point
│   ├── service-info.ts   # Service information logic
│   ├── shared/           # Shared utilities, types, and modules
│   └── user/             # User management logic
├── tsconfig.json         # TypeScript configuration

Core Components

Configuration (src/config.ts)

• Loads environment variables and defines application settings.
• Contains configuration for database connections, API keys (placeholders/retrieval methods), logging levels, rate limits, etc.
• Uses dotenv and potentially a schema validation library (like Zod) to ensure required variables are present.

Server Entry Point (src/server.ts)

• Initializes the Express application.
• Sets up core middleware (e.g., body parsing, CORS, logging).
• Mounts routers for different parts of the application (admin, user, proxy).
• Starts the HTTP server.

Proxy Layer (src/proxy/)

• The heart of the application, handling requests to downstream AI APIs.
• Contains individual modules for each supported provider (e.g., openai.ts, anthropic.ts).
• Handles request transformation, authentication against the target API, and response handling.
• Uses middleware for common proxy tasks.

User Management (src/user/)

• Handles user registration, login, session management, and potentially API key generation/management for end-users.
• Likely interacts with the database (src/shared/database/).

Admin Interface (src/admin/)

• Provides an interface for administrators to manage users, monitor usage, configure settings, etc.
• May have its own set of routes and views.

Shared Utilities (src/shared/)

• Contains reusable code across different modules.
  • api-schemas/: Zod schemas for API request/response validation.
  • database/: Database connection, schemas (e.g., Prisma), and query logic.
  • errors.ts: Custom error classes.
  • key-management/: Logic for managing API keys (if applicable).
  • models.ts: Core data models/types used throughout the application.
  • prompt-logging/: Logic for logging prompts and responses.
  • tokenization/: Utilities for counting tokens.
  • utils.ts: General utility functions.

Proxy Functionality

Proxy Routing (src/proxy/routes.ts)

• Defines the API endpoints for the proxy service (e.g., /v1/chat/completions).
• Maps incoming requests to the appropriate provider-specific handler based on the request path, headers, or body content (e.g., model requested).
• Applies relevant middleware (authentication, rate limiting, queuing, etc.).

Supported Models & Providers

• OpenAI: Handled in src/proxy/openai.ts. Supports models like GPT-4, GPT-3.5-turbo, as well as o-series models (o1, o1-mini, o1-pro, o3, o3-mini, o3-pro, o4-mini). Handles chat completions and potentially image generation (src/proxy/openai-image.ts).
• Anthropic: Handled in src/proxy/anthropic.ts. Supports Claude models. May use AWS Bedrock (src/proxy/aws-claude.ts) or Anthropic's direct API.
• Google AI / Vertex AI: Handled in src/proxy/google-ai.ts and src/proxy/gcp.ts. Supports Gemini models (gemini-flash, gemini-pro, gemini-ultra).
• Mistral AI: Handled in src/proxy/mistral-ai.ts. Supports Mistral models via their API or potentially AWS (src/proxy/aws-mistral.ts).
• Azure OpenAI: Handled in src/proxy/azure.ts. Provides an alternative endpoint for OpenAI models via Azure.
• Deepseek: Handled in src/proxy/deepseek.ts.
• Xai: Handled in src/proxy/xai.ts.
• AWS (General): src/proxy/aws.ts might contain shared AWS logic (e.g., authentication).

Middleware (src/proxy/middleware/)

• gatekeeper.ts: Likely handles initial request validation, authentication, and authorization checks before hitting provider logic. Checks origin (check-origin.ts), potentially custom tokens (check-risu-token.ts).
• rate-limit.ts: Implements rate limiting logic, potentially per-user or per-key.
• queue.ts: Manages request queuing, possibly to handle concurrency limits or prioritize requests.

Adding New Models

1. Identify the Provider: Determine if the new model belongs to an existing provider (e.g., a new OpenAI model) or a new one.
2. Update Provider Logic (if existing):
   • Modify the relevant provider file (e.g., src/proxy/openai.ts).
   • Update model lists or logic that selects/validates models.
   • Adjust any request/response transformations if the new model has a different API schema.
   • Update model information in shared files like src/shared/models.ts if necessary.
3. Update Routing (if necessary): Modify src/proxy/routes.ts if the new model requires a different endpoint or routing logic.
4. Configuration: Add any new API keys or configuration parameters to .env.example and src/config.ts.
5. Testing: Add unit or integration tests for the new model.

Adding New APIs/Providers

1. Create Provider Module: Create a new file in src/proxy/ (e.g., src/proxy/new-provider.ts).
2. Implement Handler:
   • Write the core logic to handle requests for this provider. This typically involves:
     • Receiving the standardized request from the router.
     • Transforming the request into the format expected by the new provider's API.
     • Authenticating with the new provider's API (fetching keys from config).
     • Making the API call (consider using a robust HTTP client like axios or node-fetch).
     • Handling streaming responses if applicable (using helpers from src/shared/streaming.ts).
     • Transforming the provider's response back into a standardized format.
     • Handling errors gracefully.
3. Add Routing:
   • Import the new handler in src/proxy/routes.ts.
   • Add new routes or modify existing routing logic to direct requests to the new handler based on model name, path, or other criteria.
   • Apply necessary middleware (gatekeeper, rate limiter, queue).
4. Create Key Management:
   • Create a new directory in src/shared/key-management/ for the provider.
   • Implement provider-specific key management (key checkers, token counters).
5. Configuration:
   • Add configuration variables (API keys, base URLs) to .env.example and src/config.ts.
   • Update src/config.ts to load and validate the new variables.
6. Model Information: Add details about the new provider and its models to src/shared/models.ts or similar shared locations.
7. Tokenization (if applicable): If token counting is needed, add or update tokenization logic in src/shared/tokenization/.
8. Testing: Implement thorough tests for the new provider integration.
9. Documentation: Update this guide and any other relevant documentation.

Model Management

Model Family Definitions

• Model Family Definitions: The project uses a family-based approach to group similar models together. These are defined in src/shared/models.ts.
• Each model is part of a model family (e.g., "gpt4", "claude", "gemini-pro") which helps with routing, key management, and feature support.
• The MODEL_FAMILIES array contains all supported model families, and the MODEL_FAMILY_SERVICE mapping connects each family to its provider service.

Adding OpenAI Models

When adding new OpenAI models to the codebase, there are several files that must be updated:

1. Update Model Types (src/shared/models.ts):

   • Add the new model to the OpenAIModelFamily type
   • Add the model to the MODEL_FAMILIES array
   • Add the Azure variants for the model if applicable
   • Add the model to MODEL_FAMILY_SERVICE mapping
   • Update OPENAI_MODEL_FAMILY_MAP with regex patterns to match the model names

2. Update Context Size Limits (src/proxy/middleware/request/preprocessors/validate-context-size.ts):

   • Add regex matching for the new model
   • Set the appropriate context token limit for the model

3. Update Token Cost Tracking (src/shared/stats.ts):

   • Add pricing information for the new model in the getTokenCostUsd function
   • Include both input and output prices in the comments for clarity

4. Update Feature Support Checks (src/proxy/openai.ts):

   • If the model supports special features like the reasoning API parameter (isO1Model function), update the appropriate function
   • For model feature detection, prefer using regex patterns over explicit lists when possible, as this handles date-stamped versions better

5. Update Display Names (src/info-page.ts):

   • Add friendly display names for the new models in the MODEL_FAMILY_FRIENDLY_NAME object

6. Update Key Management Provider Files:

   • For OpenAI keys in src/shared/key-management/openai/provider.ts, add token counters for the new models
   • For Azure OpenAI keys in src/shared/key-management/azure/provider.ts, add token counters for the Azure versions

Model Patterns and Versioning

The codebase handles several patterns for model naming and versioning:

1. Date-stamped Models: Many models include date stamps (e.g., gpt-4-0125-preview). The regex patterns in OPENAI_MODEL_FAMILY_MAP account for these with patterns like ^gpt-4o(-\\d{4}-\\d{2}-\\d{2})?$.

2. O-Series Models: OpenAI's o-series models (o1, o1-mini, o1-pro, o3, o3-mini, o3-pro, o4-mini) follow a different naming convention. The codebase handles these with dedicated model families and regex patterns.

3. Preview/Non-Preview Variants: Some models have preview variants (e.g., gpt-4.5-preview). The regex patterns in OPENAI_MODEL_FAMILY_MAP account for these with patterns like ^gpt-4\\.5(-preview)?(-\\d{4}-\\d{2}-\\d{2})?$.

When adding new models, try to follow the existing patterns for consistency.

Response Format Handling

The codebase includes special handling for different API response formats:

1. Chat vs. Text Completions: There's transformation logic in openai.ts to convert between chat completions and text completions formats (transformTurboInstructResponse).

2. Newer API Formats: For newer APIs like the Responses API, there's transformation logic (transformResponsesApiResponse) to convert responses to a format compatible with existing clients.

When adding support for new models or APIs, consider whether transformation is needed to maintain compatibility with existing clients.

Key Management

Key Pool System

The project uses a sophisticated key pool system (src/shared/key-management/key-pool.ts) to manage API keys for different providers. Key features include:

• Key Selection: The system selects the appropriate key based on model family, region preferences, and other criteria.
• Rotation: Keys are rotated to distribute usage and avoid hitting rate limits.
• Health Checks: Keys are checked periodically to ensure they're still valid and within rate limits.

Provider-Specific Key Management

Each provider has its own key management module in src/shared/key-management/:

• Key Checkers: Each provider implements key checkers to validate keys and check their status.
• Token Counters: Providers implement token counting logic specific to their pricing model.
• Models Support: Keys are associated with specific model families they support.

When adding a new model or provider, you'll need to update or create the appropriate key management files.

Key Rotation and Health Checks

The key pool system includes logic for:

• Rotation Strategy: Keys are selected based on a prioritization strategy (prioritize-keys.ts).
• Disabling Unhealthy Keys: Keys that fail health checks are temporarily disabled.
• Rate Limit Awareness: The system tracks usage to avoid hitting provider rate limits.

Data Management

Database (src/shared/database/)

• Likely uses Prisma or a similar ORM.
• Defines database schemas (e.g., for users, API keys, usage logs).
• Provides functions for interacting with the database.
• Configuration is managed in src/config.ts.

File Storage (src/shared/file-storage/)

• May be used for storing logs, cached data, or user-uploaded files.
• Could integrate with local storage or cloud providers (e.g., S3, GCS).

Authentication & Authorization

• User Auth: Handled in src/user/ potentially using sessions (src/shared/with-session.ts) or JWTs.
• Proxy Auth: The gatekeeper.ts middleware likely verifies incoming requests to the proxy endpoints. This could involve checking:
  • Custom API keys stored in the database (src/shared/database/).
  • Specific tokens (check-risu-token.ts).
  • HMAC signatures (src/shared/hmac-signing.ts).
  • Origin checks (check-origin.ts).
• Downstream Auth: Each provider module (src/proxy/*.ts) handles authentication with the actual AI service API using keys from the configuration.

Logging & Monitoring

• Logging: Configured in src/logger.ts, likely using a library like pino or winston. Logs requests, errors, and important events.
• Prompt Logging: Specific logic for logging prompts and responses might exist in src/shared/prompt-logging/.
• Stats/Monitoring: src/shared/stats.ts might handle collecting and exposing application metrics.

Deployment

• Docker: The project likely includes Docker configuration for containerized deployment.
• Render: The render.yaml file suggests the project is or can be deployed on Render.
• Environment Variables: The .env.example file provides a template for required environment variables in production.

Contributing

When contributing to this project:

1. Follow Coding Standards: Use the established patterns and standards in the codebase. The .prettierrc file defines code formatting rules.
2. Update Documentation: Keep this guide updated when adding new components or changing existing ones.
3. Add Tests: Ensure your changes are tested appropriately.
4. Update Configuration: If your changes require new environment variables, update .env.example.

This guide provides a high-level overview. For detailed information, refer to the specific source code files.