nlproxy/docs/overview.md

NLProxy Documentation Overview

This document provides a high-level architecture map for the nlproxy project and points to package-level documentation modules.

Purpose

NLProxy is an enterprise-grade prompt compression and LLM proxy SDK. It is built to reduce prompt cost, enforce prompt safety, and orchestrate multiple LLM providers with caching, firewall controls, and production-ready FastAPI deployment.

Architecture

The repository is organized into the following logical modules:

• core/
  • Implements the compression pipeline, safety checks, and prompt reshaping logic.
• llm/
  • Orchestrates multi-provider LLM interaction with retry, rate limiting, circuit breaking, and fallback.
• firewall/
  • Detects prompt injection and jailbreak attacks using regex and semantic matching.
• cache/
  • Provides Redis-based semantic caching for repeated prompt responses.
• service/
  • Coordinates full prompt compression workflow and thread pool execution.
• server/
  • Contains the FastAPI application, API routes, dependency initialization, and lifecycle management.
• cli/
  • Includes CLI commands for starting the server, compressing prompts, downloading models, and running tests.
• utils/
  • Shares constants and logging configuration across the project.

Documentation Modules

Each package has dedicated reference documentation in the docs/ directory:

• docs/cache.md
• docs/cli.md
• docs/core.md
• docs/firewall.md
• docs/llm.md
• docs/server.md
• docs/service.md
• docs/utils.md
• docs/tests.md

Deployment Notes

• The FastAPI application is created in server/main.py and exposed as app.
• Configuration is loaded from environment variables with the NLPROXY_ prefix in server/config.py.
• The CLI module includes runserver, download_models, and compress commands for operational control.
• Local model files must be provisioned in nlproxy/models/ or by running the model downloader.

Runtime Flow

1. Server startup triggers server/dependencies.startup().
2. The compression pipeline is initialized in service/compression.py.
3. API requests are handled by server/apis/chat.py.
4. Prompts pass through firewall analysis, compression, LLM generation, and post-LLM verification.
5. Observability data is exposed on /metrics and /health paths.

Conventions

• Configuration values are strongly typed using Pydantic models.
• The service avoids automatic external model downloads at runtime; the downloader CLI is used instead.
• Shared constants are centralized under utils/constants.py.
• Caching and external services use connection pooling and predictable timeout settings.