README.md

metadata

title: LiteLLM Gateway
emoji: 🛡️
colorFrom: blue
colorTo: green
sdk: docker
app_port: 8000
pinned: false

🛡️ LiteLLM Gateway Console — Production-Ready AI Proxy

An enterprise-grade AI gateway with intelligent LLM routing, real-time PII shielding, and a built-in Streamlit dashboard — accessible from a single URL in any browser, no local setup required.

Built with FastAPI, LiteLLM, DeBERTa-v3, and Streamlit, deployed as a single Docker container.

---

☁️ One-Click Deploy

Deploy to your preferred cloud platform by clicking a button below. Set your API key secrets in the platform dashboard after deployment.

Railway (Recommended)

After deploy: Go to Variables → add GROQ_API_KEY, CEREBRAS_API_KEY, TOGETHERAI_API_KEY

Render

After deploy: Go to Environment → add your API keys as secret variables

Fly.io

# Install Fly CLI, then:
fly launch --no-deploy
fly secrets set GROQ_API_KEY=gsk_... CEREBRAS_API_KEY=csk_... TOGETHERAI_API_KEY=...
fly deploy

---

🔑 Required Environment Variables

Variable	Description	Required
GROQ_API_KEY	Groq API key (console.groq.com)	✅
CEREBRAS_API_KEY	Cerebras API key (cloud.cerebras.ai)	✅
TOGETHERAI_API_KEY	Together AI key (api.together.ai)	✅
MISTRAL_API_KEY	Mistral API key (optional)	⬜
PORT	HTTP port (auto-set by cloud platform)	auto
OLLAMA_API_BASE	Local Ollama URL (local only)	⬜

See .env.example for a full template.

---

✨ Key Features

Feature	Details
🔀 Intelligent Routing	usage-based-routing — automatically balances load across Groq, Cerebras, Together AI
🛡️ PII Shield (DeBERTa-v3)	Real-time detection of 10+ PII entity types with BLOCK / MASK / REWRITE controls per entity
📊 Live Dashboard	Streamlit UI served at / — no separate port, works on any cloud
⚡ OpenAI-Compatible API	Drop-in replacement for any OpenAI SDK client
🔁 Auto Fallbacks	primary-cluster → backup-cluster with context-window escalation
🏗️ Single Container	One Docker image, one port, one URL

---

🏗️ Architecture

Browser → https://myapp.com/           (single public URL)
              │
              ▼
     ┌─────────────────────┐
     │   FastAPI Gateway   │  port $PORT (e.g. 8000)
     │                     │
     │  ┌───────────────┐  │  HTTP + WebSocket reverse proxy
     │  │  Streamlit UI │◄─┼─── / (root) and /_stcore/* paths
     │  │  (port 8501)  │  │
     │  └───────────────┘  │
     │                     │
     │  ┌───────────────┐  │
     │  │  LLM Router   │◄─┼─── /v1/chat/completions
     │  │  DeBERTa PII  │  │
     │  └───────────────┘  │
     └─────────────────────┘
              │
   ┌──────────┼──────────┐
   ▼          ▼          ▼
 Groq     Cerebras   Together AI

---

📁 Directory Structure

LiteLLM_Proxy/
├── Dockerfile           # Multi-stage build (model pre-cache, slim runtime)
├── .dockerignore        # Keeps secrets out of the image
├── railway.toml         # Railway deployment config
├── render.yaml          # Render.com deployment blueprint
├── fly.toml             # Fly.io app configuration
├── Procfile             # Heroku/Railway fallback entrypoint
├── .env.example         # Environment variable template (safe to commit)
├── config.yaml          # LiteLLM routing, capacity, and fallback settings
├── requirements.txt     # Python production dependencies
├── main.py              # Unified entrypoint: spawns Streamlit + starts FastAPI
├── app.py               # Streamlit dashboard frontend
├── test_proxy.py        # Automated unit & integration test suite
└── proxy/               # FastAPI package
    ├── app.py           # Routes, Streamlit reverse proxy, PII config endpoints
    ├── router.py        # Token estimation, routing, DeBERTa PII guardrail
    ├── config.py        # Pydantic config models
    └── guardrails/      # DeBERTa-v3 PII detection engine

---

🖥️ Local Development

1. Clone & Install

git clone https://github.com/sanvity/LiteLLM_Proxy.git
cd LiteLLM_Proxy

python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

2. Configure API Keys

cp .env.example .env
# Edit .env and add your API keys

3. Start the Application

python main.py

The app will:

1. Spawn the Streamlit dashboard on internal port 8501
2. Wait for Streamlit to be ready
3. Start FastAPI on port 8000

Then visit http://localhost:8000 — the full UI loads in your browser.

---

🐳 Local Docker Build

docker build -t litellm-gateway .
docker run -p 8000:8000 --env-file .env litellm-gateway

# Visit http://localhost:8000

---

🧪 Running Tests

python -m unittest test_proxy.py

---

📡 API Reference

Endpoint	Method	Description
/	GET	Streamlit dashboard (proxied)
/health	GET	Liveness probe — returns {"status": "healthy"}
/metrics	GET	Real-time routing and token metrics
/v1/models	GET	List virtual and physical models
/v1/chat/completions	POST	OpenAI-compatible chat completion
/ui/pii-config	GET/POST	Read/update PII guardrail configuration
/preference-config	GET/POST	Read/update model preference routing
/old-ui	GET	Legacy HTML dashboard (debugging)

Example Request

curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "primary-cluster",
    "messages": [{"role": "user", "content": "Hello!"}],
    "temperature": 0.7,
    "max_tokens": 500
  }'