README.md

metadata

title: Unified AI Proxy
emoji: 🔀
colorFrom: blue
colorTo: indigo
sdk: docker
pinned: false

🔀 Unified AI Proxy

A self-hosted, OpenAI-compatible API gateway that routes requests to any AI provider from a single endpoint. Manage all your provider keys through a clean dashboard and hit one URL for everything.

---

Features

• Single endpoint for OpenAI, Anthropic, Groq, Together, Mistral, Cohere, DeepSeek, OpenRouter, or any OpenAI-compatible API
• Provider routing via providerName/model-name prefix in the model field
• Streaming support — SSE responses forwarded without buffering
• Unified model list — GET /v1/models aggregates all providers, prefixed by name
• Dashboard — add/edit/delete providers, view request logs, manage your master key
• Persistent SQLite — data survives Space restarts (mount /data as persistent storage)
• Env var seeding — pre-seed providers via PROVIDER_<NAME>_URL + PROVIDER_<NAME>_KEY

---

Quick Start

1. Deploy to Hugging Face Spaces

1. Create a new Space → SDK: Docker
2. Upload these files: Dockerfile, main.py, requirements.txt, README.md
3. Enable Persistent Storage in Space settings → mount at /data
4. The Space starts and prints your Master API Key to the build logs — copy it

2. Open the Dashboard

Navigate to your Space URL. Enter your master key to unlock the dashboard.

3. Add Providers

In the Providers tab, add any OpenAI-compatible backend:

Provider	Base URL
OpenAI	https://api.openai.com/v1
Anthropic	https://api.anthropic.com/v1
Groq	https://api.groq.com/openai/v1
Together	https://api.together.xyz/v1
Mistral	https://api.mistral.ai/v1
Cohere	https://api.cohere.ai/compatibility/v1
DeepSeek	https://api.deepseek.com/v1
OpenRouter	https://openrouter.ai/api/v1

4. Make Requests

Use your Space URL as the base URL in any OpenAI-compatible client:

# Route to a specific provider using providerName/ prefix
curl https://your-space.hf.space/v1/chat/completions \
  -H "Authorization: Bearer YOUR_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "groq/llama3-8b-8192",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

# Together AI with a nested model path
curl https://your-space.hf.space/v1/chat/completions \
  -H "Authorization: Bearer YOUR_MASTER_KEY" \
  -d '{
    "model": "together/meta-llama/Llama-3-70b-chat-hf",
    "messages": [{"role": "user", "content": "Hi"}],
    "stream": true
  }'

# Use default provider (no prefix needed)
curl https://your-space.hf.space/v1/chat/completions \
  -H "Authorization: Bearer YOUR_MASTER_KEY" \
  -d '{"model": "gpt-4o", "messages": [{"role": "user", "content": "Hello"}]}'

# List all models across all providers
curl https://your-space.hf.space/v1/models \
  -H "Authorization: Bearer YOUR_MASTER_KEY"

Use with the OpenAI Python SDK

from openai import OpenAI

client = OpenAI(
    base_url="https://your-space.hf.space/v1",
    api_key="YOUR_MASTER_KEY",
)

# Route to Groq
response = client.chat.completions.create(
    model="groq/llama3-8b-8192",
    messages=[{"role": "user", "content": "Hello!"}],
)

# Route to Anthropic (via OpenAI-compat layer)
response = client.chat.completions.create(
    model="anthropic/claude-3-5-sonnet-20241022",
    messages=[{"role": "user", "content": "Hello!"}],
)

---

Environment Variable Seeding

Pre-seed providers without touching the dashboard by setting Space secrets:

PROVIDER_OPENAI_URL=https://api.openai.com/v1
PROVIDER_OPENAI_KEY=sk-...

PROVIDER_GROQ_URL=https://api.groq.com/openai/v1
PROVIDER_GROQ_KEY=gsk_...

PROVIDER_TOGETHER_URL=https://api.together.xyz/v1
PROVIDER_TOGETHER_KEY=...

These are imported on startup if the provider name doesn't already exist in the database.

---

API Reference

Route	Auth	Description
GET /	Master key (dashboard login)	Web dashboard
GET /health	None	Health check + provider count
GET /v1/models	Master key	Aggregated model list from all providers
POST /v1/chat/completions	Master key	Proxied chat completions
POST /v1/completions	Master key	Proxied text completions
POST /v1/embeddings	Master key	Proxied embeddings
GET /v1/*	Master key	Any other OpenAI-compatible endpoint

---

Model Routing

The proxy determines which provider to use based on the model field:

1. With prefix — groq/llama3-8b-8192 → routes to the provider named Groq, forwards model as llama3-8b-8192
2. Nested paths — together/meta-llama/Llama-3-70b → routes to Together, forwards meta-llama/Llama-3-70b
3. No prefix — uses the provider marked as Default in Settings
4. No default set — uses the first enabled provider

Matching is case-insensitive: GROQ/, groq/, and Groq/ all route to a provider named Groq.

---

Dashboard Tabs

Providers

Add, edit, enable/disable, and delete providers. Set one as the default. API keys are stored encrypted in SQLite and only the last 4 characters are ever displayed.

Logs

Auto-refreshing table (every 5s) of the last 200 requests: timestamp, provider, model, endpoint, status code, and latency. Automatically truncated at 500 stored rows.

Settings

View and copy your master key, regenerate it (invalidates immediately), and set the default provider.

---

Security Notes

• Provider API keys are never returned by any API endpoint — only the last 4 characters are shown in the dashboard
• The master key protects all /v1/* endpoints and the dashboard
• CORS is open (*) so the proxy can be called from any frontend — restrict this in production if needed
• The dashboard login stores the master key in sessionStorage (cleared on tab close)

---

Local Development

pip install fastapi uvicorn httpx sqlalchemy
uvicorn main:app --reload --port 7860
# Open http://localhost:7860
# Master key printed to terminal on first run

Data is stored in ./db.sqlite when /data doesn't exist.