docs/OPERATIONS.md

Operations and security

Runtime configuration

All configuration comes from environment variables (.env is a local convenience). ENCRYPTION_KEYS is mandatory in every environment. Production additionally enforces PostgreSQL, Redis, secure cookies and HTTPS allowed origins.

Setting	Default	Meaning
ENVIRONMENT	development	Set production for enforced secure configuration
ENCRYPTION_KEYS	required	Comma-separated Fernet keys; first key encrypts, all decrypt
DATABASE_URL	local SQLite	Async SQLAlchemy URL; PostgreSQL required in production
REDIS_URL	absent	Redis URL; absent is allowed only in development
ALLOWED_ORIGINS	["http://localhost:7860"]	JSON array; exact origins for dashboard mutations
COOKIE_SECURE	false	Must be true with HTTPS production
ALLOW_REGISTRATION	true	Close after onboarding if appropriate
REGISTRATION_CODE	empty	Optional deployment-wide invitation code
SESSION_HOURS	24	Dashboard session lifetime
REQUESTS_PER_MINUTE	60	Per-user gateway request limit across all their keys
MAX_RETRIES	2	Max extra candidate attempts; request can reduce this
UPSTREAM_TIMEOUT_SECONDS	60	Per-attempt upstream timeout; stream phase also bounded
MAX_REQUEST_BYTES	26214400	Entire request body, including multipart overhead
MAX_RESPONSE_BYTES	52428800	Response/stream ceiling
DISCOVERY_INTERVAL_SECONDS	3600	Refresh due model catalogs
HEALTH_INTERVAL_SECONDS	60	Background refresh/maintenance loop
CIRCUIT_COOLDOWN_SECONDS	30	Base failure cooldown; Retry-After can extend it
HIGH_LATENCY_MS	15000	Marks observed model latency degraded
LOG_RETENTION_DAYS	30	Request and audit metadata retention
PRIVATE_UPSTREAM_HOSTS	[]	Exact operator-allowlisted hosts permitted to resolve privately
PORT	7860	Container public port
WEB_CONCURRENCY	1	Uvicorn workers; use shared PostgreSQL and Redis for multiple workers
MAX_CONCURRENCY	32	Uvicorn connection/request concurrency ceiling
TRUSTED_PROXY_IPS	empty	Exact trusted proxy networks; never * on an exposed listener
DATA_DIR	/data	Embedded services data directory
PERSISTENT_STORAGE_CONFIRMED	absent	Required true for embedded production PostgreSQL

Set integer, boolean and JSON-list values as strings in hosted environment settings. Do not put keys in the frontend environment or public build arguments. A rootless image is used; persistent mount directories must be writable by UID 1000.

TLS and reverse proxies

The gateway expects TLS termination at the hosting platform or a trusted reverse proxy. For a VPS use deploy/Caddyfile, configure DNS, and set ALLOWED_ORIGINS and COOKIE_SECURE=true. Compose binds to localhost, appropriate for a reverse proxy on the host. If Caddy is in another container, change network wiring explicitly.

SSE requires unbuffered responses. Caddy’s example sets flush_interval -1. With nginx set proxy_buffering off and a suitable read timeout. Requests can span (MAX_RETRIES + 1) attempts; streaming also has a bounded stream phase. Size your outer proxy timeout accordingly.

Do not trust arbitrary X-Forwarded-For headers. If no trusted proxy configuration is supplied, authentication throttling groups clients under the actual connecting proxy address. Configure only the ingress networks you control. Authentication endpoints have a fixed ten-mutations-per-minute limit per connecting address.

Platform files

• Hugging Face: root Dockerfile and README metadata; see main guide for persistence/networking.
• Railway: use deploy/railway.json as the service config, with external PostgreSQL/Redis URLs and the production secrets. Adapt standard postgresql:// connection URLs to postgresql+asyncpg://.
• Render: deploy/render.yaml is a template; supply secure URLs/secrets and use the Docker runtime. It does not provision billable databases automatically.
• Fly.io: set a unique app name in deploy/fly.toml, supply secrets and external databases, then deploy from the repository root with fly deploy --config deploy/fly.toml.
• VPS: Docker Compose plus HTTPS reverse proxy, backups and restart policy.

These are deployment templates, not claims that accounts were provisioned or deployments completed. Provider credentials and hosting credentials are never embedded in the project.

Secrets and isolation

Provider API keys and custom headers are encrypted together using authenticated Fernet encryption. User passwords use Argon2id. Gateway keys have 256 bits of random entropy and are only stored as SHA-256 hashes plus a short display prefix; database sessions are random hashed opaque tokens. The encryption key stays outside the database.

Every provider, model, key, usage, log and audit query is scoped to the authenticated user, including mutations. A leaked gw_ key can call the inference API for that account but cannot manage providers through the dashboard API. Key revocation is checked against the database on every request.

The dashboard uses HttpOnly SameSite=Lax sessions and a separately supplied CSRF token for unsafe requests. Origin checks also protect login and registration. There are no provider keys, session bearer tokens or Gateway keys saved to browser localStorage. A freshly generated Gateway key exists in frontend memory only until dismissed. TLS and trusted device/browser security remain necessary.

Upstream errors, validation input values, credentials, prompts, completions and audio bodies are not recorded in application logs. Access logs are disabled in the container command. Error responses give a request ID and sanitized code. Registry metadata, custom connection names and user emails are still sensitive account information. Audit metadata is retained for the same configured period as request metadata.

Custom endpoints and network safety

Upstream URLs must be HTTP(S), contain no credentials/query/fragment, and use HTTPS unless an exact host is operator-allowlisted. Hop-by-hop and forwarding headers are rejected. Redirects are never followed and ambient HTTP proxy variables are ignored for provider requests.

At socket connection time, a custom HTTPcore network backend resolves the hostname, rejects private/reserved addresses unless explicitly allowed, and connects directly to a validated IP. HTTPcore retains the original TLS SNI and validates the certificate. Mixed public/private DNS answers are rejected. The integration uses HTTPcore’s internal network backend hook; HTTPx/HTTPcore versions are constrained and transport tests must pass before upgrading them. Existing persistent connections remain pinned to their original destination.

For local Ollama/LM Studio, add only the intended hostname to PRIVATE_UPSTREAM_HOSTS, and use its OpenAI-compatible /v1 URL. Reachability depends on your Docker network; localhost inside the container is not your laptop. The allowlist is an operator environment setting and cannot be changed through the dashboard. Consider an egress firewall as an additional network boundary.

Backups

Back up both the PostgreSQL database and ENCRYPTION_KEYS. Losing encryption keys makes provider credentials unrecoverable; Gateway keys also cannot be reconstructed from stored hashes.

For Compose, a custom-format dump can be made locally:

docker compose exec -T postgres pg_dump -U gateway -d gateway -Fc > gateway.dump

Treat the dump as private even though provider credentials are encrypted. Retain it outside the application host. Redis holds rate/circuit/lease state; it can be rebuilt, but resetting it resets active limits and cooldowns.

For embedded PostgreSQL, execute pg_dump inside the running container against the Unix socket /data/run as database user gateway. Do not copy live PostgreSQL data files without a consistent database backup procedure. The included image does not install an automated off-host backup service.

Restore into a separate database first, set the matching ENCRYPTION_KEYS, run migrations, and verify that you can sign in and decrypt/test one connection. Test your restore workflow before relying on it. Do not overwrite a running production database as an unreviewed recovery step.

Encryption-key rotation

1. Back up database and current keys.
2. Generate a new Fernet key. Configure ENCRYPTION_KEYS=NEW_KEY,OLD_KEY on all instances. New writes use the new key, old ciphertext remains readable.
3. Stop credential edits or use a maintenance window to avoid a rotation/edit race.
4. Run from the project root with the same production environment:

PYTHONPATH=backend python scripts/rotate_keys.py

5. Verify, restart all replicas, and only then remove the old key from the active list. Retain old keys securely with backups that need them.

Gateway keys rotate by issuing a replacement, updating the client and revoking the old key. The plaintext of saved Gateway keys is not recoverable.

Migrations and scaling

Alembic runs before Uvicorn in the container entrypoint. PostgreSQL migration execution is serialized with an advisory lock. The initial migration is frozen source; future changes should be generated/reviewed as new revisions. Development-only table creation is not a substitute for production migration management.

Use external PostgreSQL and Redis for multiple app replicas. Embedded services are one-node deployments and do not provide replication or failover. A fixed Redis discovery lease avoids duplicate refreshes across workers. Rate counters use atomic Redis operations; an unavailable Redis fails request authorization closed instead of disabling limits.

The router reads registry candidates from PostgreSQL per request, with a per-user limit of fifty connections. This is suitable for an initial deployment, not an unbounded high-QPS promise. Large catalogs and concurrency should be profiled before capacity commitments. Bound connections at the edge and configure host resources. There are no paid synthetic health probes or automatic benchmark costs.

Data accuracy and failure semantics

• Missing capability, price, token count or latency is unknown, not false, zero/free, or a successful measurement.
• Manually configured model metadata is preserved across discovery refreshes.
• Successful discovery marks disappeared non-manual models unavailable. Failed discovery retains the previous registry and shows a sanitized error.
• Model catalog visibility is not proof of account entitlement, quotas, inference health or tool support.
• Costs cover reported tokens on the returned attempt when both token prices are known. Failed attempts, media charges, caching discounts, taxes and upstream routing differences can make invoices differ.
• Mid-stream failures retain HTTP 200 because headers have already been committed, emit an SSE error and are recorded as failed in analytics.
• Automatic failover can cause multiple billable provider attempts. There is no idempotency guarantee across providers.
• Responses continuation is upstream-owned: use a connection-qualified model ID and do not assume IDs work across providers.
• The system has no password-reset email, email verification or administrator UI in v1. Use an invitation code/closed registration for controlled deployments. Account recovery requires an operator-controlled process.