docs/scaling_reference.md

# Scaling Reference

**Purpose:** What to know and what to change when traffic grows. Written at V1 (sync, low-traffic SME demo). Read this before making any scaling decision.

---

## Current Architecture (V1 Baseline)

| Layer         | Technology                   | Limit                                                 |
| ------------- | ---------------------------- | ----------------------------------------------------- |
| Web framework | FastAPI (sync, thread pool)  | ~200–500 concurrent requests before thread exhaustion |
| DB driver     | psycopg2 (sync)              | Fine up to ~100 concurrent DB queries                 |
| DB            | PostgreSQL (single instance) | Fine up to ~500–1000 connections with PgBouncer       |
| LLM           | Groq (sync HTTP call, 1–3s)  | **Real bottleneck — async won't help here**           |
| Vector DB     | ChromaDB (in-process)        | Fine for low-volume; becomes a bottleneck at scale    |
| Deployment    | Single Docker container      | No horizontal scaling                                 |

---

## The Real Bottleneck: LLM Latency

Every `/chat` request waits 1–3 seconds for the LLM response. At V1 traffic (5–50 concurrent users), this is fine. At scale:

- Async DB saves ~5ms per request
- LLM call costs 1,000–3,000ms per request

**Conclusion:** Optimizing the DB layer before solving LLM latency is premature. Address LLM first.

---

## Scale Thresholds and What to Do

### < 100 concurrent users — Do nothing

Current sync architecture handles this comfortably. Focus on product.

### 100–500 concurrent users

1. **Add PgBouncer** — connection pooling in front of PostgreSQL. Prevents DB connection exhaustion. Config change only, no code change.
2. **Add Redis caching** — cache system prompts and KB entries per tenant. Eliminates redundant DB reads on every `/chat` call.
3. **Horizontal scale** — run 2–3 replicas behind a load balancer (Nginx/Traefik). Stateless FastAPI app supports this with zero code changes.

### 500–5,000 concurrent users

4. **LLM request queue** — move LLM calls to a background worker (Celery + Redis). `/chat` returns a job ID immediately; client polls for the result. Eliminates thread exhaustion from long-running LLM calls.
5. **ChromaDB → managed vector DB** — replace in-process ChromaDB with Qdrant or Weaviate (dedicated service). In-process ChromaDB doesn't scale horizontally.
6. **Read replicas** — add PostgreSQL read replicas for conversation history and KB reads.

### > 5,000 concurrent users

7. **Async migration** — switch to `asyncpg` driver + async SQLAlchemy + async FastAPI. Requires rewriting all `get_db` dependencies, session management, and test fixtures. High cost — only justified at this scale.
8. **Kubernetes** — horizontal pod autoscaling, per-tenant rate limiting at ingress, proper resource limits.
9. **LLM provider strategy** — provider failover, per-model rate limits, tenant-level concurrency caps.

---

## Async Migration Path (when justified)

When async is finally needed, the migration order is:

1. Change `DATABASE_URL` scheme: `postgresql://` → `postgresql+asyncpg://`
2. Replace `SessionLocal` with `AsyncSession` in `database.py`
3. Replace `get_db` dependency with async generator
4. Change all router functions from `def` to `async def`
5. Replace all `db.scalars(...)` with `await db.scalars(...)`
6. Rewrite all test fixtures in `conftest.py` (async session, async client)
7. Run full test suite — expect failures in every file that touches the DB

**Cost:** ~2–3 days of pure refactoring with no new features. Only do this when load data proves it's needed.

---

## One-Way Doors to Avoid Before Scale

These decisions become very expensive to undo at scale — make them correctly from the start:

| Decision                        | Risk if wrong                                                           |
| ------------------------------- | ----------------------------------------------------------------------- |
| No `credit_events` table        | Can't reconstruct billing history — dispute with no evidence            |
| No tenant isolation in ChromaDB | Data leakage between tenants — legal liability                          |
| No API versioning (`/api/v1/`)  | Can never change request/response contract without breaking all clients |
| No `llm_models` versioned rates | Can't prove which rate applied to a historical charge                   |

---

## Summary

> Sync is fine for V1 and beyond. The LLM call is the bottleneck, not the DB driver.
> Add PgBouncer + Redis + horizontal replicas before ever touching async.
> Migrate to async only when data shows you need it — and budget 2–3 days for the rewrite.