docs/scaling_reference.md

Scaling Reference

Purpose: What to know and what to change when traffic grows. Written at V1 (sync, low-traffic SME demo). Read this before making any scaling decision.

---

Current Architecture (V1 Baseline)

Layer	Technology	Limit
Web framework	FastAPI (sync, thread pool)	~200–500 concurrent requests before thread exhaustion
DB driver	psycopg2 (sync)	Fine up to ~100 concurrent DB queries
DB	PostgreSQL (single instance)	Fine up to ~500–1000 connections with PgBouncer
LLM	Groq (sync HTTP call, 1–3s)	Real bottleneck — async won't help here
Vector DB	ChromaDB (in-process)	Fine for low-volume; becomes a bottleneck at scale
Deployment	Single Docker container	No horizontal scaling

---

The Real Bottleneck: LLM Latency

Every /chat request waits 1–3 seconds for the LLM response. At V1 traffic (5–50 concurrent users), this is fine. At scale:

• Async DB saves ~5ms per request
• LLM call costs 1,000–3,000ms per request

Conclusion: Optimizing the DB layer before solving LLM latency is premature. Address LLM first.

---

Scale Thresholds and What to Do

< 100 concurrent users — Do nothing

Current sync architecture handles this comfortably. Focus on product.

100–500 concurrent users

1. Add PgBouncer — connection pooling in front of PostgreSQL. Prevents DB connection exhaustion. Config change only, no code change.
2. Add Redis caching — cache system prompts and KB entries per tenant. Eliminates redundant DB reads on every /chat call.
3. Horizontal scale — run 2–3 replicas behind a load balancer (Nginx/Traefik). Stateless FastAPI app supports this with zero code changes.

500–5,000 concurrent users

4. LLM request queue — move LLM calls to a background worker (Celery + Redis). /chat returns a job ID immediately; client polls for the result. Eliminates thread exhaustion from long-running LLM calls.
5. ChromaDB → managed vector DB — replace in-process ChromaDB with Qdrant or Weaviate (dedicated service). In-process ChromaDB doesn't scale horizontally.
6. Read replicas — add PostgreSQL read replicas for conversation history and KB reads.

> 5,000 concurrent users

7. Async migration — switch to asyncpg driver + async SQLAlchemy + async FastAPI. Requires rewriting all get_db dependencies, session management, and test fixtures. High cost — only justified at this scale.
8. Kubernetes — horizontal pod autoscaling, per-tenant rate limiting at ingress, proper resource limits.
9. LLM provider strategy — provider failover, per-model rate limits, tenant-level concurrency caps.

---

Async Migration Path (when justified)

When async is finally needed, the migration order is:

1. Change DATABASE_URL scheme: postgresql:// → postgresql+asyncpg://
2. Replace SessionLocal with AsyncSession in database.py
3. Replace get_db dependency with async generator
4. Change all router functions from def to async def
5. Replace all db.scalars(...) with await db.scalars(...)
6. Rewrite all test fixtures in conftest.py (async session, async client)
7. Run full test suite — expect failures in every file that touches the DB

Cost: ~2–3 days of pure refactoring with no new features. Only do this when load data proves it's needed.

---

One-Way Doors to Avoid Before Scale

These decisions become very expensive to undo at scale — make them correctly from the start:

Decision	Risk if wrong
No credit_events table	Can't reconstruct billing history — dispute with no evidence
No tenant isolation in ChromaDB	Data leakage between tenants — legal liability
No API versioning (/api/v1/)	Can never change request/response contract without breaking all clients
No llm_models versioned rates	Can't prove which rate applied to a historical charge

---

Summary

Sync is fine for V1 and beyond. The LLM call is the bottleneck, not the DB driver. Add PgBouncer + Redis + horizontal replicas before ever touching async. Migrate to async only when data shows you need it — and budget 2–3 days for the rewrite.