Deploy from GitHub Actions (2026-05-31 09:04 UTC)

.dockerignore

@@ -0,0 +1,31 @@
+# Secrets — never bake into image (COPY . . in Dockerfile would include these)
+.env
+.env.*
+
+# Virtualenv — host Python (macOS) is incompatible with container Python (Linux)
+.venv/
+
+# Python bytecode — regenerated automatically, OS-specific
+__pycache__/
+*.pyc
+*.pyo
+*.pyd
+.pytest_cache/
+
+# Git history — app never needs it at runtime, and it can be hundreds of MB
+.git/
+
+# Frontend build tools — node_modules contains esbuild/vite (build-time only)
+# The compiled output (chatbot-widget.js) is already built; these are never needed at runtime
+frontend-widget/node_modules/
+
+# Tests — bind-mounted in dev via docker-compose.dev.yml; not needed in prod image
+tests/
+
+# Project management and documentation — runtime irrelevant
+archive/
+docs/
+current_state/
+scripts/
+.claude/
+*.md

.geminiignore

@@ -0,0 +1,7 @@
+.venv/
+.git/
+node_modules/
+__pycache__/
+dist/
+build/
+*.log

.gitignore

@@ -133,6 +133,7 @@ celerybeat.pid
 .env.staging
 .env.production
 .venv
+.venv-linux
 env/
 venv/
 ENV/
@@ -189,18 +190,22 @@ cython_debug/
 # Generated frontend configuration — tracked so CI/CD (GitHub Actions → HF Spaces) has it
 # frontend/config.js
 
+# ChromaDB persistent storage (runtime data)
+chromadb/
+chroma_data/
+
 # Other Files or Directories
-CLAUDE*.md
-GEMINI*.md
-AGENTS*.md
-PROJECT.md
 .claude/
 .vscode/
+*CLAUDE.md
+*AGENTS.md
+*GEMINI.md
+docker-compose.dev.local.yml
 
 # Everything inside `archive` except `daily_log.md`
 archive/*
 !archive/daily_log.md
-!archive/development_notes/
+!archive/knowledge_assessments/
 
 # Microservice credentials (sensitive)
 services/credentials/*.json
@@ -231,3 +236,6 @@ nginx/logs/
 
 # Node.js (frontend-widget Vite/Preact source project)
 frontend-widget/node_modules/
+
+# Mac's metadata
+**/.DS_Store

DOCKER_SETUP.md

@@ -126,13 +126,76 @@ docker-compose -f docker-compose.dev.yml up -d
 docker-compose -f docker-compose.dev.yml logs -f smart-chatbot
 ```
 
-#### 6. Stop Development Environment
+#### 6. Named Volumes (Dev)
+
+| Volume | Purpose |
+|--------|---------|
+| `chatbot_postgres_dev` | PostgreSQL data — persists DB across restarts |
+| `qdrant_dev_data` | Qdrant vector index data — production-target vector backend, mounted at `/qdrant/storage` |
+| `redis_dev_data` | Redis append-only data for Taskiq WordPress sync jobs |
+
+The production stack uses the same Qdrant storage path with `qdrant_prod_data` and stores Taskiq broker data in `redis_prod_data`. Startup runs the active vector-store readiness probe, so read-only volume or vector-service availability problems fail before tenant KB writes.
+
+Vector backend environment variables:
+
+| Variable | Purpose |
+|----------|---------|
+| `HF_HUB_DISABLE_XET` | Legacy HuggingFace Hub workaround; harmless if left set, no longer required for the Gemini embedding path |
+| `GEMINI_EMBEDDING_API_KEY` | Gemini API key used by `EmbeddingService` for `gemini-embedding-2` requests |
+| `VECTOR_STORE_BACKEND` | `qdrant` for V1 runtime; `chroma` remains available only as legacy fallback code |
+| `QDRANT_URL` | Qdrant HTTP URL from app/worker containers, usually `http://qdrant:6333` |
+| `QDRANT_COLLECTION_NAME` | Shared Qdrant collection for tenant KB vectors |
+| `QDRANT_VECTOR_SIZE` | Embedding dimension; `3072` for `gemini-embedding-2` |
+| `QDRANT_TIMEOUT` | Qdrant client timeout in seconds |
+| `QDRANT_CONTAINER_NAME` | Docker container name for the Qdrant service |
+
+Chat billing / completion envelope variables:
+
+| Variable | Purpose |
+|----------|---------|
+| `CHAT_MODEL_PRIMARY` | Primary routed chat model alias target used by LiteLLM |
+| `CHAT_MODEL_FALLBACKS` | Ordered fallback chat models used when the primary fails |
+| `CHAT_MAX_OUTPUT_TOKENS` | Hard cap passed to chat completions and used by the low-balance preflight reserve check |
+
+Rebuild one tenant's Qdrant index from PostgreSQL:
+
+```bash
+docker compose --env-file .env -f docker-compose.dev.yml exec smart-chatbot \
+  uv run python scripts/rebuild_tenant_index.py <tenant_uuid>
+```
+
+**Running tests — host vs Docker:**
+
+`uv run pytest` on the host will produce 5 failures:
+- `test_api_versioning.py::test_versioned_url_works`
+- `test_input_token_limit.py::test_input_token_limit`
+- `test_chat_billing_wiring.py::test_deducts_credits_on_credits_billing_mode`
+- `test_chat_billing_wiring.py::test_returns_402_on_insufficient_credits`
+- `test_chat_billing_wiring.py::test_skips_deduction_on_subscription_billing_mode`
+
+Root cause: these tests call `retrieval_service.retrieve()` which connects to `QDRANT_URL=http://qdrant:6333`. That hostname only resolves inside the Docker network. No ports are exposed to the host, so the connection fails locally.
+
+The full test suite must be run inside the container:
+
+```bash
+docker compose -f docker-compose.dev.yml exec chatbot_backend_dev uv run pytest tests -rAq --tb=no
+```
+
+Run the PostgreSQL-backed concurrent billing release-gate verification:
+
+```bash
+docker exec -w /project chatbot_backend_dev uv run python scripts/verify_postgres_concurrent_billing.py
+```
+
+Expected result: one concurrent `/chat` request returns `200`, all other burst requests return `402`, final tenant balance is `0.000000`, and exactly one `credit_events` row is persisted.
+
+#### 7. Stop Development Environment
 
 ```bash
 # Stop containers (keep data)
 docker-compose -f docker-compose.dev.yml down
 
-# Stop and remove volumes (fresh database)
+# Stop and remove volumes (fresh database + vector store)
 docker-compose -f docker-compose.dev.yml down -v
 ```
 
@@ -181,6 +244,8 @@ docker-compose -f docker-compose.prod.yml up -d
 - **Public API:** http://your-domain.com (via nginx)
 - **Backend:** NOT accessible from outside (internal network only)
 - **PostgreSQL:** NOT accessible from outside (internal network only)
+- **Redis:** NOT accessible from outside (internal network only; used by Taskiq)
+- **WordPress sync worker:** background Taskiq worker, not HTTP-accessible
 
 #### 5. Verify Services are Running
 
@@ -197,6 +262,8 @@ docker-compose -f docker-compose.prod.yml ps | grep healthy
 chatbot_backend_prod   healthy
 chatbot_nginx_prod     healthy
 chatbot_postgres_prod  healthy
+chatbot_redis_prod     healthy
+wp_sync_worker_prod    running
 ```
 
 #### 6. View Logs

Dockerfile

@@ -24,7 +24,10 @@
 # - Faster builds (shared layers cached once)
 # ============================================================================
 
-FROM python:3.11-slim AS base
+FROM cgr.dev/chainguard/python:latest-dev AS base
+
+# Chainguard sets ENTRYPOINT ["python"] — clear it so exec-form CMD works directly.
+ENTRYPOINT []
 
 # Set working directory to project root
 # NOTE: Changed from /app to /project to support absolute imports (app.*)
@@ -48,45 +51,13 @@ COPY --from=ghcr.io/astral-sh/uv:latest /uv /usr/local/bin/uv
 # - uv.lock = Exact pinned versions of all dependencies (like package-lock.json)
 COPY pyproject.toml uv.lock ./
 
-# Install dependencies using uv
-# Using "uv pip install" instead of "uv sync" for system-wide installation
-#
-# Flags explained:
-# pip install      = Use pip-compatible interface (supports --system)
-# --system         = Install to system Python (not venv) - CRITICAL for Docker
-# -r pyproject.toml = Install from project dependencies
-#
-# Why --system is required in Docker:
-#
-# WITHOUT --system (default behavior):
-#   - uv creates a virtual environment (.venv/) and installs packages there
-#   - Executables end up in: .venv/bin/uvicorn (hidden location)
-#   - Docker can't find them in PATH → Error: "No such file or directory"
-#   - You'd need to use "uv run uvicorn ..." which adds complexity
-#
-# WITH --system:
-#   - uv installs directly to system Python (no .venv/ folder created)
-#   - Executables go to: /usr/local/bin/uvicorn (standard PATH location)
-#   - Docker finds them immediately → uvicorn command just works
-#   - Cleaner CMD: just "uvicorn ..." instead of "uv run uvicorn ..."
-#
-# Why virtual environments (.venv/) are "extra" in Docker:
-#   On your laptop: .venv/ needed to separate multiple projects
-#     - Project A needs pandas 1.0
-#     - Project B needs pandas 2.0
-#     - .venv/ prevents conflicts
-#
-#   In Docker: Container itself provides isolation
-#     - Container 1 has its own Python + packages (completely isolated)
-#     - Container 2 has its own Python + packages (can't see Container 1)
-#     - .venv/ inside container = locking a safe inside another safe (redundant)
-#
-# Analogy:
-#   Without --system = Hiding keys in a drawer (Docker can't find them)
-#   With --system    = Putting keys on the key hook (Docker finds them easily)
-#
-# This replaces traditional: pip install -r requirements.txt
-RUN uv pip install --system -r pyproject.toml
+# Chainguard runs as nonroot (uid 65532) — cannot write to system Python paths.
+# Install into /project/.venv (owned by nonroot) and put it on PATH.
+RUN uv venv /project/.venv && \
+    uv pip install --python /project/.venv/bin/python -r pyproject.toml
+ENV PATH="/project/.venv/bin:$PATH"
+# Avoid HuggingFace Hub Xet transfer stalls during startup embedding warmup.
+ENV HF_HUB_DISABLE_XET=1
 
 # Copy application code
 # NOTE: Changed to copy entire project instead of just app/ directory
@@ -97,8 +68,14 @@ RUN uv pip install --system -r pyproject.toml
 # Destination: . (current WORKDIR, which is /project inside container)
 #
 # Result: Files land at /project/app/main.py, /project/app/models/, etc.
+# Note: chroma_data/ is excluded via .dockerignore — created below with correct ownership.
 COPY . .
 
+# Pre-create ChromaDB data directory as nonroot so the named volume is initialised
+# with correct ownership. Without this, Docker creates the volume mount point as root,
+# and the nonroot process cannot write the SQLite file — causing 500 errors on all KB writes.
+RUN mkdir -p /project/chroma_data
+
 # ============================================================================
 # Why we DON'T set CMD or ENV here?
 # - Base stage is INCOMPLETE - not meant to run directly
@@ -130,7 +107,7 @@ ENV DEBUG=true
 
 # Development-specific command
 # Flags explained:
-# uvicorn             = ASGI server for FastAPI (installed to system with --system flag)
+# uvicorn             = ASGI server for FastAPI (available via /project/.venv on PATH)
 # app.main:app        = Import path (app/main.py file, app object) - using absolute import
 # --reload            = Watch for code changes, auto-restart server (hot reload)
 # --host 0.0.0.0      = Listen on ALL network interfaces (required for Docker)
@@ -143,7 +120,7 @@ ENV DEBUG=true
 # - --reload is EXPENSIVE (file watching, auto-restart) - only for development
 # - Production uses gunicorn (multi-worker, more robust)
 #
-# Note: No "uv run" needed because we used --system flag during install
+# Note: No "uv run" needed — venv bin is on PATH via ENV PATH above
 CMD ["uvicorn", "app.main:app", "--reload", "--host", "0.0.0.0", "--port", "8000"]
 
 # Port documentation (doesn't actually open ports)
@@ -190,7 +167,7 @@ ENV DEBUG=false
 # - Better stability, graceful shutdowns, worker health monitoring
 #
 # Flags explained:
-# gunicorn                     = WSGI server (wraps uvicorn workers, installed to system)
+# gunicorn                     = WSGI server (wraps uvicorn workers, available via /project/.venv on PATH)
 # app.main:app                 = Import path (app/main.py file, app object) - using absolute import
 # -k uvicorn.workers.UvicornWorker  = Use Uvicorn worker class (for ASGI support)
 # -w 4                         = 4 worker processes (adjust based on CPU cores)
@@ -202,7 +179,7 @@ ENV DEBUG=false
 # Worker count rule of thumb: (2 × CPU_cores) + 1
 # Example: 2-core server → 5 workers, 4-core server → 9 workers
 #
-# Note: No "uv run" needed because we used --system flag during install
+# Note: No "uv run" needed — venv bin is on PATH via ENV PATH above
 CMD ["gunicorn", "app.main:app", "-k", "uvicorn.workers.UvicornWorker", "-w", "4", "-b", "0.0.0.0:8000"]
 
 # Port documentation (doesn't actually open ports)
@@ -253,7 +230,7 @@ EXPOSE 7860
 #   docker images | grep smart-chatbot
 #
 # RUN DEVELOPMENT CONTAINER (standalone, no compose):
-#   docker run -p 8000:8000 -v $(pwd)/app:/app smart-chatbot:dev
+#   docker run -p 8000:8000 -v $(pwd)/app:/project/app smart-chatbot:dev
 #
 # RUN PRODUCTION CONTAINER (standalone, no compose):
 #   docker run -p 8000:8000 smart-chatbot:prod

PROJECT.md

@@ -0,0 +1,141 @@
+# Smart Chatbot — Project Navigation
+
+**What it is:** A chatbot-as-a-service (CaaS) microservice for SMEs. Tenants embed the widget on their site; the chatbot answers domain questions grounded in the tenant's knowledge base (RAG). Sold as prepaid token credits or monthly subscription.
+
+---
+
+## Current Focus
+
+**Next:** Phase 3 release gate — Simulated Tenant QA Pipeline (all prior gates including Qdrant migration are complete).
+Full task spec: `current_state/project_status.md`
+
+---
+
+## Tech Stack
+
+| Layer | Technology |
+|---|---|
+| Backend | FastAPI, Python, SQLAlchemy, Alembic |
+| Database | PostgreSQL (dev/prod), SQLite (HuggingFace Spaces) |
+| Vector DB | Qdrant — shared collection with required `tenant_id` payload filter; ChromaDB legacy fallback code only |
+| Embeddings | Google Gemini Embedding 2 (`gemini-embedding-2`, 3072-dim) |
+| LLM | LiteLLM Router — provider-agnostic, model aliases in env vars, auto-fallback chains |
+| Frontend | Preact + Shadow DOM widget, Vite build |
+| Auth | API key (widget) + Google OAuth + TOTP 2FA + JWT (admin) |
+| Infra | Docker multi-env, Nginx, Redis/Celery workers, Qdrant, HuggingFace Spaces |
+| CI/CD | GitHub Actions |
+
+---
+
+## Project Structure
+
+```
+app/
+  routers/      # Route handlers (knowledge_base, system_prompt)
+  schemas/      # Pydantic request/response models
+  middleware/   # TenantAuthMiddleware (API key → tenant_id)
+  models/       # SQLAlchemy models
+  services/     # Business logic (EmbeddingService, VectorStore, RetrievalService, LLMService, CreditService, WordPress sync, kb_reindex)
+  workers/      # Celery background workers (wordpress_sync, kb_reindex)
+  utils/        # api_key, token_counter, seeding
+  dependencies/ # FastAPI dependencies (admin_auth)
+alembic/        # DB migrations
+tests/          # 469 tests, organized by domain
+  auth/         # Admin + super-admin auth, JWT, TOTP, OAuth
+  knowledge_base/ # KB CRUD, chunking, pagination, reindex, sync
+  chat/         # Conversations, RAG context, input limits, history
+  billing/      # Credits, billing modes, concurrency
+  vector_store/ # Qdrant, ChromaDB, embedding service
+  workers/      # Summarization worker, WP sync jobs
+  llm/          # LLM service, retrieval, AI verdict, RAG
+  infra/        # Middleware, rate limiter, seeding, widget, system prompt
+frontend/       # Preact chat widget (Vite build → chatbot-widget.js)
+current_state/  # Architecture and status documents (see below)
+archive/        # Historical notes and discussion logs (never retroactively edited)
+docs/           # Specs and error logs
+```
+
+---
+
+## Key Architecture Decisions
+
+**Multi-tenancy:** API key → `tenant_id` extracted in middleware. All DB queries and vector store operations are scoped to `tenant_id`. Qdrant uses one shared collection with required `tenant_id` payload filters inside the adapter; PostgreSQL remains the source of truth and Qdrant is a rebuildable search index.
+
+**RAG pipeline:** `/chat` → `RetrievalService.retrieve(query, tenant_id)` → top-k KB docs injected into system message → LLM response grounded in KB.
+
+**User auth model:** Two modes — `anonymous` (most SMEs: no end-user login, visitors are anonymous) and `authenticated` (SaaS/e-commerce: host page passes authenticated user identity). Toggle via admin panel config.
+
+**LLM provider abstraction:** `LLMService` wraps LiteLLM Router. Model aliases (`chat-model`, `summarization-model`, `judge-llm`) declared in env vars — swapping providers requires only an env var change, no code change. Auto-fallback chains configured per alias. `complete_with_usage()` returns `LLMResult` with token counts for billing.
+
+**API versioning:** All public endpoints prefixed `/api/v1/` — in place before first client embeds widget (one-way door).
+
+**Widget embedding:** Preact + Shadow DOM — 3KB bundle, fully isolated from host-site CSS/JS, works on any tech stack (WordPress, Vue, React, plain HTML). Admin console generates per-tenant snippet: `<script src="widget.js" data-api-key="tenant_abc123"></script>`. Widget reads `data-api-key` at runtime.
+
+**Per-tenant customization:** All theme/branding/chatbot identity settings stored as `customization JSONB` on `tenants` table — zero schema migrations for new settings. Only fields that need filtering (e.g. `credits`, `is_active`) get proper columns.
+
+**Multi-tenant secrets:** Shared secrets (`GROQ_API_KEY`, `DATABASE_URL`) live in backend env. Per-tenant config (system prompt, KB, theme) lives in DB. BYOK tenants store their own LLM API key encrypted in `customization JSONB`.
+
+**Credit deduction atomicity:** Single SQL `UPDATE tenants SET balance_usd = balance_usd - :amount WHERE id = :tenant_id AND balance_usd >= :amount` — check rows affected (1 = success, 0 = insufficient balance). No race condition possible without application-level locking.
+
+---
+
+## Key Files
+
+| File | Purpose |
+|---|---|
+| `app/main.py` | Entry point — router registration, middleware, startup |
+| `app/models/conversation_manager.py` | LLM call lives here — chat logic, context assembly |
+| `app/models/nlp_engine.py` | Analytics and routing only — keep as-is |
+| `app/utils/config.py` | Singleton config — safe to defer refactor |
+| `.env` / `.env.staging` / `.env.production` | Environment-specific values |
+
+---
+
+## Business Model
+
+| Tier | Best for | How it works |
+|---|---|---|
+| Prepaid credits | Informational sites, low-volume SMEs | Buy credits upfront, deducted per message |
+| Monthly subscription | E-commerce, SaaS, high-volume SMEs | Fixed fee, predictable revenue |
+
+Gifted accounts (V1 only): Monireach tops up manually via super-admin; no Stripe needed. Full payment integration is post-V1.
+
+---
+
+## Deployment Roadmap
+
+| Stage | Tenants | Stack | Monthly cost |
+|---|---|---|---|
+| Now | 0–10 | HuggingFace Spaces (demo/portfolio) | Free |
+| Early | 10–100 | Railway or Fly.io (managed, zero DevOps) | $10–30 |
+| Growth | 100–1000 | Hetzner VPS + Docker Compose | €10–20 |
+| Scale | 1000+ | Hetzner + K3s (lightweight Kubernetes) | €30–80 |
+
+Target: Hetzner CAX31 ARM (4 vCPU, 8GB RAM, €14.10/month). PostgreSQL: Supabase managed at small scale → self-hosted at large scale. Admin panels: Cloudflare Pages (free). HF Space kept as public demo proxy only.
+
+**K3s local practice (after V1 stable):** Deploy the full stack to a local K3s cluster in parallel with Docker Compose — not a replacement for the dev workflow, but a learning environment for writing real K8s manifests (`Deployment`, `Service`, `Ingress`, `ConfigMap`, `Secret`) against a real multi-service app. Makes the eventual Hetzner → K3s production migration familiar ground.
+
+---
+
+## Where to Find Things
+
+| What you need | File |
+|---|---|
+| Task status, V1 checklist, current progress | `current_state/project_status.md` |
+| Full details on completed tasks (sub-tasks, bugs, decisions) | `current_state/milestone.md` |
+| Auth model, tenant isolation, API key design | `docs/security_architecture.md` |
+| Frontend widget architecture, embedding, theming | `docs/frontend_architecture.md` |
+| Khmer language strategy, LLM provider selection | `docs/khmer_llm_strategy.md` |
+| Admin console scope and integration status | `~/projects/smart_chatbot_admin_console/PROJECT.md` |
+| Super-admin console scope and task status | `~/projects/smart_chatbot_super_admin/current_state/project_status.md` |
+| Docker setup, env vars, service topology | `DOCKER_SETUP.md` |
+| Historical discussions (system expansion, billing design) | `archive/` |
+
+---
+
+## Live Deployments
+
+| Environment | URL |
+|---|---|
+| HuggingFace Spaces (public API) | `https://huggingface.co/spaces/monireach88/smart-chatbot-api` |
+| Portfolio (widget embedded) | `https://monireach.com` |

alembic/versions/00f16d0affc6_add_auth_providers_table_and_phone_to_.py

@@ -0,0 +1,46 @@
+"""add auth_providers table and phone to tenants
+
+Revision ID: 00f16d0affc6
+Revises: 752c239e659a
+Create Date: 2026-05-28 11:35:39.306737
+
+"""
+from typing import Sequence, Union
+
+from alembic import op
+import sqlalchemy as sa
+
+
+# revision identifiers, used by Alembic.
+revision: str = '00f16d0affc6'
+down_revision: Union[str, Sequence[str], None] = '752c239e659a'
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    """Upgrade schema."""
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.create_table('tenant_auth_providers',
+    sa.Column('id', sa.UUID(), nullable=False),
+    sa.Column('tenant_id', sa.UUID(), nullable=False),
+    sa.Column('provider', sa.String(length=20), nullable=False),
+    sa.Column('provider_user_id', sa.String(length=255), nullable=False),
+    sa.Column('linked_at', sa.DateTime(), nullable=False),
+    sa.CheckConstraint("provider IN ('google', 'telegram')", name=op.f('ck_tenant_auth_providers_chk_tenant_auth_providers_provider')),
+    sa.ForeignKeyConstraint(['tenant_id'], ['tenants.id'], name=op.f('fk_tenant_auth_providers_tenant_id_tenants'), ondelete='CASCADE'),
+    sa.PrimaryKeyConstraint('id', name=op.f('pk_tenant_auth_providers')),
+    sa.UniqueConstraint('tenant_id', 'provider', name='uq_tenant_auth_providers_tenant_id_provider')
+    )
+    op.create_index(op.f('ix_tenant_auth_providers_tenant_id'), 'tenant_auth_providers', ['tenant_id'], unique=False)
+    op.add_column('tenants', sa.Column('phone', sa.String(length=20), nullable=True))
+    # ### end Alembic commands ###
+
+
+def downgrade() -> None:
+    """Downgrade schema."""
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.drop_column('tenants', 'phone')
+    op.drop_index(op.f('ix_tenant_auth_providers_tenant_id'), table_name='tenant_auth_providers')
+    op.drop_table('tenant_auth_providers')
+    # ### end Alembic commands ###

alembic/versions/05aee5d64d1b_add_context_summary_and_summarization_.py

@@ -0,0 +1,47 @@
+"""add_context_summary_and_summarization_job
+
+Revision ID: 05aee5d64d1b
+Revises: d2d812e2c226
+Create Date: 2026-03-31 15:38:57.730352
+
+"""
+from typing import Sequence, Union
+
+from alembic import op
+import sqlalchemy as sa
+
+
+# revision identifiers, used by Alembic.
+revision: str = '05aee5d64d1b'
+down_revision: Union[str, Sequence[str], None] = 'd2d812e2c226'
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    """Upgrade schema."""
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.create_table('summarization_jobs',
+    sa.Column('id', sa.UUID(), nullable=False),
+    sa.Column('conversation_id', sa.UUID(), nullable=False),
+    sa.Column('status', sa.String(length=20), nullable=False),
+    sa.Column('attempt_count', sa.Integer(), nullable=False),
+    sa.Column('created_at', sa.DateTime(), nullable=False),
+    sa.Column('last_attempted_at', sa.DateTime(), nullable=True),
+    sa.ForeignKeyConstraint(['conversation_id'], ['conversations.id'], name=op.f('fk_summarization_jobs_conversation_id_conversations')),
+    sa.PrimaryKeyConstraint('id', name=op.f('pk_summarization_jobs'))
+    )
+    op.create_index(op.f('ix_summarization_jobs_conversation_id'), 'summarization_jobs', ['conversation_id'], unique=False)
+    op.create_index('uq_summarization_jobs_pending', 'summarization_jobs', ['conversation_id'], unique=True, postgresql_where=sa.text("status = 'pending'"))
+    op.add_column('conversations', sa.Column('context_summary', sa.Text(), nullable=True))
+    # ### end Alembic commands ###
+
+
+def downgrade() -> None:
+    """Downgrade schema."""
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.drop_column('conversations', 'context_summary')
+    op.drop_index('uq_summarization_jobs_pending', table_name='summarization_jobs', postgresql_where=sa.text("status = 'pending'"))
+    op.drop_index(op.f('ix_summarization_jobs_conversation_id'), table_name='summarization_jobs')
+    op.drop_table('summarization_jobs')
+    # ### end Alembic commands ###

alembic/versions/0704837a1594_add_backup_email_to_tenants.py

@@ -0,0 +1,32 @@
+"""add backup email to tenants
+
+Revision ID: 0704837a1594
+Revises: d1e126fba5fb
+Create Date: 2026-04-24 17:27:09.794919
+
+"""
+from typing import Sequence, Union
+
+from alembic import op
+import sqlalchemy as sa
+
+
+# revision identifiers, used by Alembic.
+revision: str = '0704837a1594'
+down_revision: Union[str, Sequence[str], None] = 'd1e126fba5fb'
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    """Upgrade schema."""
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.add_column('tenants', sa.Column('backup_email', sa.String(length=255), nullable=True))
+    # ### end Alembic commands ###
+
+
+def downgrade() -> None:
+    """Downgrade schema."""
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.drop_column('tenants', 'backup_email')
+    # ### end Alembic commands ###

alembic/versions/126f5b1610a6_add_billing_tables.py

@@ -0,0 +1,62 @@
+"""add billing tables
+
+Revision ID: 126f5b1610a6
+Revises: d7470fcd70c6
+Create Date: 2026-04-18 10:02:12.372002
+
+"""
+from typing import Sequence, Union
+
+from alembic import op
+import sqlalchemy as sa
+
+
+# revision identifiers, used by Alembic.
+revision: str = '126f5b1610a6'
+down_revision: Union[str, Sequence[str], None] = 'd7470fcd70c6'
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    """Upgrade schema."""
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.create_table('llm_models',
+    sa.Column('id', sa.Integer(), autoincrement=True, nullable=False),
+    sa.Column('provider', sa.String(length=100), nullable=False),
+    sa.Column('model_name', sa.String(length=255), nullable=False),
+    sa.Column('credits_per_1k_input', sa.Numeric(precision=20, scale=6), nullable=False),
+    sa.Column('credits_per_1k_output', sa.Numeric(precision=20, scale=6), nullable=False),
+    sa.Column('effective_from', sa.Date(), nullable=False),
+    sa.Column('effective_to', sa.Date(), nullable=True),
+    sa.PrimaryKeyConstraint('id', name=op.f('pk_llm_models'))
+    )
+    op.create_table('credit_events',
+    sa.Column('id', sa.UUID(), nullable=False),
+    sa.Column('tenant_id', sa.UUID(), nullable=False),
+    sa.Column('event_type', sa.String(length=20), nullable=False),
+    sa.Column('amount', sa.Numeric(precision=20, scale=6), nullable=False),
+    sa.Column('balance_after', sa.Numeric(precision=20, scale=6), nullable=False),
+    sa.Column('created_at', sa.DateTime(), nullable=False),
+    sa.Column('request_id', sa.String(length=255), nullable=True),
+    sa.Column('llm_model_id', sa.Integer(), nullable=True),
+    sa.ForeignKeyConstraint(['llm_model_id'], ['llm_models.id'], name=op.f('fk_credit_events_llm_model_id_llm_models')),
+    sa.ForeignKeyConstraint(['tenant_id'], ['tenants.id'], name=op.f('fk_credit_events_tenant_id_tenants')),
+    sa.PrimaryKeyConstraint('id', name=op.f('pk_credit_events')),
+    sa.UniqueConstraint('tenant_id', 'request_id', name='uq_credit_events_tenant_id_request_id')
+    )
+    op.create_index(op.f('ix_credit_events_tenant_id'), 'credit_events', ['tenant_id'], unique=False)
+    op.add_column('tenants', sa.Column('credits', sa.Numeric(precision=20, scale=6), server_default='0', nullable=False))
+    op.add_column('tenants', sa.Column('billing_mode', sa.String(length=20), server_default='credits', nullable=False))
+    # ### end Alembic commands ###
+
+
+def downgrade() -> None:
+    """Downgrade schema."""
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.drop_column('tenants', 'billing_mode')
+    op.drop_column('tenants', 'credits')
+    op.drop_index(op.f('ix_credit_events_tenant_id'), table_name='credit_events')
+    op.drop_table('credit_events')
+    op.drop_table('llm_models')
+    # ### end Alembic commands ###

alembic/versions/3d1418b32af0_retire_gemini_2_0_flash_add_gemini_2_5_.py

@@ -0,0 +1,64 @@
+"""retire_gemini_2_0_flash_add_gemini_2_5_flash_rates
+
+Revision ID: 3d1418b32af0
+Revises: 9d051350268a
+Create Date: 2026-04-19 02:38:42.878677
+
+Pricing basis (2026-04-19): $0.30/1M input, $2.50/1M output (paid tier) × 3x markup.
+1 credit ≈ $0.001 USD. gemini-2.0-flash retired per Google shutdown (2026-06-01).
+"""
+from typing import Sequence, Union
+
+from alembic import op
+
+
+# revision identifiers, used by Alembic.
+revision: str = '3d1418b32af0'
+down_revision: Union[str, Sequence[str], None] = '9d051350268a'
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    # Close out gemini-2.0-flash row
+    op.execute("""
+        UPDATE llm_models
+        SET effective_to = '2026-04-19'
+        WHERE provider = 'gemini'
+          AND model_name = 'gemini-2.0-flash'
+          AND effective_to IS NULL
+    """)
+
+    # Insert gemini-2.5-flash row (idempotent)
+    op.execute("""
+        INSERT INTO llm_models
+            (provider, model_name, credits_per_1k_input, credits_per_1k_output,
+             effective_from, effective_to)
+        SELECT 'gemini', 'gemini-2.5-flash', 0.900000, 7.500000, '2026-04-19', NULL
+        WHERE NOT EXISTS (
+            SELECT 1 FROM llm_models
+            WHERE provider = 'gemini'
+              AND model_name = 'gemini-2.5-flash'
+              AND effective_to IS NULL
+        )
+    """)
+
+
+def downgrade() -> None:
+    # Remove gemini-2.5-flash row
+    op.execute("""
+        DELETE FROM llm_models
+        WHERE provider = 'gemini'
+          AND model_name = 'gemini-2.5-flash'
+          AND effective_from = '2026-04-19'
+          AND effective_to IS NULL
+    """)
+
+    # Reopen gemini-2.0-flash row
+    op.execute("""
+        UPDATE llm_models
+        SET effective_to = NULL
+        WHERE provider = 'gemini'
+          AND model_name = 'gemini-2.0-flash'
+          AND effective_to = '2026-04-19'
+    """)

alembic/versions/485fbe495e6b_add_feedback_columns_to_messages.py

@@ -0,0 +1,34 @@
+"""add feedback columns to messages
+
+Revision ID: 485fbe495e6b
+Revises: 05aee5d64d1b
+Create Date: 2026-04-05 11:57:40.860101
+
+"""
+from typing import Sequence, Union
+
+from alembic import op
+import sqlalchemy as sa
+
+
+# revision identifiers, used by Alembic.
+revision: str = '485fbe495e6b'
+down_revision: Union[str, Sequence[str], None] = '05aee5d64d1b'
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    """Upgrade schema."""
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.add_column('messages', sa.Column('user_feedback', sa.SmallInteger(), nullable=True))
+    op.add_column('messages', sa.Column('tenant_feedback', sa.SmallInteger(), nullable=True))
+    # ### end Alembic commands ###
+
+
+def downgrade() -> None:
+    """Downgrade schema."""
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.drop_column('messages', 'tenant_feedback')
+    op.drop_column('messages', 'user_feedback')
+    # ### end Alembic commands ###

alembic/versions/73ae38d1200c_add_retrieved_doc_ids_to_messages.py

@@ -0,0 +1,32 @@
+"""add_retrieved_doc_ids_to_messages
+
+Revision ID: 73ae38d1200c
+Revises: e5cc44e2040a
+Create Date: 2026-03-30 02:26:27.464417
+
+"""
+from typing import Sequence, Union
+
+from alembic import op
+import sqlalchemy as sa
+from sqlalchemy.dialects import postgresql
+
+# revision identifiers, used by Alembic.
+revision: str = '73ae38d1200c'
+down_revision: Union[str, Sequence[str], None] = 'e5cc44e2040a'
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    """Upgrade schema."""
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.add_column('messages', sa.Column('retrieved_doc_ids', postgresql.JSONB(astext_type=sa.Text()).with_variant(sa.JSON(), 'sqlite'), nullable=True))
+    # ### end Alembic commands ###
+
+
+def downgrade() -> None:
+    """Downgrade schema."""
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.drop_column('messages', 'retrieved_doc_ids')
+    # ### end Alembic commands ###

alembic/versions/752c239e659a_add_rest_base_to_sync_jobs_expand_.py

@@ -0,0 +1,40 @@
+"""add rest_base to sync_jobs, expand status constraint
+
+Revision ID: 752c239e659a
+Revises: b58e26e702df
+Create Date: 2026-05-28 04:52:54.564900
+
+"""
+from typing import Sequence, Union
+
+from alembic import op
+import sqlalchemy as sa
+
+
+# revision identifiers, used by Alembic.
+revision: str = '752c239e659a'
+down_revision: Union[str, Sequence[str], None] = 'b58e26e702df'
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    """Upgrade schema."""
+    op.add_column('sync_jobs', sa.Column('rest_base', sa.String(length=50), server_default='posts', nullable=False))
+    op.drop_constraint('sync_jobs_status_valid', 'sync_jobs', type_='check')
+    op.create_check_constraint(
+        'sync_jobs_status_valid',
+        'sync_jobs',
+        "status IN ('cancelled', 'completed', 'expired', 'failed', 'paused', 'queued', 'running')",
+    )
+
+
+def downgrade() -> None:
+    """Downgrade schema."""
+    op.drop_constraint('sync_jobs_status_valid', 'sync_jobs', type_='check')
+    op.create_check_constraint(
+        'sync_jobs_status_valid',
+        'sync_jobs',
+        "status IN ('queued', 'running', 'completed', 'failed')",
+    )
+    op.drop_column('sync_jobs', 'rest_base')

alembic/versions/7992949ccdf5_add_industries_table.py

@@ -0,0 +1,56 @@
+"""add industries table
+
+Revision ID: 7992949ccdf5
+Revises: 0704837a1594
+Create Date: 2026-04-26 01:56:13.303817
+
+"""
+from typing import Sequence, Union
+
+from alembic import op
+import sqlalchemy as sa
+
+
+# revision identifiers, used by Alembic.
+revision: str = '7992949ccdf5'
+down_revision: Union[str, Sequence[str], None] = '0704837a1594'
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    industries_table = op.create_table(
+        "industries",
+        sa.Column("id", sa.Integer(), autoincrement=True, nullable=False),
+        sa.Column("code", sa.String(length=100), nullable=False),
+        sa.Column("display_name", sa.String(length=100), nullable=False),
+        sa.Column("is_active", sa.Boolean(), nullable=False, server_default=sa.true()),
+        sa.PrimaryKeyConstraint("id", name=op.f("pk_industries")),
+        sa.UniqueConstraint("code", name=op.f("uq_industries_code")),
+    )
+    op.bulk_insert(
+        industries_table,
+        [
+            {"code": "restaurant-food-beverage", "display_name": "Restaurant / Food & Beverage", "is_active": True},
+            {"code": "retail",                   "display_name": "Retail",                        "is_active": True},
+            {"code": "e-commerce",               "display_name": "E-Commerce",                    "is_active": True},
+            {"code": "healthcare",               "display_name": "Healthcare",                    "is_active": True},
+            {"code": "legal",                    "display_name": "Legal",                         "is_active": True},
+            {"code": "education",                "display_name": "Education",                     "is_active": True},
+            {"code": "real-estate",              "display_name": "Real Estate",                   "is_active": True},
+            {"code": "financial-services",       "display_name": "Financial Services",            "is_active": True},
+            {"code": "technology",               "display_name": "Technology",                    "is_active": True},
+            {"code": "travel-hospitality",       "display_name": "Travel & Hospitality",          "is_active": True},
+            {"code": "beauty-wellness",          "display_name": "Beauty & Wellness",             "is_active": True},
+            {"code": "professional-services",    "display_name": "Professional Services",         "is_active": True},
+            {"code": "non-profit",               "display_name": "Non-Profit",                    "is_active": True},
+            {"code": "other",                    "display_name": "Other",                         "is_active": True},
+            # Legacy codes — kept for backward compatibility; hidden from new dropdowns
+            {"code": "saas",                     "display_name": "SaaS",                          "is_active": False},
+            {"code": "finance",                  "display_name": "Finance",                       "is_active": False},
+        ],
+    )
+
+
+def downgrade() -> None:
+    op.drop_table("industries")

alembic/versions/7aec975a4140_credits_to_usd_refactor.py

@@ -0,0 +1,64 @@
+"""credits to usd refactor
+
+Revision ID: 7aec975a4140
+Revises: ac67923a3324
+Create Date: 2026-05-17 04:08:19.969428
+
+"""
+from typing import Sequence, Union
+
+from alembic import op
+import sqlalchemy as sa
+
+
+# revision identifiers, used by Alembic.
+revision: str = '7aec975a4140'
+down_revision: Union[str, Sequence[str], None] = 'ac67923a3324'
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    """Upgrade schema."""
+    op.add_column(
+        "llm_models",
+        sa.Column("usd_per_1k_input", sa.Numeric(precision=20, scale=6), nullable=True),
+    )
+    op.add_column(
+        "llm_models",
+        sa.Column("usd_per_1k_output", sa.Numeric(precision=20, scale=6), nullable=True),
+    )
+    op.add_column(
+        "tenants",
+        sa.Column("balance_usd", sa.Numeric(precision=20, scale=6), server_default="0", nullable=True),
+    )
+
+    op.execute(
+        """
+        UPDATE llm_models
+        SET usd_per_1k_input = credits_per_1k_input,
+            usd_per_1k_output = credits_per_1k_output
+        WHERE usd_per_1k_input IS NULL
+           OR usd_per_1k_output IS NULL
+        """
+    )
+    op.execute(
+        """
+        UPDATE tenants
+        SET balance_usd = credits
+        WHERE balance_usd IS NULL
+        """
+    )
+
+    op.alter_column("llm_models", "usd_per_1k_input", nullable=False)
+    op.alter_column("llm_models", "usd_per_1k_output", nullable=False)
+    op.alter_column("tenants", "balance_usd", nullable=False)
+    op.alter_column("llm_models", "credits_per_1k_input", server_default="0")
+    op.alter_column("llm_models", "credits_per_1k_output", server_default="0")
+
+
+def downgrade() -> None:
+    """Downgrade schema."""
+    op.drop_column("tenants", "balance_usd")
+    op.drop_column("llm_models", "usd_per_1k_output")
+    op.drop_column("llm_models", "usd_per_1k_input")

alembic/versions/84de4a13cffd_add_ai_verdict_scores_to_messages.py

@@ -0,0 +1,36 @@
+"""add_ai_verdict_scores_to_messages
+
+Revision ID: 84de4a13cffd
+Revises: 485fbe495e6b
+Create Date: 2026-04-11 04:30:21.707463
+
+"""
+from typing import Sequence, Union
+
+from alembic import op
+import sqlalchemy as sa
+
+
+# revision identifiers, used by Alembic.
+revision: str = '84de4a13cffd'
+down_revision: Union[str, Sequence[str], None] = '485fbe495e6b'
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    """Upgrade schema."""
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.add_column('messages', sa.Column('accuracy_score', sa.Float(), nullable=True))
+    op.add_column('messages', sa.Column('relevance_score', sa.Float(), nullable=True))
+    op.add_column('messages', sa.Column('safety_score', sa.Float(), nullable=True))
+    # ### end Alembic commands ###
+
+
+def downgrade() -> None:
+    """Downgrade schema."""
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.drop_column('messages', 'safety_score')
+    op.drop_column('messages', 'relevance_score')
+    op.drop_column('messages', 'accuracy_score')
+    # ### end Alembic commands ###

alembic/versions/9ad31e299e42_add_is_indexed_to_knowledge_entries.py

@@ -0,0 +1,38 @@
+"""add is_indexed to knowledge_entries
+
+Revision ID: 9ad31e299e42
+Revises: 7aec975a4140
+Create Date: 2026-05-17 16:26:35.228332
+
+"""
+from typing import Sequence, Union
+
+from alembic import op
+import sqlalchemy as sa
+
+
+# revision identifiers, used by Alembic.
+revision: str = '9ad31e299e42'
+down_revision: Union[str, Sequence[str], None] = '7aec975a4140'
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    """Upgrade schema."""
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.add_column('knowledge_base', sa.Column('is_indexed', sa.Boolean(), server_default='false', nullable=False))
+    op.drop_column('llm_models', 'credits_per_1k_output')
+    op.drop_column('llm_models', 'credits_per_1k_input')
+    op.drop_column('tenants', 'credits')
+    # ### end Alembic commands ###
+
+
+def downgrade() -> None:
+    """Downgrade schema."""
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.add_column('tenants', sa.Column('credits', sa.NUMERIC(precision=20, scale=6), server_default=sa.text("'0'::numeric"), autoincrement=False, nullable=False))
+    op.add_column('llm_models', sa.Column('credits_per_1k_input', sa.NUMERIC(precision=20, scale=6), server_default=sa.text("'0'::numeric"), autoincrement=False, nullable=False))
+    op.add_column('llm_models', sa.Column('credits_per_1k_output', sa.NUMERIC(precision=20, scale=6), server_default=sa.text("'0'::numeric"), autoincrement=False, nullable=False))
+    op.drop_column('knowledge_base', 'is_indexed')
+    # ### end Alembic commands ###

alembic/versions/9b95a5ce2ac1_add_gifted_billing_mode_check_constraint.py

@@ -0,0 +1,30 @@
+"""add gifted billing mode check constraint
+
+Revision ID: 9b95a5ce2ac1
+Revises: bf4016f7dd2a
+Create Date: 2026-04-24 09:16:41.390223
+
+"""
+from typing import Sequence, Union
+
+from alembic import op
+import sqlalchemy as sa
+
+
+# revision identifiers, used by Alembic.
+revision: str = '9b95a5ce2ac1'
+down_revision: Union[str, Sequence[str], None] = 'bf4016f7dd2a'
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    op.create_check_constraint(
+        "ck_tenants_chk_tenant_billing_mode",
+        "tenants",
+        "billing_mode IN ('credits', 'subscription', 'gifted')",
+    )
+
+
+def downgrade() -> None:
+    op.drop_constraint("ck_tenants_chk_tenant_billing_mode", "tenants", type_="check")

alembic/versions/9d051350268a_seed_llm_models_initial_rates.py

@@ -0,0 +1,59 @@
+"""seed_llm_models_initial_rates
+
+Revision ID: 9d051350268a
+Revises: 126f5b1610a6
+Create Date: 2026-04-18 16:51:58.821173
+
+Pricing basis: 2026-04-18 approximate API costs × ~3x business markup.
+1 credit ≈ $0.001 USD. To update rates: set effective_to on old rows,
+insert new rows — never overwrite.
+"""
+from typing import Sequence, Union
+
+from alembic import op
+
+
+# revision identifiers, used by Alembic.
+revision: str = '9d051350268a'
+down_revision: Union[str, Sequence[str], None] = '126f5b1610a6'
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+_ROWS = [
+    ("gemini",    "gemini-2.0-flash",            "0.300000", "1.200000"),
+    ("groq",      "qwen3-32b",                   "0.870000", "1.770000"),
+    ("groq",      "llama-3.3-70b-versatile",     "1.770000", "2.370000"),
+    ("anthropic", "claude-3-5-haiku-20241022",   "2.400000", "12.000000"),
+]
+
+
+def upgrade() -> None:
+    for provider, model_name, inp, out in _ROWS:
+        op.execute(
+            f"""
+            INSERT INTO llm_models
+                (provider, model_name, credits_per_1k_input, credits_per_1k_output,
+                 effective_from, effective_to)
+            SELECT '{provider}', '{model_name}', {inp}, {out}, '2026-04-18', NULL
+            WHERE NOT EXISTS (
+                SELECT 1 FROM llm_models
+                WHERE provider = '{provider}'
+                  AND model_name = '{model_name}'
+                  AND effective_to IS NULL
+            )
+            """
+        )
+
+
+def downgrade() -> None:
+    for provider, model_name, _, _ in _ROWS:
+        op.execute(
+            f"""
+            DELETE FROM llm_models
+            WHERE provider = '{provider}'
+              AND model_name = '{model_name}'
+              AND effective_from = '2026-04-18'
+              AND effective_to IS NULL
+            """
+        )

alembic/versions/a53f4c8d9b21_kb_schema_redesign_and_sync_jobs.py

@@ -0,0 +1,95 @@
+"""kb schema redesign and sync jobs
+
+Revision ID: a53f4c8d9b21
+Revises: 7992949ccdf5
+Create Date: 2026-04-29 23:05:00.000000
+
+"""
+from typing import Sequence, Union
+
+from alembic import op
+import sqlalchemy as sa
+
+
+# revision identifiers, used by Alembic.
+revision: str = "a53f4c8d9b21"
+down_revision: Union[str, Sequence[str], None] = "7992949ccdf5"
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    with op.batch_alter_table("knowledge_base") as batch_op:
+        batch_op.alter_column(
+            "entry",
+            new_column_name="content",
+            existing_type=sa.Text(),
+            existing_nullable=False,
+        )
+        batch_op.add_column(sa.Column("source_url", sa.String(length=500), nullable=True))
+        batch_op.add_column(
+            sa.Column("origin", sa.String(length=20), server_default="manual", nullable=False)
+        )
+        batch_op.create_check_constraint(
+            "ck_knowledge_base_kb_origin_valid",
+            "origin IN ('manual', 'synced')",
+        )
+        batch_op.drop_column("tag")
+        batch_op.drop_column("category")
+
+    inspector = sa.inspect(op.get_bind())
+    if not inspector.has_table("sync_jobs"):
+        op.create_table(
+            "sync_jobs",
+            sa.Column("id", sa.UUID(), nullable=False),
+            sa.Column("created_at", sa.DateTime(), nullable=True),
+            sa.Column("updated_at", sa.DateTime(), nullable=True),
+            sa.Column("tenant_id", sa.UUID(), nullable=False),
+            sa.Column("source_url", sa.String(length=500), nullable=False),
+            sa.Column("status", sa.String(length=30), server_default="queued", nullable=False),
+            sa.Column("total", sa.Integer(), server_default="0", nullable=False),
+            sa.Column("processed", sa.Integer(), server_default="0", nullable=False),
+            sa.Column("failed", sa.Integer(), server_default="0", nullable=False),
+            sa.Column("error_message", sa.Text(), nullable=True),
+            sa.CheckConstraint(
+                "failed >= 0",
+                name=op.f("ck_sync_jobs_sync_jobs_failed_nonnegative"),
+            ),
+            sa.CheckConstraint(
+                "processed >= 0",
+                name=op.f("ck_sync_jobs_sync_jobs_processed_nonnegative"),
+            ),
+            sa.CheckConstraint(
+                "processed + failed <= total",
+                name=op.f("ck_sync_jobs_sync_jobs_progress_within_total"),
+            ),
+            sa.CheckConstraint(
+                "status IN ('queued', 'running', 'completed', 'failed')",
+                name=op.f("ck_sync_jobs_sync_jobs_status_valid"),
+            ),
+            sa.CheckConstraint(
+                "total >= 0",
+                name=op.f("ck_sync_jobs_sync_jobs_total_nonnegative"),
+            ),
+            sa.ForeignKeyConstraint(["tenant_id"], ["tenants.id"], name=op.f("fk_sync_jobs_tenant_id_tenants")),
+            sa.PrimaryKeyConstraint("id", name=op.f("pk_sync_jobs")),
+        )
+
+
+def downgrade() -> None:
+    op.drop_table("sync_jobs")
+
+    with op.batch_alter_table("knowledge_base") as batch_op:
+        batch_op.add_column(
+            sa.Column("category", sa.String(length=100), server_default="Uncategorized", nullable=False)
+        )
+        batch_op.add_column(sa.Column("tag", sa.String(length=100), nullable=True))
+        batch_op.drop_constraint("ck_knowledge_base_kb_origin_valid", type_="check")
+        batch_op.drop_column("origin")
+        batch_op.drop_column("source_url")
+        batch_op.alter_column(
+            "content",
+            new_column_name="entry",
+            existing_type=sa.Text(),
+            existing_nullable=False,
+        )

alembic/versions/ac67923a3324_add_created_updated_skipped_to_sync_jobs.py

@@ -0,0 +1,36 @@
+"""add created updated skipped to sync_jobs
+
+Revision ID: ac67923a3324
+Revises: a53f4c8d9b21
+Create Date: 2026-05-02 15:38:19.664806
+
+"""
+from typing import Sequence, Union
+
+from alembic import op
+import sqlalchemy as sa
+
+
+# revision identifiers, used by Alembic.
+revision: str = 'ac67923a3324'
+down_revision: Union[str, Sequence[str], None] = 'a53f4c8d9b21'
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    """Upgrade schema."""
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.add_column('sync_jobs', sa.Column('created', sa.Integer(), server_default='0', nullable=False))
+    op.add_column('sync_jobs', sa.Column('updated', sa.Integer(), server_default='0', nullable=False))
+    op.add_column('sync_jobs', sa.Column('skipped', sa.Integer(), server_default='0', nullable=False))
+    # ### end Alembic commands ###
+
+
+def downgrade() -> None:
+    """Downgrade schema."""
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.drop_column('sync_jobs', 'skipped')
+    op.drop_column('sync_jobs', 'updated')
+    op.drop_column('sync_jobs', 'created')
+    # ### end Alembic commands ###

alembic/versions/b58e26e702df_add_is_blocked_to_tenants_and_note_.py

@@ -0,0 +1,36 @@
+"""add is_blocked to tenants and note created_by to credit_events
+
+Revision ID: b58e26e702df
+Revises: 9ad31e299e42
+Create Date: 2026-05-27 16:58:30.128906
+
+"""
+from typing import Sequence, Union
+
+from alembic import op
+import sqlalchemy as sa
+
+
+# revision identifiers, used by Alembic.
+revision: str = 'b58e26e702df'
+down_revision: Union[str, Sequence[str], None] = '9ad31e299e42'
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    """Upgrade schema."""
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.add_column('credit_events', sa.Column('note', sa.Text(), nullable=True))
+    op.add_column('credit_events', sa.Column('created_by', sa.String(length=100), nullable=True))
+    op.add_column('tenants', sa.Column('is_blocked', sa.Boolean(), server_default='false', nullable=False))
+    # ### end Alembic commands ###
+
+
+def downgrade() -> None:
+    """Downgrade schema."""
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.drop_column('tenants', 'is_blocked')
+    op.drop_column('credit_events', 'created_by')
+    op.drop_column('credit_events', 'note')
+    # ### end Alembic commands ###

alembic/versions/bf4016f7dd2a_add_website_url_to_tenants.py

@@ -0,0 +1,32 @@
+"""add website_url to tenants
+
+Revision ID: bf4016f7dd2a
+Revises: 3d1418b32af0
+Create Date: 2026-04-24 08:49:11.717539
+
+"""
+from typing import Sequence, Union
+
+from alembic import op
+import sqlalchemy as sa
+
+
+# revision identifiers, used by Alembic.
+revision: str = 'bf4016f7dd2a'
+down_revision: Union[str, Sequence[str], None] = '3d1418b32af0'
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    """Upgrade schema."""
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.add_column('tenants', sa.Column('website_url', sa.String(length=255), nullable=True))
+    # ### end Alembic commands ###
+
+
+def downgrade() -> None:
+    """Downgrade schema."""
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.drop_column('tenants', 'website_url')
+    # ### end Alembic commands ###

alembic/versions/c9e1f8a2b7d4_add_auth_fields_to_tenants.py

@@ -0,0 +1,39 @@
+"""add_auth_fields_to_tenants
+
+Revision ID: c9e1f8a2b7d4
+Revises: 84de4a13cffd
+Create Date: 2026-04-17 00:00:00.000000
+
+"""
+from typing import Sequence, Union
+
+from alembic import op
+import sqlalchemy as sa
+
+
+# revision identifiers, used by Alembic.
+revision: str = 'c9e1f8a2b7d4'
+down_revision: Union[str, Sequence[str], None] = '84de4a13cffd'
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    """Upgrade schema."""
+    op.add_column('tenants', sa.Column('owner_email', sa.String(255), nullable=True))
+    op.add_column('tenants', sa.Column('google_id', sa.String(255), nullable=True))
+    op.add_column('tenants', sa.Column('totp_secret', sa.String(512), nullable=True))
+    op.add_column('tenants', sa.Column('totp_enabled', sa.Boolean(), nullable=True))
+
+    op.create_unique_constraint('uq_tenants_owner_email', 'tenants', ['owner_email'])
+    op.create_unique_constraint('uq_tenants_google_id', 'tenants', ['google_id'])
+
+
+def downgrade() -> None:
+    """Downgrade schema."""
+    op.drop_constraint('uq_tenants_google_id', 'tenants', type_='unique')
+    op.drop_constraint('uq_tenants_owner_email', 'tenants', type_='unique')
+    op.drop_column('tenants', 'totp_enabled')
+    op.drop_column('tenants', 'totp_secret')
+    op.drop_column('tenants', 'google_id')
+    op.drop_column('tenants', 'owner_email')

alembic/versions/d1e126fba5fb_add_title_category_tag_to_kb_entries.py

@@ -0,0 +1,36 @@
+"""add title category tag to kb entries
+
+Revision ID: d1e126fba5fb
+Revises: 9b95a5ce2ac1
+Create Date: 2026-04-24 09:32:20.768068
+
+"""
+from typing import Sequence, Union
+
+from alembic import op
+import sqlalchemy as sa
+
+
+# revision identifiers, used by Alembic.
+revision: str = 'd1e126fba5fb'
+down_revision: Union[str, Sequence[str], None] = '9b95a5ce2ac1'
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    """Upgrade schema."""
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.add_column('knowledge_base', sa.Column('title', sa.String(length=255), server_default='Untitled', nullable=False))
+    op.add_column('knowledge_base', sa.Column('category', sa.String(length=100), server_default='Uncategorized', nullable=False))
+    op.add_column('knowledge_base', sa.Column('tag', sa.String(length=100), nullable=True))
+    # ### end Alembic commands ###
+
+
+def downgrade() -> None:
+    """Downgrade schema."""
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.drop_column('knowledge_base', 'tag')
+    op.drop_column('knowledge_base', 'category')
+    op.drop_column('knowledge_base', 'title')
+    # ### end Alembic commands ###

alembic/versions/d2d812e2c226_add_last_message_preview_to_.py

@@ -0,0 +1,32 @@
+"""add_last_message_preview_to_conversations
+
+Revision ID: d2d812e2c226
+Revises: 73ae38d1200c
+Create Date: 2026-03-30 10:33:10.409942
+
+"""
+from typing import Sequence, Union
+
+from alembic import op
+import sqlalchemy as sa
+
+
+# revision identifiers, used by Alembic.
+revision: str = 'd2d812e2c226'
+down_revision: Union[str, Sequence[str], None] = '73ae38d1200c'
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    """Upgrade schema."""
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.add_column('conversations', sa.Column('last_message_preview', sa.String(length=255), nullable=True))
+    # ### end Alembic commands ###
+
+
+def downgrade() -> None:
+    """Downgrade schema."""
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.drop_column('conversations', 'last_message_preview')
+    # ### end Alembic commands ###

alembic/versions/d7470fcd70c6_add_backup_codes_to_tenants.py

@@ -0,0 +1,32 @@
+"""add_backup_codes_to_tenants
+
+Revision ID: d7470fcd70c6
+Revises: c9e1f8a2b7d4
+Create Date: 2026-04-17 15:21:51.999768
+
+"""
+from typing import Sequence, Union
+
+from alembic import op
+import sqlalchemy as sa
+
+
+# revision identifiers, used by Alembic.
+revision: str = 'd7470fcd70c6'
+down_revision: Union[str, Sequence[str], None] = 'c9e1f8a2b7d4'
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    """Upgrade schema."""
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.add_column('tenants', sa.Column('backup_codes', sa.Text(), nullable=True))
+    # ### end Alembic commands ###
+
+
+def downgrade() -> None:
+    """Downgrade schema."""
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.drop_column('tenants', 'backup_codes')
+    # ### end Alembic commands ###

alembic/versions/e5cc44e2040a_make_tenant_id_non_nullable_on_users_.py

@@ -0,0 +1,44 @@
+"""make tenant_id non-nullable on users; change entities to JSON on messages
+
+Revision ID: e5cc44e2040a
+Revises: f6988c25696e
+Create Date: 2026-03-27 15:22:44.809002
+
+"""
+from typing import Sequence, Union
+
+from alembic import op
+import sqlalchemy as sa
+from sqlalchemy.dialects import postgresql
+
+# revision identifiers, used by Alembic.
+revision: str = 'e5cc44e2040a'
+down_revision: Union[str, Sequence[str], None] = 'f6988c25696e'
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    """Upgrade schema."""
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.execute("ALTER TABLE messages ALTER COLUMN entities TYPE JSONB USING entities::jsonb")
+    op.execute("DELETE FROM messages WHERE conversation_id IN (SELECT id FROM conversations WHERE user_id IN (SELECT id FROM users WHERE tenant_id IS NULL))")
+    op.execute("DELETE FROM conversations WHERE user_id IN (SELECT id FROM users WHERE tenant_id IS NULL)")
+    op.execute("DELETE FROM users WHERE tenant_id IS NULL")
+    op.alter_column('users', 'tenant_id',
+               existing_type=sa.UUID(),
+               nullable=False)
+    # ### end Alembic commands ###
+
+
+def downgrade() -> None:
+    """Downgrade schema."""
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.alter_column('users', 'tenant_id',
+               existing_type=sa.UUID(),
+               nullable=True)
+    op.alter_column('messages', 'entities',
+               existing_type=postgresql.JSONB(astext_type=sa.Text()),
+               type_=sa.TEXT(),
+               existing_nullable=True)
+    # ### end Alembic commands ###

alembic/versions/f6988c25696e_add_customization_to_tenants.py

@@ -0,0 +1,29 @@
+"""add_customization_to_tenants
+
+Revision ID: f6988c25696e
+Revises: 5f697ba37cb7
+Create Date: 2026-03-24 23:33:21.014824
+
+"""
+from typing import Sequence, Union
+
+from alembic import op
+import sqlalchemy as sa
+from sqlalchemy.dialects import postgresql
+
+
+# revision identifiers, used by Alembic.
+revision: str = 'f6988c25696e'
+down_revision: Union[str, Sequence[str], None] = '5f697ba37cb7'
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    """Upgrade schema."""
+    op.add_column("tenants", sa.Column("customization", postgresql.JSONB(), nullable=True))
+
+
+def downgrade() -> None:
+    """Downgrade schema."""
+    op.drop_column("tenants", "customization")

app/dependencies/admin_auth.py

@@ -0,0 +1,23 @@
+import uuid
+
+from fastapi import HTTPException, Request
+
+from app.services.google_auth import decode_full_jwt
+from app.utils.config import get_config
+
+config = get_config()
+
+
+def require_admin_jwt(request: Request) -> uuid.UUID:
+    token = request.cookies.get("access_token")
+    if not token:
+        raise HTTPException(status_code=401, detail="Unauthorized")
+
+    try:
+        tenant_id_str = decode_full_jwt(token, config.jwt_secret_key)
+        tenant_id = uuid.UUID(tenant_id_str)
+    except (ValueError, AttributeError):
+        raise HTTPException(status_code=401, detail="Unauthorized")
+
+    request.state.tenant_id = tenant_id
+    return tenant_id

app/dependencies/super_admin_auth.py

@@ -0,0 +1,27 @@
+from typing import NoReturn
+
+from fastapi import HTTPException, Request
+
+from app.services.google_auth import decode_super_admin_jwt
+from app.utils.config import get_config
+
+config = get_config()
+
+
+def require_super_admin_jwt(request: Request) -> str:
+    def _reject() -> NoReturn:
+        raise HTTPException(status_code=403, detail="Forbidden")
+
+    token = request.cookies.get("super_admin_token")
+    if not token:
+        _reject()
+
+    try:
+        email = decode_super_admin_jwt(token, config.jwt_secret_key)
+    except ValueError:
+        _reject()
+
+    if email.lower() not in [e.lower() for e in config.super_admin_emails]:
+        _reject()
+
+    return email

app/main.py

@@ -1,10 +1,15 @@
 # app/main.py
 # pylint: disable=unused-import
+from decimal import Decimal
+from typing import Optional
 import uuid
 import time
+from collections.abc import AsyncIterator
 from contextlib import asynccontextmanager
 from pathlib import Path
-from fastapi import FastAPI, HTTPException
+from fastapi import APIRouter, BackgroundTasks, Depends, FastAPI, HTTPException, Request
+from sqlalchemy import select
+from sqlalchemy.orm import Session
 from fastapi.middleware.cors import CORSMiddleware
 from fastapi.staticfiles import StaticFiles
 from pydantic import BaseModel
@@ -12,27 +17,68 @@ from pydantic import BaseModel
 # NOTE: Using absolute imports from project root (app.*) instead of relative imports
 # This ensures imports work consistently in both IDE and Docker environments
 from app.middleware.tenant_auth import TenantAuthMiddleware
+from app.services.credit_service import (
+    CreditService,
+    InsufficientCreditsError,
+    ModelRateNotFoundError,
+)
+from app.services.dependencies import (
+    get_embedding_service,
+    get_retrieval_service,
+    get_vector_store,
+)
+from app.services.retrieval_service import RetrievalService
 from app.utils.config import get_config, get_logger, get_version
 from app.models.nlp_engine import NLPEngine
 from app.models.conversation_manager import ConversationManager
 from app.models.database import (
+    LLMModel,
     SessionLocal,
+    Tenant,
+    get_db,
     init_database,
     health_check as db_health_check,
 )
-from app.models.smart_models import UserPreference, UserInsight, ConversationTopic
+from app.models.smart_models import (
+    UserPreference,
+    UserInsight,
+    ConversationTopic,
+)  # this ensures Base in database.py imports these tables
 from app.data.training_data import TRAINING_DATA
-from app.routers import knowledge_base
+from app.routers import (
+    admin_attention,
+    admin_auth,
+    admin_apikey,
+    admin_me,
+    admin_settings,
+    billing,
+    conversations,
+    knowledge_base,
+    super_admin_auth,
+    super_admin_tenants,
+    super_admin_rates,
+    widget_settings,
+    system_prompt as system_prompt_router,
+)
 from app.utils.seeding import seed_demo_tenant
+from app.utils.token_counter import InputTooLargeError
+from app.services.ai_verdict import run_ai_verdict
 
 # Load configuration (this sets up logging automatically)
 config = get_config()
 logger = get_logger(__name__)
 
+API_V1_PREFIX = f"/api/{config.api_version}"
+api_router = APIRouter(prefix=API_V1_PREFIX)
+
+_qdrant_ready: bool = False
+
 
 # Train the mode on startup
 @asynccontextmanager
-async def lifespan(fastapi_app: FastAPI):  # pylint: disable=unused-argument
+async def lifespan(
+    fastapi_app: FastAPI,
+) -> AsyncIterator[None]:  # pylint: disable=unused-argument
     """Application lifespan manager for startup and shutdown events"""
     # Startup
     logger.info("Starting application..")
@@ -42,7 +88,8 @@ async def lifespan(fastapi_app: FastAPI):  # pylint: disable=unused-argument
     try:
         init_database()
         with SessionLocal() as session:
-            seed_demo_tenant(session)
+            if config.env.name == "development":
+                seed_demo_tenant(session)
 
         if db_health_check():
             logger.info("Database connection established successfully")
@@ -52,9 +99,25 @@ async def lifespan(fastapi_app: FastAPI):  # pylint: disable=unused-argument
         logger.error("Database initialization failed: %s: %s", type(e).__name__, str(e))
         logger.warning("Continuing without database - some features may not work...")
 
+    global _qdrant_ready
+    try:
+        get_vector_store().health_check()
+        _qdrant_ready = True
+        logger.info("Vector store readiness check passed")
+    except Exception as e:  # pylint: disable=broad-exception-caught
+        _qdrant_ready = False
+        logger.error(
+            "Vector store readiness check failed: %s — starting in degraded mode", str(e)
+        )
+
     # Train NLP model
     nlp_engine.train_intent_classifier(TRAINING_DATA)
     logger.info("NLP model training completed")
+
+    # Remote Gemini embeddings do not require a local model warmup call.
+    get_embedding_service()
+    logger.info("Embedding service initialized successfully")
+
     logger.info("Application started successfully in %s mode", config.env.name)
 
     yield  # App runs here
@@ -64,7 +127,20 @@ async def lifespan(fastapi_app: FastAPI):  # pylint: disable=unused-argument
 
 
 app = FastAPI(title=config.api.title, debug=config.api.debug, lifespan=lifespan)
-app.include_router(knowledge_base.router)
+
+api_router.include_router(admin_attention.router)
+api_router.include_router(admin_auth.router)
+api_router.include_router(admin_apikey.router)
+api_router.include_router(admin_me.router)
+api_router.include_router(admin_settings.router)
+api_router.include_router(knowledge_base.router)
+api_router.include_router(billing.router)
+api_router.include_router(widget_settings.router)
+api_router.include_router(system_prompt_router.router)
+api_router.include_router(conversations.router)
+api_router.include_router(super_admin_auth.router)
+api_router.include_router(super_admin_tenants.router)
+api_router.include_router(super_admin_rates.router)
 
 # Enable CORS
 app.add_middleware(
@@ -115,34 +191,51 @@ async def health_check():
     # Check database health
     db_status = "healthy" if db_health_check() else "unhealthy"
 
+    qdrant_status = "healthy" if _qdrant_ready else "degraded"
+    overall_status = "healthy" if (db_status == "healthy" and _qdrant_ready) else "degraded"
     return {
-        "status": "healthy",
+        "status": overall_status,
         "environment": config.env.name,
         "version": get_version(),  # Read from pyproject.toml
         "database": db_status,
+        "qdrant": qdrant_status,
         "components": {
             "nlp_engine": "ready",
             "conversation_manager": "ready",
             "database": db_status,
+            "qdrant": qdrant_status,
         },
     }
 
 
-@app.post("/chat", response_model=ChatResponse)
-async def chat(request: ChatRequest):
-    logger.debug("Chat request received: %s ...", request.message[:50])
+@api_router.post("/chat", response_model=ChatResponse)
+async def chat(
+    request: Request,
+    body: ChatRequest,
+    background_tasks: BackgroundTasks,
+    retrieval_service: RetrievalService = Depends(get_retrieval_service),
+    db: Session = Depends(get_db),
+) -> ChatResponse:
+    logger.debug("Chat request received: %s ...", body.message[:50])
     start_time = time.time()
 
+    # Get tenant_id from the TenantAuthMiddleware
+    tenant_id = request.state.tenant_id
+
+    request_id = str(uuid.uuid4())
+
     try:
+        credit_service = CreditService()
+
         # Generate session ID if not provided
-        session_id = request.session_id or str(uuid.uuid4())
+        session_id = body.session_id or str(uuid.uuid4())
 
         # Detect language
-        language = nlp_engine.detect_language(request.message)
+        language = nlp_engine.detect_language(body.message)
 
         # Process message
-        intent, confidence = nlp_engine.classify_intent(request.message)
-        entities = nlp_engine.extract_entities(request.message, language)
+        intent, confidence = nlp_engine.classify_intent(body.message)
+        entities = nlp_engine.extract_entities(body.message, language)
 
         # Convert NumPy types to Python types
         intent = str(intent)
@@ -164,26 +257,107 @@ async def chat(request: ChatRequest):
             )
             intent = "low_confidence"
 
+        # Fetch tenant (used for system prompt and billing)
+        tenant = None
+        try:
+            tenant = db.get(Tenant, tenant_id)
+        except Exception:  # pylint: disable=broad-exception-caught
+            pass
+        tenant_system_prompt = (
+            (tenant.customization or {}).get("system_prompt") if tenant else None
+        )
+
+        if tenant and tenant.billing_mode == "credits":
+            try:
+                reserve_cost = credit_service.estimate_max_chat_cost(
+                    model_refs=[
+                        config.chat_model_primary,
+                        *config.chat_model_fallbacks,
+                    ],
+                    max_input_tokens=config.nlp["max_input_tokens"],
+                    max_output_tokens=config.chat_max_output_tokens,
+                    db=db,
+                )
+            except ModelRateNotFoundError:
+                raise HTTPException(status_code=500, detail="LLM model rate not found")
+
+            if tenant.balance_usd < reserve_cost:
+                raise HTTPException(status_code=402, detail="Insufficient credits")
+
+        retrieved_docs, retrieved_ids = retrieval_service.retrieve(
+            tenant_id=str(tenant_id),
+            query=body.message,
+            n_results=config.rag_n_results,
+        )
+
         # Generate response (now with context awareness)
-        thinking_block, response = conversation_manager.get_response(
-            intent, language, request.message, session_id, entities
+        thinking_block, response, should_summarize, llm_result = conversation_manager.get_response(
+            intent=intent,
+            language=language,
+            user_input=body.message,
+            tenant_id=tenant_id,
+            retrieved_docs=retrieved_docs,
+            session_id=session_id,
+            entities=entities,
+            system_prompt=tenant_system_prompt,
         )
 
+        # get_response() returns str | None; narrow to str before passing downstream
+        response = response or "I'm sorry, I couldn't generate a response."
+
+        # Billing: deduct credits before saving the conversation
+        if tenant and tenant.billing_mode == "credits":
+            try:
+                llm_model = credit_service.get_llm_model_for_response(
+                    llm_result.model_used,
+                    db,
+                )
+            except ModelRateNotFoundError:
+                raise HTTPException(status_code=500, detail="LLM model rate not found")
+
+            amount = credit_service.calculate_chat_cost(
+                input_tokens=llm_result.input_tokens,
+                output_tokens=llm_result.output_tokens,
+                llm_model=llm_model,
+            )
+            try:
+                credit_service.deduct_for_chat(
+                    tenant_id,
+                    amount,
+                    request_id,
+                    llm_model.id,
+                    db,
+                )
+            except InsufficientCreditsError:
+                raise HTTPException(status_code=402, detail="Insufficient credits")
+
         # Calculate response time
         response_time_ms = int((time.time() - start_time) * 1000)
 
         # Save conversation (now to database)
-        conversation_manager.save_conversation(
-            session_id,
-            request.message,
-            response,
-            intent,
-            confidence,
-            entities,
-            language,
-            response_time_ms,
+        message_id = conversation_manager.save_conversation(
+            session_id=session_id,
+            user_input=body.message,
+            bot_response=response,
+            intent=intent,
+            tenant_id=tenant_id,
+            confidence=confidence,
+            entities=entities,
+            language=language,
+            response_time_ms=response_time_ms,
+            retrieved_doc_ids=retrieved_ids,
+            should_summarize=should_summarize,
         )
 
+        if message_id:
+            background_tasks.add_task(
+                run_ai_verdict,
+                message_id=message_id,
+                user_query=body.message,
+                retrieved_docs=retrieved_docs,
+                bot_response=response,
+            )
+
         # Prepare response
         chat_response = ChatResponse(
             thinking=thinking_block,
@@ -208,12 +382,24 @@ async def chat(request: ChatRequest):
         logger.info("Chat response generated successfully for session %s", session_id)
         return chat_response
 
+    except HTTPException:
+        raise
+
+    except InputTooLargeError as e:
+        logger.error("Input Too Large error: %s: %s", type(e).__name__, str(e))
+        raise HTTPException(status_code=413, detail="Input too large") from e
+
     except Exception as e:
-        logger.error("Error processing chat request: %s", str(e), exc_info=True)
+        logger.error(
+            "Error processing chat request: %s: %s",
+            type(e).__name__,
+            str(e),
+            exc_info=True,
+        )
         raise HTTPException(status_code=500, detail="Internal server error") from e
 
 
-@app.get("/analytics/{session_id}")
+@api_router.get("/analytics/{session_id}")
 async def get_conversation_history(session_id: str):
     logger.debug("Analytics requested for session: %s", session_id)
 
@@ -237,7 +423,7 @@ async def get_conversation_history(session_id: str):
         raise HTTPException(status_code=500, detail="Internal server error") from e
 
 
-@app.get("/analytics/{session_id}/stats")
+@api_router.get("/analytics/{session_id}/stats")
 async def get_user_stats(session_id: str):
     logger.debug("User stats requested for session: %s", session_id)
 
@@ -257,6 +443,8 @@ async def get_user_stats(session_id: str):
         raise HTTPException(status_code=500, detail="Internal server error") from e
 
 
+app.include_router(api_router)
+
 # Serve frontend static files (HF Spaces has no separate nginx)
 # Must be mounted LAST so API routes above take precedence
 _frontend_dir = Path(__file__).parent.parent / "frontend"

app/middleware/tenant_auth.py

@@ -5,8 +5,7 @@ from sqlalchemy import select
 from starlette.middleware.base import BaseHTTPMiddleware
 from starlette.responses import JSONResponse
 
-from app.models.database import SessionLocal
-from app.models.smart_models import Tenant
+from app.models.database import SessionLocal, Tenant
 
 
 class TenantAuthMiddleware(BaseHTTPMiddleware):
@@ -23,6 +22,13 @@ class TenantAuthMiddleware(BaseHTTPMiddleware):
             request.method == "OPTIONS"
             or path in excluded_paths
             or "." in path.split("/")[-1]  # static files (.js, .css, .html, etc.)
+            or "/admin" in path  # admin auth and other admin endpoints use JWT
+            or "/billing" in path  # billing endpoints use JWT
+            or "/kb/entries" in path  # admin KB management → JWT auth
+            or "/kb/bulk-upload" in path # admin KB bulk upload
+            or "/kb/sync" in path # admin KB sync
+            or "/system-prompt" in path  # admin system-prompt management → JWT auth
+            or "/conversations" in path  # admin conversations → JWT auth
         ):
             return await call_next(request)
 
@@ -46,6 +52,11 @@ class TenantAuthMiddleware(BaseHTTPMiddleware):
                 status_code=403, content={"detail": "Tenant is inactive"}
             )
 
+        if tenant.is_blocked:  # type: ignore[truthy-bool]
+            return JSONResponse(
+                status_code=403, content={"detail": "Tenant is blocked"}
+            )
+
         # this makes `tenant_id` available to use. If we need to use other properties like `api_key` we need to explicitly set it this way.
         request.state.tenant_id = tenant.id
 

app/models/conversation_manager.py

@@ -1,20 +1,51 @@
-import json
 import re
 from datetime import datetime, timezone, timedelta
-from typing import Dict, List, Optional
-from sqlalchemy import func
+from typing import Optional
+import uuid
+from sqlalchemy import func, select
 from sqlalchemy.exc import SQLAlchemyError, OperationalError
-from groq import Groq
 
-# Absolute imports from project root
-from app.utils.config import get_logger
-from app.models.database import User, Conversation, Message, SessionLocal
+from app.utils.config import Config, get_logger
+from app.utils.token_counter import count_tokens
+from app.models.database import (
+    SummarizationJob,
+    User,
+    Conversation,
+    Message,
+    SessionLocal,
+)
 from app.models.information_extractor import InformationExtractor
+from app.services.llm_service import LLMResult, LLMService
+
+
+CONVERSATION_TIMEOUT_MINUTES = 30
+CONTEXT_MESSAGE_LIMIT = 5
+
+# Hierarchical Context Memory
+CHARS_PER_TOKEN = 4
+DB_FETCH_LIMIT = 100
+SUMMARIZATION_SYSTEM_PROMPT = (
+    "You are a conversation summarizer. You will be given a conversation history "
+    "between a user and an assistant. Summarize the key points of the conversation "
+    "in concise prose, covering both what the user asked and what the assistant "
+    "responded. Focus on topics, decisions, and information exchanged. "
+    "Do not include greetings or filler. Write in third person. "
+    "Keep the summary under 200 words."
+)
+
+RAG_SYSTEM_PROMPT = (
+    "Use the knowledge base context as the primary source of truth for any "
+    "business-specific question. If the answer is present in the knowledge base "
+    "context, answer from it directly and do not fall back to generic assistant "
+    "memory. If the context does not contain the answer, say that you could not "
+    "find it in the knowledge base and avoid guessing."
+)
 
 
 class ConversationManager:
-    def __init__(self, config=None):
+    def __init__(self, config: Config, llm_service: LLMService | None = None):
         self.config = config
+        self.llm_service = llm_service or LLMService()
         self.logger = get_logger(__name__)
         self.info_extractor = InformationExtractor()
 
@@ -25,13 +56,13 @@ class ConversationManager:
             max_history,
         )
 
-    def get_or_create_user(self, session_id: str) -> User:
+    def get_or_create_user(self, session_id: str, tenant_id: uuid.UUID) -> User:
         """Get existing user or create new one"""
         with SessionLocal() as db_session:
-            user = db_session.query(User).filter(User.session_id == session_id).first()
+            user = db_session.scalar(select(User).where(User.session_id == session_id))
 
             if not user:
-                user = User(session_id=session_id)
+                user = User(session_id=session_id, tenant_id=tenant_id)
                 db_session.add(user)
                 db_session.commit()
                 db_session.refresh(user)
@@ -41,6 +72,11 @@ class ConversationManager:
                 user.last_seen = datetime.now(timezone.utc)
                 db_session.commit()
 
+            # Reload all columns into memory before the session closes.
+            # After commit(), SQLAlchemy marks all attributes as expired.
+            # Without this, accessing user.preferred_language outside the
+            # `with` block triggers a lazy-load on a detached object → DetachedInstanceError.
+            db_session.refresh(user)
             return user
 
     def get_active_conversation(self, user: User) -> Optional[Conversation]:
@@ -49,15 +85,14 @@ class ConversationManager:
             # Refresh user object in this session
             user = db_session.merge(user)
 
-            conversation = (
-                db_session.query(Conversation)
-                .filter(
+            conversation = db_session.scalar(
+                select(Conversation)
+                .where(
                     Conversation.user_id == user.id,
                     Conversation.is_active  # pylint: disable=singleton-comparison
                     == True,
                 )  # cannot write `Conversation.is_active is True` here, because it's not a Python's `True``, but a SQLAlchemy's `InstrumentedAttribute`` (a special obj that represents the column). Using `is` will always turn False because they are not the same object in Memory
                 .order_by(Conversation.last_message_at.desc())
-                .first()
             )
 
             # If conversation is older than 30 minutes, consider it inactive
@@ -71,20 +106,20 @@ class ConversationManager:
                     last_message_utc = conversation.last_message_at
                 time_diff = datetime.now(timezone.utc) - last_message_utc
 
-                if time_diff > timedelta(minutes=30):
+                if time_diff > timedelta(minutes=CONVERSATION_TIMEOUT_MINUTES):
                     conversation.is_active = False
                     db_session.commit()
                     conversation = None
 
             return conversation
 
-    def create_conversation(self, user: User) -> Conversation:
+    def create_conversation(self, user: User, tenant_id: uuid.UUID) -> Conversation:
         """Create a new conversation for the user"""
         with SessionLocal() as db_session:
             # Refresh user object in this session
             user = db_session.merge(user)
 
-            conversation = Conversation(user_id=user.id)
+            conversation = Conversation(user_id=user.id, tenant_id=tenant_id)
             db_session.add(conversation)
             db_session.commit()
             db_session.refresh(conversation)
@@ -93,20 +128,19 @@ class ConversationManager:
             return conversation
 
     def get_conversation_context(
-        self, conversation: Conversation, limit: int = 5
-    ) -> List[Dict]:
+        self, conversation: Conversation, limit: int = CONTEXT_MESSAGE_LIMIT
+    ) -> list[dict]:
         """Get recent messages from conversation for context"""
         with SessionLocal() as db_session:
             # Refresh conversation object in this section
             conversation = db_session.merge(conversation)
 
-            messages = (
-                db_session.query(Message)
-                .filter(Message.conversation_id == conversation.id)
+            messages = db_session.scalars(
+                select(Message)
+                .where(Message.conversation_id == conversation.id)
                 .order_by(Message.timestamp.desc())
                 .limit(limit)
-                .all()
-            )
+            ).all()
 
             context = []
             for msg in reversed(messages):  # Reverse to get chronological order
@@ -115,7 +149,9 @@ class ConversationManager:
                         "user_input": msg.user_input,
                         "bot_response": msg.bot_response,
                         "intent": msg.intent,
-                        "timestamp": msg.timestamp.isoformat(),
+                        "timestamp": (
+                            msg.timestamp.isoformat() if msg.timestamp else None
+                        ),
                     }
                 )
             return context
@@ -137,7 +173,9 @@ class ConversationManager:
                 )
 
                 thinking_block = thinking_block_match.group(1)
-                bot_response = bot_response_match.group(1)
+                bot_response = (
+                    bot_response_match.group(1) if bot_response_match else content
+                )
             else:
                 bot_response = content
 
@@ -148,32 +186,70 @@ class ConversationManager:
         intent: str,
         language: str,
         user_input: str,
-        session_id: str = None,
-        entities: Dict = None,
-    ) -> str:
+        tenant_id: uuid.UUID,
+        retrieved_docs: Optional[list[str]] = None,
+        session_id: Optional[str] = None,
+        entities: Optional[dict] = None,
+        system_prompt: Optional[str] = None,
+    ) -> tuple[Optional[str], Optional[str], bool, LLMResult]:
         """Generate response with context awareness"""
 
         history_messages = []  # Initialize history messages for multi-turn context
+        summary_message = ""
+        should_summarize = False
+
+        base_prompt = system_prompt or self.config.system_prompt
+        system_content = base_prompt
+        if retrieved_docs:
+            knowledge_base_entries = "\n\n".join(retrieved_docs)
+            system_content = (
+                f"{base_prompt}\n\n"
+                f"{RAG_SYSTEM_PROMPT}\n\n"
+                f"Relevant knowledge base context:\n{knowledge_base_entries}"
+            )
+
+        messages = [
+            {"role": "system", "content": system_content},
+            {"role": "user", "content": user_input},
+        ]
 
         if session_id:
+            user = None
             try:
-                user = self.get_or_create_user(session_id)
+                user = self.get_or_create_user(
+                    session_id=session_id, tenant_id=tenant_id
+                )
                 conversation = self.get_active_conversation(user)
 
                 if conversation:
-                    context = self.get_conversation_context(conversation)
+                    context = self.build_hierarchical_context(
+                        conversation=conversation, system_prompt=system_content
+                    )
+
+                    if context["summary"] is not None:
+                        summary_message = context["summary"]
+                        should_summarize = True
 
                     # Prepare history messages for LLM context
-                    for conv in context:
-                        history_messages.extend(
-                            [
-                                {"role": "user", "content": conv["user_input"]},
-                                {
-                                    "role": "assistant",
-                                    "content": conv["bot_response"],
-                                },
-                            ]
-                        )
+                    history_messages.extend(context["verbatim"])
+
+                # Reassign messages now — before the language update — so a
+                # failure there can never prevent history from being passed.
+                messages = (
+                    [{"role": "system", "content": system_content}]
+                    + (
+                        [
+                            {
+                                "role": "system",
+                                "content": f"Summary of earlier conversation:\n{summary_message}",
+                            }
+                        ]
+                        if summary_message
+                        else []
+                    )
+                    + history_messages
+                    + [{"role": "user", "content": user_input}]
+                )
 
                 # Update user's preferred language
                 if user and user.preferred_language != language:
@@ -194,21 +270,19 @@ class ConversationManager:
                 )
                 raise
 
-        # Create LLM Client
-        client = Groq(api_key=self.config.llm_api_key)
+        count_tokens(messages)
 
-        llm_response = client.chat.completions.create(
-            messages=[{"role": "system", "content": self.config.system_prompt}]
-            + history_messages
-            + [{"role": "user", "content": user_input}],
-            model="qwen/qwen3-32b",
-        )
+        self.logger.info("History messages count: %d", len(history_messages))
 
-        thinking_block, response = self.parse_thinking_block(
-            llm_response.choices[0].message.content
+        llm_result = self.llm_service.complete_with_usage(
+            "chat-model",
+            messages,
+            max_tokens=self.config.chat_max_output_tokens,
         )
 
-        return thinking_block, response
+        thinking_block, response = self.parse_thinking_block(llm_result.content)
+
+        return thinking_block, response, should_summarize, llm_result
 
     def save_conversation(
         self,
@@ -216,20 +290,23 @@ class ConversationManager:
         user_input: str,
         bot_response: str,
         intent: str,
-        confidence: float = None,
-        entities: Dict = None,
+        tenant_id: uuid.UUID,
+        confidence: Optional[float] = None,
+        entities: Optional[dict] = None,
         language: str = "en",
-        response_time_ms: int = None,
-    ):
+        response_time_ms: Optional[int] = None,
+        retrieved_doc_ids: Optional[list[str]] = None,
+        should_summarize: bool = False,
+    ) -> Optional[uuid.UUID]:
         """Save conversation to database"""
         try:
             # Get or create user
-            user = self.get_or_create_user(session_id)
+            user = self.get_or_create_user(session_id=session_id, tenant_id=tenant_id)
 
             # Get or create active conversation
             conversation = self.get_active_conversation(user)
             if not conversation:
-                conversation = self.create_conversation(user)
+                conversation = self.create_conversation(user=user, tenant_id=tenant_id)
 
             # Create message record
             with SessionLocal() as db_session:
@@ -241,26 +318,39 @@ class ConversationManager:
                     conversation_id=conversation.id,
                     user_input=user_input,
                     bot_response=bot_response,
+                    retrieved_doc_ids=retrieved_doc_ids,
                     detected_language=language,
                     intent=intent,
                     confidence=confidence,
-                    entities=json.dumps(entities) if entities else None,
+                    entities=entities if entities else None,
                     response_time_ms=response_time_ms,
                 )
 
                 db_session.add(message)
 
+                # Check if SummarizationJob should be executed
+                if should_summarize:
+                    db_session.add(SummarizationJob(conversation_id=conversation.id))
+
                 # Update conversation stats
                 conversation.last_message_at = datetime.now(timezone.utc)
-                conversation.message_count += 1
+                conversation.last_message_preview = user_input[:255]
+                conversation.message_count = (conversation.message_count or 0) + 1
 
                 # Update user stats
-                user.total_messages += 1
+                user.total_messages = (user.total_messages or 0) + 1
                 user.last_seen = datetime.now(timezone.utc)
 
                 db_session.commit()
 
-            self.logger.debug("Conversation saved for session %s", session_id)
+                # Capture message.id
+                message_id: Optional[uuid.UUID] = message.id
+
+            self.logger.debug(
+                "Conversation saved for session: %s, under tenant: %s",
+                session_id,
+                str(tenant_id),
+            )
 
             # Test the extracted info (temp)
             extracted_info = self.info_extractor.extract_user_information(
@@ -268,78 +358,180 @@ class ConversationManager:
             )
             self.logger.info("Extracted info: %s", extracted_info)
 
-        except Exception as e:
+            return message_id
+
+        except Exception as e:  # pylint: disable=broad-exception-caught
             self.logger.error("Failed to save conversation: %s", str(e), exc_info=True)
 
-    def get_conversation_history(self, session_id: str) -> List[Dict]:
+    def get_conversation_history(self, session_id: str) -> list[dict]:
         """Get conversation history for a session (for analytics endpoint)"""
         try:
             with SessionLocal() as db_session:
-                user = (
-                    db_session.query(User).filter(User.session_id == session_id).first()
+                user = db_session.scalar(
+                    select(User).where(User.session_id == session_id)
                 )
                 if not user:
                     return []
 
-            messages = (
-                db_session.query(Message)
-                .join(Conversation)
-                .filter(Conversation.user_id == user.id)
-                .order_by(Message.timestamp.desc())
-                .limit(self.config.nlp["max_history"] if self.config else 50)
-                .all()
-            )
+                messages = db_session.scalars(
+                    select(Message)
+                    .join(Conversation)
+                    .where(Conversation.user_id == user.id)
+                    .order_by(Message.timestamp.desc())
+                    .limit(self.config.nlp["max_history"] if self.config else 50)
+                ).all()
 
             history = []
             for msg in reversed(messages):  # Reverse for chronological order
                 history.append(
                     {
-                        "timestamp": msg.timestamp.isoformat(),
+                        "timestamp": (
+                            msg.timestamp.isoformat() if msg.timestamp else None
+                        ),
                         "user_input": msg.user_input,
                         "bot_response": msg.bot_response,
                         "intent": msg.intent,
                         "confidence": msg.confidence,
                         "language": msg.detected_language,
-                        "entities": json.loads(msg.entities) if msg.entities else {},
+                        "entities": msg.entities if msg.entities else {},
                     }
                 )
 
             return history
 
-        except Exception as e:
+        except Exception as e:  # pylint: disable=broad-exception-caught
             self.logger.error(
                 "Failed to get conversation history: %s", str(e), exc_info=True
             )
             return []
 
-    def get_user_stats(self, session_id: str) -> Dict:
+    def get_user_stats(self, session_id: str) -> dict:
         """Get user statistics"""
         try:
             with SessionLocal() as db_session:
-                user = (
-                    db_session.query(User).filter(User.session_id == session_id).first()
+                user = db_session.scalar(
+                    select(User).where(User.session_id == session_id)
                 )
                 if not user:
                     return {}
 
                 # Get conversation count
-                conversation_count = (
-                    db_session.query(
+                conversation_count = db_session.scalar(
+                    select(
                         func.count(Conversation.id)  # pylint: disable=not-callable
-                    )
-                    .filter(Conversation.user_id == user.id)
-                    .scalar()
+                    ).where(Conversation.user_id == user.id)
                 )
 
                 return {
                     "session_id": user.session_id,
                     "preferred_language": user.preferred_language,
-                    "first_seen": user.first_seen.isoformat(),
-                    "last_seen": user.last_seen.isoformat(),
+                    "first_seen": (
+                        user.first_seen.isoformat() if user.first_seen else None
+                    ),
+                    "last_seen": user.last_seen.isoformat() if user.last_seen else None,
                     "total_messages": user.total_messages,
                     "total_conversations": conversation_count,
                 }
 
-        except Exception as e:
+        except Exception as e:  # pylint: disable=broad-exception-caught
             self.logger.error("Failed to get user stats: %s", str(e), exc_info=True)
             return {}
+
+    # Methods for Hierarchical Context Memory
+    def _split_verbatim_and_displaced(
+        self, messages: list[dict], budget: int
+    ) -> tuple[list[dict], list[dict]]:
+
+        token_count = 0
+        cutoff = 0
+
+        for msg in reversed(messages):
+            msg_tokens = (
+                len(msg["user_input"]) + len(msg["bot_response"])
+            ) // CHARS_PER_TOKEN
+
+            if token_count + msg_tokens > budget:
+                break
+
+            token_count += msg_tokens
+            cutoff += 1
+
+        verbatim = messages[len(messages) - cutoff :]
+        displaced = messages[: len(messages) - cutoff]
+
+        return verbatim, displaced
+
+    def build_hierarchical_context(
+        self,
+        conversation: Conversation,
+        system_prompt: str,
+    ) -> dict:
+
+        # Get a list of formatted messages
+        with SessionLocal() as db_session:
+            db_session.merge(conversation)
+
+            messages = db_session.scalars(
+                select(Message)
+                .where(Message.conversation_id == conversation.id)
+                .limit(DB_FETCH_LIMIT)
+                .order_by(Message.timestamp.asc())
+            ).all()
+
+            formatted_messages = [
+                {"user_input": msg.user_input, "bot_response": msg.bot_response}
+                for msg in messages
+            ]
+
+        # Calculate the budget tokens
+        max_input_tokens = self.config.nlp["max_input_tokens"]
+        context_reserve_tokens = self.config.nlp["context_reserve_tokens"]
+        system_tokens = count_tokens(
+            [{"role": "system", "content": system_prompt or self.config.system_prompt}]
+        )
+        summary_tokens = (
+            len(conversation.context_summary) // CHARS_PER_TOKEN
+            if conversation.context_summary
+            else 0
+        )
+
+        budget = (
+            max_input_tokens - system_tokens - summary_tokens - context_reserve_tokens
+        )
+
+        verbatim, displaced = self._split_verbatim_and_displaced(
+            messages=formatted_messages, budget=budget
+        )
+
+        summary = self._summarize_messages(displaced) if displaced else None
+        verbatim_formatted = [
+            item
+            for msg in verbatim
+            for item in (
+                {"role": "user", "content": msg["user_input"]},
+                {"role": "assistant", "content": msg["bot_response"]},
+            )
+        ]
+
+        return {"verbatim": verbatim_formatted, "summary": summary}
+
+    def _summarize_messages(self, messages: list[dict]) -> Optional[str]:
+
+        formatted_messages: list[dict] = []
+        for msg in messages:
+            formatted_messages.extend(
+                [
+                    {"role": "user", "content": msg["user_input"]},
+                    {"role": "assistant", "content": msg["bot_response"]},
+                ]
+            )
+
+        formatted_messages.insert(
+            0, {"role": "system", "content": SUMMARIZATION_SYSTEM_PROMPT}
+        )
+
+        content = self.llm_service.complete("summarization-model", formatted_messages)
+
+        _, response = self.parse_thinking_block(content)
+
+        return response

app/models/database.py

@@ -1,9 +1,17 @@
 # app/models/database.py
-from datetime import datetime, timezone
+from datetime import datetime, timezone, date
+from decimal import Decimal
+from typing import Optional, TYPE_CHECKING
 import uuid
 from sqlalchemy import (
+    JSON,
+    Index,
+    Numeric,
+    Date,
+    SmallInteger,
+    UniqueConstraint,
+    CheckConstraint,
     create_engine,
-    Column,
     Integer,
     String,
     Text,
@@ -12,19 +20,30 @@ from sqlalchemy import (
     Boolean,
     ForeignKey,
     MetaData,
+    text,
 )
 
 # from sqlalchemy.ext.declarative import declarative_base
-from sqlalchemy.orm import sessionmaker, relationship, declarative_base
-from sqlalchemy.dialects.postgresql import UUID
+from sqlalchemy.orm import (
+    sessionmaker,
+    relationship,
+    DeclarativeBase,
+    Mapped,
+    mapped_column,
+)
+from sqlalchemy.dialects.postgresql import UUID, JSONB
 
-# Absolute import from project root
 from app.utils.config import get_logger, config_manager
 
+if TYPE_CHECKING:
+    from app.models.smart_models import UserInsight, UserPreference, ConversationTopic
+
 config = config_manager.get_config()
 
 logger = get_logger(__name__)
 
+# Naming conventions for auto-generated constraint names (indexes, unique constraints,
+# foreign keys, etc.) so Alembic can reliably identify and diff them across migrations.
 convention = {
     "ix": "ix_%(column_0_label)s",
     "uq": "uq_%(table_name)s_%(column_0_name)s",
@@ -33,7 +52,21 @@ convention = {
     "pk": "pk_%(table_name)s",
 }
 
-Base = declarative_base(metadata=MetaData(naming_convention=convention))
+
+class Base(DeclarativeBase):
+    """Declarative base class for all ORM models.
+
+    Defined as a class (subclassing DeclarativeBase) rather than the legacy
+    `Base = declarative_base()` pattern because SQLAlchemy 2.0 requires the
+    class-based form to support `Mapped[...]` type annotations and
+    `mapped_column()`. The old factory function does not support these features.
+
+    All ORM models inherit from this class to register themselves with the
+    shared metadata and gain SQLAlchemy's mapping machinery.
+    """
+
+    metadata = MetaData(naming_convention=convention)
+
 
 engine = create_engine(
     config.database_url,
@@ -48,25 +81,33 @@ SessionLocal = sessionmaker(autocommit=False, autoflush=False, bind=engine)
 class User(Base):
     __tablename__ = "users"
 
-    id = Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
-    session_id = Column(String(255), unique=True, index=True)
-    preferred_language = Column(String(10), default="en")
-    first_seen = Column(DateTime, default=lambda: datetime.now(timezone.utc))
-    last_seen = Column(DateTime, default=lambda: datetime.now(timezone.utc))
-    total_messages = Column(Integer, default=0)
-    tenant_id = Column(UUID(as_uuid=True), ForeignKey("tenants.id"), nullable=True)
+    id: Mapped[uuid.UUID] = mapped_column(
+        UUID(as_uuid=True), primary_key=True, default=uuid.uuid4
+    )
+    session_id: Mapped[str] = mapped_column(String(255), unique=True, index=True)
+    tenant_id: Mapped[uuid.UUID] = mapped_column(
+        UUID(as_uuid=True), ForeignKey("tenants.id"), nullable=False
+    )
+    preferred_language: Mapped[Optional[str]] = mapped_column(String(10), default="en")
+    first_seen: Mapped[Optional[datetime]] = mapped_column(
+        DateTime, default=lambda: datetime.now(timezone.utc)
+    )
+    last_seen: Mapped[Optional[datetime]] = mapped_column(
+        DateTime, default=lambda: datetime.now(timezone.utc)
+    )
+    total_messages: Mapped[Optional[int]] = mapped_column(Integer, default=0)
 
     # Relationships
-    conversations = relationship(
+    tenant: Mapped["Tenant"] = relationship("Tenant", back_populates="users")
+    conversations: Mapped[list["Conversation"]] = relationship(
         "Conversation", back_populates="user", cascade="all, delete-orphan"
     )
-    preferences = relationship(
+    preferences: Mapped[list["UserPreference"]] = relationship(
         "UserPreference", back_populates="user", cascade="all, delete-orphan"
     )
-    insights = relationship(
+    insights: Mapped[list["UserInsight"]] = relationship(
         "UserInsight", back_populates="user", cascade="all, delete-orphan"
     )
-    tenant = relationship("Tenant", back_populates="users")
 
     def __repr__(self):
         return (
@@ -77,23 +118,37 @@ class User(Base):
 class Conversation(Base):
     __tablename__ = "conversations"
 
-    id = Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
-    user_id = Column(UUID(as_uuid=True), ForeignKey("users.id"), nullable=False)
-    started_at = Column(DateTime, default=lambda: datetime.now(timezone.utc))
-    last_message_at = Column(DateTime, default=lambda: datetime.now(timezone.utc))
-    message_count = Column(Integer, default=0)
-    is_active = Column(Boolean, default=True)
-    tenant_id = Column(UUID(as_uuid=True), ForeignKey("tenants.id"), nullable=True)
+    id: Mapped[uuid.UUID] = mapped_column(
+        UUID(as_uuid=True), primary_key=True, default=uuid.uuid4
+    )
+    user_id: Mapped[uuid.UUID] = mapped_column(
+        UUID(as_uuid=True), ForeignKey("users.id"), nullable=False
+    )
+    tenant_id: Mapped[uuid.UUID] = mapped_column(
+        UUID(as_uuid=True), ForeignKey("tenants.id"), nullable=True
+    )
+    started_at: Mapped[Optional[datetime]] = mapped_column(
+        DateTime, default=lambda: datetime.now(timezone.utc)
+    )
+    last_message_at: Mapped[Optional[datetime]] = mapped_column(
+        DateTime, default=lambda: datetime.now(timezone.utc)
+    )
+    last_message_preview: Mapped[Optional[str]] = mapped_column(
+        String(255), nullable=True
+    )
+    message_count: Mapped[Optional[int]] = mapped_column(Integer, default=0)
+    is_active: Mapped[Optional[bool]] = mapped_column(Boolean, default=True)
+    context_summary: Mapped[Optional[str]] = mapped_column(Text, nullable=True)
 
     # Relationship
-    user = relationship("User", back_populates="conversations")
-    messages = relationship(
+    user: Mapped["User"] = relationship("User", back_populates="conversations")
+    messages: Mapped[list["Message"]] = relationship(
         "Message", back_populates="conversation", cascade="all, delete-orphan"
     )
-    topics = relationship(
+    topics: Mapped[list["ConversationTopic"]] = relationship(
         "ConversationTopic", back_populates="conversation", cascade="all, delete-orphan"
     )
-    tenant = relationship("Tenant", back_populates="conversations")
+    tenant: Mapped["Tenant"] = relationship("Tenant", back_populates="conversations")
 
     def __repr__(self):
         return f"<Conversation(id={self.id}, user_id={self.user_id}, messages={self.message_count})>"
@@ -102,36 +157,258 @@ class Conversation(Base):
 class Message(Base):
     __tablename__ = "messages"
 
-    id = Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
-    conversation_id = Column(
+    id: Mapped[uuid.UUID] = mapped_column(
+        UUID(as_uuid=True), primary_key=True, default=uuid.uuid4
+    )
+    conversation_id: Mapped[uuid.UUID] = mapped_column(
         UUID(as_uuid=True), ForeignKey("conversations.id"), nullable=False
     )
 
     # Message content
-    user_input = Column(Text, nullable=False)
-    bot_response = Column(Text, nullable=False)
+    user_input: Mapped[str] = mapped_column(Text, nullable=False)
+    bot_response: Mapped[str] = mapped_column(Text, nullable=False)
 
     # NLP analysis
-    detected_language = Column(String(10))
-    intent = Column(String(100))
-    confidence = Column(Float)
-    entities = Column(Text)  # JSON string
+    retrieved_doc_ids: Mapped[Optional[list]] = mapped_column(
+        JSONB().with_variant(JSON(), "sqlite"), nullable=True
+    )
+    detected_language: Mapped[Optional[str]] = mapped_column(String(10))
+    intent: Mapped[Optional[str]] = mapped_column(String(100))
+    confidence: Mapped[Optional[float]] = mapped_column(Float)
+    entities: Mapped[Optional[dict]] = mapped_column(
+        JSONB().with_variant(JSON(), "sqlite")
+    )
 
     # Metadata
-    timestamp = Column(DateTime, default=lambda: datetime.now(timezone.utc))
-    response_time_ms = Column(Integer)  # How long the bot took to respond
+    timestamp: Mapped[Optional[datetime]] = mapped_column(
+        DateTime, default=lambda: datetime.now(timezone.utc)
+    )
+    response_time_ms: Mapped[Optional[int]] = mapped_column(
+        Integer
+    )  # How long the bot took to respond
+
+    # Feedback
+    user_feedback: Mapped[Optional[int]] = mapped_column(SmallInteger, nullable=True)
+    tenant_feedback: Mapped[Optional[int]] = mapped_column(SmallInteger, nullable=True)
+
+    # AI verdict scores (filled async by judge LLM after each response)
+    accuracy_score: Mapped[Optional[float]] = mapped_column(
+        Float, nullable=True
+    )  # factual correctness vs retrieved docs
+    relevance_score: Mapped[Optional[float]] = mapped_column(
+        Float, nullable=True
+    )  # how well the response addressed the user query
+    safety_score: Mapped[Optional[float]] = mapped_column(
+        Float, nullable=True
+    )  # did bot stay on-domain and avoid harmful/misleading claims (0.0 = unsafe, 1.0 = safe)
 
     # Relationships
-    conversation = relationship("Conversation", back_populates="messages")
+    conversation: Mapped["Conversation"] = relationship(
+        "Conversation", back_populates="messages"
+    )
 
     def __repr__(self):
         return f"<Message(id={self.id}, intent={self.intent}, confidence={self.confidence})>"
 
 
+class Tenant(Base):
+    __tablename__ = "tenants"
+    __table_args__ = (
+        CheckConstraint("billing_mode IN ('credits', 'subscription', 'gifted')", name="chk_tenant_billing_mode"),
+    )
+
+    id: Mapped[uuid.UUID] = mapped_column(
+        UUID(as_uuid=True), primary_key=True, default=uuid.uuid4
+    )
+    name: Mapped[str] = mapped_column(String(255), nullable=False)
+    website_url: Mapped[Optional[str]] = mapped_column(String(255), nullable=True)
+    api_key: Mapped[str] = mapped_column(String(255), unique=True, nullable=False)
+    is_active: Mapped[Optional[bool]] = mapped_column(Boolean, default=True)
+    created_at: Mapped[Optional[datetime]] = mapped_column(
+        DateTime, default=lambda: datetime.now(timezone.utc)
+    )
+    updated_at: Mapped[Optional[datetime]] = mapped_column(
+        DateTime,
+        default=lambda: datetime.now(timezone.utc),
+        onupdate=lambda: datetime.now(timezone.utc),
+    )
+    customization: Mapped[Optional[dict]] = mapped_column(
+        JSONB().with_variant(JSON(), "sqlite"), nullable=True
+    )
+
+    # Admin panel auth fields
+    owner_email: Mapped[Optional[str]] = mapped_column(
+        String(255), nullable=True, unique=True
+    )
+    google_id: Mapped[Optional[str]] = mapped_column(
+        String(255), nullable=True, unique=True
+    )
+    # Stores Fernet-encrypted TOTP seed — never the raw secret
+    totp_secret: Mapped[Optional[str]] = mapped_column(String(512), nullable=True)
+    totp_enabled: Mapped[Optional[bool]] = mapped_column(Boolean, default=False)
+    # Stores JSON array of SHA-256-hashed backup codes (plaintext shown once at setup)
+    backup_codes: Mapped[Optional[str]] = mapped_column(Text, nullable=True)
+
+    backup_email: Mapped[Optional[str]] = mapped_column(String(255), nullable=True)
+    phone: Mapped[Optional[str]] = mapped_column(String(20), nullable=True)
+
+    # Super-admin controls
+    is_blocked: Mapped[bool] = mapped_column(
+        Boolean, default=False, nullable=False, server_default="false"
+    )
+
+    # Billing
+    balance_usd: Mapped[Decimal] = mapped_column(
+        Numeric(20, 6), nullable=False, server_default="0"
+    )
+    billing_mode: Mapped[str] = mapped_column(
+        String(20), nullable=False, server_default="credits"
+    )
+
+    # Relationships
+    users: Mapped[list["User"]] = relationship("User", back_populates="tenant")
+    conversations: Mapped[list["Conversation"]] = relationship(
+        "Conversation", back_populates="tenant"
+    )
+    credit_events: Mapped[list["CreditEvent"]] = relationship(
+        "CreditEvent", back_populates="tenant"
+    )
+    auth_providers: Mapped[list["TenantAuthProvider"]] = relationship(
+        "TenantAuthProvider", back_populates="tenant", cascade="all, delete-orphan"
+    )
+
+    def __repr__(self):
+        return f"<Tenant(id={self.id} name={self.name})>"
+
+
+class LLMModel(Base):
+    __tablename__ = "llm_models"
+
+    id: Mapped[int] = mapped_column(Integer, primary_key=True, autoincrement=True)
+    provider: Mapped[str] = mapped_column(String(100), nullable=False)
+    model_name: Mapped[str] = mapped_column(String(255), nullable=False)
+    usd_per_1k_input: Mapped[Decimal] = mapped_column(Numeric(20, 6), nullable=False)
+    usd_per_1k_output: Mapped[Decimal] = mapped_column(Numeric(20, 6), nullable=False)
+    effective_from: Mapped[date] = mapped_column(Date, nullable=False)
+    effective_to: Mapped[Optional[date]] = mapped_column(Date, nullable=True)
+
+    def __repr__(self):
+        return f"<LLMModel(provider={self.provider}, model={self.model_name}, from={self.effective_from})>"
+
+
+class CreditEvent(Base):
+    __tablename__ = "credit_events"
+    __table_args__ = (
+        UniqueConstraint(
+            "tenant_id", "request_id", name="uq_credit_events_tenant_id_request_id"
+        ),
+    )
+
+    id: Mapped[uuid.UUID] = mapped_column(
+        UUID(as_uuid=True), primary_key=True, default=uuid.uuid4
+    )
+    tenant_id: Mapped[uuid.UUID] = mapped_column(
+        UUID(as_uuid=True), ForeignKey("tenants.id"), nullable=False, index=True
+    )
+    event_type: Mapped[str] = mapped_column(String(20), nullable=False)  # "deduct" | "topup"
+    amount: Mapped[Decimal] = mapped_column(Numeric(20, 6), nullable=False)
+    balance_after: Mapped[Decimal] = mapped_column(Numeric(20, 6), nullable=False)
+    note: Mapped[Optional[str]] = mapped_column(Text, nullable=True)
+    created_by: Mapped[Optional[str]] = mapped_column(String(100), nullable=True)
+    created_at: Mapped[datetime] = mapped_column(
+        DateTime, default=lambda: datetime.now(timezone.utc)
+    )
+    request_id: Mapped[Optional[str]] = mapped_column(String(255), nullable=True)
+    llm_model_id: Mapped[Optional[int]] = mapped_column(
+        Integer, ForeignKey("llm_models.id"), nullable=True
+    )
+
+    # Relationships
+    tenant: Mapped["Tenant"] = relationship("Tenant", back_populates="credit_events")
+    llm_model: Mapped[Optional["LLMModel"]] = relationship("LLMModel")
+
+    def __repr__(self):
+        return f"<CreditEvent(tenant_id={self.tenant_id}, type={self.event_type}, amount={self.amount})>"
+
+
+class TenantAuthProvider(Base):
+    __tablename__ = "tenant_auth_providers"
+    __table_args__ = (
+        UniqueConstraint("tenant_id", "provider", name="uq_tenant_auth_providers_tenant_id_provider"),
+        CheckConstraint("provider IN ('google', 'telegram')", name="chk_tenant_auth_providers_provider"),
+    )
+
+    id: Mapped[uuid.UUID] = mapped_column(
+        UUID(as_uuid=True), primary_key=True, default=uuid.uuid4
+    )
+    tenant_id: Mapped[uuid.UUID] = mapped_column(
+        UUID(as_uuid=True), ForeignKey("tenants.id", ondelete="CASCADE"), nullable=False, index=True
+    )
+    provider: Mapped[str] = mapped_column(String(20), nullable=False)
+    provider_user_id: Mapped[str] = mapped_column(String(255), nullable=False)
+    linked_at: Mapped[datetime] = mapped_column(
+        DateTime, default=lambda: datetime.now(timezone.utc)
+    )
+
+    tenant: Mapped["Tenant"] = relationship("Tenant", back_populates="auth_providers")
+
+    def __repr__(self):
+        return f"<TenantAuthProvider(tenant_id={self.tenant_id}, provider={self.provider})>"
+
+
+class SummarizationJob(Base):
+    __tablename__ = "summarization_jobs"
+    __table_args__ = (
+        Index(
+            "uq_summarization_jobs_pending",
+            "conversation_id",
+            unique=True,
+            postgresql_where=text("status = 'pending'"),
+        ),
+    )
+
+    id: Mapped[uuid.UUID] = mapped_column(
+        UUID(as_uuid=True),
+        primary_key=True,
+        default=uuid.uuid4,
+    )
+    conversation_id: Mapped[uuid.UUID] = mapped_column(
+        UUID(as_uuid=True),
+        ForeignKey("conversations.id"),
+        nullable=False,
+        index=True,
+    )
+    status: Mapped[str] = mapped_column(
+        String(20),
+        default="pending",
+    )
+    attempt_count: Mapped[int] = mapped_column(Integer, default=0)
+    created_at: Mapped[datetime] = mapped_column(
+        DateTime,
+        default=lambda: datetime.now(timezone.utc),
+    )
+    last_attempted_at: Mapped[Optional[datetime]] = mapped_column(DateTime)
+
+
+class Industry(Base):
+    __tablename__ = "industries"
+
+    id: Mapped[int] = mapped_column(Integer, primary_key=True, autoincrement=True)
+    code: Mapped[str] = mapped_column(String(100), unique=True, nullable=False)
+    display_name: Mapped[str] = mapped_column(String(100), nullable=False)
+    is_active: Mapped[bool] = mapped_column(Boolean, default=True, nullable=False)
+
+    def __repr__(self):
+        return f"<Industry(code={self.code}, display_name={self.display_name}, is_active={self.is_active})>"
+
+
 def init_database():
-    # create all tables in the database
-    Base.metadata.create_all(bind=engine)
-    logger.info("Database tables created successfully")
+    """Create all tables on SQLite (HF Spaces / tests). PostgreSQL uses Alembic migrations only."""
+    if engine.dialect.name == "sqlite":
+        Base.metadata.create_all(bind=engine)
+        logger.info("Database tables created successfully (SQLite)")
+    else:
+        logger.info("PostgreSQL detected — skipping create_all(); use 'alembic upgrade head' to apply migrations")
 
 
 def health_check():
@@ -149,6 +426,11 @@ def health_check():
 
 
 def get_db():
+    """FastAPI dependency that provides a database session per request.
+
+    Yields a session and guarantees it is closed after the request finishes,
+    whether or not an exception occurred. Inject with `db: Session = Depends(get_db)`.
+    """
     db = SessionLocal()
     try:
         yield db

app/models/information_extractor.py

@@ -50,7 +50,7 @@ class InformationExtractor:
         except OSError:
             return False
 
-    def _build_patterns(self) -> Dict[str, List[str]]:
+    def _build_patterns(self) -> dict[str, list[str] | dict[str, list[str]]]:
         """Build regex patterns for information extraction"""
         return {
             # Name patterns
@@ -157,14 +157,16 @@ class InformationExtractor:
         """Extract product interests and preferences"""
         interests = []
 
-        for category, patterns in self.patterns["product_mentions"].items():
-            for pattern in patterns:
-                if re.search(pattern, message, re.IGNORECASE):
-                    category_name = category.split("_", maxsplit=1)[
-                        0
-                    ]  # "product_x_y" -> "product", "x_y" -> "product"
-                    if category_name not in interests:
-                        interests.append(category_name)
+        product_mentions = self.patterns["product_mentions"]
+        if isinstance(product_mentions, dict):
+            for category, patterns in product_mentions.items():
+                for pattern in patterns:
+                    if re.search(pattern, message, re.IGNORECASE):
+                        category_name = category.split("_", maxsplit=1)[
+                            0
+                        ]  # "product_x_y" -> "product", "x_y" -> "product"
+                        if category_name not in interests:
+                            interests.append(category_name)
 
         # Alternative approach - direct pattern matching
         product_categories = {
@@ -336,6 +338,6 @@ class InformationExtractor:
 
         # Convert sets to lists for JSON serialization
         summary["topics_discussed"] = list(summary["topics_discussed"])
-        summary["user_interests"] = List(summary["user_interests"])
+        summary["user_interests"] = list(summary["user_interests"])
 
         return summary

app/models/smart_models.py

@@ -1,7 +1,19 @@
+"""ORM models for the smart layer — features that learn from and adapt to users.
+
+These models are separate from the core chatbot models (User, Conversation, Message)
+in database.py. They extend the system with intelligence: tracking what users prefer,
+what topics they discuss, what behavioral patterns emerge, and what knowledge the
+tenant has loaded into the system.
+
+All models inherit from Base defined in database.py and are imported in main.py to
+ensure their tables are registered with Base.metadata before init_database() runs.
+"""
+
 from datetime import datetime, timezone
+from typing import Optional, TYPE_CHECKING
 import uuid
 from sqlalchemy import (
-    Column,
+    CheckConstraint,
     String,
     Text,
     DateTime,
@@ -10,139 +22,252 @@ from sqlalchemy import (
     Boolean,
     ForeignKey,
 )
-from sqlalchemy.orm import relationship
+from sqlalchemy.orm import relationship, mapped_column, Mapped
 from sqlalchemy.dialects.postgresql import UUID
 
 # Absolute import from project root
 from app.models.database import Base
 
+if TYPE_CHECKING:
+    from app.models.database import User, Conversation
+
 
 class UserPreference(Base):
+    """Stores a single learned preference for a user.
+
+    Each row represents one thing the system has inferred about a user —
+    for example, their name, a product category they care about, or their
+    preferred communication style. A user can have many preferences, each
+    identified by a (preference_type, preference_key) pair.
+    """
+
     __tablename__ = "user_preferences"
 
-    id = Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
-    user_id = Column(UUID(as_uuid=True), ForeignKey("users.id"), nullable=False)
+    id: Mapped[uuid.UUID] = mapped_column(
+        UUID(as_uuid=True), primary_key=True, default=uuid.uuid4
+    )
+    user_id: Mapped[uuid.UUID] = mapped_column(
+        UUID(as_uuid=True), ForeignKey("users.id"), nullable=False
+    )
 
     # Preference details
-    preference_type = Column(
+    preference_type: Mapped[str] = mapped_column(
         String(50), nullable=False
     )  # e.g., "name", "product_interest", "communication_style"
-    preference_key = Column(
+    preference_key: Mapped[str] = mapped_column(
         String(100), nullable=False
     )  # e.g., "user_name", "preferred_products", "formality_level"
-    preference_value = Column(
+    preference_value: Mapped[str] = mapped_column(
         Text, nullable=False
     )  # e.g., "John", "toys,gifts", "casual"
 
     # Learning metadata
-    confidence_score = Column(
+    confidence_score: Mapped[Optional[float]] = mapped_column(
         Float, default=1.0
     )  # How confident the model is (0.0-1.0)
-    learned_from_messages = Column(
+    learned_from_messages: Mapped[Optional[int]] = mapped_column(
         Integer, default=1
     )  # Number of messages that taught us this
-    first_learned = Column(DateTime, default=lambda: datetime.now(timezone.utc))
-    last_updated = Column(DateTime, default=lambda: datetime.now(timezone.utc))
+    first_learned: Mapped[Optional[datetime]] = mapped_column(
+        DateTime, default=lambda: datetime.now(timezone.utc)
+    )
+    last_updated: Mapped[Optional[datetime]] = mapped_column(
+        DateTime, default=lambda: datetime.now(timezone.utc)
+    )
 
     # Relationship
-    user = relationship("User", back_populates="preferences")
+    user: Mapped["User"] = relationship("User", back_populates="preferences")
 
     def __repr__(self):
         return f"<UserPreference(id={self.id} user_id={self.user_id} {self.preference_key}={self.preference_value})>"
 
 
 class ConversationTopic(Base):
+    """Tracks what topics were discussed within a single conversation.
+
+    Each row tags a conversation with a topic (e.g. "product_inquiry") and
+    optional subtopic (e.g. "toys"). Used to understand what a conversation
+    was about without re-reading its messages, enabling routing, analytics,
+    and context-aware follow-ups.
+    """
+
     __tablename__ = "conversation_topics"
 
-    id = Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
-    conversation_id = Column(
+    id: Mapped[uuid.UUID] = mapped_column(
+        UUID(as_uuid=True), primary_key=True, default=uuid.uuid4
+    )
+    conversation_id: Mapped[uuid.UUID] = mapped_column(
         UUID(as_uuid=True), ForeignKey("conversations.id"), nullable=False
     )
 
     # Topic details
-    topic = Column(
+    topic: Mapped[str] = mapped_column(
         String(100), nullable=False
     )  # "product_inquiry", "order_management", "support"
-    subtopic = Column(String(100))  # "toys", "cancellation", "shipping_issue"
-    keywords = Column(Text)  # JSON list of relevant keywords
+    subtopic: Mapped[Optional[str]] = mapped_column(
+        String(100)
+    )  # "toys", "cancellation", "shipping_issue"
+    keywords: Mapped[Optional[str]] = mapped_column(
+        Text
+    )  # JSON list of relevant keywords
 
     # Topic metadata
-    first_mentioned = Column(DateTime, default=lambda: datetime.now(timezone.utc))
-    message_count = Column(Integer, default=1)  # How many messages discussed this topic
-    importance_score = Column(
+    first_mentioned: Mapped[Optional[datetime]] = mapped_column(
+        DateTime, default=lambda: datetime.now(timezone.utc)
+    )
+    message_count: Mapped[Optional[int]] = mapped_column(
+        Integer, default=1
+    )  # How many messages discussed this topic
+    importance_score: Mapped[Optional[float]] = mapped_column(
         Float, default=1.0
     )  # How important this topic was (0.0-1.0)
 
     # Relationships
-    conversation = relationship("Conversation", back_populates="topics")
+    conversation: Mapped["Conversation"] = relationship(
+        "Conversation", back_populates="topics"
+    )
 
     def __repr__(self):
         return f"<ConversationTopic(topic={self.topic}, subtopic={self.subtopic})>"
 
 
 class UserInsight(Base):
+    """Stores higher-level behavioral patterns derived from a user's history.
+
+    Unlike UserPreference (which records stated or inferred facts), UserInsight
+    records computed patterns — e.g. "most active in the morning", "asks complex
+    questions", "prefers toys category". Insights are recalculated over time and
+    can be deactivated when they are no longer supported by recent data.
+    """
+
     __tablename__ = "user_insights"
 
-    id = Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
-    user_id = Column(UUID(as_uuid=True), ForeignKey("users.id"), nullable=False)
+    id: Mapped[uuid.UUID] = mapped_column(
+        UUID(as_uuid=True), primary_key=True, default=uuid.uuid4
+    )
+    user_id: Mapped[uuid.UUID] = mapped_column(
+        UUID(as_uuid=True), ForeignKey("users.id"), nullable=False
+    )
 
     # Insight details
-    insight_type = Column(
+    insight_type: Mapped[str] = mapped_column(
         String(50), nullable=False
     )  # "behavior_pattern", "preference_trend", "interaction_style"
-    insight_key = Column(
+    insight_key: Mapped[str] = mapped_column(
         String(100), nullable=False
     )  # "most_active_time", "preferred_topics", "question_complexity"
-    insight_value = Column(Text, nullable=False)  # "morning", "toys,gifts", "simple"
-    insight_description = Column(Text)  # Human-readable explanation
+    insight_value: Mapped[str] = mapped_column(
+        Text, nullable=False
+    )  # "morning", "toys,gifts", "simple"
+    insight_description: Mapped[Optional[str]] = mapped_column(
+        Text
+    )  # Human-readable explanation
 
     # Analytics metadata
-    confidence_level = Column(Float, default=1.0)  # Statistical confidence (0.0-1.0)
-    based_on_messages = Column(Integer, default=0)  # Number of messages analyzed
-    last_calculated = Column(DateTime, default=lambda: datetime.now(timezone.utc))
-    is_active = Column(Boolean, default=True)  # Is this insight still relevant?
+    confidence_level: Mapped[Optional[float]] = mapped_column(
+        Float, default=1.0
+    )  # Statistical confidence (0.0-1.0)
+    based_on_messages: Mapped[Optional[int]] = mapped_column(
+        Integer, default=0
+    )  # Number of messages analyzed
+    last_calculated: Mapped[Optional[datetime]] = mapped_column(
+        DateTime, default=lambda: datetime.now(timezone.utc)
+    )
+    is_active: Mapped[Optional[bool]] = mapped_column(
+        Boolean, default=True
+    )  # Is this insight still relevant?
 
     # Relationships
-    user = relationship("User", back_populates="insights")
+    user: Mapped["User"] = relationship("User", back_populates="insights")
 
     def __repr__(self):
         return f"<UserInsight(id={self.id} user_id={self.user_id}, {self.insight_key}={self.insight_value})>"
 
 
 class KnowledgeBase(Base):
+    """Stores tenant-specific knowledge entries used to ground the chatbot's responses.
+
+    Each row is a plain-text passage belonging to a tenant. The chatbot retrieves
+    relevant entries at query time (RAG) and includes them in the LLM prompt so
+    responses are based on the tenant's actual content rather than general knowledge.
+    """
+
     __tablename__ = "knowledge_base"
 
-    id = Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
-    created_at = Column(DateTime, default=lambda: datetime.now(timezone.utc))
-    updated_at = Column(
+    id: Mapped[uuid.UUID] = mapped_column(
+        UUID(as_uuid=True), primary_key=True, default=uuid.uuid4
+    )
+    created_at: Mapped[Optional[datetime]] = mapped_column(
+        DateTime, default=lambda: datetime.now(timezone.utc)
+    )
+    updated_at: Mapped[Optional[datetime]] = mapped_column(
         DateTime,
         default=lambda: datetime.now(timezone.utc),
         onupdate=lambda: datetime.now(timezone.utc),
     )
-    tenant_id = Column(UUID(as_uuid=True), ForeignKey("tenants.id"), nullable=False)
-    entry = Column(Text, nullable=False)
+    tenant_id: Mapped[uuid.UUID] = mapped_column(
+        UUID(as_uuid=True), ForeignKey("tenants.id"), nullable=False
+    )
+    title: Mapped[str] = mapped_column(String(255), nullable=False, server_default="Untitled")
+    content: Mapped[str] = mapped_column(Text, nullable=False)
+    source_url: Mapped[Optional[str]] = mapped_column(String(500), nullable=True)
+    origin: Mapped[str] = mapped_column(String(20), nullable=False, server_default="manual")
+    is_indexed: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false")
+
+    __table_args__ = (
+        CheckConstraint("origin IN ('manual', 'synced')", name="kb_origin_valid"),
+    )
 
     def __repr__(self):
         return f"<KnowledgeBase(id={self.id} tenant_id={self.tenant_id})>"
 
 
-class Tenant(Base):
-    __tablename__ = "tenants"
+class SyncJob(Base):
+    """Tracks tenant-owned WordPress sync progress for async background jobs."""
 
-    id = Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
-    name = Column(String(255), nullable=False)
-    api_key = Column(String(255), unique=True, nullable=False)
-    is_active = Column(Boolean, default=True)
-    created_at = Column(DateTime, default=lambda: datetime.now(timezone.utc))
-    updated_at = Column(
+    __tablename__ = "sync_jobs"
+
+    id: Mapped[uuid.UUID] = mapped_column(
+        UUID(as_uuid=True), primary_key=True, default=uuid.uuid4
+    )
+    created_at: Mapped[Optional[datetime]] = mapped_column(
+        DateTime, default=lambda: datetime.now(timezone.utc)
+    )
+    updated_at: Mapped[Optional[datetime]] = mapped_column(
         DateTime,
         default=lambda: datetime.now(timezone.utc),
         onupdate=lambda: datetime.now(timezone.utc),
     )
+    tenant_id: Mapped[uuid.UUID] = mapped_column(
+        UUID(as_uuid=True), ForeignKey("tenants.id"), nullable=False
+    )
+    source_url: Mapped[str] = mapped_column(String(500), nullable=False)
+    rest_base: Mapped[str] = mapped_column(String(50), nullable=False, default="posts", server_default="posts")
+    status: Mapped[str] = mapped_column(String(30), nullable=False, server_default="queued")
+    total: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
+    processed: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
+    failed: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
+    created: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
+    updated: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
+    skipped: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0")
+    error_message: Mapped[Optional[str]] = mapped_column(Text, nullable=True)
 
-    # Relationships
-    users = relationship("User", back_populates="tenant")
-    conversations = relationship("Conversation", back_populates="tenant")
+    __table_args__ = (
+        CheckConstraint(
+            "status IN ('cancelled', 'completed', 'expired', 'failed', 'paused', 'queued', 'running')",
+            name="sync_jobs_status_valid",
+        ),
+        CheckConstraint("total >= 0", name="sync_jobs_total_nonnegative"),
+        CheckConstraint("processed >= 0", name="sync_jobs_processed_nonnegative"),
+        CheckConstraint("failed >= 0", name="sync_jobs_failed_nonnegative"),
+        CheckConstraint(
+            "processed + failed <= total",
+            name="sync_jobs_progress_within_total",
+        ),
+        CheckConstraint("created >= 0", name="sync_jobs_created_nonnegative"),
+        CheckConstraint("updated >= 0", name="sync_jobs_updated_nonnegative"),
+        CheckConstraint("skipped >= 0", name="sync_jobs_skipped_nonnegative"),
+    )
 
     def __repr__(self):
-        return f"<Tenant(id={self.id} name={self.name})>"
+        return f"<SyncJob(id={self.id} tenant_id={self.tenant_id} status={self.status})>"

app/routers/admin_apikey.py

@@ -0,0 +1,28 @@
+from fastapi import APIRouter, Depends, HTTPException
+from sqlalchemy import select
+from sqlalchemy.orm import Session
+from app.models.database import get_db, Tenant
+from app.dependencies.admin_auth import require_admin_jwt
+from app.schemas.admin_apikey import ApiKeyRotateResponse
+from app.utils.api_key import generate_api_key
+import uuid
+
+router = APIRouter(prefix="/admin", tags=["admin"])
+
+@router.post("/api-key/rotate", response_model=ApiKeyRotateResponse)
+def rotate_api_key(
+    tenant_id: uuid.UUID = Depends(require_admin_jwt),
+    db: Session = Depends(get_db)
+):
+    """
+    Generates a new API key, stores it hashed, and returns the plaintext exactly once.
+    """
+    tenant = db.scalar(select(Tenant).where(Tenant.id == tenant_id))
+    if not tenant:
+        raise HTTPException(status_code=404, detail="Tenant not found")
+    
+    plaintext, hashed = generate_api_key()
+    tenant.api_key = hashed
+    db.commit()
+    
+    return {"new_api_key": plaintext}

app/routers/admin_attention.py

@@ -0,0 +1,26 @@
+import uuid
+
+from fastapi import APIRouter, Depends
+from sqlalchemy import func, select
+from sqlalchemy.orm import Session
+
+from app.dependencies.admin_auth import require_admin_jwt
+from app.models.database import get_db
+from app.models.smart_models import KnowledgeBase
+from app.schemas.admin import AttentionResponse
+
+router = APIRouter(prefix="/admin", tags=["admin"])
+
+
+@router.get("/attention", response_model=AttentionResponse)
+def get_admin_attention(
+    tenant_id: uuid.UUID = Depends(require_admin_jwt),
+    db: Session = Depends(get_db),
+) -> AttentionResponse:
+    count = db.scalar(
+        select(func.count()).where(
+            KnowledgeBase.tenant_id == tenant_id,
+            KnowledgeBase.is_indexed == False,  # noqa: E712
+        )
+    )
+    return AttentionResponse(kb_has_unindexed=(count or 0) > 0)

app/routers/admin_auth.py

@@ -0,0 +1,446 @@
+import hashlib
+import hmac as hmac_mod
+import re
+import time
+import uuid
+from typing import Optional
+
+from fastapi import APIRouter, Depends, HTTPException, Request, Response
+from fastapi.responses import JSONResponse
+from pydantic import BaseModel
+from sqlalchemy import select
+from sqlalchemy.orm import Session
+
+from app.dependencies.admin_auth import require_admin_jwt
+from app.models.database import Tenant, TenantAuthProvider, get_db
+from app.services.google_auth import (
+    decode_partial_jwt,
+    issue_full_jwt,
+    issue_partial_jwt,
+    lookup_and_bind_tenant,
+    verify_google_token,
+)
+from app.services.totp_service import (
+    consume_backup_code,
+    encode_backup_codes,
+    encrypt_secret,
+    generate_backup_codes,
+    generate_totp_secret,
+    hash_backup_code,
+    make_provisioning_uri,
+    verify_totp_code,
+)
+from app.utils.config import get_config
+
+config = get_config()
+router = APIRouter(prefix="/admin/auth", tags=["admin-auth"])
+
+_TOTP_RE = re.compile(r"^\d{6}$")
+_BACKUP_RE = re.compile(r"^[0-9a-f]{16}$")
+
+
+class GoogleAuthRequest(BaseModel):
+    id_token: str
+
+
+class GoogleAuthResponse(BaseModel):
+    totp_enabled: bool
+    partial_token: Optional[str] = None
+
+
+class TOTPSetupResponse(BaseModel):
+    provisioning_uri: str
+    backup_codes: list[str]  # plaintext — shown once, not retrievable again
+
+
+class TOTPVerifyRequest(BaseModel):
+    code: str
+
+
+class TelegramAuthRequest(BaseModel):
+    id: int
+    first_name: str
+    last_name: Optional[str] = None
+    username: Optional[str] = None
+    photo_url: Optional[str] = None
+    auth_date: int
+    hash: str
+    phone_number: Optional[str] = None
+
+
+def _verify_telegram_hmac(data: dict, bot_token: str) -> bool:
+    """Verify Telegram Login Widget data using HMAC-SHA256."""
+    received_hash = data.get("hash", "")
+    check_pairs = sorted(
+        (k, str(v)) for k, v in data.items() if k != "hash" and v is not None
+    )
+    data_check_string = "\n".join(f"{k}={v}" for k, v in check_pairs)
+    secret_key = hashlib.sha256(bot_token.encode()).digest()
+    expected_hash = hmac_mod.new(
+        secret_key, data_check_string.encode(), hashlib.sha256
+    ).hexdigest()
+    return hmac_mod.compare_digest(expected_hash, received_hash)
+
+
+def _set_access_cookie(response: Response, tenant_id: uuid.UUID) -> None:
+    """Issue a full JWT and set it as an HttpOnly cookie on the response."""
+    access_token = issue_full_jwt(tenant_id, config.jwt_secret_key)
+    response.set_cookie(
+        key="access_token",
+        value=access_token,
+        httponly=True,
+        secure=config.env.name == "production",
+        samesite="strict",
+        path="/api",
+        max_age=86400,
+    )
+
+
+@router.post("/telegram")
+async def telegram_auth(body: TelegramAuthRequest, db: Session = Depends(get_db)) -> JSONResponse:
+    # 1. Guard: bot token must be configured
+    if not config.telegram_bot_token:
+        raise HTTPException(status_code=401, detail="Unauthorized")
+
+    # 2. Verify HMAC
+    data = body.model_dump(exclude_none=True)
+    if not _verify_telegram_hmac(data, config.telegram_bot_token):
+        raise HTTPException(status_code=401, detail="Unauthorized")
+
+    # 3. Check auth_date freshness (24 hours)
+    if time.time() - body.auth_date > 86400:
+        raise HTTPException(status_code=401, detail="Unauthorized")
+
+    telegram_user_id = str(body.id)
+    phone = body.phone_number
+
+    # 3. Try direct match via auth_providers
+    ap_row = db.scalars(
+        select(TenantAuthProvider).where(
+            TenantAuthProvider.provider == "telegram",
+            TenantAuthProvider.provider_user_id == telegram_user_id,
+        )
+    ).first()
+
+    if ap_row:
+        tenant = db.get(Tenant, ap_row.tenant_id)
+        if not tenant or not tenant.is_active:
+            raise HTTPException(status_code=401, detail="Unauthorized")
+    elif phone:
+        # 4. Try phone match
+        tenant = db.scalars(select(Tenant).where(Tenant.phone == phone)).first()
+        if not tenant or not tenant.is_active:
+            raise HTTPException(status_code=401, detail="Unauthorized")
+
+        # Check for collision: tenant already linked to a different auth method
+        has_google = tenant.google_id is not None
+        existing_ap = db.scalars(
+            select(TenantAuthProvider).where(TenantAuthProvider.tenant_id == tenant.id)
+        ).first()
+        if has_google or existing_ap:
+            existing_provider = existing_ap.provider if existing_ap else "google"
+            raise HTTPException(
+                status_code=409,
+                detail={"collision": True, "existing_provider": existing_provider},
+            )
+        # First Telegram login: auto-link
+        ap_row = None  # will be created below
+    else:
+        raise HTTPException(status_code=401, detail="Unauthorized")
+
+    # 5. Create auth_providers row if this is a first-time phone-matched login
+    if ap_row is None:
+        db.add(TenantAuthProvider(
+            tenant_id=tenant.id,
+            provider="telegram",
+            provider_user_id=telegram_user_id,
+        ))
+
+    # 6. Store phone on tenant if provided and not yet set
+    if phone and not tenant.phone:
+        tenant.phone = phone
+
+    db.commit()
+
+    # 7. Issue full JWT via cookie (set cookie on the returned JSONResponse, not the injected Response)
+    result = JSONResponse(content={"ok": True})
+    _set_access_cookie(result, tenant.id)
+    return result
+
+
+@router.post("/google", response_model=GoogleAuthResponse)
+async def google_auth(body: GoogleAuthRequest, response: Response, db: Session = Depends(get_db)) -> GoogleAuthResponse:
+    try:
+        google_payload = verify_google_token(body.id_token, config.google_client_id)
+    except ValueError:
+        raise HTTPException(status_code=401, detail="Unauthorized")
+
+    email = google_payload["email"]
+    google_id = google_payload["google_id"]
+
+    # Pre-check for collision: detect Telegram-only account before binding google_id
+    pre_tenant = db.scalars(select(Tenant).where(Tenant.owner_email == email)).first()
+    if pre_tenant and pre_tenant.google_id is None:
+        existing_ap = db.scalars(
+            select(TenantAuthProvider).where(
+                TenantAuthProvider.tenant_id == pre_tenant.id,
+                TenantAuthProvider.provider != "google",
+            )
+        ).first()
+        if existing_ap:
+            raise HTTPException(
+                status_code=409,
+                detail={"collision": True, "existing_provider": existing_ap.provider},
+            )
+
+    tenant = lookup_and_bind_tenant(email=email, google_id=google_id, db=db)
+
+    # Register Google in auth_providers on first successful login
+    has_google_ap = db.scalars(
+        select(TenantAuthProvider).where(
+            TenantAuthProvider.tenant_id == tenant.id,
+            TenantAuthProvider.provider == "google",
+        )
+    ).first()
+    if not has_google_ap:
+        db.add(TenantAuthProvider(
+            tenant_id=tenant.id,
+            provider="google",
+            provider_user_id=google_id,
+        ))
+        db.commit()
+
+    if not tenant.totp_enabled:
+        _set_access_cookie(response, tenant.id)
+        return GoogleAuthResponse(totp_enabled=False)
+
+    partial_token = issue_partial_jwt(tenant.id, config.jwt_secret_key)
+    return GoogleAuthResponse(totp_enabled=True, partial_token=partial_token)
+
+
+@router.post("/link/telegram")
+async def link_telegram(
+    body: TelegramAuthRequest,
+    tenant_id: uuid.UUID = Depends(require_admin_jwt),
+    db: Session = Depends(get_db),
+) -> JSONResponse:
+    if not config.telegram_bot_token:
+        raise HTTPException(status_code=401, detail="Unauthorized")
+
+    data = body.model_dump(exclude_none=True)
+    if not _verify_telegram_hmac(data, config.telegram_bot_token):
+        raise HTTPException(status_code=401, detail="Unauthorized")
+
+    if time.time() - body.auth_date > 86400:
+        raise HTTPException(status_code=401, detail="Unauthorized")
+
+    telegram_user_id = str(body.id)
+
+    # Already linked to this tenant
+    if db.scalars(
+        select(TenantAuthProvider).where(
+            TenantAuthProvider.tenant_id == tenant_id,
+            TenantAuthProvider.provider == "telegram",
+        )
+    ).first():
+        raise HTTPException(status_code=409, detail={"already_linked": True})
+
+    # Telegram ID already used by another tenant
+    if db.scalars(
+        select(TenantAuthProvider).where(
+            TenantAuthProvider.provider == "telegram",
+            TenantAuthProvider.provider_user_id == telegram_user_id,
+            TenantAuthProvider.tenant_id != tenant_id,
+        )
+    ).first():
+        raise HTTPException(status_code=409, detail={"already_linked": True, "used_by_other": True})
+
+    db.add(TenantAuthProvider(
+        tenant_id=tenant_id,
+        provider="telegram",
+        provider_user_id=telegram_user_id,
+    ))
+
+    tenant = db.get(Tenant, tenant_id)
+    if body.phone_number and tenant and not tenant.phone:
+        tenant.phone = body.phone_number
+
+    db.commit()
+    return JSONResponse(content={"linked": True})
+
+
+@router.post("/link/google")
+async def link_google(
+    body: GoogleAuthRequest,
+    tenant_id: uuid.UUID = Depends(require_admin_jwt),
+    db: Session = Depends(get_db),
+) -> JSONResponse:
+    try:
+        google_payload = verify_google_token(body.id_token, config.google_client_id)
+    except ValueError:
+        raise HTTPException(status_code=401, detail="Unauthorized")
+
+    google_id = google_payload["google_id"]
+    email = google_payload["email"]
+
+    # Already linked to this tenant
+    if db.scalars(
+        select(TenantAuthProvider).where(
+            TenantAuthProvider.tenant_id == tenant_id,
+            TenantAuthProvider.provider == "google",
+        )
+    ).first():
+        raise HTTPException(status_code=409, detail={"already_linked": True})
+
+    # Google ID already used by another tenant
+    if db.scalars(
+        select(TenantAuthProvider).where(
+            TenantAuthProvider.provider == "google",
+            TenantAuthProvider.provider_user_id == google_id,
+            TenantAuthProvider.tenant_id != tenant_id,
+        )
+    ).first():
+        raise HTTPException(status_code=409, detail={"already_linked": True, "used_by_other": True})
+
+    db.add(TenantAuthProvider(
+        tenant_id=tenant_id,
+        provider="google",
+        provider_user_id=google_id,
+    ))
+
+    tenant = db.get(Tenant, tenant_id)
+    if tenant:
+        if not tenant.owner_email:
+            tenant.owner_email = email
+        if not tenant.google_id:
+            tenant.google_id = google_id
+
+    db.commit()
+    return JSONResponse(content={"linked": True})
+
+
+@router.post("/totp/setup", response_model=TOTPSetupResponse)
+@router.post("/totp/enroll", response_model=TOTPSetupResponse)
+async def totp_setup(request: Request, db: Session = Depends(get_db)) -> TOTPSetupResponse:
+    auth_header = request.headers.get("Authorization", "")
+    if not auth_header.startswith("Bearer "):
+        raise HTTPException(status_code=401, detail="Unauthorized")
+
+    token = auth_header.removeprefix("Bearer ")
+
+    try:
+        tenant_id_str = decode_partial_jwt(token, config.jwt_secret_key)
+        tenant_id = uuid.UUID(tenant_id_str)
+    except (ValueError, AttributeError):
+        raise HTTPException(status_code=401, detail="Unauthorized")
+
+    tenant = db.get(Tenant, tenant_id)
+    if not tenant:
+        raise HTTPException(status_code=401, detail="Unauthorized")
+
+    if tenant.totp_enabled:
+        raise HTTPException(status_code=403, detail="TOTP already configured")
+
+    raw_secret = generate_totp_secret()
+    plain_codes = generate_backup_codes()
+
+    tenant.totp_secret = encrypt_secret(raw_secret, config.fernet_key)
+    tenant.backup_codes = encode_backup_codes([hash_backup_code(c) for c in plain_codes])
+    tenant.totp_enabled = False
+    db.commit()
+
+    return TOTPSetupResponse(
+        provisioning_uri=make_provisioning_uri(raw_secret, tenant.owner_email or ""),
+        backup_codes=plain_codes,
+    )
+
+
+@router.post("/totp/verify")
+async def totp_verify(body: TOTPVerifyRequest, request: Request, db: Session = Depends(get_db)) -> JSONResponse:
+    auth_header = request.headers.get("Authorization", "")
+    if not auth_header.startswith("Bearer "):
+        raise HTTPException(status_code=401, detail="Unauthorized")
+
+    token = auth_header.removeprefix("Bearer ")
+
+    try:
+        tenant_id_str = decode_partial_jwt(token, config.jwt_secret_key)
+        tenant_id = uuid.UUID(tenant_id_str)
+    except (ValueError, AttributeError):
+        raise HTTPException(status_code=401, detail="Unauthorized")
+
+    tenant = db.get(Tenant, tenant_id)
+    if not tenant or not tenant.totp_secret:
+        raise HTTPException(status_code=401, detail="Unauthorized")
+
+    code = body.code.strip().lower()
+    verified = False
+
+    if _TOTP_RE.match(code):
+        verified = verify_totp_code(tenant.totp_secret, code, config.fernet_key)
+    elif _BACKUP_RE.match(code) and tenant.backup_codes:
+        updated = consume_backup_code(tenant.backup_codes, code)
+        if updated is not None:
+            tenant.backup_codes = encode_backup_codes(updated)
+            verified = True
+
+    if not verified:
+        raise HTTPException(status_code=401, detail="Unauthorized")
+
+    tenant.totp_enabled = True
+    db.commit()
+
+    access_token = issue_full_jwt(tenant.id, config.jwt_secret_key)
+    response = JSONResponse(content={"ok": True})
+    # SameSite=Strict provides CSRF protection without a CSRF token
+    response.set_cookie(
+        key="access_token",
+        value=access_token,
+        httponly=True,
+        secure=config.env.name == "production",
+        samesite="strict",
+        path="/api",
+        max_age=86400,
+    )
+    return response
+
+
+@router.delete("/totp")
+async def totp_disable(
+    body: TOTPVerifyRequest,
+    tenant_id: uuid.UUID = Depends(require_admin_jwt),
+    db: Session = Depends(get_db),
+) -> JSONResponse:
+    tenant = db.get(Tenant, tenant_id)
+    if not tenant:
+        raise HTTPException(status_code=401, detail="Unauthorized")
+
+    if not tenant.totp_enabled or not tenant.totp_secret:
+        raise HTTPException(status_code=400, detail="TOTP not configured")
+
+    code = body.code.strip().lower()
+    verified = False
+
+    if _TOTP_RE.match(code):
+        verified = verify_totp_code(tenant.totp_secret, code, config.fernet_key)
+    elif _BACKUP_RE.match(code) and tenant.backup_codes:
+        updated = consume_backup_code(tenant.backup_codes, code)
+        if updated is not None:
+            tenant.backup_codes = encode_backup_codes(updated)
+            verified = True
+
+    if not verified:
+        raise HTTPException(status_code=401, detail="Unauthorized")
+
+    tenant.totp_enabled = False
+    tenant.totp_secret = None
+    tenant.backup_codes = None
+    db.commit()
+
+    return JSONResponse(content={"totp_enabled": False})
+
+
+@router.post("/logout")
+async def logout(response: Response) -> dict:
+    response.delete_cookie(key="access_token", path="/api")
+    return {"ok": True}

app/routers/admin_me.py

@@ -0,0 +1,23 @@
+from fastapi import APIRouter, Depends, HTTPException
+from sqlalchemy import select
+from sqlalchemy.orm import Session
+from app.models.database import get_db, Tenant
+from app.dependencies.admin_auth import require_admin_jwt
+from app.schemas.admin import AdminMeResponse
+import uuid
+
+router = APIRouter(prefix="/admin", tags=["admin"])
+
+@router.get("/me", response_model=AdminMeResponse)
+def get_admin_me(
+    tenant_id: uuid.UUID = Depends(require_admin_jwt),
+    db: Session = Depends(get_db)
+):
+    """
+    Returns the authenticated tenant's profile.
+    Used by the frontend to restore session state.
+    """
+    tenant = db.scalar(select(Tenant).where(Tenant.id == tenant_id))
+    if not tenant:
+        raise HTTPException(status_code=404, detail="Tenant not found")
+    return tenant

app/routers/admin_settings.py

@@ -0,0 +1,68 @@
+import uuid
+from fastapi import APIRouter, Depends, HTTPException
+from sqlalchemy import select
+from sqlalchemy.orm import Session
+from app.models.database import get_db, Tenant
+from app.dependencies.admin_auth import require_admin_jwt
+from app.schemas.admin_settings import GoLiveRequest, TenantSettingsUpdate, TenantResponse
+
+router = APIRouter(prefix="/admin", tags=["admin"])
+
+
+@router.patch("/go-live", response_model=TenantResponse)
+def patch_admin_go_live(
+    request: GoLiveRequest,
+    tenant_id: uuid.UUID = Depends(require_admin_jwt),
+    db: Session = Depends(get_db)
+):
+    tenant = db.scalar(select(Tenant).where(Tenant.id == tenant_id))
+    if not tenant:
+        raise HTTPException(status_code=404, detail="Tenant not found")
+
+    tenant.is_active = request.is_active
+    db.commit()
+    db.refresh(tenant)
+    return tenant
+
+
+@router.put("/settings", response_model=TenantResponse)
+def put_admin_settings(
+    request: TenantSettingsUpdate,
+    tenant_id: uuid.UUID = Depends(require_admin_jwt),
+    db: Session = Depends(get_db)
+):
+    tenant = db.scalar(select(Tenant).where(Tenant.id == tenant_id))
+    if not tenant:
+        raise HTTPException(status_code=404, detail="Tenant not found")
+
+    if request.name is not None:
+        tenant.name = request.name
+    if request.website_url is not None:
+        tenant.website_url = str(request.website_url)
+
+    if tenant.customization is None:
+        tenant.customization = {}
+
+    customization_update = {}
+    if request.industry is not None:
+        customization_update["industry"] = request.industry
+    if request.timezone is not None:
+        customization_update["timezone"] = request.timezone
+    if request.chatbot_name is not None:
+        customization_update["chatbot_name"] = request.chatbot_name
+    if request.chatbot_greeting is not None:
+        customization_update["chatbot_greeting"] = request.chatbot_greeting
+
+    if customization_update:
+        # SQLAlchemy requires a new dict object for JSON column mutation tracking
+        new_customization = dict(tenant.customization)
+        new_customization.update(customization_update)
+        tenant.customization = new_customization
+
+    if "backup_email" in request.model_fields_set:
+        tenant.backup_email = str(request.backup_email) if request.backup_email else None
+
+    db.commit()
+    db.refresh(tenant)
+    return tenant
+

app/routers/billing.py

@@ -0,0 +1,33 @@
+from fastapi import APIRouter, Depends, HTTPException
+from sqlalchemy import select
+from sqlalchemy.orm import Session
+from app.models.database import get_db, Tenant, CreditEvent
+from app.dependencies.admin_auth import require_admin_jwt
+from app.schemas.billing import BillingSummaryResponse
+import uuid
+
+router = APIRouter(prefix="/billing", tags=["billing"])
+
+@router.get("/summary", response_model=BillingSummaryResponse)
+def get_billing_summary(
+    tenant_id: uuid.UUID = Depends(require_admin_jwt),
+    db: Session = Depends(get_db)
+):
+    """
+    Returns current balance and credit event history for the billing page.
+    """
+    tenant = db.scalar(select(Tenant).where(Tenant.id == tenant_id))
+    if not tenant:
+        raise HTTPException(status_code=404, detail="Tenant not found")
+    
+    events = db.scalars(
+        select(CreditEvent)
+        .where(CreditEvent.tenant_id == tenant_id)
+        .order_by(CreditEvent.created_at.desc())
+        .limit(50)
+    ).all()
+
+    return BillingSummaryResponse(
+        balance_usd=tenant.balance_usd,
+        events=list(events)
+    )

app/routers/conversations.py

@@ -0,0 +1,190 @@
+from typing import Literal
+import uuid
+
+from fastapi import APIRouter, Depends, HTTPException, Request, Query
+from sqlalchemy import select
+from sqlalchemy.exc import SQLAlchemyError
+from sqlalchemy.orm import Session
+
+from app.dependencies.admin_auth import require_admin_jwt
+from app.models.database import Conversation, Message, get_db
+from app.schemas.conversations import (
+    ConversationSummary,
+    FeedbackRequest,
+    FeedbackResponse,
+    MessageItem,
+)
+
+
+router = APIRouter(tags=["conversations"])
+
+
+@router.get("/conversations", status_code=200, response_model=list[ConversationSummary])
+def list_conversation(
+    request: Request,
+    page: int = Query(default=1, ge=1),
+    page_size: int = Query(default=50, ge=1, le=100),
+    db: Session = Depends(get_db),
+    _: uuid.UUID = Depends(require_admin_jwt),
+) -> list[ConversationSummary]:
+
+    tenant_id = request.state.tenant_id
+
+    conversations = db.scalars(
+        select(Conversation)
+        .where(Conversation.tenant_id == tenant_id)
+        .order_by(Conversation.last_message_at.desc())
+        .limit(page_size)
+        .offset((page - 1) * page_size)
+    ).all()
+
+    result: list[ConversationSummary] = []
+    for conversation in conversations:
+        result.append(
+            ConversationSummary(
+                id=conversation.id,
+                started_at=conversation.started_at,
+                last_message_at=conversation.last_message_at,
+                message_count=(
+                    conversation.message_count if conversation.message_count else 0
+                ),
+                last_message=conversation.last_message_preview,
+            )
+        )
+
+    return result
+
+
+@router.get(
+    "/conversations/{conversation_id}/messages",
+    status_code=200,
+    response_model=list[MessageItem],
+)
+def get_conversation_messages(
+    request: Request,
+    conversation_id: uuid.UUID,
+    page: int = Query(default=1, ge=1),
+    page_size: int = Query(default=100, ge=1, le=200),
+    db: Session = Depends(get_db),
+    _: uuid.UUID = Depends(require_admin_jwt),
+) -> list[MessageItem]:
+
+    result: list[MessageItem] = []
+    tenant_id = request.state.tenant_id
+
+    conv = db.get(Conversation, conversation_id)
+
+    if conv is None or conv.tenant_id != tenant_id:
+        raise HTTPException(status_code=404, detail="Conversation doesn't exist")
+
+    messages = db.scalars(
+        select(Message)
+        .where(
+            Message.conversation_id == conversation_id,
+        )
+        .order_by(Message.timestamp.asc())
+        .limit(page_size)
+        .offset((page - 1) * page_size)
+    ).all()
+
+    for msg in messages:
+        result.append(
+            MessageItem(
+                role="user",
+                text=msg.user_input,
+                timestamp=msg.timestamp,
+                intent=msg.intent,
+            )
+        )
+        result.append(
+            MessageItem(
+                role="assistant",
+                text=msg.bot_response,
+                timestamp=msg.timestamp,
+                intent=msg.intent,
+            )
+        )
+
+    return result
+
+
+def _apply_feedback(
+    message_id: uuid.UUID,
+    score: Literal[-1, 1],
+    field: Literal["user_feedback", "tenant_feedback"],
+    tenant_id: uuid.UUID,
+    db: Session,
+) -> FeedbackResponse:
+
+    message = db.get(Message, message_id)
+
+    if not message:
+        raise HTTPException(status_code=404, detail="Message not found")
+
+    # Verify the message's conversation belongs to tenant_id
+    conv = db.get(Conversation, message.conversation_id)
+
+    if not conv or conv.tenant_id != tenant_id:
+        raise HTTPException(status_code=403, detail="Access denied")
+
+    try:
+        setattr(message, field, score)
+        db.commit()
+        db.refresh(message)
+    except SQLAlchemyError as e:
+        db.rollback()
+        raise HTTPException(status_code=500, detail="Database error") from e
+
+    return FeedbackResponse(
+        message_id=message.id, score=score
+    )  # use body.score because it's already validated as Literal[-1, 1]; if message.user_feedback, the system flags it as an error because its return type is `int | None`
+
+
+@router.patch(
+    "/messages/{message_id}/feedback/user",
+    status_code=200,
+    response_model=FeedbackResponse,
+)
+async def submit_user_feedback(
+    message_id: uuid.UUID,
+    body: FeedbackRequest,
+    request: Request,
+    db: Session = Depends(get_db),
+) -> FeedbackResponse:
+
+    tenant_id = request.state.tenant_id
+
+    feedback_response = _apply_feedback(
+        message_id=message_id,
+        score=body.score,
+        field="user_feedback",
+        tenant_id=tenant_id,
+        db=db,
+    )
+
+    return feedback_response
+
+
+@router.patch(
+    "/messages/{message_id}/feedback/tenant",
+    status_code=200,
+    response_model=FeedbackResponse,
+)
+async def submit_tenant_feedback(
+    message_id: uuid.UUID,
+    body: FeedbackRequest,
+    request: Request,
+    db: Session = Depends(get_db),
+) -> FeedbackResponse:
+
+    tenant_id = request.state.tenant_id
+
+    feedback_response = _apply_feedback(
+        message_id=message_id,
+        score=body.score,
+        field="tenant_feedback",
+        tenant_id=tenant_id,
+        db=db,
+    )
+
+    return feedback_response

app/routers/knowledge_base.py

@@ -1,31 +1,502 @@
+import urllib.parse
 import uuid
+import csv
+import io
+import json
+import time
+from collections import defaultdict
+from datetime import datetime, timezone
 
-from fastapi import APIRouter, Depends, HTTPException, Request
+from fastapi import APIRouter, BackgroundTasks, Depends, HTTPException, Query, Request, UploadFile, File
 from sqlalchemy import select
 from sqlalchemy.orm import Session
 
-from app.models.smart_models import KnowledgeBase
-from app.schemas.knowledge_base import KBEntryCreate, KBEntryResponse, KBEntryUpdate
-from app.models.database import get_db
+from app.dependencies.admin_auth import require_admin_jwt
+from app.models.smart_models import KnowledgeBase, SyncJob
+from app.schemas.knowledge_base import (
+    KBEntryCreate, KBEntryResponse, KBEntryUpdate,
+    BulkUploadResponse, BulkUploadError,
+    WPSyncRequest, WPSyncResponse, WPSyncPreviewResponse,
+    WPSyncPreviewEntry, WPSyncConfirmResponse, WPSyncStatusResponse,
+    WPSingleSyncRequest, WPSingleSyncResponse,
+    KBReindexResponse, WPSyncJobControlResponse, WPSyncResumeResponse,
+)
+from app.models.database import SessionLocal, get_db
+from app.services.dependencies import get_embedding_service, get_vector_store
+from app.services.embedding_service import EmbeddingService
+from app.services.kb_chunking import embed_entry_for_retrieval
+from app.services.vector_store import VectorStore
+from app.services.wordpress_sync import (
+    WordPressSyncError,
+    WordPressPostTypeError,
+    fetch_post_type_rest_base,
+    fetch_single_wordpress_post,
+    fetch_single_wordpress_post_by_url,
+    preview_wordpress_entries,
+    validate_public_https_url,
+)
+import redis as redis_lib
+from app.utils.sync_control import set_cancel_flag, set_pause_flag, clear_pause_flag
+from app.workers.kb_reindex import reindex_kb_for_tenant
+from app.workers.wordpress_sync import sync_wordpress_site
 
-router = APIRouter()
+router = APIRouter(dependencies=[Depends(require_admin_jwt)])
+
+_SYNC_RATE_LIMIT: dict[str, list[float]] = defaultdict(list)
+_SYNC_RATE_LIMIT_WINDOW_SECONDS = 60
+_SYNC_RATE_LIMIT_MAX_REQUESTS = 10
+
+_TITLE_SYNONYMS = frozenset({"title", "headline", "name", "subject", "heading"})
+
+
+def _extract_csv_fields(row: dict) -> tuple[str, str, str | None]:
+    """Extract (title, content, source_url) from a CSV row using smart column detection.
+
+    - Title: first column whose lowercased name is in _TITLE_SYNONYMS; value used as title.
+    - source_url: column whose lowercased name is exactly "source_url".
+    - Content: all remaining columns' non-empty values joined with newline.
+    """
+    title_col = next((k for k in row if k.lower().strip() in _TITLE_SYNONYMS), None)
+    title_val = (row[title_col] or "").strip() if title_col else ""
+    title = title_val if title_val else "Untitled"
+
+    source_url_col = next((k for k in row if k.lower().strip() == "source_url"), None)
+    source_url_val = (row[source_url_col] or "").strip() if source_url_col else ""
+    source_url = source_url_val if source_url_val else None
+
+    skip = {title_col, source_url_col} - {None}
+    parts = [v.strip() for k, v in row.items() if k not in skip and v and v.strip()]
+    content = "\n".join(parts)
+
+    return title, content, source_url
+
+
+def _embed_entry(
+    embedding_service: EmbeddingService,
+    vector_store: VectorStore,
+    tenant_id: str,
+    entry_id: str,
+    text: str,
+    metadata: dict[str, str | int | float | bool | None] | None = None,
+) -> None:
+    embed_entry_for_retrieval(
+        embedding_service=embedding_service,
+        vector_store=vector_store,
+        tenant_id=tenant_id,
+        entry_id=entry_id,
+        text=text,
+        metadata=metadata,
+    )
+    with SessionLocal() as db:
+        kb = db.get(KnowledgeBase, uuid.UUID(entry_id))
+        if kb is not None:
+            kb.is_indexed = True
+            db.commit()
+
+
+def _enforce_sync_rate_limit(tenant_id: uuid.UUID) -> None:
+    now = time.time()
+    key = str(tenant_id)
+    recent = [
+        timestamp
+        for timestamp in _SYNC_RATE_LIMIT[key]
+        if now - timestamp < _SYNC_RATE_LIMIT_WINDOW_SECONDS
+    ]
+    if len(recent) >= _SYNC_RATE_LIMIT_MAX_REQUESTS:
+        raise HTTPException(status_code=429, detail="WordPress sync rate limit exceeded")
+    recent.append(now)
+    _SYNC_RATE_LIMIT[key] = recent
+
+
+def _http_error_from_sync_error(exc: WordPressSyncError) -> HTTPException:
+    return HTTPException(status_code=400, detail=str(exc))
+
+
+@router.post("/kb/sync/wordpress/preview", status_code=200, response_model=WPSyncPreviewResponse)
+async def preview_wordpress_posts(
+    request: Request,
+    payload: WPSyncRequest,
+) -> WPSyncPreviewResponse:
+    tenant_id = request.state.tenant_id
+    _enforce_sync_rate_limit(tenant_id)
+    try:
+        site_url, entries, errors = await preview_wordpress_entries(payload.site_url, post_type=payload.post_type)
+    except WordPressPostTypeError as exc:
+        raise HTTPException(status_code=422, detail=str(exc)) from exc
+    except WordPressSyncError as exc:
+        raise _http_error_from_sync_error(exc) from exc
+
+    return WPSyncPreviewResponse(
+        site_url=site_url,
+        entries=[
+            WPSyncPreviewEntry(
+                title=entry.title,
+                content=entry.content,
+                source_url=entry.source_url,
+                warnings=entry.warnings,
+            )
+            for entry in entries
+        ],
+        errors=errors,
+    )
+
+
+@router.post("/kb/sync/wordpress/confirm", status_code=202, response_model=WPSyncConfirmResponse)
+async def confirm_wordpress_sync(
+    request: Request,
+    payload: WPSyncRequest,
+    db: Session = Depends(get_db),
+) -> WPSyncConfirmResponse:
+    tenant_id = request.state.tenant_id
+    _enforce_sync_rate_limit(tenant_id)
+    try:
+        site_url = validate_public_https_url(payload.site_url)
+    except WordPressSyncError as exc:
+        raise _http_error_from_sync_error(exc) from exc
+
+    try:
+        rest_base = await fetch_post_type_rest_base(site_url, payload.post_type)
+    except WordPressPostTypeError as exc:
+        raise HTTPException(status_code=422, detail=str(exc)) from exc
+    except WordPressSyncError as exc:
+        raise _http_error_from_sync_error(exc) from exc
+
+    existing_job = db.scalars(
+        select(SyncJob).where(
+            SyncJob.tenant_id == tenant_id,
+            SyncJob.source_url == site_url,
+            SyncJob.status.in_(["queued", "running"]),
+        )
+    ).first()
+    if existing_job is not None:
+        return WPSyncConfirmResponse(job_id=existing_job.id)
+
+    job = SyncJob(tenant_id=tenant_id, source_url=site_url, status="queued")
+    db.add(job)
+    db.commit()
+    db.refresh(job)
+    try:
+        sync_wordpress_site.apply_async(args=[str(tenant_id), str(job.id), site_url, rest_base, payload.force])
+    except Exception as exc:  # pylint: disable=broad-exception-caught
+        job.status = "failed"
+        job.error_message = f"Task enqueue failed: {type(exc).__name__}: {exc}"
+        job.updated_at = datetime.now(timezone.utc)
+        db.commit()
+        raise HTTPException(status_code=503, detail="Failed to enqueue WordPress sync job") from exc
+    return WPSyncConfirmResponse(job_id=job.id)
+
+
+@router.get("/kb/sync/status/{job_id}", status_code=200, response_model=WPSyncStatusResponse)
+def read_wordpress_sync_status(
+    request: Request,
+    job_id: uuid.UUID,
+    db: Session = Depends(get_db),
+) -> WPSyncStatusResponse:
+    tenant_id = request.state.tenant_id
+    job = db.scalars(
+        select(SyncJob).where(SyncJob.id == job_id, SyncJob.tenant_id == tenant_id)
+    ).first()
+    if job is None:
+        raise HTTPException(status_code=404, detail="Sync job not found")
+    return WPSyncStatusResponse(
+        job_id=job.id,
+        status=job.status,
+        source_url=job.source_url,
+        total=job.total,
+        processed=job.processed,
+        failed=job.failed,
+        error_message=job.error_message,
+        created=job.created,
+        updated=job.updated,
+        skipped=job.skipped,
+    )
+
+
+_CANCELLABLE_STATUSES = {"queued", "running"}
+_TERMINAL_STATUSES = {"completed", "failed", "cancelled", "expired"}
+
+
+@router.post("/kb/sync/wordpress/{job_id}/cancel", status_code=200, response_model=WPSyncJobControlResponse)
+def cancel_wordpress_sync(
+    request: Request,
+    job_id: uuid.UUID,
+    db: Session = Depends(get_db),
+) -> WPSyncJobControlResponse:
+    tenant_id = request.state.tenant_id
+    job = db.scalars(
+        select(SyncJob).where(SyncJob.id == job_id, SyncJob.tenant_id == tenant_id)
+    ).first()
+    if job is None:
+        raise HTTPException(status_code=404, detail="Sync job not found")
+    if job.status in _TERMINAL_STATUSES:
+        raise HTTPException(status_code=409, detail=f"Cannot cancel a job with status '{job.status}'")
+    try:
+        set_cancel_flag(str(job_id))
+    except redis_lib.RedisError as exc:
+        raise HTTPException(status_code=503, detail="Job control unavailable — Redis unreachable") from exc
+    job.status = "cancelled"
+    job.updated_at = datetime.now(timezone.utc)
+    db.commit()
+    return WPSyncJobControlResponse(job_id=job.id, status=job.status)
+
+
+@router.post("/kb/sync/wordpress/{job_id}/pause", status_code=202, response_model=WPSyncJobControlResponse)
+def pause_wordpress_sync(
+    request: Request,
+    job_id: uuid.UUID,
+    db: Session = Depends(get_db),
+) -> WPSyncJobControlResponse:
+    tenant_id = request.state.tenant_id
+    job = db.scalars(
+        select(SyncJob).where(SyncJob.id == job_id, SyncJob.tenant_id == tenant_id)
+    ).first()
+    if job is None:
+        raise HTTPException(status_code=404, detail="Sync job not found")
+    if job.status != "running":
+        raise HTTPException(status_code=409, detail=f"Cannot pause a job with status '{job.status}'")
+    try:
+        set_pause_flag(str(job_id))
+    except redis_lib.RedisError as exc:
+        raise HTTPException(status_code=503, detail="Job control unavailable — Redis unreachable") from exc
+    return WPSyncJobControlResponse(job_id=job.id, status="pausing")
+
+
+@router.post("/kb/sync/wordpress/{job_id}/resume", status_code=202, response_model=WPSyncResumeResponse)
+def resume_wordpress_sync(
+    request: Request,
+    job_id: uuid.UUID,
+    db: Session = Depends(get_db),
+) -> WPSyncResumeResponse:
+    tenant_id = request.state.tenant_id
+    original_job = db.scalars(
+        select(SyncJob).where(SyncJob.id == job_id, SyncJob.tenant_id == tenant_id)
+    ).first()
+    if original_job is None:
+        raise HTTPException(status_code=404, detail="Sync job not found")
+    if original_job.status != "paused":
+        raise HTTPException(status_code=409, detail=f"Cannot resume a job with status '{original_job.status}'")
+
+    # Clear any stale pause flag (non-fatal if Redis is unavailable)
+    try:
+        clear_pause_flag(str(job_id))
+    except redis_lib.RedisError:
+        pass  # Stale flag will expire on its own; proceed with resume
+
+    new_job = SyncJob(
+        tenant_id=tenant_id,
+        source_url=original_job.source_url,
+        rest_base=original_job.rest_base,
+        status="queued",
+    )
+    db.add(new_job)
+    db.flush()
+    try:
+        sync_wordpress_site.apply_async(
+            args=[str(tenant_id), str(new_job.id), original_job.source_url, original_job.rest_base, False]
+        )
+    except Exception as exc:  # pylint: disable=broad-exception-caught
+        new_job.status = "failed"
+        db.commit()
+        raise HTTPException(status_code=503, detail="Failed to enqueue resume job") from exc
+    db.commit()
+    return WPSyncResumeResponse(original_job_id=original_job.id, new_job_id=new_job.id)
+
+
+@router.post("/kb/sync/wordpress/single", status_code=200, response_model=WPSingleSyncResponse)
+async def sync_single_wordpress_post(
+    request: Request,
+    payload: WPSingleSyncRequest,
+    db: Session = Depends(get_db),
+    embedding_service: EmbeddingService = Depends(get_embedding_service),
+    vector_store: VectorStore = Depends(get_vector_store),
+) -> WPSingleSyncResponse:
+    tenant_id = request.state.tenant_id
+    _enforce_sync_rate_limit(tenant_id)
+
+    try:
+        site_url = validate_public_https_url(payload.site_url)
+        validate_public_https_url(payload.source_url)
+    except WordPressSyncError as exc:
+        raise _http_error_from_sync_error(exc) from exc
+
+    if urllib.parse.urlparse(payload.source_url).netloc != urllib.parse.urlparse(site_url).netloc:
+        raise HTTPException(status_code=400, detail="source_url must be on the same domain as site_url")
+
+    try:
+        rest_base = await fetch_post_type_rest_base(site_url, payload.post_type)
+    except WordPressPostTypeError as exc:
+        raise HTTPException(status_code=422, detail=str(exc)) from exc
+    except WordPressSyncError as exc:
+        raise _http_error_from_sync_error(exc) from exc
+
+    try:
+        wp_entry = await fetch_single_wordpress_post(site_url, payload.source_url, rest_base)
+    except WordPressSyncError as exc:
+        raise _http_error_from_sync_error(exc) from exc
+
+    kb = db.scalars(
+        select(KnowledgeBase).where(
+            KnowledgeBase.tenant_id == tenant_id,
+            KnowledgeBase.source_url == wp_entry.source_url,
+        )
+    ).first()
+
+    if kb is None:
+        kb = KnowledgeBase(
+            tenant_id=tenant_id,
+            title=wp_entry.title,
+            content=wp_entry.content,
+            source_url=wp_entry.source_url,
+            origin="synced",
+        )
+        db.add(kb)
+        db.commit()
+        db.refresh(kb)
+        _embed_entry(
+            embedding_service,
+            vector_store,
+            str(tenant_id),
+            str(kb.id),
+            kb.content,
+            metadata={"origin": kb.origin, "source_url": kb.source_url},
+        )
+        action = "created"
+    elif payload.force:
+        kb.title = wp_entry.title
+        kb.content = wp_entry.content
+        kb.is_indexed = False
+        kb.updated_at = datetime.now(timezone.utc)
+        db.commit()
+        db.refresh(kb)
+        _embed_entry(
+            embedding_service,
+            vector_store,
+            str(tenant_id),
+            str(kb.id),
+            kb.content,
+            metadata={"origin": kb.origin, "source_url": kb.source_url},
+        )
+        action = "updated"
+    else:
+        action = "skipped"
+
+    return WPSingleSyncResponse(action=action, entry=KBEntryResponse.model_validate(kb))
+
+
+@router.post("/kb/bulk-upload", status_code=201, response_model=BulkUploadResponse)
+def bulk_upload_entries(
+    request: Request,
+    background_tasks: BackgroundTasks,
+    file: UploadFile = File(...),
+    db: Session = Depends(get_db),
+    embedding_service: EmbeddingService = Depends(get_embedding_service),
+    vector_store: VectorStore = Depends(get_vector_store),
+) -> BulkUploadResponse:
+    tenant_id = request.state.tenant_id
+    created_count = 0
+    errors = []
+
+    parsed_data = []
+    filename = file.filename or ""
+
+    content_type = file.content_type or ""
+    is_csv = filename.endswith(".csv") or content_type == "text/csv"
+    is_json = filename.endswith(".json") or content_type == "application/json"
+
+    if not is_csv and not is_json:
+        raise HTTPException(status_code=400, detail="Unsupported file format. Use CSV or JSON.")
+
+    try:
+        content = file.file.read()
+        if is_csv:
+            parsed_data = list(csv.DictReader(io.StringIO(content.decode("utf-8"))))
+        else:
+            parsed_data = json.loads(content.decode("utf-8"))
+            if not isinstance(parsed_data, list):
+                raise ValueError("JSON must be an array of objects")
+    except (UnicodeDecodeError, ValueError) as e:
+        raise HTTPException(status_code=400, detail=f"Failed to parse file: {str(e)}")
+
+    valid_entries = []
+
+    for idx, row in enumerate(parsed_data):
+        row_num = idx + 1
+        if not isinstance(row, dict):
+            errors.append(BulkUploadError(row=row_num, reason="Row is not an object/dictionary"))
+            continue
+
+        if is_csv:
+            row_title, row_content, row_source_url = _extract_csv_fields(row)
+        else:
+            row_content = row.get("content")
+            row_title = row.get("title") or "Untitled"
+            row_source_url = row.get("source_url") or None
+
+        if not row_content:
+            errors.append(BulkUploadError(row=row_num, reason="Missing 'content' field"))
+            continue
+
+        kb = KnowledgeBase(
+            tenant_id=tenant_id,
+            content=row_content,
+            title=row_title,
+            source_url=row_source_url,
+            origin="manual",
+        )
+        valid_entries.append(kb)
+
+    if valid_entries:
+        db.add_all(valid_entries)
+        db.commit()
+        for kb in valid_entries:
+            db.refresh(kb)
+            background_tasks.add_task(
+                _embed_entry,
+                embedding_service,
+                vector_store,
+                str(kb.tenant_id),
+                str(kb.id),
+                kb.content,
+                {"origin": kb.origin, "source_url": kb.source_url},
+            )
+            created_count += 1
+
+    return BulkUploadResponse(created=created_count, errors=errors)
 
 
 @router.post("/kb/entries", status_code=201, response_model=KBEntryResponse)
 def create_entry(
-    request: Request, entry: KBEntryCreate, db: Session = Depends(get_db)
+    request: Request,
+    entry: KBEntryCreate,
+    background_tasks: BackgroundTasks,
+    db: Session = Depends(get_db),
+    embedding_service: EmbeddingService = Depends(get_embedding_service),
+    vector_store: VectorStore = Depends(get_vector_store),
 ) -> KBEntryResponse:
 
     knowledge_base = KnowledgeBase()
     knowledge_base.tenant_id = (
         request.state.tenant_id
     )  # get the tenant_id from the middleware
-    knowledge_base.entry = entry.entry  # type: ignore[truthy-bool]
+    knowledge_base.title = entry.title
+    knowledge_base.content = entry.content
+    knowledge_base.source_url = entry.source_url
+    knowledge_base.origin = "manual"
     db.add(knowledge_base)
     db.commit()
     db.refresh(knowledge_base)
 
-    return knowledge_base
+    background_tasks.add_task(
+        _embed_entry,
+        embedding_service,
+        vector_store,
+        str(knowledge_base.tenant_id),
+        str(knowledge_base.id),
+        knowledge_base.content,
+        {"origin": knowledge_base.origin, "source_url": knowledge_base.source_url},
+    )
+
+    return KBEntryResponse.model_validate(knowledge_base)
 
 
 @router.get("/kb/entries/{entry_id}", status_code=200, response_model=KBEntryResponse)
@@ -43,17 +514,23 @@ def read_entry(
     if kb is None:
         raise HTTPException(status_code=404, detail="Entry not found")
 
-    return kb
+    return KBEntryResponse.model_validate(kb)
 
 
 @router.get("/kb/entries", status_code=200, response_model=list[KBEntryResponse])
 def read_entries(
-    request: Request, db: Session = Depends(get_db)
+    request: Request,
+    page: int = Query(default=1, ge=1),
+    page_size: int = Query(default=50, ge=1, le=100),
+    db: Session = Depends(get_db),
 ) -> list[KBEntryResponse]:
 
     tenant_id = request.state.tenant_id
     kb_entries = db.scalars(
-        select(KnowledgeBase).where(KnowledgeBase.tenant_id == tenant_id)
+        select(KnowledgeBase)
+        .where(KnowledgeBase.tenant_id == tenant_id)
+        .limit(page_size)
+        .offset((page - 1) * page_size)
     ).all()
 
     return kb_entries  # type: ignore[truthy-bool]
@@ -64,7 +541,10 @@ def update_entry(
     request: Request,
     entry_id: uuid.UUID,
     entry: KBEntryUpdate,
+    background_tasks: BackgroundTasks,
     db: Session = Depends(get_db),
+    embedding_service: EmbeddingService = Depends(get_embedding_service),
+    vector_store: VectorStore = Depends(get_vector_store),
 ) -> KBEntryResponse:
 
     tenant_id = request.state.tenant_id
@@ -76,16 +556,43 @@ def update_entry(
 
     if kb is None:
         raise HTTPException(status_code=404, detail="Entry not found")
-    kb.entry = entry.entry  # type: ignore[truthy-bool]
+
+    if kb.origin == "synced":
+        raise HTTPException(
+            status_code=409,
+            detail="Synced entries are read-only through the normal KB update endpoint",
+        )
+
+    if entry.title is not None:
+        kb.title = entry.title
+    if entry.content is not None:
+        kb.content = entry.content
+    if "source_url" in entry.model_fields_set:
+        kb.source_url = entry.source_url
+    kb.is_indexed = False
+
     db.commit()
     db.refresh(kb)
 
-    return kb
+    background_tasks.add_task(
+        _embed_entry,
+        embedding_service,
+        vector_store,
+        str(kb.tenant_id),
+        str(kb.id),
+        kb.content,
+        {"origin": kb.origin, "source_url": kb.source_url},
+    )
+
+    return KBEntryResponse.model_validate(kb)
 
 
 @router.delete("/kb/entries/{entry_id}", status_code=204)
 def delete_entry(
-    request: Request, entry_id: uuid.UUID, db: Session = Depends(get_db)
+    request: Request,
+    entry_id: uuid.UUID,
+    db: Session = Depends(get_db),
+    vector_store: VectorStore = Depends(get_vector_store),
 ) -> None:
 
     tenant_id = request.state.tenant_id
@@ -100,3 +607,67 @@ def delete_entry(
 
     db.delete(kb)
     db.commit()
+
+    vector_store.delete_docs(tenant_id=str(tenant_id), entry_ids=[str(entry_id)])
+
+
+@router.post("/kb/entries/{entry_id}/resync", status_code=200, response_model=KBEntryResponse)
+async def resync_entry(
+    request: Request,
+    entry_id: uuid.UUID,
+    background_tasks: BackgroundTasks,
+    db: Session = Depends(get_db),
+    embedding_service: EmbeddingService = Depends(get_embedding_service),
+    vector_store: VectorStore = Depends(get_vector_store),
+) -> KBEntryResponse:
+    tenant_id = request.state.tenant_id
+
+    kb = db.scalars(
+        select(KnowledgeBase).where(
+            KnowledgeBase.id == entry_id, KnowledgeBase.tenant_id == tenant_id
+        )
+    ).first()
+    if kb is None:
+        raise HTTPException(status_code=404, detail="Entry not found")
+
+    if kb.origin != "synced":
+        raise HTTPException(status_code=400, detail="Only synced entries can be resynced")
+
+    if kb.source_url is None:
+        raise HTTPException(status_code=400, detail="Entry has no source URL to resync from")
+
+    try:
+        validate_public_https_url(kb.source_url)
+    except WordPressSyncError as exc:
+        raise HTTPException(status_code=400, detail=str(exc)) from exc
+
+    try:
+        wp_entry = await fetch_single_wordpress_post_by_url(kb.source_url)
+    except WordPressSyncError as exc:
+        raise HTTPException(status_code=400, detail=str(exc)) from exc
+
+    kb.title = wp_entry.title
+    kb.content = wp_entry.content
+    kb.is_indexed = False
+    kb.updated_at = datetime.now(timezone.utc)
+    db.commit()
+    db.refresh(kb)
+
+    background_tasks.add_task(
+        _embed_entry,
+        embedding_service,
+        vector_store,
+        str(kb.tenant_id),
+        str(kb.id),
+        kb.content,
+        {"origin": kb.origin, "source_url": kb.source_url},
+    )
+
+    return KBEntryResponse.model_validate(kb)
+
+
+@router.post("/kb/reindex", status_code=202, response_model=KBReindexResponse)
+def trigger_reindex(request: Request) -> KBReindexResponse:
+    tenant_id = request.state.tenant_id
+    task = reindex_kb_for_tenant.delay(str(tenant_id))
+    return KBReindexResponse(task_id=task.id)

app/routers/super_admin_auth.py

@@ -0,0 +1,58 @@
+from fastapi import APIRouter, Depends, HTTPException, Response
+from fastapi.responses import JSONResponse
+from pydantic import BaseModel
+
+from app.dependencies.super_admin_auth import require_super_admin_jwt
+from app.services.google_auth import (
+    SUPER_ADMIN_TOKEN_EXPIRE_HOURS,
+    issue_super_admin_jwt,
+    verify_google_token,
+)
+from app.utils.config import get_config
+
+config = get_config()
+router = APIRouter(prefix="/super-admin", tags=["super-admin-auth"])
+
+_TOKEN_MAX_AGE = SUPER_ADMIN_TOKEN_EXPIRE_HOURS * 3600
+
+
+class SuperAdminGoogleAuthRequest(BaseModel):
+    id_token: str
+
+
+@router.post("/auth/google")
+async def super_admin_google_auth(body: SuperAdminGoogleAuthRequest) -> JSONResponse:
+    try:
+        google_payload = verify_google_token(body.id_token, config.google_client_id)
+    except ValueError:
+        raise HTTPException(status_code=401, detail="Unauthorized")
+
+    email = google_payload.get("email")
+    if not email:
+        raise HTTPException(status_code=401, detail="Unauthorized")
+    if email.lower() not in [e.lower() for e in config.super_admin_emails]:
+        raise HTTPException(status_code=403, detail="Forbidden")
+
+    token = issue_super_admin_jwt(email, config.jwt_secret_key)
+    response = JSONResponse(content={"ok": True})
+    response.set_cookie(
+        key="super_admin_token",
+        value=token,
+        httponly=True,
+        secure=config.env.name == "production",
+        samesite="lax",
+        path="/api",
+        max_age=_TOKEN_MAX_AGE,
+    )
+    return response
+
+
+@router.post("/auth/logout")
+async def super_admin_logout(response: Response) -> dict:
+    response.delete_cookie(key="super_admin_token", path="/api")
+    return {"ok": True}
+
+
+@router.get("/me")
+async def super_admin_me(email: str = Depends(require_super_admin_jwt)) -> dict:
+    return {"email": email}

app/routers/super_admin_rates.py

@@ -0,0 +1,116 @@
+# app/routers/super_admin_rates.py
+"""Super-admin endpoints for managing LLM model rates.
+
+GET  /super-admin/llm-rates  — returns active rates and full history
+POST /super-admin/llm-rates  — adds a new rate, closes the previous active rate
+"""
+from datetime import date
+from decimal import Decimal
+from typing import Optional
+
+from fastapi import APIRouter, Depends, HTTPException
+from pydantic import BaseModel
+from sqlalchemy import select
+from sqlalchemy.orm import Session
+
+from app.dependencies.super_admin_auth import require_super_admin_jwt
+from app.models.database import LLMModel, get_db
+
+router = APIRouter(prefix="/super-admin", tags=["super-admin-rates"])
+
+
+# ---------------------------------------------------------------------------
+# Pydantic schemas
+# ---------------------------------------------------------------------------
+
+
+class LLMRateOut(BaseModel):
+    """Serialises one row of llm_models using only columns that exist in the DB."""
+
+    id: int
+    provider: str
+    model_name: str
+    usd_per_1k_input: Decimal
+    usd_per_1k_output: Decimal
+    effective_from: date
+    effective_to: Optional[date]
+
+    model_config = {"from_attributes": True}
+
+
+class LLMRatesResponse(BaseModel):
+    active: list[LLMRateOut]
+    history: list[LLMRateOut]
+
+
+class LLMRateCreate(BaseModel):
+    provider: str
+    model_name: str
+    usd_per_1k_input: Decimal
+    usd_per_1k_output: Decimal
+
+
+# ---------------------------------------------------------------------------
+# GET /llm-rates
+# ---------------------------------------------------------------------------
+
+
+@router.get("/llm-rates", response_model=LLMRatesResponse)
+def list_llm_rates(
+    _email: str = Depends(require_super_admin_jwt),
+    db: Session = Depends(get_db),
+) -> LLMRatesResponse:
+    """Return all active rates and the full rate history."""
+    all_rates = db.execute(
+        select(LLMModel).order_by(LLMModel.effective_from.desc())
+    ).scalars().all()
+
+    active = [r for r in all_rates if r.effective_to is None]
+    history = list(all_rates)  # already ordered by effective_from desc
+
+    return LLMRatesResponse(
+        active=[LLMRateOut.model_validate(r) for r in active],
+        history=[LLMRateOut.model_validate(r) for r in history],
+    )
+
+
+# ---------------------------------------------------------------------------
+# POST /llm-rates
+# ---------------------------------------------------------------------------
+
+
+@router.post("/llm-rates", response_model=LLMRateOut, status_code=201)
+def create_llm_rate(
+    body: LLMRateCreate,
+    _email: str = Depends(require_super_admin_jwt),
+    db: Session = Depends(get_db),
+) -> LLMRateOut:
+    """Add a new rate for a model, closing the current active rate if one exists."""
+    today = date.today()
+
+    # Close the current active rate for this provider + model_name, if any
+    existing_active = db.execute(
+        select(LLMModel).where(
+            LLMModel.provider == body.provider,
+            LLMModel.model_name == body.model_name,
+            LLMModel.effective_to.is_(None),
+        )
+    ).scalars().first()
+
+    if existing_active is not None:
+        existing_active.effective_to = today
+
+    # Insert the new rate
+    new_rate = LLMModel(
+        provider=body.provider,
+        model_name=body.model_name,
+        usd_per_1k_input=body.usd_per_1k_input,
+        usd_per_1k_output=body.usd_per_1k_output,
+        effective_from=today,
+        effective_to=None,
+    )
+    db.add(new_rate)
+    db.commit()
+    db.refresh(new_rate)
+
+    return LLMRateOut.model_validate(new_rate)