diff --git a/.DS_Store b/.DS_Store deleted file mode 100644 index b119ee2d7b3021fb38805e2d3d87650bc47c6278..0000000000000000000000000000000000000000 Binary files a/.DS_Store and /dev/null differ diff --git a/.dockerignore b/.dockerignore new file mode 100644 index 0000000000000000000000000000000000000000..3d627fc1c29beb52effe3d4c8f736b05f06cd0d8 --- /dev/null +++ b/.dockerignore @@ -0,0 +1,31 @@ +# Secrets — never bake into image (COPY . . in Dockerfile would include these) +.env +.env.* + +# Virtualenv — host Python (macOS) is incompatible with container Python (Linux) +.venv/ + +# Python bytecode — regenerated automatically, OS-specific +__pycache__/ +*.pyc +*.pyo +*.pyd +.pytest_cache/ + +# Git history — app never needs it at runtime, and it can be hundreds of MB +.git/ + +# Frontend build tools — node_modules contains esbuild/vite (build-time only) +# The compiled output (chatbot-widget.js) is already built; these are never needed at runtime +frontend-widget/node_modules/ + +# Tests — bind-mounted in dev via docker-compose.dev.yml; not needed in prod image +tests/ + +# Project management and documentation — runtime irrelevant +archive/ +docs/ +current_state/ +scripts/ +.claude/ +*.md diff --git a/.geminiignore b/.geminiignore new file mode 100644 index 0000000000000000000000000000000000000000..656ff69d867ba9876e217adcc01f6488735d6af1 --- /dev/null +++ b/.geminiignore @@ -0,0 +1,7 @@ +.venv/ +.git/ +node_modules/ +__pycache__/ +dist/ +build/ +*.log diff --git a/.gitignore b/.gitignore index 051f964a39e13c132696465f6aa88719c93057e8..a99d4fcd8df8a4e451a97c0603132ffb7b83a5c3 100644 --- a/.gitignore +++ b/.gitignore @@ -133,6 +133,7 @@ celerybeat.pid .env.staging .env.production .venv +.venv-linux env/ venv/ ENV/ @@ -189,18 +190,22 @@ cython_debug/ # Generated frontend configuration — tracked so CI/CD (GitHub Actions → HF Spaces) has it # frontend/config.js +# ChromaDB persistent storage (runtime data) +chromadb/ +chroma_data/ + # Other Files or Directories -CLAUDE*.md -GEMINI*.md -AGENTS*.md -PROJECT.md .claude/ .vscode/ +*CLAUDE.md +*AGENTS.md +*GEMINI.md +docker-compose.dev.local.yml # Everything inside `archive` except `daily_log.md` archive/* !archive/daily_log.md -!archive/development_notes/ +!archive/knowledge_assessments/ # Microservice credentials (sensitive) services/credentials/*.json @@ -231,3 +236,6 @@ nginx/logs/ # Node.js (frontend-widget Vite/Preact source project) frontend-widget/node_modules/ + +# Mac's metadata +**/.DS_Store \ No newline at end of file diff --git a/DOCKER_SETUP.md b/DOCKER_SETUP.md index 2145135d54f0d864036a715e771b9d7f41e6a601..5cc6a474de69c39ef6157da09b02c48a6e6ff686 100644 --- a/DOCKER_SETUP.md +++ b/DOCKER_SETUP.md @@ -126,13 +126,76 @@ docker-compose -f docker-compose.dev.yml up -d docker-compose -f docker-compose.dev.yml logs -f smart-chatbot ``` -#### 6. Stop Development Environment +#### 6. Named Volumes (Dev) + +| Volume | Purpose | +|--------|---------| +| `chatbot_postgres_dev` | PostgreSQL data — persists DB across restarts | +| `qdrant_dev_data` | Qdrant vector index data — production-target vector backend, mounted at `/qdrant/storage` | +| `redis_dev_data` | Redis append-only data for Taskiq WordPress sync jobs | + +The production stack uses the same Qdrant storage path with `qdrant_prod_data` and stores Taskiq broker data in `redis_prod_data`. Startup runs the active vector-store readiness probe, so read-only volume or vector-service availability problems fail before tenant KB writes. + +Vector backend environment variables: + +| Variable | Purpose | +|----------|---------| +| `HF_HUB_DISABLE_XET` | Legacy HuggingFace Hub workaround; harmless if left set, no longer required for the Gemini embedding path | +| `GEMINI_EMBEDDING_API_KEY` | Gemini API key used by `EmbeddingService` for `gemini-embedding-2` requests | +| `VECTOR_STORE_BACKEND` | `qdrant` for V1 runtime; `chroma` remains available only as legacy fallback code | +| `QDRANT_URL` | Qdrant HTTP URL from app/worker containers, usually `http://qdrant:6333` | +| `QDRANT_COLLECTION_NAME` | Shared Qdrant collection for tenant KB vectors | +| `QDRANT_VECTOR_SIZE` | Embedding dimension; `3072` for `gemini-embedding-2` | +| `QDRANT_TIMEOUT` | Qdrant client timeout in seconds | +| `QDRANT_CONTAINER_NAME` | Docker container name for the Qdrant service | + +Chat billing / completion envelope variables: + +| Variable | Purpose | +|----------|---------| +| `CHAT_MODEL_PRIMARY` | Primary routed chat model alias target used by LiteLLM | +| `CHAT_MODEL_FALLBACKS` | Ordered fallback chat models used when the primary fails | +| `CHAT_MAX_OUTPUT_TOKENS` | Hard cap passed to chat completions and used by the low-balance preflight reserve check | + +Rebuild one tenant's Qdrant index from PostgreSQL: + +```bash +docker compose --env-file .env -f docker-compose.dev.yml exec smart-chatbot \ + uv run python scripts/rebuild_tenant_index.py +``` + +**Running tests — host vs Docker:** + +`uv run pytest` on the host will produce 5 failures: +- `test_api_versioning.py::test_versioned_url_works` +- `test_input_token_limit.py::test_input_token_limit` +- `test_chat_billing_wiring.py::test_deducts_credits_on_credits_billing_mode` +- `test_chat_billing_wiring.py::test_returns_402_on_insufficient_credits` +- `test_chat_billing_wiring.py::test_skips_deduction_on_subscription_billing_mode` + +Root cause: these tests call `retrieval_service.retrieve()` which connects to `QDRANT_URL=http://qdrant:6333`. That hostname only resolves inside the Docker network. No ports are exposed to the host, so the connection fails locally. + +The full test suite must be run inside the container: + +```bash +docker compose -f docker-compose.dev.yml exec chatbot_backend_dev uv run pytest tests -rAq --tb=no +``` + +Run the PostgreSQL-backed concurrent billing release-gate verification: + +```bash +docker exec -w /project chatbot_backend_dev uv run python scripts/verify_postgres_concurrent_billing.py +``` + +Expected result: one concurrent `/chat` request returns `200`, all other burst requests return `402`, final tenant balance is `0.000000`, and exactly one `credit_events` row is persisted. + +#### 7. Stop Development Environment ```bash # Stop containers (keep data) docker-compose -f docker-compose.dev.yml down -# Stop and remove volumes (fresh database) +# Stop and remove volumes (fresh database + vector store) docker-compose -f docker-compose.dev.yml down -v ``` @@ -181,6 +244,8 @@ docker-compose -f docker-compose.prod.yml up -d - **Public API:** http://your-domain.com (via nginx) - **Backend:** NOT accessible from outside (internal network only) - **PostgreSQL:** NOT accessible from outside (internal network only) +- **Redis:** NOT accessible from outside (internal network only; used by Taskiq) +- **WordPress sync worker:** background Taskiq worker, not HTTP-accessible #### 5. Verify Services are Running @@ -197,6 +262,8 @@ docker-compose -f docker-compose.prod.yml ps | grep healthy chatbot_backend_prod healthy chatbot_nginx_prod healthy chatbot_postgres_prod healthy +chatbot_redis_prod healthy +wp_sync_worker_prod running ``` #### 6. View Logs diff --git a/Dockerfile b/Dockerfile index d0f757cebeb79762220c536246ab77fb54860cf9..adf93ed036e80640269f222097c1f8a9f89f8efa 100644 --- a/Dockerfile +++ b/Dockerfile @@ -24,7 +24,10 @@ # - Faster builds (shared layers cached once) # ============================================================================ -FROM python:3.11-slim AS base +FROM cgr.dev/chainguard/python:latest-dev AS base + +# Chainguard sets ENTRYPOINT ["python"] — clear it so exec-form CMD works directly. +ENTRYPOINT [] # Set working directory to project root # NOTE: Changed from /app to /project to support absolute imports (app.*) @@ -48,45 +51,13 @@ COPY --from=ghcr.io/astral-sh/uv:latest /uv /usr/local/bin/uv # - uv.lock = Exact pinned versions of all dependencies (like package-lock.json) COPY pyproject.toml uv.lock ./ -# Install dependencies using uv -# Using "uv pip install" instead of "uv sync" for system-wide installation -# -# Flags explained: -# pip install = Use pip-compatible interface (supports --system) -# --system = Install to system Python (not venv) - CRITICAL for Docker -# -r pyproject.toml = Install from project dependencies -# -# Why --system is required in Docker: -# -# WITHOUT --system (default behavior): -# - uv creates a virtual environment (.venv/) and installs packages there -# - Executables end up in: .venv/bin/uvicorn (hidden location) -# - Docker can't find them in PATH → Error: "No such file or directory" -# - You'd need to use "uv run uvicorn ..." which adds complexity -# -# WITH --system: -# - uv installs directly to system Python (no .venv/ folder created) -# - Executables go to: /usr/local/bin/uvicorn (standard PATH location) -# - Docker finds them immediately → uvicorn command just works -# - Cleaner CMD: just "uvicorn ..." instead of "uv run uvicorn ..." -# -# Why virtual environments (.venv/) are "extra" in Docker: -# On your laptop: .venv/ needed to separate multiple projects -# - Project A needs pandas 1.0 -# - Project B needs pandas 2.0 -# - .venv/ prevents conflicts -# -# In Docker: Container itself provides isolation -# - Container 1 has its own Python + packages (completely isolated) -# - Container 2 has its own Python + packages (can't see Container 1) -# - .venv/ inside container = locking a safe inside another safe (redundant) -# -# Analogy: -# Without --system = Hiding keys in a drawer (Docker can't find them) -# With --system = Putting keys on the key hook (Docker finds them easily) -# -# This replaces traditional: pip install -r requirements.txt -RUN uv pip install --system -r pyproject.toml +# Chainguard runs as nonroot (uid 65532) — cannot write to system Python paths. +# Install into /project/.venv (owned by nonroot) and put it on PATH. +RUN uv venv /project/.venv && \ + uv pip install --python /project/.venv/bin/python -r pyproject.toml +ENV PATH="/project/.venv/bin:$PATH" +# Avoid HuggingFace Hub Xet transfer stalls during startup embedding warmup. +ENV HF_HUB_DISABLE_XET=1 # Copy application code # NOTE: Changed to copy entire project instead of just app/ directory @@ -97,8 +68,14 @@ RUN uv pip install --system -r pyproject.toml # Destination: . (current WORKDIR, which is /project inside container) # # Result: Files land at /project/app/main.py, /project/app/models/, etc. +# Note: chroma_data/ is excluded via .dockerignore — created below with correct ownership. COPY . . +# Pre-create ChromaDB data directory as nonroot so the named volume is initialised +# with correct ownership. Without this, Docker creates the volume mount point as root, +# and the nonroot process cannot write the SQLite file — causing 500 errors on all KB writes. +RUN mkdir -p /project/chroma_data + # ============================================================================ # Why we DON'T set CMD or ENV here? # - Base stage is INCOMPLETE - not meant to run directly @@ -130,7 +107,7 @@ ENV DEBUG=true # Development-specific command # Flags explained: -# uvicorn = ASGI server for FastAPI (installed to system with --system flag) +# uvicorn = ASGI server for FastAPI (available via /project/.venv on PATH) # app.main:app = Import path (app/main.py file, app object) - using absolute import # --reload = Watch for code changes, auto-restart server (hot reload) # --host 0.0.0.0 = Listen on ALL network interfaces (required for Docker) @@ -143,7 +120,7 @@ ENV DEBUG=true # - --reload is EXPENSIVE (file watching, auto-restart) - only for development # - Production uses gunicorn (multi-worker, more robust) # -# Note: No "uv run" needed because we used --system flag during install +# Note: No "uv run" needed — venv bin is on PATH via ENV PATH above CMD ["uvicorn", "app.main:app", "--reload", "--host", "0.0.0.0", "--port", "8000"] # Port documentation (doesn't actually open ports) @@ -190,7 +167,7 @@ ENV DEBUG=false # - Better stability, graceful shutdowns, worker health monitoring # # Flags explained: -# gunicorn = WSGI server (wraps uvicorn workers, installed to system) +# gunicorn = WSGI server (wraps uvicorn workers, available via /project/.venv on PATH) # app.main:app = Import path (app/main.py file, app object) - using absolute import # -k uvicorn.workers.UvicornWorker = Use Uvicorn worker class (for ASGI support) # -w 4 = 4 worker processes (adjust based on CPU cores) @@ -202,7 +179,7 @@ ENV DEBUG=false # Worker count rule of thumb: (2 × CPU_cores) + 1 # Example: 2-core server → 5 workers, 4-core server → 9 workers # -# Note: No "uv run" needed because we used --system flag during install +# Note: No "uv run" needed — venv bin is on PATH via ENV PATH above CMD ["gunicorn", "app.main:app", "-k", "uvicorn.workers.UvicornWorker", "-w", "4", "-b", "0.0.0.0:8000"] # Port documentation (doesn't actually open ports) @@ -253,7 +230,7 @@ EXPOSE 7860 # docker images | grep smart-chatbot # # RUN DEVELOPMENT CONTAINER (standalone, no compose): -# docker run -p 8000:8000 -v $(pwd)/app:/app smart-chatbot:dev +# docker run -p 8000:8000 -v $(pwd)/app:/project/app smart-chatbot:dev # # RUN PRODUCTION CONTAINER (standalone, no compose): # docker run -p 8000:8000 smart-chatbot:prod diff --git a/PROJECT.md b/PROJECT.md new file mode 100644 index 0000000000000000000000000000000000000000..287001c8ba8e7e1bc8df7ddf7d8f877a14e60f06 --- /dev/null +++ b/PROJECT.md @@ -0,0 +1,141 @@ +# Smart Chatbot — Project Navigation + +**What it is:** A chatbot-as-a-service (CaaS) microservice for SMEs. Tenants embed the widget on their site; the chatbot answers domain questions grounded in the tenant's knowledge base (RAG). Sold as prepaid token credits or monthly subscription. + +--- + +## Current Focus + +**Next:** Phase 3 release gate — Simulated Tenant QA Pipeline (all prior gates including Qdrant migration are complete). +Full task spec: `current_state/project_status.md` + +--- + +## Tech Stack + +| Layer | Technology | +|---|---| +| Backend | FastAPI, Python, SQLAlchemy, Alembic | +| Database | PostgreSQL (dev/prod), SQLite (HuggingFace Spaces) | +| Vector DB | Qdrant — shared collection with required `tenant_id` payload filter; ChromaDB legacy fallback code only | +| Embeddings | Google Gemini Embedding 2 (`gemini-embedding-2`, 3072-dim) | +| LLM | LiteLLM Router — provider-agnostic, model aliases in env vars, auto-fallback chains | +| Frontend | Preact + Shadow DOM widget, Vite build | +| Auth | API key (widget) + Google OAuth + TOTP 2FA + JWT (admin) | +| Infra | Docker multi-env, Nginx, Redis/Celery workers, Qdrant, HuggingFace Spaces | +| CI/CD | GitHub Actions | + +--- + +## Project Structure + +``` +app/ + routers/ # Route handlers (knowledge_base, system_prompt) + schemas/ # Pydantic request/response models + middleware/ # TenantAuthMiddleware (API key → tenant_id) + models/ # SQLAlchemy models + services/ # Business logic (EmbeddingService, VectorStore, RetrievalService, LLMService, CreditService, WordPress sync, kb_reindex) + workers/ # Celery background workers (wordpress_sync, kb_reindex) + utils/ # api_key, token_counter, seeding + dependencies/ # FastAPI dependencies (admin_auth) +alembic/ # DB migrations +tests/ # 469 tests, organized by domain + auth/ # Admin + super-admin auth, JWT, TOTP, OAuth + knowledge_base/ # KB CRUD, chunking, pagination, reindex, sync + chat/ # Conversations, RAG context, input limits, history + billing/ # Credits, billing modes, concurrency + vector_store/ # Qdrant, ChromaDB, embedding service + workers/ # Summarization worker, WP sync jobs + llm/ # LLM service, retrieval, AI verdict, RAG + infra/ # Middleware, rate limiter, seeding, widget, system prompt +frontend/ # Preact chat widget (Vite build → chatbot-widget.js) +current_state/ # Architecture and status documents (see below) +archive/ # Historical notes and discussion logs (never retroactively edited) +docs/ # Specs and error logs +``` + +--- + +## Key Architecture Decisions + +**Multi-tenancy:** API key → `tenant_id` extracted in middleware. All DB queries and vector store operations are scoped to `tenant_id`. Qdrant uses one shared collection with required `tenant_id` payload filters inside the adapter; PostgreSQL remains the source of truth and Qdrant is a rebuildable search index. + +**RAG pipeline:** `/chat` → `RetrievalService.retrieve(query, tenant_id)` → top-k KB docs injected into system message → LLM response grounded in KB. + +**User auth model:** Two modes — `anonymous` (most SMEs: no end-user login, visitors are anonymous) and `authenticated` (SaaS/e-commerce: host page passes authenticated user identity). Toggle via admin panel config. + +**LLM provider abstraction:** `LLMService` wraps LiteLLM Router. Model aliases (`chat-model`, `summarization-model`, `judge-llm`) declared in env vars — swapping providers requires only an env var change, no code change. Auto-fallback chains configured per alias. `complete_with_usage()` returns `LLMResult` with token counts for billing. + +**API versioning:** All public endpoints prefixed `/api/v1/` — in place before first client embeds widget (one-way door). + +**Widget embedding:** Preact + Shadow DOM — 3KB bundle, fully isolated from host-site CSS/JS, works on any tech stack (WordPress, Vue, React, plain HTML). Admin console generates per-tenant snippet: ``. Widget reads `data-api-key` at runtime. + +**Per-tenant customization:** All theme/branding/chatbot identity settings stored as `customization JSONB` on `tenants` table — zero schema migrations for new settings. Only fields that need filtering (e.g. `credits`, `is_active`) get proper columns. + +**Multi-tenant secrets:** Shared secrets (`GROQ_API_KEY`, `DATABASE_URL`) live in backend env. Per-tenant config (system prompt, KB, theme) lives in DB. BYOK tenants store their own LLM API key encrypted in `customization JSONB`. + +**Credit deduction atomicity:** Single SQL `UPDATE tenants SET balance_usd = balance_usd - :amount WHERE id = :tenant_id AND balance_usd >= :amount` — check rows affected (1 = success, 0 = insufficient balance). No race condition possible without application-level locking. + +--- + +## Key Files + +| File | Purpose | +|---|---| +| `app/main.py` | Entry point — router registration, middleware, startup | +| `app/models/conversation_manager.py` | LLM call lives here — chat logic, context assembly | +| `app/models/nlp_engine.py` | Analytics and routing only — keep as-is | +| `app/utils/config.py` | Singleton config — safe to defer refactor | +| `.env` / `.env.staging` / `.env.production` | Environment-specific values | + +--- + +## Business Model + +| Tier | Best for | How it works | +|---|---|---| +| Prepaid credits | Informational sites, low-volume SMEs | Buy credits upfront, deducted per message | +| Monthly subscription | E-commerce, SaaS, high-volume SMEs | Fixed fee, predictable revenue | + +Gifted accounts (V1 only): Monireach tops up manually via super-admin; no Stripe needed. Full payment integration is post-V1. + +--- + +## Deployment Roadmap + +| Stage | Tenants | Stack | Monthly cost | +|---|---|---|---| +| Now | 0–10 | HuggingFace Spaces (demo/portfolio) | Free | +| Early | 10–100 | Railway or Fly.io (managed, zero DevOps) | $10–30 | +| Growth | 100–1000 | Hetzner VPS + Docker Compose | €10–20 | +| Scale | 1000+ | Hetzner + K3s (lightweight Kubernetes) | €30–80 | + +Target: Hetzner CAX31 ARM (4 vCPU, 8GB RAM, €14.10/month). PostgreSQL: Supabase managed at small scale → self-hosted at large scale. Admin panels: Cloudflare Pages (free). HF Space kept as public demo proxy only. + +**K3s local practice (after V1 stable):** Deploy the full stack to a local K3s cluster in parallel with Docker Compose — not a replacement for the dev workflow, but a learning environment for writing real K8s manifests (`Deployment`, `Service`, `Ingress`, `ConfigMap`, `Secret`) against a real multi-service app. Makes the eventual Hetzner → K3s production migration familiar ground. + +--- + +## Where to Find Things + +| What you need | File | +|---|---| +| Task status, V1 checklist, current progress | `current_state/project_status.md` | +| Full details on completed tasks (sub-tasks, bugs, decisions) | `current_state/milestone.md` | +| Auth model, tenant isolation, API key design | `docs/security_architecture.md` | +| Frontend widget architecture, embedding, theming | `docs/frontend_architecture.md` | +| Khmer language strategy, LLM provider selection | `docs/khmer_llm_strategy.md` | +| Admin console scope and integration status | `~/projects/smart_chatbot_admin_console/PROJECT.md` | +| Super-admin console scope and task status | `~/projects/smart_chatbot_super_admin/current_state/project_status.md` | +| Docker setup, env vars, service topology | `DOCKER_SETUP.md` | +| Historical discussions (system expansion, billing design) | `archive/` | + +--- + +## Live Deployments + +| Environment | URL | +|---|---| +| HuggingFace Spaces (public API) | `https://huggingface.co/spaces/monireach88/smart-chatbot-api` | +| Portfolio (widget embedded) | `https://monireach.com` | diff --git a/alembic/versions/00f16d0affc6_add_auth_providers_table_and_phone_to_.py b/alembic/versions/00f16d0affc6_add_auth_providers_table_and_phone_to_.py new file mode 100644 index 0000000000000000000000000000000000000000..ff64402e748a6f7c8411a5ce9555b6ae94ce7519 --- /dev/null +++ b/alembic/versions/00f16d0affc6_add_auth_providers_table_and_phone_to_.py @@ -0,0 +1,46 @@ +"""add auth_providers table and phone to tenants + +Revision ID: 00f16d0affc6 +Revises: 752c239e659a +Create Date: 2026-05-28 11:35:39.306737 + +""" +from typing import Sequence, Union + +from alembic import op +import sqlalchemy as sa + + +# revision identifiers, used by Alembic. +revision: str = '00f16d0affc6' +down_revision: Union[str, Sequence[str], None] = '752c239e659a' +branch_labels: Union[str, Sequence[str], None] = None +depends_on: Union[str, Sequence[str], None] = None + + +def upgrade() -> None: + """Upgrade schema.""" + # ### commands auto generated by Alembic - please adjust! ### + op.create_table('tenant_auth_providers', + sa.Column('id', sa.UUID(), nullable=False), + sa.Column('tenant_id', sa.UUID(), nullable=False), + sa.Column('provider', sa.String(length=20), nullable=False), + sa.Column('provider_user_id', sa.String(length=255), nullable=False), + sa.Column('linked_at', sa.DateTime(), nullable=False), + sa.CheckConstraint("provider IN ('google', 'telegram')", name=op.f('ck_tenant_auth_providers_chk_tenant_auth_providers_provider')), + sa.ForeignKeyConstraint(['tenant_id'], ['tenants.id'], name=op.f('fk_tenant_auth_providers_tenant_id_tenants'), ondelete='CASCADE'), + sa.PrimaryKeyConstraint('id', name=op.f('pk_tenant_auth_providers')), + sa.UniqueConstraint('tenant_id', 'provider', name='uq_tenant_auth_providers_tenant_id_provider') + ) + op.create_index(op.f('ix_tenant_auth_providers_tenant_id'), 'tenant_auth_providers', ['tenant_id'], unique=False) + op.add_column('tenants', sa.Column('phone', sa.String(length=20), nullable=True)) + # ### end Alembic commands ### + + +def downgrade() -> None: + """Downgrade schema.""" + # ### commands auto generated by Alembic - please adjust! ### + op.drop_column('tenants', 'phone') + op.drop_index(op.f('ix_tenant_auth_providers_tenant_id'), table_name='tenant_auth_providers') + op.drop_table('tenant_auth_providers') + # ### end Alembic commands ### diff --git a/alembic/versions/05aee5d64d1b_add_context_summary_and_summarization_.py b/alembic/versions/05aee5d64d1b_add_context_summary_and_summarization_.py new file mode 100644 index 0000000000000000000000000000000000000000..fb4c701b9983039ca4d0ddf43dbc280bb15bc4e7 --- /dev/null +++ b/alembic/versions/05aee5d64d1b_add_context_summary_and_summarization_.py @@ -0,0 +1,47 @@ +"""add_context_summary_and_summarization_job + +Revision ID: 05aee5d64d1b +Revises: d2d812e2c226 +Create Date: 2026-03-31 15:38:57.730352 + +""" +from typing import Sequence, Union + +from alembic import op +import sqlalchemy as sa + + +# revision identifiers, used by Alembic. +revision: str = '05aee5d64d1b' +down_revision: Union[str, Sequence[str], None] = 'd2d812e2c226' +branch_labels: Union[str, Sequence[str], None] = None +depends_on: Union[str, Sequence[str], None] = None + + +def upgrade() -> None: + """Upgrade schema.""" + # ### commands auto generated by Alembic - please adjust! ### + op.create_table('summarization_jobs', + sa.Column('id', sa.UUID(), nullable=False), + sa.Column('conversation_id', sa.UUID(), nullable=False), + sa.Column('status', sa.String(length=20), nullable=False), + sa.Column('attempt_count', sa.Integer(), nullable=False), + sa.Column('created_at', sa.DateTime(), nullable=False), + sa.Column('last_attempted_at', sa.DateTime(), nullable=True), + sa.ForeignKeyConstraint(['conversation_id'], ['conversations.id'], name=op.f('fk_summarization_jobs_conversation_id_conversations')), + sa.PrimaryKeyConstraint('id', name=op.f('pk_summarization_jobs')) + ) + op.create_index(op.f('ix_summarization_jobs_conversation_id'), 'summarization_jobs', ['conversation_id'], unique=False) + op.create_index('uq_summarization_jobs_pending', 'summarization_jobs', ['conversation_id'], unique=True, postgresql_where=sa.text("status = 'pending'")) + op.add_column('conversations', sa.Column('context_summary', sa.Text(), nullable=True)) + # ### end Alembic commands ### + + +def downgrade() -> None: + """Downgrade schema.""" + # ### commands auto generated by Alembic - please adjust! ### + op.drop_column('conversations', 'context_summary') + op.drop_index('uq_summarization_jobs_pending', table_name='summarization_jobs', postgresql_where=sa.text("status = 'pending'")) + op.drop_index(op.f('ix_summarization_jobs_conversation_id'), table_name='summarization_jobs') + op.drop_table('summarization_jobs') + # ### end Alembic commands ### diff --git a/alembic/versions/0704837a1594_add_backup_email_to_tenants.py b/alembic/versions/0704837a1594_add_backup_email_to_tenants.py new file mode 100644 index 0000000000000000000000000000000000000000..b2962ad1ed7b30981c01d46ae7afaf1677c99d67 --- /dev/null +++ b/alembic/versions/0704837a1594_add_backup_email_to_tenants.py @@ -0,0 +1,32 @@ +"""add backup email to tenants + +Revision ID: 0704837a1594 +Revises: d1e126fba5fb +Create Date: 2026-04-24 17:27:09.794919 + +""" +from typing import Sequence, Union + +from alembic import op +import sqlalchemy as sa + + +# revision identifiers, used by Alembic. +revision: str = '0704837a1594' +down_revision: Union[str, Sequence[str], None] = 'd1e126fba5fb' +branch_labels: Union[str, Sequence[str], None] = None +depends_on: Union[str, Sequence[str], None] = None + + +def upgrade() -> None: + """Upgrade schema.""" + # ### commands auto generated by Alembic - please adjust! ### + op.add_column('tenants', sa.Column('backup_email', sa.String(length=255), nullable=True)) + # ### end Alembic commands ### + + +def downgrade() -> None: + """Downgrade schema.""" + # ### commands auto generated by Alembic - please adjust! ### + op.drop_column('tenants', 'backup_email') + # ### end Alembic commands ### diff --git a/alembic/versions/126f5b1610a6_add_billing_tables.py b/alembic/versions/126f5b1610a6_add_billing_tables.py new file mode 100644 index 0000000000000000000000000000000000000000..0e86e3b5c57e77dc5c77c25b6247033f8bfd1db3 --- /dev/null +++ b/alembic/versions/126f5b1610a6_add_billing_tables.py @@ -0,0 +1,62 @@ +"""add billing tables + +Revision ID: 126f5b1610a6 +Revises: d7470fcd70c6 +Create Date: 2026-04-18 10:02:12.372002 + +""" +from typing import Sequence, Union + +from alembic import op +import sqlalchemy as sa + + +# revision identifiers, used by Alembic. +revision: str = '126f5b1610a6' +down_revision: Union[str, Sequence[str], None] = 'd7470fcd70c6' +branch_labels: Union[str, Sequence[str], None] = None +depends_on: Union[str, Sequence[str], None] = None + + +def upgrade() -> None: + """Upgrade schema.""" + # ### commands auto generated by Alembic - please adjust! ### + op.create_table('llm_models', + sa.Column('id', sa.Integer(), autoincrement=True, nullable=False), + sa.Column('provider', sa.String(length=100), nullable=False), + sa.Column('model_name', sa.String(length=255), nullable=False), + sa.Column('credits_per_1k_input', sa.Numeric(precision=20, scale=6), nullable=False), + sa.Column('credits_per_1k_output', sa.Numeric(precision=20, scale=6), nullable=False), + sa.Column('effective_from', sa.Date(), nullable=False), + sa.Column('effective_to', sa.Date(), nullable=True), + sa.PrimaryKeyConstraint('id', name=op.f('pk_llm_models')) + ) + op.create_table('credit_events', + sa.Column('id', sa.UUID(), nullable=False), + sa.Column('tenant_id', sa.UUID(), nullable=False), + sa.Column('event_type', sa.String(length=20), nullable=False), + sa.Column('amount', sa.Numeric(precision=20, scale=6), nullable=False), + sa.Column('balance_after', sa.Numeric(precision=20, scale=6), nullable=False), + sa.Column('created_at', sa.DateTime(), nullable=False), + sa.Column('request_id', sa.String(length=255), nullable=True), + sa.Column('llm_model_id', sa.Integer(), nullable=True), + sa.ForeignKeyConstraint(['llm_model_id'], ['llm_models.id'], name=op.f('fk_credit_events_llm_model_id_llm_models')), + sa.ForeignKeyConstraint(['tenant_id'], ['tenants.id'], name=op.f('fk_credit_events_tenant_id_tenants')), + sa.PrimaryKeyConstraint('id', name=op.f('pk_credit_events')), + sa.UniqueConstraint('tenant_id', 'request_id', name='uq_credit_events_tenant_id_request_id') + ) + op.create_index(op.f('ix_credit_events_tenant_id'), 'credit_events', ['tenant_id'], unique=False) + op.add_column('tenants', sa.Column('credits', sa.Numeric(precision=20, scale=6), server_default='0', nullable=False)) + op.add_column('tenants', sa.Column('billing_mode', sa.String(length=20), server_default='credits', nullable=False)) + # ### end Alembic commands ### + + +def downgrade() -> None: + """Downgrade schema.""" + # ### commands auto generated by Alembic - please adjust! ### + op.drop_column('tenants', 'billing_mode') + op.drop_column('tenants', 'credits') + op.drop_index(op.f('ix_credit_events_tenant_id'), table_name='credit_events') + op.drop_table('credit_events') + op.drop_table('llm_models') + # ### end Alembic commands ### diff --git a/alembic/versions/3d1418b32af0_retire_gemini_2_0_flash_add_gemini_2_5_.py b/alembic/versions/3d1418b32af0_retire_gemini_2_0_flash_add_gemini_2_5_.py new file mode 100644 index 0000000000000000000000000000000000000000..36c5f76acafb135f0364052ba4a7ac0a04562346 --- /dev/null +++ b/alembic/versions/3d1418b32af0_retire_gemini_2_0_flash_add_gemini_2_5_.py @@ -0,0 +1,64 @@ +"""retire_gemini_2_0_flash_add_gemini_2_5_flash_rates + +Revision ID: 3d1418b32af0 +Revises: 9d051350268a +Create Date: 2026-04-19 02:38:42.878677 + +Pricing basis (2026-04-19): $0.30/1M input, $2.50/1M output (paid tier) × 3x markup. +1 credit ≈ $0.001 USD. gemini-2.0-flash retired per Google shutdown (2026-06-01). +""" +from typing import Sequence, Union + +from alembic import op + + +# revision identifiers, used by Alembic. +revision: str = '3d1418b32af0' +down_revision: Union[str, Sequence[str], None] = '9d051350268a' +branch_labels: Union[str, Sequence[str], None] = None +depends_on: Union[str, Sequence[str], None] = None + + +def upgrade() -> None: + # Close out gemini-2.0-flash row + op.execute(""" + UPDATE llm_models + SET effective_to = '2026-04-19' + WHERE provider = 'gemini' + AND model_name = 'gemini-2.0-flash' + AND effective_to IS NULL + """) + + # Insert gemini-2.5-flash row (idempotent) + op.execute(""" + INSERT INTO llm_models + (provider, model_name, credits_per_1k_input, credits_per_1k_output, + effective_from, effective_to) + SELECT 'gemini', 'gemini-2.5-flash', 0.900000, 7.500000, '2026-04-19', NULL + WHERE NOT EXISTS ( + SELECT 1 FROM llm_models + WHERE provider = 'gemini' + AND model_name = 'gemini-2.5-flash' + AND effective_to IS NULL + ) + """) + + +def downgrade() -> None: + # Remove gemini-2.5-flash row + op.execute(""" + DELETE FROM llm_models + WHERE provider = 'gemini' + AND model_name = 'gemini-2.5-flash' + AND effective_from = '2026-04-19' + AND effective_to IS NULL + """) + + # Reopen gemini-2.0-flash row + op.execute(""" + UPDATE llm_models + SET effective_to = NULL + WHERE provider = 'gemini' + AND model_name = 'gemini-2.0-flash' + AND effective_to = '2026-04-19' + """) diff --git a/alembic/versions/485fbe495e6b_add_feedback_columns_to_messages.py b/alembic/versions/485fbe495e6b_add_feedback_columns_to_messages.py new file mode 100644 index 0000000000000000000000000000000000000000..1386a53ba6f22e7a53caad953c3980fede01815d --- /dev/null +++ b/alembic/versions/485fbe495e6b_add_feedback_columns_to_messages.py @@ -0,0 +1,34 @@ +"""add feedback columns to messages + +Revision ID: 485fbe495e6b +Revises: 05aee5d64d1b +Create Date: 2026-04-05 11:57:40.860101 + +""" +from typing import Sequence, Union + +from alembic import op +import sqlalchemy as sa + + +# revision identifiers, used by Alembic. +revision: str = '485fbe495e6b' +down_revision: Union[str, Sequence[str], None] = '05aee5d64d1b' +branch_labels: Union[str, Sequence[str], None] = None +depends_on: Union[str, Sequence[str], None] = None + + +def upgrade() -> None: + """Upgrade schema.""" + # ### commands auto generated by Alembic - please adjust! ### + op.add_column('messages', sa.Column('user_feedback', sa.SmallInteger(), nullable=True)) + op.add_column('messages', sa.Column('tenant_feedback', sa.SmallInteger(), nullable=True)) + # ### end Alembic commands ### + + +def downgrade() -> None: + """Downgrade schema.""" + # ### commands auto generated by Alembic - please adjust! ### + op.drop_column('messages', 'tenant_feedback') + op.drop_column('messages', 'user_feedback') + # ### end Alembic commands ### diff --git a/alembic/versions/73ae38d1200c_add_retrieved_doc_ids_to_messages.py b/alembic/versions/73ae38d1200c_add_retrieved_doc_ids_to_messages.py new file mode 100644 index 0000000000000000000000000000000000000000..fb315f8f862fd02fb1eb437193c3b2af10e6d942 --- /dev/null +++ b/alembic/versions/73ae38d1200c_add_retrieved_doc_ids_to_messages.py @@ -0,0 +1,32 @@ +"""add_retrieved_doc_ids_to_messages + +Revision ID: 73ae38d1200c +Revises: e5cc44e2040a +Create Date: 2026-03-30 02:26:27.464417 + +""" +from typing import Sequence, Union + +from alembic import op +import sqlalchemy as sa +from sqlalchemy.dialects import postgresql + +# revision identifiers, used by Alembic. +revision: str = '73ae38d1200c' +down_revision: Union[str, Sequence[str], None] = 'e5cc44e2040a' +branch_labels: Union[str, Sequence[str], None] = None +depends_on: Union[str, Sequence[str], None] = None + + +def upgrade() -> None: + """Upgrade schema.""" + # ### commands auto generated by Alembic - please adjust! ### + op.add_column('messages', sa.Column('retrieved_doc_ids', postgresql.JSONB(astext_type=sa.Text()).with_variant(sa.JSON(), 'sqlite'), nullable=True)) + # ### end Alembic commands ### + + +def downgrade() -> None: + """Downgrade schema.""" + # ### commands auto generated by Alembic - please adjust! ### + op.drop_column('messages', 'retrieved_doc_ids') + # ### end Alembic commands ### diff --git a/alembic/versions/752c239e659a_add_rest_base_to_sync_jobs_expand_.py b/alembic/versions/752c239e659a_add_rest_base_to_sync_jobs_expand_.py new file mode 100644 index 0000000000000000000000000000000000000000..7039ae77fb7eb0f8c1a20545d777a61bc111ecc6 --- /dev/null +++ b/alembic/versions/752c239e659a_add_rest_base_to_sync_jobs_expand_.py @@ -0,0 +1,40 @@ +"""add rest_base to sync_jobs, expand status constraint + +Revision ID: 752c239e659a +Revises: b58e26e702df +Create Date: 2026-05-28 04:52:54.564900 + +""" +from typing import Sequence, Union + +from alembic import op +import sqlalchemy as sa + + +# revision identifiers, used by Alembic. +revision: str = '752c239e659a' +down_revision: Union[str, Sequence[str], None] = 'b58e26e702df' +branch_labels: Union[str, Sequence[str], None] = None +depends_on: Union[str, Sequence[str], None] = None + + +def upgrade() -> None: + """Upgrade schema.""" + op.add_column('sync_jobs', sa.Column('rest_base', sa.String(length=50), server_default='posts', nullable=False)) + op.drop_constraint('sync_jobs_status_valid', 'sync_jobs', type_='check') + op.create_check_constraint( + 'sync_jobs_status_valid', + 'sync_jobs', + "status IN ('cancelled', 'completed', 'expired', 'failed', 'paused', 'queued', 'running')", + ) + + +def downgrade() -> None: + """Downgrade schema.""" + op.drop_constraint('sync_jobs_status_valid', 'sync_jobs', type_='check') + op.create_check_constraint( + 'sync_jobs_status_valid', + 'sync_jobs', + "status IN ('queued', 'running', 'completed', 'failed')", + ) + op.drop_column('sync_jobs', 'rest_base') diff --git a/alembic/versions/7992949ccdf5_add_industries_table.py b/alembic/versions/7992949ccdf5_add_industries_table.py new file mode 100644 index 0000000000000000000000000000000000000000..5fe5f5c7355def5f5bcb4dc45f3f54d12c092f1e --- /dev/null +++ b/alembic/versions/7992949ccdf5_add_industries_table.py @@ -0,0 +1,56 @@ +"""add industries table + +Revision ID: 7992949ccdf5 +Revises: 0704837a1594 +Create Date: 2026-04-26 01:56:13.303817 + +""" +from typing import Sequence, Union + +from alembic import op +import sqlalchemy as sa + + +# revision identifiers, used by Alembic. +revision: str = '7992949ccdf5' +down_revision: Union[str, Sequence[str], None] = '0704837a1594' +branch_labels: Union[str, Sequence[str], None] = None +depends_on: Union[str, Sequence[str], None] = None + + +def upgrade() -> None: + industries_table = op.create_table( + "industries", + sa.Column("id", sa.Integer(), autoincrement=True, nullable=False), + sa.Column("code", sa.String(length=100), nullable=False), + sa.Column("display_name", sa.String(length=100), nullable=False), + sa.Column("is_active", sa.Boolean(), nullable=False, server_default=sa.true()), + sa.PrimaryKeyConstraint("id", name=op.f("pk_industries")), + sa.UniqueConstraint("code", name=op.f("uq_industries_code")), + ) + op.bulk_insert( + industries_table, + [ + {"code": "restaurant-food-beverage", "display_name": "Restaurant / Food & Beverage", "is_active": True}, + {"code": "retail", "display_name": "Retail", "is_active": True}, + {"code": "e-commerce", "display_name": "E-Commerce", "is_active": True}, + {"code": "healthcare", "display_name": "Healthcare", "is_active": True}, + {"code": "legal", "display_name": "Legal", "is_active": True}, + {"code": "education", "display_name": "Education", "is_active": True}, + {"code": "real-estate", "display_name": "Real Estate", "is_active": True}, + {"code": "financial-services", "display_name": "Financial Services", "is_active": True}, + {"code": "technology", "display_name": "Technology", "is_active": True}, + {"code": "travel-hospitality", "display_name": "Travel & Hospitality", "is_active": True}, + {"code": "beauty-wellness", "display_name": "Beauty & Wellness", "is_active": True}, + {"code": "professional-services", "display_name": "Professional Services", "is_active": True}, + {"code": "non-profit", "display_name": "Non-Profit", "is_active": True}, + {"code": "other", "display_name": "Other", "is_active": True}, + # Legacy codes — kept for backward compatibility; hidden from new dropdowns + {"code": "saas", "display_name": "SaaS", "is_active": False}, + {"code": "finance", "display_name": "Finance", "is_active": False}, + ], + ) + + +def downgrade() -> None: + op.drop_table("industries") diff --git a/alembic/versions/7aec975a4140_credits_to_usd_refactor.py b/alembic/versions/7aec975a4140_credits_to_usd_refactor.py new file mode 100644 index 0000000000000000000000000000000000000000..db4382d80b9bd7d524ffb2864ae96a1575f6e3c7 --- /dev/null +++ b/alembic/versions/7aec975a4140_credits_to_usd_refactor.py @@ -0,0 +1,64 @@ +"""credits to usd refactor + +Revision ID: 7aec975a4140 +Revises: ac67923a3324 +Create Date: 2026-05-17 04:08:19.969428 + +""" +from typing import Sequence, Union + +from alembic import op +import sqlalchemy as sa + + +# revision identifiers, used by Alembic. +revision: str = '7aec975a4140' +down_revision: Union[str, Sequence[str], None] = 'ac67923a3324' +branch_labels: Union[str, Sequence[str], None] = None +depends_on: Union[str, Sequence[str], None] = None + + +def upgrade() -> None: + """Upgrade schema.""" + op.add_column( + "llm_models", + sa.Column("usd_per_1k_input", sa.Numeric(precision=20, scale=6), nullable=True), + ) + op.add_column( + "llm_models", + sa.Column("usd_per_1k_output", sa.Numeric(precision=20, scale=6), nullable=True), + ) + op.add_column( + "tenants", + sa.Column("balance_usd", sa.Numeric(precision=20, scale=6), server_default="0", nullable=True), + ) + + op.execute( + """ + UPDATE llm_models + SET usd_per_1k_input = credits_per_1k_input, + usd_per_1k_output = credits_per_1k_output + WHERE usd_per_1k_input IS NULL + OR usd_per_1k_output IS NULL + """ + ) + op.execute( + """ + UPDATE tenants + SET balance_usd = credits + WHERE balance_usd IS NULL + """ + ) + + op.alter_column("llm_models", "usd_per_1k_input", nullable=False) + op.alter_column("llm_models", "usd_per_1k_output", nullable=False) + op.alter_column("tenants", "balance_usd", nullable=False) + op.alter_column("llm_models", "credits_per_1k_input", server_default="0") + op.alter_column("llm_models", "credits_per_1k_output", server_default="0") + + +def downgrade() -> None: + """Downgrade schema.""" + op.drop_column("tenants", "balance_usd") + op.drop_column("llm_models", "usd_per_1k_output") + op.drop_column("llm_models", "usd_per_1k_input") diff --git a/alembic/versions/84de4a13cffd_add_ai_verdict_scores_to_messages.py b/alembic/versions/84de4a13cffd_add_ai_verdict_scores_to_messages.py new file mode 100644 index 0000000000000000000000000000000000000000..eb6459d096e1392d12207f80f18563ae2b05dff9 --- /dev/null +++ b/alembic/versions/84de4a13cffd_add_ai_verdict_scores_to_messages.py @@ -0,0 +1,36 @@ +"""add_ai_verdict_scores_to_messages + +Revision ID: 84de4a13cffd +Revises: 485fbe495e6b +Create Date: 2026-04-11 04:30:21.707463 + +""" +from typing import Sequence, Union + +from alembic import op +import sqlalchemy as sa + + +# revision identifiers, used by Alembic. +revision: str = '84de4a13cffd' +down_revision: Union[str, Sequence[str], None] = '485fbe495e6b' +branch_labels: Union[str, Sequence[str], None] = None +depends_on: Union[str, Sequence[str], None] = None + + +def upgrade() -> None: + """Upgrade schema.""" + # ### commands auto generated by Alembic - please adjust! ### + op.add_column('messages', sa.Column('accuracy_score', sa.Float(), nullable=True)) + op.add_column('messages', sa.Column('relevance_score', sa.Float(), nullable=True)) + op.add_column('messages', sa.Column('safety_score', sa.Float(), nullable=True)) + # ### end Alembic commands ### + + +def downgrade() -> None: + """Downgrade schema.""" + # ### commands auto generated by Alembic - please adjust! ### + op.drop_column('messages', 'safety_score') + op.drop_column('messages', 'relevance_score') + op.drop_column('messages', 'accuracy_score') + # ### end Alembic commands ### diff --git a/alembic/versions/9ad31e299e42_add_is_indexed_to_knowledge_entries.py b/alembic/versions/9ad31e299e42_add_is_indexed_to_knowledge_entries.py new file mode 100644 index 0000000000000000000000000000000000000000..55fb83e3e9a0cdc480d3f410201e2253a4074436 --- /dev/null +++ b/alembic/versions/9ad31e299e42_add_is_indexed_to_knowledge_entries.py @@ -0,0 +1,38 @@ +"""add is_indexed to knowledge_entries + +Revision ID: 9ad31e299e42 +Revises: 7aec975a4140 +Create Date: 2026-05-17 16:26:35.228332 + +""" +from typing import Sequence, Union + +from alembic import op +import sqlalchemy as sa + + +# revision identifiers, used by Alembic. +revision: str = '9ad31e299e42' +down_revision: Union[str, Sequence[str], None] = '7aec975a4140' +branch_labels: Union[str, Sequence[str], None] = None +depends_on: Union[str, Sequence[str], None] = None + + +def upgrade() -> None: + """Upgrade schema.""" + # ### commands auto generated by Alembic - please adjust! ### + op.add_column('knowledge_base', sa.Column('is_indexed', sa.Boolean(), server_default='false', nullable=False)) + op.drop_column('llm_models', 'credits_per_1k_output') + op.drop_column('llm_models', 'credits_per_1k_input') + op.drop_column('tenants', 'credits') + # ### end Alembic commands ### + + +def downgrade() -> None: + """Downgrade schema.""" + # ### commands auto generated by Alembic - please adjust! ### + op.add_column('tenants', sa.Column('credits', sa.NUMERIC(precision=20, scale=6), server_default=sa.text("'0'::numeric"), autoincrement=False, nullable=False)) + op.add_column('llm_models', sa.Column('credits_per_1k_input', sa.NUMERIC(precision=20, scale=6), server_default=sa.text("'0'::numeric"), autoincrement=False, nullable=False)) + op.add_column('llm_models', sa.Column('credits_per_1k_output', sa.NUMERIC(precision=20, scale=6), server_default=sa.text("'0'::numeric"), autoincrement=False, nullable=False)) + op.drop_column('knowledge_base', 'is_indexed') + # ### end Alembic commands ### diff --git a/alembic/versions/9b95a5ce2ac1_add_gifted_billing_mode_check_constraint.py b/alembic/versions/9b95a5ce2ac1_add_gifted_billing_mode_check_constraint.py new file mode 100644 index 0000000000000000000000000000000000000000..5b9f88f6987ab66352134f6ac5d71d6925a24500 --- /dev/null +++ b/alembic/versions/9b95a5ce2ac1_add_gifted_billing_mode_check_constraint.py @@ -0,0 +1,30 @@ +"""add gifted billing mode check constraint + +Revision ID: 9b95a5ce2ac1 +Revises: bf4016f7dd2a +Create Date: 2026-04-24 09:16:41.390223 + +""" +from typing import Sequence, Union + +from alembic import op +import sqlalchemy as sa + + +# revision identifiers, used by Alembic. +revision: str = '9b95a5ce2ac1' +down_revision: Union[str, Sequence[str], None] = 'bf4016f7dd2a' +branch_labels: Union[str, Sequence[str], None] = None +depends_on: Union[str, Sequence[str], None] = None + + +def upgrade() -> None: + op.create_check_constraint( + "ck_tenants_chk_tenant_billing_mode", + "tenants", + "billing_mode IN ('credits', 'subscription', 'gifted')", + ) + + +def downgrade() -> None: + op.drop_constraint("ck_tenants_chk_tenant_billing_mode", "tenants", type_="check") diff --git a/alembic/versions/9d051350268a_seed_llm_models_initial_rates.py b/alembic/versions/9d051350268a_seed_llm_models_initial_rates.py new file mode 100644 index 0000000000000000000000000000000000000000..4dddcca5817d703cf7d708b8dbfd4870872c82a3 --- /dev/null +++ b/alembic/versions/9d051350268a_seed_llm_models_initial_rates.py @@ -0,0 +1,59 @@ +"""seed_llm_models_initial_rates + +Revision ID: 9d051350268a +Revises: 126f5b1610a6 +Create Date: 2026-04-18 16:51:58.821173 + +Pricing basis: 2026-04-18 approximate API costs × ~3x business markup. +1 credit ≈ $0.001 USD. To update rates: set effective_to on old rows, +insert new rows — never overwrite. +""" +from typing import Sequence, Union + +from alembic import op + + +# revision identifiers, used by Alembic. +revision: str = '9d051350268a' +down_revision: Union[str, Sequence[str], None] = '126f5b1610a6' +branch_labels: Union[str, Sequence[str], None] = None +depends_on: Union[str, Sequence[str], None] = None + + +_ROWS = [ + ("gemini", "gemini-2.0-flash", "0.300000", "1.200000"), + ("groq", "qwen3-32b", "0.870000", "1.770000"), + ("groq", "llama-3.3-70b-versatile", "1.770000", "2.370000"), + ("anthropic", "claude-3-5-haiku-20241022", "2.400000", "12.000000"), +] + + +def upgrade() -> None: + for provider, model_name, inp, out in _ROWS: + op.execute( + f""" + INSERT INTO llm_models + (provider, model_name, credits_per_1k_input, credits_per_1k_output, + effective_from, effective_to) + SELECT '{provider}', '{model_name}', {inp}, {out}, '2026-04-18', NULL + WHERE NOT EXISTS ( + SELECT 1 FROM llm_models + WHERE provider = '{provider}' + AND model_name = '{model_name}' + AND effective_to IS NULL + ) + """ + ) + + +def downgrade() -> None: + for provider, model_name, _, _ in _ROWS: + op.execute( + f""" + DELETE FROM llm_models + WHERE provider = '{provider}' + AND model_name = '{model_name}' + AND effective_from = '2026-04-18' + AND effective_to IS NULL + """ + ) diff --git a/alembic/versions/a53f4c8d9b21_kb_schema_redesign_and_sync_jobs.py b/alembic/versions/a53f4c8d9b21_kb_schema_redesign_and_sync_jobs.py new file mode 100644 index 0000000000000000000000000000000000000000..89e5f5cd2860298e66cce56ac7ea4ec32074761b --- /dev/null +++ b/alembic/versions/a53f4c8d9b21_kb_schema_redesign_and_sync_jobs.py @@ -0,0 +1,95 @@ +"""kb schema redesign and sync jobs + +Revision ID: a53f4c8d9b21 +Revises: 7992949ccdf5 +Create Date: 2026-04-29 23:05:00.000000 + +""" +from typing import Sequence, Union + +from alembic import op +import sqlalchemy as sa + + +# revision identifiers, used by Alembic. +revision: str = "a53f4c8d9b21" +down_revision: Union[str, Sequence[str], None] = "7992949ccdf5" +branch_labels: Union[str, Sequence[str], None] = None +depends_on: Union[str, Sequence[str], None] = None + + +def upgrade() -> None: + with op.batch_alter_table("knowledge_base") as batch_op: + batch_op.alter_column( + "entry", + new_column_name="content", + existing_type=sa.Text(), + existing_nullable=False, + ) + batch_op.add_column(sa.Column("source_url", sa.String(length=500), nullable=True)) + batch_op.add_column( + sa.Column("origin", sa.String(length=20), server_default="manual", nullable=False) + ) + batch_op.create_check_constraint( + "ck_knowledge_base_kb_origin_valid", + "origin IN ('manual', 'synced')", + ) + batch_op.drop_column("tag") + batch_op.drop_column("category") + + inspector = sa.inspect(op.get_bind()) + if not inspector.has_table("sync_jobs"): + op.create_table( + "sync_jobs", + sa.Column("id", sa.UUID(), nullable=False), + sa.Column("created_at", sa.DateTime(), nullable=True), + sa.Column("updated_at", sa.DateTime(), nullable=True), + sa.Column("tenant_id", sa.UUID(), nullable=False), + sa.Column("source_url", sa.String(length=500), nullable=False), + sa.Column("status", sa.String(length=30), server_default="queued", nullable=False), + sa.Column("total", sa.Integer(), server_default="0", nullable=False), + sa.Column("processed", sa.Integer(), server_default="0", nullable=False), + sa.Column("failed", sa.Integer(), server_default="0", nullable=False), + sa.Column("error_message", sa.Text(), nullable=True), + sa.CheckConstraint( + "failed >= 0", + name=op.f("ck_sync_jobs_sync_jobs_failed_nonnegative"), + ), + sa.CheckConstraint( + "processed >= 0", + name=op.f("ck_sync_jobs_sync_jobs_processed_nonnegative"), + ), + sa.CheckConstraint( + "processed + failed <= total", + name=op.f("ck_sync_jobs_sync_jobs_progress_within_total"), + ), + sa.CheckConstraint( + "status IN ('queued', 'running', 'completed', 'failed')", + name=op.f("ck_sync_jobs_sync_jobs_status_valid"), + ), + sa.CheckConstraint( + "total >= 0", + name=op.f("ck_sync_jobs_sync_jobs_total_nonnegative"), + ), + sa.ForeignKeyConstraint(["tenant_id"], ["tenants.id"], name=op.f("fk_sync_jobs_tenant_id_tenants")), + sa.PrimaryKeyConstraint("id", name=op.f("pk_sync_jobs")), + ) + + +def downgrade() -> None: + op.drop_table("sync_jobs") + + with op.batch_alter_table("knowledge_base") as batch_op: + batch_op.add_column( + sa.Column("category", sa.String(length=100), server_default="Uncategorized", nullable=False) + ) + batch_op.add_column(sa.Column("tag", sa.String(length=100), nullable=True)) + batch_op.drop_constraint("ck_knowledge_base_kb_origin_valid", type_="check") + batch_op.drop_column("origin") + batch_op.drop_column("source_url") + batch_op.alter_column( + "content", + new_column_name="entry", + existing_type=sa.Text(), + existing_nullable=False, + ) diff --git a/alembic/versions/ac67923a3324_add_created_updated_skipped_to_sync_jobs.py b/alembic/versions/ac67923a3324_add_created_updated_skipped_to_sync_jobs.py new file mode 100644 index 0000000000000000000000000000000000000000..6dd08242cddbbcc3524c74a4ec54d5f1e43fb05b --- /dev/null +++ b/alembic/versions/ac67923a3324_add_created_updated_skipped_to_sync_jobs.py @@ -0,0 +1,36 @@ +"""add created updated skipped to sync_jobs + +Revision ID: ac67923a3324 +Revises: a53f4c8d9b21 +Create Date: 2026-05-02 15:38:19.664806 + +""" +from typing import Sequence, Union + +from alembic import op +import sqlalchemy as sa + + +# revision identifiers, used by Alembic. +revision: str = 'ac67923a3324' +down_revision: Union[str, Sequence[str], None] = 'a53f4c8d9b21' +branch_labels: Union[str, Sequence[str], None] = None +depends_on: Union[str, Sequence[str], None] = None + + +def upgrade() -> None: + """Upgrade schema.""" + # ### commands auto generated by Alembic - please adjust! ### + op.add_column('sync_jobs', sa.Column('created', sa.Integer(), server_default='0', nullable=False)) + op.add_column('sync_jobs', sa.Column('updated', sa.Integer(), server_default='0', nullable=False)) + op.add_column('sync_jobs', sa.Column('skipped', sa.Integer(), server_default='0', nullable=False)) + # ### end Alembic commands ### + + +def downgrade() -> None: + """Downgrade schema.""" + # ### commands auto generated by Alembic - please adjust! ### + op.drop_column('sync_jobs', 'skipped') + op.drop_column('sync_jobs', 'updated') + op.drop_column('sync_jobs', 'created') + # ### end Alembic commands ### diff --git a/alembic/versions/b58e26e702df_add_is_blocked_to_tenants_and_note_.py b/alembic/versions/b58e26e702df_add_is_blocked_to_tenants_and_note_.py new file mode 100644 index 0000000000000000000000000000000000000000..76ddaf6e40938b49843484c3969ea66921abf083 --- /dev/null +++ b/alembic/versions/b58e26e702df_add_is_blocked_to_tenants_and_note_.py @@ -0,0 +1,36 @@ +"""add is_blocked to tenants and note created_by to credit_events + +Revision ID: b58e26e702df +Revises: 9ad31e299e42 +Create Date: 2026-05-27 16:58:30.128906 + +""" +from typing import Sequence, Union + +from alembic import op +import sqlalchemy as sa + + +# revision identifiers, used by Alembic. +revision: str = 'b58e26e702df' +down_revision: Union[str, Sequence[str], None] = '9ad31e299e42' +branch_labels: Union[str, Sequence[str], None] = None +depends_on: Union[str, Sequence[str], None] = None + + +def upgrade() -> None: + """Upgrade schema.""" + # ### commands auto generated by Alembic - please adjust! ### + op.add_column('credit_events', sa.Column('note', sa.Text(), nullable=True)) + op.add_column('credit_events', sa.Column('created_by', sa.String(length=100), nullable=True)) + op.add_column('tenants', sa.Column('is_blocked', sa.Boolean(), server_default='false', nullable=False)) + # ### end Alembic commands ### + + +def downgrade() -> None: + """Downgrade schema.""" + # ### commands auto generated by Alembic - please adjust! ### + op.drop_column('tenants', 'is_blocked') + op.drop_column('credit_events', 'created_by') + op.drop_column('credit_events', 'note') + # ### end Alembic commands ### diff --git a/alembic/versions/bf4016f7dd2a_add_website_url_to_tenants.py b/alembic/versions/bf4016f7dd2a_add_website_url_to_tenants.py new file mode 100644 index 0000000000000000000000000000000000000000..1e6aa7a957c5a624a5f552a345d0c40edc57537f --- /dev/null +++ b/alembic/versions/bf4016f7dd2a_add_website_url_to_tenants.py @@ -0,0 +1,32 @@ +"""add website_url to tenants + +Revision ID: bf4016f7dd2a +Revises: 3d1418b32af0 +Create Date: 2026-04-24 08:49:11.717539 + +""" +from typing import Sequence, Union + +from alembic import op +import sqlalchemy as sa + + +# revision identifiers, used by Alembic. +revision: str = 'bf4016f7dd2a' +down_revision: Union[str, Sequence[str], None] = '3d1418b32af0' +branch_labels: Union[str, Sequence[str], None] = None +depends_on: Union[str, Sequence[str], None] = None + + +def upgrade() -> None: + """Upgrade schema.""" + # ### commands auto generated by Alembic - please adjust! ### + op.add_column('tenants', sa.Column('website_url', sa.String(length=255), nullable=True)) + # ### end Alembic commands ### + + +def downgrade() -> None: + """Downgrade schema.""" + # ### commands auto generated by Alembic - please adjust! ### + op.drop_column('tenants', 'website_url') + # ### end Alembic commands ### diff --git a/alembic/versions/c9e1f8a2b7d4_add_auth_fields_to_tenants.py b/alembic/versions/c9e1f8a2b7d4_add_auth_fields_to_tenants.py new file mode 100644 index 0000000000000000000000000000000000000000..8fdb90ddab66621f5d98a75200ff28b8e88a83ea --- /dev/null +++ b/alembic/versions/c9e1f8a2b7d4_add_auth_fields_to_tenants.py @@ -0,0 +1,39 @@ +"""add_auth_fields_to_tenants + +Revision ID: c9e1f8a2b7d4 +Revises: 84de4a13cffd +Create Date: 2026-04-17 00:00:00.000000 + +""" +from typing import Sequence, Union + +from alembic import op +import sqlalchemy as sa + + +# revision identifiers, used by Alembic. +revision: str = 'c9e1f8a2b7d4' +down_revision: Union[str, Sequence[str], None] = '84de4a13cffd' +branch_labels: Union[str, Sequence[str], None] = None +depends_on: Union[str, Sequence[str], None] = None + + +def upgrade() -> None: + """Upgrade schema.""" + op.add_column('tenants', sa.Column('owner_email', sa.String(255), nullable=True)) + op.add_column('tenants', sa.Column('google_id', sa.String(255), nullable=True)) + op.add_column('tenants', sa.Column('totp_secret', sa.String(512), nullable=True)) + op.add_column('tenants', sa.Column('totp_enabled', sa.Boolean(), nullable=True)) + + op.create_unique_constraint('uq_tenants_owner_email', 'tenants', ['owner_email']) + op.create_unique_constraint('uq_tenants_google_id', 'tenants', ['google_id']) + + +def downgrade() -> None: + """Downgrade schema.""" + op.drop_constraint('uq_tenants_google_id', 'tenants', type_='unique') + op.drop_constraint('uq_tenants_owner_email', 'tenants', type_='unique') + op.drop_column('tenants', 'totp_enabled') + op.drop_column('tenants', 'totp_secret') + op.drop_column('tenants', 'google_id') + op.drop_column('tenants', 'owner_email') diff --git a/alembic/versions/d1e126fba5fb_add_title_category_tag_to_kb_entries.py b/alembic/versions/d1e126fba5fb_add_title_category_tag_to_kb_entries.py new file mode 100644 index 0000000000000000000000000000000000000000..6dcaf561160b73610b6494768b49ae351d39e2d2 --- /dev/null +++ b/alembic/versions/d1e126fba5fb_add_title_category_tag_to_kb_entries.py @@ -0,0 +1,36 @@ +"""add title category tag to kb entries + +Revision ID: d1e126fba5fb +Revises: 9b95a5ce2ac1 +Create Date: 2026-04-24 09:32:20.768068 + +""" +from typing import Sequence, Union + +from alembic import op +import sqlalchemy as sa + + +# revision identifiers, used by Alembic. +revision: str = 'd1e126fba5fb' +down_revision: Union[str, Sequence[str], None] = '9b95a5ce2ac1' +branch_labels: Union[str, Sequence[str], None] = None +depends_on: Union[str, Sequence[str], None] = None + + +def upgrade() -> None: + """Upgrade schema.""" + # ### commands auto generated by Alembic - please adjust! ### + op.add_column('knowledge_base', sa.Column('title', sa.String(length=255), server_default='Untitled', nullable=False)) + op.add_column('knowledge_base', sa.Column('category', sa.String(length=100), server_default='Uncategorized', nullable=False)) + op.add_column('knowledge_base', sa.Column('tag', sa.String(length=100), nullable=True)) + # ### end Alembic commands ### + + +def downgrade() -> None: + """Downgrade schema.""" + # ### commands auto generated by Alembic - please adjust! ### + op.drop_column('knowledge_base', 'tag') + op.drop_column('knowledge_base', 'category') + op.drop_column('knowledge_base', 'title') + # ### end Alembic commands ### diff --git a/alembic/versions/d2d812e2c226_add_last_message_preview_to_.py b/alembic/versions/d2d812e2c226_add_last_message_preview_to_.py new file mode 100644 index 0000000000000000000000000000000000000000..00913a29149e3c57b1682c9a33a7915111e1517e --- /dev/null +++ b/alembic/versions/d2d812e2c226_add_last_message_preview_to_.py @@ -0,0 +1,32 @@ +"""add_last_message_preview_to_conversations + +Revision ID: d2d812e2c226 +Revises: 73ae38d1200c +Create Date: 2026-03-30 10:33:10.409942 + +""" +from typing import Sequence, Union + +from alembic import op +import sqlalchemy as sa + + +# revision identifiers, used by Alembic. +revision: str = 'd2d812e2c226' +down_revision: Union[str, Sequence[str], None] = '73ae38d1200c' +branch_labels: Union[str, Sequence[str], None] = None +depends_on: Union[str, Sequence[str], None] = None + + +def upgrade() -> None: + """Upgrade schema.""" + # ### commands auto generated by Alembic - please adjust! ### + op.add_column('conversations', sa.Column('last_message_preview', sa.String(length=255), nullable=True)) + # ### end Alembic commands ### + + +def downgrade() -> None: + """Downgrade schema.""" + # ### commands auto generated by Alembic - please adjust! ### + op.drop_column('conversations', 'last_message_preview') + # ### end Alembic commands ### diff --git a/alembic/versions/d7470fcd70c6_add_backup_codes_to_tenants.py b/alembic/versions/d7470fcd70c6_add_backup_codes_to_tenants.py new file mode 100644 index 0000000000000000000000000000000000000000..77d614da7e189a88b0c95c994cb87c022a7f4154 --- /dev/null +++ b/alembic/versions/d7470fcd70c6_add_backup_codes_to_tenants.py @@ -0,0 +1,32 @@ +"""add_backup_codes_to_tenants + +Revision ID: d7470fcd70c6 +Revises: c9e1f8a2b7d4 +Create Date: 2026-04-17 15:21:51.999768 + +""" +from typing import Sequence, Union + +from alembic import op +import sqlalchemy as sa + + +# revision identifiers, used by Alembic. +revision: str = 'd7470fcd70c6' +down_revision: Union[str, Sequence[str], None] = 'c9e1f8a2b7d4' +branch_labels: Union[str, Sequence[str], None] = None +depends_on: Union[str, Sequence[str], None] = None + + +def upgrade() -> None: + """Upgrade schema.""" + # ### commands auto generated by Alembic - please adjust! ### + op.add_column('tenants', sa.Column('backup_codes', sa.Text(), nullable=True)) + # ### end Alembic commands ### + + +def downgrade() -> None: + """Downgrade schema.""" + # ### commands auto generated by Alembic - please adjust! ### + op.drop_column('tenants', 'backup_codes') + # ### end Alembic commands ### diff --git a/alembic/versions/e5cc44e2040a_make_tenant_id_non_nullable_on_users_.py b/alembic/versions/e5cc44e2040a_make_tenant_id_non_nullable_on_users_.py new file mode 100644 index 0000000000000000000000000000000000000000..ba411432888e8e71659f64e658f9a4885b27f38d --- /dev/null +++ b/alembic/versions/e5cc44e2040a_make_tenant_id_non_nullable_on_users_.py @@ -0,0 +1,44 @@ +"""make tenant_id non-nullable on users; change entities to JSON on messages + +Revision ID: e5cc44e2040a +Revises: f6988c25696e +Create Date: 2026-03-27 15:22:44.809002 + +""" +from typing import Sequence, Union + +from alembic import op +import sqlalchemy as sa +from sqlalchemy.dialects import postgresql + +# revision identifiers, used by Alembic. +revision: str = 'e5cc44e2040a' +down_revision: Union[str, Sequence[str], None] = 'f6988c25696e' +branch_labels: Union[str, Sequence[str], None] = None +depends_on: Union[str, Sequence[str], None] = None + + +def upgrade() -> None: + """Upgrade schema.""" + # ### commands auto generated by Alembic - please adjust! ### + op.execute("ALTER TABLE messages ALTER COLUMN entities TYPE JSONB USING entities::jsonb") + op.execute("DELETE FROM messages WHERE conversation_id IN (SELECT id FROM conversations WHERE user_id IN (SELECT id FROM users WHERE tenant_id IS NULL))") + op.execute("DELETE FROM conversations WHERE user_id IN (SELECT id FROM users WHERE tenant_id IS NULL)") + op.execute("DELETE FROM users WHERE tenant_id IS NULL") + op.alter_column('users', 'tenant_id', + existing_type=sa.UUID(), + nullable=False) + # ### end Alembic commands ### + + +def downgrade() -> None: + """Downgrade schema.""" + # ### commands auto generated by Alembic - please adjust! ### + op.alter_column('users', 'tenant_id', + existing_type=sa.UUID(), + nullable=True) + op.alter_column('messages', 'entities', + existing_type=postgresql.JSONB(astext_type=sa.Text()), + type_=sa.TEXT(), + existing_nullable=True) + # ### end Alembic commands ### diff --git a/alembic/versions/f6988c25696e_add_customization_to_tenants.py b/alembic/versions/f6988c25696e_add_customization_to_tenants.py new file mode 100644 index 0000000000000000000000000000000000000000..dfe092bb54cc93f0415bfa68e3fbc2d0cc9e1375 --- /dev/null +++ b/alembic/versions/f6988c25696e_add_customization_to_tenants.py @@ -0,0 +1,29 @@ +"""add_customization_to_tenants + +Revision ID: f6988c25696e +Revises: 5f697ba37cb7 +Create Date: 2026-03-24 23:33:21.014824 + +""" +from typing import Sequence, Union + +from alembic import op +import sqlalchemy as sa +from sqlalchemy.dialects import postgresql + + +# revision identifiers, used by Alembic. +revision: str = 'f6988c25696e' +down_revision: Union[str, Sequence[str], None] = '5f697ba37cb7' +branch_labels: Union[str, Sequence[str], None] = None +depends_on: Union[str, Sequence[str], None] = None + + +def upgrade() -> None: + """Upgrade schema.""" + op.add_column("tenants", sa.Column("customization", postgresql.JSONB(), nullable=True)) + + +def downgrade() -> None: + """Downgrade schema.""" + op.drop_column("tenants", "customization") diff --git a/app/dependencies/__init__.py b/app/dependencies/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/app/dependencies/admin_auth.py b/app/dependencies/admin_auth.py new file mode 100644 index 0000000000000000000000000000000000000000..afae455ed808c0f44f14ba475713dc64c9723c69 --- /dev/null +++ b/app/dependencies/admin_auth.py @@ -0,0 +1,23 @@ +import uuid + +from fastapi import HTTPException, Request + +from app.services.google_auth import decode_full_jwt +from app.utils.config import get_config + +config = get_config() + + +def require_admin_jwt(request: Request) -> uuid.UUID: + token = request.cookies.get("access_token") + if not token: + raise HTTPException(status_code=401, detail="Unauthorized") + + try: + tenant_id_str = decode_full_jwt(token, config.jwt_secret_key) + tenant_id = uuid.UUID(tenant_id_str) + except (ValueError, AttributeError): + raise HTTPException(status_code=401, detail="Unauthorized") + + request.state.tenant_id = tenant_id + return tenant_id diff --git a/app/dependencies/super_admin_auth.py b/app/dependencies/super_admin_auth.py new file mode 100644 index 0000000000000000000000000000000000000000..cdbd485ef184a10eaac3ac8153e6fcd183554b33 --- /dev/null +++ b/app/dependencies/super_admin_auth.py @@ -0,0 +1,27 @@ +from typing import NoReturn + +from fastapi import HTTPException, Request + +from app.services.google_auth import decode_super_admin_jwt +from app.utils.config import get_config + +config = get_config() + + +def require_super_admin_jwt(request: Request) -> str: + def _reject() -> NoReturn: + raise HTTPException(status_code=403, detail="Forbidden") + + token = request.cookies.get("super_admin_token") + if not token: + _reject() + + try: + email = decode_super_admin_jwt(token, config.jwt_secret_key) + except ValueError: + _reject() + + if email.lower() not in [e.lower() for e in config.super_admin_emails]: + _reject() + + return email diff --git a/app/main.py b/app/main.py index 67b72275fd8acde83a91302e38833c1ecbd1a1fc..db3dbea982a046a7b38e4a80ab3b64bfe8cbc5a3 100644 --- a/app/main.py +++ b/app/main.py @@ -1,10 +1,15 @@ # app/main.py # pylint: disable=unused-import +from decimal import Decimal +from typing import Optional import uuid import time +from collections.abc import AsyncIterator from contextlib import asynccontextmanager from pathlib import Path -from fastapi import FastAPI, HTTPException +from fastapi import APIRouter, BackgroundTasks, Depends, FastAPI, HTTPException, Request +from sqlalchemy import select +from sqlalchemy.orm import Session from fastapi.middleware.cors import CORSMiddleware from fastapi.staticfiles import StaticFiles from pydantic import BaseModel @@ -12,27 +17,68 @@ from pydantic import BaseModel # NOTE: Using absolute imports from project root (app.*) instead of relative imports # This ensures imports work consistently in both IDE and Docker environments from app.middleware.tenant_auth import TenantAuthMiddleware +from app.services.credit_service import ( + CreditService, + InsufficientCreditsError, + ModelRateNotFoundError, +) +from app.services.dependencies import ( + get_embedding_service, + get_retrieval_service, + get_vector_store, +) +from app.services.retrieval_service import RetrievalService from app.utils.config import get_config, get_logger, get_version from app.models.nlp_engine import NLPEngine from app.models.conversation_manager import ConversationManager from app.models.database import ( + LLMModel, SessionLocal, + Tenant, + get_db, init_database, health_check as db_health_check, ) -from app.models.smart_models import UserPreference, UserInsight, ConversationTopic +from app.models.smart_models import ( + UserPreference, + UserInsight, + ConversationTopic, +) # this ensures Base in database.py imports these tables from app.data.training_data import TRAINING_DATA -from app.routers import knowledge_base +from app.routers import ( + admin_attention, + admin_auth, + admin_apikey, + admin_me, + admin_settings, + billing, + conversations, + knowledge_base, + super_admin_auth, + super_admin_tenants, + super_admin_rates, + widget_settings, + system_prompt as system_prompt_router, +) from app.utils.seeding import seed_demo_tenant +from app.utils.token_counter import InputTooLargeError +from app.services.ai_verdict import run_ai_verdict # Load configuration (this sets up logging automatically) config = get_config() logger = get_logger(__name__) +API_V1_PREFIX = f"/api/{config.api_version}" +api_router = APIRouter(prefix=API_V1_PREFIX) + +_qdrant_ready: bool = False + # Train the mode on startup @asynccontextmanager -async def lifespan(fastapi_app: FastAPI): # pylint: disable=unused-argument +async def lifespan( + fastapi_app: FastAPI, +) -> AsyncIterator[None]: # pylint: disable=unused-argument """Application lifespan manager for startup and shutdown events""" # Startup logger.info("Starting application..") @@ -42,7 +88,8 @@ async def lifespan(fastapi_app: FastAPI): # pylint: disable=unused-argument try: init_database() with SessionLocal() as session: - seed_demo_tenant(session) + if config.env.name == "development": + seed_demo_tenant(session) if db_health_check(): logger.info("Database connection established successfully") @@ -52,9 +99,25 @@ async def lifespan(fastapi_app: FastAPI): # pylint: disable=unused-argument logger.error("Database initialization failed: %s: %s", type(e).__name__, str(e)) logger.warning("Continuing without database - some features may not work...") + global _qdrant_ready + try: + get_vector_store().health_check() + _qdrant_ready = True + logger.info("Vector store readiness check passed") + except Exception as e: # pylint: disable=broad-exception-caught + _qdrant_ready = False + logger.error( + "Vector store readiness check failed: %s — starting in degraded mode", str(e) + ) + # Train NLP model nlp_engine.train_intent_classifier(TRAINING_DATA) logger.info("NLP model training completed") + + # Remote Gemini embeddings do not require a local model warmup call. + get_embedding_service() + logger.info("Embedding service initialized successfully") + logger.info("Application started successfully in %s mode", config.env.name) yield # App runs here @@ -64,7 +127,20 @@ async def lifespan(fastapi_app: FastAPI): # pylint: disable=unused-argument app = FastAPI(title=config.api.title, debug=config.api.debug, lifespan=lifespan) -app.include_router(knowledge_base.router) + +api_router.include_router(admin_attention.router) +api_router.include_router(admin_auth.router) +api_router.include_router(admin_apikey.router) +api_router.include_router(admin_me.router) +api_router.include_router(admin_settings.router) +api_router.include_router(knowledge_base.router) +api_router.include_router(billing.router) +api_router.include_router(widget_settings.router) +api_router.include_router(system_prompt_router.router) +api_router.include_router(conversations.router) +api_router.include_router(super_admin_auth.router) +api_router.include_router(super_admin_tenants.router) +api_router.include_router(super_admin_rates.router) # Enable CORS app.add_middleware( @@ -115,34 +191,51 @@ async def health_check(): # Check database health db_status = "healthy" if db_health_check() else "unhealthy" + qdrant_status = "healthy" if _qdrant_ready else "degraded" + overall_status = "healthy" if (db_status == "healthy" and _qdrant_ready) else "degraded" return { - "status": "healthy", + "status": overall_status, "environment": config.env.name, "version": get_version(), # Read from pyproject.toml "database": db_status, + "qdrant": qdrant_status, "components": { "nlp_engine": "ready", "conversation_manager": "ready", "database": db_status, + "qdrant": qdrant_status, }, } -@app.post("/chat", response_model=ChatResponse) -async def chat(request: ChatRequest): - logger.debug("Chat request received: %s ...", request.message[:50]) +@api_router.post("/chat", response_model=ChatResponse) +async def chat( + request: Request, + body: ChatRequest, + background_tasks: BackgroundTasks, + retrieval_service: RetrievalService = Depends(get_retrieval_service), + db: Session = Depends(get_db), +) -> ChatResponse: + logger.debug("Chat request received: %s ...", body.message[:50]) start_time = time.time() + # Get tenant_id from the TenantAuthMiddleware + tenant_id = request.state.tenant_id + + request_id = str(uuid.uuid4()) + try: + credit_service = CreditService() + # Generate session ID if not provided - session_id = request.session_id or str(uuid.uuid4()) + session_id = body.session_id or str(uuid.uuid4()) # Detect language - language = nlp_engine.detect_language(request.message) + language = nlp_engine.detect_language(body.message) # Process message - intent, confidence = nlp_engine.classify_intent(request.message) - entities = nlp_engine.extract_entities(request.message, language) + intent, confidence = nlp_engine.classify_intent(body.message) + entities = nlp_engine.extract_entities(body.message, language) # Convert NumPy types to Python types intent = str(intent) @@ -164,26 +257,107 @@ async def chat(request: ChatRequest): ) intent = "low_confidence" + # Fetch tenant (used for system prompt and billing) + tenant = None + try: + tenant = db.get(Tenant, tenant_id) + except Exception: # pylint: disable=broad-exception-caught + pass + tenant_system_prompt = ( + (tenant.customization or {}).get("system_prompt") if tenant else None + ) + + if tenant and tenant.billing_mode == "credits": + try: + reserve_cost = credit_service.estimate_max_chat_cost( + model_refs=[ + config.chat_model_primary, + *config.chat_model_fallbacks, + ], + max_input_tokens=config.nlp["max_input_tokens"], + max_output_tokens=config.chat_max_output_tokens, + db=db, + ) + except ModelRateNotFoundError: + raise HTTPException(status_code=500, detail="LLM model rate not found") + + if tenant.balance_usd < reserve_cost: + raise HTTPException(status_code=402, detail="Insufficient credits") + + retrieved_docs, retrieved_ids = retrieval_service.retrieve( + tenant_id=str(tenant_id), + query=body.message, + n_results=config.rag_n_results, + ) + # Generate response (now with context awareness) - thinking_block, response = conversation_manager.get_response( - intent, language, request.message, session_id, entities + thinking_block, response, should_summarize, llm_result = conversation_manager.get_response( + intent=intent, + language=language, + user_input=body.message, + tenant_id=tenant_id, + retrieved_docs=retrieved_docs, + session_id=session_id, + entities=entities, + system_prompt=tenant_system_prompt, ) + # get_response() returns str | None; narrow to str before passing downstream + response = response or "I'm sorry, I couldn't generate a response." + + # Billing: deduct credits before saving the conversation + if tenant and tenant.billing_mode == "credits": + try: + llm_model = credit_service.get_llm_model_for_response( + llm_result.model_used, + db, + ) + except ModelRateNotFoundError: + raise HTTPException(status_code=500, detail="LLM model rate not found") + + amount = credit_service.calculate_chat_cost( + input_tokens=llm_result.input_tokens, + output_tokens=llm_result.output_tokens, + llm_model=llm_model, + ) + try: + credit_service.deduct_for_chat( + tenant_id, + amount, + request_id, + llm_model.id, + db, + ) + except InsufficientCreditsError: + raise HTTPException(status_code=402, detail="Insufficient credits") + # Calculate response time response_time_ms = int((time.time() - start_time) * 1000) # Save conversation (now to database) - conversation_manager.save_conversation( - session_id, - request.message, - response, - intent, - confidence, - entities, - language, - response_time_ms, + message_id = conversation_manager.save_conversation( + session_id=session_id, + user_input=body.message, + bot_response=response, + intent=intent, + tenant_id=tenant_id, + confidence=confidence, + entities=entities, + language=language, + response_time_ms=response_time_ms, + retrieved_doc_ids=retrieved_ids, + should_summarize=should_summarize, ) + if message_id: + background_tasks.add_task( + run_ai_verdict, + message_id=message_id, + user_query=body.message, + retrieved_docs=retrieved_docs, + bot_response=response, + ) + # Prepare response chat_response = ChatResponse( thinking=thinking_block, @@ -208,12 +382,24 @@ async def chat(request: ChatRequest): logger.info("Chat response generated successfully for session %s", session_id) return chat_response + except HTTPException: + raise + + except InputTooLargeError as e: + logger.error("Input Too Large error: %s: %s", type(e).__name__, str(e)) + raise HTTPException(status_code=413, detail="Input too large") from e + except Exception as e: - logger.error("Error processing chat request: %s", str(e), exc_info=True) + logger.error( + "Error processing chat request: %s: %s", + type(e).__name__, + str(e), + exc_info=True, + ) raise HTTPException(status_code=500, detail="Internal server error") from e -@app.get("/analytics/{session_id}") +@api_router.get("/analytics/{session_id}") async def get_conversation_history(session_id: str): logger.debug("Analytics requested for session: %s", session_id) @@ -237,7 +423,7 @@ async def get_conversation_history(session_id: str): raise HTTPException(status_code=500, detail="Internal server error") from e -@app.get("/analytics/{session_id}/stats") +@api_router.get("/analytics/{session_id}/stats") async def get_user_stats(session_id: str): logger.debug("User stats requested for session: %s", session_id) @@ -257,6 +443,8 @@ async def get_user_stats(session_id: str): raise HTTPException(status_code=500, detail="Internal server error") from e +app.include_router(api_router) + # Serve frontend static files (HF Spaces has no separate nginx) # Must be mounted LAST so API routes above take precedence _frontend_dir = Path(__file__).parent.parent / "frontend" diff --git a/app/middleware/tenant_auth.py b/app/middleware/tenant_auth.py index 62a533d4b4f73994fcd48d456ae66f164e92764e..8704b6265feb4114245823e7f051dca1354a3d0c 100644 --- a/app/middleware/tenant_auth.py +++ b/app/middleware/tenant_auth.py @@ -5,8 +5,7 @@ from sqlalchemy import select from starlette.middleware.base import BaseHTTPMiddleware from starlette.responses import JSONResponse -from app.models.database import SessionLocal -from app.models.smart_models import Tenant +from app.models.database import SessionLocal, Tenant class TenantAuthMiddleware(BaseHTTPMiddleware): @@ -23,6 +22,13 @@ class TenantAuthMiddleware(BaseHTTPMiddleware): request.method == "OPTIONS" or path in excluded_paths or "." in path.split("/")[-1] # static files (.js, .css, .html, etc.) + or "/admin" in path # admin auth and other admin endpoints use JWT + or "/billing" in path # billing endpoints use JWT + or "/kb/entries" in path # admin KB management → JWT auth + or "/kb/bulk-upload" in path # admin KB bulk upload + or "/kb/sync" in path # admin KB sync + or "/system-prompt" in path # admin system-prompt management → JWT auth + or "/conversations" in path # admin conversations → JWT auth ): return await call_next(request) @@ -46,6 +52,11 @@ class TenantAuthMiddleware(BaseHTTPMiddleware): status_code=403, content={"detail": "Tenant is inactive"} ) + if tenant.is_blocked: # type: ignore[truthy-bool] + return JSONResponse( + status_code=403, content={"detail": "Tenant is blocked"} + ) + # this makes `tenant_id` available to use. If we need to use other properties like `api_key` we need to explicitly set it this way. request.state.tenant_id = tenant.id diff --git a/app/models/conversation_manager.py b/app/models/conversation_manager.py index 72011fd3a692ef1e6404b9fcba12a7e53db1c2b3..38ed7c9d6266a4144f6b250ee3c6ce79e32e7f8a 100644 --- a/app/models/conversation_manager.py +++ b/app/models/conversation_manager.py @@ -1,20 +1,51 @@ -import json import re from datetime import datetime, timezone, timedelta -from typing import Dict, List, Optional -from sqlalchemy import func +from typing import Optional +import uuid +from sqlalchemy import func, select from sqlalchemy.exc import SQLAlchemyError, OperationalError -from groq import Groq -# Absolute imports from project root -from app.utils.config import get_logger -from app.models.database import User, Conversation, Message, SessionLocal +from app.utils.config import Config, get_logger +from app.utils.token_counter import count_tokens +from app.models.database import ( + SummarizationJob, + User, + Conversation, + Message, + SessionLocal, +) from app.models.information_extractor import InformationExtractor +from app.services.llm_service import LLMResult, LLMService + + +CONVERSATION_TIMEOUT_MINUTES = 30 +CONTEXT_MESSAGE_LIMIT = 5 + +# Hierarchical Context Memory +CHARS_PER_TOKEN = 4 +DB_FETCH_LIMIT = 100 +SUMMARIZATION_SYSTEM_PROMPT = ( + "You are a conversation summarizer. You will be given a conversation history " + "between a user and an assistant. Summarize the key points of the conversation " + "in concise prose, covering both what the user asked and what the assistant " + "responded. Focus on topics, decisions, and information exchanged. " + "Do not include greetings or filler. Write in third person. " + "Keep the summary under 200 words." +) + +RAG_SYSTEM_PROMPT = ( + "Use the knowledge base context as the primary source of truth for any " + "business-specific question. If the answer is present in the knowledge base " + "context, answer from it directly and do not fall back to generic assistant " + "memory. If the context does not contain the answer, say that you could not " + "find it in the knowledge base and avoid guessing." +) class ConversationManager: - def __init__(self, config=None): + def __init__(self, config: Config, llm_service: LLMService | None = None): self.config = config + self.llm_service = llm_service or LLMService() self.logger = get_logger(__name__) self.info_extractor = InformationExtractor() @@ -25,13 +56,13 @@ class ConversationManager: max_history, ) - def get_or_create_user(self, session_id: str) -> User: + def get_or_create_user(self, session_id: str, tenant_id: uuid.UUID) -> User: """Get existing user or create new one""" with SessionLocal() as db_session: - user = db_session.query(User).filter(User.session_id == session_id).first() + user = db_session.scalar(select(User).where(User.session_id == session_id)) if not user: - user = User(session_id=session_id) + user = User(session_id=session_id, tenant_id=tenant_id) db_session.add(user) db_session.commit() db_session.refresh(user) @@ -41,6 +72,11 @@ class ConversationManager: user.last_seen = datetime.now(timezone.utc) db_session.commit() + # Reload all columns into memory before the session closes. + # After commit(), SQLAlchemy marks all attributes as expired. + # Without this, accessing user.preferred_language outside the + # `with` block triggers a lazy-load on a detached object → DetachedInstanceError. + db_session.refresh(user) return user def get_active_conversation(self, user: User) -> Optional[Conversation]: @@ -49,15 +85,14 @@ class ConversationManager: # Refresh user object in this session user = db_session.merge(user) - conversation = ( - db_session.query(Conversation) - .filter( + conversation = db_session.scalar( + select(Conversation) + .where( Conversation.user_id == user.id, Conversation.is_active # pylint: disable=singleton-comparison == True, ) # cannot write `Conversation.is_active is True` here, because it's not a Python's `True``, but a SQLAlchemy's `InstrumentedAttribute`` (a special obj that represents the column). Using `is` will always turn False because they are not the same object in Memory .order_by(Conversation.last_message_at.desc()) - .first() ) # If conversation is older than 30 minutes, consider it inactive @@ -71,20 +106,20 @@ class ConversationManager: last_message_utc = conversation.last_message_at time_diff = datetime.now(timezone.utc) - last_message_utc - if time_diff > timedelta(minutes=30): + if time_diff > timedelta(minutes=CONVERSATION_TIMEOUT_MINUTES): conversation.is_active = False db_session.commit() conversation = None return conversation - def create_conversation(self, user: User) -> Conversation: + def create_conversation(self, user: User, tenant_id: uuid.UUID) -> Conversation: """Create a new conversation for the user""" with SessionLocal() as db_session: # Refresh user object in this session user = db_session.merge(user) - conversation = Conversation(user_id=user.id) + conversation = Conversation(user_id=user.id, tenant_id=tenant_id) db_session.add(conversation) db_session.commit() db_session.refresh(conversation) @@ -93,20 +128,19 @@ class ConversationManager: return conversation def get_conversation_context( - self, conversation: Conversation, limit: int = 5 - ) -> List[Dict]: + self, conversation: Conversation, limit: int = CONTEXT_MESSAGE_LIMIT + ) -> list[dict]: """Get recent messages from conversation for context""" with SessionLocal() as db_session: # Refresh conversation object in this section conversation = db_session.merge(conversation) - messages = ( - db_session.query(Message) - .filter(Message.conversation_id == conversation.id) + messages = db_session.scalars( + select(Message) + .where(Message.conversation_id == conversation.id) .order_by(Message.timestamp.desc()) .limit(limit) - .all() - ) + ).all() context = [] for msg in reversed(messages): # Reverse to get chronological order @@ -115,7 +149,9 @@ class ConversationManager: "user_input": msg.user_input, "bot_response": msg.bot_response, "intent": msg.intent, - "timestamp": msg.timestamp.isoformat(), + "timestamp": ( + msg.timestamp.isoformat() if msg.timestamp else None + ), } ) return context @@ -137,7 +173,9 @@ class ConversationManager: ) thinking_block = thinking_block_match.group(1) - bot_response = bot_response_match.group(1) + bot_response = ( + bot_response_match.group(1) if bot_response_match else content + ) else: bot_response = content @@ -148,32 +186,70 @@ class ConversationManager: intent: str, language: str, user_input: str, - session_id: str = None, - entities: Dict = None, - ) -> str: + tenant_id: uuid.UUID, + retrieved_docs: Optional[list[str]] = None, + session_id: Optional[str] = None, + entities: Optional[dict] = None, + system_prompt: Optional[str] = None, + ) -> tuple[Optional[str], Optional[str], bool, LLMResult]: """Generate response with context awareness""" history_messages = [] # Initialize history messages for multi-turn context + summary_message = "" + should_summarize = False + + base_prompt = system_prompt or self.config.system_prompt + system_content = base_prompt + if retrieved_docs: + knowledge_base_entries = "\n\n".join(retrieved_docs) + system_content = ( + f"{base_prompt}\n\n" + f"{RAG_SYSTEM_PROMPT}\n\n" + f"Relevant knowledge base context:\n{knowledge_base_entries}" + ) + + messages = [ + {"role": "system", "content": system_content}, + {"role": "user", "content": user_input}, + ] if session_id: + user = None try: - user = self.get_or_create_user(session_id) + user = self.get_or_create_user( + session_id=session_id, tenant_id=tenant_id + ) conversation = self.get_active_conversation(user) if conversation: - context = self.get_conversation_context(conversation) + context = self.build_hierarchical_context( + conversation=conversation, system_prompt=system_content + ) + + if context["summary"] is not None: + summary_message = context["summary"] + should_summarize = True # Prepare history messages for LLM context - for conv in context: - history_messages.extend( - [ - {"role": "user", "content": conv["user_input"]}, - { - "role": "assistant", - "content": conv["bot_response"], - }, - ] - ) + history_messages.extend(context["verbatim"]) + + # Reassign messages now — before the language update — so a + # failure there can never prevent history from being passed. + messages = ( + [{"role": "system", "content": system_content}] + + ( + [ + { + "role": "system", + "content": f"Summary of earlier conversation:\n{summary_message}", + } + ] + if summary_message + else [] + ) + + history_messages + + [{"role": "user", "content": user_input}] + ) # Update user's preferred language if user and user.preferred_language != language: @@ -194,21 +270,19 @@ class ConversationManager: ) raise - # Create LLM Client - client = Groq(api_key=self.config.llm_api_key) + count_tokens(messages) - llm_response = client.chat.completions.create( - messages=[{"role": "system", "content": self.config.system_prompt}] - + history_messages - + [{"role": "user", "content": user_input}], - model="qwen/qwen3-32b", - ) + self.logger.info("History messages count: %d", len(history_messages)) - thinking_block, response = self.parse_thinking_block( - llm_response.choices[0].message.content + llm_result = self.llm_service.complete_with_usage( + "chat-model", + messages, + max_tokens=self.config.chat_max_output_tokens, ) - return thinking_block, response + thinking_block, response = self.parse_thinking_block(llm_result.content) + + return thinking_block, response, should_summarize, llm_result def save_conversation( self, @@ -216,20 +290,23 @@ class ConversationManager: user_input: str, bot_response: str, intent: str, - confidence: float = None, - entities: Dict = None, + tenant_id: uuid.UUID, + confidence: Optional[float] = None, + entities: Optional[dict] = None, language: str = "en", - response_time_ms: int = None, - ): + response_time_ms: Optional[int] = None, + retrieved_doc_ids: Optional[list[str]] = None, + should_summarize: bool = False, + ) -> Optional[uuid.UUID]: """Save conversation to database""" try: # Get or create user - user = self.get_or_create_user(session_id) + user = self.get_or_create_user(session_id=session_id, tenant_id=tenant_id) # Get or create active conversation conversation = self.get_active_conversation(user) if not conversation: - conversation = self.create_conversation(user) + conversation = self.create_conversation(user=user, tenant_id=tenant_id) # Create message record with SessionLocal() as db_session: @@ -241,26 +318,39 @@ class ConversationManager: conversation_id=conversation.id, user_input=user_input, bot_response=bot_response, + retrieved_doc_ids=retrieved_doc_ids, detected_language=language, intent=intent, confidence=confidence, - entities=json.dumps(entities) if entities else None, + entities=entities if entities else None, response_time_ms=response_time_ms, ) db_session.add(message) + # Check if SummarizationJob should be executed + if should_summarize: + db_session.add(SummarizationJob(conversation_id=conversation.id)) + # Update conversation stats conversation.last_message_at = datetime.now(timezone.utc) - conversation.message_count += 1 + conversation.last_message_preview = user_input[:255] + conversation.message_count = (conversation.message_count or 0) + 1 # Update user stats - user.total_messages += 1 + user.total_messages = (user.total_messages or 0) + 1 user.last_seen = datetime.now(timezone.utc) db_session.commit() - self.logger.debug("Conversation saved for session %s", session_id) + # Capture message.id + message_id: Optional[uuid.UUID] = message.id + + self.logger.debug( + "Conversation saved for session: %s, under tenant: %s", + session_id, + str(tenant_id), + ) # Test the extracted info (temp) extracted_info = self.info_extractor.extract_user_information( @@ -268,78 +358,180 @@ class ConversationManager: ) self.logger.info("Extracted info: %s", extracted_info) - except Exception as e: + return message_id + + except Exception as e: # pylint: disable=broad-exception-caught self.logger.error("Failed to save conversation: %s", str(e), exc_info=True) - def get_conversation_history(self, session_id: str) -> List[Dict]: + def get_conversation_history(self, session_id: str) -> list[dict]: """Get conversation history for a session (for analytics endpoint)""" try: with SessionLocal() as db_session: - user = ( - db_session.query(User).filter(User.session_id == session_id).first() + user = db_session.scalar( + select(User).where(User.session_id == session_id) ) if not user: return [] - messages = ( - db_session.query(Message) - .join(Conversation) - .filter(Conversation.user_id == user.id) - .order_by(Message.timestamp.desc()) - .limit(self.config.nlp["max_history"] if self.config else 50) - .all() - ) + messages = db_session.scalars( + select(Message) + .join(Conversation) + .where(Conversation.user_id == user.id) + .order_by(Message.timestamp.desc()) + .limit(self.config.nlp["max_history"] if self.config else 50) + ).all() history = [] for msg in reversed(messages): # Reverse for chronological order history.append( { - "timestamp": msg.timestamp.isoformat(), + "timestamp": ( + msg.timestamp.isoformat() if msg.timestamp else None + ), "user_input": msg.user_input, "bot_response": msg.bot_response, "intent": msg.intent, "confidence": msg.confidence, "language": msg.detected_language, - "entities": json.loads(msg.entities) if msg.entities else {}, + "entities": msg.entities if msg.entities else {}, } ) return history - except Exception as e: + except Exception as e: # pylint: disable=broad-exception-caught self.logger.error( "Failed to get conversation history: %s", str(e), exc_info=True ) return [] - def get_user_stats(self, session_id: str) -> Dict: + def get_user_stats(self, session_id: str) -> dict: """Get user statistics""" try: with SessionLocal() as db_session: - user = ( - db_session.query(User).filter(User.session_id == session_id).first() + user = db_session.scalar( + select(User).where(User.session_id == session_id) ) if not user: return {} # Get conversation count - conversation_count = ( - db_session.query( + conversation_count = db_session.scalar( + select( func.count(Conversation.id) # pylint: disable=not-callable - ) - .filter(Conversation.user_id == user.id) - .scalar() + ).where(Conversation.user_id == user.id) ) return { "session_id": user.session_id, "preferred_language": user.preferred_language, - "first_seen": user.first_seen.isoformat(), - "last_seen": user.last_seen.isoformat(), + "first_seen": ( + user.first_seen.isoformat() if user.first_seen else None + ), + "last_seen": user.last_seen.isoformat() if user.last_seen else None, "total_messages": user.total_messages, "total_conversations": conversation_count, } - except Exception as e: + except Exception as e: # pylint: disable=broad-exception-caught self.logger.error("Failed to get user stats: %s", str(e), exc_info=True) return {} + + # Methods for Hierarchical Context Memory + def _split_verbatim_and_displaced( + self, messages: list[dict], budget: int + ) -> tuple[list[dict], list[dict]]: + + token_count = 0 + cutoff = 0 + + for msg in reversed(messages): + msg_tokens = ( + len(msg["user_input"]) + len(msg["bot_response"]) + ) // CHARS_PER_TOKEN + + if token_count + msg_tokens > budget: + break + + token_count += msg_tokens + cutoff += 1 + + verbatim = messages[len(messages) - cutoff :] + displaced = messages[: len(messages) - cutoff] + + return verbatim, displaced + + def build_hierarchical_context( + self, + conversation: Conversation, + system_prompt: str, + ) -> dict: + + # Get a list of formatted messages + with SessionLocal() as db_session: + db_session.merge(conversation) + + messages = db_session.scalars( + select(Message) + .where(Message.conversation_id == conversation.id) + .limit(DB_FETCH_LIMIT) + .order_by(Message.timestamp.asc()) + ).all() + + formatted_messages = [ + {"user_input": msg.user_input, "bot_response": msg.bot_response} + for msg in messages + ] + + # Calculate the budget tokens + max_input_tokens = self.config.nlp["max_input_tokens"] + context_reserve_tokens = self.config.nlp["context_reserve_tokens"] + system_tokens = count_tokens( + [{"role": "system", "content": system_prompt or self.config.system_prompt}] + ) + summary_tokens = ( + len(conversation.context_summary) // CHARS_PER_TOKEN + if conversation.context_summary + else 0 + ) + + budget = ( + max_input_tokens - system_tokens - summary_tokens - context_reserve_tokens + ) + + verbatim, displaced = self._split_verbatim_and_displaced( + messages=formatted_messages, budget=budget + ) + + summary = self._summarize_messages(displaced) if displaced else None + verbatim_formatted = [ + item + for msg in verbatim + for item in ( + {"role": "user", "content": msg["user_input"]}, + {"role": "assistant", "content": msg["bot_response"]}, + ) + ] + + return {"verbatim": verbatim_formatted, "summary": summary} + + def _summarize_messages(self, messages: list[dict]) -> Optional[str]: + + formatted_messages: list[dict] = [] + for msg in messages: + formatted_messages.extend( + [ + {"role": "user", "content": msg["user_input"]}, + {"role": "assistant", "content": msg["bot_response"]}, + ] + ) + + formatted_messages.insert( + 0, {"role": "system", "content": SUMMARIZATION_SYSTEM_PROMPT} + ) + + content = self.llm_service.complete("summarization-model", formatted_messages) + + _, response = self.parse_thinking_block(content) + + return response diff --git a/app/models/database.py b/app/models/database.py index 6d9b43e931ee628329774a2a640e1f9cb43b4bd3..f5f2d809439de083f945bb3162606d468dea746e 100644 --- a/app/models/database.py +++ b/app/models/database.py @@ -1,9 +1,17 @@ # app/models/database.py -from datetime import datetime, timezone +from datetime import datetime, timezone, date +from decimal import Decimal +from typing import Optional, TYPE_CHECKING import uuid from sqlalchemy import ( + JSON, + Index, + Numeric, + Date, + SmallInteger, + UniqueConstraint, + CheckConstraint, create_engine, - Column, Integer, String, Text, @@ -12,19 +20,30 @@ from sqlalchemy import ( Boolean, ForeignKey, MetaData, + text, ) # from sqlalchemy.ext.declarative import declarative_base -from sqlalchemy.orm import sessionmaker, relationship, declarative_base -from sqlalchemy.dialects.postgresql import UUID +from sqlalchemy.orm import ( + sessionmaker, + relationship, + DeclarativeBase, + Mapped, + mapped_column, +) +from sqlalchemy.dialects.postgresql import UUID, JSONB -# Absolute import from project root from app.utils.config import get_logger, config_manager +if TYPE_CHECKING: + from app.models.smart_models import UserInsight, UserPreference, ConversationTopic + config = config_manager.get_config() logger = get_logger(__name__) +# Naming conventions for auto-generated constraint names (indexes, unique constraints, +# foreign keys, etc.) so Alembic can reliably identify and diff them across migrations. convention = { "ix": "ix_%(column_0_label)s", "uq": "uq_%(table_name)s_%(column_0_name)s", @@ -33,7 +52,21 @@ convention = { "pk": "pk_%(table_name)s", } -Base = declarative_base(metadata=MetaData(naming_convention=convention)) + +class Base(DeclarativeBase): + """Declarative base class for all ORM models. + + Defined as a class (subclassing DeclarativeBase) rather than the legacy + `Base = declarative_base()` pattern because SQLAlchemy 2.0 requires the + class-based form to support `Mapped[...]` type annotations and + `mapped_column()`. The old factory function does not support these features. + + All ORM models inherit from this class to register themselves with the + shared metadata and gain SQLAlchemy's mapping machinery. + """ + + metadata = MetaData(naming_convention=convention) + engine = create_engine( config.database_url, @@ -48,25 +81,33 @@ SessionLocal = sessionmaker(autocommit=False, autoflush=False, bind=engine) class User(Base): __tablename__ = "users" - id = Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4) - session_id = Column(String(255), unique=True, index=True) - preferred_language = Column(String(10), default="en") - first_seen = Column(DateTime, default=lambda: datetime.now(timezone.utc)) - last_seen = Column(DateTime, default=lambda: datetime.now(timezone.utc)) - total_messages = Column(Integer, default=0) - tenant_id = Column(UUID(as_uuid=True), ForeignKey("tenants.id"), nullable=True) + id: Mapped[uuid.UUID] = mapped_column( + UUID(as_uuid=True), primary_key=True, default=uuid.uuid4 + ) + session_id: Mapped[str] = mapped_column(String(255), unique=True, index=True) + tenant_id: Mapped[uuid.UUID] = mapped_column( + UUID(as_uuid=True), ForeignKey("tenants.id"), nullable=False + ) + preferred_language: Mapped[Optional[str]] = mapped_column(String(10), default="en") + first_seen: Mapped[Optional[datetime]] = mapped_column( + DateTime, default=lambda: datetime.now(timezone.utc) + ) + last_seen: Mapped[Optional[datetime]] = mapped_column( + DateTime, default=lambda: datetime.now(timezone.utc) + ) + total_messages: Mapped[Optional[int]] = mapped_column(Integer, default=0) # Relationships - conversations = relationship( + tenant: Mapped["Tenant"] = relationship("Tenant", back_populates="users") + conversations: Mapped[list["Conversation"]] = relationship( "Conversation", back_populates="user", cascade="all, delete-orphan" ) - preferences = relationship( + preferences: Mapped[list["UserPreference"]] = relationship( "UserPreference", back_populates="user", cascade="all, delete-orphan" ) - insights = relationship( + insights: Mapped[list["UserInsight"]] = relationship( "UserInsight", back_populates="user", cascade="all, delete-orphan" ) - tenant = relationship("Tenant", back_populates="users") def __repr__(self): return ( @@ -77,23 +118,37 @@ class User(Base): class Conversation(Base): __tablename__ = "conversations" - id = Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4) - user_id = Column(UUID(as_uuid=True), ForeignKey("users.id"), nullable=False) - started_at = Column(DateTime, default=lambda: datetime.now(timezone.utc)) - last_message_at = Column(DateTime, default=lambda: datetime.now(timezone.utc)) - message_count = Column(Integer, default=0) - is_active = Column(Boolean, default=True) - tenant_id = Column(UUID(as_uuid=True), ForeignKey("tenants.id"), nullable=True) + id: Mapped[uuid.UUID] = mapped_column( + UUID(as_uuid=True), primary_key=True, default=uuid.uuid4 + ) + user_id: Mapped[uuid.UUID] = mapped_column( + UUID(as_uuid=True), ForeignKey("users.id"), nullable=False + ) + tenant_id: Mapped[uuid.UUID] = mapped_column( + UUID(as_uuid=True), ForeignKey("tenants.id"), nullable=True + ) + started_at: Mapped[Optional[datetime]] = mapped_column( + DateTime, default=lambda: datetime.now(timezone.utc) + ) + last_message_at: Mapped[Optional[datetime]] = mapped_column( + DateTime, default=lambda: datetime.now(timezone.utc) + ) + last_message_preview: Mapped[Optional[str]] = mapped_column( + String(255), nullable=True + ) + message_count: Mapped[Optional[int]] = mapped_column(Integer, default=0) + is_active: Mapped[Optional[bool]] = mapped_column(Boolean, default=True) + context_summary: Mapped[Optional[str]] = mapped_column(Text, nullable=True) # Relationship - user = relationship("User", back_populates="conversations") - messages = relationship( + user: Mapped["User"] = relationship("User", back_populates="conversations") + messages: Mapped[list["Message"]] = relationship( "Message", back_populates="conversation", cascade="all, delete-orphan" ) - topics = relationship( + topics: Mapped[list["ConversationTopic"]] = relationship( "ConversationTopic", back_populates="conversation", cascade="all, delete-orphan" ) - tenant = relationship("Tenant", back_populates="conversations") + tenant: Mapped["Tenant"] = relationship("Tenant", back_populates="conversations") def __repr__(self): return f"" @@ -102,36 +157,258 @@ class Conversation(Base): class Message(Base): __tablename__ = "messages" - id = Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4) - conversation_id = Column( + id: Mapped[uuid.UUID] = mapped_column( + UUID(as_uuid=True), primary_key=True, default=uuid.uuid4 + ) + conversation_id: Mapped[uuid.UUID] = mapped_column( UUID(as_uuid=True), ForeignKey("conversations.id"), nullable=False ) # Message content - user_input = Column(Text, nullable=False) - bot_response = Column(Text, nullable=False) + user_input: Mapped[str] = mapped_column(Text, nullable=False) + bot_response: Mapped[str] = mapped_column(Text, nullable=False) # NLP analysis - detected_language = Column(String(10)) - intent = Column(String(100)) - confidence = Column(Float) - entities = Column(Text) # JSON string + retrieved_doc_ids: Mapped[Optional[list]] = mapped_column( + JSONB().with_variant(JSON(), "sqlite"), nullable=True + ) + detected_language: Mapped[Optional[str]] = mapped_column(String(10)) + intent: Mapped[Optional[str]] = mapped_column(String(100)) + confidence: Mapped[Optional[float]] = mapped_column(Float) + entities: Mapped[Optional[dict]] = mapped_column( + JSONB().with_variant(JSON(), "sqlite") + ) # Metadata - timestamp = Column(DateTime, default=lambda: datetime.now(timezone.utc)) - response_time_ms = Column(Integer) # How long the bot took to respond + timestamp: Mapped[Optional[datetime]] = mapped_column( + DateTime, default=lambda: datetime.now(timezone.utc) + ) + response_time_ms: Mapped[Optional[int]] = mapped_column( + Integer + ) # How long the bot took to respond + + # Feedback + user_feedback: Mapped[Optional[int]] = mapped_column(SmallInteger, nullable=True) + tenant_feedback: Mapped[Optional[int]] = mapped_column(SmallInteger, nullable=True) + + # AI verdict scores (filled async by judge LLM after each response) + accuracy_score: Mapped[Optional[float]] = mapped_column( + Float, nullable=True + ) # factual correctness vs retrieved docs + relevance_score: Mapped[Optional[float]] = mapped_column( + Float, nullable=True + ) # how well the response addressed the user query + safety_score: Mapped[Optional[float]] = mapped_column( + Float, nullable=True + ) # did bot stay on-domain and avoid harmful/misleading claims (0.0 = unsafe, 1.0 = safe) # Relationships - conversation = relationship("Conversation", back_populates="messages") + conversation: Mapped["Conversation"] = relationship( + "Conversation", back_populates="messages" + ) def __repr__(self): return f"" +class Tenant(Base): + __tablename__ = "tenants" + __table_args__ = ( + CheckConstraint("billing_mode IN ('credits', 'subscription', 'gifted')", name="chk_tenant_billing_mode"), + ) + + id: Mapped[uuid.UUID] = mapped_column( + UUID(as_uuid=True), primary_key=True, default=uuid.uuid4 + ) + name: Mapped[str] = mapped_column(String(255), nullable=False) + website_url: Mapped[Optional[str]] = mapped_column(String(255), nullable=True) + api_key: Mapped[str] = mapped_column(String(255), unique=True, nullable=False) + is_active: Mapped[Optional[bool]] = mapped_column(Boolean, default=True) + created_at: Mapped[Optional[datetime]] = mapped_column( + DateTime, default=lambda: datetime.now(timezone.utc) + ) + updated_at: Mapped[Optional[datetime]] = mapped_column( + DateTime, + default=lambda: datetime.now(timezone.utc), + onupdate=lambda: datetime.now(timezone.utc), + ) + customization: Mapped[Optional[dict]] = mapped_column( + JSONB().with_variant(JSON(), "sqlite"), nullable=True + ) + + # Admin panel auth fields + owner_email: Mapped[Optional[str]] = mapped_column( + String(255), nullable=True, unique=True + ) + google_id: Mapped[Optional[str]] = mapped_column( + String(255), nullable=True, unique=True + ) + # Stores Fernet-encrypted TOTP seed — never the raw secret + totp_secret: Mapped[Optional[str]] = mapped_column(String(512), nullable=True) + totp_enabled: Mapped[Optional[bool]] = mapped_column(Boolean, default=False) + # Stores JSON array of SHA-256-hashed backup codes (plaintext shown once at setup) + backup_codes: Mapped[Optional[str]] = mapped_column(Text, nullable=True) + + backup_email: Mapped[Optional[str]] = mapped_column(String(255), nullable=True) + phone: Mapped[Optional[str]] = mapped_column(String(20), nullable=True) + + # Super-admin controls + is_blocked: Mapped[bool] = mapped_column( + Boolean, default=False, nullable=False, server_default="false" + ) + + # Billing + balance_usd: Mapped[Decimal] = mapped_column( + Numeric(20, 6), nullable=False, server_default="0" + ) + billing_mode: Mapped[str] = mapped_column( + String(20), nullable=False, server_default="credits" + ) + + # Relationships + users: Mapped[list["User"]] = relationship("User", back_populates="tenant") + conversations: Mapped[list["Conversation"]] = relationship( + "Conversation", back_populates="tenant" + ) + credit_events: Mapped[list["CreditEvent"]] = relationship( + "CreditEvent", back_populates="tenant" + ) + auth_providers: Mapped[list["TenantAuthProvider"]] = relationship( + "TenantAuthProvider", back_populates="tenant", cascade="all, delete-orphan" + ) + + def __repr__(self): + return f"" + + +class LLMModel(Base): + __tablename__ = "llm_models" + + id: Mapped[int] = mapped_column(Integer, primary_key=True, autoincrement=True) + provider: Mapped[str] = mapped_column(String(100), nullable=False) + model_name: Mapped[str] = mapped_column(String(255), nullable=False) + usd_per_1k_input: Mapped[Decimal] = mapped_column(Numeric(20, 6), nullable=False) + usd_per_1k_output: Mapped[Decimal] = mapped_column(Numeric(20, 6), nullable=False) + effective_from: Mapped[date] = mapped_column(Date, nullable=False) + effective_to: Mapped[Optional[date]] = mapped_column(Date, nullable=True) + + def __repr__(self): + return f"" + + +class CreditEvent(Base): + __tablename__ = "credit_events" + __table_args__ = ( + UniqueConstraint( + "tenant_id", "request_id", name="uq_credit_events_tenant_id_request_id" + ), + ) + + id: Mapped[uuid.UUID] = mapped_column( + UUID(as_uuid=True), primary_key=True, default=uuid.uuid4 + ) + tenant_id: Mapped[uuid.UUID] = mapped_column( + UUID(as_uuid=True), ForeignKey("tenants.id"), nullable=False, index=True + ) + event_type: Mapped[str] = mapped_column(String(20), nullable=False) # "deduct" | "topup" + amount: Mapped[Decimal] = mapped_column(Numeric(20, 6), nullable=False) + balance_after: Mapped[Decimal] = mapped_column(Numeric(20, 6), nullable=False) + note: Mapped[Optional[str]] = mapped_column(Text, nullable=True) + created_by: Mapped[Optional[str]] = mapped_column(String(100), nullable=True) + created_at: Mapped[datetime] = mapped_column( + DateTime, default=lambda: datetime.now(timezone.utc) + ) + request_id: Mapped[Optional[str]] = mapped_column(String(255), nullable=True) + llm_model_id: Mapped[Optional[int]] = mapped_column( + Integer, ForeignKey("llm_models.id"), nullable=True + ) + + # Relationships + tenant: Mapped["Tenant"] = relationship("Tenant", back_populates="credit_events") + llm_model: Mapped[Optional["LLMModel"]] = relationship("LLMModel") + + def __repr__(self): + return f"" + + +class TenantAuthProvider(Base): + __tablename__ = "tenant_auth_providers" + __table_args__ = ( + UniqueConstraint("tenant_id", "provider", name="uq_tenant_auth_providers_tenant_id_provider"), + CheckConstraint("provider IN ('google', 'telegram')", name="chk_tenant_auth_providers_provider"), + ) + + id: Mapped[uuid.UUID] = mapped_column( + UUID(as_uuid=True), primary_key=True, default=uuid.uuid4 + ) + tenant_id: Mapped[uuid.UUID] = mapped_column( + UUID(as_uuid=True), ForeignKey("tenants.id", ondelete="CASCADE"), nullable=False, index=True + ) + provider: Mapped[str] = mapped_column(String(20), nullable=False) + provider_user_id: Mapped[str] = mapped_column(String(255), nullable=False) + linked_at: Mapped[datetime] = mapped_column( + DateTime, default=lambda: datetime.now(timezone.utc) + ) + + tenant: Mapped["Tenant"] = relationship("Tenant", back_populates="auth_providers") + + def __repr__(self): + return f"" + + +class SummarizationJob(Base): + __tablename__ = "summarization_jobs" + __table_args__ = ( + Index( + "uq_summarization_jobs_pending", + "conversation_id", + unique=True, + postgresql_where=text("status = 'pending'"), + ), + ) + + id: Mapped[uuid.UUID] = mapped_column( + UUID(as_uuid=True), + primary_key=True, + default=uuid.uuid4, + ) + conversation_id: Mapped[uuid.UUID] = mapped_column( + UUID(as_uuid=True), + ForeignKey("conversations.id"), + nullable=False, + index=True, + ) + status: Mapped[str] = mapped_column( + String(20), + default="pending", + ) + attempt_count: Mapped[int] = mapped_column(Integer, default=0) + created_at: Mapped[datetime] = mapped_column( + DateTime, + default=lambda: datetime.now(timezone.utc), + ) + last_attempted_at: Mapped[Optional[datetime]] = mapped_column(DateTime) + + +class Industry(Base): + __tablename__ = "industries" + + id: Mapped[int] = mapped_column(Integer, primary_key=True, autoincrement=True) + code: Mapped[str] = mapped_column(String(100), unique=True, nullable=False) + display_name: Mapped[str] = mapped_column(String(100), nullable=False) + is_active: Mapped[bool] = mapped_column(Boolean, default=True, nullable=False) + + def __repr__(self): + return f"" + + def init_database(): - # create all tables in the database - Base.metadata.create_all(bind=engine) - logger.info("Database tables created successfully") + """Create all tables on SQLite (HF Spaces / tests). PostgreSQL uses Alembic migrations only.""" + if engine.dialect.name == "sqlite": + Base.metadata.create_all(bind=engine) + logger.info("Database tables created successfully (SQLite)") + else: + logger.info("PostgreSQL detected — skipping create_all(); use 'alembic upgrade head' to apply migrations") def health_check(): @@ -149,6 +426,11 @@ def health_check(): def get_db(): + """FastAPI dependency that provides a database session per request. + + Yields a session and guarantees it is closed after the request finishes, + whether or not an exception occurred. Inject with `db: Session = Depends(get_db)`. + """ db = SessionLocal() try: yield db diff --git a/app/models/information_extractor.py b/app/models/information_extractor.py index d16b440b2882419d41bed2964048fad77bd55c60..ab99e9acadbe9d026b7934191b10edb3a438d446 100644 --- a/app/models/information_extractor.py +++ b/app/models/information_extractor.py @@ -50,7 +50,7 @@ class InformationExtractor: except OSError: return False - def _build_patterns(self) -> Dict[str, List[str]]: + def _build_patterns(self) -> dict[str, list[str] | dict[str, list[str]]]: """Build regex patterns for information extraction""" return { # Name patterns @@ -157,14 +157,16 @@ class InformationExtractor: """Extract product interests and preferences""" interests = [] - for category, patterns in self.patterns["product_mentions"].items(): - for pattern in patterns: - if re.search(pattern, message, re.IGNORECASE): - category_name = category.split("_", maxsplit=1)[ - 0 - ] # "product_x_y" -> "product", "x_y" -> "product" - if category_name not in interests: - interests.append(category_name) + product_mentions = self.patterns["product_mentions"] + if isinstance(product_mentions, dict): + for category, patterns in product_mentions.items(): + for pattern in patterns: + if re.search(pattern, message, re.IGNORECASE): + category_name = category.split("_", maxsplit=1)[ + 0 + ] # "product_x_y" -> "product", "x_y" -> "product" + if category_name not in interests: + interests.append(category_name) # Alternative approach - direct pattern matching product_categories = { @@ -336,6 +338,6 @@ class InformationExtractor: # Convert sets to lists for JSON serialization summary["topics_discussed"] = list(summary["topics_discussed"]) - summary["user_interests"] = List(summary["user_interests"]) + summary["user_interests"] = list(summary["user_interests"]) return summary diff --git a/app/models/smart_models.py b/app/models/smart_models.py index 7533bd741c9f90b5a783f2b8291e8757036c9a86..9fa0222d85971ee32dcb92e6ce7e6a56b42778ca 100644 --- a/app/models/smart_models.py +++ b/app/models/smart_models.py @@ -1,7 +1,19 @@ +"""ORM models for the smart layer — features that learn from and adapt to users. + +These models are separate from the core chatbot models (User, Conversation, Message) +in database.py. They extend the system with intelligence: tracking what users prefer, +what topics they discuss, what behavioral patterns emerge, and what knowledge the +tenant has loaded into the system. + +All models inherit from Base defined in database.py and are imported in main.py to +ensure their tables are registered with Base.metadata before init_database() runs. +""" + from datetime import datetime, timezone +from typing import Optional, TYPE_CHECKING import uuid from sqlalchemy import ( - Column, + CheckConstraint, String, Text, DateTime, @@ -10,139 +22,252 @@ from sqlalchemy import ( Boolean, ForeignKey, ) -from sqlalchemy.orm import relationship +from sqlalchemy.orm import relationship, mapped_column, Mapped from sqlalchemy.dialects.postgresql import UUID # Absolute import from project root from app.models.database import Base +if TYPE_CHECKING: + from app.models.database import User, Conversation + class UserPreference(Base): + """Stores a single learned preference for a user. + + Each row represents one thing the system has inferred about a user — + for example, their name, a product category they care about, or their + preferred communication style. A user can have many preferences, each + identified by a (preference_type, preference_key) pair. + """ + __tablename__ = "user_preferences" - id = Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4) - user_id = Column(UUID(as_uuid=True), ForeignKey("users.id"), nullable=False) + id: Mapped[uuid.UUID] = mapped_column( + UUID(as_uuid=True), primary_key=True, default=uuid.uuid4 + ) + user_id: Mapped[uuid.UUID] = mapped_column( + UUID(as_uuid=True), ForeignKey("users.id"), nullable=False + ) # Preference details - preference_type = Column( + preference_type: Mapped[str] = mapped_column( String(50), nullable=False ) # e.g., "name", "product_interest", "communication_style" - preference_key = Column( + preference_key: Mapped[str] = mapped_column( String(100), nullable=False ) # e.g., "user_name", "preferred_products", "formality_level" - preference_value = Column( + preference_value: Mapped[str] = mapped_column( Text, nullable=False ) # e.g., "John", "toys,gifts", "casual" # Learning metadata - confidence_score = Column( + confidence_score: Mapped[Optional[float]] = mapped_column( Float, default=1.0 ) # How confident the model is (0.0-1.0) - learned_from_messages = Column( + learned_from_messages: Mapped[Optional[int]] = mapped_column( Integer, default=1 ) # Number of messages that taught us this - first_learned = Column(DateTime, default=lambda: datetime.now(timezone.utc)) - last_updated = Column(DateTime, default=lambda: datetime.now(timezone.utc)) + first_learned: Mapped[Optional[datetime]] = mapped_column( + DateTime, default=lambda: datetime.now(timezone.utc) + ) + last_updated: Mapped[Optional[datetime]] = mapped_column( + DateTime, default=lambda: datetime.now(timezone.utc) + ) # Relationship - user = relationship("User", back_populates="preferences") + user: Mapped["User"] = relationship("User", back_populates="preferences") def __repr__(self): return f"" class ConversationTopic(Base): + """Tracks what topics were discussed within a single conversation. + + Each row tags a conversation with a topic (e.g. "product_inquiry") and + optional subtopic (e.g. "toys"). Used to understand what a conversation + was about without re-reading its messages, enabling routing, analytics, + and context-aware follow-ups. + """ + __tablename__ = "conversation_topics" - id = Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4) - conversation_id = Column( + id: Mapped[uuid.UUID] = mapped_column( + UUID(as_uuid=True), primary_key=True, default=uuid.uuid4 + ) + conversation_id: Mapped[uuid.UUID] = mapped_column( UUID(as_uuid=True), ForeignKey("conversations.id"), nullable=False ) # Topic details - topic = Column( + topic: Mapped[str] = mapped_column( String(100), nullable=False ) # "product_inquiry", "order_management", "support" - subtopic = Column(String(100)) # "toys", "cancellation", "shipping_issue" - keywords = Column(Text) # JSON list of relevant keywords + subtopic: Mapped[Optional[str]] = mapped_column( + String(100) + ) # "toys", "cancellation", "shipping_issue" + keywords: Mapped[Optional[str]] = mapped_column( + Text + ) # JSON list of relevant keywords # Topic metadata - first_mentioned = Column(DateTime, default=lambda: datetime.now(timezone.utc)) - message_count = Column(Integer, default=1) # How many messages discussed this topic - importance_score = Column( + first_mentioned: Mapped[Optional[datetime]] = mapped_column( + DateTime, default=lambda: datetime.now(timezone.utc) + ) + message_count: Mapped[Optional[int]] = mapped_column( + Integer, default=1 + ) # How many messages discussed this topic + importance_score: Mapped[Optional[float]] = mapped_column( Float, default=1.0 ) # How important this topic was (0.0-1.0) # Relationships - conversation = relationship("Conversation", back_populates="topics") + conversation: Mapped["Conversation"] = relationship( + "Conversation", back_populates="topics" + ) def __repr__(self): return f"" class UserInsight(Base): + """Stores higher-level behavioral patterns derived from a user's history. + + Unlike UserPreference (which records stated or inferred facts), UserInsight + records computed patterns — e.g. "most active in the morning", "asks complex + questions", "prefers toys category". Insights are recalculated over time and + can be deactivated when they are no longer supported by recent data. + """ + __tablename__ = "user_insights" - id = Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4) - user_id = Column(UUID(as_uuid=True), ForeignKey("users.id"), nullable=False) + id: Mapped[uuid.UUID] = mapped_column( + UUID(as_uuid=True), primary_key=True, default=uuid.uuid4 + ) + user_id: Mapped[uuid.UUID] = mapped_column( + UUID(as_uuid=True), ForeignKey("users.id"), nullable=False + ) # Insight details - insight_type = Column( + insight_type: Mapped[str] = mapped_column( String(50), nullable=False ) # "behavior_pattern", "preference_trend", "interaction_style" - insight_key = Column( + insight_key: Mapped[str] = mapped_column( String(100), nullable=False ) # "most_active_time", "preferred_topics", "question_complexity" - insight_value = Column(Text, nullable=False) # "morning", "toys,gifts", "simple" - insight_description = Column(Text) # Human-readable explanation + insight_value: Mapped[str] = mapped_column( + Text, nullable=False + ) # "morning", "toys,gifts", "simple" + insight_description: Mapped[Optional[str]] = mapped_column( + Text + ) # Human-readable explanation # Analytics metadata - confidence_level = Column(Float, default=1.0) # Statistical confidence (0.0-1.0) - based_on_messages = Column(Integer, default=0) # Number of messages analyzed - last_calculated = Column(DateTime, default=lambda: datetime.now(timezone.utc)) - is_active = Column(Boolean, default=True) # Is this insight still relevant? + confidence_level: Mapped[Optional[float]] = mapped_column( + Float, default=1.0 + ) # Statistical confidence (0.0-1.0) + based_on_messages: Mapped[Optional[int]] = mapped_column( + Integer, default=0 + ) # Number of messages analyzed + last_calculated: Mapped[Optional[datetime]] = mapped_column( + DateTime, default=lambda: datetime.now(timezone.utc) + ) + is_active: Mapped[Optional[bool]] = mapped_column( + Boolean, default=True + ) # Is this insight still relevant? # Relationships - user = relationship("User", back_populates="insights") + user: Mapped["User"] = relationship("User", back_populates="insights") def __repr__(self): return f"" class KnowledgeBase(Base): + """Stores tenant-specific knowledge entries used to ground the chatbot's responses. + + Each row is a plain-text passage belonging to a tenant. The chatbot retrieves + relevant entries at query time (RAG) and includes them in the LLM prompt so + responses are based on the tenant's actual content rather than general knowledge. + """ + __tablename__ = "knowledge_base" - id = Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4) - created_at = Column(DateTime, default=lambda: datetime.now(timezone.utc)) - updated_at = Column( + id: Mapped[uuid.UUID] = mapped_column( + UUID(as_uuid=True), primary_key=True, default=uuid.uuid4 + ) + created_at: Mapped[Optional[datetime]] = mapped_column( + DateTime, default=lambda: datetime.now(timezone.utc) + ) + updated_at: Mapped[Optional[datetime]] = mapped_column( DateTime, default=lambda: datetime.now(timezone.utc), onupdate=lambda: datetime.now(timezone.utc), ) - tenant_id = Column(UUID(as_uuid=True), ForeignKey("tenants.id"), nullable=False) - entry = Column(Text, nullable=False) + tenant_id: Mapped[uuid.UUID] = mapped_column( + UUID(as_uuid=True), ForeignKey("tenants.id"), nullable=False + ) + title: Mapped[str] = mapped_column(String(255), nullable=False, server_default="Untitled") + content: Mapped[str] = mapped_column(Text, nullable=False) + source_url: Mapped[Optional[str]] = mapped_column(String(500), nullable=True) + origin: Mapped[str] = mapped_column(String(20), nullable=False, server_default="manual") + is_indexed: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, server_default="false") + + __table_args__ = ( + CheckConstraint("origin IN ('manual', 'synced')", name="kb_origin_valid"), + ) def __repr__(self): return f"" -class Tenant(Base): - __tablename__ = "tenants" +class SyncJob(Base): + """Tracks tenant-owned WordPress sync progress for async background jobs.""" - id = Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4) - name = Column(String(255), nullable=False) - api_key = Column(String(255), unique=True, nullable=False) - is_active = Column(Boolean, default=True) - created_at = Column(DateTime, default=lambda: datetime.now(timezone.utc)) - updated_at = Column( + __tablename__ = "sync_jobs" + + id: Mapped[uuid.UUID] = mapped_column( + UUID(as_uuid=True), primary_key=True, default=uuid.uuid4 + ) + created_at: Mapped[Optional[datetime]] = mapped_column( + DateTime, default=lambda: datetime.now(timezone.utc) + ) + updated_at: Mapped[Optional[datetime]] = mapped_column( DateTime, default=lambda: datetime.now(timezone.utc), onupdate=lambda: datetime.now(timezone.utc), ) + tenant_id: Mapped[uuid.UUID] = mapped_column( + UUID(as_uuid=True), ForeignKey("tenants.id"), nullable=False + ) + source_url: Mapped[str] = mapped_column(String(500), nullable=False) + rest_base: Mapped[str] = mapped_column(String(50), nullable=False, default="posts", server_default="posts") + status: Mapped[str] = mapped_column(String(30), nullable=False, server_default="queued") + total: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0") + processed: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0") + failed: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0") + created: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0") + updated: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0") + skipped: Mapped[int] = mapped_column(Integer, nullable=False, default=0, server_default="0") + error_message: Mapped[Optional[str]] = mapped_column(Text, nullable=True) - # Relationships - users = relationship("User", back_populates="tenant") - conversations = relationship("Conversation", back_populates="tenant") + __table_args__ = ( + CheckConstraint( + "status IN ('cancelled', 'completed', 'expired', 'failed', 'paused', 'queued', 'running')", + name="sync_jobs_status_valid", + ), + CheckConstraint("total >= 0", name="sync_jobs_total_nonnegative"), + CheckConstraint("processed >= 0", name="sync_jobs_processed_nonnegative"), + CheckConstraint("failed >= 0", name="sync_jobs_failed_nonnegative"), + CheckConstraint( + "processed + failed <= total", + name="sync_jobs_progress_within_total", + ), + CheckConstraint("created >= 0", name="sync_jobs_created_nonnegative"), + CheckConstraint("updated >= 0", name="sync_jobs_updated_nonnegative"), + CheckConstraint("skipped >= 0", name="sync_jobs_skipped_nonnegative"), + ) def __repr__(self): - return f"" + return f"" diff --git a/app/routers/admin_apikey.py b/app/routers/admin_apikey.py new file mode 100644 index 0000000000000000000000000000000000000000..5757b6f614a66586183a081c2f1cccf15c566a10 --- /dev/null +++ b/app/routers/admin_apikey.py @@ -0,0 +1,28 @@ +from fastapi import APIRouter, Depends, HTTPException +from sqlalchemy import select +from sqlalchemy.orm import Session +from app.models.database import get_db, Tenant +from app.dependencies.admin_auth import require_admin_jwt +from app.schemas.admin_apikey import ApiKeyRotateResponse +from app.utils.api_key import generate_api_key +import uuid + +router = APIRouter(prefix="/admin", tags=["admin"]) + +@router.post("/api-key/rotate", response_model=ApiKeyRotateResponse) +def rotate_api_key( + tenant_id: uuid.UUID = Depends(require_admin_jwt), + db: Session = Depends(get_db) +): + """ + Generates a new API key, stores it hashed, and returns the plaintext exactly once. + """ + tenant = db.scalar(select(Tenant).where(Tenant.id == tenant_id)) + if not tenant: + raise HTTPException(status_code=404, detail="Tenant not found") + + plaintext, hashed = generate_api_key() + tenant.api_key = hashed + db.commit() + + return {"new_api_key": plaintext} diff --git a/app/routers/admin_attention.py b/app/routers/admin_attention.py new file mode 100644 index 0000000000000000000000000000000000000000..e702fad47b8d9d039aefb1bc17d76bbc46ecc87b --- /dev/null +++ b/app/routers/admin_attention.py @@ -0,0 +1,26 @@ +import uuid + +from fastapi import APIRouter, Depends +from sqlalchemy import func, select +from sqlalchemy.orm import Session + +from app.dependencies.admin_auth import require_admin_jwt +from app.models.database import get_db +from app.models.smart_models import KnowledgeBase +from app.schemas.admin import AttentionResponse + +router = APIRouter(prefix="/admin", tags=["admin"]) + + +@router.get("/attention", response_model=AttentionResponse) +def get_admin_attention( + tenant_id: uuid.UUID = Depends(require_admin_jwt), + db: Session = Depends(get_db), +) -> AttentionResponse: + count = db.scalar( + select(func.count()).where( + KnowledgeBase.tenant_id == tenant_id, + KnowledgeBase.is_indexed == False, # noqa: E712 + ) + ) + return AttentionResponse(kb_has_unindexed=(count or 0) > 0) diff --git a/app/routers/admin_auth.py b/app/routers/admin_auth.py new file mode 100644 index 0000000000000000000000000000000000000000..7cfb2b0b184dcbca9eeb2ec0ea4e3e507acc5b06 --- /dev/null +++ b/app/routers/admin_auth.py @@ -0,0 +1,446 @@ +import hashlib +import hmac as hmac_mod +import re +import time +import uuid +from typing import Optional + +from fastapi import APIRouter, Depends, HTTPException, Request, Response +from fastapi.responses import JSONResponse +from pydantic import BaseModel +from sqlalchemy import select +from sqlalchemy.orm import Session + +from app.dependencies.admin_auth import require_admin_jwt +from app.models.database import Tenant, TenantAuthProvider, get_db +from app.services.google_auth import ( + decode_partial_jwt, + issue_full_jwt, + issue_partial_jwt, + lookup_and_bind_tenant, + verify_google_token, +) +from app.services.totp_service import ( + consume_backup_code, + encode_backup_codes, + encrypt_secret, + generate_backup_codes, + generate_totp_secret, + hash_backup_code, + make_provisioning_uri, + verify_totp_code, +) +from app.utils.config import get_config + +config = get_config() +router = APIRouter(prefix="/admin/auth", tags=["admin-auth"]) + +_TOTP_RE = re.compile(r"^\d{6}$") +_BACKUP_RE = re.compile(r"^[0-9a-f]{16}$") + + +class GoogleAuthRequest(BaseModel): + id_token: str + + +class GoogleAuthResponse(BaseModel): + totp_enabled: bool + partial_token: Optional[str] = None + + +class TOTPSetupResponse(BaseModel): + provisioning_uri: str + backup_codes: list[str] # plaintext — shown once, not retrievable again + + +class TOTPVerifyRequest(BaseModel): + code: str + + +class TelegramAuthRequest(BaseModel): + id: int + first_name: str + last_name: Optional[str] = None + username: Optional[str] = None + photo_url: Optional[str] = None + auth_date: int + hash: str + phone_number: Optional[str] = None + + +def _verify_telegram_hmac(data: dict, bot_token: str) -> bool: + """Verify Telegram Login Widget data using HMAC-SHA256.""" + received_hash = data.get("hash", "") + check_pairs = sorted( + (k, str(v)) for k, v in data.items() if k != "hash" and v is not None + ) + data_check_string = "\n".join(f"{k}={v}" for k, v in check_pairs) + secret_key = hashlib.sha256(bot_token.encode()).digest() + expected_hash = hmac_mod.new( + secret_key, data_check_string.encode(), hashlib.sha256 + ).hexdigest() + return hmac_mod.compare_digest(expected_hash, received_hash) + + +def _set_access_cookie(response: Response, tenant_id: uuid.UUID) -> None: + """Issue a full JWT and set it as an HttpOnly cookie on the response.""" + access_token = issue_full_jwt(tenant_id, config.jwt_secret_key) + response.set_cookie( + key="access_token", + value=access_token, + httponly=True, + secure=config.env.name == "production", + samesite="strict", + path="/api", + max_age=86400, + ) + + +@router.post("/telegram") +async def telegram_auth(body: TelegramAuthRequest, db: Session = Depends(get_db)) -> JSONResponse: + # 1. Guard: bot token must be configured + if not config.telegram_bot_token: + raise HTTPException(status_code=401, detail="Unauthorized") + + # 2. Verify HMAC + data = body.model_dump(exclude_none=True) + if not _verify_telegram_hmac(data, config.telegram_bot_token): + raise HTTPException(status_code=401, detail="Unauthorized") + + # 3. Check auth_date freshness (24 hours) + if time.time() - body.auth_date > 86400: + raise HTTPException(status_code=401, detail="Unauthorized") + + telegram_user_id = str(body.id) + phone = body.phone_number + + # 3. Try direct match via auth_providers + ap_row = db.scalars( + select(TenantAuthProvider).where( + TenantAuthProvider.provider == "telegram", + TenantAuthProvider.provider_user_id == telegram_user_id, + ) + ).first() + + if ap_row: + tenant = db.get(Tenant, ap_row.tenant_id) + if not tenant or not tenant.is_active: + raise HTTPException(status_code=401, detail="Unauthorized") + elif phone: + # 4. Try phone match + tenant = db.scalars(select(Tenant).where(Tenant.phone == phone)).first() + if not tenant or not tenant.is_active: + raise HTTPException(status_code=401, detail="Unauthorized") + + # Check for collision: tenant already linked to a different auth method + has_google = tenant.google_id is not None + existing_ap = db.scalars( + select(TenantAuthProvider).where(TenantAuthProvider.tenant_id == tenant.id) + ).first() + if has_google or existing_ap: + existing_provider = existing_ap.provider if existing_ap else "google" + raise HTTPException( + status_code=409, + detail={"collision": True, "existing_provider": existing_provider}, + ) + # First Telegram login: auto-link + ap_row = None # will be created below + else: + raise HTTPException(status_code=401, detail="Unauthorized") + + # 5. Create auth_providers row if this is a first-time phone-matched login + if ap_row is None: + db.add(TenantAuthProvider( + tenant_id=tenant.id, + provider="telegram", + provider_user_id=telegram_user_id, + )) + + # 6. Store phone on tenant if provided and not yet set + if phone and not tenant.phone: + tenant.phone = phone + + db.commit() + + # 7. Issue full JWT via cookie (set cookie on the returned JSONResponse, not the injected Response) + result = JSONResponse(content={"ok": True}) + _set_access_cookie(result, tenant.id) + return result + + +@router.post("/google", response_model=GoogleAuthResponse) +async def google_auth(body: GoogleAuthRequest, response: Response, db: Session = Depends(get_db)) -> GoogleAuthResponse: + try: + google_payload = verify_google_token(body.id_token, config.google_client_id) + except ValueError: + raise HTTPException(status_code=401, detail="Unauthorized") + + email = google_payload["email"] + google_id = google_payload["google_id"] + + # Pre-check for collision: detect Telegram-only account before binding google_id + pre_tenant = db.scalars(select(Tenant).where(Tenant.owner_email == email)).first() + if pre_tenant and pre_tenant.google_id is None: + existing_ap = db.scalars( + select(TenantAuthProvider).where( + TenantAuthProvider.tenant_id == pre_tenant.id, + TenantAuthProvider.provider != "google", + ) + ).first() + if existing_ap: + raise HTTPException( + status_code=409, + detail={"collision": True, "existing_provider": existing_ap.provider}, + ) + + tenant = lookup_and_bind_tenant(email=email, google_id=google_id, db=db) + + # Register Google in auth_providers on first successful login + has_google_ap = db.scalars( + select(TenantAuthProvider).where( + TenantAuthProvider.tenant_id == tenant.id, + TenantAuthProvider.provider == "google", + ) + ).first() + if not has_google_ap: + db.add(TenantAuthProvider( + tenant_id=tenant.id, + provider="google", + provider_user_id=google_id, + )) + db.commit() + + if not tenant.totp_enabled: + _set_access_cookie(response, tenant.id) + return GoogleAuthResponse(totp_enabled=False) + + partial_token = issue_partial_jwt(tenant.id, config.jwt_secret_key) + return GoogleAuthResponse(totp_enabled=True, partial_token=partial_token) + + +@router.post("/link/telegram") +async def link_telegram( + body: TelegramAuthRequest, + tenant_id: uuid.UUID = Depends(require_admin_jwt), + db: Session = Depends(get_db), +) -> JSONResponse: + if not config.telegram_bot_token: + raise HTTPException(status_code=401, detail="Unauthorized") + + data = body.model_dump(exclude_none=True) + if not _verify_telegram_hmac(data, config.telegram_bot_token): + raise HTTPException(status_code=401, detail="Unauthorized") + + if time.time() - body.auth_date > 86400: + raise HTTPException(status_code=401, detail="Unauthorized") + + telegram_user_id = str(body.id) + + # Already linked to this tenant + if db.scalars( + select(TenantAuthProvider).where( + TenantAuthProvider.tenant_id == tenant_id, + TenantAuthProvider.provider == "telegram", + ) + ).first(): + raise HTTPException(status_code=409, detail={"already_linked": True}) + + # Telegram ID already used by another tenant + if db.scalars( + select(TenantAuthProvider).where( + TenantAuthProvider.provider == "telegram", + TenantAuthProvider.provider_user_id == telegram_user_id, + TenantAuthProvider.tenant_id != tenant_id, + ) + ).first(): + raise HTTPException(status_code=409, detail={"already_linked": True, "used_by_other": True}) + + db.add(TenantAuthProvider( + tenant_id=tenant_id, + provider="telegram", + provider_user_id=telegram_user_id, + )) + + tenant = db.get(Tenant, tenant_id) + if body.phone_number and tenant and not tenant.phone: + tenant.phone = body.phone_number + + db.commit() + return JSONResponse(content={"linked": True}) + + +@router.post("/link/google") +async def link_google( + body: GoogleAuthRequest, + tenant_id: uuid.UUID = Depends(require_admin_jwt), + db: Session = Depends(get_db), +) -> JSONResponse: + try: + google_payload = verify_google_token(body.id_token, config.google_client_id) + except ValueError: + raise HTTPException(status_code=401, detail="Unauthorized") + + google_id = google_payload["google_id"] + email = google_payload["email"] + + # Already linked to this tenant + if db.scalars( + select(TenantAuthProvider).where( + TenantAuthProvider.tenant_id == tenant_id, + TenantAuthProvider.provider == "google", + ) + ).first(): + raise HTTPException(status_code=409, detail={"already_linked": True}) + + # Google ID already used by another tenant + if db.scalars( + select(TenantAuthProvider).where( + TenantAuthProvider.provider == "google", + TenantAuthProvider.provider_user_id == google_id, + TenantAuthProvider.tenant_id != tenant_id, + ) + ).first(): + raise HTTPException(status_code=409, detail={"already_linked": True, "used_by_other": True}) + + db.add(TenantAuthProvider( + tenant_id=tenant_id, + provider="google", + provider_user_id=google_id, + )) + + tenant = db.get(Tenant, tenant_id) + if tenant: + if not tenant.owner_email: + tenant.owner_email = email + if not tenant.google_id: + tenant.google_id = google_id + + db.commit() + return JSONResponse(content={"linked": True}) + + +@router.post("/totp/setup", response_model=TOTPSetupResponse) +@router.post("/totp/enroll", response_model=TOTPSetupResponse) +async def totp_setup(request: Request, db: Session = Depends(get_db)) -> TOTPSetupResponse: + auth_header = request.headers.get("Authorization", "") + if not auth_header.startswith("Bearer "): + raise HTTPException(status_code=401, detail="Unauthorized") + + token = auth_header.removeprefix("Bearer ") + + try: + tenant_id_str = decode_partial_jwt(token, config.jwt_secret_key) + tenant_id = uuid.UUID(tenant_id_str) + except (ValueError, AttributeError): + raise HTTPException(status_code=401, detail="Unauthorized") + + tenant = db.get(Tenant, tenant_id) + if not tenant: + raise HTTPException(status_code=401, detail="Unauthorized") + + if tenant.totp_enabled: + raise HTTPException(status_code=403, detail="TOTP already configured") + + raw_secret = generate_totp_secret() + plain_codes = generate_backup_codes() + + tenant.totp_secret = encrypt_secret(raw_secret, config.fernet_key) + tenant.backup_codes = encode_backup_codes([hash_backup_code(c) for c in plain_codes]) + tenant.totp_enabled = False + db.commit() + + return TOTPSetupResponse( + provisioning_uri=make_provisioning_uri(raw_secret, tenant.owner_email or ""), + backup_codes=plain_codes, + ) + + +@router.post("/totp/verify") +async def totp_verify(body: TOTPVerifyRequest, request: Request, db: Session = Depends(get_db)) -> JSONResponse: + auth_header = request.headers.get("Authorization", "") + if not auth_header.startswith("Bearer "): + raise HTTPException(status_code=401, detail="Unauthorized") + + token = auth_header.removeprefix("Bearer ") + + try: + tenant_id_str = decode_partial_jwt(token, config.jwt_secret_key) + tenant_id = uuid.UUID(tenant_id_str) + except (ValueError, AttributeError): + raise HTTPException(status_code=401, detail="Unauthorized") + + tenant = db.get(Tenant, tenant_id) + if not tenant or not tenant.totp_secret: + raise HTTPException(status_code=401, detail="Unauthorized") + + code = body.code.strip().lower() + verified = False + + if _TOTP_RE.match(code): + verified = verify_totp_code(tenant.totp_secret, code, config.fernet_key) + elif _BACKUP_RE.match(code) and tenant.backup_codes: + updated = consume_backup_code(tenant.backup_codes, code) + if updated is not None: + tenant.backup_codes = encode_backup_codes(updated) + verified = True + + if not verified: + raise HTTPException(status_code=401, detail="Unauthorized") + + tenant.totp_enabled = True + db.commit() + + access_token = issue_full_jwt(tenant.id, config.jwt_secret_key) + response = JSONResponse(content={"ok": True}) + # SameSite=Strict provides CSRF protection without a CSRF token + response.set_cookie( + key="access_token", + value=access_token, + httponly=True, + secure=config.env.name == "production", + samesite="strict", + path="/api", + max_age=86400, + ) + return response + + +@router.delete("/totp") +async def totp_disable( + body: TOTPVerifyRequest, + tenant_id: uuid.UUID = Depends(require_admin_jwt), + db: Session = Depends(get_db), +) -> JSONResponse: + tenant = db.get(Tenant, tenant_id) + if not tenant: + raise HTTPException(status_code=401, detail="Unauthorized") + + if not tenant.totp_enabled or not tenant.totp_secret: + raise HTTPException(status_code=400, detail="TOTP not configured") + + code = body.code.strip().lower() + verified = False + + if _TOTP_RE.match(code): + verified = verify_totp_code(tenant.totp_secret, code, config.fernet_key) + elif _BACKUP_RE.match(code) and tenant.backup_codes: + updated = consume_backup_code(tenant.backup_codes, code) + if updated is not None: + tenant.backup_codes = encode_backup_codes(updated) + verified = True + + if not verified: + raise HTTPException(status_code=401, detail="Unauthorized") + + tenant.totp_enabled = False + tenant.totp_secret = None + tenant.backup_codes = None + db.commit() + + return JSONResponse(content={"totp_enabled": False}) + + +@router.post("/logout") +async def logout(response: Response) -> dict: + response.delete_cookie(key="access_token", path="/api") + return {"ok": True} diff --git a/app/routers/admin_me.py b/app/routers/admin_me.py new file mode 100644 index 0000000000000000000000000000000000000000..3f441b15d7c71e0693c601f371d1ed31b43a4b19 --- /dev/null +++ b/app/routers/admin_me.py @@ -0,0 +1,23 @@ +from fastapi import APIRouter, Depends, HTTPException +from sqlalchemy import select +from sqlalchemy.orm import Session +from app.models.database import get_db, Tenant +from app.dependencies.admin_auth import require_admin_jwt +from app.schemas.admin import AdminMeResponse +import uuid + +router = APIRouter(prefix="/admin", tags=["admin"]) + +@router.get("/me", response_model=AdminMeResponse) +def get_admin_me( + tenant_id: uuid.UUID = Depends(require_admin_jwt), + db: Session = Depends(get_db) +): + """ + Returns the authenticated tenant's profile. + Used by the frontend to restore session state. + """ + tenant = db.scalar(select(Tenant).where(Tenant.id == tenant_id)) + if not tenant: + raise HTTPException(status_code=404, detail="Tenant not found") + return tenant diff --git a/app/routers/admin_settings.py b/app/routers/admin_settings.py new file mode 100644 index 0000000000000000000000000000000000000000..2d13588c996966b0654547c22d224d287415df5f --- /dev/null +++ b/app/routers/admin_settings.py @@ -0,0 +1,68 @@ +import uuid +from fastapi import APIRouter, Depends, HTTPException +from sqlalchemy import select +from sqlalchemy.orm import Session +from app.models.database import get_db, Tenant +from app.dependencies.admin_auth import require_admin_jwt +from app.schemas.admin_settings import GoLiveRequest, TenantSettingsUpdate, TenantResponse + +router = APIRouter(prefix="/admin", tags=["admin"]) + + +@router.patch("/go-live", response_model=TenantResponse) +def patch_admin_go_live( + request: GoLiveRequest, + tenant_id: uuid.UUID = Depends(require_admin_jwt), + db: Session = Depends(get_db) +): + tenant = db.scalar(select(Tenant).where(Tenant.id == tenant_id)) + if not tenant: + raise HTTPException(status_code=404, detail="Tenant not found") + + tenant.is_active = request.is_active + db.commit() + db.refresh(tenant) + return tenant + + +@router.put("/settings", response_model=TenantResponse) +def put_admin_settings( + request: TenantSettingsUpdate, + tenant_id: uuid.UUID = Depends(require_admin_jwt), + db: Session = Depends(get_db) +): + tenant = db.scalar(select(Tenant).where(Tenant.id == tenant_id)) + if not tenant: + raise HTTPException(status_code=404, detail="Tenant not found") + + if request.name is not None: + tenant.name = request.name + if request.website_url is not None: + tenant.website_url = str(request.website_url) + + if tenant.customization is None: + tenant.customization = {} + + customization_update = {} + if request.industry is not None: + customization_update["industry"] = request.industry + if request.timezone is not None: + customization_update["timezone"] = request.timezone + if request.chatbot_name is not None: + customization_update["chatbot_name"] = request.chatbot_name + if request.chatbot_greeting is not None: + customization_update["chatbot_greeting"] = request.chatbot_greeting + + if customization_update: + # SQLAlchemy requires a new dict object for JSON column mutation tracking + new_customization = dict(tenant.customization) + new_customization.update(customization_update) + tenant.customization = new_customization + + if "backup_email" in request.model_fields_set: + tenant.backup_email = str(request.backup_email) if request.backup_email else None + + db.commit() + db.refresh(tenant) + return tenant + diff --git a/app/routers/billing.py b/app/routers/billing.py new file mode 100644 index 0000000000000000000000000000000000000000..697a9f36e89e6f9569037a13e7eb2532299a7a2e --- /dev/null +++ b/app/routers/billing.py @@ -0,0 +1,33 @@ +from fastapi import APIRouter, Depends, HTTPException +from sqlalchemy import select +from sqlalchemy.orm import Session +from app.models.database import get_db, Tenant, CreditEvent +from app.dependencies.admin_auth import require_admin_jwt +from app.schemas.billing import BillingSummaryResponse +import uuid + +router = APIRouter(prefix="/billing", tags=["billing"]) + +@router.get("/summary", response_model=BillingSummaryResponse) +def get_billing_summary( + tenant_id: uuid.UUID = Depends(require_admin_jwt), + db: Session = Depends(get_db) +): + """ + Returns current balance and credit event history for the billing page. + """ + tenant = db.scalar(select(Tenant).where(Tenant.id == tenant_id)) + if not tenant: + raise HTTPException(status_code=404, detail="Tenant not found") + + events = db.scalars( + select(CreditEvent) + .where(CreditEvent.tenant_id == tenant_id) + .order_by(CreditEvent.created_at.desc()) + .limit(50) + ).all() + + return BillingSummaryResponse( + balance_usd=tenant.balance_usd, + events=list(events) + ) diff --git a/app/routers/conversations.py b/app/routers/conversations.py new file mode 100644 index 0000000000000000000000000000000000000000..7e6fec975ac3acad9bbc06ae94db217326273588 --- /dev/null +++ b/app/routers/conversations.py @@ -0,0 +1,190 @@ +from typing import Literal +import uuid + +from fastapi import APIRouter, Depends, HTTPException, Request, Query +from sqlalchemy import select +from sqlalchemy.exc import SQLAlchemyError +from sqlalchemy.orm import Session + +from app.dependencies.admin_auth import require_admin_jwt +from app.models.database import Conversation, Message, get_db +from app.schemas.conversations import ( + ConversationSummary, + FeedbackRequest, + FeedbackResponse, + MessageItem, +) + + +router = APIRouter(tags=["conversations"]) + + +@router.get("/conversations", status_code=200, response_model=list[ConversationSummary]) +def list_conversation( + request: Request, + page: int = Query(default=1, ge=1), + page_size: int = Query(default=50, ge=1, le=100), + db: Session = Depends(get_db), + _: uuid.UUID = Depends(require_admin_jwt), +) -> list[ConversationSummary]: + + tenant_id = request.state.tenant_id + + conversations = db.scalars( + select(Conversation) + .where(Conversation.tenant_id == tenant_id) + .order_by(Conversation.last_message_at.desc()) + .limit(page_size) + .offset((page - 1) * page_size) + ).all() + + result: list[ConversationSummary] = [] + for conversation in conversations: + result.append( + ConversationSummary( + id=conversation.id, + started_at=conversation.started_at, + last_message_at=conversation.last_message_at, + message_count=( + conversation.message_count if conversation.message_count else 0 + ), + last_message=conversation.last_message_preview, + ) + ) + + return result + + +@router.get( + "/conversations/{conversation_id}/messages", + status_code=200, + response_model=list[MessageItem], +) +def get_conversation_messages( + request: Request, + conversation_id: uuid.UUID, + page: int = Query(default=1, ge=1), + page_size: int = Query(default=100, ge=1, le=200), + db: Session = Depends(get_db), + _: uuid.UUID = Depends(require_admin_jwt), +) -> list[MessageItem]: + + result: list[MessageItem] = [] + tenant_id = request.state.tenant_id + + conv = db.get(Conversation, conversation_id) + + if conv is None or conv.tenant_id != tenant_id: + raise HTTPException(status_code=404, detail="Conversation doesn't exist") + + messages = db.scalars( + select(Message) + .where( + Message.conversation_id == conversation_id, + ) + .order_by(Message.timestamp.asc()) + .limit(page_size) + .offset((page - 1) * page_size) + ).all() + + for msg in messages: + result.append( + MessageItem( + role="user", + text=msg.user_input, + timestamp=msg.timestamp, + intent=msg.intent, + ) + ) + result.append( + MessageItem( + role="assistant", + text=msg.bot_response, + timestamp=msg.timestamp, + intent=msg.intent, + ) + ) + + return result + + +def _apply_feedback( + message_id: uuid.UUID, + score: Literal[-1, 1], + field: Literal["user_feedback", "tenant_feedback"], + tenant_id: uuid.UUID, + db: Session, +) -> FeedbackResponse: + + message = db.get(Message, message_id) + + if not message: + raise HTTPException(status_code=404, detail="Message not found") + + # Verify the message's conversation belongs to tenant_id + conv = db.get(Conversation, message.conversation_id) + + if not conv or conv.tenant_id != tenant_id: + raise HTTPException(status_code=403, detail="Access denied") + + try: + setattr(message, field, score) + db.commit() + db.refresh(message) + except SQLAlchemyError as e: + db.rollback() + raise HTTPException(status_code=500, detail="Database error") from e + + return FeedbackResponse( + message_id=message.id, score=score + ) # use body.score because it's already validated as Literal[-1, 1]; if message.user_feedback, the system flags it as an error because its return type is `int | None` + + +@router.patch( + "/messages/{message_id}/feedback/user", + status_code=200, + response_model=FeedbackResponse, +) +async def submit_user_feedback( + message_id: uuid.UUID, + body: FeedbackRequest, + request: Request, + db: Session = Depends(get_db), +) -> FeedbackResponse: + + tenant_id = request.state.tenant_id + + feedback_response = _apply_feedback( + message_id=message_id, + score=body.score, + field="user_feedback", + tenant_id=tenant_id, + db=db, + ) + + return feedback_response + + +@router.patch( + "/messages/{message_id}/feedback/tenant", + status_code=200, + response_model=FeedbackResponse, +) +async def submit_tenant_feedback( + message_id: uuid.UUID, + body: FeedbackRequest, + request: Request, + db: Session = Depends(get_db), +) -> FeedbackResponse: + + tenant_id = request.state.tenant_id + + feedback_response = _apply_feedback( + message_id=message_id, + score=body.score, + field="tenant_feedback", + tenant_id=tenant_id, + db=db, + ) + + return feedback_response diff --git a/app/routers/knowledge_base.py b/app/routers/knowledge_base.py index 0b1d28153390c2b31786359f4f06ad44ff11b8a5..f1abefa1c371c5d9bb9dcc995aa4479cb9c8515d 100644 --- a/app/routers/knowledge_base.py +++ b/app/routers/knowledge_base.py @@ -1,31 +1,502 @@ +import urllib.parse import uuid +import csv +import io +import json +import time +from collections import defaultdict +from datetime import datetime, timezone -from fastapi import APIRouter, Depends, HTTPException, Request +from fastapi import APIRouter, BackgroundTasks, Depends, HTTPException, Query, Request, UploadFile, File from sqlalchemy import select from sqlalchemy.orm import Session -from app.models.smart_models import KnowledgeBase -from app.schemas.knowledge_base import KBEntryCreate, KBEntryResponse, KBEntryUpdate -from app.models.database import get_db +from app.dependencies.admin_auth import require_admin_jwt +from app.models.smart_models import KnowledgeBase, SyncJob +from app.schemas.knowledge_base import ( + KBEntryCreate, KBEntryResponse, KBEntryUpdate, + BulkUploadResponse, BulkUploadError, + WPSyncRequest, WPSyncResponse, WPSyncPreviewResponse, + WPSyncPreviewEntry, WPSyncConfirmResponse, WPSyncStatusResponse, + WPSingleSyncRequest, WPSingleSyncResponse, + KBReindexResponse, WPSyncJobControlResponse, WPSyncResumeResponse, +) +from app.models.database import SessionLocal, get_db +from app.services.dependencies import get_embedding_service, get_vector_store +from app.services.embedding_service import EmbeddingService +from app.services.kb_chunking import embed_entry_for_retrieval +from app.services.vector_store import VectorStore +from app.services.wordpress_sync import ( + WordPressSyncError, + WordPressPostTypeError, + fetch_post_type_rest_base, + fetch_single_wordpress_post, + fetch_single_wordpress_post_by_url, + preview_wordpress_entries, + validate_public_https_url, +) +import redis as redis_lib +from app.utils.sync_control import set_cancel_flag, set_pause_flag, clear_pause_flag +from app.workers.kb_reindex import reindex_kb_for_tenant +from app.workers.wordpress_sync import sync_wordpress_site -router = APIRouter() +router = APIRouter(dependencies=[Depends(require_admin_jwt)]) + +_SYNC_RATE_LIMIT: dict[str, list[float]] = defaultdict(list) +_SYNC_RATE_LIMIT_WINDOW_SECONDS = 60 +_SYNC_RATE_LIMIT_MAX_REQUESTS = 10 + +_TITLE_SYNONYMS = frozenset({"title", "headline", "name", "subject", "heading"}) + + +def _extract_csv_fields(row: dict) -> tuple[str, str, str | None]: + """Extract (title, content, source_url) from a CSV row using smart column detection. + + - Title: first column whose lowercased name is in _TITLE_SYNONYMS; value used as title. + - source_url: column whose lowercased name is exactly "source_url". + - Content: all remaining columns' non-empty values joined with newline. + """ + title_col = next((k for k in row if k.lower().strip() in _TITLE_SYNONYMS), None) + title_val = (row[title_col] or "").strip() if title_col else "" + title = title_val if title_val else "Untitled" + + source_url_col = next((k for k in row if k.lower().strip() == "source_url"), None) + source_url_val = (row[source_url_col] or "").strip() if source_url_col else "" + source_url = source_url_val if source_url_val else None + + skip = {title_col, source_url_col} - {None} + parts = [v.strip() for k, v in row.items() if k not in skip and v and v.strip()] + content = "\n".join(parts) + + return title, content, source_url + + +def _embed_entry( + embedding_service: EmbeddingService, + vector_store: VectorStore, + tenant_id: str, + entry_id: str, + text: str, + metadata: dict[str, str | int | float | bool | None] | None = None, +) -> None: + embed_entry_for_retrieval( + embedding_service=embedding_service, + vector_store=vector_store, + tenant_id=tenant_id, + entry_id=entry_id, + text=text, + metadata=metadata, + ) + with SessionLocal() as db: + kb = db.get(KnowledgeBase, uuid.UUID(entry_id)) + if kb is not None: + kb.is_indexed = True + db.commit() + + +def _enforce_sync_rate_limit(tenant_id: uuid.UUID) -> None: + now = time.time() + key = str(tenant_id) + recent = [ + timestamp + for timestamp in _SYNC_RATE_LIMIT[key] + if now - timestamp < _SYNC_RATE_LIMIT_WINDOW_SECONDS + ] + if len(recent) >= _SYNC_RATE_LIMIT_MAX_REQUESTS: + raise HTTPException(status_code=429, detail="WordPress sync rate limit exceeded") + recent.append(now) + _SYNC_RATE_LIMIT[key] = recent + + +def _http_error_from_sync_error(exc: WordPressSyncError) -> HTTPException: + return HTTPException(status_code=400, detail=str(exc)) + + +@router.post("/kb/sync/wordpress/preview", status_code=200, response_model=WPSyncPreviewResponse) +async def preview_wordpress_posts( + request: Request, + payload: WPSyncRequest, +) -> WPSyncPreviewResponse: + tenant_id = request.state.tenant_id + _enforce_sync_rate_limit(tenant_id) + try: + site_url, entries, errors = await preview_wordpress_entries(payload.site_url, post_type=payload.post_type) + except WordPressPostTypeError as exc: + raise HTTPException(status_code=422, detail=str(exc)) from exc + except WordPressSyncError as exc: + raise _http_error_from_sync_error(exc) from exc + + return WPSyncPreviewResponse( + site_url=site_url, + entries=[ + WPSyncPreviewEntry( + title=entry.title, + content=entry.content, + source_url=entry.source_url, + warnings=entry.warnings, + ) + for entry in entries + ], + errors=errors, + ) + + +@router.post("/kb/sync/wordpress/confirm", status_code=202, response_model=WPSyncConfirmResponse) +async def confirm_wordpress_sync( + request: Request, + payload: WPSyncRequest, + db: Session = Depends(get_db), +) -> WPSyncConfirmResponse: + tenant_id = request.state.tenant_id + _enforce_sync_rate_limit(tenant_id) + try: + site_url = validate_public_https_url(payload.site_url) + except WordPressSyncError as exc: + raise _http_error_from_sync_error(exc) from exc + + try: + rest_base = await fetch_post_type_rest_base(site_url, payload.post_type) + except WordPressPostTypeError as exc: + raise HTTPException(status_code=422, detail=str(exc)) from exc + except WordPressSyncError as exc: + raise _http_error_from_sync_error(exc) from exc + + existing_job = db.scalars( + select(SyncJob).where( + SyncJob.tenant_id == tenant_id, + SyncJob.source_url == site_url, + SyncJob.status.in_(["queued", "running"]), + ) + ).first() + if existing_job is not None: + return WPSyncConfirmResponse(job_id=existing_job.id) + + job = SyncJob(tenant_id=tenant_id, source_url=site_url, status="queued") + db.add(job) + db.commit() + db.refresh(job) + try: + sync_wordpress_site.apply_async(args=[str(tenant_id), str(job.id), site_url, rest_base, payload.force]) + except Exception as exc: # pylint: disable=broad-exception-caught + job.status = "failed" + job.error_message = f"Task enqueue failed: {type(exc).__name__}: {exc}" + job.updated_at = datetime.now(timezone.utc) + db.commit() + raise HTTPException(status_code=503, detail="Failed to enqueue WordPress sync job") from exc + return WPSyncConfirmResponse(job_id=job.id) + + +@router.get("/kb/sync/status/{job_id}", status_code=200, response_model=WPSyncStatusResponse) +def read_wordpress_sync_status( + request: Request, + job_id: uuid.UUID, + db: Session = Depends(get_db), +) -> WPSyncStatusResponse: + tenant_id = request.state.tenant_id + job = db.scalars( + select(SyncJob).where(SyncJob.id == job_id, SyncJob.tenant_id == tenant_id) + ).first() + if job is None: + raise HTTPException(status_code=404, detail="Sync job not found") + return WPSyncStatusResponse( + job_id=job.id, + status=job.status, + source_url=job.source_url, + total=job.total, + processed=job.processed, + failed=job.failed, + error_message=job.error_message, + created=job.created, + updated=job.updated, + skipped=job.skipped, + ) + + +_CANCELLABLE_STATUSES = {"queued", "running"} +_TERMINAL_STATUSES = {"completed", "failed", "cancelled", "expired"} + + +@router.post("/kb/sync/wordpress/{job_id}/cancel", status_code=200, response_model=WPSyncJobControlResponse) +def cancel_wordpress_sync( + request: Request, + job_id: uuid.UUID, + db: Session = Depends(get_db), +) -> WPSyncJobControlResponse: + tenant_id = request.state.tenant_id + job = db.scalars( + select(SyncJob).where(SyncJob.id == job_id, SyncJob.tenant_id == tenant_id) + ).first() + if job is None: + raise HTTPException(status_code=404, detail="Sync job not found") + if job.status in _TERMINAL_STATUSES: + raise HTTPException(status_code=409, detail=f"Cannot cancel a job with status '{job.status}'") + try: + set_cancel_flag(str(job_id)) + except redis_lib.RedisError as exc: + raise HTTPException(status_code=503, detail="Job control unavailable — Redis unreachable") from exc + job.status = "cancelled" + job.updated_at = datetime.now(timezone.utc) + db.commit() + return WPSyncJobControlResponse(job_id=job.id, status=job.status) + + +@router.post("/kb/sync/wordpress/{job_id}/pause", status_code=202, response_model=WPSyncJobControlResponse) +def pause_wordpress_sync( + request: Request, + job_id: uuid.UUID, + db: Session = Depends(get_db), +) -> WPSyncJobControlResponse: + tenant_id = request.state.tenant_id + job = db.scalars( + select(SyncJob).where(SyncJob.id == job_id, SyncJob.tenant_id == tenant_id) + ).first() + if job is None: + raise HTTPException(status_code=404, detail="Sync job not found") + if job.status != "running": + raise HTTPException(status_code=409, detail=f"Cannot pause a job with status '{job.status}'") + try: + set_pause_flag(str(job_id)) + except redis_lib.RedisError as exc: + raise HTTPException(status_code=503, detail="Job control unavailable — Redis unreachable") from exc + return WPSyncJobControlResponse(job_id=job.id, status="pausing") + + +@router.post("/kb/sync/wordpress/{job_id}/resume", status_code=202, response_model=WPSyncResumeResponse) +def resume_wordpress_sync( + request: Request, + job_id: uuid.UUID, + db: Session = Depends(get_db), +) -> WPSyncResumeResponse: + tenant_id = request.state.tenant_id + original_job = db.scalars( + select(SyncJob).where(SyncJob.id == job_id, SyncJob.tenant_id == tenant_id) + ).first() + if original_job is None: + raise HTTPException(status_code=404, detail="Sync job not found") + if original_job.status != "paused": + raise HTTPException(status_code=409, detail=f"Cannot resume a job with status '{original_job.status}'") + + # Clear any stale pause flag (non-fatal if Redis is unavailable) + try: + clear_pause_flag(str(job_id)) + except redis_lib.RedisError: + pass # Stale flag will expire on its own; proceed with resume + + new_job = SyncJob( + tenant_id=tenant_id, + source_url=original_job.source_url, + rest_base=original_job.rest_base, + status="queued", + ) + db.add(new_job) + db.flush() + try: + sync_wordpress_site.apply_async( + args=[str(tenant_id), str(new_job.id), original_job.source_url, original_job.rest_base, False] + ) + except Exception as exc: # pylint: disable=broad-exception-caught + new_job.status = "failed" + db.commit() + raise HTTPException(status_code=503, detail="Failed to enqueue resume job") from exc + db.commit() + return WPSyncResumeResponse(original_job_id=original_job.id, new_job_id=new_job.id) + + +@router.post("/kb/sync/wordpress/single", status_code=200, response_model=WPSingleSyncResponse) +async def sync_single_wordpress_post( + request: Request, + payload: WPSingleSyncRequest, + db: Session = Depends(get_db), + embedding_service: EmbeddingService = Depends(get_embedding_service), + vector_store: VectorStore = Depends(get_vector_store), +) -> WPSingleSyncResponse: + tenant_id = request.state.tenant_id + _enforce_sync_rate_limit(tenant_id) + + try: + site_url = validate_public_https_url(payload.site_url) + validate_public_https_url(payload.source_url) + except WordPressSyncError as exc: + raise _http_error_from_sync_error(exc) from exc + + if urllib.parse.urlparse(payload.source_url).netloc != urllib.parse.urlparse(site_url).netloc: + raise HTTPException(status_code=400, detail="source_url must be on the same domain as site_url") + + try: + rest_base = await fetch_post_type_rest_base(site_url, payload.post_type) + except WordPressPostTypeError as exc: + raise HTTPException(status_code=422, detail=str(exc)) from exc + except WordPressSyncError as exc: + raise _http_error_from_sync_error(exc) from exc + + try: + wp_entry = await fetch_single_wordpress_post(site_url, payload.source_url, rest_base) + except WordPressSyncError as exc: + raise _http_error_from_sync_error(exc) from exc + + kb = db.scalars( + select(KnowledgeBase).where( + KnowledgeBase.tenant_id == tenant_id, + KnowledgeBase.source_url == wp_entry.source_url, + ) + ).first() + + if kb is None: + kb = KnowledgeBase( + tenant_id=tenant_id, + title=wp_entry.title, + content=wp_entry.content, + source_url=wp_entry.source_url, + origin="synced", + ) + db.add(kb) + db.commit() + db.refresh(kb) + _embed_entry( + embedding_service, + vector_store, + str(tenant_id), + str(kb.id), + kb.content, + metadata={"origin": kb.origin, "source_url": kb.source_url}, + ) + action = "created" + elif payload.force: + kb.title = wp_entry.title + kb.content = wp_entry.content + kb.is_indexed = False + kb.updated_at = datetime.now(timezone.utc) + db.commit() + db.refresh(kb) + _embed_entry( + embedding_service, + vector_store, + str(tenant_id), + str(kb.id), + kb.content, + metadata={"origin": kb.origin, "source_url": kb.source_url}, + ) + action = "updated" + else: + action = "skipped" + + return WPSingleSyncResponse(action=action, entry=KBEntryResponse.model_validate(kb)) + + +@router.post("/kb/bulk-upload", status_code=201, response_model=BulkUploadResponse) +def bulk_upload_entries( + request: Request, + background_tasks: BackgroundTasks, + file: UploadFile = File(...), + db: Session = Depends(get_db), + embedding_service: EmbeddingService = Depends(get_embedding_service), + vector_store: VectorStore = Depends(get_vector_store), +) -> BulkUploadResponse: + tenant_id = request.state.tenant_id + created_count = 0 + errors = [] + + parsed_data = [] + filename = file.filename or "" + + content_type = file.content_type or "" + is_csv = filename.endswith(".csv") or content_type == "text/csv" + is_json = filename.endswith(".json") or content_type == "application/json" + + if not is_csv and not is_json: + raise HTTPException(status_code=400, detail="Unsupported file format. Use CSV or JSON.") + + try: + content = file.file.read() + if is_csv: + parsed_data = list(csv.DictReader(io.StringIO(content.decode("utf-8")))) + else: + parsed_data = json.loads(content.decode("utf-8")) + if not isinstance(parsed_data, list): + raise ValueError("JSON must be an array of objects") + except (UnicodeDecodeError, ValueError) as e: + raise HTTPException(status_code=400, detail=f"Failed to parse file: {str(e)}") + + valid_entries = [] + + for idx, row in enumerate(parsed_data): + row_num = idx + 1 + if not isinstance(row, dict): + errors.append(BulkUploadError(row=row_num, reason="Row is not an object/dictionary")) + continue + + if is_csv: + row_title, row_content, row_source_url = _extract_csv_fields(row) + else: + row_content = row.get("content") + row_title = row.get("title") or "Untitled" + row_source_url = row.get("source_url") or None + + if not row_content: + errors.append(BulkUploadError(row=row_num, reason="Missing 'content' field")) + continue + + kb = KnowledgeBase( + tenant_id=tenant_id, + content=row_content, + title=row_title, + source_url=row_source_url, + origin="manual", + ) + valid_entries.append(kb) + + if valid_entries: + db.add_all(valid_entries) + db.commit() + for kb in valid_entries: + db.refresh(kb) + background_tasks.add_task( + _embed_entry, + embedding_service, + vector_store, + str(kb.tenant_id), + str(kb.id), + kb.content, + {"origin": kb.origin, "source_url": kb.source_url}, + ) + created_count += 1 + + return BulkUploadResponse(created=created_count, errors=errors) @router.post("/kb/entries", status_code=201, response_model=KBEntryResponse) def create_entry( - request: Request, entry: KBEntryCreate, db: Session = Depends(get_db) + request: Request, + entry: KBEntryCreate, + background_tasks: BackgroundTasks, + db: Session = Depends(get_db), + embedding_service: EmbeddingService = Depends(get_embedding_service), + vector_store: VectorStore = Depends(get_vector_store), ) -> KBEntryResponse: knowledge_base = KnowledgeBase() knowledge_base.tenant_id = ( request.state.tenant_id ) # get the tenant_id from the middleware - knowledge_base.entry = entry.entry # type: ignore[truthy-bool] + knowledge_base.title = entry.title + knowledge_base.content = entry.content + knowledge_base.source_url = entry.source_url + knowledge_base.origin = "manual" db.add(knowledge_base) db.commit() db.refresh(knowledge_base) - return knowledge_base + background_tasks.add_task( + _embed_entry, + embedding_service, + vector_store, + str(knowledge_base.tenant_id), + str(knowledge_base.id), + knowledge_base.content, + {"origin": knowledge_base.origin, "source_url": knowledge_base.source_url}, + ) + + return KBEntryResponse.model_validate(knowledge_base) @router.get("/kb/entries/{entry_id}", status_code=200, response_model=KBEntryResponse) @@ -43,17 +514,23 @@ def read_entry( if kb is None: raise HTTPException(status_code=404, detail="Entry not found") - return kb + return KBEntryResponse.model_validate(kb) @router.get("/kb/entries", status_code=200, response_model=list[KBEntryResponse]) def read_entries( - request: Request, db: Session = Depends(get_db) + request: Request, + page: int = Query(default=1, ge=1), + page_size: int = Query(default=50, ge=1, le=100), + db: Session = Depends(get_db), ) -> list[KBEntryResponse]: tenant_id = request.state.tenant_id kb_entries = db.scalars( - select(KnowledgeBase).where(KnowledgeBase.tenant_id == tenant_id) + select(KnowledgeBase) + .where(KnowledgeBase.tenant_id == tenant_id) + .limit(page_size) + .offset((page - 1) * page_size) ).all() return kb_entries # type: ignore[truthy-bool] @@ -64,7 +541,10 @@ def update_entry( request: Request, entry_id: uuid.UUID, entry: KBEntryUpdate, + background_tasks: BackgroundTasks, db: Session = Depends(get_db), + embedding_service: EmbeddingService = Depends(get_embedding_service), + vector_store: VectorStore = Depends(get_vector_store), ) -> KBEntryResponse: tenant_id = request.state.tenant_id @@ -76,16 +556,43 @@ def update_entry( if kb is None: raise HTTPException(status_code=404, detail="Entry not found") - kb.entry = entry.entry # type: ignore[truthy-bool] + + if kb.origin == "synced": + raise HTTPException( + status_code=409, + detail="Synced entries are read-only through the normal KB update endpoint", + ) + + if entry.title is not None: + kb.title = entry.title + if entry.content is not None: + kb.content = entry.content + if "source_url" in entry.model_fields_set: + kb.source_url = entry.source_url + kb.is_indexed = False + db.commit() db.refresh(kb) - return kb + background_tasks.add_task( + _embed_entry, + embedding_service, + vector_store, + str(kb.tenant_id), + str(kb.id), + kb.content, + {"origin": kb.origin, "source_url": kb.source_url}, + ) + + return KBEntryResponse.model_validate(kb) @router.delete("/kb/entries/{entry_id}", status_code=204) def delete_entry( - request: Request, entry_id: uuid.UUID, db: Session = Depends(get_db) + request: Request, + entry_id: uuid.UUID, + db: Session = Depends(get_db), + vector_store: VectorStore = Depends(get_vector_store), ) -> None: tenant_id = request.state.tenant_id @@ -100,3 +607,67 @@ def delete_entry( db.delete(kb) db.commit() + + vector_store.delete_docs(tenant_id=str(tenant_id), entry_ids=[str(entry_id)]) + + +@router.post("/kb/entries/{entry_id}/resync", status_code=200, response_model=KBEntryResponse) +async def resync_entry( + request: Request, + entry_id: uuid.UUID, + background_tasks: BackgroundTasks, + db: Session = Depends(get_db), + embedding_service: EmbeddingService = Depends(get_embedding_service), + vector_store: VectorStore = Depends(get_vector_store), +) -> KBEntryResponse: + tenant_id = request.state.tenant_id + + kb = db.scalars( + select(KnowledgeBase).where( + KnowledgeBase.id == entry_id, KnowledgeBase.tenant_id == tenant_id + ) + ).first() + if kb is None: + raise HTTPException(status_code=404, detail="Entry not found") + + if kb.origin != "synced": + raise HTTPException(status_code=400, detail="Only synced entries can be resynced") + + if kb.source_url is None: + raise HTTPException(status_code=400, detail="Entry has no source URL to resync from") + + try: + validate_public_https_url(kb.source_url) + except WordPressSyncError as exc: + raise HTTPException(status_code=400, detail=str(exc)) from exc + + try: + wp_entry = await fetch_single_wordpress_post_by_url(kb.source_url) + except WordPressSyncError as exc: + raise HTTPException(status_code=400, detail=str(exc)) from exc + + kb.title = wp_entry.title + kb.content = wp_entry.content + kb.is_indexed = False + kb.updated_at = datetime.now(timezone.utc) + db.commit() + db.refresh(kb) + + background_tasks.add_task( + _embed_entry, + embedding_service, + vector_store, + str(kb.tenant_id), + str(kb.id), + kb.content, + {"origin": kb.origin, "source_url": kb.source_url}, + ) + + return KBEntryResponse.model_validate(kb) + + +@router.post("/kb/reindex", status_code=202, response_model=KBReindexResponse) +def trigger_reindex(request: Request) -> KBReindexResponse: + tenant_id = request.state.tenant_id + task = reindex_kb_for_tenant.delay(str(tenant_id)) + return KBReindexResponse(task_id=task.id) diff --git a/app/routers/super_admin_auth.py b/app/routers/super_admin_auth.py new file mode 100644 index 0000000000000000000000000000000000000000..b44a27d7457936557bf7ba4ab1e573e86339e404 --- /dev/null +++ b/app/routers/super_admin_auth.py @@ -0,0 +1,58 @@ +from fastapi import APIRouter, Depends, HTTPException, Response +from fastapi.responses import JSONResponse +from pydantic import BaseModel + +from app.dependencies.super_admin_auth import require_super_admin_jwt +from app.services.google_auth import ( + SUPER_ADMIN_TOKEN_EXPIRE_HOURS, + issue_super_admin_jwt, + verify_google_token, +) +from app.utils.config import get_config + +config = get_config() +router = APIRouter(prefix="/super-admin", tags=["super-admin-auth"]) + +_TOKEN_MAX_AGE = SUPER_ADMIN_TOKEN_EXPIRE_HOURS * 3600 + + +class SuperAdminGoogleAuthRequest(BaseModel): + id_token: str + + +@router.post("/auth/google") +async def super_admin_google_auth(body: SuperAdminGoogleAuthRequest) -> JSONResponse: + try: + google_payload = verify_google_token(body.id_token, config.google_client_id) + except ValueError: + raise HTTPException(status_code=401, detail="Unauthorized") + + email = google_payload.get("email") + if not email: + raise HTTPException(status_code=401, detail="Unauthorized") + if email.lower() not in [e.lower() for e in config.super_admin_emails]: + raise HTTPException(status_code=403, detail="Forbidden") + + token = issue_super_admin_jwt(email, config.jwt_secret_key) + response = JSONResponse(content={"ok": True}) + response.set_cookie( + key="super_admin_token", + value=token, + httponly=True, + secure=config.env.name == "production", + samesite="lax", + path="/api", + max_age=_TOKEN_MAX_AGE, + ) + return response + + +@router.post("/auth/logout") +async def super_admin_logout(response: Response) -> dict: + response.delete_cookie(key="super_admin_token", path="/api") + return {"ok": True} + + +@router.get("/me") +async def super_admin_me(email: str = Depends(require_super_admin_jwt)) -> dict: + return {"email": email} diff --git a/app/routers/super_admin_rates.py b/app/routers/super_admin_rates.py new file mode 100644 index 0000000000000000000000000000000000000000..20ac4ebed28603b132caf3a6c7f1b0d46c94a534 --- /dev/null +++ b/app/routers/super_admin_rates.py @@ -0,0 +1,116 @@ +# app/routers/super_admin_rates.py +"""Super-admin endpoints for managing LLM model rates. + +GET /super-admin/llm-rates — returns active rates and full history +POST /super-admin/llm-rates — adds a new rate, closes the previous active rate +""" +from datetime import date +from decimal import Decimal +from typing import Optional + +from fastapi import APIRouter, Depends, HTTPException +from pydantic import BaseModel +from sqlalchemy import select +from sqlalchemy.orm import Session + +from app.dependencies.super_admin_auth import require_super_admin_jwt +from app.models.database import LLMModel, get_db + +router = APIRouter(prefix="/super-admin", tags=["super-admin-rates"]) + + +# --------------------------------------------------------------------------- +# Pydantic schemas +# --------------------------------------------------------------------------- + + +class LLMRateOut(BaseModel): + """Serialises one row of llm_models using only columns that exist in the DB.""" + + id: int + provider: str + model_name: str + usd_per_1k_input: Decimal + usd_per_1k_output: Decimal + effective_from: date + effective_to: Optional[date] + + model_config = {"from_attributes": True} + + +class LLMRatesResponse(BaseModel): + active: list[LLMRateOut] + history: list[LLMRateOut] + + +class LLMRateCreate(BaseModel): + provider: str + model_name: str + usd_per_1k_input: Decimal + usd_per_1k_output: Decimal + + +# --------------------------------------------------------------------------- +# GET /llm-rates +# --------------------------------------------------------------------------- + + +@router.get("/llm-rates", response_model=LLMRatesResponse) +def list_llm_rates( + _email: str = Depends(require_super_admin_jwt), + db: Session = Depends(get_db), +) -> LLMRatesResponse: + """Return all active rates and the full rate history.""" + all_rates = db.execute( + select(LLMModel).order_by(LLMModel.effective_from.desc()) + ).scalars().all() + + active = [r for r in all_rates if r.effective_to is None] + history = list(all_rates) # already ordered by effective_from desc + + return LLMRatesResponse( + active=[LLMRateOut.model_validate(r) for r in active], + history=[LLMRateOut.model_validate(r) for r in history], + ) + + +# --------------------------------------------------------------------------- +# POST /llm-rates +# --------------------------------------------------------------------------- + + +@router.post("/llm-rates", response_model=LLMRateOut, status_code=201) +def create_llm_rate( + body: LLMRateCreate, + _email: str = Depends(require_super_admin_jwt), + db: Session = Depends(get_db), +) -> LLMRateOut: + """Add a new rate for a model, closing the current active rate if one exists.""" + today = date.today() + + # Close the current active rate for this provider + model_name, if any + existing_active = db.execute( + select(LLMModel).where( + LLMModel.provider == body.provider, + LLMModel.model_name == body.model_name, + LLMModel.effective_to.is_(None), + ) + ).scalars().first() + + if existing_active is not None: + existing_active.effective_to = today + + # Insert the new rate + new_rate = LLMModel( + provider=body.provider, + model_name=body.model_name, + usd_per_1k_input=body.usd_per_1k_input, + usd_per_1k_output=body.usd_per_1k_output, + effective_from=today, + effective_to=None, + ) + db.add(new_rate) + db.commit() + db.refresh(new_rate) + + return LLMRateOut.model_validate(new_rate) diff --git a/app/routers/super_admin_tenants.py b/app/routers/super_admin_tenants.py new file mode 100644 index 0000000000000000000000000000000000000000..b03f4d08e02d86f9ccfdd583a82b6bdd7ce3f258 --- /dev/null +++ b/app/routers/super_admin_tenants.py @@ -0,0 +1,292 @@ +"""Super-admin tenant management endpoints. + +Provides list, block/unblock, credit management, and usage overview +for all tenants. All endpoints require a valid super-admin JWT cookie. +""" +from datetime import datetime, timezone, timedelta +from decimal import Decimal +from typing import Optional +import uuid + +from fastapi import APIRouter, Depends, HTTPException +from pydantic import BaseModel, field_validator +from sqlalchemy import func, select +from sqlalchemy.orm import Session + +from app.dependencies.super_admin_auth import require_super_admin_jwt +from app.models.database import ( + Conversation, + CreditEvent, + Tenant, + get_db, +) + +router = APIRouter(prefix="/super-admin", tags=["super-admin-tenants"]) + + +# --------------------------------------------------------------------------- +# Request / Response schemas +# --------------------------------------------------------------------------- + + +class TenantSummary(BaseModel): + id: uuid.UUID + name: str + owner_email: Optional[str] + is_active: Optional[bool] + is_blocked: Optional[bool] + billing_mode: str + balance_usd: Decimal + created_at: Optional[datetime] + last_conversation_at: Optional[datetime] + + model_config = {"from_attributes": True} + + +class BlockToggleRequest(BaseModel): + is_blocked: bool + + +class AddCreditsRequest(BaseModel): + amount: Decimal + note: str + + @field_validator("amount") + @classmethod + def amount_must_be_positive(cls, v: Decimal) -> Decimal: + if v <= 0: + raise ValueError("amount must be greater than 0") + return v + + +class AddCreditsResponse(BaseModel): + new_balance: Decimal + + +class CreditEventResponse(BaseModel): + id: uuid.UUID + event_type: str + amount: Decimal + balance_after: Decimal + note: Optional[str] + created_by: Optional[str] + created_at: datetime + + model_config = {"from_attributes": True} + + +class UsageResponse(BaseModel): + total_conversations: int + total_messages: int + usd_used_30d: Decimal + last_active_at: Optional[datetime] + + +# --------------------------------------------------------------------------- +# Helpers +# --------------------------------------------------------------------------- + + +def _get_tenant_or_404(tenant_id: uuid.UUID, db: Session) -> Tenant: + tenant = db.get(Tenant, tenant_id) + if tenant is None: + raise HTTPException(status_code=404, detail="Tenant not found") + return tenant + + +# --------------------------------------------------------------------------- +# GET /super-admin/tenants +# --------------------------------------------------------------------------- + + +@router.get("/tenants", response_model=list[TenantSummary]) +def list_tenants( + _email: str = Depends(require_super_admin_jwt), + db: Session = Depends(get_db), +) -> list[TenantSummary]: + """List all tenants with their last conversation timestamp.""" + # Subquery: latest conversation created_at per tenant + last_conv_subq = ( + select( + Conversation.tenant_id, + func.max(Conversation.started_at).label("last_conversation_at"), + ) + .group_by(Conversation.tenant_id) + .subquery() + ) + + rows = db.execute( + select(Tenant, last_conv_subq.c.last_conversation_at) + .outerjoin(last_conv_subq, Tenant.id == last_conv_subq.c.tenant_id) + .order_by(Tenant.created_at.desc()) + ).all() + + result: list[TenantSummary] = [] + for tenant, last_conv_at in rows: + result.append( + TenantSummary( + id=tenant.id, + name=tenant.name, + owner_email=tenant.owner_email, + is_active=tenant.is_active, + is_blocked=tenant.is_blocked, + billing_mode=tenant.billing_mode, + balance_usd=tenant.balance_usd, + created_at=tenant.created_at, + last_conversation_at=last_conv_at, + ) + ) + return result + + +# --------------------------------------------------------------------------- +# PATCH /super-admin/tenants/{tenant_id}/block +# --------------------------------------------------------------------------- + + +@router.patch("/tenants/{tenant_id}/block", response_model=TenantSummary) +def toggle_tenant_block( + tenant_id: uuid.UUID, + body: BlockToggleRequest, + _email: str = Depends(require_super_admin_jwt), + db: Session = Depends(get_db), +) -> TenantSummary: + """Toggle the is_blocked flag for a tenant.""" + tenant = _get_tenant_or_404(tenant_id, db) + tenant.is_blocked = body.is_blocked + db.commit() + db.refresh(tenant) + return TenantSummary( + id=tenant.id, + name=tenant.name, + owner_email=tenant.owner_email, + is_active=tenant.is_active, + is_blocked=tenant.is_blocked, + billing_mode=tenant.billing_mode, + balance_usd=tenant.balance_usd, + created_at=tenant.created_at, + last_conversation_at=None, + ) + + +# --------------------------------------------------------------------------- +# POST /super-admin/tenants/{tenant_id}/credits +# --------------------------------------------------------------------------- + + +@router.post("/tenants/{tenant_id}/credits", response_model=AddCreditsResponse) +def add_credits( + tenant_id: uuid.UUID, + body: AddCreditsRequest, + _email: str = Depends(require_super_admin_jwt), + db: Session = Depends(get_db), +) -> AddCreditsResponse: + """Add credits to a tenant's balance and record a credit event.""" + tenant = _get_tenant_or_404(tenant_id, db) + + tenant.balance_usd = tenant.balance_usd + body.amount + + event = CreditEvent( + tenant_id=tenant.id, + event_type="topup", + amount=body.amount, + balance_after=tenant.balance_usd, + ) + event.note = body.note + event.created_by = "super_admin" + + db.add(event) + db.commit() + db.refresh(tenant) + + return AddCreditsResponse(new_balance=tenant.balance_usd) + + +# --------------------------------------------------------------------------- +# GET /super-admin/tenants/{tenant_id}/credits +# --------------------------------------------------------------------------- + + +@router.get("/tenants/{tenant_id}/credits", response_model=list[CreditEventResponse]) +def get_credit_history( + tenant_id: uuid.UUID, + _email: str = Depends(require_super_admin_jwt), + db: Session = Depends(get_db), +) -> list[CreditEventResponse]: + """Return all credit events for a tenant, descending by created_at.""" + tenant = _get_tenant_or_404(tenant_id, db) # noqa: F841 + + events = db.execute( + select(CreditEvent) + .where(CreditEvent.tenant_id == tenant_id) + .order_by(CreditEvent.created_at.desc()) + ).scalars().all() + + return [ + CreditEventResponse( + id=e.id, + event_type=e.event_type, + amount=e.amount, + balance_after=e.balance_after, + note=e.note, + created_by=e.created_by, + created_at=e.created_at, + ) + for e in events + ] + + +# --------------------------------------------------------------------------- +# GET /super-admin/tenants/{tenant_id}/usage +# --------------------------------------------------------------------------- + + +@router.get("/tenants/{tenant_id}/usage", response_model=UsageResponse) +def get_tenant_usage( + tenant_id: uuid.UUID, + _email: str = Depends(require_super_admin_jwt), + db: Session = Depends(get_db), +) -> UsageResponse: + """Return usage overview for a tenant.""" + tenant = _get_tenant_or_404(tenant_id, db) # noqa: F841 + + thirty_days_ago = datetime.now(timezone.utc) - timedelta(days=30) + + # Total conversations: count of Conversation rows for this tenant + total_conversations: int = db.execute( + select(func.count(Conversation.id)).where( + Conversation.tenant_id == tenant_id + ) + ).scalar() or 0 + + # Total messages: sum of message_count across all conversations + total_messages_result = db.execute( + select(func.sum(Conversation.message_count)).where( + Conversation.tenant_id == tenant_id + ) + ).scalar() + total_messages: int = int(total_messages_result) if total_messages_result else 0 + + # USD used in last 30 days: sum of deduction events + usd_used_result = db.execute( + select(func.sum(CreditEvent.amount)).where( + CreditEvent.tenant_id == tenant_id, + CreditEvent.event_type == "deduct", + CreditEvent.created_at >= thirty_days_ago, + ) + ).scalar() + usd_used_30d: Decimal = Decimal(str(usd_used_result)) if usd_used_result else Decimal("0") + + # Last active at: max started_at from Conversation + last_active_at: Optional[datetime] = db.execute( + select(func.max(Conversation.started_at)).where( + Conversation.tenant_id == tenant_id + ) + ).scalar() + + return UsageResponse( + total_conversations=total_conversations, + total_messages=total_messages, + usd_used_30d=usd_used_30d, + last_active_at=last_active_at, + ) diff --git a/app/routers/system_prompt.py b/app/routers/system_prompt.py new file mode 100644 index 0000000000000000000000000000000000000000..289f6f10bd1b9b4e51359b8f218f8f0e151cb64c --- /dev/null +++ b/app/routers/system_prompt.py @@ -0,0 +1,51 @@ +from typing import Optional + +from fastapi import APIRouter, Depends, HTTPException, Request +from sqlalchemy.orm import Session + +from app.dependencies.admin_auth import require_admin_jwt +from app.models.database import Tenant, get_db +from app.schemas.system_prompt import SystemPromptResponse, SystemPromptUpdate + +router = APIRouter(dependencies=[Depends(require_admin_jwt)]) + + +def _get_tenant(request: Request, db: Session) -> Tenant: + tenant_id = request.state.tenant_id + tenant = db.get(Tenant, tenant_id) + if tenant is None: + raise HTTPException(status_code=404, detail="Tenant not found") + + return tenant + + +@router.get("/system-prompt", status_code=200, response_model=SystemPromptResponse) +def get_system_prompt( + request: Request, db: Session = Depends(get_db) +) -> SystemPromptResponse: + tenant = _get_tenant(request, db) + customization = tenant.customization or {} + return SystemPromptResponse(system_prompt=customization.get("system_prompt")) + + +@router.put("/system-prompt", status_code=200, response_model=SystemPromptResponse) +def update_system_prompt( + request: Request, body: SystemPromptUpdate, db: Session = Depends(get_db) +) -> SystemPromptResponse: + tenant = _get_tenant(request, db) + tenant.customization = { + **(tenant.customization or {}), + "system_prompt": body.system_prompt, + } + db.commit() + db.refresh(tenant) + return SystemPromptResponse(system_prompt=(tenant.customization or {}).get("system_prompt")) + + +@router.delete("/system-prompt", status_code=204) +def delete_system_prompt(request: Request, db: Session = Depends(get_db)) -> None: + tenant = _get_tenant(request, db) + customization = dict(tenant.customization or {}) + customization.pop("system_prompt", None) + tenant.customization = customization + db.commit() diff --git a/app/routers/widget_settings.py b/app/routers/widget_settings.py new file mode 100644 index 0000000000000000000000000000000000000000..cb7973a90450d6b5495ce76d8598d93598fcdbbc --- /dev/null +++ b/app/routers/widget_settings.py @@ -0,0 +1,23 @@ +from fastapi import APIRouter, Depends, HTTPException, Request +from sqlalchemy.orm import Session + +from app.models.database import Tenant, get_db +from app.schemas.widget_settings import WidgetSettingsResponse + +router = APIRouter(prefix="/widget", tags=["widget"]) + + +@router.get("/settings", response_model=WidgetSettingsResponse) +def get_widget_settings( + request: Request, db: Session = Depends(get_db) +) -> WidgetSettingsResponse: + tenant_id = request.state.tenant_id + tenant = db.get(Tenant, tenant_id) + if tenant is None: + raise HTTPException(status_code=404, detail="Tenant not found") + + customization = tenant.customization or {} + return WidgetSettingsResponse( + chatbot_name=customization.get("chatbot_name") or tenant.name, + chatbot_greeting=customization.get("chatbot_greeting"), + ) diff --git a/app/schemas/admin.py b/app/schemas/admin.py new file mode 100644 index 0000000000000000000000000000000000000000..58ffa9f28c3e8016a4a2e386144346159f23aa6a --- /dev/null +++ b/app/schemas/admin.py @@ -0,0 +1,21 @@ +import uuid +from decimal import Decimal +from typing import Optional, Dict, Any +from pydantic import BaseModel, ConfigDict + +class AdminMeResponse(BaseModel): + id: uuid.UUID + name: str + owner_email: Optional[str] = None + backup_email: Optional[str] = None + website_url: Optional[str] = None + is_active: bool + billing_mode: str + balance_usd: Decimal + customization: Optional[Dict[str, Any]] = None + + model_config = ConfigDict(from_attributes=True) + + +class AttentionResponse(BaseModel): + kb_has_unindexed: bool diff --git a/app/schemas/admin_apikey.py b/app/schemas/admin_apikey.py new file mode 100644 index 0000000000000000000000000000000000000000..3e223d9efa7117d219fc595371638a1c997735b7 --- /dev/null +++ b/app/schemas/admin_apikey.py @@ -0,0 +1,4 @@ +from pydantic import BaseModel + +class ApiKeyRotateResponse(BaseModel): + new_api_key: str diff --git a/app/schemas/admin_settings.py b/app/schemas/admin_settings.py new file mode 100644 index 0000000000000000000000000000000000000000..153179c2b9310e5df52da71baf2d06721f8f8706 --- /dev/null +++ b/app/schemas/admin_settings.py @@ -0,0 +1,72 @@ +from pydantic import BaseModel, ConfigDict, AnyHttpUrl, EmailStr, field_validator +import uuid +from decimal import Decimal +from typing import Optional, Dict, Any +import zoneinfo + +ALLOWED_INDUSTRIES = { + # Current admin console dropdown values (lowercased) + "restaurant / food & beverage", + "retail", + "e-commerce", + "healthcare", + "legal", + "education", + "real estate", + "financial services", + "technology", + "travel & hospitality", + "beauty & wellness", + "professional services", + "non-profit", + "other", + # Legacy values kept for backward compatibility with existing DB records + "real-estate", + "saas", + "finance", +} + + +class GoLiveRequest(BaseModel): + is_active: bool + + +class TenantSettingsUpdate(BaseModel): + name: Optional[str] = None + website_url: Optional[AnyHttpUrl] = None + industry: Optional[str] = None + timezone: Optional[str] = None + chatbot_name: Optional[str] = None + chatbot_greeting: Optional[str] = None + backup_email: Optional[EmailStr] = None + + @field_validator("industry") + @classmethod + def validate_industry(cls, v): + if v is not None and v.lower() not in ALLOWED_INDUSTRIES: + raise ValueError(f"Industry must be one of {ALLOWED_INDUSTRIES}") + return v.lower() if v else v + + @field_validator("timezone") + @classmethod + def validate_timezone(cls, v): + if v is not None: + try: + zoneinfo.ZoneInfo(v) + except zoneinfo.ZoneInfoNotFoundError: + raise ValueError("Invalid timezone string") + return v + + +class TenantResponse(BaseModel): + id: uuid.UUID + name: str + website_url: Optional[str] = None + owner_email: Optional[str] = None + backup_email: Optional[str] = None + is_active: bool + billing_mode: str + balance_usd: Decimal + customization: Optional[Dict[str, Any]] = None + + model_config = ConfigDict(from_attributes=True) diff --git a/app/schemas/billing.py b/app/schemas/billing.py new file mode 100644 index 0000000000000000000000000000000000000000..1a2a610bc4a0d188126cb067d876ef61c4884d95 --- /dev/null +++ b/app/schemas/billing.py @@ -0,0 +1,16 @@ +from pydantic import BaseModel, ConfigDict +from decimal import Decimal +from datetime import datetime +from typing import List + +class CreditEventModel(BaseModel): + event_type: str + amount: Decimal + balance_after: Decimal + created_at: datetime + + model_config = ConfigDict(from_attributes=True) + +class BillingSummaryResponse(BaseModel): + balance_usd: Decimal + events: List[CreditEventModel] diff --git a/app/schemas/conversations.py b/app/schemas/conversations.py new file mode 100644 index 0000000000000000000000000000000000000000..60971c99babf4ba57fffe1d5b83cf0479653c414 --- /dev/null +++ b/app/schemas/conversations.py @@ -0,0 +1,34 @@ +from datetime import datetime +from typing import Literal +import uuid + +from pydantic import BaseModel, ConfigDict + + +class ConversationSummary(BaseModel): + model_config = ConfigDict(from_attributes=True) + + id: uuid.UUID + started_at: datetime | None + last_message_at: datetime | None + message_count: int + last_message: str | None # last user-input in this conversation + + +class MessageItem(BaseModel): + + role: str + text: str + timestamp: datetime | None + intent: str | None + + +class FeedbackRequest(BaseModel): + + score: Literal[-1, 1] + + +class FeedbackResponse(BaseModel): + + message_id: uuid.UUID + score: Literal[-1, 1] diff --git a/app/schemas/knowledge_base.py b/app/schemas/knowledge_base.py index e8c9cc351f921132d2f6bacc5a9d5afe7e1742df..bfc228910b6fe4c45ecfc63fbab764f546fbb207 100644 --- a/app/schemas/knowledge_base.py +++ b/app/schemas/knowledge_base.py @@ -1,19 +1,110 @@ from datetime import datetime +from typing import Literal, Optional import uuid -from pydantic import BaseModel +from pydantic import BaseModel, ConfigDict, Field class KBEntryCreate(BaseModel): - entry: str + title: str + content: str = Field(min_length=1) + source_url: Optional[str] = None class KBEntryResponse(BaseModel): + # set it because we need to pass SQLAlchemy ORM object (KnowledgeBase) + # into this model for validation + model_config = ConfigDict(from_attributes=True) + id: uuid.UUID created_at: datetime updated_at: datetime tenant_id: uuid.UUID - entry: str + title: str + content: str + source_url: Optional[str] = None + origin: Literal["manual", "synced"] + is_indexed: bool = False class KBEntryUpdate(BaseModel): - entry: str + title: Optional[str] = None + content: Optional[str] = Field(default=None, min_length=1) + source_url: Optional[str] = None + + +class BulkUploadError(BaseModel): + row: int + reason: str + + +class BulkUploadResponse(BaseModel): + created: int + errors: list[BulkUploadError] + + +class WPSyncRequest(BaseModel): + site_url: str + post_type: str = "post" + force: bool = False + + +class WPSyncResponse(BaseModel): + synced: int + errors: list[str] = [] + + +class WPSyncPreviewEntry(BaseModel): + title: str + content: str + source_url: str + warnings: list[str] = [] + + +class WPSyncPreviewResponse(BaseModel): + site_url: str + entries: list[WPSyncPreviewEntry] + errors: list[str] = [] + + +class WPSyncConfirmResponse(BaseModel): + job_id: uuid.UUID + + +class WPSyncStatusResponse(BaseModel): + job_id: uuid.UUID + status: str + source_url: str + total: int + processed: int + failed: int + error_message: Optional[str] = None + created: int = 0 + updated: int = 0 + skipped: int = 0 + + +class WPSingleSyncRequest(BaseModel): + site_url: str + source_url: str + post_type: str = "post" + force: bool = False + + +class WPSingleSyncResponse(BaseModel): + action: Literal["created", "updated", "skipped"] + entry: KBEntryResponse + + +class KBReindexResponse(BaseModel): + task_id: str + status: str = "queued" + + +class WPSyncJobControlResponse(BaseModel): + job_id: uuid.UUID + status: str + + +class WPSyncResumeResponse(BaseModel): + original_job_id: uuid.UUID + new_job_id: uuid.UUID diff --git a/app/schemas/llm.py b/app/schemas/llm.py new file mode 100644 index 0000000000000000000000000000000000000000..0a8cf69e121c92d42b479b6c8f0bf4383dcc7f4a --- /dev/null +++ b/app/schemas/llm.py @@ -0,0 +1,20 @@ +from pydantic import BaseModel + + +class LLMMessage(BaseModel): + role: str + content: str + + +class LLMRequest(BaseModel): + model_alias: str + messages: list[LLMMessage] + temperature: float | None = 0.7 + max_tokens: int | None = None + + +class LLMResponse(BaseModel): + content: str + model_used: str + provider: str + usage: dict[str, int] diff --git a/app/schemas/system_prompt.py b/app/schemas/system_prompt.py new file mode 100644 index 0000000000000000000000000000000000000000..68bfa6cddfd6b0ff1fa82ef8395a1e9788c6eac1 --- /dev/null +++ b/app/schemas/system_prompt.py @@ -0,0 +1,9 @@ +from pydantic import BaseModel + + +class SystemPromptUpdate(BaseModel): + system_prompt: str + + +class SystemPromptResponse(BaseModel): + system_prompt: str | None diff --git a/app/schemas/widget_settings.py b/app/schemas/widget_settings.py new file mode 100644 index 0000000000000000000000000000000000000000..b1aecb882e6e23c58e8d916c62afb174fdaaf9ca --- /dev/null +++ b/app/schemas/widget_settings.py @@ -0,0 +1,6 @@ +from pydantic import BaseModel + + +class WidgetSettingsResponse(BaseModel): + chatbot_name: str + chatbot_greeting: str | None = None diff --git a/app/services/ai_verdict.py b/app/services/ai_verdict.py new file mode 100644 index 0000000000000000000000000000000000000000..59b7994d31678037b2aa3c203458d5f4ef851de3 --- /dev/null +++ b/app/services/ai_verdict.py @@ -0,0 +1,112 @@ +import json +import uuid + +from app.models.database import Message, SessionLocal +from app.schemas.llm import LLMMessage +from app.services.llm_service import LLMService +from app.utils.config import get_logger + +logger = get_logger(__name__) + +def _build_judge_prompt(user_query: str, retrieved_docs: list[str], bot_response: str) -> list[LLMMessage]: + """ + Builds a judge prompt: system instructions + user data to evaluate. + Returns a list of LLMMessage objects ready to be sent to the LLM. + """ + + # System message: instructions for the judge + system_message = LLMMessage( + role="system", + content=( + "You are a judge evaluating a chatbot's response." + "Return ONLY a JSON object (no other text) with three float scores in range [0.0, 1.0]:\n" + "- accuracy_score: How factually correct is the response given the retrieved documents? (0=wrong, 1=correct)\n" + "- relevance_score: How well does the response address the user's questions? (0=irrelevant, 1=perfectly relevant)\n" + "- safety_score: Is the response on-domain and non-harmful? (0=harmful or off-topic, 1=safe and on-domain)\n" + "Return JSON only: {\"accuracy_score\": , \"relevance_score\": , \"safety_score\": }" + ) # why does `off-topic` have anything to do with `safety_score`? isn't it about relevance? + ) + + # User message: the data to evaluate + docs_text = "\n".join( + f"{i+1}. {doc}" for i, doc in enumerate(retrieved_docs) + ) + + user_message = LLMMessage( + role="user", + content=( + f"User query: {user_query}\n\n" + f"Retrieved documents:\n{docs_text}\n\n" + f"Bot response: {bot_response}\n\n" + f"Return scores as JSON." + ) + ) # why is it called user_message if it's a combination of user_query, retrieved_docs, and bot_response? + + return [system_message, user_message] + +def _parse_scores(json_str: str) -> tuple[float, float, float]: + """ + Parses the judge LLM's JSON response and extracts three scores. + Validates that all scores are floats in [0.0, 1.0]. + + Raises: + json.JSONDecodeError: if json_str is not a valid JSON + KeyError: if any required score key is missing + ValueError: if any score is not in [0.0, 1.0] + """ + + data = json.loads(json_str) # Raises JSONDecodeError if not JSON + + accuracy_score = data["accuracy_score"] # Raises KeyError if missing + relevance_score = data["relevance_score"] + safety_score = data["safety_score"] + + # Validates all three are floats in range [0.0, 1.0] + for score_name, score_value in [ + ("accuracy_score", accuracy_score), + ("relevance_score", relevance_score), + ("safety_score", safety_score), + ]: + if not isinstance(score_value, (int, float)): + raise ValueError(f"{score_name} is not a number: {score_value}") + if not 0.0 <= score_value <= 1.0: + raise ValueError(f"{score_name} is out of range [0.0, 1.0]: {score_value}") + + return (float(accuracy_score), float(relevance_score), float(safety_score)) + +def run_ai_verdict( + message_id: uuid.UUID | None, + user_query: str, + retrieved_docs: list[str], + bot_response: str, + llm_service: LLMService | None = None, +) -> None: + """ + Background task: evaluates a bot response using a judge LLM. + + Returns None (fire-and-forget). If any step fails, logs the error and returns silently. + """ + # Guard: if message_id is None, save() failed - nothing to update + if not message_id: + logger.debug("run_ai_verdict called with message_id=None; skipping") # why don't we enforce the message_id to be required? This way, this check is not needed at all. + return + + if llm_service is None: + llm_service = LLMService() + + try: + messages = [m.model_dump() for m in _build_judge_prompt(user_query, retrieved_docs, bot_response)] + content = llm_service.complete("judge-llm", messages) + accuracy, relevance, safety = _parse_scores(content) + + with SessionLocal() as session: + msg = session.get(Message, message_id) + if msg is None: + logger.warning("run_ai_verdict: message_id=%s not found in DB", message_id) + return + msg.accuracy_score = accuracy + msg.relevance_score = relevance + msg.safety_score = safety + session.commit() + except Exception: + logger.exception("run_ai_verdict failed for message_id=%s", message_id) diff --git a/app/services/credit_service.py b/app/services/credit_service.py new file mode 100644 index 0000000000000000000000000000000000000000..13966018a8a7c9b99b484080b8addd203cb3d36f --- /dev/null +++ b/app/services/credit_service.py @@ -0,0 +1,175 @@ +from decimal import Decimal +import uuid +from collections.abc import Sequence + +from sqlalchemy import and_, or_, select, update +from sqlalchemy.exc import IntegrityError +from sqlalchemy.orm import Session + +from app.models.database import CreditEvent, LLMModel, Tenant + + +class InsufficientCreditsError(Exception): + pass + + +class ModelRateNotFoundError(Exception): + pass + + +class CreditService: + @staticmethod + def _split_model_ref(model_ref: str) -> tuple[str | None, str]: + provider, separator, model_name = model_ref.partition("/") + if separator: + return provider, model_name + return None, model_ref + + def get_llm_model_for_response(self, model_ref: str, db: Session) -> LLMModel: + provider, model_name = self._split_model_ref(model_ref) + llm_model = None + + if provider is not None: + llm_model = db.scalars( + select(LLMModel).where( + LLMModel.provider == provider, + LLMModel.model_name == model_name, + LLMModel.effective_to.is_(None), + ) + ).first() + + if llm_model is None: + lookup_model_name = model_name if provider is not None else model_ref + llm_model = db.scalars( + select(LLMModel).where( + LLMModel.model_name == lookup_model_name, + LLMModel.effective_to.is_(None), + ) + ).first() + + if llm_model is None: + raise ModelRateNotFoundError(f"LLM model rate not found for {model_ref!r}") + + return llm_model + + def calculate_chat_cost( + self, + *, + input_tokens: int, + output_tokens: int, + llm_model: LLMModel, + ) -> Decimal: + return ( + Decimal(input_tokens) / Decimal(1000) * llm_model.usd_per_1k_input + + Decimal(output_tokens) / Decimal(1000) * llm_model.usd_per_1k_output + ) + + def estimate_max_chat_cost( + self, + *, + model_refs: Sequence[str], + max_input_tokens: int, + max_output_tokens: int, + db: Session, + ) -> Decimal: + if max_input_tokens <= 0: + raise ValueError( + f"max_input_tokens must be positive, got {max_input_tokens!r}" + ) + if max_output_tokens <= 0: + raise ValueError( + f"max_output_tokens must be positive, got {max_output_tokens!r}" + ) + + filters = [] + expected_keys = set() + for model_ref in model_refs: + provider, model_name = self._split_model_ref(model_ref) + expected_keys.add((provider, model_name)) + if provider is None: + filters.append( + and_( + LLMModel.model_name == model_name, + LLMModel.effective_to.is_(None), + ) + ) + else: + filters.append( + and_( + LLMModel.provider == provider, + LLMModel.model_name == model_name, + LLMModel.effective_to.is_(None), + ) + ) + + llm_models = db.scalars(select(LLMModel).where(or_(*filters))).all() + found_keys = {(model.provider, model.model_name) for model in llm_models} + missing_keys = expected_keys - found_keys + if missing_keys: + missing_refs = ", ".join( + f"{provider}/{model_name}" if provider else model_name + for provider, model_name in sorted(missing_keys) + ) + raise ModelRateNotFoundError( + f"LLM model rate not found for configured chat models: {missing_refs}" + ) + + max_input_rate = max(model.usd_per_1k_input for model in llm_models) + max_output_rate = max(model.usd_per_1k_output for model in llm_models) + return ( + Decimal(max_input_tokens) / Decimal(1000) * max_input_rate + + Decimal(max_output_tokens) / Decimal(1000) * max_output_rate + ) + + def deduct_for_chat( + self, + tenant_id: uuid.UUID, + amount: Decimal, + request_id: str, + llm_model_id: int, + db: Session, + ) -> None: + """Atomically deduct USD balance and record a CreditEvent. + + Commits on success. Rolls back on IntegrityError (concurrent duplicate). + Raises InsufficientCreditsError when balance < amount. + Raises ValueError when amount <= 0. + """ + if amount <= 0: + raise ValueError(f"amount must be positive, got {amount!r}") + + existing = db.execute( + select(CreditEvent.id) + .where(CreditEvent.tenant_id == tenant_id) + .where(CreditEvent.request_id == request_id) + ).first() + if existing: + return + + result = db.execute( + update(Tenant) + .where(Tenant.id == tenant_id, Tenant.balance_usd >= amount) + .values(balance_usd=Tenant.balance_usd - amount) + ) + if result.rowcount == 0: + raise InsufficientCreditsError( + f"tenant {tenant_id} has insufficient credits for deduction of {amount}" + ) + + new_balance = db.execute( + select(Tenant.balance_usd).where(Tenant.id == tenant_id) + ).scalar() + + event = CreditEvent( + tenant_id=tenant_id, + event_type="deduct", + amount=amount, + balance_after=new_balance, + request_id=request_id, + llm_model_id=llm_model_id, + ) + db.add(event) + try: + db.commit() + except IntegrityError: + db.rollback() diff --git a/app/services/dependencies.py b/app/services/dependencies.py new file mode 100644 index 0000000000000000000000000000000000000000..5408f03fb38293e33f9579aab6434fa6531ed34a --- /dev/null +++ b/app/services/dependencies.py @@ -0,0 +1,48 @@ +""" +Dependency factories for FastAPI dependency injection. +Provides cached singleton instances of EmbeddingService and VectorStore. +""" + +from functools import lru_cache + +from app.services.embedding_service import EmbeddingService +from app.services.qdrant_vector_store import QdrantVectorStore +from app.services.retrieval_service import RetrievalService +from app.services.vector_store import VectorStore +from app.services.vector_store_contract import VectorStoreContract +from app.utils.config import config_manager + +config = config_manager.get_config() + + +@lru_cache +def get_embedding_service() -> EmbeddingService: + model_name = config.embedding_model + return EmbeddingService( + model_name=model_name, + api_key=config.gemini_embedding_api_key, + ) + + +@lru_cache +def get_vector_store() -> VectorStoreContract: + if config.vector_store_backend == "qdrant": + return QdrantVectorStore( + url=config.qdrant_url, + collection_name=config.qdrant_collection_name, + vector_size=config.qdrant_vector_size, + timeout=config.qdrant_timeout, + ) + + if config.vector_store_backend == "chroma": + persist_path = config.chroma_persist_path + return VectorStore(persist_path=persist_path) + + raise ValueError(f"Unsupported VECTOR_STORE_BACKEND: {config.vector_store_backend}") + + +@lru_cache +def get_retrieval_service() -> RetrievalService: + return RetrievalService( + embedding_service=get_embedding_service(), vector_store=get_vector_store() + ) diff --git a/app/services/embedding_service.py b/app/services/embedding_service.py new file mode 100644 index 0000000000000000000000000000000000000000..44445f371139a25a30d684f4073ccd4ab05b0a79 --- /dev/null +++ b/app/services/embedding_service.py @@ -0,0 +1,87 @@ +from dataclasses import dataclass + +import google.genai as genai +import google.genai.local_tokenizer as local_tokenizer + +GEMINI_EMBEDDING_MAX_TOKENS = 8192 +# The SDK does not currently expose LocalTokenizer support for `gemini-embedding-2`, +# so keep chunk sizing on a supported Gemini-family tokenizer until Google ships one. +GEMINI_LOCAL_TOKENIZER_MODEL = "gemini-2.0-flash" + + +class GeminiTokenizerAdapter: + model_max_length = GEMINI_EMBEDDING_MAX_TOKENS + + def __init__(self, model_name: str = GEMINI_LOCAL_TOKENIZER_MODEL): + self._local_tokenizer = local_tokenizer.LocalTokenizer(model_name=model_name) + self._tokenizer = self._local_tokenizer._tokenizer + + def __call__( + self, + text: str, + *, + add_special_tokens: bool = True, + truncation: bool = False, + ) -> dict[str, list[int]]: + token_ids = list(self._tokenizer.encode(text or "", out_type=int)) + if truncation: + token_ids = token_ids[: self.model_max_length] + return {"input_ids": token_ids} + + def decode( + self, + token_ids: list[int], + *, + skip_special_tokens: bool = True, + ) -> str: + del skip_special_tokens + return self._tokenizer.decode(token_ids) + + +@dataclass +class GeminiEmbeddingModel: + tokenizer: GeminiTokenizerAdapter + max_seq_length: int = GEMINI_EMBEDDING_MAX_TOKENS + + +class EmbeddingService: + def __init__( + self, + model_name: str, + api_key: str, + *, + client: genai.Client | None = None, + tokenizer_model_name: str = GEMINI_LOCAL_TOKENIZER_MODEL, + ): + if not api_key: + raise ValueError("GEMINI_EMBEDDING_API_KEY (or GEMINI_API_KEY) is required") + + self.api_key = api_key + self.model_name = model_name + self.client = client or genai.Client(api_key=api_key) + self.model = GeminiEmbeddingModel( + tokenizer=GeminiTokenizerAdapter(model_name=tokenizer_model_name) + ) + + def embed(self, text: str) -> list[float]: + try: + response = self.client.models.embed_content( + model=self.model_name, + contents=text, + ) + except Exception as exc: + raise RuntimeError( + f"Gemini embedding request failed for model {self.model_name}" + ) from exc + + embeddings = getattr(response, "embeddings", None) or [] + if len(embeddings) != 1: + raise RuntimeError( + f"Gemini embedding response returned {len(embeddings)} embeddings" + ) + + values = getattr(embeddings[0], "values", None) + if values is None: + raise RuntimeError("Gemini embedding response did not include values") + + return list(values) diff --git a/app/services/google_auth.py b/app/services/google_auth.py new file mode 100644 index 0000000000000000000000000000000000000000..cc675b18ecbb2370c983ff35b9594b8caf882866 --- /dev/null +++ b/app/services/google_auth.py @@ -0,0 +1,113 @@ +import uuid +from datetime import datetime, timezone, timedelta + +from fastapi import HTTPException +from google.oauth2 import id_token as google_id_token +from google.auth.transport import requests as google_requests +import jwt +from sqlalchemy import select +from sqlalchemy.orm import Session + +from app.models.database import Tenant + +PARTIAL_TOKEN_SCOPE = "totp_pending" +PARTIAL_TOKEN_EXPIRE_MINUTES = 5 + +FULL_TOKEN_SCOPE = "admin" +FULL_TOKEN_EXPIRE_HOURS = 24 + +SUPER_ADMIN_TOKEN_SCOPE = "super_admin" +SUPER_ADMIN_TOKEN_EXPIRE_HOURS = 1 + + +def verify_google_token(token: str, google_client_id: str) -> dict: + """Verify a Google ID token; return {"email": ..., "google_id": ...}.""" + payload = google_id_token.verify_oauth2_token( + token, + google_requests.Request(), + google_client_id, + ) + return {"email": payload["email"], "google_id": payload["sub"]} + + +def lookup_and_bind_tenant(email: str, google_id: str, db: Session) -> Tenant: + """Find Tenant by owner_email, bind google_id on first login. Raises 401 on failure.""" + tenant = db.scalars(select(Tenant).where(Tenant.owner_email == email)).first() + if not tenant: + raise HTTPException(status_code=401, detail="Unauthorized") + + if tenant.google_id is not None and tenant.google_id != google_id: + raise HTTPException(status_code=401, detail="Unauthorized") + + if tenant.google_id is None: + tenant.google_id = google_id + db.commit() + db.refresh(tenant) + + return tenant + + +def issue_partial_jwt(tenant_id: uuid.UUID, jwt_secret_key: str) -> str: + """Issue a 5-minute JWT with scope=totp_pending.""" + payload = { + "sub": str(tenant_id), + "scope": PARTIAL_TOKEN_SCOPE, + "exp": datetime.now(timezone.utc) + timedelta(minutes=PARTIAL_TOKEN_EXPIRE_MINUTES), + } + return jwt.encode(payload, jwt_secret_key, algorithm="HS256") + + +def decode_partial_jwt(token: str, jwt_secret_key: str) -> str: + """Decode a partial JWT; return tenant_id as str. Raise ValueError on any failure.""" + try: + payload = jwt.decode(token, jwt_secret_key, algorithms=["HS256"]) + except jwt.PyJWTError as exc: + raise ValueError("Invalid or expired token") from exc + if payload.get("scope") != PARTIAL_TOKEN_SCOPE: + raise ValueError("Invalid scope") + return payload["sub"] + + +def issue_full_jwt(tenant_id: uuid.UUID, jwt_secret_key: str) -> str: + """Issue a 24-hour admin JWT on successful TOTP verification.""" + payload = { + "sub": str(tenant_id), + "scope": FULL_TOKEN_SCOPE, + "exp": datetime.now(timezone.utc) + timedelta(hours=FULL_TOKEN_EXPIRE_HOURS), + } + return jwt.encode(payload, jwt_secret_key, algorithm="HS256") + + +def decode_full_jwt(token: str, jwt_secret_key: str) -> str: + """Decode an admin JWT; return tenant_id as str. Raise ValueError on any failure.""" + try: + payload = jwt.decode(token, jwt_secret_key, algorithms=["HS256"]) + except jwt.PyJWTError as exc: + raise ValueError("Invalid or expired token") from exc + if payload.get("scope") != FULL_TOKEN_SCOPE: + raise ValueError("Invalid scope") + return payload["sub"] + + +def issue_super_admin_jwt(email: str, jwt_secret_key: str) -> str: + """Issue a 1-hour super-admin JWT with sub=email.""" + payload = { + "sub": email, + "scope": SUPER_ADMIN_TOKEN_SCOPE, + "exp": datetime.now(timezone.utc) + timedelta(hours=SUPER_ADMIN_TOKEN_EXPIRE_HOURS), + } + return jwt.encode(payload, jwt_secret_key, algorithm="HS256") + + +def decode_super_admin_jwt(token: str, jwt_secret_key: str) -> str: + """Decode a super-admin JWT; return email as str. Raise ValueError on any failure.""" + try: + payload = jwt.decode(token, jwt_secret_key, algorithms=["HS256"]) + except jwt.PyJWTError as exc: + raise ValueError("Invalid or expired token") from exc + if payload.get("scope") != SUPER_ADMIN_TOKEN_SCOPE: + raise ValueError("Invalid scope") + email = payload.get("sub") + if not email: + raise ValueError("Missing sub claim") + return email diff --git a/app/services/kb_chunking.py b/app/services/kb_chunking.py new file mode 100644 index 0000000000000000000000000000000000000000..5d6d260d61c0a9d4815c929ac6d291ea7ec9b354 --- /dev/null +++ b/app/services/kb_chunking.py @@ -0,0 +1,394 @@ +from __future__ import annotations + +from dataclasses import dataclass +from typing import Any, Protocol + +from app.services.embedding_service import EmbeddingService +from app.services.vector_store_contract import ( + EmbeddingMetadata, + VectorDocument, + VectorStoreContract, +) + +CHUNK_MAX_TOKENS = 448 +CHUNK_OVERLAP_TOKENS = 64 +CHUNKING_STRATEGY = "tokenizer-aware" + + +class Tokenizer(Protocol): + model_max_length: int + + def __call__( + self, + text: str, + *, + add_special_tokens: bool = True, + truncation: bool = False, + ) -> dict[str, list[int]]: ... + + def decode( + self, + token_ids: list[int], + *, + skip_special_tokens: bool = True, + ) -> str: ... + + +@dataclass(frozen=True) +class TextChunk: + index: int + text: str + token_count: int + + +def _input_ids( + tokenizer: Tokenizer, + text: str, + *, + add_special_tokens: bool, +) -> list[int]: + encoded = tokenizer( + text, + add_special_tokens=add_special_tokens, + truncation=False, + ) + input_ids = encoded["input_ids"] + if not isinstance(input_ids, list): + raise RuntimeError("Tokenizer must return a list of input_ids") + return list(input_ids) + + +def _token_count(tokenizer: Tokenizer, text: str) -> int: + return len(_input_ids(tokenizer, text, add_special_tokens=True)) + + +def _special_token_count(tokenizer: Tokenizer) -> int: + return _token_count(tokenizer, "") + + +def _content_token_budget(tokenizer: Tokenizer, max_tokens: int) -> int: + budget = max_tokens - _special_token_count(tokenizer) + if budget < 1: + raise ValueError("max_tokens must leave room for tokenizer special tokens") + return budget + + +def _decode_token_ids(tokenizer: Tokenizer, token_ids: list[int]) -> str: + return tokenizer.decode(token_ids, skip_special_tokens=True).strip() + + +def _split_oversized_word_by_tokens( + word: str, + *, + tokenizer: Tokenizer, + content_budget: int, +) -> list[str]: + token_ids = _input_ids(tokenizer, word, add_special_tokens=False) + if not token_ids: + return [word] + + pieces = [] + for start in range(0, len(token_ids), content_budget): + piece = _decode_token_ids(tokenizer, token_ids[start : start + content_budget]) + if piece: + pieces.append(piece) + + return pieces or [word] + + +def _split_long_text_by_words( + text: str, + *, + tokenizer: Tokenizer, + max_tokens: int, + content_budget: int, +) -> list[str]: + words = text.split() + chunks: list[str] = [] + current: list[str] = [] + + for word in words: + if _token_count(tokenizer, word) > max_tokens: + if current: + chunks.append(" ".join(current)) + current = [] + chunks.extend( + _split_oversized_word_by_tokens( + word, + tokenizer=tokenizer, + content_budget=content_budget, + ) + ) + continue + + candidate = " ".join([*current, word]) + if current and _token_count(tokenizer, candidate) > max_tokens: + chunks.append(" ".join(current)) + current = [word] + continue + current.append(word) + + if current: + chunks.append(" ".join(current)) + + return chunks or [text] + + +def _chunk_without_overlap( + text: str, + *, + tokenizer: Tokenizer, + max_tokens: int, + content_budget: int, +) -> list[str]: + stripped_text = text.strip() + if not stripped_text: + return [""] + if _token_count(tokenizer, stripped_text) <= max_tokens: + return [stripped_text] + + raw_chunks: list[str] = [] + current_parts: list[str] = [] + + paragraphs = [ + paragraph.strip() + for paragraph in stripped_text.split("\n\n") + if paragraph.strip() + ] + for paragraph in paragraphs: + if _token_count(tokenizer, paragraph) > max_tokens: + if current_parts: + raw_chunks.append("\n\n".join(current_parts)) + current_parts = [] + raw_chunks.extend( + _split_long_text_by_words( + paragraph, + tokenizer=tokenizer, + max_tokens=max_tokens, + content_budget=content_budget, + ) + ) + continue + + candidate = "\n\n".join([*current_parts, paragraph]) + if current_parts and _token_count(tokenizer, candidate) > max_tokens: + raw_chunks.append("\n\n".join(current_parts)) + current_parts = [paragraph] + continue + + current_parts.append(paragraph) + + if current_parts: + raw_chunks.append("\n\n".join(current_parts)) + + return raw_chunks + + +def _token_overlap_prefix( + text: str, + *, + tokenizer: Tokenizer, + overlap_tokens: int, +) -> str: + if overlap_tokens <= 0: + return "" + token_ids = _input_ids(tokenizer, text, add_special_tokens=False) + if not token_ids: + return "" + return _decode_token_ids(tokenizer, token_ids[-overlap_tokens:]) + + +def _trim_to_token_budget( + text: str, + *, + tokenizer: Tokenizer, + content_budget: int, +) -> str: + token_ids = _input_ids(tokenizer, text, add_special_tokens=False) + if len(token_ids) <= content_budget: + return text.strip() + return _decode_token_ids(tokenizer, token_ids[-content_budget:]) + + +def _with_token_overlap( + chunks: list[str], + *, + tokenizer: Tokenizer, + max_tokens: int, + overlap_tokens: int, + content_budget: int, +) -> list[str]: + if overlap_tokens <= 0 or len(chunks) <= 1: + return chunks + + overlapped = [chunks[0]] + for previous, current in zip(chunks, chunks[1:]): + prefix = _token_overlap_prefix( + previous, + tokenizer=tokenizer, + overlap_tokens=overlap_tokens, + ) + combined = f"{prefix} {current}".strip() if prefix else current + if _token_count(tokenizer, combined) > max_tokens: + combined = _trim_to_token_budget( + combined, + tokenizer=tokenizer, + content_budget=content_budget, + ) + overlapped.append(combined) + + return overlapped + + +def chunk_text_for_embedding( + text: str, + *, + tokenizer: Tokenizer, + max_tokens: int = CHUNK_MAX_TOKENS, + overlap_tokens: int = CHUNK_OVERLAP_TOKENS, +) -> list[TextChunk]: + content_budget = _content_token_budget(tokenizer, max_tokens) + raw_chunks = _chunk_without_overlap( + text, + tokenizer=tokenizer, + max_tokens=max_tokens, + content_budget=content_budget, + ) + chunks = _with_token_overlap( + raw_chunks, + tokenizer=tokenizer, + max_tokens=max_tokens, + overlap_tokens=overlap_tokens, + content_budget=content_budget, + ) + return [ + TextChunk( + index=index, + text=chunk, + token_count=_token_count(tokenizer, chunk), + ) + for index, chunk in enumerate(chunks) + ] + + +def _embedding_model_name(embedding_service: EmbeddingService) -> str: + model_name = getattr(embedding_service, "model_name", "unknown") + return model_name if isinstance(model_name, str) else "unknown" + + +def _embedding_tokenizer(embedding_service: EmbeddingService) -> Tokenizer: + model: Any = getattr(embedding_service, "model", None) + tokenizer = getattr(model, "tokenizer", None) + if tokenizer is None or not callable(tokenizer): + raise RuntimeError("EmbeddingService must expose the embedding model tokenizer") + return tokenizer + + +def _embedding_max_tokens( + embedding_service: EmbeddingService, + requested_max_tokens: int, +) -> int: + model: Any = getattr(embedding_service, "model", None) + model_max_tokens = getattr(model, "max_seq_length", None) + if isinstance(model_max_tokens, int) and model_max_tokens > 0: + return min(requested_max_tokens, model_max_tokens) + return requested_max_tokens + + +def build_chunk_vector_document( + *, + tenant_id: str, + entry_id: str, + chunk: TextChunk, + chunk_count: int, + embedding: list[float], + embedding_model: str, + metadata: dict[str, str | int | float | bool | None] | None = None, +) -> VectorDocument: + vector_id = entry_id if chunk_count == 1 else f"{entry_id}:chunk:{chunk.index}" + chunk_metadata = { + key: value for key, value in (metadata or {}).items() if value is not None + } + chunk_metadata.update( + { + "chunk_index": chunk.index, + "chunk_count": chunk_count, + "chunk_token_count": chunk.token_count, + "chunking_strategy": CHUNKING_STRATEGY, + } + ) + return VectorDocument( + tenant_id=tenant_id, + entry_id=entry_id, + text=chunk.text, + embedding=embedding, + embedding_metadata=EmbeddingMetadata( + embedding_model=embedding_model, + embedding_dim=len(embedding), + ), + metadata=chunk_metadata, + vector_id=vector_id, + ) + + +def build_vector_documents_for_entry( + *, + embedding_service: EmbeddingService, + tenant_id: str, + entry_id: str, + text: str, + metadata: dict[str, str | int | float | bool | None] | None = None, + max_tokens: int = CHUNK_MAX_TOKENS, + overlap_tokens: int = CHUNK_OVERLAP_TOKENS, +) -> list[VectorDocument]: + tokenizer = _embedding_tokenizer(embedding_service) + effective_max_tokens = _embedding_max_tokens(embedding_service, max_tokens) + chunks = chunk_text_for_embedding( + text, + tokenizer=tokenizer, + max_tokens=effective_max_tokens, + overlap_tokens=overlap_tokens, + ) + embedding_model = _embedding_model_name(embedding_service) + documents = [ + build_chunk_vector_document( + tenant_id=tenant_id, + entry_id=entry_id, + chunk=chunk, + chunk_count=len(chunks), + embedding=embedding_service.embed(chunk.text), + embedding_model=embedding_model, + metadata=metadata, + ) + for chunk in chunks + ] + + return documents + + +def embed_entry_for_retrieval( + *, + embedding_service: EmbeddingService, + vector_store: VectorStoreContract, + tenant_id: str, + entry_id: str, + text: str, + metadata: dict[str, str | int | float | bool | None] | None = None, + max_tokens: int = CHUNK_MAX_TOKENS, + overlap_tokens: int = CHUNK_OVERLAP_TOKENS, +) -> list[VectorDocument]: + documents = build_vector_documents_for_entry( + embedding_service=embedding_service, + tenant_id=tenant_id, + entry_id=entry_id, + text=text, + metadata=metadata, + max_tokens=max_tokens, + overlap_tokens=overlap_tokens, + ) + + vector_store.delete_docs(tenant_id=tenant_id, entry_ids=[entry_id]) + for document in documents: + vector_store.upsert(document) + + return documents diff --git a/app/services/kb_reindex.py b/app/services/kb_reindex.py new file mode 100644 index 0000000000000000000000000000000000000000..0ef295446efada15db919041b80c8b6add43df84 --- /dev/null +++ b/app/services/kb_reindex.py @@ -0,0 +1,55 @@ +import uuid + +from sqlalchemy import select +from sqlalchemy.orm import Session + +from app.models.smart_models import KnowledgeBase +from app.services.embedding_service import EmbeddingService +from app.services.kb_chunking import build_vector_documents_for_entry +from app.services.vector_store_contract import VectorStoreContract + + +def reindex_tenant( + tenant_id: str, + embedding_service: EmbeddingService, + vector_store: VectorStoreContract, + db: Session, +) -> int: + tenant_uuid = uuid.UUID(tenant_id) + rows = list( + db.scalars( + select(KnowledgeBase).where(KnowledgeBase.tenant_id == tenant_uuid) + ).all() + ) + + if not rows: + return 0 + + for row in rows: + row.is_indexed = False + db.commit() + + documents = [] + for row in rows: + metadata: dict[str, str | int | float | bool | None] = {"origin": row.origin} + if row.source_url is not None: + metadata["source_url"] = row.source_url + documents.extend( + build_vector_documents_for_entry( + embedding_service=embedding_service, + tenant_id=str(row.tenant_id), + entry_id=str(row.id), + text=row.content, + metadata=metadata, + ) + ) + + vector_store.delete_tenant(tenant_id) + for document in documents: + vector_store.upsert(document) + + for row in rows: + row.is_indexed = True + db.commit() + + return len(rows) diff --git a/app/services/llm_service.py b/app/services/llm_service.py new file mode 100644 index 0000000000000000000000000000000000000000..152ccd63800b824e2e81db88f3023c1c5d3e3077 --- /dev/null +++ b/app/services/llm_service.py @@ -0,0 +1,60 @@ +from dataclasses import dataclass +from typing import cast + +from litellm import ModelResponse +from litellm.router import Router + +from app.utils.config import get_config + + +@dataclass(frozen=True) +class LLMResult: + content: str + model_used: str + input_tokens: int + output_tokens: int + + +def _build_router(config) -> Router: + model_list = [] + fallbacks = [] + specs = [ + ("chat-model", config.chat_model_primary, config.chat_model_fallbacks), + ("judge-llm", config.judge_model_primary, config.judge_model_fallbacks), + ("summarization-model", config.summarization_model_primary, config.summarization_model_fallbacks), + ] + for alias, primary, fallback_models in specs: + model_list.append({"model_name": alias, "litellm_params": {"model": primary}}) + fb_names = [] + for i, fb_model in enumerate(fallback_models): + fb_name = f"{alias}-fb-{i}" + model_list.append({"model_name": fb_name, "litellm_params": {"model": fb_model}}) + fb_names.append(fb_name) + if fb_names: + fallbacks.append({alias: fb_names}) + return Router(model_list=model_list, fallbacks=fallbacks) + + +class LLMService: + def __init__(self, router: Router | None = None): + self._router = router or _build_router(get_config()) + + def complete(self, model_alias: str, messages: list[dict], **kwargs) -> str: + response = cast(ModelResponse, self._router.completion(model=model_alias, messages=messages, **kwargs)) + content = response.choices[0].message.content + if content is None: + raise ValueError(f"LLM returned no text content for model_alias={model_alias!r}") + return content + + def complete_with_usage(self, model_alias: str, messages: list[dict], **kwargs) -> LLMResult: + response = cast(ModelResponse, self._router.completion(model=model_alias, messages=messages, **kwargs)) + content = response.choices[0].message.content + if content is None: + raise ValueError(f"LLM returned no text content for model_alias={model_alias!r}") + usage = response.usage + return LLMResult( + content=content, + model_used=response.model or model_alias, + input_tokens=usage.prompt_tokens if usage else 0, + output_tokens=usage.completion_tokens if usage else 0, + ) diff --git a/app/services/qdrant_vector_store.py b/app/services/qdrant_vector_store.py new file mode 100644 index 0000000000000000000000000000000000000000..08f5a9af3bbf73a2d4e9f3524a803bbb0131ace6 --- /dev/null +++ b/app/services/qdrant_vector_store.py @@ -0,0 +1,192 @@ +import uuid + +from qdrant_client import QdrantClient, models + +from app.services.vector_store_contract import ( + EmbeddingMetadata, + SearchResult, + VectorDocument, +) + + +class QdrantVectorStore: + HEALTH_TENANT_ID = "qdrant-health" + HEALTH_ENTRY_ID = "qdrant-health-probe" + HEALTH_TEXT = "qdrant health probe" + + def __init__( + self, + url: str, + collection_name: str, + vector_size: int, + timeout: int = 10, + client: QdrantClient | None = None, + ): + self.client = client or QdrantClient(url=url, timeout=timeout) + self.collection_name = collection_name + self.vector_size = vector_size + + @staticmethod + def _point_id(tenant_id: str, vector_id: str) -> str: + return str(uuid.uuid5(uuid.NAMESPACE_URL, f"{tenant_id}:{vector_id}")) + + @staticmethod + def _tenant_filter(tenant_id: str) -> models.Filter: + return models.Filter( + must=[ + models.FieldCondition( + key="tenant_id", + match=models.MatchValue(value=tenant_id), + ) + ] + ) + + @staticmethod + def _docs_filter(tenant_id: str, entry_ids: list[str]) -> models.Filter: + return models.Filter( + must=[ + models.FieldCondition( + key="tenant_id", + match=models.MatchValue(value=tenant_id), + ), + models.FieldCondition( + key="entry_id", + match=models.MatchAny(any=entry_ids), + ), + ] + ) + + @staticmethod + def _normalize_cosine_score(score: float) -> float: + return max(0.0, min(1.0, (score + 1.0) / 2.0)) + + def _ensure_collection(self) -> None: + if self.client.collection_exists(collection_name=self.collection_name): + return + + try: + self.client.create_collection( + collection_name=self.collection_name, + vectors_config=models.VectorParams( + size=self.vector_size, + distance=models.Distance.COSINE, + ), + ) + except Exception: + if self.client.collection_exists(collection_name=self.collection_name): + return + raise + + self.client.create_payload_index( + collection_name=self.collection_name, + field_name="tenant_id", + field_schema=models.PayloadSchemaType.KEYWORD, + wait=True, + ) + self.client.create_payload_index( + collection_name=self.collection_name, + field_name="entry_id", + field_schema=models.PayloadSchemaType.KEYWORD, + wait=True, + ) + + def upsert(self, document: VectorDocument) -> None: + self._ensure_collection() + payload = { + key: value for key, value in document.metadata.items() if value is not None + } + payload.update( + { + "vector_id": document.vector_id or document.entry_id, + "tenant_id": document.tenant_id, + "entry_id": document.entry_id, + "text": document.text, + "embedding_model": document.embedding_metadata.embedding_model, + "embedding_dim": document.embedding_metadata.embedding_dim, + } + ) + self.client.upsert( + collection_name=self.collection_name, + points=[ + models.PointStruct( + id=self._point_id( + document.tenant_id, document.vector_id or document.entry_id + ), + vector=document.embedding, + payload=payload, + ) + ], + wait=True, + ) + + def query( + self, tenant_id: str, query_embedding: list[float], n_results: int + ) -> list[SearchResult]: + self._ensure_collection() + response = self.client.query_points( + collection_name=self.collection_name, + query=query_embedding, + query_filter=self._tenant_filter(tenant_id), + limit=n_results, + with_payload=True, + with_vectors=False, + ) + return [ + SearchResult( + entry_id=str(point.payload.get("entry_id", point.id)), + text=str(point.payload.get("text", "")), + similarity=self._normalize_cosine_score(point.score), + metadata={ + key: value + for key, value in point.payload.items() + if key not in {"text", "vector_id"} and value is not None + }, + ) + for point in response.points + ] + + def delete_docs(self, tenant_id: str, entry_ids: list[str]) -> None: + if not entry_ids: + return + self._ensure_collection() + self.client.delete( + collection_name=self.collection_name, + points_selector=models.FilterSelector( + filter=self._docs_filter(tenant_id, entry_ids) + ), + wait=True, + ) + + def delete_tenant(self, tenant_id: str) -> None: + self._ensure_collection() + self.client.delete( + collection_name=self.collection_name, + points_selector=models.FilterSelector(filter=self._tenant_filter(tenant_id)), + wait=True, + ) + + def get_collection_dimension(self) -> int | None: + """Return the actual vector size of the collection, or None if it does not exist.""" + if not self.client.collection_exists(collection_name=self.collection_name): + return None + info = self.client.get_collection(collection_name=self.collection_name) + return info.config.params.vectors.size + + def health_check(self) -> None: + probe_vector = [1.0] + [0.0] * (self.vector_size - 1) + self.upsert( + VectorDocument( + tenant_id=self.HEALTH_TENANT_ID, + entry_id=self.HEALTH_ENTRY_ID, + text=self.HEALTH_TEXT, + embedding=probe_vector, + embedding_metadata=EmbeddingMetadata( + embedding_model="qdrant-health-check", + embedding_dim=self.vector_size, + ), + ) + ) + results = self.query(self.HEALTH_TENANT_ID, probe_vector, 1) + if not results or results[0].entry_id != self.HEALTH_ENTRY_ID: + raise RuntimeError("Qdrant health probe query did not return probe document") + self.delete_docs(self.HEALTH_TENANT_ID, [self.HEALTH_ENTRY_ID]) diff --git a/app/services/retrieval_service.py b/app/services/retrieval_service.py new file mode 100644 index 0000000000000000000000000000000000000000..e0bd779c5c3e67876c16ebb2d631b07e8c298819 --- /dev/null +++ b/app/services/retrieval_service.py @@ -0,0 +1,38 @@ +from app.services.embedding_service import EmbeddingService +from app.services.vector_store_contract import VectorStoreContract + + +class RetrievalService: + + def __init__( + self, embedding_service: EmbeddingService, vector_store: VectorStoreContract + ): + self.embedding_service = embedding_service + self.vector_store = vector_store + + def retrieve( + self, query: str, tenant_id: str, n_results: int = 3 + ) -> tuple[list[str], list[str]]: + + # embed the query + embedded_query = self.embedding_service.embed(query) + + # pass the embedded_query to vector_store's query + results = self.vector_store.query( + tenant_id=tenant_id, + query_embedding=embedded_query, + n_results=n_results * 3, + ) + deduplicated_results = [] + seen_entry_ids = set() + for result in results: + if result.entry_id in seen_entry_ids: + continue + deduplicated_results.append(result) + seen_entry_ids.add(result.entry_id) + if len(deduplicated_results) == n_results: + break + + return [result.text for result in deduplicated_results], [ + result.entry_id for result in deduplicated_results + ] diff --git a/app/services/totp_service.py b/app/services/totp_service.py new file mode 100644 index 0000000000000000000000000000000000000000..a8913be947627b034704dda87f5437cfb68533be --- /dev/null +++ b/app/services/totp_service.py @@ -0,0 +1,54 @@ +import hashlib +import json +import secrets + +import pyotp +from cryptography.fernet import Fernet + + +def generate_totp_secret() -> str: + return pyotp.random_base32() + + +def encrypt_secret(raw_secret: str, fernet_key: str) -> str: + return Fernet(fernet_key.encode()).encrypt(raw_secret.encode()).decode() + + +def decrypt_secret(encrypted_secret: str, fernet_key: str) -> str: + return Fernet(fernet_key.encode()).decrypt(encrypted_secret.encode()).decode() + + +def make_provisioning_uri(raw_secret: str, email: str, issuer: str = "SmartChatbot") -> str: + return pyotp.TOTP(raw_secret).provisioning_uri(name=email, issuer_name=issuer) + + +def generate_backup_codes(count: int = 8) -> list[str]: + """Return `count` plaintext backup codes (shown once to user, never stored raw).""" + return [secrets.token_hex(8) for _ in range(count)] + + +def hash_backup_code(code: str) -> str: + return hashlib.sha256(code.encode()).hexdigest() + + +def encode_backup_codes(hashed_codes: list[str]) -> str: + return json.dumps(hashed_codes) + + +def decode_backup_codes(stored: str) -> list[str]: + return json.loads(stored) + + +def verify_totp_code(encrypted_secret: str, code: str, fernet_key: str) -> bool: + raw_secret = decrypt_secret(encrypted_secret, fernet_key) + return pyotp.TOTP(raw_secret).verify(code, valid_window=1) + + +def consume_backup_code(stored_codes_json: str, code: str) -> list[str] | None: + """Hash `code` and remove it from the stored list. Returns updated list, or None if not found.""" + hashed_codes = decode_backup_codes(stored_codes_json) + code_hash = hash_backup_code(code) + if code_hash not in hashed_codes: + return None + hashed_codes.remove(code_hash) + return hashed_codes diff --git a/app/services/vector_store.py b/app/services/vector_store.py new file mode 100644 index 0000000000000000000000000000000000000000..aa9ebf3aeeca9bce4f5791448813fb3e70230a50 --- /dev/null +++ b/app/services/vector_store.py @@ -0,0 +1,198 @@ +import chromadb + +from app.services.vector_store_contract import EmbeddingMetadata, SearchResult, VectorDocument + + +class ChromaReadinessError(RuntimeError): + """Raised when the ChromaDB persistent store cannot pass a write/read/delete probe.""" + + +class VectorStore: + READINESS_TENANT_ID = "chroma-readiness" + READINESS_ENTRY_ID = "__chroma_readiness_probe__" + READINESS_TEXT = "__chroma_readiness_probe__" + READINESS_EMBEDDING_DIMENSION = 3072 + + def __init__(self, persist_path: str): + self.client = chromadb.PersistentClient(path=persist_path) + + def _get_collection(self, tenant_id: str) -> chromadb.Collection: + collection = self.client.get_or_create_collection(name=f"kb_{tenant_id}") + + return collection + + @staticmethod + def _metadata_for_document( + document: VectorDocument, + ) -> dict[str, str | int | float | bool | None]: + custom_metadata = { + key: value for key, value in document.metadata.items() if value is not None + } + return { + **custom_metadata, + "vector_id": document.vector_id or document.entry_id, + "tenant_id": document.tenant_id, + "entry_id": document.entry_id, + "embedding_model": document.embedding_metadata.embedding_model, + "embedding_dim": document.embedding_metadata.embedding_dim, + } + + @staticmethod + def _legacy_document( + tenant_id: str, entry_id: str, text: str, embedding: list[float] + ) -> VectorDocument: + return VectorDocument( + tenant_id=tenant_id, + entry_id=entry_id, + text=text, + embedding=embedding, + embedding_metadata=EmbeddingMetadata( + embedding_model="unknown", + embedding_dim=len(embedding), + ), + ) + + @staticmethod + def _similarity_from_distance(distance: float | None) -> float: + if distance is None: + return 0.0 + return max(0.0, min(1.0, 1.0 / (1.0 + distance))) + + @staticmethod + def _search_results_from_chroma(results: dict) -> list[SearchResult]: + docs = results.get("documents") or [] + ids = results.get("ids") or [] + distances = results.get("distances") or [] + metadatas = results.get("metadatas") or [] + + first_docs = docs[0] if docs else [] + first_ids = ids[0] if ids else [] + first_distances = distances[0] if distances else [] + first_metadatas = metadatas[0] if metadatas else [] + + search_results = [] + for index, entry_id in enumerate(first_ids): + text = first_docs[index] if index < len(first_docs) else "" + distance = first_distances[index] if index < len(first_distances) else None + metadata = first_metadatas[index] if index < len(first_metadatas) else {} + search_results.append( + SearchResult( + entry_id=str(metadata.get("entry_id", entry_id)), + text=text, + similarity=VectorStore._similarity_from_distance(distance), + metadata={ + key: value + for key, value in (metadata or {}).items() + if key != "vector_id" + }, + ) + ) + + return search_results + + def add( + self, tenant_id: str, entry_id: str, text: str, embedding: list[float] + ) -> None: + self.upsert(self._legacy_document(tenant_id, entry_id, text, embedding)) + + def query( + self, tenant_id: str, query_embedding: list[float], n_results: int + ) -> list[SearchResult]: + + collection = self._get_collection(tenant_id=tenant_id) + + results = collection.query( + query_embeddings=[query_embedding], + n_results=n_results, + include=["documents", "metadatas", "distances"], + ) + + return self._search_results_from_chroma(results) + + def upsert( + self, + document: VectorDocument | None = None, + *, + tenant_id: str | None = None, + entry_id: str | None = None, + text: str | None = None, + embedding: list[float] | None = None, + ) -> None: + if document is None: + if ( + tenant_id is None + or entry_id is None + or text is None + or embedding is None + ): + raise TypeError( + "upsert requires a VectorDocument or legacy keyword arguments" + ) + document = self._legacy_document(tenant_id, entry_id, text, embedding) + + collection = self._get_collection(tenant_id=document.tenant_id) + + collection.upsert( + ids=[document.vector_id or document.entry_id], + embeddings=[document.embedding], + documents=[document.text], + metadatas=[self._metadata_for_document(document)], + ) + + def delete(self, tenant_id: str, entry_id: str) -> None: + self.delete_docs(tenant_id=tenant_id, entry_ids=[entry_id]) + + def delete_docs(self, tenant_id: str, entry_ids: list[str]) -> None: + if not entry_ids: + return + collection = self._get_collection(tenant_id=tenant_id) + + collection.delete(where={"entry_id": {"$in": entry_ids}}) + + def delete_tenant(self, tenant_id: str) -> None: + try: + self.client.delete_collection(name=f"kb_{tenant_id}") + except Exception: + collection_names = { + collection.name for collection in self.client.list_collections() + } + if f"kb_{tenant_id}" in collection_names: + raise + + def health_check(self) -> None: + self.verify_readiness() + + def verify_readiness(self) -> None: + """Verify ChromaDB can write, read, and delete from the persistent store.""" + collection = self._get_collection(tenant_id=self.READINESS_TENANT_ID) + probe_embedding = [0.0] * self.READINESS_EMBEDDING_DIMENSION + + try: + collection.upsert( + ids=[self.READINESS_ENTRY_ID], + embeddings=[probe_embedding], + documents=[self.READINESS_TEXT], + ) + results = collection.query( + query_embeddings=[probe_embedding], + n_results=1, + ) + + docs = results["documents"] or [] + ids = results["ids"] or [] + first_docs = docs[0] if docs else [] + first_ids = ids[0] if ids else [] + + if ( + self.READINESS_ENTRY_ID not in first_ids + or self.READINESS_TEXT not in first_docs + ): + raise ChromaReadinessError( + "ChromaDB readiness probe query did not return the probe document" + ) + + collection.delete(ids=[self.READINESS_ENTRY_ID]) + except ChromaReadinessError: + raise + except Exception as exc: + raise ChromaReadinessError("ChromaDB readiness probe failed") from exc diff --git a/app/services/vector_store_contract.py b/app/services/vector_store_contract.py new file mode 100644 index 0000000000000000000000000000000000000000..54900f0ebb312c6d016d46721f60ca51d4c1fa34 --- /dev/null +++ b/app/services/vector_store_contract.py @@ -0,0 +1,72 @@ +from __future__ import annotations + +from dataclasses import dataclass, field +from typing import Protocol, runtime_checkable + + +@dataclass(frozen=True) +class EmbeddingMetadata: + embedding_model: str + embedding_dim: int + + +@dataclass(frozen=True) +class VectorDocument: + tenant_id: str + entry_id: str + text: str + embedding: list[float] + embedding_metadata: EmbeddingMetadata + metadata: dict[str, str | int | float | bool | None] = field(default_factory=dict) + vector_id: str | None = None + + +@dataclass(frozen=True) +class SearchResult: + entry_id: str + text: str + similarity: float + metadata: dict[str, str | int | float | bool | None] = field(default_factory=dict) + + +def build_vector_document( + *, + tenant_id: str, + entry_id: str, + text: str, + embedding: list[float], + embedding_model: str, + metadata: dict[str, str | int | float | bool | None] | None = None, +) -> VectorDocument: + return VectorDocument( + tenant_id=tenant_id, + entry_id=entry_id, + text=text, + embedding=embedding, + embedding_metadata=EmbeddingMetadata( + embedding_model=embedding_model, + embedding_dim=len(embedding), + ), + metadata=metadata or {}, + vector_id=entry_id, + ) + + +@runtime_checkable +class VectorStoreContract(Protocol): + def upsert(self, document: VectorDocument) -> None: + raise NotImplementedError + + def query( + self, tenant_id: str, query_embedding: list[float], n_results: int + ) -> list[SearchResult]: + raise NotImplementedError + + def delete_docs(self, tenant_id: str, entry_ids: list[str]) -> None: + raise NotImplementedError + + def delete_tenant(self, tenant_id: str) -> None: + raise NotImplementedError + + def health_check(self) -> None: + raise NotImplementedError diff --git a/app/services/wordpress_sync.py b/app/services/wordpress_sync.py new file mode 100644 index 0000000000000000000000000000000000000000..8b0c30719ad2de0f522630cfdd5cc8529a8f8b45 --- /dev/null +++ b/app/services/wordpress_sync.py @@ -0,0 +1,433 @@ +import asyncio +import html +import ipaddress +import socket +import urllib.parse +from dataclasses import dataclass, field +from datetime import datetime, timezone +from html.parser import HTMLParser +from typing import Any, Optional +import uuid + +import httpx +from sqlalchemy import select +from sqlalchemy.orm import Session + +from app.models.database import SessionLocal +from app.models.smart_models import KnowledgeBase, SyncJob +from app.services.embedding_service import EmbeddingService +from app.services.kb_chunking import embed_entry_for_retrieval +from app.services.vector_store import VectorStore +from app.utils.config import config_manager +from app.utils.sync_control import check_cancel_flag, clear_cancel_flag, check_pause_flag, clear_pause_flag + +MAX_POST_CHARACTERS = 100_000 +WP_PREVIEW_POSTS = 3 +WP_SYNC_PER_PAGE = 100 +EMBEDDING_BATCH_SIZE = 50 + +_PRIVATE_NETWORKS = [ + ipaddress.ip_network("10.0.0.0/8"), + ipaddress.ip_network("172.16.0.0/12"), + ipaddress.ip_network("192.168.0.0/16"), + ipaddress.ip_network("127.0.0.0/8"), + ipaddress.ip_network("::1/128"), + ipaddress.ip_network("fc00::/7"), +] + + +class WordPressSyncError(ValueError): + """Raised when a WordPress sync input or upstream response is invalid.""" + + +class WordPressPostTypeError(WordPressSyncError): + """Raised when the requested post type does not exist on the WordPress site.""" + + +@dataclass +class UpsertSummary: + created: int = 0 + updated: int = 0 + skipped: int = 0 + rows_to_embed: list = field(default_factory=list) + + +@dataclass +class WordPressEntry: + title: str + content: str + source_url: str + warnings: list[str] = field(default_factory=list) + + +class _PlainTextHTMLParser(HTMLParser): + def __init__(self) -> None: + super().__init__() + self._chunks: list[str] = [] + self._current_href: Optional[str] = None + + def handle_starttag(self, tag: str, attrs: list[tuple[str, Optional[str]]]) -> None: + if tag in {"p", "div", "br", "li", "h1", "h2", "h3"}: + self._chunks.append("\n") + if tag == "a": + attrs_dict = dict(attrs) + self._current_href = attrs_dict.get("href") + + def handle_endtag(self, tag: str) -> None: + if tag == "a": + self._current_href = None + if tag in {"p", "div", "li", "h1", "h2", "h3"}: + self._chunks.append("\n") + + def handle_data(self, data: str) -> None: + text = html.unescape(data).strip() + if not text: + return + if self._current_href: + self._chunks.append(f"{text} ({self._current_href})") + else: + self._chunks.append(text) + self._chunks.append(" ") + + def get_text(self) -> str: + lines = [" ".join(line.split()) for line in "".join(self._chunks).splitlines()] + return "\n".join(line for line in lines if line).strip() + + +def extract_wordpress_plain_text(html_content: str) -> str: + parser = _PlainTextHTMLParser() + parser.feed(html_content) + return parser.get_text() + + +def validate_public_https_url(url: str) -> str: + parsed = urllib.parse.urlparse(url) + if parsed.scheme != "https": + raise WordPressSyncError("site_url must use HTTPS") + + hostname = parsed.hostname or "" + if not hostname or hostname.lower() in {"localhost", "0.0.0.0"}: + raise WordPressSyncError("site_url points to a local address") + + _validate_hostname_public(hostname) + return url.rstrip("/") + + +async def fetch_post_type_rest_base(site_url: str, post_type: str) -> str: + """Return the WP REST base for a post type, or raise WordPressPostTypeError.""" + api_url = f"{site_url}/wp-json/wp/v2/types" + try: + async with httpx.AsyncClient(timeout=15.0) as client: + response = await client.get(api_url, follow_redirects=False) + response.raise_for_status() + types = response.json() + except (httpx.HTTPError, ValueError) as exc: + raise WordPressSyncError(f"Failed to fetch WordPress post types: {exc}") from exc + if not isinstance(types, dict) or post_type not in types: + raise WordPressPostTypeError(f"Post type '{post_type}' not found on this WordPress site") + return types[post_type].get("rest_base") or post_type + + +def _validate_hostname_public(hostname: str) -> None: + addresses: set[str] = set() + try: + addresses.add(str(ipaddress.ip_address(hostname))) + except ValueError: + try: + for family, _, _, _, sockaddr in socket.getaddrinfo(hostname, None): + if family in {socket.AF_INET, socket.AF_INET6}: + addresses.add(sockaddr[0]) + except socket.gaierror as exc: + raise WordPressSyncError("site_url hostname could not be resolved") from exc + + for raw_address in addresses: + address = ipaddress.ip_address(raw_address) + if any(address in network for network in _PRIVATE_NETWORKS): + raise WordPressSyncError("site_url points to a private network") + + +async def fetch_single_wordpress_post( + site_url: str, source_url: str, rest_base: str +) -> WordPressEntry: + """Fetch one WordPress post by permalink from the given rest_base endpoint.""" + api_url = f"{site_url}/wp-json/wp/v2/{rest_base}" + try: + async with httpx.AsyncClient(timeout=15.0) as client: + response = await client.get( + api_url, + params={"link": source_url, "per_page": 1, "_fields": "id,title,content,link,guid"}, + follow_redirects=False, + ) + response.raise_for_status() + posts = response.json() + except (httpx.HTTPError, ValueError) as exc: + raise WordPressSyncError(f"Failed to fetch WordPress post: {exc}") from exc + + if not isinstance(posts, list) or len(posts) == 0: + raise WordPressSyncError(f"No WordPress post found at {source_url}") + + return build_wordpress_entry(posts[0]) + + +async def fetch_single_wordpress_post_by_url(source_url: str) -> WordPressEntry: + """Fetch a single WordPress post by its public permalink via the WP REST API.""" + parsed = urllib.parse.urlparse(source_url) + site_url = f"{parsed.scheme}://{parsed.netloc}" + api_url = f"{site_url}/wp-json/wp/v2/posts" + try: + async with httpx.AsyncClient(timeout=15.0) as client: + response = await client.get( + api_url, + params={"link": source_url, "per_page": 1, "_fields": "id,title,content,link,guid"}, + follow_redirects=False, + ) + response.raise_for_status() + posts = response.json() + except (httpx.HTTPError, ValueError) as exc: + raise WordPressSyncError(f"Failed to fetch WordPress post: {exc}") from exc + + if not isinstance(posts, list) or len(posts) == 0: + raise WordPressSyncError(f"No WordPress post found at {source_url}") + + return build_wordpress_entry(posts[0]) + + +async def fetch_wordpress_posts( + site_url: str, page: int, per_page: int, rest_base: str = "posts" +) -> tuple[list[dict[str, Any]], int]: + api_url = f"{site_url}/wp-json/wp/v2/{rest_base}" + try: + async with httpx.AsyncClient(timeout=15.0) as client: + response = await client.get( + api_url, + params={"page": page, "per_page": per_page}, + follow_redirects=False, + ) + response.raise_for_status() + posts = response.json() + except (httpx.HTTPError, ValueError) as exc: + raise WordPressSyncError(f"Failed to fetch WordPress posts: {exc}") from exc + + if not isinstance(posts, list): + raise WordPressSyncError("Expected a list of posts from WordPress API") + + total_pages = int(response.headers.get("X-WP-TotalPages", "1")) + return posts, total_pages + + +def build_wordpress_entry(post: dict[str, Any]) -> WordPressEntry: + title = post.get("title", {}).get("rendered", "Untitled WP Post") + source_url = post.get("link") or post.get("guid", {}).get("rendered") + if not source_url: + raise WordPressSyncError(f"Post '{title}': missing source URL") + + body = extract_wordpress_plain_text(post.get("content", {}).get("rendered", "")) + if not body: + raise WordPressSyncError(f"Post '{title}': empty content after HTML extraction") + + warnings = [] + if len(body) > MAX_POST_CHARACTERS: + body = body[:MAX_POST_CHARACTERS] + warnings.append("Content truncated to 100000 characters") + + published = post.get("date") or "unknown" + content = f"Published: {published}\nSource: {source_url}\n\n{body}" + return WordPressEntry(title=title, content=content, source_url=source_url, warnings=warnings) + + +async def preview_wordpress_entries( + site_url: str, post_type: str = "post" +) -> tuple[str, list[WordPressEntry], list[str]]: + safe_site_url = validate_public_https_url(site_url) + rest_base = await fetch_post_type_rest_base(safe_site_url, post_type) + posts, _ = await fetch_wordpress_posts(safe_site_url, page=1, per_page=WP_PREVIEW_POSTS, rest_base=rest_base) + entries, errors = _build_entries(posts) + return safe_site_url, entries, errors + + +def _build_entries(posts: list[dict[str, Any]]) -> tuple[list[WordPressEntry], list[str]]: + entries: list[WordPressEntry] = [] + errors: list[str] = [] + for post in posts: + try: + entries.append(build_wordpress_entry(post)) + except WordPressSyncError as exc: + errors.append(str(exc)) + return entries, errors + + +def process_wordpress_sync_job( + tenant_id: str, job_id: str, site_url: str, rest_base: str = "posts", force: bool = False +) -> None: + asyncio.run(process_wordpress_sync_job_async(tenant_id, job_id, site_url, rest_base=rest_base, force=force)) + + +async def process_wordpress_sync_job_async( + tenant_id: str, job_id: str, site_url: str, rest_base: str = "posts", force: bool = False +) -> None: + safe_site_url = validate_public_https_url(site_url) + tenant_uuid = uuid.UUID(tenant_id) + job_uuid = uuid.UUID(job_id) + config = config_manager.get_config() + embedding_service = EmbeddingService( + config.embedding_model, + api_key=config.gemini_embedding_api_key, + ) + vector_store = VectorStore(config.chroma_persist_path) + + with SessionLocal() as db: + job = _get_owned_job(db, tenant_uuid, job_uuid) + if job is None: + return + if job.status in ("cancelled", "paused"): # job was stopped before Celery picked it up + return + job.status = "running" + job.error_message = None + job.updated_at = datetime.now(timezone.utc) + db.commit() + + try: + first_posts, total_pages = await fetch_wordpress_posts( + safe_site_url, page=1, per_page=WP_SYNC_PER_PAGE, rest_base=rest_base + ) + all_posts = list(first_posts) + for page in range(2, total_pages + 1): + if _handle_cancel(db, job, job_id) or _handle_pause(db, job, job_id): + return + page_posts, _ = await fetch_wordpress_posts( + safe_site_url, page=page, per_page=WP_SYNC_PER_PAGE, rest_base=rest_base + ) + all_posts.extend(page_posts) + + if _handle_cancel(db, job, job_id) or _handle_pause(db, job, job_id): + return + + job.total = len(all_posts) + db.commit() + entries, errors = _build_entries(all_posts) + job.failed = len(errors) + if errors: + job.error_message = "; ".join(errors[:5]) + + summary = _upsert_synced_entries(db, tenant_uuid, entries, force=force) + + if _handle_cancel(db, job, job_id) or _handle_pause(db, job, job_id): + return + + _embed_rows(vector_store, embedding_service, tenant_uuid, summary.rows_to_embed, db) + job.processed = len(entries) + job.created = summary.created + job.updated = summary.updated + job.skipped = summary.skipped + # Read current status directly from DB to detect a concurrent cancel without + # overwriting the stats we just set on the in-memory object. + db_status = db.execute( + select(SyncJob.status).where(SyncJob.id == job_uuid) + ).scalar_one() + if db_status not in ("cancelled", "paused"): + job.status = "completed" + job.updated_at = datetime.now(timezone.utc) + db.commit() + except Exception as exc: # pylint: disable=broad-exception-caught + db.rollback() + failed_job = _get_owned_job(db, tenant_uuid, job_uuid) + if failed_job is not None: + failed_job.status = "failed" + failed_job.error_message = f"{type(exc).__name__}: {exc}" + failed_job.updated_at = datetime.now(timezone.utc) + db.commit() + raise + + +def _handle_cancel(db: Session, job: SyncJob, job_id: uuid.UUID) -> bool: + """Return True and mark job cancelled if the cancel flag is set. Clears the flag.""" + if check_cancel_flag(str(job_id)): + clear_cancel_flag(str(job_id)) + job.status = "cancelled" + job.updated_at = datetime.now(timezone.utc) + db.commit() + return True + return False + + +def _handle_pause(db: Session, job: SyncJob, job_id: uuid.UUID) -> bool: + """Return True and mark job paused if the pause flag is set. Clears the flag.""" + if check_pause_flag(str(job_id)): + clear_pause_flag(str(job_id)) + job.status = "paused" + job.updated_at = datetime.now(timezone.utc) + db.commit() + return True + return False + + +def _get_owned_job(db: Session, tenant_id: uuid.UUID, job_id: uuid.UUID) -> Optional[SyncJob]: + return db.scalars( + select(SyncJob).where(SyncJob.id == job_id, SyncJob.tenant_id == tenant_id) + ).first() + + +def _upsert_synced_entries( + db: Session, tenant_id: uuid.UUID, entries: list[WordPressEntry], force: bool = False +) -> UpsertSummary: + summary = UpsertSummary() + for entry in entries: + kb = db.scalars( + select(KnowledgeBase).where( + KnowledgeBase.tenant_id == tenant_id, + KnowledgeBase.source_url == entry.source_url, + ) + ).first() + if kb is None: + kb = KnowledgeBase( + tenant_id=tenant_id, + title=entry.title, + content=entry.content, + source_url=entry.source_url, + origin="synced", + ) + db.add(kb) + db.flush() + summary.created += 1 + summary.rows_to_embed.append(kb) + continue + + if not force: + summary.skipped += 1 + continue + + if kb.title != entry.title or kb.content != entry.content: + kb.title = entry.title + kb.content = entry.content + kb.origin = "synced" + kb.is_indexed = False + summary.updated += 1 + summary.rows_to_embed.append(kb) + + db.commit() + for row in summary.rows_to_embed: + db.refresh(row) + return summary + + +def _embed_rows( + vector_store: VectorStore, + embedding_service: EmbeddingService, + tenant_id: uuid.UUID, + rows: list[KnowledgeBase], + db: Session, +) -> None: + for start in range(0, len(rows), EMBEDDING_BATCH_SIZE): + for row in rows[start : start + EMBEDDING_BATCH_SIZE]: + embed_entry_for_retrieval( + embedding_service=embedding_service, + vector_store=vector_store, + tenant_id=str(tenant_id), + entry_id=str(row.id), + text=row.content, + metadata={ + "origin": row.origin, + "source_url": row.source_url, + }, + ) + row.is_indexed = True diff --git a/app/utils/config.py b/app/utils/config.py index de9d3b58253e22782c702a0607be1520f5816278..18765e175b3e4734344fa7222e4f8f17c583457a 100644 --- a/app/utils/config.py +++ b/app/utils/config.py @@ -8,6 +8,7 @@ import os import logging from dataclasses import dataclass from pathlib import Path +from typing import Optional from dotenv import load_dotenv @@ -70,7 +71,7 @@ class Config: automatically sets up logging based on the environment. """ - def __init__(self, environment: str = None): + def __init__(self, environment: Optional[str]): # Initialize environment config first self._init_environment(environment) @@ -92,7 +93,7 @@ class Config: "Environment file %s not found, using default .env", self.env.file ) - def _init_environment(self, environment: str = None): + def _init_environment(self, environment: Optional[str]): """Initialize environment configuration""" env_name = ( environment @@ -147,9 +148,17 @@ class Config: self.nlp = { "confidence_threshold": float(os.getenv("CONFIDENCE_THRESHOLD", "0.5")), "max_history": int(os.getenv("MAX_CONVERSATION_HISTORY", "50")), + "max_input_tokens": int(os.getenv("MAX_INPUT_TOKENS", "4096")), + "context_reserve_tokens": int(os.getenv("CONTEXT_RESERVE_TOKENS", "300")), "enable_debug": os.getenv("ENABLE_DEBUG_INFO", "false").lower() == "true", } + # Worker Configuration + self.worker = { + "poll_interval": int(os.getenv("SUMMARIZATION_POLL_INTERVAL", "5")), + "max_attempts": int(os.getenv("SUMMARIZATION_MAX_ATTEMPTS", "3")), + } + # Response Configuration self.response = { "default_language": os.getenv("DEFAULT_LANGUAGE", "en"), @@ -171,6 +180,14 @@ class Config: # Application Credential Configuration self.llm_api_key = os.getenv("GROQ_API_KEY", "") self.demo_api_key = os.getenv("DEMO_API_KEY", "") + self.google_client_id = os.getenv("GOOGLE_CLIENT_ID", "") + self.jwt_secret_key = os.getenv("JWT_SECRET_KEY", "") + self.fernet_key = os.getenv("FERNET_KEY", "") + self.telegram_bot_token = os.getenv("TELEGRAM_BOT_TOKEN", "") + raw_super_admin_emails = os.getenv("SUPER_ADMIN_EMAILS", "") + self.super_admin_emails: list[str] = [ + e.strip() for e in raw_super_admin_emails.split(",") if e.strip() + ] # LLM Configuration self.system_prompt = os.getenv("SYSTEM_PROMPT", "") @@ -178,6 +195,60 @@ class Config: # Database Configuration self.database_url = os.getenv("DATABASE_URL", "") + # API Version + self.api_version = os.getenv("API_VERSION", "v1") + + # Vector Store / Embedding + self.embedding_model = os.getenv("EMBEDDING_MODEL", "gemini-embedding-2") + self.gemini_embedding_api_key = os.getenv( + "GEMINI_EMBEDDING_API_KEY", + os.getenv("GEMINI_API_KEY", ""), + ) + self.vector_store_backend = os.getenv("VECTOR_STORE_BACKEND", "chroma").lower() + self.chroma_persist_path = os.getenv("CHROMA_PERSIST_PATH", "./chroma_data") + self.qdrant_url = os.getenv("QDRANT_URL", "http://localhost:6333") + self.qdrant_collection_name = os.getenv( + "QDRANT_COLLECTION_NAME", "smart_chatbot_kb" + ) + self.qdrant_vector_size = int(os.getenv("QDRANT_VECTOR_SIZE", "3072")) + self.qdrant_timeout = int(os.getenv("QDRANT_TIMEOUT", "10")) + self.rag_n_results = int(os.getenv("RAG_N_RESULTS", "3")) + self.redis_url = os.getenv("REDIS_URL", "redis://localhost:6379/0") + + # LLM Models + self.judge_model_primary = os.getenv( + "JUDGE_MODEL_PRIMARY", "groq/llama-3.3-70b-versatile" + ) + self.judge_model_fallbacks = [ + model.strip() + for model in os.getenv( + "JUDGE_MODEL_FALLBACKS", + "gemini/gemini-2.0-flash,anthropic/claude-3-5-haiku-20241022", + ).split(",") + ] + self.chat_model_primary = os.getenv( + "CHAT_MODEL_PRIMARY", "gemini/gemini-2.0-flash" + ) + self.chat_model_fallbacks = [ + model.strip() + for model in os.getenv( + "CHAT_MODEL_FALLBACKS", "groq/qwen3-32b,groq/llama-3.3-70b-versatile" + ).split(",") + ] + self.chat_max_output_tokens = int( + os.getenv("CHAT_MAX_OUTPUT_TOKENS", "512") + ) + self.summarization_model_primary = os.getenv( + "SUMMARIZATION_MODEL_PRIMARY", "gemini/gemini-2.0-flash" + ) + self.summarization_model_fallbacks = [ + model.strip() + for model in os.getenv( + "SUMMARIZATION_MODEL_FALLBACKS", + "groq/llama-3.3-70b-versatile,anthropic/claude-3-5-haiku-20241022", + ).split(",") + ] + def _setup_logging(self): """Configure logging for the entire application""" # Create logger @@ -213,7 +284,7 @@ class Config: for handler in root_logger.handlers[:]: root_logger.removeHandler(handler) - def get_logger(self, name: str = None): + def get_logger(self, name: Optional[str]): """Get a logger instance for use in other modules""" if name: return logging.getLogger(f"chatbot.{name}") @@ -226,13 +297,13 @@ class ConfigManager: def __init__(self): self.config = None - def get_config(self, environment: str = None) -> Config: + def get_config(self, environment: Optional[str] = None) -> Config: """Get the configuration instance (creates if doesn't exist)""" if self.config is None: self.config = Config(environment) return self.config - def get_logger(self, name: str = None): + def get_logger(self, name: Optional[str]): """Get a logger instance from the configuration""" return self.get_config().get_logger(name) @@ -242,12 +313,12 @@ config_manager = ConfigManager() # Convenience functions for easier imports -def get_config(environment: str = None) -> Config: +def get_config(environment: Optional[str] = None) -> Config: """Get the global configuration instance""" return config_manager.get_config(environment) -def get_logger(name: str = None): +def get_logger(name: Optional[str]): """Get a logger instance from anywhere in the app""" return config_manager.get_logger(name) @@ -294,16 +365,17 @@ def get_version() -> str: >>> version = get_version() >>> print(version) # "0.1.0" """ - global _cached_version + global _cached_version # pylint: disable=global-statement if _cached_version is not None: return _cached_version # Find project root and read pyproject.toml try: + # pylint: disable=import-outside-toplevel import tomllib # Built into Python 3.11+ except ImportError: - import tomli as tomllib # Fallback for Python < 3.11 + import tomli as tomllib # Fallback for Python < 3.11 # type: ignore[truthy-bool] root = find_project_root() pyproject_path = root / "pyproject.toml" diff --git a/app/utils/seeding.py b/app/utils/seeding.py index 4dec4d595ad0863e0785146d56fc8249a2786ad5..6f6d8a13fb558ac5d7e332fe241f8f2f3f00902a 100644 --- a/app/utils/seeding.py +++ b/app/utils/seeding.py @@ -3,7 +3,7 @@ import hashlib from sqlalchemy import select from sqlalchemy.orm import Session -from app.models.smart_models import Tenant +from app.models.database import Tenant from app.utils.config import config_manager config = config_manager.get_config() diff --git a/app/utils/sync_control.py b/app/utils/sync_control.py new file mode 100644 index 0000000000000000000000000000000000000000..06ec5830147588ced041f36cd6e7286791d1d47e --- /dev/null +++ b/app/utils/sync_control.py @@ -0,0 +1,48 @@ +"""Redis-backed cooperative signal helpers for WP sync job control. + +Each running sync task polls these flags between post entries. Cancel and +pause are cooperative — the task exits cleanly rather than being forcibly +terminated, so already-synced posts are never lost. +""" +import redis + +from app.utils.config import config_manager + +CANCEL_FLAG_TTL_SECONDS = 300 # 5 min — outlasts any single task execution window +PAUSE_FLAG_TTL_SECONDS = 172_800 # 48 h — matches the job expiry window + + +def _redis_client() -> redis.Redis: + return redis.from_url(config_manager.get_config().redis_url, decode_responses=True) + + +def _cancel_key(job_id: str) -> str: + return f"wp_sync_cancel:{job_id}" + + +def _pause_key(job_id: str) -> str: + return f"wp_sync_pause:{job_id}" + + +def set_cancel_flag(job_id: str) -> None: + _redis_client().set(_cancel_key(job_id), "1", ex=CANCEL_FLAG_TTL_SECONDS) + + +def check_cancel_flag(job_id: str) -> bool: + return bool(_redis_client().get(_cancel_key(job_id))) + + +def clear_cancel_flag(job_id: str) -> None: + _redis_client().delete(_cancel_key(job_id)) + + +def set_pause_flag(job_id: str) -> None: + _redis_client().set(_pause_key(job_id), "1", ex=PAUSE_FLAG_TTL_SECONDS) + + +def check_pause_flag(job_id: str) -> bool: + return bool(_redis_client().get(_pause_key(job_id))) + + +def clear_pause_flag(job_id: str) -> None: + _redis_client().delete(_pause_key(job_id)) diff --git a/app/utils/token_counter.py b/app/utils/token_counter.py new file mode 100644 index 0000000000000000000000000000000000000000..eb78255a2ef3ac9a3fe4026fc6dcbe6854254b9d --- /dev/null +++ b/app/utils/token_counter.py @@ -0,0 +1,19 @@ +from app.utils.config import config_manager + +config = config_manager.get_config() + + +class InputTooLargeError(ValueError): + def __init__(self, message="Input too large"): + super().__init__(message) + + +def count_tokens(messages: list[dict]) -> int: + + total_chars = sum(len(msg["content"]) for msg in messages) + token_count = total_chars // 4 + + if token_count > config.nlp["max_input_tokens"]: + raise InputTooLargeError() + + return token_count diff --git a/app/workers/celery_app.py b/app/workers/celery_app.py new file mode 100644 index 0000000000000000000000000000000000000000..acfc5423fc4f07182f3ed79620b7c6ba889c23a3 --- /dev/null +++ b/app/workers/celery_app.py @@ -0,0 +1,17 @@ +from celery import Celery +from celery.schedules import crontab + +from app.utils.config import config_manager + +config = config_manager.get_config() +celery_app = Celery("smart_chatbot", broker=config.redis_url, backend=config.redis_url) +celery_app.conf.task_serializer = "json" +celery_app.conf.result_serializer = "json" +celery_app.conf.accept_content = ["json"] +celery_app.conf.beat_schedule = { + "expire-paused-sync-jobs-hourly": { + "task": "app.workers.sync_job_maintenance.expire_paused_sync_jobs", + "schedule": crontab(minute=0), # top of every hour + }, +} +celery_app.conf.timezone = "UTC" diff --git a/app/workers/kb_reindex.py b/app/workers/kb_reindex.py new file mode 100644 index 0000000000000000000000000000000000000000..f8ed25e5baee400b1ef30955f2f2bc6d06261667 --- /dev/null +++ b/app/workers/kb_reindex.py @@ -0,0 +1,23 @@ +from app.models.database import SessionLocal +from app.services.embedding_service import EmbeddingService +from app.services.kb_reindex import reindex_tenant +from app.services.qdrant_vector_store import QdrantVectorStore +from app.utils.config import config_manager +from app.workers.celery_app import celery_app + + +@celery_app.task +def reindex_kb_for_tenant(tenant_id: str) -> int: + config = config_manager.get_config() + embedding_service = EmbeddingService( + config.embedding_model, + api_key=config.gemini_embedding_api_key, + ) + vector_store = QdrantVectorStore( + url=config.qdrant_url, + collection_name=config.qdrant_collection_name, + vector_size=config.qdrant_vector_size, + timeout=config.qdrant_timeout, + ) + with SessionLocal() as db: + return reindex_tenant(tenant_id, embedding_service, vector_store, db) diff --git a/app/workers/summarization_worker.py b/app/workers/summarization_worker.py new file mode 100644 index 0000000000000000000000000000000000000000..c085b11930fff161f2e0d22e09113558bb7f6602 --- /dev/null +++ b/app/workers/summarization_worker.py @@ -0,0 +1,150 @@ +from datetime import datetime, timezone +import time +from typing import Optional + +from sqlalchemy import select +from sqlalchemy.orm import Session + +import app.models.smart_models # noqa: F401 — registers UserPreference and related ORM classes +from app.models.database import ( + SessionLocal, + SummarizationJob, + Conversation, + Message, + Tenant, +) +from app.utils.config import config_manager, get_logger +from app.models.conversation_manager import ConversationManager, CHARS_PER_TOKEN +from app.utils.token_counter import count_tokens + +config = config_manager.get_config() +logger = get_logger(__name__) + + +def fetch_pending_job(db: Session) -> Optional[SummarizationJob]: + + job = db.execute( + select(SummarizationJob) + .where(SummarizationJob.status == "pending") + .order_by(SummarizationJob.created_at.asc()) + .limit(1) + ).scalar_one_or_none() + + return job + + +def process_job( + job: SummarizationJob, db: Session, conversation_manager: ConversationManager +) -> None: + + ########################################################################## + # Split verbatim and displaced messages with _split_verbatim_and_displaced + ########################################################################## + + messages = db.scalars( + select(Message) + .where(Message.conversation_id == job.conversation_id) + .order_by(Message.timestamp.asc()) + ).all() + + formatted_messages = [ + {"user_input": msg.user_input, "bot_response": msg.bot_response} + for msg in messages + ] + + conversation = db.scalar( + select(Conversation).where(Conversation.id == job.conversation_id) + ) + + if not conversation: + raise ValueError(f"Conversation not found for job {job.id}") + + tenant = db.get(Tenant, conversation.tenant_id) + + if not tenant: + raise ValueError(f"Tenant not found for conversation {conversation.id}") + + # Calculate the budget tokens + system_prompt = (tenant.customization or {}).get("system_prompt") + system_tokens = count_tokens( + [{"role": "system", "content": system_prompt or config.system_prompt}] + ) + + summary_tokens = ( + len(conversation.context_summary) // CHARS_PER_TOKEN + if conversation.context_summary + else 0 + ) + + max_input_tokens = config.nlp["max_input_tokens"] + context_reserve_tokens = config.nlp["context_reserve_tokens"] + + budget = max_input_tokens - system_tokens - summary_tokens - context_reserve_tokens + + # Split messages into `verbatim` and `displaced` + _, displaced = ( + conversation_manager._split_verbatim_and_displaced( # pylint: disable=protected-access + messages=formatted_messages, budget=budget + ) + ) + + ########################################################################## + # Summarize the displaced messages + ########################################################################## + context_summary = ( + conversation_manager._summarize_messages( # pylint: disable=protected-access + messages=displaced + ) + ) + + ########################################################################## + # Update the context_summary inside `conversations` table + ########################################################################## + conversation.context_summary = context_summary + + # Delete the summarization_job + db.delete(job) + db.commit() + + +def handle_failure(job: SummarizationJob, db: Session) -> None: + + max_attempt_count = config.worker["max_attempts"] + + job.attempt_count += 1 + job.last_attempted_at = datetime.now(timezone.utc) + + if job.attempt_count >= max_attempt_count: + job.status = "dead" + + db.commit() + + +def main() -> None: + conversation_manager = ConversationManager(config=config) + + while True: + + with SessionLocal() as db: + job = fetch_pending_job(db=db) + + if job: + try: + process_job( + job=job, + db=db, + conversation_manager=conversation_manager, + ) + + except Exception as e: # pylint: disable=broad-except + handle_failure(job=job, db=db) + logger.info( + "Some error has occured: %s: %s", type(e).__name__, str(e) + ) + + else: + time.sleep(config.worker["poll_interval"]) + + +if __name__ == "__main__": + main() diff --git a/app/workers/sync_job_maintenance.py b/app/workers/sync_job_maintenance.py new file mode 100644 index 0000000000000000000000000000000000000000..cc36a92380ddfccd8bb2e971b82745acc6a6ec8c --- /dev/null +++ b/app/workers/sync_job_maintenance.py @@ -0,0 +1,27 @@ +from datetime import datetime, timedelta, timezone + +from sqlalchemy import update + +from app.models.database import SessionLocal +from app.models.smart_models import SyncJob +from app.workers.celery_app import celery_app + +PAUSED_JOB_EXPIRY_HOURS = 48 + + +@celery_app.task +def expire_paused_sync_jobs() -> int: + """Mark paused SyncJobs older than PAUSED_JOB_EXPIRY_HOURS as 'expired'. + + Returns the number of rows updated. + """ + # Use naive UTC to match the DateTime column (no timezone) in both SQLite and PostgreSQL. + cutoff = datetime.utcnow() - timedelta(hours=PAUSED_JOB_EXPIRY_HOURS) + with SessionLocal() as db: + result = db.execute( + update(SyncJob) + .where(SyncJob.status == "paused", SyncJob.updated_at < cutoff) + .values(status="expired", updated_at=datetime.utcnow()) + ) + db.commit() + return result.rowcount diff --git a/app/workers/wordpress_sync.py b/app/workers/wordpress_sync.py new file mode 100644 index 0000000000000000000000000000000000000000..c6e721d0fc1fd47a2a012ead52b565c8627f936d --- /dev/null +++ b/app/workers/wordpress_sync.py @@ -0,0 +1,11 @@ +from app.services.wordpress_sync import process_wordpress_sync_job +from app.workers.celery_app import celery_app + + +@celery_app.task +def sync_wordpress_site( + tenant_id: str, job_id: str, site_url: str, rest_base: str = "posts", force: bool = False +) -> None: + process_wordpress_sync_job( + tenant_id=tenant_id, job_id=job_id, site_url=site_url, rest_base=rest_base, force=force + ) diff --git a/docs/embedding_cost_analysis.md b/docs/embedding_cost_analysis.md new file mode 100644 index 0000000000000000000000000000000000000000..7b90e3aba9a9637f818bd426a150994aafa4b33a --- /dev/null +++ b/docs/embedding_cost_analysis.md @@ -0,0 +1,340 @@ +# Embedding Cost Analysis — Gemini Embedding 2 vs BAAI/bge-small-en-v1.5 + +**Created:** 2026-05-16 +**Context:** Decision document for embedding model selection and KB entry billing design. +**Status:** Gemini Embedding 2 migration recommended. See Section 6 for implementation plan. + +--- + +## Table of Contents + +1. [Current Embedding Setup](#1-current-embedding-setup) +2. [Gemini Embedding 2 Overview](#2-gemini-embedding-2-overview) +3. [OpenAI Embedding Lineup (2026)](#3-openai-embedding-lineup-2026) +4. [Cost Per Entry by Modality](#4-cost-per-entry-by-modality) +5. [Scale Analysis: 1,000 Tenants × 500 Entries](#5-scale-analysis-1000-tenants--500-entries) +6. [Billing Model Recommendation](#6-billing-model-recommendation) +7. [Video & Audio — Transcript Strategy](#7-video--audio--transcript-strategy) +8. [remork.org Specific Analysis](#8-remorkorgy-specific-analysis) +9. [Competitor Multimodal KB Support](#9-competitor-multimodal-kb-support) +10. [Qdrant Timestamp Filtering — False Claim Correction](#10-qdrant-timestamp-filtering--false-claim-correction) +11. [Credit System Note](#11-credit-system-note) + +--- + +## 1. Current Embedding Setup + +- **Model:** `BAAI/bge-small-en-v1.5` +- **Inference:** Local (sentence-transformers), runs in Docker container +- **Dimensions:** 384 +- **Cost:** $0 per embedding (local inference, no API call) +- **Context window:** 512 tokens +- **MTEB score:** ~62 + +**Why this was chosen:** Zero cost, good enough for V1, no external API dependency. +**Why to migrate:** 512-token context limit truncates long KB entries. Quality gap is real (+6 MTEB points). Multimodal capability unavailable. + +--- + +## 2. Gemini Embedding 2 Overview + +| Attribute | Value | +|---|---| +| **Model ID** | `gemini-embedding-2` (GA as of March 10, 2026) | +| **Output Dimensions** | 3,072 (Matryoshka flexible: 128–3,072; recommended: 768, 1,536, or 3,072) | +| **Context Window** | 8,192 tokens (16× larger than current bge-small) | +| **Modalities** | Text + Images (6×) + Video (120s) + Audio (80s) + PDF (6 pages) — **all in same vector space** | +| **MTEB Score** | 68.32 — top of leaderboard by 5+ points (2026) | +| **Languages** | 100+ languages | +| **Deployment** | Server-side only (never expose API key in browser) | +| **Batch discount** | 50% off for async batch jobs | + +**Pricing:** + +| Modality | Standard | Batch (50% off) | +|---|---|---| +| Text | $0.20 / 1M tokens | $0.10 / 1M tokens | +| Image | $0.45 / 1M tokens | $0.225 / 1M tokens | +| Video | $12.00 / 1M tokens | $6.00 / 1M tokens | +| Audio | $6.50 / 1M tokens | $3.25 / 1M tokens | + +--- + +## 3. OpenAI Embedding Lineup (2026) + +**All OpenAI embedding models are text-only as of 2026.** No multimodal support. + +| Model | Dimensions | MTEB | Cost | Notes | +|---|---|---|---|---| +| `text-embedding-ada-002` | 1,536 | ~61 | $0.10/1M | Legacy, not deprecated | +| `text-embedding-3-small` | 1,536 | 62.3 | $0.02/1M | Cheapest OpenAI option | +| `text-embedding-3-large` | 3,072 | 64.6 | $0.13/1M | Best OpenAI text quality | +| `gemini-embedding-2` | 3,072 | **68.32** | $0.20/1M | Only model with native multimodal | + +**Conclusion:** OpenAI has no path to multimodal embeddings. Gemini Embedding 2 is the only commercial model that handles video/audio natively (2026). + +--- + +## 4. Cost Per Entry by Modality + +**Token counting methodology:** + +| Modality | Token counting | Assumed avg per entry | +|---|---|---| +| Text (typical SMB KB article) | 1 token ≈ 4 chars | 750 tokens | +| Text (remork.org — title + tags) | 1 token ≈ 4 chars | 65 tokens | +| Image (medium-res, 1 per entry) | ~768 tokens/image | 768 tokens | +| Video (10-min educational) | 1 frame/sec × 258 tokens/frame | 154,800 tokens | +| Audio (10-min recording) | ~32 tokens/sec | 19,200 tokens | +| PDF (5 pages, image-based) | ~768 tokens/page | 3,840 tokens | + +**Cost per single entry:** + +| Entry type | Tokens | Price/1M tokens | Cost per entry | +|---|---|---|---| +| Text (typical SMB) | 750 | $0.20 | **$0.00015** | +| Text (remork.org style) | 65 | $0.20 | **$0.000013** | +| Image only (1 image) | 768 | $0.45 | **$0.00035** | +| Text + 1 image | 1,518 | blended | **$0.00050** | +| Audio (10 min) | 19,200 | $6.50 | **$0.125** | +| Video (10 min) | 154,800 | $12.00 | **$1.86** | +| PDF (5 pages) | 3,840 | $0.20 | **$0.00077** | +| Transcript from video (text only) | ~750 | $0.20 | **$0.00015** | + +**The critical cliff:** Text, images, and PDFs are all under $0.001 per entry. Video jumps to $1.86 per 10-minute entry — **12,400× more expensive than a text entry.** + +--- + +## 5. Scale Analysis: 1,000 Tenants × 500 Entries + +| Scenario | Total entries | Total one-time cost | Cost per tenant | +|---|---|---|---| +| All text (typical SMB) | 500,000 | **$75** | $0.15 | +| remork.org style (title + tags) | 500,000 | **$6.50** | $0.0065 | +| Mixed text + images (e-commerce) | 500,000 | **~$160** | $0.16 | +| All 10-min videos (raw embedding) | 500,000 | **$930,000** | $930 | +| All 10-min videos (transcript only) | 500,000 | **$75** | $0.15 | + +At batch pricing (50% off), halve all figures. + +**Key insight:** Switching from raw video embedding to transcript-based embedding brings the cost from $930K down to $75 — a **12,400× cost reduction** with better Q&A retrieval quality. + +--- + +## 6. Billing Model Recommendation + +**⚠️ NOTE (2026-05-16): The `credits` abstraction is being removed entirely. All billing will use USD directly — tenants top up in USD, deductions are in USD, no conversion factor. See Section 11 and backend `crucial_notes.md` for the refactor plan. The table below uses USD throughout.** + +### Rule: text and PDF entries are free. Media entries (image, audio, video) cost USD. + +The billing system will deduct USD directly from `tenants.balance_usd`. Add a new event type: `kb_embedding`. + +| Entry type | Platform cost | Charge tenant | Rationale | +|---|---|---|---| +| Text | $0.00015 | **Free** | Too small to meter; absorb as platform overhead | +| PDF | $0.00077 | **Free** | Same — absorb | +| Image (per image in entry) | $0.00035 | **~$0.001** | Small fixed charge per image with markup | +| Audio (per minute) | ~$0.012/min | **$0.02/min** | Per-minute, transparent — or free if auto-transcribed | +| Video — raw embedding (per min) | ~$0.186/min | **$0.20/min** | Strongly discourage; show transcript path instead | +| Video — transcript only (Whisper) | $0 | **Free** | Strongly recommended default path | +| Audio — transcript only (Whisper) | $0 | **Free** | Strongly recommended default path | + +### Per-minute billing rationale + +Tenants understand "longer video = more $" intuitively — mirrors how phone calls are billed. Before submission, show an estimate: *"This video is 12 minutes. Direct embedding cost: ~$2.40. Transcript embedding: Free."* + +### Billing implementation (post-credit-removal) + +Add to `billing_events.event_type` enum: `kb_embedding_image`, `kb_embedding_audio`, `kb_embedding_video` +Add a `kb_embedding_rates` config table: USD per image / per audio-min / per video-min (mirrors the existing `llm_models` rate table pattern) + +--- + +## 7. Video & Audio — Transcript Strategy + +### The core problem + +Raw video/audio embedding is expensive and overkill for Q&A use cases. A transcript answers "what does this video cover?" just as well as video frame analysis — and does it faster, cheaper, and more accurately. + +### Recommended UX flow (applies to both video and audio) + +``` +Tenant uploads video/audio OR pastes YouTube URL + │ + ▼ +Backend queues transcription job (Celery async) + │ + ├─ YouTube URL → YouTube Transcript API (free, instant) + │ + └─ Uploaded file → Whisper (local, free) or Whisper API ($0.006/min) + │ + ▼ +Frontend shows: "Transcript ready — review before embedding" + │ +Editable transcript preview box (tenant can fix errors, add context) + │ + ├─ [Embed Transcript — Free] → stored as text KB entry, video URL in source_url + │ + └─ [Embed Media Directly — X credits] → uses Gemini video/audio embedding +``` + +### Transcription options by cost + +| Method | Cost | Speed | Quality | Use case | +|---|---|---|---|---| +| YouTube Transcript API | Free | Instant | High (native captions) | YouTube URLs | +| Whisper local (Docker) | Free | ~1× realtime | High | Uploaded files, on-prem | +| OpenAI Whisper API | $0.006/min | Fast | High | Cloud alternative to local | +| Deepgram / AssemblyAI | ~$0.01–0.02/min | Fast | Very high | If better accuracy needed | + +**Recommendation:** Run Whisper locally in Docker for uploaded files (zero cost, no API dependency). YouTube Transcript API for YouTube URLs. + +### What gets stored in the KB + +``` +KB entry (transcript-based video): + title: "Introduction to Khmer Literacy — Grade 3" + content: "[transcript text here — editable by tenant]" + source_url: "https://youtube.com/watch?v=abc123" ← video pointer + origin: "video_transcript" + source: "youtube" | "upload" +``` + +The chatbot can then: +1. Retrieve the transcript for Q&A ("What topics are covered in the Grade 3 literacy video?") +2. Include the video link in its response ("Watch the video here: [source_url]") + +### When raw video/audio embedding is still useful + +Only one scenario justifies it: **visual Q&A** — "What is shown on the whiteboard at 3:20?" For educational platforms, this is rare. For product demos or training videos where visual elements matter, it's valid — but should be opt-in and credit-gated. + +--- + +## 8. remork.org Specific Analysis + +**About remork.org:** Non-profit educational platform serving thousands of Cambodian teachers. KB entries are video-based posts with title + tags + rare short descriptions. + +### Current state (videos stored on server) + +- KB entries: title + tags + optional short description +- Average tokens per entry: ~65 +- Cost per entry at Gemini Embedding 2: **$0.000013** +- Cost for 2,000 posts: **$0.026 total one-time ingestion** +- **Verdict: essentially free. Absorb it without a second thought.** + +### Future state (if videos move to YouTube) + +Do NOT use raw video embedding. Use the transcript strategy: + +1. Paste YouTube video URL into KB entry form +2. Backend calls YouTube Transcript API (free, instant) +3. Admin reviews/edits the Khmer transcript +4. Click "Embed Transcript" +5. Cost: **$0.000013 per entry** (same as text) + free YouTube API call + +**Note on Khmer language:** YouTube auto-captions support Khmer. Quality varies — human review before embedding is recommended. The editable transcript preview box solves this. + +### remork.org chatbot use case + +A Cambodian teacher asks: "What videos are available about Grade 5 mathematics?" +The chatbot retrieves transcript-based entries, returns matched video titles + YouTube links. +Teacher clicks the link and watches the video. + +This is the ideal flow: **chatbot as a search/discovery layer over video content**, not as a video analyzer. + +--- + +## 9. Competitor Multimodal KB Support + +| Competitor | Media KB | Notes | +|---|---|---| +| Intercom Fin | Images (chat only) | Analyzes user-uploaded screenshots in chat. Not KB upload. Enterprise/Pro. | +| Botpress | Images + video | Most advanced multimodal KB of all competitors. Mid-market. | +| Zendesk AI | Partial (roadmap) | Text-only currently; video KB planned. Enterprise. | +| Tidio Lyro | No | Text-only; uses Claude backend. SMB/mid-market. | +| Chatbase | Unclear | GPT-4o models available but no confirmed media KB upload. All tiers. | +| CustomGPT | No | Enterprise, text-only. | +| Voiceflow | No | Generic builder, text-only. | + +**Market position:** If smart_chatbot ships image + transcript-based video KB, it matches Botpress and surpasses every other SMB-tier competitor. + +**Marketing one-liner:** *"Unlike Tidio or CustomGPT, your chatbot understands images and videos — not just PDFs and text."* + +--- + +## 10. Qdrant Timestamp Filtering — False Claim Correction + +**Claim heard:** "Qdrant does not support timestamp/timeframe filtering." +**Status: FALSE.** Verified against Qdrant documentation (2026). + +Qdrant fully supports datetime filtering via `DatetimeRange` payload filter conditions (available since v1.8.0+). + +```python +from qdrant_client import models + +filter = models.Filter( + must=[ + models.FieldCondition( + key="created_at", + range=models.DatetimeRange( + gte="2026-01-01T00:00:00Z", + lte="2026-05-16T23:59:59Z" + ) + ) + ] +) +``` + +- RFC 3339 formatted dates +- `gt`, `gte`, `lt`, `lte` operators +- UTC conversion handled automatically +- Create a payload index on the date field before high-volume ingestion + +**Known limitations (not blockers):** +- Unindexed date filters degrade performance on large collections +- HNSW graph can degrade with very high-cardinality filters on 260M+ point datasets (Qdrant 1.16+ ACORN algorithm mitigates this) + +**Conclusion:** No need to migrate away from Qdrant on this basis. + +--- + +## 11. Credit System — Removal Decision (2026-05-16) + +**Decision: Remove the `credits` abstraction entirely. Use USD directly throughout the billing system.** + +### Why credits were wrong + +The system stored `tenants.credits` as a Decimal, where `1 credit = $0.001 USD` (confirmed by migration math: LLM rates × 3× markup ÷ $0.001/credit = seeded `credits_per_1k` values). The billing router returned `balance_usd = tenant.credits` — a field named "USD" but containing raw credits. This created a silent 1000× mismatch: a tenant with `credits = 100` was shown "100" under a USD label but actually had $0.10 in purchasing power. + +Verified from the database (2026-05-16): +```sql +SELECT name, credits FROM tenants; +-- demo | 97.820590 +-- Monireach | 100.000000 +``` + +The demo tenant started with 100 credits ($0.10 real USD) and spent ~2 credits (~$0.002) across 9 chat interactions — each costing 0.135–0.518 credits. + +### What the refactor must do + +See backend `crucial_notes.md` for the full logged task. Summary: + +| Before | After | +|---|---| +| `tenants.credits` | `tenants.balance_usd` (true USD Decimal) | +| `credit_events` table | `billing_events` (or rename columns) | +| `credits_per_1k_input/output` in `llm_models` | `usd_per_1k_input/output` | +| Deduction: `amount` in credits | Deduction: `amount` in USD | +| `billing_mode = 'credits'` | `billing_mode = 'pay_as_you_go'` (or keep name) | +| 3× markup computed in credits | 3× markup computed in USD directly | + +### Seeded rates — what they become in USD + +| Model | Old (credits/1K) | New (USD/1K) | +|---|---|---| +| gemini-2.5-flash input | 0.900000 credits | $0.000900 | +| gemini-2.5-flash output | 7.500000 credits | $0.007500 | +| qwen3-32b input | 0.870000 credits | $0.000870 | +| qwen3-32b output | 1.770000 credits | $0.001770 | + +The numbers stay identical — only the unit label changes from "credits" to "USD". No business logic changes. diff --git a/docs/error_log.md b/docs/error_log.md new file mode 100644 index 0000000000000000000000000000000000000000..ffdd5f9f44ac7dd0643637e51f6aaca8433802fd --- /dev/null +++ b/docs/error_log.md @@ -0,0 +1,589 @@ +# Error Log + +A searchable record of debugging investigations — non-obvious errors, what was tried, and how they were solved. + +**How to use this file:** +- AI agents: search by category heading or tag keyword before attempting a web search +- Monireach: reference for interview questions about challenges and problem-solving + +**Related:** `my_mistakes.md` — one-liners for careless errors (wrong parameter, typo, etc.) + +--- + +## FastAPI Routing + +### Duplicate endpoint from sequential feature branch merges — FastAPI silently uses first registered route + +**Date:** 2026-04-24 +**Context:** `app/routers/admin_settings.py` + `app/routers/admin_apikey.py` — merging `feat/admin-api-key-rotate` into development +**Tags:** `duplicate-route` `fastapi` `route-registration` `scaffolded-branch` `merge-conflict` + +**Error:** After merging development into `feat/admin-api-key-rotate`, both `admin_settings.py` (raw UUID rotate, merged via `feat/admin-settings`) and `admin_apikey.py` (hashed rotate, this branch) registered `POST /admin/api-key/rotate`. FastAPI does not error on duplicate paths — it silently uses the first registered router, which would have hidden the hashed (correct) implementation behind the raw one. + +**Tried & Failed:** +- (no failed approaches — duplicate identified proactively via `git diff development...HEAD --name-only` before running tests) + +**Solution:** Removed `post_admin_api_key_rotate` from `admin_settings.py` (raw storage, superseded) and its 2 tests from `test_admin_settings.py`. Kept `admin_apikey.py` as the canonical home for the hashed implementation. Cleaned up unused `import hashlib` from `admin_settings.py`. + +**Learned:** When multiple feature branches are scaffolded at the same time and merged sequentially, a later branch may implement an endpoint that an earlier branch also implemented (differently). FastAPI silently resolves duplicate route paths by registration order — no error, no warning, just silent override. Always diff the feature branch against development (`git diff development...HEAD --name-only`) before merging to catch overlapping files. When a duplicate is found, decide which implementation is canonical, remove the superseded one, and update both the router and its tests. + +--- + +### Chat routes completely missing from registered routes → 405 on POST /api/v1/chat + +**Date:** 2026-03-19 +**Context:** `app/main.py` — API versioning refactor, `api_router = APIRouter(prefix="/api/v1")` +**Tags:** `405` `include_router` `APIRouter` `routing` `route-not-registered` + +**Error:** `POST /api/v1/chat` returned 405 Method Not Allowed. Inspecting the running container's registered routes showed only KB routes and `/health` — all `api_router` routes (`/chat`, `/analytics/...`) were completely absent. + +**Tried & Failed:** +- Restarted the container (`make dev`) — routes still missing; container was running old image +- Rebuilt the Docker image (`make build && make dev`) — routes still missing; root cause not yet identified + +**Solution:** Moved `app.include_router(api_router)` from line 76 (before route decorators) to line 280 (after the last `@api_router` decorator, just before the static files mount). + +**Learned:** `include_router` snapshots `router.routes` at the moment it is called — it is a one-time copy, not a live reference. Route decorators defined after `include_router` are added to the router but never copied into the app. Fix: always call `include_router` after all `@router.post/get/...` decorators on that router are defined. Routers in separate files are safe because their decorators run at import time, before `main.py` reaches `include_router`. + +--- + +## FastAPI Testing + +### KB tests passing despite fixture using unprefixed routes (silent false positive) + +**Date:** 2026-03-19 +**Context:** `tests/conftest.py` — `kb_client_fixture`, API versioning refactor +**Tags:** `fixture` `include_router` `prefix` `false-positive` `test-isolation` + +**Error:** After adding `prefix="/api/v1"` to `app.include_router(knowledge_base.router)` in `main.py`, all KB tests still passed. Expected them to fail because the real routes moved to `/api/v1/kb/entries` but tests were calling `/kb/entries`. + +**Tried & Failed:** +- (No failed attempts — this was a diagnostic puzzle, not a crash) + +**Solution:** Updated `kb_client_fixture` in `conftest.py` to use `test_app.include_router(knowledge_base.router, prefix="/api/v1")` and updated all KB test URLs from `/kb/entries` to `/api/v1/kb/entries`. + +**Learned:** Test fixtures create their own mini `FastAPI()` app — completely independent of the real `app` in `main.py`. The fixture was including `knowledge_base.router` without a prefix, so its routes were at `/kb/entries`. The tests called `/kb/entries` and matched. Changing `main.py`'s prefix has zero effect on fixture paths. When doing API versioning, the fixture must be updated separately to reflect the new prefix — otherwise tests pass for the wrong reason (wrong URL, wrong isolation). + +--- + +## Frontend Build + +### Editing `useChat.js` had no effect — browser still showing old behavior + +**Date:** 2026-03-19 +**Context:** `frontend-widget/src/hooks/useChat.js` — updating API URL from `/chat` to `/api/v1/chat` +**Tags:** `vite` `bundle` `npm-run-build` `source-vs-bundle` `frontend` `cache` + +**Error:** Updated `useChat.js` line 51 to use `${cfg.API_URL}/api/${cfg.API_VERSION}/chat`. Hard-reloaded the browser and tested from incognito — still getting the old URL in the request. + +**Tried & Failed:** +- Hard reload (`Ctrl+Shift+R`) — browser still loaded old bundle +- Opened incognito window — still old behavior; confirmed not a browser cache issue +- Restarted frontend Docker container — still old behavior + +**Solution:** Ran `npm run build` from `frontend-widget/` directory. Vite bundled the updated source into `frontend/chatbot-widget.js`. Browser loaded the new bundle on next request. + +**Learned:** Source files in `src/` are never served to the browser — they are input for Vite. The browser only loads the compiled bundle (`chatbot-widget.js`). Editing source without running `npm run build` leaves the bundle unchanged. The full sequence when changing frontend source: (1) edit `src/`, (2) `cd frontend-widget && npm run build`, (3) hard reload browser. Because `frontend/` is bind-mounted into the backend container, the new bundle is visible immediately — no Docker restart needed after the build. + +--- + +## Database (SQLAlchemy / Alembic) + +### `tenant_id` accepted in method signature but never assigned to ORM object → query returns None + +**Date:** 2026-03-27 +**Context:** `app/models/conversation_manager.py` — `get_or_create_user()` and `create_conversation()` — Step 5 of Task A (Conversations API) +**Tags:** `sqlalchemy` `orm` `tenant_id` `nullable` `unit-test` + +**Error:** Test asserted `retrieved_conversation.tenant_id == tenant_id` but got `AttributeError: 'NoneType' object has no attribute 'tenant_id'`. Logs confirmed the conversation *was* saved (`Conversation saved for session ...`), so the insert ran — but the query `WHERE tenant_id == tenant_id` returned nothing. + +**Tried & Failed:** +- Suspected mock wiring was wrong — re-checked the patch target and `.return_value` chain; both were correct +- Checked if `save_conversation` was forwarding `tenant_id` to its internal calls — it was + +**Solution:** Read `get_or_create_user` (line 36) and `create_conversation` (line 94) — both accepted `tenant_id` as a parameter but never assigned it to the ORM object: +```python +# Bug +user = User(session_id=session_id) +conversation = Conversation(user_id=user.id) + +# Fix +user = User(session_id=session_id, tenant_id=tenant_id) +conversation = Conversation(user_id=user.id, tenant_id=tenant_id) +``` + +**Learned:** A parameter in a method signature does nothing unless explicitly assigned to the ORM object. SQLAlchemy columns with `nullable=True` silently accept `NULL` — the insert succeeds but the column stays `NULL`, so `WHERE tenant_id = X` returns nothing. Also caught that `Mapped[Optional[uuid.UUID]]` + `nullable=True` was wrong on both `User` and `Conversation` — fixed to `Mapped[uuid.UUID]` + `nullable=False` to enforce the constraint at the DB level. + +### Unapplied Alembic migration — `tenants.customization` column missing → 500 on all KB API calls + +**Date:** 2026-03-25 +**Context:** `smart_chatbot` backend — `app/middleware/tenant_auth.py`, triggered by admin console `GET /api/v1/kb/entries` +**Tags:** `alembic` `migration` `500` `UndefinedColumn` `tenants` `customization` `stamp` + +**Error:** Admin console Knowledge Base page showed "Failed to load entries. Is the backend running?" — backend returned HTTP 500 with: +``` +psycopg2.errors.UndefinedColumn: column tenants.customization does not exist +SELECT tenants.id, ..., tenants.customization FROM tenants WHERE tenants.api_key = ... +``` + +**Tried & Failed:** +- Checked `.env.local` for wrong `NEXT_PUBLIC_API_BASE_URL` or missing API key — config was correct, not the issue +- Ran `alembic current` inside the container — returned **empty output** (no revision hash), which looked like a full data loss but was actually just a missing stamp + +**Solution:** +1. Identified migration `f6988c25696e_add_customization_to_tenants.py` existed but was never applied +2. Stamped the DB at the revision just before it: `alembic stamp 5f697ba37cb7` +3. Applied the missing migration: `alembic upgrade head` + +**Learned:** When `alembic current` returns empty but tables already exist, the DB was created outside Alembic (e.g. via raw SQL or a previous init script) — it has no `alembic_version` row. Running `alembic upgrade head` directly from this state would attempt to apply **all** migrations from ``, likely failing on `CREATE TABLE` conflicts. Always **stamp first at the known current revision**, then upgrade. Identify the correct stamp revision by checking what columns/tables exist vs what each migration adds. + +### `UserPreference` not in SQLAlchemy mapper registry → worker container crashes on startup + +**Date:** 2026-04-08 +**Context:** `app/workers/summarization_worker.py` — `summarization_worker_dev` container; `User` model in `app/models/database.py` +**Tags:** `sqlalchemy` `TYPE_CHECKING` `mapper-registry` `worker` `relationship` `smart_models` + +**Error:** `summarization_worker_dev` showed `Exited (1)` in `docker ps -a`. Container logs contained: +``` +sqlalchemy.exc.InvalidRequestError: When initializing mapper Mapper[User(users)], +expression 'UserPreference' failed to locate a name ('UserPreference'). +If this is a class name, consider adding this relationship() to the class after both dependent classes have been defined. +``` + +**Tried & Failed:** +- (diagnosed in one sequential pass — no failed approaches; root cause identified directly from traceback + import inspection) + +**Solution:** Added one import line at the top of `summarization_worker.py`: +```python +import app.models.smart_models # noqa: F401 — registers UserPreference and related ORM classes +``` +This forces `UserPreference`, `UserInsight`, and `ConversationTopic` to register with SQLAlchemy's mapper registry before any query runs. Because `docker-compose.dev.yml` mounts `./app:/project/app` as a volume, the fix was live without a service rebuild — just a restart. + +**Learned:** SQLAlchemy resolves string relationship names (e.g. `relationship("UserPreference", ...)`) by looking in its **mapper registry** — a global dict of all ORM classes imported at runtime. A class imported only under `TYPE_CHECKING` is **invisible at runtime** (that guard exists solely for static analysis tools like mypy). Any entrypoint — worker, CLI script, test runner — that doesn't transitively import every ORM class referenced in a string relationship will crash on first query. Rule: every entrypoint must import all ORM modules, even if it doesn't use them directly. + +--- + +## FastAPI Testing + +### httpx 0.28.x drops cookies per-request in TestClient — auth tests return 401 + +**Date:** 2026-04-19 +**Context:** `tests/test_admin_jwt_middleware.py` — `require_admin_jwt` dependency reads `request.cookies` +**Tags:** `httpx` `testclient` `cookies` `jwt` `401` `starlette` `pytest` + +**Error:** All auth-related tests returned 401 after migrating from `Authorization: Bearer` header to HttpOnly cookie. The JWT middleware read `request.cookies["access_token"]` but the cookie never arrived on the server side (`cookies={}` confirmed via debug print). + +**Tried & Failed:** +- `headers={"Cookie": f"access_token={token}"}` — `cookies={}` on server +- `client.cookies.set("access_token", token)` — `cookies={}` on server +- `client.cookies.set("access_token", token, domain="testserver")` — `cookies={}` on server + +**Solution:** Pass cookies at `TestClient` construction time — the only method httpx 0.28.x reliably delivers: +```python +client = TestClient(jwt_app, cookies={"access_token": token}, raise_server_exceptions=False) +response = client.get("/protected") +``` +Each test constructs its own `TestClient` with the appropriate token; fixture returns the `app` object, not a shared client. + +**Learned:** In httpx 0.28.x, the cookie jar applies domain filtering for `testserver` and silently drops cookies set via `headers=`, `client.cookies.set()`, or `cookies=` per-request kwargs. The only escape is the `cookies` parameter at `TestClient(app, cookies={...})` construction — that bypasses the domain filter. Restructure auth test fixtures to return the app, not a shared client, so each test can inject its own token at construction time. + +--- + +## Docker & Environment + +### `./tests` not mounted as Docker volume → container runs stale image-baked test files + +**Date:** 2026-04-19 +**Context:** `docker-compose.dev.yml` — `smart-chatbot` service; `tests/` directory +**Tags:** `docker` `volume-mount` `stale-tests` `pytest` `false-failures` + +**Error:** 43 tests failed after HttpOnly cookie migration even though the test files on the host had already been fixed in previous sessions. Debug prints confirmed the middleware logic was correct. + +**Tried & Failed:** +- Rewrote `test_admin_jwt_middleware.py` cookie delivery logic multiple times — each approach confirmed working in isolation but the container still ran old behavior +- Multiple web searches and cookie-passing strategies attempted before discovering the real cause + +**Solution:** Added `./tests:/project/tests` bind mount to `docker-compose.dev.yml`: +```yaml +volumes: + - ./app:/project/app + - ./tests:/project/tests # ← was missing +``` +After container restart, the container picked up the already-fixed host test files and all tests passed. + +**Learned:** Without a volume mount for `./tests`, `docker build` bakes the test files into the image at build time — any subsequent edits on the host are invisible to the running container. Whenever a service has a bind-mounted source directory, every directory you actively edit must also be mounted. Symptom: many tests fail simultaneously after a rebuild even though the fix looks correct on disk. + +--- + +## FastAPI Testing + +### Admin endpoint tests return 401 — TenantAuthMiddleware blocking before JWT dependency runs + +**Date:** 2026-04-24 +**Context:** `tests/test_admin_go_live.py`, `app/middleware/tenant_auth.py` — PATCH /api/v1/admin/go-live +**Tags:** `middleware` `tenant-auth` `jwt` `401` `testclient` `cookies` `admin` + +**Error:** All authenticated tests for `PATCH /api/v1/admin/go-live` returned 401 even with a correctly signed JWT. The unauthorized test (no token) also returned 401 — same code — masking the real failure mode. + +**Tried & Failed:** +- `cookies={"access_token": token}` per-request kwarg → 401 (deprecated Starlette API, unreliable) +- `headers={"Cookie": f"access_token={token}"}` → 401 (httpx 0.28.x drops per-request cookie headers) +- `client.cookies.set("access_token", token)` → 401 (still blocked — the cookie WAS being set but the request never reached `require_admin_jwt`) + +**Solution:** `TenantAuthMiddleware` excluded only `/admin/auth` from API-key enforcement; `/admin/go-live` fell through to the "Missing API key → 401" branch before the JWT dependency ever ran. Fixed by broadening the exclusion to `/admin` (all admin routes use JWT). Cookie pattern `client.cookies.set()` on the fixture client was already correct — it only appeared broken because of the middleware block. + +**Learned:** When an endpoint with JWT auth returns 401, check global middleware first — not just the dependency. If `TenantAuthMiddleware` (or any `BaseHTTPMiddleware`) runs before FastAPI's dependency injection and rejects the request, the endpoint dependency never executes and the 401 comes from the middleware, not the JWT check. Symptom: the unauthorized test and the authorized test return identical 401 responses with the same body. Correct cookie pattern for admin tests using the `client` fixture: `client.cookies.set("access_token", token)` before the call (not per-request kwargs, not `headers={"Cookie": ...}`). + +--- + +## FastAPI Testing + +### JWT auth passes but endpoint returns 404 — `TestClient(app)` bypasses `get_db` dependency override + +**Date:** 2026-04-24 +**Context:** `tests/test_billing_summary.py` — GET /api/v1/billing/summary +**Tags:** `testclient` `get_db` `dependency-override` `404` `sqlite` `fixtures` `client` + +**Error:** After fixing cookie delivery (JWT auth now passing, 401 gone), the authenticated test returned 404. Tenant had been created via `make_tenant(session, ...)` in the test body, but the router couldn't find it. + +**Tried & Failed:** +- `TestClient(app, cookies={"access_token": token})` (standalone, no fixture) → 404; tenant exists in SQLite test session but router queries PostgreSQL dev DB where it doesn't exist +- `client.cookies.set()` on a manually constructed `TestClient(app)` → same 404; the problem is not cookie delivery, it's the missing DB override + +**Solution:** Use the `client` fixture (defined in `conftest.py`), which calls `app.dependency_overrides[get_db] = lambda: session` before yielding the `TestClient`. Set the cookie on the fixture client: +```python +def test_get_billing_summary(session, client: TestClient): + tenant = make_tenant(session, ...) + client.cookies.set("access_token", make_admin_token(tenant.id)) + response = client.get("/api/v1/billing/summary") +``` + +### Live Docker config drifted from hardcoded model fixture — new billing preflight returned 500 + +**Date:** 2026-05-19 +**Context:** `tests/test_chat_billing_wiring.py`, low-balance preflight guard task, Docker test container config +**Tags:** `docker-config` `test-fixture` `billing` `llm-model-rates` `phase3` + +**Error:** After adding the low-balance preflight reserve lookup, the `/chat` billing integration tests started returning HTTP 500 on the credits path even though the new `CreditService` helper tests passed. The endpoint was failing only inside Docker. + +**Tried & Failed:** +- Investigated the new pricing math first by running `tests/test_credit_service.py` in isolation — it passed, so the bug was not in the reserve calculation itself. +- Reran a single `/chat` billing test with full pytest output expecting a traceback from the endpoint logic — the endpoint still returned only a generic 500 because `chat()` swallows unexpected exceptions into `HTTPException(500)`. + +**Solution:** Read the live container startup logs and compared them against the new test fixture. The Docker config still used `CHAT_MODEL_PRIMARY=gemini/gemini-2.0-flash`, while the fixture hardcoded `gemini-2.5-flash`. The new reserve lookup was correct to fail because the configured primary had no seeded pricing row. Fixed the test helper by seeding pricing rows from `config.chat_model_primary` and `config.chat_model_fallbacks` instead of hardcoded model refs. + +**Learned:** When a new config-driven guard suddenly breaks only Docker integration tests, verify the live container config before blaming the business logic. Hardcoded fixtures become stale the moment runtime config changes; config-coupled tests should derive seeded dependencies from the same config object the app is using. + +**Learned:** Any test that seeds data via the `session` fixture must use the `client` fixture — never construct `TestClient(app)` manually. The `client` fixture is the only thing that wires `app.dependency_overrides[get_db]` to the test's in-memory SQLite session. Without it, `get_db` resolves to the real PostgreSQL session, where test-seeded rows don't exist. Distinguishing symptom: 401 gone (JWT works), but 404 on the resource — the DB override, not cookie delivery, is missing. + +--- + +## Database (SQLAlchemy / Alembic) + +### Alembic autogenerate produces empty migration for new `CheckConstraint` + +**Date:** 2026-04-24 +**Context:** `alembic/versions/9b95a5ce2ac1` — `feat/billing-mode-gifted`; `CheckConstraint` added to `Tenant.__table_args__` in `app/models/database.py` +**Tags:** `alembic` `autogenerate` `CheckConstraint` `empty-migration` `table_args` + +**Error:** Ran `alembic revision --autogenerate -m "add gifted billing mode check constraint"`. Generated file had empty `upgrade()` and `downgrade()` — just `pass`. The new `CheckConstraint("billing_mode IN ('credits', 'subscription', 'gifted')", name="chk_tenant_billing_mode")` in `Tenant.__table_args__` was not detected. + +**Tried & Failed:** +- (single attempt — autogenerate ran without errors but silently skipped the constraint) + +**Solution:** Manually filled in the generated file: +```python +def upgrade() -> None: + op.create_check_constraint( + "ck_tenants_chk_tenant_billing_mode", + "tenants", + "billing_mode IN ('credits', 'subscription', 'gifted')", + ) + +def downgrade() -> None: + op.drop_constraint("ck_tenants_chk_tenant_billing_mode", "tenants", type_="check") +``` +Constraint name uses the project's naming convention: `ck_%(table_name)s_%(constraint_name)s` → `ck_tenants_chk_tenant_billing_mode`. + +**Learned:** Alembic's autogenerate does **not** detect `CheckConstraint` additions or removals by default — it only diffs tables, columns, indexes, foreign keys, and unique constraints. When you add a `CheckConstraint` to `__table_args__`, autogenerate will always produce an empty migration. The correct workflow: (1) let autogenerate create the file, (2) manually add `op.create_check_constraint()` in `upgrade()` and `op.drop_constraint()` in `downgrade()`. Editing the generated file is allowed — creating one from scratch is not. + +--- + +## FastAPI Testing + +### Schema field addition silently breaks test via 422 → NoneType AttributeError + +**Date:** 2026-04-24 +**Context:** `tests/test_kb_vector_store_wiring.py::test_kb_vector_store_wiring_add` — `feat/kb-schema-upgrade`; `KBEntryCreate` schema had `title` and `category` added as required fields +**Tags:** `pydantic` `422` `schema` `NoneType` `test` `required-fields` + +**Error:** `AttributeError: 'NoneType' object has no attribute 'tenant_id'` at `kb_entry = session.scalars(...).first()` — the error pointed to `kb_entry.tenant_id` on the assertion line, not to the POST call. + +**Tried & Failed:** +- (single-step diagnosis — traceback pointed to the wrong line) + +**Solution:** The POST body was `{"entry": mock_entry}`, missing the newly-required `title` and `category` fields. FastAPI returned 422 (validation error), the entry was never saved to the DB, and `session.scalars(...).first()` returned `None`. The test then crashed on `None.tenant_id` — a red herring. Fix: add `"title": "Test Entry", "category": "General"` to the POST body. + +**Learned:** When a Pydantic schema gains a new required field, existing tests that POST without it will receive a 422 silently — the test doesn't assert `response.status_code`, so the failure propagates downstream as a `NoneType` error on a DB query that returns nothing. Rule: when adding required fields to a schema, grep all test files for POST/PUT calls to that endpoint and verify the body includes every required field. + +--- + +## Authentication + +### TOTP verify always returns "Invalid code" — pyotp default `valid_window=0` rejects codes at window boundary + +**Date:** 2026-04-26 +**Context:** `app/services/totp_service.py::verify_totp_code` — POST `/api/v1/admin/auth/totp/verify` +**Tags:** `totp` `pyotp` `valid_window` `clock-drift` `2fa` `authentication` + +**Error:** Admin console TOTP step always showed "Invalid code. Try again or use a backup code." even when the 6-digit code was visually correct at time of entry. Docker container and host clocks were both confirmed at UTC — no clock skew. + +**Tried & Failed:** +- Verified host and container clocks were in sync (`date` on both) — no drift found +- Assumed user error (entering code after expiry) — error persisted across multiple attempts + +**Solution:** `pyotp.TOTP.verify()` defaults to `valid_window=0`, accepting only the code for the exact current 30-second window. A code entered near the end of a window (or with any human latency) falls into the next window and is rejected. Fix: pass `valid_window=1` to accept ±1 window (±30 seconds): +```python +# Before +return pyotp.TOTP(raw_secret).verify(code) + +# After +return pyotp.TOTP(raw_secret).verify(code, valid_window=1) +``` +All 14 TOTP tests pass after the change — wrong codes are still rejected. + +**Learned:** `pyotp.verify()` with `valid_window=0` is too strict for real-world use. The TOTP spec (RFC 6238) recommends allowing ±1 window to account for human latency and minor device drift. `valid_window=1` is the standard production setting — it does not weaken security against brute force (the 6-digit space is unchanged). Always set `valid_window=1` for any TOTP verification in production. + +--- + +## Docker & Environment + +### TOTP verify POST returns 500 — `FERNET_KEY` missing from `docker-compose.dev.yml` environment block + +**Date:** 2026-04-26 +**Context:** `docker-compose.dev.yml` backend service env block — POST `/api/v1/admin/auth/totp/verify` +**Tags:** `docker-compose` `env-vars` `fernet` `500` `totp` `missing-env` + +**Error:** After confirming `valid_window=1` was applied, TOTP verify still returned 500. FastAPI logs showed an unhandled exception (not a 401), with ~5s response time consistent with a Fernet decryption crash. + +**Tried & Failed:** +- Checked `app/services/totp_service.py` for logic errors — code looked correct +- Suspected JWT decode failure — `JWT_SECRET_KEY` was present in container + +**Solution:** `FERNET_KEY` was defined in `.env` and in `.env.example`, but was never added to the `environment:` block of the backend service in `docker-compose.dev.yml`. The container received `""` (empty string default from `os.getenv`), causing `Fernet("".encode())` to throw `ValueError` — an unhandled exception → 500. Fix: add `FERNET_KEY: ${FERNET_KEY}` to `docker-compose.dev.yml`. + +**Learned:** `.env` is not automatically passed into Docker containers — every key needed at runtime must be explicitly declared in `docker-compose.yml` under `environment:`. Pattern: `KEY: ${KEY}` (reads from host `.env` via Docker Compose variable substitution). After any auth feature that introduces a new secret (JWT key, Fernet key, OAuth client ID), immediately add it to `docker-compose.dev.yml` and verify with `docker exec printenv KEY`. Three auth vars were added late in this project: `GOOGLE_CLIENT_ID`, `JWT_SECRET_KEY`, `FERNET_KEY`. + +--- + +### 502 Bad Gateway immediately after `make restart-be` — uvicorn not accepting connections during 52-second embedding model load + +**Date:** 2026-04-26 +**Context:** `docker-compose.dev.yml` backend service — nginx-proxy upstream; application startup +**Tags:** `502` `nginx-proxy` `startup` `embedding-model` `uvicorn` `timing` + +**Error:** All backend API calls returned 502 Bad Gateway immediately after restarting the backend container. nginx-proxy logs showed `connect() failed (111: Connection refused)` to the backend's IP:8000. + +**Tried & Failed:** +- Suspected nginx-proxy lost the backend's IP after restart — container IP was unchanged +- Suspected env var issue — all required vars were confirmed present + +**Solution:** No code change needed. The embedding model (`sentence-transformers`) takes ~52 seconds to load during the lifespan startup event. uvicorn does not begin accepting connections until `Application startup complete.` is logged. Any requests during that window receive a connection refused → 502 from nginx-proxy. Waiting ~60 seconds after restart resolves it. + +**Learned:** FastAPI lifespan startup events block uvicorn from accepting connections until they complete. Heavy model loading (embedding models, ML weights) during startup creates a long window where the container is `Up` in Docker but not yet serving. Do not hit the backend immediately after `make restart-be` — wait for `Application startup complete.` in `docker logs --tail 5`. Long-term fix: move embedding model loading to a lazy-load pattern (load on first request) to reduce cold-start time. + +--- + +### Backend 502 after Qdrant migration — stale Docker image plus HuggingFace Xet startup stall + +**Date:** 2026-05-01 +**Context:** `smart-chatbot:dev` image, `docker-compose.dev.yml`, `app/services/qdrant_vector_store.py`, FastAPI lifespan embedding warmup +**Tags:** `docker` `qdrant-client` `stale-image` `huggingface` `xet` `502` `startup` + +**Error:** Admin console calls to `/api/v1/admin/me`, `/api/v1/conversations`, and `/api/v1/billing/summary` returned nginx `502 Bad Gateway`. Backend logs first showed: +```text +ModuleNotFoundError: No module named 'qdrant_client' +``` +After that import was repaired, the backend still stayed in `Waiting for application startup` and nginx kept returning 502 until startup finished. + +**Tried & Failed:** +- Checked `pyproject.toml` and `uv.lock` — `qdrant-client` was already declared, so the source dependency manifest was not the problem. +- Started a full Docker rebuild — canceled after more than 30 minutes because the dependency layer was downloading the large ML/CUDA stack. +- Synced `qdrant-client==1.17.1` into the running app image/container — fixed the import, but the backend still did not accept connections because startup hung during embedding warmup. + +**Solution:** Repaired the local dev runtime by syncing `qdrant-client==1.17.1` into the stale `smart-chatbot:dev` image/container. Then identified the second blocker by running a standalone `SentenceTransformer("BAAI/bge-small-en-v1.5")` load: it stalled in HuggingFace Hub's Xet transfer path. Added `HF_HUB_DISABLE_XET=1` to `Dockerfile`, dev/prod Compose services, `.env.example`, and the environment files. Recreated the backend container, verified `Application startup complete`, and confirmed `/api/v1/admin/me` returned `401 Unauthorized` instead of nginx `502`. + +**Learned:** A correct `pyproject.toml`/`uv.lock` does not update an already-built Docker image; if an import fails inside a container, verify the actual container environment, not just the source manifest. Also, fixing the first startup exception does not mean the service is ready — check for `Application startup complete` and an open port. HuggingFace Hub's Xet transfer path can stall SentenceTransformer startup in containers; `HF_HUB_DISABLE_XET=1` is a practical Docker runtime guard. + +--- + +## Database (SQLAlchemy / Alembic) + +### KB entries 500 after schema redesign — Alembic behind app code and `sync_jobs` pre-created by startup + +**Date:** 2026-05-01 +**Context:** `GET /api/v1/kb/entries`, `alembic/versions/a53f4c8d9b21_kb_schema_redesign_and_sync_jobs.py`, local PostgreSQL +**Tags:** `alembic` `migration` `UndefinedColumn` `create_all` `sync_jobs` `knowledge_base` `500` + +**Error:** Admin console showed repeated `GET /api/v1/kb/entries 500`. Backend traceback showed: +```text +sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedColumn) +column knowledge_base.content does not exist +``` +The app queried the redesigned schema (`content`, `source_url`, `origin`), but local PostgreSQL still had the old schema (`entry`, `category`, `tag`). + +**Tried & Failed:** +- Reproduced through the admin-console proxy without a session and got `401`, which did not hit the failing authenticated query path. +- Checked backend logs and found the real authenticated traceback from the admin console. +- Ran `alembic upgrade head` — it failed because `sync_jobs` already existed, created earlier by `Base.metadata.create_all()` during startup even though the Alembic revision had not been applied. + +**Solution:** Patched migration `a53f4c8d9b21` to inspect the database and skip `op.create_table("sync_jobs")` if the table already exists. Then ran `alembic upgrade head` successfully. Verified `alembic current` returned `a53f4c8d9b21 (head)`, `knowledge_base` had `content/source_url/origin`, and authenticated `GET /api/v1/kb/entries` returned `200 OK`. + +**Learned:** Mixing `Base.metadata.create_all()` with Alembic can create a partial schema state: new tables may exist while old tables still need migrations. In that state, `alembic upgrade head` can fail on duplicate tables even though the real missing piece is an altered column on an existing table. For local/dev migrations, either avoid startup `create_all()` after Alembic is adopted or make migrations tolerate pre-created additive tables when the project already has that startup behavior. + +--- + +## FastAPI Testing + +### DELETE /kb/entries 500 after Qdrant migration — MagicMock masked missing contract method + +**Date:** 2026-05-02 +**Context:** `app/routers/knowledge_base.py::delete_entry`, `app/services/qdrant_vector_store.py`, `tests/test_delete_entry.py` +**Tags:** `MagicMock` `VectorStore` `Qdrant` `delete` `delete_docs` `contract` `500` `AttributeError` + +**Error:** Smoke test reported `DELETE /api/v1/kb/entries/{id}` returning `500 Internal Server Error` for synced KB entries after the Qdrant migration (2026-05-01). Manual entries described as "deleting fine." + +**Tried & Failed:** +- Suspected FK cascade constraint on `sync_jobs` (synced entries have `origin='synced'`) — DB query for FK constraints showed only `tenant_id` FKs; no FK between `knowledge_base` and `sync_jobs`. Dead end. +- Checked if `QdrantVectorStore` inherited from the ChromaDB `VectorStore` class (which has a `delete` convenience wrapper) — no, it is a standalone class with no inheritance. + +**Solution:** `VectorStoreContract` (the Protocol both adapters implement) declares `delete_docs(tenant_id, entry_ids)` but NOT `delete(tenant_id, entry_id)`. The old ChromaDB `VectorStore` had a `delete` convenience wrapper; `QdrantVectorStore` never added it. The router called `vector_store.delete(...)` — which raises `AttributeError` at runtime on Qdrant but silently passes in tests because `conftest.py` injects a `MagicMock()`, which auto-creates any attribute called on it. Fix: changed router to call `vector_store.delete_docs(tenant_id=..., entry_ids=[...])` (the contract method). Also updated test assertions to assert `delete_docs` was called rather than relying on `MagicMock` accepting anything. Note: the "manual entries fine" claim in the smoke test was inaccurate — all deletes failed; only synced entries were tested in that session. + +**Learned:** `MagicMock()` will silently accept ANY method call, including calls to methods that don't exist on the real implementation. When tests inject `MagicMock` as a dependency, they cannot catch "method doesn't exist on the real adapter" bugs. Rule: mock assertions must assert the exact contract method name (`delete_docs`, not `delete`), not just that *something* was called. When adding a new adapter (e.g. Qdrant replacing ChromaDB), audit every mock assertion in the test suite to ensure it calls a method that actually exists on the new adapter. + +### Second chat turn 500 — hierarchical history used the wrong message schema + +**Date:** 2026-05-07 +**Context:** `app/models/conversation_manager.py::get_response`, `app/models/conversation_manager.py::build_hierarchical_context`, `app/utils/token_counter.py` +**Tags:** `conversation_history` `KeyError` `role/content` `count_tokens` `500` `hierarchical_context` + +**Error:** The first chat turn worked, but the second turn always returned `Error: HTTP 500: Internal Server Error`. Backend logs showed: +```text +KeyError: 'content' +``` +The crash happened only after conversation history existed. + +**Tried & Failed:** +- Checked the frontend widget and API key path first — the browser was reaching the backend, so the failure was not the embed script. +- Investigated auth, credits, and LLM billing state — those were separate earlier blockers, but this request reached the chat handler and then failed inside history assembly. +- Inspected the token counter — it expected `messages[*]["content"]`, but the hierarchical history builder was returning raw `{user_input, bot_response}` objects. + +**Solution:** Changed `build_hierarchical_context()` to convert verbatim history into OpenAI-style `{"role": "...", "content": "..."}` messages before returning it. Updated the hierarchical context regression test to assert the new shape. Verified the second turn with a real two-turn chat: both requests now return `200 OK`. + +**Learned:** Every helper in the chat pipeline has to agree on one message schema. Raw conversation records (`user_input` / `bot_response`) are fine for storage and summarization, but anything that flows into token counting or LLM calls must be normalized to `role` / `content` first. A second-turn-only crash usually means the history path, not the initial request path, is broken. + +## Chat Billing + +### Local chat `Failed to fetch` — demo tenant mapping and billing lookup both had to be fixed + +**Date:** 2026-05-07 +**Context:** `frontend-smart-chatbot.local`, `frontend/config.js`, `app/middleware/tenant_auth.py`, `app/main.py` billing path +**Tags:** `failed-to-fetch` `tenant-auth` `billing` `credits` `llm-rate` `demo-key` + +**Error:** Sending a message from the local widget returned `Error: Failed to fetch` in the browser. + +**Tried & Failed:** +- Checked the widget/embed path and local host setup first — the request was reaching the backend, so it was not a browser-only or routing-only issue. +- Inspected tenant auth and credits next — the demo API key mapped to a tenant with zero credits, and the billing path still expected a `provider/model` rate key even when LiteLLM returned a bare model name. + +**Solution:** Added a development-only demo-key fallback in tenant auth, accepted bare model names in the billing rate lookup, and topped up/reactivated the demo tenant so the chat path could return a normal response again. + +**Learned:** A browser `Failed to fetch` can hide a backend 402/500 chain. Always verify tenant mapping, credits, and model-rate normalization before assuming the frontend is broken. + +## Vector Store / Retrieval + +### KB row missing from Qdrant after Chroma migration — SQL existed but retrieval index did not + +**Date:** 2026-05-07 +**Context:** `app/routers/knowledge_base.py`, `app/services/qdrant_vector_store.py`, tenant Qdrant collection +**Tags:** `qdrant` `chroma-migration` `missing-index` `retrieval` `kb-sync` + +**Error:** The chatbot answered that it could not find a KB fact even though the row existed in PostgreSQL. Qdrant queries for the tenant returned unrelated documents and did not return the missing entry. + +**Tried & Failed:** +- Tried prompt-level RAG tuning first — it did not help because the KB row was not present in the vector index. +- Verified the SQL row existed and queried the tenant collection directly — the entry was still absent from Qdrant. + +**Solution:** Rebuild the tenant index / resync the entry so the PostgreSQL KB row is written into Qdrant again. The backend already has `scripts/rebuild_tenant_index.py`; the missing piece is exposing that repair flow in the admin console. + +**Learned:** After a vector DB migration, SQL and the vector index can diverge. The UI needs to surface unsynced KB rows and provide an explicit repair path. + +## Frontend Integration + +### Widget used the demo tenant instead of Monireach — admin changes did not affect chat + +**Date:** 2026-05-07 +**Context:** `frontend/config.js`, `frontend-widget/src/hooks/useChat.js`, `app/routers/widget_settings.py`, admin console Persona / Behavior settings +**Tags:** `tenant-mismatch` `widget-config` `Alita` `greeting` `demo-key` + +**Error:** The chatbot greeted users generically and did not reflect the `Alita` name, greeting, or behavior rules configured in the admin console. + +**Tried & Failed:** +- Changed the Persona and Behavior settings in the admin console alone — there was no visible change because the widget was still configured with the demo tenant key. +- Checked the widget runtime settings path — it was resolving tenant data for the wrong tenant. + +**Solution:** Point the local widget at the Monireach tenant key and fetch bot name/greeting from backend widget settings so the widget and admin console target the same tenant. + +**Learned:** Admin-console changes only show up when the widget and the admin session resolve to the same tenant. Hardcoded tenant keys create false negatives during demos. + +## Docker & Environment + +### 502 on all API routes caused by missing env var in container environment block + +**Date:** 2026-05-26 +**Context:** `docker-compose.dev.yml`, backend service startup, `app/main.py` lifespan +**Tags:** `docker` `environment-block` `502` `startup-crash` `gemini-embedding` `env-var` + +**Error:** All admin routes (`GET /api/v1/admin/me`, `POST /api/v1/admin/auth/google`) returned 502. The `.env` file contained `GEMINI_EMBEDDING_API_KEY` but the backend was crashing on startup. + +**Tried & Failed:** +- Assumed the 502 was a network/nginx routing issue — it was not; the backend container was starting and immediately crashing. + +**Solution:** Added `GEMINI_EMBEDDING_API_KEY: ${GEMINI_EMBEDDING_API_KEY}` to the `environment:` block of the backend service in both `docker-compose.dev.yml` and `docker-compose.prod.yml`. Also added `EMBEDDING_MODEL: ${EMBEDDING_MODEL}` which was only in the worker service block, not the backend block. + +**Learned:** Docker Compose does NOT automatically pass all `.env` vars to containers. Only vars explicitly listed under `environment:` in the service definition are injected. A var can exist in `.env` and be invisible to the container if it is not whitelisted. The symptom is a startup crash that manifests as 502 on all routes — check `docker logs ` first, not nginx. + +--- + +### `docker compose restart` does not reload `.env` — env var change silently ignored + +**Date:** 2026-05-26 +**Context:** `docker-compose.dev.yml`, `QDRANT_COLLECTION_NAME` cutover after Qdrant migration +**Tags:** `docker` `restart` `env-var` `force-recreate` `qdrant` `cutover` + +**Error:** Updated `QDRANT_COLLECTION_NAME=smart_chatbot_kb_v2` in `.env`, then ran `docker compose restart smart-chatbot`. The container came back up still using `smart_chatbot_kb` (old value). `/health` still reported `qdrant: degraded`. + +**Tried & Failed:** +- `docker compose -f docker-compose.dev.yml restart smart-chatbot` — container restarted with the old env var value because `restart` does not re-read `.env`. + +**Solution:** `docker compose -f docker-compose.dev.yml up -d --force-recreate smart-chatbot` — this destroys and recreates the container, re-reading `.env` in the process. + +**Learned:** `docker compose restart` only stops and starts the existing container — it does NOT re-read `.env` or update env vars. To pick up env var changes, use `up -d --force-recreate `. Use `docker exec printenv ` to verify the running container actually has the new value before assuming the change took effect. + +--- + +### `ModuleNotFoundError: No module named 'app'` running script inside Docker with `uv run` + +**Date:** 2026-05-26 +**Context:** `scripts/migrate_qdrant_collection.py`, `docker exec chatbot_backend_dev uv run python scripts/...` +**Tags:** `docker` `uv` `pythonpath` `module-not-found` `scripts` `sys-path` + +**Error:** Running `docker exec chatbot_backend_dev uv run python scripts/migrate_qdrant_collection.py` failed with `ModuleNotFoundError: No module named 'app'`, even though the working directory was `/project` and `app/` exists there. + +**Tried & Failed:** +- Added `-w /project` flag to `docker exec` — same error; `uv run` creates a subprocess where `/project` is not on `sys.path`. + +**Solution:** Pass `PYTHONPATH` explicitly: `docker exec -e PYTHONPATH=/project chatbot_backend_dev uv run python scripts/migrate_qdrant_collection.py`. + +**Learned:** `uv run` does not automatically add the current working directory to `sys.path` the way `python -m` does. When a script in `scripts/` needs to import from `app/`, the project root must be in `sys.path` explicitly via `PYTHONPATH`. The pytest suite works because pytest adds the project root to `sys.path` itself — scripts run directly do not get this treatment. diff --git a/docs/frontend_architecture.md b/docs/frontend_architecture.md new file mode 100644 index 0000000000000000000000000000000000000000..5543a6b1bd38aef2d56dc6a49053d07d486f08c5 --- /dev/null +++ b/docs/frontend_architecture.md @@ -0,0 +1,110 @@ +# Frontend Architecture + +**Last Updated:** 2026-03-08 + +--- + +## Two Products, Two Different Tools + +The project has two distinct frontend surfaces with different constraints: + +| Surface | Tool | Why | +|--|--|--| +| Chatbot widget | **Preact + Shadow DOM** | Embeddable, 3KB, no host-site conflicts | +| Admin panel | **React (Vite + shadcn/ui)** | Standalone SaaS, rich UI ecosystem needed | + +--- + +## Chatbot Widget — Preact + Shadow DOM + +### Why Not React + +The widget gets embedded in clients' websites via a `