backend/README.md

# Backend Documentation

This folder contains the production-ready FastAPI stack plus the companion MCP servers that power IntegraChat.

## Directory Overview

- `api/` – FastAPI application (routes, services, storage helpers, MCP clients)
- `mcp_server/` – Unified MCP server exposing rag/web/admin tools via namespaces
- `workers/` – Celery workers and schedulers for async ingestion + analytics maintenance

## Prerequisites

- Python 3.10+
- PostgreSQL (with the `vector` extension) for RAG data, or Supabase with pgvector enabled
- SQLite (auto-created in `data/`) for analytics and admin rules
- Optional: Ollama running locally (default) or Groq API credentials for remote LLMs

Create a virtual environment at the repo root, then:

```bash
pip install -r requirements.txt
cp env.example .env   # update MCP URLs + LLM settings
```

## Running the Services Locally

1. **FastAPI core**  
   ```bash
   uvicorn backend.api.main:app --port 8000 --reload
   ```

2. **Unified MCP server (rag/web/admin)**
   ```bash
   python backend/mcp_server/server.py
   ```
   This single endpoint exposes the following namespaced tools:
   - `rag.search`, `rag.ingest`, `rag.delete`
   - `web.search`
   - `admin.getRules`, `admin.addRule`, `admin.deleteRule`, `admin.logViolation`

3. **Optional workers** (if running Celery-based ingestion/analytics jobs):
   ```bash
   celery -A backend.workers.ingestion_worker worker --loglevel=info
   celery -A backend.workers.analytics_worker worker --loglevel=info
   ```

The Gradio UI (`python app.py`) and the Next.js operator console (see `frontend/README.md`) both talk to the FastAPI layer at `http://localhost:8000`.

## Key Endpoints

All endpoints require the `x-tenant-id` header unless otherwise noted.

| Service | Path | Notes |
| --- | --- | --- |
| Agent | `POST /agent/message` | Autonomous orchestration (RAG/Web/Admin/LLM) |
| Agent Debug | `POST /agent/debug` | Full reasoning trace + tool plan |
| Agent Plan | `POST /agent/plan` | Dry-run planning without executing tools |
| RAG | `POST /rag/ingest-document` | Rich ingestion (text, URL, metadata) |
| RAG | `GET /rag/list` | Paginated document listing per tenant |
| Admin | `POST /admin/rules` | Regex + severity rule ingestion |
| Analytics | `GET /analytics/overview` | Summary metrics (queries, tokens, red flags) |

Refer to the root `README.md` for the complete endpoint tables.

## Diagnostics & Tenant Isolation

Use the helper scripts in the repo root when validating backend changes:

- `python verify_tenant_isolation.py` – Exercises analytics logging, admin rule CRUD, API reachability, and proves RAG tenant isolation by ingesting + querying as multiple tenants.
- `python check_rag_database.py` – Talks directly to the pgvector database to list tenant IDs, preview stored chunks, and run safeguarded searches via `search_vectors()`. Helpful when troubleshooting suspected cross-tenant leakage.
- `python test_manual.py` – Legacy manual smoke test harness (analytics store, admin rules, API surface).

> **Troubleshooting tip:** If the isolation script reports a failure, first run `check_rag_database.py` to confirm documents are tagged with the correct `tenant_id`, then restart the unified MCP server so it reloads the updated SQL filtering logic.

## Environment Variables (excerpt)

Defined in `env.example`:

- `RAG_MCP_URL`, `WEB_MCP_URL`, `ADMIN_MCP_URL`
- `OLLAMA_URL`, `OLLAMA_MODEL` (or `GROQ_API_KEY` + `LLM_BACKEND=groq`)
- `SUPABASE_URL`, `SUPABASE_SERVICE_KEY` (optional admin integrations)
- `APP_ENV`, `LOG_LEVEL`, `API_PORT`

Update these before starting the servers to ensure the agent can reach every MCP endpoint and LLM runtime.

## Unified MCP tool instructions

Agents that speak the Model Context Protocol should connect to the `integrachat` server id defined in `backend/mcp_server/server.py` and call the namespaced tools directly:

| Namespace | Tool | Purpose |
| --- | --- | --- |
| `rag` | `search` | Retrieve tenant-scoped document chunks |
| `rag` | `ingest` | Chunk + store new knowledge |
| `rag` | `delete` | Remove one/all stored documents |
| `web` | `search` | DuckDuckGo English-biased search |
| `admin` | `getRules` | Fetch tenant governance rules (list or detailed) |
| `admin` | `addRule` | Insert or update a rule |
| `admin` | `deleteRule` | Remove a rule by text |
| `admin` | `logViolation` | Persist a red-flag event into analytics |

Always send `tenant_id`, and optionally `user_id`, in the payload so the shared middleware can enforce isolation and log analytics.