update the readme.md files

README.md

@@ -177,6 +177,29 @@ All calls are proxied through the FastAPI backend running at `http://localhost:8
 
 ---
 
+## Testing & Diagnostics
+
+IntegraChat ships with several helper scripts to validate the full stack end-to-end:
+
+- `python verify_tenant_isolation.py`  
+  Runs a comprehensive suite that covers analytics logging, admin rule storage, API reachability, and—most importantly—multi-tenant RAG isolation.  
+  - ✅ **Prerequisites:** FastAPI backend plus all MCP servers (RAG/Web/Admin) running locally.  
+  - ✅ **What it checks:**  
+    1. Direct database writes via the analytics and rules stores  
+    2. CRUD over the `/admin/*` and `/analytics/*` endpoints  
+    3. RAG ingestion and isolation by issuing queries as multiple tenants and ensuring secrets never leak across IDs  
+  - ✅ **Pass criteria:** At least 80 % of the sub-tests succeed (the RAG isolation test must pass for overall success).
+
+- `python check_rag_database.py`  
+  Provides a low-level inspection of the RAG datastore. It connects straight to the pgvector/Postgres instance, lists all tenant IDs, prints sample chunks, and runs `search_vectors()` directly to ensure the SQL `WHERE tenant_id = …` filter is behaving as expected. Use this script when diagnosing suspected cross-tenant leakage or when seeding demo data.
+
+- `python test_manual.py`  
+  The existing manual test runner remains useful for smoke-testing analytics logging, admin rule CRUD, and API response codes. Run it whenever you adjust schemas or update MCP endpoints.
+
+> **Tip:** All scripts assume the Python virtual environment is active (`source venv/bin/activate` or `.\venv\Scripts\activate`) and that `.env` contains the MCP server URLs/LLM settings noted earlier.
+
+---
+
 ## Demo Video
 
 🎥 **[Demo Video Placeholder]** - Coming soon!

backend/README.md

@@ -1,28 +1,91 @@
 # Backend Documentation
 
-This directory contains the IntegraChat backend implementation.
+This folder contains the production-ready FastAPI stack plus the companion MCP servers that power IntegraChat.
 
-## Structure
+## Directory Overview
 
-- `api/` - FastAPI application with routes and services
-- `mcp_servers/` - MCP server implementations (RAG, Web Search, Admin)
-- `workers/` - Celery workers for async task processing
+- `api/` – FastAPI application (routes, services, storage helpers, MCP clients)
+- `mcp_servers/` – Stand-alone MCP servers:
+  - `rag_server.py` / `main.py` – pgvector-backed retrieval over tenant documents
+  - `web_server.py` – DuckDuckGo-powered search with English bias
+  - `admin_server.py` – Governance utilities (regex rules, violation logging, tenant registry)
+- `workers/` – Celery workers and schedulers for async ingestion + analytics maintenance
 
-## Setup
+## Prerequisites
 
-For full backend setup instructions, refer to the main project README.md in the root directory.
+- Python 3.10+
+- PostgreSQL (with the `vector` extension) for RAG data, or Supabase with pgvector enabled
+- SQLite (auto-created in `data/`) for analytics and admin rules
+- Optional: Ollama running locally (default) or Groq API credentials for remote LLMs
 
-## API Endpoints
+Create a virtual environment at the repo root, then:
 
-The main API endpoint used by the Gradio interface:
+```bash
+pip install -r requirements.txt
+cp env.example .env   # update MCP URLs + LLM settings
+```
 
-- `POST /agent/chat` - Chat with the MCP agent
-  - Headers: `x-tenant-id: <tenant-id>`
-  - Body: `{ "message": "<text>" }`
-  - Response: `{ "response": "<agent-response>" }`
+## Running the Services Locally
 
-## Note
+1. **FastAPI core**  
+   ```bash
+   uvicorn backend.api.main:app --port 8000 --reload
+   ```
 
-This Hugging Face Space submission includes only placeholder files for the backend structure.
-The full backend codebase exists in the main IntegraChat repository.
+2. **MCP servers** (each in its own shell):
+   ```bash
+   # RAG / knowledge base
+   python -m backend.mcp_servers.main        # or python -m backend.mcp_servers.rag_server
+
+   # Web search
+   python -m backend.mcp_servers.web_server
+
+   # Admin / governance
+   python -m backend.mcp_servers.admin_server
+   ```
+
+3. **Optional workers** (if running Celery-based ingestion/analytics jobs):
+   ```bash
+   celery -A backend.workers.ingestion_worker worker --loglevel=info
+   celery -A backend.workers.analytics_worker worker --loglevel=info
+   ```
+
+The Gradio UI (`python app.py`) and the Next.js operator console (see `frontend/README.md`) both talk to the FastAPI layer at `http://localhost:8000`.
+
+## Key Endpoints
+
+All endpoints require the `x-tenant-id` header unless otherwise noted.
+
+| Service | Path | Notes |
+| --- | --- | --- |
+| Agent | `POST /agent/message` | Autonomous orchestration (RAG/Web/Admin/LLM) |
+| Agent Debug | `POST /agent/debug` | Full reasoning trace + tool plan |
+| Agent Plan | `POST /agent/plan` | Dry-run planning without executing tools |
+| RAG | `POST /rag/ingest-document` | Rich ingestion (text, URL, metadata) |
+| RAG | `GET /rag/list` | Paginated document listing per tenant |
+| Admin | `POST /admin/rules` | Regex + severity rule ingestion |
+| Analytics | `GET /analytics/overview` | Summary metrics (queries, tokens, red flags) |
+
+Refer to the root `README.md` for the complete endpoint tables.
+
+## Diagnostics & Tenant Isolation
+
+Use the helper scripts in the repo root when validating backend changes:
+
+- `python verify_tenant_isolation.py` – Exercises analytics logging, admin rule CRUD, API reachability, and proves RAG tenant isolation by ingesting + querying as multiple tenants.
+- `python check_rag_database.py` – Talks directly to the pgvector database to list tenant IDs, preview stored chunks, and run safeguarded searches via `search_vectors()`. Helpful when troubleshooting suspected cross-tenant leakage.
+- `python test_manual.py` – Legacy manual smoke test harness (analytics store, admin rules, API surface).
+
+> **Troubleshooting tip:** If the isolation script reports a failure, first run `check_rag_database.py` to confirm documents are tagged with the correct `tenant_id`, then restart the RAG MCP server so it reloads the updated SQL filtering logic.
+
+## Environment Variables (excerpt)
+
+Defined in `env.example`:
+
+- `RAG_MCP_URL`, `WEB_MCP_URL`, `ADMIN_MCP_URL`
+- `OLLAMA_URL`, `OLLAMA_MODEL` (or `GROQ_API_KEY` + `LLM_BACKEND=groq`)
+- `SUPABASE_URL`, `SUPABASE_SERVICE_KEY` (optional admin integrations)
+- `APP_ENV`, `LOG_LEVEL`, `API_PORT`
+
+Update these before starting the servers to ensure the agent can reach every MCP endpoint and LLM runtime.