update the Readme.md files

README.md

@@ -48,8 +48,10 @@ This platform showcases how MCP can power intelligent, governed, multi-tenant AI
 
 1. **Backend services running**:
    - FastAPI API (`uvicorn backend.api.main:app --port 8000`)
-   - MCP servers (RAG 8001, Web 8002, Admin 8003) as described in `backend/README.md`
+   - Unified MCP server (port 8900) as described in `backend/README.md`
    - Optional: Ollama / Groq credentials for the LLM client
+   
+   **Quick Start**: Run `start.bat` (Windows) to launch all services automatically.
 2. **Python 3.10+** with the dependencies in `requirements.txt`
 
 ### Installation
@@ -122,11 +124,11 @@ Then open `http://localhost:3000`. The navbar links on the landing page route to
 
 | Purpose | Method & Path | Description |
 | --- | --- | --- |
-| Ingest document | `POST /rag/ingest-document` | Accepts `source_type`, `content`, metadata (filename, URL, doc_id) |
+| Ingest document | `POST /rag/ingest-document` | Accepts `source_type`, `content`, metadata (filename, URL, doc_id). Supports raw text, URLs, PDFs, DOCX, TXT, and Markdown files |
 | Ingest file | `POST /rag/ingest-file` | Multipart upload with `x-tenant-id` header (PDF/DOCX/TXT/MD) |
-| List documents | `GET /rag/list` | Returns all documents for a tenant with pagination |
-| Delete document | `DELETE /rag/delete/{document_id}` | Deletes a specific document by ID |
-| Delete all documents | `DELETE /rag/delete-all` | Deletes all documents for a tenant |
+| List documents | `GET /rag/list?tenant_id={id}&limit={n}&offset={n}` | Returns all documents for a tenant with pagination. Requires `x-tenant-id` header or `tenant_id` query parameter |
+| Delete document | `DELETE /rag/delete/{document_id}?tenant_id={id}` | Deletes a specific document by ID. Requires `x-tenant-id` header or `tenant_id` query parameter |
+| Delete all documents | `DELETE /rag/delete-all?tenant_id={id}` | Deletes all documents for a tenant. Requires `x-tenant-id` header or `tenant_id` query parameter |
 
 ### Admin & Governance Endpoints
 
@@ -235,7 +237,22 @@ This project is licensed under the MIT License - see the [LICENSE](LICENSE) file
 - **LLM Integration**: Ollama (local) or Groq (cloud) via configurable backend
 - **Vector Store**: pgvector (via Supabase) or SQLite embeddings
 - **Analytics**: SQLite with indexed queries for fast analytics
-- **MCP Servers**: RAG (8001), Web (8002), Admin (8003)
+- **MCP Server**: Unified MCP server (port 8900) exposing all tools via namespaces
+- **Database**: PostgreSQL with pgvector extension for RAG embeddings, SQLite for analytics
+
+## Key Technical Features
+
+### Tenant Isolation & Normalization
+- **Strict tenant isolation** enforced at database level with `WHERE tenant_id = ...` filters
+- **Automatic tenant ID normalization** handles whitespace and formatting differences
+- Documents can be listed and deleted consistently across different tenant_id formats
+- All operations validate tenant ownership before execution
+
+### MCP Server Architecture
+- **Unified server** running on a single port (default 8900) for all namespaced tools
+- **Dual protocol support**: Both MCP protocol (POST with JSON) and RESTful HTTP (GET/DELETE)
+- **Response wrapping**: Standardized response format with automatic unwrapping in clients
+- **Error handling**: Comprehensive error responses with detailed messages for debugging
 
 ## Acknowledgments
 

backend/README.md

@@ -33,10 +33,27 @@ cp env.example .env   # update MCP URLs + LLM settings
    ```bash
    python backend/mcp_server/server.py
    ```
-   This single endpoint exposes the following namespaced tools:
-   - `rag.search`, `rag.ingest`, `rag.delete`
-   - `web.search`
+   Or use the provided startup script:
+   ```bash
+   start.bat  # Windows - launches MCP server on port 8900 and FastAPI on port 8000
+   ```
+   
+   This single server (default port 8900) exposes the following namespaced tools:
+   - `rag.search` - Semantic search across tenant documents
+   - `rag.ingest` - Ingest text content into knowledge base
+   - `rag.delete` - Delete individual or all documents for a tenant
+   - `rag.list` - List all documents for a tenant with pagination
+   - `web.search` - DuckDuckGo-based web search
    - `admin.getRules`, `admin.addRule`, `admin.deleteRule`, `admin.logViolation`
+   
+   **HTTP Endpoints** (for direct API access):
+   - `GET /rag/list?tenant_id={id}&limit={n}&offset={n}` - List documents
+   - `POST /rag/ingest` - Ingest content
+   - `POST /rag/search` - Search documents
+   - `DELETE /rag/delete/{document_id}?tenant_id={id}` - Delete specific document
+   - `DELETE /rag/delete-all?tenant_id={id}` - Delete all documents
+   - `POST /web/search` - Web search
+   - `POST /admin/*` - Admin operations
 
 3. **Optional workers** (if running Celery-based ingestion/analytics jobs):
    ```bash
@@ -56,7 +73,10 @@ All endpoints require the `x-tenant-id` header unless otherwise noted.
 | Agent Debug | `POST /agent/debug` | Full reasoning trace + tool plan |
 | Agent Plan | `POST /agent/plan` | Dry-run planning without executing tools |
 | RAG | `POST /rag/ingest-document` | Rich ingestion (text, URL, metadata) |
-| RAG | `GET /rag/list` | Paginated document listing per tenant |
+| RAG | `POST /rag/ingest-file` | File upload (PDF/DOCX/TXT/MD) |
+| RAG | `GET /rag/list` | Paginated document listing per tenant (requires `x-tenant-id` header) |
+| RAG | `DELETE /rag/delete/{document_id}` | Delete specific document (requires `x-tenant-id` header) |
+| RAG | `DELETE /rag/delete-all` | Delete all documents for tenant (requires `x-tenant-id` header) |
 | Admin | `POST /admin/rules` | Regex + severity rule ingestion |
 | Analytics | `GET /analytics/overview` | Summary metrics (queries, tokens, red flags) |
 
@@ -72,31 +92,60 @@ Use the helper scripts in the repo root when validating backend changes:
 
 > **Troubleshooting tip:** If the isolation script reports a failure, first run `check_rag_database.py` to confirm documents are tagged with the correct `tenant_id`, then restart the unified MCP server so it reloads the updated SQL filtering logic.
 
+## Recent Improvements
+
+### Tenant ID Normalization
+- All database operations now normalize tenant IDs to handle whitespace and formatting differences
+- Documents can be listed and deleted consistently even if stored with slightly different tenant_id formatting
+- The system automatically matches tenant IDs after normalization, ensuring operations work across different input formats
+
+### HTTP Endpoint Support
+- Added GET support for `/rag/list` endpoint (previously POST-only)
+- Added DELETE support for `/rag/delete/{document_id}` and `/rag/delete-all` endpoints
+- All endpoints support both MCP protocol (POST with JSON payload) and direct HTTP methods (GET/DELETE with query parameters)
+
+### Response Format
+- MCP server responses are wrapped in a standard format with `status`, `data`, and `metadata` fields
+- RAG client automatically unwraps responses for seamless integration
+- Error responses include detailed messages for better debugging
+
 ## Environment Variables (excerpt)
 
 Defined in `env.example`:
 
-- `RAG_MCP_URL`, `WEB_MCP_URL`, `ADMIN_MCP_URL`
+- `RAG_MCP_URL` - Default: `http://localhost:8900/rag` (unified MCP server)
+- `WEB_MCP_URL` - Default: `http://localhost:8900/web` (unified MCP server)
+- `ADMIN_MCP_URL` - Default: `http://localhost:8900/admin` (unified MCP server)
+- `MCP_PORT` - Port for unified MCP server (default: 8900)
+- `MCP_HOST` - Host for unified MCP server (default: 0.0.0.0)
+- `POSTGRESQL_URL` - PostgreSQL connection string with pgvector extension
 - `OLLAMA_URL`, `OLLAMA_MODEL` (or `GROQ_API_KEY` + `LLM_BACKEND=groq`)
 - `SUPABASE_URL`, `SUPABASE_SERVICE_KEY` (optional admin integrations)
 - `APP_ENV`, `LOG_LEVEL`, `API_PORT`
 
 Update these before starting the servers to ensure the agent can reach every MCP endpoint and LLM runtime.
 
+**Note**: The unified MCP server runs on a single port (default 8900) and handles all namespaced tools. The `start.bat` script automatically configures the correct URLs.
+
 ## Unified MCP tool instructions
 
 Agents that speak the Model Context Protocol should connect to the `integrachat` server id defined in `backend/mcp_server/server.py` and call the namespaced tools directly:
 
-| Namespace | Tool | Purpose |
-| --- | --- | --- |
-| `rag` | `search` | Retrieve tenant-scoped document chunks |
-| `rag` | `ingest` | Chunk + store new knowledge |
-| `rag` | `delete` | Remove one/all stored documents |
-| `web` | `search` | DuckDuckGo English-biased search |
-| `admin` | `getRules` | Fetch tenant governance rules (list or detailed) |
-| `admin` | `addRule` | Insert or update a rule |
-| `admin` | `deleteRule` | Remove a rule by text |
-| `admin` | `logViolation` | Persist a red-flag event into analytics |
-
-Always send `tenant_id`, and optionally `user_id`, in the payload so the shared middleware can enforce isolation and log analytics.
+| Namespace | Tool | Purpose | HTTP Endpoint |
+| --- | --- | --- | --- |
+| `rag` | `search` | Retrieve tenant-scoped document chunks | `POST /rag/search` |
+| `rag` | `ingest` | Chunk + store new knowledge | `POST /rag/ingest` |
+| `rag` | `list` | List all documents for tenant | `GET /rag/list?tenant_id={id}` |
+| `rag` | `delete` | Remove one/all stored documents | `DELETE /rag/delete/{id}?tenant_id={id}` or `DELETE /rag/delete-all?tenant_id={id}` |
+| `web` | `search` | DuckDuckGo English-biased search | `POST /web/search` |
+| `admin` | `getRules` | Fetch tenant governance rules (list or detailed) | `POST /admin/getRules` |
+| `admin` | `addRule` | Insert or update a rule | `POST /admin/addRule` |
+| `admin` | `deleteRule` | Remove a rule by text | `POST /admin/deleteRule` |
+| `admin` | `logViolation` | Persist a red-flag event into analytics | `POST /admin/logViolation` |
+
+**Important Notes:**
+- Always send `tenant_id` in the payload (or as query parameter for GET/DELETE requests) so the shared middleware can enforce isolation and log analytics
+- The MCP server automatically normalizes tenant IDs to ensure consistent matching across operations
+- All endpoints support both POST (with JSON payload) and direct HTTP methods (GET for list, DELETE for delete operations)
+- Tenant ID normalization handles whitespace and ensures documents can be listed and deleted consistently
 

frontend/README.md

@@ -39,14 +39,19 @@ NEXT_PUBLIC_API_URL=http://localhost:8000
 - **Ingestion card** for quick document uploads
 
 ### Knowledge Base Page (`/knowledge-base`)
-- **Document listing** with pagination and filtering
+- **Document listing** with pagination and filtering by type (text, PDF, FAQ, link)
 - **Search interface** for semantic search across documents
 - **Document ingestion** with support for:
   - Raw text input
+  - URL ingestion (automatic content fetching)
   - PDF file uploads
   - DOCX file uploads
-  - TXT file uploads
-- **Document management** with tenant isolation
+  - TXT and Markdown file uploads
+- **Document management** with tenant isolation:
+  - Delete individual documents by ID
+  - Delete all documents for a tenant (with confirmation)
+  - Real-time document list updates after operations
+  - Error handling with clear user feedback
 
 ### Components