update the readme files

LICENSE

@@ -13,7 +13,7 @@ The above copyright notice and this permission notice shall be included in all
 copies or substantial portions of the Software.
 
 THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,got 
 FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
 AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
 LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,

README.md

@@ -22,7 +22,15 @@ This platform showcases how MCP can power intelligent, governed, multi-tenant AI
 - 📚 **Enhanced Knowledge Base Management** – Upload raw text, URLs, or documents (PDF/DOCX/TXT/MD) with rich metadata (source URL, timestamp, document type) and optimized chunking (400-600 tokens)
 - 🔍 **Optimized RAG Search** – Semantic search with configurable similarity threshold (default 0.3) for better recall, with fallback to return top results even if below threshold
 - 🗑️ **Document Management** – Delete individual documents or bulk delete all documents for a tenant with confirmation dialogs
-- 🛡️ **Enterprise Admin Governance** – Regex-based red-flag pattern matching with severity levels (low/medium/high/critical) and automatic admin alerts
+- 🛡️ **Enterprise Admin Governance** – Advanced rule management system with:
+  - Regex-based red-flag pattern matching with severity levels (low/medium/high/critical)
+  - Automatic admin alerts for violations
+  - **LLM-Enhanced Rules**: Rules are automatically analyzed and enhanced to identify edge cases, improve regex patterns, and suggest appropriate severity levels
+  - **File Upload Support**: Upload rules from TXT, PDF, DOC, or DOCX files with drag-and-drop interface
+  - **Chunk Processing**: Large rule sets processed in manageable chunks (5 rules at a time) to prevent timeouts
+  - **Rule-Based Behavior Control**: Rules checked FIRST - brief response rules return quick answers, blocking rules prevent requests
+  - **Comment Filtering**: Comment lines (starting with #) automatically ignored when uploading rules
+  - **Supabase Integration**: Rules stored in Supabase for production scalability (with SQLite fallback)
 - 📊 **Comprehensive Analytics & Observability** – Full tenant-level analytics logging with SQLite backend:
   - Tool usage breakdown (RAG, Web, Admin, LLM) with latency and token tracking
   - RAG recall/precision indicators (average hits, scores, top scores)
@@ -33,14 +41,20 @@ This platform showcases how MCP can power intelligent, governed, multi-tenant AI
 - 🏢 **Multi-Tenant Isolation** – Complete tenant isolation with centralized tenant ID management; backend enforces strict isolation for chat, ingestion, and admin ops
 - 🔄 **Intelligent Multi-Tool Orchestration** – MCP agent orchestrator autonomously selects optimal tool chains (RAG + Web + LLM, etc.) based on query intent and context
 - ⚡ **Robust Error Handling** – Structured error responses, retry mechanisms, and graceful fallbacks (e.g., if RAG fails → fallback to LLM-only)
+- 📡 **Streaming Responses** – Chat responses stream word-by-word using Server-Sent Events (SSE) for real-time user experience
+- 🎯 **Rule-First Processing** – Admin rules checked before intent classification - rules can trigger brief responses or block requests entirely
 
 ### Enterprise Features
 
 - 🔍 **Regex-Based Red-Flag Detection** – Support for complex regex patterns with keyword fallback and semantic scoring
+- 🤖 **LLM-Enhanced Rule Management** – Rules automatically enhanced by LLM to identify edge cases, improve patterns, and suggest severity levels
+- 📄 **File Upload & Drag-and-Drop** – Upload rules from files (TXT, PDF, DOC, DOCX) with intuitive drag-and-drop interface
+- ⚡ **Chunk-Wise Processing** – Large rule sets processed in chunks to prevent timeouts and ensure reliable processing
 - 📈 **Real-Time Analytics Dashboard** – Per-tenant analytics with configurable time windows (7, 30, 90 days)
 - 🛠️ **Admin API Endpoints** – `/admin/violations`, `/admin/tools/logs`, `/admin/tenants` for comprehensive governance
 - 🧠 **Agent Debug & Planning** – `/agent/debug` and `/agent/plan` endpoints for observability and tool selection inspection
 - 💾 **Persistent Analytics Storage** – SQLite-based analytics store with indexes for fast queries
+- 🗄️ **Supabase Integration** – Production-ready Supabase support for admin rules with automatic table creation
 
 ---
 
@@ -97,7 +111,14 @@ The Gradio UI exposes four tabs once you launch `app.py`:
    - **Formatted Summary**: Key metrics displayed in an easy-to-read format
    - Click "🔄 Fetch Analytics Snapshot" to load the latest data
 
-5. **Admin Rules & Compliance** – upload/delete governance rules that are stored via the backend `/admin/rules` API.
+5. **Admin Rules & Compliance** – comprehensive rule management with:
+   - **Text Input**: Paste rules one per line (comment lines starting with # are automatically ignored)
+   - **File Upload**: Upload rules from TXT, PDF, DOC, or DOCX files with drag-and-drop support
+   - **LLM Enhancement**: Rules are automatically enhanced by LLM to identify edge cases, improve patterns, and suggest severity levels
+   - **Chunk Processing**: Large rule sets processed in chunks (5 rules at a time) to avoid timeouts
+   - **Rule-Based Behavior**: Rules are checked FIRST before normal processing - brief response rules (low severity) return quick responses, blocking rules (high severity) block requests
+   - **Streaming Responses**: Chat responses stream word-by-word for better user experience
+   - **Refresh Button**: Refresh rules directly from the Rule Set table
 
 **Tip:** Every action requires a tenant ID. The tenant ID is now managed centrally and persists across page refreshes. The Knowledge Base Library and Admin Analytics tabs feature beautiful, modern UI with dark theme styling and interactive Plotly visualizations.
 
@@ -137,6 +158,7 @@ Then open `http://localhost:3000`. The navbar links on the landing page route to
 | Purpose | Method & Path | Description |
 | --- | --- | --- |
 | Chat with agent | `POST /agent/message` | Main chat endpoint with `tenant_id`, `message`, optional history |
+| Chat with agent (streaming) | `POST /agent/message/stream` | Streams response word-by-word using Server-Sent Events (SSE). Returns status messages and tokens as they're generated |
 | Agent debug | `POST /agent/debug` | Returns detailed debugging info: reasoning trace, tool selection, intent classification |
 | Agent plan | `POST /agent/plan` | Returns tool selection plan without execution (intent, tool scores, planned steps) |
 
@@ -155,11 +177,14 @@ Then open `http://localhost:3000`. The navbar links on the landing page route to
 | Purpose | Method & Path | Description |
 | --- | --- | --- |
 | List rules | `GET /admin/rules?detailed=true` | Get all rules (use `detailed=true` for regex/severity metadata) |
-| Add rule | `POST /admin/rules` | Add rule with optional `pattern` (regex), `severity` (low/medium/high/critical), `description` |
+| Add rule | `POST /admin/rules?enhance=true` | Add single rule with optional `pattern` (regex), `severity` (low/medium/high/critical), `description`. Set `enhance=true` for LLM enhancement |
+| Add rules bulk | `POST /admin/rules/bulk?enhance=true` | Add multiple rules at once. Processed in chunks of 5 to avoid timeouts. LLM enhancement applied automatically |
+| Upload rules file | `POST /admin/rules/upload-file?enhance=true` | Upload rules from file (TXT, PDF, DOC, DOCX). Text extracted server-side, rules processed with LLM enhancement |
 | Delete rule | `DELETE /admin/rules/{rule}` | Delete a specific rule |
 | List violations | `GET /admin/violations?days=30&limit=50` | Get red-flag violations with timestamps and confidence scores |
 | Tool logs | `GET /admin/tools/logs?tool_name=rag&days=7` | Get detailed tool usage logs with latency and token counts |
 | Manage tenants | `GET/POST/DELETE /admin/tenants` | Tenant management endpoints (placeholder implementation) |
+| Setup Supabase table | `POST /admin/setup/table` | Create admin_rules table in Supabase if it doesn't exist |
 
 ### Analytics Endpoints
 
@@ -255,11 +280,31 @@ This project is licensed under the MIT License - see the [LICENSE](LICENSE) file
 - **Backend**: FastAPI with async/await for high-performance MCP orchestration
 - **Frontend**: Gradio interface with Plotly visualizations + Next.js operator console
 - **UI Libraries**: Plotly for interactive charts, Gradio for web interface
-- **LLM Integration**: Ollama (local) or Groq (cloud) via configurable backend
+- **LLM Integration**: Ollama (local) or Groq (cloud) via configurable backend with streaming support
 - **Vector Store**: pgvector (via Supabase) or SQLite embeddings
 - **Analytics**: SQLite with indexed queries for fast analytics
+- **Rules Storage**: Supabase (production) or SQLite (development) with automatic table creation
 - **MCP Server**: Unified MCP server (port 8900) exposing all tools via namespaces
 - **Database**: PostgreSQL with pgvector extension for RAG embeddings, SQLite for analytics
+- **File Processing**: Support for TXT, PDF, DOC, DOCX with server-side text extraction (PyPDF2, python-docx)
+- **Streaming**: Server-Sent Events (SSE) for real-time word-by-word response streaming
+
+## Recent Enhancements
+
+### Admin Rules System (Latest)
+- **File Upload Support**: Upload rules from TXT, PDF, DOC, DOCX files with drag-and-drop interface
+- **LLM Enhancement**: Automatic rule enhancement identifies edge cases, improves regex patterns, and suggests severity levels
+- **Chunk Processing**: Large rule sets processed in chunks of 5 to prevent timeouts (handles 100+ rules efficiently)
+- **Comment Filtering**: Comment lines (starting with #) automatically ignored when uploading rules
+- **Rule-First Processing**: Admin rules checked before intent classification - enables behavior control (brief responses vs blocking)
+- **Supabase Integration**: Production-ready Supabase support with automatic table creation
+- **Streaming Responses**: Word-by-word streaming for chat responses using Server-Sent Events (SSE)
+
+### UI Improvements
+- **Modern Drag-and-Drop**: Intuitive file upload with visual feedback
+- **Enhanced Status Messages**: Clear success/error messages with icons
+- **Refresh Button in Table**: Quick refresh directly from the Rule Set section
+- **Better Visual Hierarchy**: Improved spacing, colors, and layout
 
 ## Key Technical Features
 

RULES_EXAMPLES.md

@@ -4,9 +4,10 @@ This document provides examples of rules you can use with the IntegraChat admin
 
 ## Quick Start
 
-1. **Simple Rules** - Copy from `example_rules.txt` and paste into Gradio UI
-2. **Detailed Rules** - Use `example_rules_detailed.json` for rules with patterns and severity
-3. **API** - Use the `/admin/rules` or `/admin/rules/bulk` endpoints
+1. **Simple Rules** - Copy from `example_rules.txt` and paste into Gradio UI or Next.js frontend
+2. **File Upload** - Drag and drop or upload TXT, PDF, DOC, or DOCX files directly
+3. **Detailed Rules** - Use `example_rules_detailed.json` for rules with patterns and severity
+4. **API** - Use the `/admin/rules`, `/admin/rules/bulk`, or `/admin/rules/upload-file` endpoints
 
 ## Rule Categories
 
@@ -101,10 +102,20 @@ For more precise matching, you can specify regex patterns:
 1. Open the IntegraChat Gradio interface
 2. Go to "Admin Rules & Compliance" tab
 3. Enter your tenant ID
-4. Paste rules from `example_rules.txt` (one per line)
-5. Click "Upload / Append Rules"
+4. **Option A - Text Input**: Paste rules from `example_rules.txt` (one per line) and click "Upload / Append Rules"
+5. **Option B - File Upload**: Drag and drop or click to upload a TXT, PDF, DOC, or DOCX file containing rules
+6. Rules are automatically enhanced by LLM (identifies edge cases, improves patterns)
+7. Comment lines (starting with #) are automatically ignored
 
-### Method 2: Via API (Programmatic)
+### Method 2: Via Next.js Frontend
+
+1. Navigate to `/admin-rules` page
+2. Enter your tenant ID in the navbar
+3. **Text Input**: Paste rules in the text area and click "Upload / Append Rules"
+4. **File Upload**: Drag and drop files or click the drop zone to upload
+5. Click "Refresh Rules" to see your uploaded rules
+
+### Method 3: Via API (Programmatic)
 
 **Single Rule:**
 ```bash
@@ -121,7 +132,7 @@ curl -X POST http://localhost:8000/admin/rules \
 
 **Bulk Rules:**
 ```bash
-curl -X POST http://localhost:8000/admin/rules/bulk \
+curl -X POST "http://localhost:8000/admin/rules/bulk?enhance=true" \
   -H "Content-Type: application/json" \
   -H "x-tenant-id: your_tenant_id" \
   -d '{
@@ -133,7 +144,14 @@ curl -X POST http://localhost:8000/admin/rules/bulk \
   }'
 ```
 
-### Method 3: Using Python
+**File Upload:**
+```bash
+curl -X POST "http://localhost:8000/admin/rules/upload-file?enhance=true" \
+  -H "x-tenant-id: your_tenant_id" \
+  -F "file=@example_rules.txt"
+```
+
+### Method 4: Using Python
 
 ```python
 import requests
@@ -172,6 +190,10 @@ When you add rules, the LLM will automatically:
 - ✅ Improve regex patterns for better matching
 - ✅ Suggest appropriate severity levels
 - ✅ Write clear descriptions
+- ✅ Process rules in chunks (5 at a time) to avoid timeouts
+- ✅ Handle large rule sets efficiently
+
+**Note**: Enhancement can be disabled by setting `enhance=false` in the API query parameter, but it's enabled by default for better rule quality.
 
 **Example:**
 - **Input:** `Block password queries`
@@ -197,11 +219,15 @@ After adding rules, test them by asking questions that should be blocked:
 ## Best Practices
 
 1. **Start Simple** - Begin with basic rules, then add patterns
-2. **Test Thoroughly** - Test rules with various phrasings
-3. **Review Edge Cases** - Check if rules block legitimate queries
-4. **Use Appropriate Severity** - Match severity to risk level
-5. **Regular Updates** - Review and update rules periodically
-6. **Document Patterns** - Add descriptions explaining what each rule blocks
+2. **Use File Upload** - For large rule sets, upload from files instead of typing manually
+3. **Leverage LLM Enhancement** - Let the system enhance your rules automatically
+4. **Test Thoroughly** - Test rules with various phrasings
+5. **Review Edge Cases** - Check if rules block legitimate queries
+6. **Use Appropriate Severity** - Match severity to risk level (low for brief responses, high for blocking)
+7. **Comment Lines** - Use `#` for comments in rule files - they're automatically ignored
+8. **Regular Updates** - Review and update rules periodically
+9. **Document Patterns** - Add descriptions explaining what each rule blocks
+10. **Chunk Processing** - Large uploads are automatically chunked - be patient for 20+ rules
 
 ## Common Patterns