update the readme file

backend/README.md

@@ -49,7 +49,7 @@ cp env.example .env   # update MCP URLs + LLM settings
    **HTTP Endpoints** (for direct API access):
    - `GET /rag/list?tenant_id={id}&limit={n}&offset={n}` - List documents
    - `POST /rag/ingest` - Ingest content
-   - `POST /rag/search` - Search documents
+   - `POST /rag/search` - Search documents (supports `threshold` parameter, default: 0.3)
    - `DELETE /rag/delete/{document_id}?tenant_id={id}` - Delete specific document
    - `DELETE /rag/delete-all?tenant_id={id}` - Delete all documents
    - `POST /web/search` - Web search
@@ -109,6 +109,13 @@ Use the helper scripts in the repo root when validating backend changes:
 - RAG client automatically unwraps responses for seamless integration
 - Error responses include detailed messages for better debugging
 
+### RAG Search Enhancements
+- **Lowered default threshold** from 0.5 to 0.3 for improved recall of relevant documents
+- **Intelligent fallback mechanism** returns the top result even if similarity score is below threshold, ensuring knowledge base content is always accessible
+- **Configurable threshold** via `threshold` parameter in search requests (default: 0.3)
+- **Enhanced tool selection** automatically triggers RAG for admin questions, fact lookups ("who is", "what is"), and internal knowledge queries
+- **Response unwrapping** in MCP client ensures orchestrator receives properly formatted results for tool scoring and prompt building
+
 ## Environment Variables (excerpt)
 
 Defined in `env.example`:
@@ -148,4 +155,24 @@ Agents that speak the Model Context Protocol should connect to the `integrachat`
 - The MCP server automatically normalizes tenant IDs to ensure consistent matching across operations
 - All endpoints support both POST (with JSON payload) and direct HTTP methods (GET for list, DELETE for delete operations)
 - Tenant ID normalization handles whitespace and ensures documents can be listed and deleted consistently
+- RAG search uses a default threshold of 0.3 for better recall; adjust via `threshold` parameter if needed
+
+## Troubleshooting
+
+### RAG Search Not Returning Results
+- **Check similarity threshold**: The default threshold is 0.3. If results are still not found, try lowering it to 0.2 or 0.1
+- **Verify documents are ingested**: Use `GET /rag/list?tenant_id={id}` to confirm documents exist for the tenant
+- **Check tenant ID matching**: Ensure the tenant_id used for search matches the one used for ingestion (normalization handles whitespace automatically)
+- **Review search logs**: Check MCP server logs for search metrics (hits_count, avg_score, top_score)
+
+### Agent Not Using RAG for Knowledge Base Questions
+- **Verify RAG results are being found**: Check the agent debug endpoint (`POST /agent/debug`) to see if RAG results are being pre-fetched
+- **Check tool scores**: The debug output shows `rag_fitness` score; if it's low (< 0.4), the agent may skip RAG
+- **Ensure knowledge base content exists**: Questions like "who is the admin" require relevant content in the knowledge base
+- **Pattern matching**: The tool selector automatically triggers RAG for patterns like "admin", "who is", "what is", but semantic similarity also plays a role
+
+### Document Deletion Issues
+- **404 Not Found**: Verify the document_id exists and belongs to the correct tenant
+- **Tenant ID mismatch**: The system normalizes tenant IDs, but ensure you're using the same tenant_id format as when documents were ingested
+- **Check logs**: Database deletion logs show detailed information about tenant ID matching and document existence