--- title: CX Bot emoji: πŸ€– colorFrom: purple colorTo: blue sdk: docker pinned: false --- # Domain-Aware Multi-Agent CX Bot A local, fully self-hosted customer experience platform that routes incoming support queries to domain-specific RAG agents (billing, returns, escalation), grounds every answer in retrieved policy documents, scores its own confidence, and logs every interaction for auditing and continuous evaluation. Everything runs on your own machine β€” local LLM via Ollama, local vector index via FAISS, local cache/session store via Redis. No external API keys required. ## Why this exists Most "RAG chatbot" demos stop at retrieve β†’ generate. This project adds the pieces that make a support bot actually trustworthy and operable in production: - **Routing instead of one giant prompt** β€” a lightweight classifier sends each query to a specialized agent with its own system prompt and document set. - **Hybrid retrieval** β€” BM25 keyword search and FAISS semantic search are fused, so exact policy terms (e.g. "Section 5") aren't lost to pure embedding similarity. - **Graph-based multi-hop expansion** β€” chunks that cross-reference each other ("see Section 3", "as per the returns policy") are linked in a graph so the agent can follow that reference automatically instead of missing it. - **Confidence isn't a single number from the LLM** β€” it's a weighted fusion of router confidence, retrieval relevance, and an LLM self-rating, so low-confidence answers can be reliably flagged for escalation. - **Every query is audited** β€” full request/response metadata is logged to disk as JSONL, which a nightly evaluator and prompt optimizer both read from. - **PII is redacted before it touches the LLM or the logs**, with a regex fallback if Presidio/spaCy aren't installed. ## Stack LangChain Β· FAISS Β· HuggingFace BGE embeddings Β· BM25 Β· NetworkX (graph RAG) Β· Ollama (Llama 3.2) Β· FastAPI Β· Redis Β· Presidio (PII redaction) ## Architecture ``` User Query β”‚ β–Ό [PII Redactor] (Presidio, regex fallback) β”‚ β–Ό [Redis Semantic Cache] ──HIT──→ Cached Response β”‚ MISS β–Ό [Session Memory] (Redis, last N turns) β”‚ β–Ό [Router Agent] (Llama 3.2 zero-shot classification) β”œβ”€β”€β†’ BillingAgent β”œβ”€β”€β†’ ReturnsAgent └──→ EscalationAgent β”‚ β–Ό [Hybrid Retriever] BM25 + FAISS/BGE, score-fused β”‚ β–Ό [Graph RAG] NetworkX multi-hop expansion (cross-references) β”‚ β–Ό [Llama 3.2 via Ollama] grounded generation β”‚ β–Ό [Confidence Scorer] router + retrieval + LLM self-rating β”‚ β–Ό [Audit Logger] β†’ logs/audit_.jsonl β”‚ β–Ό Structured Response ``` ## Project structure ``` cx_bot/ β”œβ”€β”€ api/main.py # FastAPI app β€” all HTTP endpoints β”œβ”€β”€ agents/ β”‚ β”œβ”€β”€ base_agent.py # Shared retrieve β†’ expand β†’ generate β†’ score pipeline β”‚ β”œβ”€β”€ billing_agent.py # Billing domain system prompt β”‚ β”œβ”€β”€ returns_agent.py # Returns domain system prompt β”‚ β”œβ”€β”€ escalation_agent.py # Escalation domain system prompt β”‚ └── router.py # LLM-based query classifier β”œβ”€β”€ rag/ β”‚ β”œβ”€β”€ indexer.py # Loads docs, chunks, embeds, builds/saves FAISS indexes β”‚ β”œβ”€β”€ retriever.py # Hybrid BM25 + FAISS retrieval with score fusion β”‚ β”œβ”€β”€ bm25_retriever.py # BM25 keyword search per domain β”‚ β”œβ”€β”€ graph_rag.py # Builds cross-reference graph, multi-hop expansion β”‚ └── embeddings.py # Embedding model loader (HuggingFace BGE) β”œβ”€β”€ pipeline/ β”‚ β”œβ”€β”€ pii_redactor.py # Presidio-based PII redaction, regex fallback β”‚ └── confidence_scorer.py # Fuses router/retrieval/LLM signals into one score β”œβ”€β”€ cache/redis_cache.py # Semantic response cache (cosine similarity over embeddings) β”œβ”€β”€ memory/session_store.py # Per-session conversation history (Redis, TTL-bound) β”œβ”€β”€ audit/logger.py # Append-only JSONL audit logging + read-back β”œβ”€β”€ eval/ β”‚ β”œβ”€β”€ auto_eval.py # Nightly benchmark: samples audit logs, computes metrics β”‚ └── prompt_optimizer.py # Tests prompt variants against low-confidence queries β”œβ”€β”€ documents/ # Source-of-truth policy docs, by domain β”‚ β”œβ”€β”€ billing/ β”‚ β”œβ”€β”€ returns/ β”‚ └── escalation/ β”œβ”€β”€ faiss_indexes/ # Generated β€” FAISS indexes + chunk pickles + graph cache β”œβ”€β”€ logs/ # Generated β€” daily audit logs, benchmark/optimizer reports β”œβ”€β”€ config.py # All tunables, loaded from .env β”œβ”€β”€ docker-compose.yml # Redis service β”œβ”€β”€ run.sh # One-shot startup script └── test_client.py # Manual smoke-test script ``` ## Prerequisites - Python 3.10+ - [Ollama](https://ollama.com) installed and on PATH - Docker (for Redis) β€” or a Redis instance you point `config.py` at - ~4GB free disk for the Llama 3.2 model + BGE embedding model on first run ## Quick Start ### 1. Set up the environment ```bash python -m venv venv source venv/bin/activate # Windows: venv\Scripts\activate pip install -r requirements.txt cp .env.example .env # adjust values if needed ``` ### 2. Pull and serve the Ollama model ```bash ollama pull llama3.2 ollama serve ``` ### 3. Start Redis ```bash docker-compose up -d redis ``` ### 4. Build document indexes ```bash python rag/indexer.py --domain all ``` ### 5. Start the API ```bash uvicorn api.main:app --host 0.0.0.0 --port 8000 --reload ``` Or, on Linux/macOS, do steps 2–5 in one go: ```bash chmod +x run.sh && ./run.sh ``` Once running, interactive API docs are available at `http://localhost:8000/docs`. ## Configuration All settings live in `.env` (see `.env.example` for the full template) and are loaded through `config.py`: | Variable | Default | Description | |---|---|---| | `OLLAMA_BASE_URL` | `http://localhost:11434` | Ollama server address | | `OLLAMA_MODEL` | `llama3.2` | Model used for routing and generation | | `REDIS_HOST` / `REDIS_PORT` / `REDIS_DB` | `localhost` / `6379` / `0` | Redis connection for cache + session memory | | `EMBED_MODEL` | `BAAI/bge-base-en-v1.5` | HuggingFace embedding model for FAISS | | `FAISS_INDEX_DIR` | `faiss_indexes` | Where indexes/chunks/graphs are persisted | | `DOCS_DIR` | `documents` | Root folder for domain policy documents | | `AUDIT_LOG_DIR` | `logs` | Where audit logs and reports are written | | `CONFIDENCE_THRESHOLD` | `0.65` | Below this, a response is flagged `low_confidence` | | `CACHE_SIM_THRESHOLD` | `0.92` | Minimum cosine similarity for a cache hit | | `CACHE_TTL_SECONDS` | `3600` | Cache entry lifetime | | `SESSION_TTL_SECONDS` | `1800` | Session memory lifetime | | `SESSION_MAX_TURNS` | `10` | Conversation turns retained per session | | `TOP_K_RETRIEVAL` | `5` | Chunks retrieved per query | | `BM25_WEIGHT` / `VECTOR_WEIGHT` | `0.4` / `0.6` | Fusion weights for hybrid retrieval | ## API Endpoints | Method | Endpoint | Description | |--------|----------|-------------| | POST | `/query` | Submit a customer query, get a routed, grounded, scored response | | GET | `/audit` | View audit log records (`?date=YYYY-MM-DD&limit=100`) | | POST | `/index` | Re-index all domain documents (runs in the background) | | DELETE | `/session/{id}` | Clear a session's conversation memory | | GET | `/health` | Health check; lists active agents | ### Example request ```bash curl -X POST http://localhost:8000/query \ -H "Content-Type: application/json" \ -d '{"session_id": "user_001", "query": "Why was I charged twice?"}' ``` ### Example response ```json { "answer": "Based on our billing policy (Section 5)...", "agent": "billing", "sources": ["billing_policy.txt"], "confidence": 0.87, "low_confidence": false, "disclaimer": "This response is based on current billing policies...", "audit_id": "uuid-here", "latency_ms": 1240, "cache_hit": false } ``` ## Adding domain documents Drop `.txt` or `.md` files into: - `documents/billing/` β€” invoices, billing policies - `documents/returns/` β€” return/exchange policies - `documents/escalation/` β€” SLA documents, escalation procedures Then re-index just that domain (or all of them): ```bash python rag/indexer.py --domain billing # or rebuild everything via the API (clears in-memory FAISS/BM25 caches too): curl -X POST http://localhost:8000/index ``` To enable cross-document reference following, phrase relationships explicitly in your docs (e.g. *"see Section 3"*, *"as per the returns policy"*) β€” `graph_rag.py` detects these patterns to link chunks for multi-hop retrieval. ## Evaluation & continuous improvement **Benchmark** β€” samples recent audit logs (across the last 7 days by default) and checks them against target thresholds: ```bash python eval/auto_eval.py --mode benchmark --samples 100 ``` | Metric | Target | |---|---| | Cache hit rate | β‰₯ 40% | | Low-confidence rate | ≀ 15% | | Avg latency | < 1500ms | | p95 latency | < 3500ms | | Avg confidence | β‰₯ 0.80 | **Prompt optimizer** β€” pulls queries that were flagged `low_confidence`, tests three system-prompt variants (precise / empathetic / structured) against them, and writes the best-performing variant to `logs/prompt_optimization_latest.json`: ```bash python eval/prompt_optimizer.py ``` ### Nightly cron (optional) ```cron 0 2 * * * cd /path/to/cx_bot && python eval/auto_eval.py >> logs/eval.log 2>&1 0 3 * * * cd /path/to/cx_bot && python eval/prompt_optimizer.py >> logs/optimizer.log 2>&1 ``` ## Troubleshooting - **Audit log line counts don't match a benchmark's `total_queries`** β€” the benchmark sums entries across the last 7 daily `audit_.jsonl` files, not just today's. Check all files with: ```powershell Get-ChildItem logs\*.jsonl | ForEach-Object { "$($_.Name): $((Get-Content $_.FullName).Count) lines" } ``` - **Presidio/spaCy not installed** β€” PII redaction silently falls back to regex patterns (email, phone, credit card, SSN). Check logs for a "Presidio not available" warning. - **Redis unreachable** β€” caching and session memory fail open (queries still work, just without caching/memory); check for "Redis unavailable" debug logs. - **First query after a fresh clone is slow** β€” indexes are built lazily on first access if missing. Run `python rag/indexer.py --domain all` ahead of time to avoid this on a live request. ## License MIT β€” see [LICENSE](LICENSE).