README.md

metadata

title: CX Bot
emoji: 🤖
colorFrom: purple
colorTo: blue
sdk: docker
pinned: false

Domain-Aware Multi-Agent CX Bot

A local, fully self-hosted customer experience platform that routes incoming support queries to domain-specific RAG agents (billing, returns, escalation), grounds every answer in retrieved policy documents, scores its own confidence, and logs every interaction for auditing and continuous evaluation.

Everything runs on your own machine — local LLM via Ollama, local vector index via FAISS, local cache/session store via Redis. No external API keys required.

Why this exists

Most "RAG chatbot" demos stop at retrieve → generate. This project adds the pieces that make a support bot actually trustworthy and operable in production:

• Routing instead of one giant prompt — a lightweight classifier sends each query to a specialized agent with its own system prompt and document set.
• Hybrid retrieval — BM25 keyword search and FAISS semantic search are fused, so exact policy terms (e.g. "Section 5") aren't lost to pure embedding similarity.
• Graph-based multi-hop expansion — chunks that cross-reference each other ("see Section 3", "as per the returns policy") are linked in a graph so the agent can follow that reference automatically instead of missing it.
• Confidence isn't a single number from the LLM — it's a weighted fusion of router confidence, retrieval relevance, and an LLM self-rating, so low-confidence answers can be reliably flagged for escalation.
• Every query is audited — full request/response metadata is logged to disk as JSONL, which a nightly evaluator and prompt optimizer both read from.
• PII is redacted before it touches the LLM or the logs, with a regex fallback if Presidio/spaCy aren't installed.

Stack

LangChain · FAISS · HuggingFace BGE embeddings · BM25 · NetworkX (graph RAG) · Ollama (Llama 3.2) · FastAPI · Redis · Presidio (PII redaction)

Architecture

User Query
    │
    ▼
[PII Redactor] (Presidio, regex fallback)
    │
    ▼
[Redis Semantic Cache] ──HIT──→ Cached Response
    │ MISS
    ▼
[Session Memory] (Redis, last N turns)
    │
    ▼
[Router Agent] (Llama 3.2 zero-shot classification)
    ├──→ BillingAgent
    ├──→ ReturnsAgent
    └──→ EscalationAgent
              │
              ▼
    [Hybrid Retriever] BM25 + FAISS/BGE, score-fused
              │
              ▼
    [Graph RAG] NetworkX multi-hop expansion (cross-references)
              │
              ▼
    [Llama 3.2 via Ollama] grounded generation
              │
              ▼
    [Confidence Scorer] router + retrieval + LLM self-rating
              │
              ▼
    [Audit Logger] → logs/audit_<date>.jsonl
              │
              ▼
    Structured Response

Project structure

cx_bot/
├── api/main.py              # FastAPI app — all HTTP endpoints
├── agents/
│   ├── base_agent.py        # Shared retrieve → expand → generate → score pipeline
│   ├── billing_agent.py     # Billing domain system prompt
│   ├── returns_agent.py     # Returns domain system prompt
│   ├── escalation_agent.py  # Escalation domain system prompt
│   └── router.py            # LLM-based query classifier
├── rag/
│   ├── indexer.py           # Loads docs, chunks, embeds, builds/saves FAISS indexes
│   ├── retriever.py         # Hybrid BM25 + FAISS retrieval with score fusion
│   ├── bm25_retriever.py    # BM25 keyword search per domain
│   ├── graph_rag.py         # Builds cross-reference graph, multi-hop expansion
│   └── embeddings.py        # Embedding model loader (HuggingFace BGE)
├── pipeline/
│   ├── pii_redactor.py      # Presidio-based PII redaction, regex fallback
│   └── confidence_scorer.py # Fuses router/retrieval/LLM signals into one score
├── cache/redis_cache.py     # Semantic response cache (cosine similarity over embeddings)
├── memory/session_store.py  # Per-session conversation history (Redis, TTL-bound)
├── audit/logger.py          # Append-only JSONL audit logging + read-back
├── eval/
│   ├── auto_eval.py         # Nightly benchmark: samples audit logs, computes metrics
│   └── prompt_optimizer.py  # Tests prompt variants against low-confidence queries
├── documents/                # Source-of-truth policy docs, by domain
│   ├── billing/
│   ├── returns/
│   └── escalation/
├── faiss_indexes/            # Generated — FAISS indexes + chunk pickles + graph cache
├── logs/                     # Generated — daily audit logs, benchmark/optimizer reports
├── config.py                 # All tunables, loaded from .env
├── docker-compose.yml        # Redis service
├── run.sh                    # One-shot startup script
└── test_client.py            # Manual smoke-test script

Prerequisites

• Python 3.10+
• Ollama installed and on PATH
• Docker (for Redis) — or a Redis instance you point config.py at
• ~4GB free disk for the Llama 3.2 model + BGE embedding model on first run

Quick Start

1. Set up the environment

python -m venv venv
source venv/bin/activate        # Windows: venv\Scripts\activate
pip install -r requirements.txt
cp .env.example .env            # adjust values if needed

2. Pull and serve the Ollama model

ollama pull llama3.2
ollama serve

3. Start Redis

docker-compose up -d redis

4. Build document indexes

python rag/indexer.py --domain all

5. Start the API

uvicorn api.main:app --host 0.0.0.0 --port 8000 --reload

Or, on Linux/macOS, do steps 2–5 in one go:

chmod +x run.sh && ./run.sh

Once running, interactive API docs are available at http://localhost:8000/docs.

Configuration

All settings live in .env (see .env.example for the full template) and are loaded through config.py:

Variable	Default	Description
OLLAMA_BASE_URL	http://localhost:11434	Ollama server address
OLLAMA_MODEL	llama3.2	Model used for routing and generation
REDIS_HOST / REDIS_PORT / REDIS_DB	localhost / 6379 / 0	Redis connection for cache + session memory
EMBED_MODEL	BAAI/bge-base-en-v1.5	HuggingFace embedding model for FAISS
FAISS_INDEX_DIR	faiss_indexes	Where indexes/chunks/graphs are persisted
DOCS_DIR	documents	Root folder for domain policy documents
AUDIT_LOG_DIR	logs	Where audit logs and reports are written
CONFIDENCE_THRESHOLD	0.65	Below this, a response is flagged low_confidence
CACHE_SIM_THRESHOLD	0.92	Minimum cosine similarity for a cache hit
CACHE_TTL_SECONDS	3600	Cache entry lifetime
SESSION_TTL_SECONDS	1800	Session memory lifetime
SESSION_MAX_TURNS	10	Conversation turns retained per session
TOP_K_RETRIEVAL	5	Chunks retrieved per query
BM25_WEIGHT / VECTOR_WEIGHT	0.4 / 0.6	Fusion weights for hybrid retrieval

API Endpoints

Method	Endpoint	Description
POST	/query	Submit a customer query, get a routed, grounded, scored response
GET	/audit	View audit log records (?date=YYYY-MM-DD&limit=100)
POST	/index	Re-index all domain documents (runs in the background)
DELETE	/session/{id}	Clear a session's conversation memory
GET	/health	Health check; lists active agents

Example request

curl -X POST http://localhost:8000/query \
  -H "Content-Type: application/json" \
  -d '{"session_id": "user_001", "query": "Why was I charged twice?"}'

Example response

{
  "answer": "Based on our billing policy (Section 5)...",
  "agent": "billing",
  "sources": ["billing_policy.txt"],
  "confidence": 0.87,
  "low_confidence": false,
  "disclaimer": "This response is based on current billing policies...",
  "audit_id": "uuid-here",
  "latency_ms": 1240,
  "cache_hit": false
}

Adding domain documents

Drop .txt or .md files into:

• documents/billing/ — invoices, billing policies
• documents/returns/ — return/exchange policies
• documents/escalation/ — SLA documents, escalation procedures

Then re-index just that domain (or all of them):

python rag/indexer.py --domain billing
# or rebuild everything via the API (clears in-memory FAISS/BM25 caches too):
curl -X POST http://localhost:8000/index

To enable cross-document reference following, phrase relationships explicitly in your docs (e.g. "see Section 3", "as per the returns policy") — graph_rag.py detects these patterns to link chunks for multi-hop retrieval.

Evaluation & continuous improvement

Benchmark — samples recent audit logs (across the last 7 days by default) and checks them against target thresholds:

python eval/auto_eval.py --mode benchmark --samples 100

Metric	Target
Cache hit rate	≥ 40%
Low-confidence rate	≤ 15%
Avg latency	< 1500ms
p95 latency	< 3500ms
Avg confidence	≥ 0.80

Prompt optimizer — pulls queries that were flagged low_confidence, tests three system-prompt variants (precise / empathetic / structured) against them, and writes the best-performing variant to logs/prompt_optimization_latest.json:

python eval/prompt_optimizer.py

Nightly cron (optional)

0 2 * * * cd /path/to/cx_bot && python eval/auto_eval.py >> logs/eval.log 2>&1
0 3 * * * cd /path/to/cx_bot && python eval/prompt_optimizer.py >> logs/optimizer.log 2>&1

Troubleshooting

• Audit log line counts don't match a benchmark's total_queries — the benchmark sums entries across the last 7 daily audit_<date>.jsonl files, not just today's. Check all files with:

  Get-ChildItem logs\*.jsonl | ForEach-Object { "$($_.Name): $((Get-Content $_.FullName).Count) lines" }
  

• Presidio/spaCy not installed — PII redaction silently falls back to regex patterns (email, phone, credit card, SSN). Check logs for a "Presidio not available" warning.
• Redis unreachable — caching and session memory fail open (queries still work, just without caching/memory); check for "Redis unavailable" debug logs.
• First query after a fresh clone is slow — indexes are built lazily on first access if missing. Run python rag/indexer.py --domain all ahead of time to avoid this on a live request.

License

MIT — see LICENSE.