data/docs/redis_worker_architecture.md

# Redis Worker & Asynchronous Background Tasks Architecture

This document explains how the Redis worker handles asynchronous background tasks and document indexing in the Support Docs Copilot architecture.

## 1. Asynchronous Background Task Queue
The Support Docs Copilot uses **ARQ (Async Redis Queue)** paired with a Redis instance to manage heavy background workloads without blocking the asynchronous FastAPI event loop. When long-running tasks such as document ingestion or batch evaluation are initiated via the API, the server pushes a task job to Redis and immediately returns a job ID to the client.

## 2. Document Indexing & Ingestion Workflow
When an ingestion request (`POST /ingest`) is received:
1. **Task Dispatch**: The API server enqueues an `ingest_documents_task` into Redis.
2. **Worker Execution**: The standalone ARQ worker process picks up the task from Redis asynchronously.
3. **Chunking & Embedding**: The worker loads files from `data/docs`, applies recursive character chunking with overlap, and generates dense vector embeddings using local ONNX models (`FastEmbedEmbeddings` with `BAAI/bge-small-en-v1.5`) as well as sparse embeddings (`Qdrant/bm25`) for hybrid retrieval.
4. **Vector Storage**: The processed points and metadata are batched and upserted into the Qdrant vector database under the `support_docs` collection.

## 3. Reliability & Scalability
By offloading document indexing and semantic cache pre-warming to the Redis worker queue, the backend ensures zero degradation in chat latency for active user sessions. The queue supports automatic retries, job status polling, and concurrency control.