ARCHITECTURE_EXPLANATION.md

Cortex RAG — Architecture & Implementation Guide

This document provides a deep dive into the architecture of Cortex, a production-grade Retrieval-Augmented Generation (RAG) system. This guide is structured to help you explain the "what", "how", and "why" of each layer during your GenAI Engineer interview.

---

🏗️ 1. High-Level Architecture Overview

Cortex follows a modular, multi-layer RAG architecture designed for high precision, scalability, and reliability. It moves beyond "naive RAG" by implementing:

• Semantic Data Ingestion (instead of fixed-size chunking)
• Hybrid Multi-Strategy Retrieval (Dense + Sparse + Knowledge Graph)
• Corrective Gating (CRAG) (to handle retrieval failures)
• Reference-Free Evaluation (using RAGAS)

---

📥 2. Ingestion Layer: "Context-Aware Processing"

Document Loading

• Supports multiple formats: PDF, HTML, and TXT.
• Implementation: DocumentLoader handles parsing and basic cleaning.

Semantic Chunker (ingestion/chunker.py)

• The Problem: Fixed-size chunking (e.g., 512 tokens) often splits mid-sentence or mid-concept, losing semantic coherence.
• The Solution: We use Sentence-Level Semantic Boundary Detection.
• How it works:
  1. Split text into individual sentences.
  2. Embed each sentence using BAAI/bge-small-en-v1.5.
  3. Compute the cosine similarity between consecutive sentence embeddings.
  4. Insert a chunk boundary whenever the similarity drops below a certain threshold (e.g., 0.82) or the token limit is reached.
• Why? This ensures each chunk contains a single, coherent topic.

Parent-Child Hierarchy

• The Problem: Small chunks are better for retrieval precision, but large chunks provide better context for generation.
• The Implementation:
  • Child Chunks (~256 tokens): These are the units indexed in the vector database. They represent a specific "nugget" of information.
  • Parent Chunks (~1024 tokens): A wider window of text centered on the child. When a child is retrieved, its parent text is what gets sent to the LLM.
• Why? It decouples retrieval granularity (find exactly what you need) from context width (give the LLM enough room to understand).

---

🔍 3. Retrieval Layer: "Multi-Strategy Orchestration"

Cortex doesn't just rely on vector search; it uses a MultiStrategyRetriever to combine different search paradigms.

A. Dense Retrieval (Milvus)

• Embeddings: bge-small-en-v1.5 (384-dim).
• Vector DB: Milvus (Dockerized).
• Indexing: IVF_FLAT with COSINE similarity metric.
• Why? Captures semantic meaning (e.g., "puppy" matches "dog").

B. Sparse Retrieval (BM25)

• Implementation: rank_bm25 library.
• Why? Essential for exact keyword matching, acronyms, and specific names (e.g., "Project Cortex-X1") where vector search might be too "fuzzy".

C. Knowledge Graph (GraphRAG)

• Extraction: During ingestion, we use spaCy for Named Entity Recognition (NER) and REBEL (or LLM) for relation extraction.
• Storage: A NetworkX graph storing triples: (Subject) --[Predicate]--> (Object).
• Retrieval:
  1. Extract entities from the user query.
  2. Traverse the graph to find related nodes (multi-hop traversal).
  3. Retrieve the chunks associated with those nodes.
• Why? Solves "multi-hop" queries where the answer requires connecting disparate pieces of information across the document.

D. Fusion & Reranking (retrieval/fusion.py)

• RRF (Reciprocal Rank Fusion): Combines the ranked lists from Milvus, BM25, and the Graph into one unified list.
• Cross-Encoder Reranker: We take the top-15 fused candidates and run them through a Cross-Encoder (e.g., BAAI/bge-reranker-base).
• Why? Cross-encoders are much more accurate (but slower) than vector search because they look at the query and chunk simultaneously. Using them as a final "filter" boosts precision significantly.

---

🧠 4. Generation Layer: "Corrective RAG (CRAG)"

The CRAGGate (generation/crag.py) acts as a "quality control" layer between retrieval and the LLM.

The CRAG Workflow

1. Grading: An LLM-as-judge assesses if the retrieved chunks are relevant to the query.
2. Action Categories:
   • GOOD: Chunks are relevant. Proceed to generation.
   • POOR: Chunks are partially relevant. Rewrite the query (using CoT) and re-retrieve to find better results.
   • ABSENT: Knowledge base doesn't have the answer. Fallback to Web Search (Tavily/DuckDuckGo).
3. LLM Generation: Uses Groq (Llama 3), OpenAI, or NVIDIA NIM to generate the final answer with inline citations (e.g., "The sky is blue [1].").

---

📊 5. Evaluation Layer: "Reference-Free Metrics"

Since production RAG systems often lack "ground truth" answers, we use the RAGAS framework (evaluation/ragas_eval.py).

Key Metrics

• Faithfulness: Does the answer stay true to the retrieved context? (Prevents hallucinations).
• Answer Relevancy: Does the answer actually address the user's question?
• Context Precision: Were the retrieved chunks actually useful?
• Context Utilisation: What % of retrieved chunks were actually cited?

Implementation

• Evaluations run asynchronously in background threads so they don't slow down the user's response time.
• Results are stored in a local SQLite DB for monitoring.

---

🛠️ 6. System & Infrastructure

• API: FastAPI for high-performance, asynchronous endpoints.
• UI: Streamlit for a clean, interactive dashboard (Ask, Ingest, Monitor).
• Cache: Redis for caching query results (TTL-based) to save LLM costs and latency.
• Deployment: Full Docker Compose setup for Milvus, Redis, API, and UI.

---

💡 Interview Tip: "Why this architecture?"

If asked why you built it this way, emphasize these three points:

1. Precision: By using Semantic Chunking and Cross-Encoder Reranking, we ensure only the most relevant context reaches the LLM.
2. Reliability: CRAG ensures the system doesn't hallucinate when the knowledge base is missing information.
3. Observability: By integrating RAGAS, we have an automated way to track performance and catch regressions.

Good luck with your interview! 🚀