docs/rag_techniques.md

Retrieval-Augmented Generation: Techniques and Architecture

Overview

Retrieval-Augmented Generation (RAG) is a framework that grounds language model outputs in a retrieved knowledge base, reducing hallucination and enabling accurate answers over private or domain-specific corpora. Introduced by Lewis et al. (2020), RAG combines a dense retrieval component with a generative language model.

Naive RAG — embed query, retrieve top-k chunks, prompt an LLM — fails predictably in three ways: retrieval misses (the right chunk isn't found), confident hallucination (the LLM generates fluently despite poor context), and multi-hop failures (questions spanning multiple documents get partial answers). Each failure mode has a principled solution.

---

Retrieval Techniques

Hybrid Dense + BM25 + RRF

Pure dense (vector) retrieval struggles with lexically specific terms: product codes, proper names, acronyms. BM25 (Best Match 25) is a classical term-frequency model that handles exact-match queries. Combining both via Reciprocal Rank Fusion (RRF) produces a merged ranking that outperforms either alone.

RRF formula: score(d) = Σ 1 / (k + rank_i(d)), where k=60 by default and rank_i is the position of document d in retrieval list i. Documents appearing in both the dense and sparse lists score higher than those in only one.

Practical impact: 15–25% recall improvement over dense-only on keyword-heavy queries.

Cross-Encoder Reranking

Bi-encoder models (used for initial retrieval) encode query and document independently for speed. Cross-encoders take the concatenated (query, document) pair as input, enabling full attention between them. This produces much higher-quality relevance scores at the cost of latency.

The typical pipeline: retrieve top-50 candidates with a bi-encoder, then rerank with a cross-encoder and keep top-10. The cross-encoder (e.g., ms-marco-MiniLM-L-6-v2) is not used for first-pass retrieval due to the O(n) inference cost over the full corpus.

Latency cost: +150–300ms. Quality gain: 10–20% improvement in MRR@10.

HyDE — Hypothetical Document Embeddings

Gao et al. (NAACL 2022) identified a fundamental asymmetry: queries are short and question-like, while relevant documents are long and answer-like. Embedding both in the same space creates a distribution mismatch.

HyDE solution: prompt an LLM to generate a hypothetical answer to the query without access to the knowledge base. Embed the hypothetical answer (not the original query) for retrieval. The hypothetical answer lives in the same embedding space as real documents.

Effect: significant recall improvement on complex, multi-part questions. Especially effective when queries are abstract or underspecified.

Multi-Query Expansion

A single query may not capture all relevant phrasings. Multi-query generates N (typically 3) semantically diverse reformulations of the input query using an LLM. Each reformulation is embedded and retrieved against independently. Results are merged and deduplicated using RRF or union with deduplication by content hash.

Effect: +10–15% recall on queries with ambiguous or domain-specific phrasing.

Contextual Retrieval — Anthropic (November 2024)

Standard chunking loses document-level context. A chunk reading "The company reported a 15% decline" provides no information about which company, which metric, or which time period — because that context is in adjacent chunks that were split away.

Contextual Retrieval prepends a 1–2 sentence LLM-generated context to each chunk before embedding. The context is generated by prompting Claude Haiku with the full document and the chunk: "Situate this excerpt within the document." The contextualized chunk is then embedded instead of the raw chunk.

Anthropic reported a 49% reduction in retrieval failures on their internal benchmarks. The approach is ingest-time, so queries run at normal speed. Cost: one Haiku API call per chunk, approximately $0.01–0.05 per document.

MMR — Maximal Marginal Relevance

Carbonell & Goldstein (1998): retrieved chunks tend to cluster around the same passage. MMR balances relevance against diversity: at each step, select the document that maximizes λ·sim(q, d) - (1-λ)·max_{d'∈S} sim(d, d'), where S is the already-selected set. λ=1 is pure relevance; λ=0 is pure diversity.

Effect: reduces redundant context in the prompt, leaving more room for actually useful chunks.

---

Hallucination Mitigation

CRAG — Corrective RAG

Yan et al. (2024): a lightweight evaluator model scores the relevance of each retrieved document against the query. Documents are classified as Correct, Ambiguous, or Incorrect. Incorrect documents trigger a web search fallback. Ambiguous documents are decomposed into finer-grained queries before retrieval.

The evaluator is a fine-tuned T5-small (77M parameters) — cheap enough to run on every query. CRAG prevents the LLM from generating answers grounded in irrelevant or misleading retrieved context.

Sufficient Context — Google (ICLR 2025)

Levy et al. (2025) introduced a framework where the system scores the retrieved context for sufficiency before generation. If the context density and coverage scores fall below a threshold, the system retrieves more rather than generating an answer.

Three components: density scoring (does the context have enough relevant tokens?), coverage scoring (does it address all sub-questions?), and an optional LLM self-rating. The system abstains ("I don't have enough information to answer this confidently") rather than hallucinating when context is insufficient.

This is a more conservative but more reliable approach than CRAG's reranking strategy.

---

Multi-Hop and Reasoning Techniques

CoT-RAG — Chain-of-Thought RAG (EMNLP 2025)

Liu et al.: complex questions requiring facts from multiple documents fail with single-pass retrieval. CoT-RAG decomposes the question into a sequence of reasoning steps using chain-of-thought prompting. For each step, it retrieves context relevant to that sub-question. The intermediate retrieval results are synthesized at each step.

Example: "What is the difference in revenue between the top-performing and bottom-performing products?" decomposes into (1) retrieve top-performing product revenue, (2) retrieve bottom-performing product revenue, (3) compute difference. Each retrieval is targeted and specific.

Adaptive RAG — Jeong et al. (NAACL 2024)

Not all questions require retrieval. Simple factual questions (capital cities, arithmetic) are better answered from parametric memory. Complex questions need iterative retrieval. Over-retrieval adds latency and noise.

Adaptive RAG trains a classifier to route queries into three strategies: NO_RETRIEVAL (answer from memory), SINGLE_STEP (one retrieval pass), or ITERATIVE (multiple passes with query rewriting). The classifier runs before any retrieval occurs.

Self-RAG — Asai et al. (ICLR 2024)

Self-RAG fine-tunes the generation model to produce special reflection tokens: [Retrieve] (should I retrieve?), [IsRel] (is this chunk relevant?), [IsSup] (does the generated text match retrieved context?), [IsUse] (is the response useful?). The model evaluates its own outputs and retrieves more if needed.

Unlike CRAG (which uses an external evaluator), Self-RAG's reflection is internal to the generation model. This requires a fine-tuned model rather than an off-the-shelf LLM.

RAPTOR — Sarthi et al. (ICLR 2024)

Long documents fail standard chunking: a 100-page report chunked at 1024 characters produces hundreds of disconnected pieces. RAPTOR builds a recursive tree: raw chunks → cluster by semantic similarity → summarize each cluster → cluster the summaries → repeat until a single root summary.

At query time, retrieve at multiple levels simultaneously: specific chunks for precise facts, cluster summaries for thematic context, root summary for global understanding. This enables answering both "what is the exact figure on page 47?" and "what is the overall conclusion of this report?"

---

Knowledge Graph Techniques

GraphRAG — Microsoft (2024)

Edge et al.: standard RAG retrieves isolated chunks with no awareness of entity relationships. GraphRAG extracts a knowledge graph (entities, relations, co-occurrence) from the corpus. Communities of related entities are detected and summarized. Queries are answered against both individual chunks and community summaries.

Effective for: global questions ("what are the main themes?"), comparative questions ("how do X and Y differ?"), and relationship questions ("what connects A to B?").

LightRAG — HKUDS (EMNLP 2025)

Guo et al.: extends GraphRAG with dual-level retrieval routing. Low-level queries (specific facts, named entities) route to entity-level graph traversal. High-level queries (themes, summaries, concepts) route to community-level summaries. Auto-routing selects the appropriate level based on query type classification.

LightRAG is faster than GraphRAG (no community detection at query time) and more flexible (incremental graph updates without full re-ingestion).

---

Infrastructure

Semantic Cache

Identical or near-identical queries should not hit the LLM twice. The cache stores (query_embedding, response) pairs. On each new query, compute cosine similarity against cached embeddings. If similarity ≥ threshold (default 0.95), return the cached response directly.

Effect: 0ms latency on cache hits. For a system with repeated query patterns, cache hit rate of 30–60% is typical.

Agentic RAG

Some queries require multiple tool invocations: search the document store, then run a SQL query against a structured database, then verify against a web search result. Agentic RAG implements a tool-use loop where the LLM can invoke:

• search_docs: query the vector store
• search_web: DuckDuckGo or Tavily fallback
• query_sql: structured database query
• calculate: arithmetic and unit conversion

The loop continues until the LLM produces a final answer or hits the max iteration limit.

Embedding Fine-Tuning

Pre-trained embedding models are trained on general corpora. Domain-specific documents (legal, medical, financial, technical) have vocabulary and semantics that diverge from the training distribution.

Fine-tuning with Multiple Negatives Ranking (MNR) loss: for each (query, relevant_document) pair, treat all other documents in the batch as negative examples. The model learns to bring query-document pairs closer in embedding space while pushing non-relevant pairs apart.

Training data comes from the feedback loop: user thumbs-up/down on responses generates contrastive pairs. Fine-tuned models show 10–30% retrieval improvement on domain-specific benchmarks.

---

Evaluation Framework

RAGAS Metrics

• Faithfulness: does the generated answer contain only facts supported by the retrieved context? Scored by an LLM judging each claim in the answer against the context. Scale: 1–5.
• Answer Relevancy: does the answer actually address the question? Measured by generating synthetic questions from the answer and computing similarity to the original question.
• Context Recall: was the relevant information retrieved? Requires ground-truth labels — what information was needed to answer the question correctly?
• Context Precision: among retrieved chunks, what fraction was actually needed? High precision means the retriever is not flooding the prompt with irrelevant context.

CI/CD Quality Gate

The evaluation harness runs on every push via GitHub Actions. The workflow:

1. Loads a synthetic QA corpus (questions, ground-truth answers, source documents)
2. Runs the full retrieval + generation stack
3. Scores Faithfulness, Recall@K, and Answer Relevancy
4. Fails the build if any metric drops below threshold (faithfulness < 2.5/5.0, recall < 0.5)

This prevents silent regressions when changing retrieval parameters, chunking strategy, or embedding model.