docs/rag_comparison.md

# RAG Retrieval Strategy Comparison

## Implemented strategies

1. **Similarity (cosine)**  
   Standard top-k retrieval by embedding cosine similarity. Chroma returns the k nearest chunks to the query embedding. No diversity adjustment.

2. **MMR (Max Marginal Relevance)**  
   Balances relevance and diversity: after selecting the top chunk, we iteratively choose the next chunk that maximizes  
   `λ * relevance(chunk, query) - (1 - λ) * max_similarity(chunk, already_selected)`.  
   Here we use a simplified MMR that approximates inter-chunk similarity from metadata (same source vs different source) and uses Chroma’s distance for relevance.  
   `MMR_LAMBDA` (default 0.7) controls the trade-off: 1.0 ≈ pure similarity, 0 ≈ maximum diversity.

## What changed in retrieved docs

- **Similarity**: Tends to return several very similar chunks (e.g. same section or same source repeated), which can make the answer repetitive and over-anchored to one part of the corpus.
- **MMR**: Favors spreading across different sources/sections, so the model sees more varied context and is less likely to over-cite one document. You may see more distinct source names in citations.

## Speed differences

- **Retrieval time**: Similarity is a single Chroma query. MMR does one Chroma query (with a larger n_results) then a small Python loop to select k chunks; the extra work is negligible (milliseconds). So retrieval time is effectively the same.
- **Generation time**: Unchanged by strategy; it depends on context length and model. MMR can slightly reduce redundancy in the context, which might marginally affect length and thus generation time.

## Timing in the app

After each assistant reply, the UI shows:

- **Retrieval: X.XXs** – time from query to having the list of chunks.
- **Generation: X.XXs** – time for the LLM to produce the answer.

These values are logged (e.g. in `/data/logs/app.log`) for comparison across runs.

## Which performed best and why

- **Similarity** is best when the user’s question is very focused and the answer is expected to come from one or two passages. It’s also the fastest in principle (one query, no post-processing).
- **MMR** is best when the question is broad (“summarize all sources”, “compare X and Y”) or when you want to avoid over-citing one document. It improves perceived quality when the corpus has multiple relevant documents that should all contribute.

Recommendation: use **similarity** for short, factual questions; use **MMR** for synthesis and multi-document comparison. The timing approach (logging retrieval + generation and displaying them in the UI) lets you compare strategies on your own data and choose accordingly.