docs/rag_comparison.md

RAG Retrieval Strategy Comparison

Implemented strategies

1. Similarity (cosine)
   Standard top-k retrieval by embedding cosine similarity. Chroma returns the k nearest chunks to the query embedding. No diversity adjustment.

2. MMR (Max Marginal Relevance)
   Balances relevance and diversity: after selecting the top chunk, we iteratively choose the next chunk that maximizes
   λ * relevance(chunk, query) - (1 - λ) * max_similarity(chunk, already_selected).
   Here we use a simplified MMR that approximates inter-chunk similarity from metadata (same source vs different source) and uses Chroma’s distance for relevance.
   MMR_LAMBDA (default 0.7) controls the trade-off: 1.0 ≈ pure similarity, 0 ≈ maximum diversity.

What changed in retrieved docs

• Similarity: Tends to return several very similar chunks (e.g. same section or same source repeated), which can make the answer repetitive and over-anchored to one part of the corpus.
• MMR: Favors spreading across different sources/sections, so the model sees more varied context and is less likely to over-cite one document. You may see more distinct source names in citations.

Speed differences

• Retrieval time: Similarity is a single Chroma query. MMR does one Chroma query (with a larger n_results) then a small Python loop to select k chunks; the extra work is negligible (milliseconds). So retrieval time is effectively the same.
• Generation time: Unchanged by strategy; it depends on context length and model. MMR can slightly reduce redundancy in the context, which might marginally affect length and thus generation time.

Timing in the app

After each assistant reply, the UI shows:

• Retrieval: X.XXs – time from query to having the list of chunks.
• Generation: X.XXs – time for the LLM to produce the answer.

These values are logged (e.g. in /data/logs/app.log) for comparison across runs.

Which performed best and why

• Similarity is best when the user’s question is very focused and the answer is expected to come from one or two passages. It’s also the fastest in principle (one query, no post-processing).
• MMR is best when the question is broad (“summarize all sources”, “compare X and Y”) or when you want to avoid over-citing one document. It improves perceived quality when the corpus has multiple relevant documents that should all contribute.

Recommendation: use similarity for short, factual questions; use MMR for synthesis and multi-document comparison. The timing approach (logging retrieval + generation and displaying them in the UI) lets you compare strategies on your own data and choose accordingly.