docs/retrieval_evaluation.md

Retrieval Evaluation Summary

This document summarizes the completed retrieval-quality stage for the Intelligent Document Query Engine. It is a lightweight committed summary; generated JSON/Markdown reports under eval/results/ remain ignored.

Benchmark

• Corpus: annual reports for Infosys, HDFC Bank, and Bajaj Finance.
• Questions: 33 labeled benchmark questions.
• Question types: lexical, paraphrase, conceptual, and distractor.
• Modes measured: MiniLM baseline, E5-small-v2, and E5+BM25 hybrid ablation.
• Metrics: Recall@3, Recall@5, MRR, needs_review count, retrieval latency, and ingestion/indexing time.
• Evaluation setting: retrieval-only (--no-llm), so metrics measure retrieval/reranking without Groq answer generation.

Final Metrics

Configuration	R@3	R@5	MRR	needs_review	p50	p95	ingest/index time
MiniLM	54.2%	62.5%	0.474	7	56 ms	144 ms	230.8s
E5-small-v2	58.3%	70.8%	0.496	4	73 ms	156 ms	534.0s
E5+BM25 hybrid	58.3%	70.8%	0.504	5	216 ms	527 ms	510.0s

Decision

E5-small-v2 is the current default embedder in the GitHub repo. It improves R@5 from 62.5% to 70.8%, improves MRR, and reduces needs_review from 7 to 4. CPU ingestion is slower than MiniLM, so MiniLM remains available as a fallback/baseline for speed or cost-sensitive runs.

Final Shipped Configuration

• Embedder: intfloat/e5-small-v2 by default.
• MiniLM fallback: all-MiniLM-L6-v2 via EMBEDDING_MODEL_NAME.
• Retrieval mode: faiss_reranker.
• Initial FAISS candidates: k_initial=20.
• Final reranked chunks: k_final=8.
• Reranker: cross-encoder/ms-marco-TinyBERT-L-2-v2.

The eval harness and production pipeline now use the same k_final=8, so the benchmark measures the shipped final-context configuration.

Default E5 run:

$env:EMBEDDING_MODEL_NAME='intfloat/e5-small-v2'

MiniLM fallback/baseline:

$env:EMBEDDING_MODEL_NAME='all-MiniLM-L6-v2'

The E5+BM25 hybrid is documented as an ablation, not as the default. It rescued one exact-table case but introduced a new regression and increased retrieval latency substantially.

Target Checks

E5 rescues over MiniLM

E5 rescued three MiniLM misses at hit@5:

• bajaj_finance_ar_2024_25_q05
• infosys_ar_2024_25_q03
• infosys_ar_2024_25_q04

These gains mostly came from better semantic/paraphrase retrieval.

q08 exact-table regression

bajaj_finance_ar_2024_25_q08 asks for total liabilities and equity from the consolidated balance sheet. MiniLM retrieved the exact supporting chunk at rank 1. E5-only missed it because it retrieved thematically similar balance-sheet and liability chunks instead of the exact table row.

The hybrid experiment fixed this case:

• Correct chunk: 2275
• BM25 rank: 1
• Final hybrid reranked rank: 1
• hit@5: true

Hybrid was still not adopted because it did not improve overall R@5 and increased p50 latency from 73 ms to 216 ms versus E5-only.

HDFC near-rescues

hdfc_bank_ar_2024_25_q03:

• Correct evidence entered the E5 candidate pool.
• In E5-only, correct chunks appeared below top-5 after reranking.
• In hybrid, correct chunks still finished outside top-5.
• Interpretation: reranker/final-selection limitation rather than an embedder-only miss.

hdfc_bank_ar_2024_25_q05:

• Correct evidence appears when the candidate pool is expanded.
• In targeted inspection, the correct chunk entered the merged pool but finished outside top-5.
• Interpretation: candidate-pool and reranker/final-selection limitation.

Hybrid regression

Hybrid introduced a new regression versus E5-only:

• infosys_ar_2024_25_q04

This was one of the E5 rescues. Hybrid reranking changed the merged candidate ordering enough to lose hit@5.

Larger Model Ablations

• GTE-base-en-v1.5 was slower and worse than E5-small-v2 in the local comparison.
• Qwen3-Embedding-0.6B was rejected because CPU ingestion was impractically slow for interactive uploads.

Recommendation

Use E5-small-v2 as the default retrieval embedder in the GitHub repo, while keeping MiniLM configurable as the fallback/baseline. Keep E5+BM25 hybrid as an ablation until there is a better merge/rerank strategy that improves q08 without losing E5 semantic rescues or adding unacceptable latency.

The Hugging Face live demo may lag behind this GitHub repo until the Space is manually synced.