--- title: Calibrated Hybrid Retrieval Masterclass emoji: πŸ”¬ colorFrom: blue colorTo: indigo sdk: static pinned: false --- # πŸ”¬ Calibrated Entropy-Weighted Hybrid Retrieval [![CI](https://github.com/ayushmath07/Semantic-Document-Retrieval-API/actions/workflows/ci.yml/badge.svg)](https://github.com/ayushmath07/Semantic-Document-Retrieval-API/actions/workflows/ci.yml) [![Python 3.11](https://img.shields.io/badge/python-3.11-blue.svg)](https://www.python.org/downloads/release/python-3110/) [![FastAPI](https://img.shields.io/badge/FastAPI-1.0.0-009688.svg?logo=fastapi)](https://fastapi.tiangolo.com/) [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE) [![Code style: Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff) A research-oriented hybrid retrieval system that tests whether **score calibration makes retrieval uncertainty measurable and useful**. Combines BM25 sparse retrieval, FAISS dense retrieval, corpus-level CDF score calibration, entropy-weighted adaptive fusion, and cross-encoder reranking β€” all behind a FastAPI service. > **Benchmark:** [BEIR SciFact](https://github.com/beir-cellar/beir) β€” 5,183 PubMed abstracts, 300 test claims, document-level qrels. --- ## ✨ Highlights - **7 retrieval modes** for systematic ablation β€” from single-retriever baselines to full calibrated reranking - **Corpus-level CDF calibration** maps BM25 and dense scores to a common probability space, making entropy comparable across retrievers - **Entropy-weighted adaptive fusion** dynamically adjusts BM25↔dense balance per query based on each retriever's confidence - **Statistical rigor** β€” paired bootstrap significance tests (1,000 resamples) and Pearson correlations with p-values - **Production-ready API** with FastAPI, Docker, CI/CD, and Swagger docs --- ## πŸ—οΈ Architecture ```mermaid flowchart LR Q["πŸ” Query"] --> BM25["BM25\n(Sparse)"] Q --> FAISS["FAISS\n(Dense)"] BM25 --> CAL_S["CDF\nCalibration"] FAISS --> CAL_D["CDF\nCalibration"] CAL_S --> ENT_S["Shannon\nEntropy H_s"] CAL_D --> ENT_D["Shannon\nEntropy H_d"] ENT_S --> ALPHA["Ξ± = H_d / (H_d + H_s + Ξ΅)"] ENT_D --> ALPHA CAL_S --> FUSE["Weighted Fusion\nΞ±Β·sparse + (1-Ξ±)Β·dense"] CAL_D --> FUSE ALPHA --> FUSE FUSE --> RERANK["Cross-Encoder\nReranking"] RERANK --> RES["πŸ“„ Results + Telemetry"] style Q fill:#4A90D9,stroke:#2C5F8A,color:#fff style BM25 fill:#E8913A,stroke:#B86E2C,color:#fff style FAISS fill:#7B68EE,stroke:#5A4DB8,color:#fff style CAL_S fill:#20B2AA,stroke:#178A82,color:#fff style CAL_D fill:#20B2AA,stroke:#178A82,color:#fff style ENT_S fill:#FF6B6B,stroke:#CC5555,color:#fff style ENT_D fill:#FF6B6B,stroke:#CC5555,color:#fff style ALPHA fill:#FFD93D,stroke:#CCB030,color:#333 style FUSE fill:#6BCB77,stroke:#4FA35C,color:#fff style RERANK fill:#9B59B6,stroke:#7D3C98,color:#fff style RES fill:#4A90D9,stroke:#2C5F8A,color:#fff ``` **Pipeline:** Query β†’ parallel BM25 + FAISS retrieval β†’ CDF calibration to corpus percentiles β†’ Shannon entropy per retriever β†’ adaptive Ξ± weighting β†’ score fusion β†’ cross-encoder reranking β†’ ranked results with full telemetry. --- ## πŸ§ͺ Research Question > Does calibrating BM25 and dense retrieval scores to a common probability space via corpus-level CDF normalization make entropy a more reliable predictor of per-query retrieval quality, and does that translate to better fusion? ### Hypotheses | ID | Hypothesis | Result | |---|---|---| | **H1** | CDF entropy has the strongest negative correlation with retrieval quality | ❌ **Not supported** β€” z-score entropy is the strongest predictor | | **H2** | Entropy fusion outperforms RRF and fixed-alpha baselines | ⚠️ **Partially supported** β€” beats fixed-alpha significantly, ties with RRF |
πŸ“ Method Details ### Corpus-Level CDFs Built offline during indexing β€” **not** per-query top-k (which would normalize away the distributional signal): ``` 50 SciFact test claims + 50 pseudo-queries from corpus chunks β”‚ β–Ό score every query against all 17,243 chunks β”‚ β–Ό sort 1,724,300 BM25 scores and 1,724,300 dense scores β”‚ β–Ό corpus_cdf_bm25.npy and corpus_cdf_dense.npy ``` ### Score Calibration ``` calibrated_score = CDF_corpus(raw_score) # maps to corpus percentile ``` ### Entropy Computation ``` H = βˆ’Ξ£α΅’ pα΅’ logβ‚‚(pα΅’) ``` ### Adaptive Fusion ``` Ξ± = H_dense / (H_dense + H_sparse + Ξ΅) fused = Ξ± Β· sparse_score + (1 βˆ’ Ξ±) Β· dense_score ``` When dense retrieval has high entropy (low confidence), Ξ± increases β†’ system trusts BM25 more. When sparse retrieval has high entropy, Ξ± decreases β†’ system trusts dense retrieval more. ### Bug Fixes from Initial Run The first toy-corpus run exposed two evaluation bugs that were fixed before SciFact benchmarking: - **Entropy invariance bug:** Entropy was subtracting the minimum score before normalization, causing raw/min-max/z-score entropy to collapse under affine transforms. Fixed: nonnegative calibrated scores are treated as probability mass directly; softmax is used only for negative-valued z-scores. - **Chunk-level evaluation:** Metrics were computed at chunk-level, but BEIR qrels are document-level. Fixed: retrieved chunks are deduplicated by source document ID before metric computation.
--- ## πŸ“Š Benchmark Results (BEIR SciFact) ```bash python scripts/build_index.py --dataset scifact python scripts/evaluate.py ``` ### Retrieval Ablation | Mode | NDCG@10 | MRR | P@3 | P@5 | R@5 | p95 Latency | |:---|---:|---:|---:|---:|---:|---:| | `dense` | 0.6715 | 0.6329 | 0.2466 | 0.1647 | 0.7495 | 33 ms | | `sparse` | 0.6151 | 0.5804 | 0.2233 | 0.1520 | 0.7089 | 130 ms | | `rrf` | 0.7028 | 0.6713 | 0.2555 | 0.1680 | 0.7744 | 249 ms | | `hybrid_fixed` | 0.6829 | 0.6527 | 0.2522 | 0.1647 | 0.7594 | 232 ms | | `hybrid_calibrated` | 0.6981 | 0.6672 | 0.2489 | 0.1660 | 0.7661 | 241 ms | | `hybrid_fixed_rerank` | 0.7006 | 0.6653 | 0.2578 | 0.1700 | 0.7714 | 2432 ms | | **`hybrid_calibrated_rerank`** | **0.7071** | **0.6719** | **0.2622** | **0.1720** | **0.7838** | 2116 ms | > **Best overall:** `hybrid_calibrated_rerank` achieves the highest NDCG@10 (0.7071) with the lowest per-query variance (Οƒ = 0.3703).
πŸ“ˆ H1: Entropy Correlation Analysis Pearson correlations β€” entropy as predictor vs. per-query retrieval quality. Stronger negative r = higher entropy better predicts lower quality. | Retriever | Calibration | r vs NDCG@10 | p-value | r vs MRR | p-value | |:---|:---|---:|---:|---:|---:| | Sparse | raw | 0.0033 | 0.9550 | 0.0057 | 0.9220 | | Sparse | min-max | 0.0033 | 0.9550 | 0.0057 | 0.9220 | | Sparse | z-score | **βˆ’0.3759** | <0.000001 | **βˆ’0.3855** | <0.000001 | | Sparse | CDF | βˆ’0.0085 | 0.8834 | βˆ’0.0031 | 0.9575 | | Dense | raw | 0.0865 | 0.1348 | 0.0636 | 0.2718 | | Dense | min-max | 0.0053 | 0.9273 | βˆ’0.0164 | 0.7775 | | Dense | z-score | **βˆ’0.4065** | <0.000001 | **βˆ’0.3970** | <0.000001 | | Dense | CDF | βˆ’0.1151 | 0.0463 | βˆ’0.1100 | 0.0571 | **Verdict:** H1 is **not supported** as originally stated. Z-score entropy is the strongest negative predictor for both retrievers on SciFact. CDF entropy shows a weak but significant correlation for dense retrieval only.
πŸ“‰ H2: Paired Bootstrap Significance Tests 1,000 paired bootstrap resamples over 300 SciFact test queries. | Comparison | Metric | Mean Ξ” | 95% CI | p-value | Significant? | |:---|:---|---:|:---|---:|:---| | `calibrated` vs `rrf` | NDCG@10 | βˆ’0.0047 | [βˆ’0.017, +0.007] | 0.426 | No | | `calibrated` vs `rrf` | P@3 | βˆ’0.0067 | [βˆ’0.012, βˆ’0.002] | 0.002 | Yes, worse | | `calibrated` vs `fixed` | NDCG@10 | **+0.0152** | [+0.003, +0.027] | **0.012** | βœ… Yes | | `calibrated` vs `fixed` | MRR | **+0.0145** | [+0.002, +0.027] | **0.028** | βœ… Yes | | `cal_rerank` vs `fix_rerank` | NDCG@10 | +0.0065 | [βˆ’0.006, +0.020] | 0.314 | No | **Verdict:** H2 is **partially supported**. CDF entropy fusion significantly outperforms fixed-alpha fusion (p=0.012 on NDCG@10). It does not beat RRF, and with reranking the improvement is not statistically significant.
--- ## πŸŽ›οΈ Retrieval Modes | Mode | Strategy | Description | |:---|:---|:---| | `dense` | Single | FAISS dense retrieval only | | `sparse` | Single | BM25 lexical retrieval only | | `rrf` | Fusion | Reciprocal Rank Fusion (k=60) | | `hybrid_fixed` | Fusion | Min-max calibration + fixed Ξ±=0.5 | | `hybrid_calibrated` | Fusion | CDF calibration + entropy-weighted Ξ± | | `hybrid_fixed_rerank` | Fusion + Rerank | Fixed hybrid β†’ cross-encoder | | `hybrid_calibrated_rerank` | Full Pipeline | CDF entropy hybrid β†’ cross-encoder | --- ## πŸš€ Quick Start ```bash # 1. Clone and set up git clone https://github.com/ayushmath07/Semantic-Document-Retrieval-API.git cd Semantic-Document-Retrieval-API python -m venv venv venv\Scripts\activate # Windows # source venv/bin/activate # macOS/Linux pip install -r requirements.txt # 2. Build the index (downloads SciFact on first run) python scripts/build_index.py --dataset scifact # 3. Start the API uvicorn app.main:app --reload ``` Open **[http://127.0.0.1:8000/docs](http://127.0.0.1:8000/docs)** for interactive Swagger docs. > **Note:** First run downloads SentenceTransformers models (~90 MB). Dense retrieval uses `all-MiniLM-L6-v2`; reranking uses `cross-encoder/ms-marco-MiniLM-L6-v2`. --- ## πŸ“‘ API Reference ### `GET /modes` β€” List retrieval modes ```bash curl http://127.0.0.1:8000/modes ``` ### `POST /query` β€” Search documents ```bash curl -X POST http://127.0.0.1:8000/query \ -H "Content-Type: application/json" \ -d '{"question": "0-dimensional biomaterials show inductive properties.", "top_k": 3, "mode": "hybrid_calibrated_rerank"}' ```
Example response ```json { "question": "0-dimensional biomaterials show inductive properties.", "answer": "[4983046] Biomaterial dimensionality ...", "sources": [ { "source": "4983046", "chunk": 1, "text": "...", "score": 0.823, "cross_encoder_score": 2.41 } ], "retrieval_latency_ms": 241.5, "telemetry": { "mode": "hybrid_calibrated_rerank", "alpha": 0.487, "h_sparse": 8.21, "h_dense": 7.79, "calibration": "cdf", "reranked": true } } ```
### `POST /upload` β€” Upload documents ```bash curl -X POST http://127.0.0.1:8000/upload \ -F "file=@paper.pdf" ``` Supports `.pdf`, `.txt`, and `.md` files. ### `GET /health` β€” Health check ### `GET /documents` β€” List indexed documents --- ## πŸ“ Project Structure ``` app/ β”œβ”€β”€ main.py FastAPI application and route handlers β”œβ”€β”€ retriever.py Seven-mode retrieval orchestrator β”œβ”€β”€ calibration.py CDF transforms, entropy, QPP predictors, statistics β”œβ”€β”€ fusion.py RRF, fixed linear fusion, entropy-weighted fusion β”œβ”€β”€ reranker.py Cross-encoder reranking (ms-marco-MiniLM-L6-v2) β”œβ”€β”€ sparse_retriever.py BM25 index, search, and full-corpus scoring └── datasets.py SciFact corpus/query/qrels loaders scripts/ β”œβ”€β”€ build_index.py Builds FAISS, BM25, CDFs, corpus LM, doc lookup └── evaluate.py Runs H1/H2 evaluation and bootstrap significance tests tests/ pytest test suite (9 tests) eval/ Golden set labels and generated results data/faiss_index/ Runtime artifacts (FAISS, BM25, CDFs, metadata) ```
Runtime artifacts | Artifact | Purpose | |:---|:---| | `index.faiss`, `index.pkl` | Dense FAISS vector store | | `bm25_index.pkl` | Tokenized BM25 corpus and metadata | | `corpus_cdf_bm25.npy` | Corpus-level BM25 score CDF | | `corpus_cdf_dense.npy` | Corpus-level dense score CDF | | `corpus_lm.pkl` | Corpus unigram language model (Clarity Score) | | `doc_lookup.pkl` | (source, chunk) β†’ text lookup | | `index_metadata.json` | Dataset and artifact build metadata |
--- ## βš™οΈ Configuration | Variable | Default | Description | |:---|:---|:---| | `EMBEDDING_MODEL` | `sentence-transformers/all-MiniLM-L6-v2` | HuggingFace embedding model | | `DATA_DIR` | `data` | Base directory for uploads and index | | `MIN_RELEVANCE_SCORE` | `0.0` | Minimum similarity threshold (dense-only mode) | | `CANDIDATE_K` | `20` | Candidates per retriever before fusion | Copy `.env.example` to `.env` to customize: ```bash cp .env.example .env ``` --- ## 🐳 Docker ```bash docker compose up --build ``` API docs at **[http://127.0.0.1:8000/docs](http://127.0.0.1:8000/docs)**. --- ## πŸ§ͺ Testing ```bash pytest tests/ -v ``` 9 tests covering: health checks, upload validation, empty-index behavior, mode listing, mode validation, query telemetry, calibration entropy behavior, and document listing. --- ## πŸ”¬ Run Full Evaluation ```bash python scripts/build_index.py --dataset scifact python scripts/evaluate.py ``` The evaluator outputs: - Aggregate metrics for all 7 modes - NDCG@10 variance and per-query metrics - H1 Pearson correlations with p-values - H2 paired bootstrap confidence intervals and p-values - Per-query rankings, relevance flags, and telemetry Full results are saved to `eval/results.json`. --- ## πŸ’‘ Interpretation The original toy CS corpus was useful for plumbing, but it was too easy and too small to validate the research claim. SciFact changes the story: **calibration is not a universal win**, RRF remains a strong baseline, and the clearest uncertainty signal comes from z-score entropy rather than CDF entropy. That is a stronger project outcome than a polished toy result because the system now produces **falsifiable, benchmarked evidence**. --- ## 🀝 Contributing See [CONTRIBUTING.md](CONTRIBUTING.md) for setup instructions, code style, and PR guidelines. ## πŸ“„ License [MIT](LICENSE)