Hugging Face's logo

Spaces:
Kind07
/
semantic-retrieval-api
Running

App Files Community
semantic-retrieval-api / index.html
Kind07's picture
Kind07
Add distinct BM25, FAISS, and Low Confidence Rejection sample buttons; update rejection text to remove query word
f28ebaf
Raw
History Blame Contribute Delete
46.5 kB
<!DOCTYPE html>
<html lang="en">
<head>
 <meta charset="UTF-8">
 <meta name="viewport" content="width=device-width, initial-scale=1.0">
 <title>Calibrated Entropy-Weighted Hybrid Retrieval — Technical Guide & Operational Engine</title>
 <link rel="preconnect" href="https://fonts.googleapis.com">
 <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
 <link href="https://fonts.googleapis.com/css2?family=Inter:wght@400;500;600;700;800&family=JetBrains+Mono:wght@400;500;700&display=swap" rel="stylesheet">
 <link rel="stylesheet" href="style.css">
 <!-- MathJax for LaTeX Rendering -->
 <script src="https://polyfill.io/v3/polyfill.min.js?features=es6"></script>
 <script id="MathJax-script" async src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js"></script>
</head>
<body>
<div class="app-container">
 <!-- Sidebar Navigation -->
 <nav class="sidebar">
 <div class="sidebar-brand">
 <div class="logo-icon">🔬</div>
 <div class="sidebar-brand-text">
 <h1>Hybrid RAG System</h1>
 <p>Project Technical Guide</p>
 </div>
 </div>
<div class="nav-section-label">Navigation</div>
 <ul class="nav-menu">
 <li><a href="#overview" class="nav-link active"><span class="icon">🚀</span> Overview</a></li>
 <li><a href="#live-demo" class="nav-link"><span class="icon">🎮</span> Operational Retrieval Engine</a></li>
 <li><a href="#techstack" class="nav-link"><span class="icon">⚡</span> Tech Stack</a></li>
 <li><a href="#architecture" class="nav-link"><span class="icon">🏗️</span> Pipeline & Data Flow</a></li>
 <li><a href="#data-journey" class="nav-link"><span class="icon">📦</span> Data State & Chunking</a></li>
 <li><a href="#mathematics" class="nav-link"><span class="icon">📐</span> Mathematics Deep-Dive</a></li>
 <li><a href="#simulator" class="nav-link"><span class="icon">🎛️</span> Interactive Simulator</a></li>
 <li><a href="#codebase" class="nav-link"><span class="icon">📁</span> File & Function Registry</a></li>
 <li><a href="#benchmarks" class="nav-link"><span class="icon">📊</span> Research Results & H1/H2</a></li>
 <li><a href="#how-to-run" class="nav-link"><span class="icon">🛠️</span> How to Run & Deploy</a></li>
 </ul>
 </nav>
<!-- Main Content -->
 <main class="main-content">
<!-- Hero Header -->
 <header id="overview" class="hero section">
 <div class="hero-badge">🔬 Calibrated Hybrid RAG Architecture</div>
 <h1 class="hero-title">Calibrated Entropy-Weighted Hybrid Retrieval</h1>
 <p class="hero-description">
 I built this research-oriented hybrid retrieval system to test whether score calibration makes retrieval uncertainty measurable and useful for search performance. This interactive masterclass explains every architectural decision, mathematical algorithm, and implementation detail I built into the system.
 </p>
<div class="hero-stats">
 <div class="stat-item">
 <h3>7 Modes</h3>
 <p>Retrieval Ablation</p>
 </div>
 <div class="stat-item">
 <h3>Computer Science</h3>
 <p>& Science Data</p>
 </div>
 <div class="stat-item">
 <h3>Strict Guard</h3>
 <p>Low-Confidence Rejection</p>
 </div>
 <div class="stat-item">
 <h3>0.7071</h3>
 <p>Peak NDCG@10 Score</p>
 </div>
 </div>
 </header>
<!-- OPERATIONAL HYBRID RETRIEVAL ENGINE DEMO -->
 <section id="live-demo" class="section">
 <div class="section-header">
 <span class="section-tag">Operational Engine</span>
 <h2 class="section-title">🎮 Operational Retrieval Engine & Live Queries</h2>
 <p class="section-subtitle">Test the actual hybrid retrieval pipeline I built across Computer Science & Biomedical corpora. If top candidates fall below minimum confidence (Score &lt; 0.15), the system explicitly rejects the result to prevent hallucination.</p>
 </div>
<div class="simulator-card">
 <div class="grid-2">
 <div>
 <label style="font-weight: 600; font-size: 0.85rem; display: block; margin-bottom: 0.4rem;">Search Question / Claim</label>
 <input type="text" id="demo-query-input" class="search-box" value="Transformer models rely on multi-head self-attention mechanisms." style="margin-bottom: 0;">
 </div>
 <div>
 <label style="font-weight: 600; font-size: 0.85rem; display: block; margin-bottom: 0.4rem;">Retrieval Mode</label>
 <select id="demo-mode-select" class="search-box" style="margin-bottom: 0; cursor: pointer; background: var(--bg-card);">
 <option value="hybrid_calibrated_rerank" selected>hybrid_calibrated_rerank (Full Calibrated Pipeline + Reranker)</option>
 <option value="hybrid_calibrated">hybrid_calibrated (CDF Calibration + Dynamic Entropy Alpha)</option>
 <option value="hybrid_fixed_rerank">hybrid_fixed_rerank (Min-Max Calibration + Reranker)</option>
 <option value="hybrid_fixed">hybrid_fixed (Min-Max Calibration + Fixed α=0.5)</option>
 <option value="rrf">rrf (Reciprocal Rank Fusion k=60)</option>
 <option value="sparse">sparse (BM25 Lexical Keyword Search)</option>
 <option value="dense">dense (FAISS Dense Vector Similarity)</option>
 </select>
 </div>
 </div>
<div style="margin-top: 0.75rem; display: flex; gap: 0.5rem; flex-wrap: wrap;">
 <span style="font-size: 0.75rem; color: var(--text-muted); font-weight: 600;">Sample Queries:</span>
 <button class="function-tag" onclick="document.getElementById('demo-query-input').value='B-Tree indexing and cost-based query optimizer'; document.getElementById('demo-mode-select').value='sparse'; document.getElementById('run-demo-btn').click();" style="cursor: pointer; background: rgba(56, 189, 248, 0.15); color: var(--accent-cyan);">🔍 BM25 Lexical Match Example</button>
 <button class="function-tag" onclick="document.getElementById('demo-query-input').value='How do neural agents learn from experience replay?'; document.getElementById('demo-mode-select').value='dense'; document.getElementById('run-demo-btn').click();" style="cursor: pointer; background: rgba(168, 85, 247, 0.15); color: var(--accent-purple);">📐 FAISS Dense Vector Example</button>
 <button class="function-tag" onclick="document.getElementById('demo-query-input').value='Transformer models rely on multi-head self-attention mechanisms.'; document.getElementById('demo-mode-select').value='hybrid_calibrated_rerank'; document.getElementById('run-demo-btn').click();" style="cursor: pointer; background: rgba(52, 211, 153, 0.15); color: var(--accent-emerald);">⚡ Full Hybrid + Reranker</button>
 <button class="function-tag" onclick="document.getElementById('demo-query-input').value='Random irrelevant cooking recipe.'; document.getElementById('demo-mode-select').value='hybrid_calibrated_rerank'; document.getElementById('run-demo-btn').click();" style="cursor: pointer; background: rgba(244, 63, 94, 0.15); color: var(--accent-rose);">🚫 Test Low Confidence Rejection</button>
 </div>
<button id="run-demo-btn" style="margin-top: 1.25rem; width: 100%; padding: 0.85rem; background: linear-gradient(135deg, var(--accent-cyan), var(--accent-blue)); border: none; border-radius: 8px; color: white; font-weight: 700; font-size: 0.95rem; cursor: pointer; box-shadow: 0 4px 12px rgba(56, 189, 248, 0.3);">
 ⚡ Execute Retrieval & Compute Telemetry
 </button>
<!-- Live Telemetry Display -->
 <div id="demo-telemetry-box" style="margin-top: 1.5rem; padding: 1rem; background: rgba(0, 0, 0, 0.4); border: 1px solid var(--border-color); border-radius: 8px; font-family: var(--font-mono); font-size: 0.85rem;">
 <!-- Populated by JavaScript -->
 </div>
<!-- Ranked Results Output -->
 <div id="demo-results-container" style="margin-top: 1.5rem;">
 <!-- Populated by JavaScript -->
 </div>
 </div>
 </section>
<!-- Tech Stack Section -->
 <section id="techstack" class="section">
 <div class="section-header">
 <span class="section-tag">Technology Stack</span>
 <h2 class="section-title">What I Built & Core Technologies</h2>
 <p class="section-subtitle">Every library, framework, and model I integrated into the system.</p>
 </div>
<div class="tech-grid">
 <div class="tech-card">
 <div class="tech-icon">🐍</div>
 <div class="tech-info">
 <h4>Python 3.11</h4>
 <p>Core runtime language, typing system, async context management, and fast scientific computation.</p>
 </div>
 </div>
<div class="tech-card">
 <div class="tech-icon">⚡</div>
 <div class="tech-info">
 <h4>FastAPI & Uvicorn</h4>
 <p>High-performance web API framework with automatic Pydantic request validation and interactive Swagger UI docs.</p>
 </div>
 </div>
<div class="tech-card">
 <div class="tech-icon">🔍</div>
 <div class="tech-info">
 <h4>BM25Okapi (rank_bm25)</h4>
 <p>Lexical keyword retriever calculating term frequency / inverse document frequency scores across tokenized corpora.</p>
 </div>
 </div>
<div class="tech-card">
 <div class="tech-icon">📐</div>
 <div class="tech-info">
 <h4>FAISS & Sentence Transformers</h4>
 <p>Dense vector search using <code>all-MiniLM-L6-v2</code> embeddings (384 dimensions) and IndexFlatL2 distance indexing.</p>
 </div>
 </div>
<div class="tech-card">
 <div class="tech-icon">🎯</div>
 <div class="tech-info">
 <h4>Cross-Encoder Reranker</h4>
 <p>High-precision reranking using <code>cross-encoder/ms-marco-MiniLM-L6-v2</code> (22M parameter pairwise cross-attention model).</p>
 </div>
 </div>
<div class="tech-card">
 <div class="tech-icon">📊</div>
 <div class="tech-info">
 <h4>NumPy & SciPy</h4>
 <p>Empirical CDF quantile mapping via <code>np.searchsorted</code>, Pearson correlation r with p-values, and bootstrap resampling.</p>
 </div>
 </div>
 </div>
 </section>
<!-- Architecture & Data Flow Section (INLINE PIPELINE DETAILS) -->
 <section id="architecture" class="section">
 <div class="section-header">
 <span class="section-tag">System Architecture</span>
 <h2 class="section-title">End-to-End Pipeline & Background Data Flow</h2>
 <p class="section-subtitle">Click any pipeline node to expand and inspect its detailed execution logic right beneath that step.</p>
 </div>
<div class="pipeline-container">
 <div class="pipeline-steps">
<!-- Node 1 -->
 <div class="pipeline-node active" data-step="1">
 <div class="node-left">
 <div class="node-number">1</div>
 <div class="node-text">
 <h4>Query Receipt & Tokenization</h4>
 <p>User query hits POST /query endpoint</p>
 </div>
 </div>
 <span class="function-tag">app/main.py</span>
<div class="node-inline-details" style="display: block; margin-top: 1rem; width: 100%;">
 <div class="node-details-box" style="margin-top: 0.5rem;">
 <h3>Step 1: User Query Input & Tokenization</h3>
 <p>The user submits a natural language search query (e.g. "Transformer models rely on multi-head self-attention mechanisms.") to the POST /query endpoint. The query is cleaned and tokenized separately for lexical and dense search streams.</p>
 </div>
 </div>
 </div>
<!-- Node 2 -->
 <div class="pipeline-node" data-step="2">
 <div class="node-left">
 <div class="node-number">2</div>
 <div class="node-text">
 <h4>Dual Parallel Retrieval (BM25 + FAISS)</h4>
 <p>Retrieves candidate_k=20 chunks from both engines</p>
 </div>
 </div>
 <span class="function-tag">app/retriever.py</span>
<div class="node-inline-details" style="display: none; margin-top: 1rem; width: 100%;">
 <div class="node-details-box" style="margin-top: 0.5rem;">
 <h3>Step 2: Dual Parallel Retrieval (Sparse BM25 + Dense FAISS)</h3>
 <p>The query runs through two independent engines in parallel:<br>• <b>BM25 (Sparse)</b>: Scores documents based on term frequency and inverse document frequency (BM25Okapi).<br>• <b>FAISS (Dense)</b>: Encodes query into a 384-dimensional vector using <code>all-MiniLM-L6-v2</code> and performs exhaustive L2 similarity search over chunk vectors.</p>
 </div>
 </div>
 </div>
<!-- Node 3 -->
 <div class="pipeline-node" data-step="3">
 <div class="node-left">
 <div class="node-number">3</div>
 <div class="node-text">
 <h4>Corpus-Level CDF Quantile Calibration</h4>
 <p>Maps raw scores to corpus percentiles via np.searchsorted</p>
 </div>
 </div>
 <span class="function-tag">app/calibration.py</span>
<div class="node-inline-details" style="display: none; margin-top: 1rem; width: 100%;">
 <div class="node-details-box" style="margin-top: 0.5rem;">
 <h3>Step 3: Corpus-Level CDF Calibration</h3>
 <p>Raw scores from BM25 (unbounded, e.g. 0–35) and FAISS (similarity, e.g. 0.3–0.9) cannot be directly compared. I look up each score's empirical quantile rank in offline pre-computed corpus CDF arrays (<code>corpus_cdf_bm25.npy</code> and <code>corpus_cdf_dense.npy</code>) built from 1.7M scores. This maps both distributions to a common [0, 1] probability percentile space.</p>
 </div>
 </div>
 </div>
<!-- Node 4 -->
 <div class="pipeline-node" data-step="4">
 <div class="node-left">
 <div class="node-number">4</div>
 <div class="node-text">
 <h4>Shannon Entropy Computation (H_sparse & H_dense)</h4>
 <p>Measures retrieval uncertainty across score distributions</p>
 </div>
 </div>
 <span class="function-tag">app/calibration.py</span>
<div class="node-inline-details" style="display: none; margin-top: 1rem; width: 100%;">
 <div class="node-details-box" style="margin-top: 0.5rem;">
 <h3>Step 4: Shannon Entropy Computation & Softmax Handling</h3>
 <p>For each retriever's calibrated score distribution over candidate documents, I compute Shannon entropy H = -Σ p_i log2(p_i). High entropy means flat, ambiguous scores (low confidence); low entropy means a sharp peak for top documents (high confidence).</p>
 </div>
 </div>
 </div>
<!-- Node 5 -->
 <div class="pipeline-node" data-step="5">
 <div class="node-left">
 <div class="node-number">5</div>
 <div class="node-text">
 <h4>Adaptive Dynamic Alpha Weight Calculation</h4>
 <p>Computes precision-weighted balance: α = H_dense / (H_dense + H_sparse + ε)</p>
 </div>
 </div>
 <span class="function-tag">app/fusion.py</span>
<div class="node-inline-details" style="display: none; margin-top: 1rem; width: 100%;">
 <div class="node-details-box" style="margin-top: 0.5rem;">
 <h3>Step 5: Adaptive Precision-Weighted Alpha Calculation</h3>
 <p>I calculate the dynamic weighting parameter α:<br><code>α = H_dense / (H_dense + H_sparse + ε)</code><br>If FAISS has high entropy (uncertainty), α increases, giving more weight to lexical BM25. If BM25 has high entropy, α decreases, placing trust in FAISS.</p>
 </div>
 </div>
 </div>
<!-- Node 6 -->
 <div class="pipeline-node" data-step="6">
 <div class="node-left">
 <div class="node-number">6</div>
 <div class="node-text">
 <h4>Weighted Linear Fusion</h4>
 <p>Score = α · BM25_cal + (1-α) · FAISS_cal</p>
 </div>
 </div>
 <span class="function-tag">app/fusion.py</span>
<div class="node-inline-details" style="display: none; margin-top: 1rem; width: 100%;">
 <div class="node-details-box" style="margin-top: 0.5rem;">
 <h3>Step 6: Weighted Linear Score Fusion</h3>
 <p>Candidate document scores are combined: <code>Score = α · S_sparse,cal + (1 - α) · S_dense,cal</code>. The resulting merged candidate list is sorted descending by fused score.</p>
 </div>
 </div>
 </div>
<!-- Node 7 -->
 <div class="pipeline-node" data-step="7">
 <div class="node-left">
 <div class="node-number">7</div>
 <div class="node-text">
 <h4>Cross-Encoder Pairwise Reranking</h4>
 <p>Full cross-attention scoring using ms-marco-MiniLM-L6-v2</p>
 </div>
 </div>
 <span class="function-tag">app/reranker.py</span>
<div class="node-inline-details" style="display: none; margin-top: 1rem; width: 100%;">
 <div class="node-details-box" style="margin-top: 0.5rem;">
 <h3>Step 7: Cross-Encoder Transformer Reranking</h3>
 <p>Top candidate pairs <code>(query, doc_text)</code> are fed into a 22M parameter Cross-Encoder model (<code>ms-marco-MiniLM-L6-v2</code>). Unlike dual-encoders, Cross-Encoders apply full pairwise self-attention across query and document tokens for maximum precision.</p>
 </div>
 </div>
 </div>
<!-- Node 8 -->
 <div class="pipeline-node" data-step="8">
 <div class="node-left">
 <div class="node-number">8</div>
 <div class="node-text">
 <h4>Document ID Deduplication</h4>
 <p>Keeps top-scoring chunk per source document</p>
 </div>
 </div>
 <span class="function-tag">scripts/evaluate.py</span>
<div class="node-inline-details" style="display: none; margin-top: 1rem; width: 100%;">
 <div class="node-details-box" style="margin-top: 0.5rem;">
 <h3>Step 8: Source Document Deduplication</h3>
 <p>Retrieved text chunks are deduplicated by source document ID (keeping the highest-scoring chunk per document) so metrics and final output represent distinct source documents rather than repeating chunks.</p>
 </div>
 </div>
 </div>
<!-- Node 9 -->
 <div class="pipeline-node" data-step="9">
 <div class="node-left">
 <div class="node-number">9</div>
 <div class="node-text">
 <h4>Final Output & Telemetry</h4>
 <p>Returns top-k snippets, sources, scores, latency & entropy metrics</p>
 </div>
 </div>
 <span class="function-tag">app/main.py</span>
<div class="node-inline-details" style="display: none; margin-top: 1rem; width: 100%;">
 <div class="node-details-box" style="margin-top: 0.5rem;">
 <h3>Step 9: Answer Generation & Telemetry Output</h3>
 <p>Returns top-k document snippets, source IDs, relevance scores, cross-encoder scores, and full execution telemetry (latency, α, H_sparse, H_dense, calibration mode).</p>
 </div>
 </div>
 </div>
</div>
 </div>
 </section>
<!-- Data State & Chunking Journey Section -->
 <section id="data-journey" class="section">
 <div class="section-header">
 <span class="section-tag">Data Lifecycle</span>
 <h2 class="section-title">How Data Was Before, How It's Split, & Persistence</h2>
 <p class="section-subtitle">From raw text to FAISS vectors, BM25 inverted indices, and corpus CDF lookup tables.</p>
 </div>
<div class="grid-2">
 <div class="card">
 <h3 class="card-title">📄 1. Input Data Structure</h3>
 <div class="card-body">
 <p><strong>Raw Sources:</strong> Uploaded PDF files, plain text (<code>.txt</code>), markdown (<code>.md</code>), Computer Science documents, or the benchmark SciFact corpus (5,183 PubMed scientific abstracts).</p>
 <p style="margin-top: 0.5rem;"><strong>Before Chunking:</strong> Long documents with full title, abstract, and body text. Raw scores across different length documents would introduce severe length bias without chunking.</p>
 </div>
 </div>
<div class="card">
 <h3 class="card-title">✂️ 2. Recursive Chunking Strategy</h3>
 <div class="card-body">
 <p>Using <code>RecursiveCharacterTextSplitter</code>:</p>
 <ul style="margin-left: 1.2rem; margin-top: 0.4rem;">
 <li><code>chunk_size = 800</code> characters</li>
 <li><code>chunk_overlap = 120</code> characters</li>
 </ul>
 <p style="margin-top: 0.5rem;"><strong>Metadata Attached:</strong> Each chunk is assigned a tuple key <code>(source_doc_id, chunk_index)</code>, e.g. <code>"CS-101:1"</code>.</p>
 </div>
 </div>
 </div>
<div class="grid-3" style="margin-top: 1.5rem;">
 <div class="card">
 <h3 class="card-title">🗂️ 3. FAISS Vector Store</h3>
 <div class="card-body">
 <p>Each chunk is embedded into a 384-dimensional vector using <code>all-MiniLM-L6-v2</code> and saved to <code>data/faiss_index/index.faiss</code>.</p>
 </div>
 </div>
<div class="card">
 <h3 class="card-title">📖 4. BM25 Tokenized Corpus</h3>
 <div class="card-body">
 <p>Chunks are lowercased, stripped of punctuation, whitespace-split, and pickled in <code>bm25_index.pkl</code> for <code>BM25Okapi</code> keyword matching.</p>
 </div>
 </div>
<div class="card">
 <h3 class="card-title">📈 5. Corpus CDF Score Arrays</h3>
 <div class="card-body">
 <p>100 sample queries are scored against ALL 17,243 corpus chunks offline (1,724,300 scores per retriever). Sorted arrays saved to <code>corpus_cdf_bm25.npy</code> and <code>corpus_cdf_dense.npy</code>.</p>
 </div>
 </div>
 </div>
 </section>
<!-- Mathematics Deep-Dive Section -->
 <section id="mathematics" class="section">
 <div class="section-header">
 <span class="section-tag">Mathematical Foundations</span>
 <h2 class="section-title">Exact Formulas & Algorithm Mechanics</h2>
 <p class="section-subtitle">Understanding score calibration, entropy, precision-weighting, and statistical testing.</p>
 </div>
<!-- Formula 1: BM25 -->
 <div class="math-card">
 <h3>1. BM25 Okapi Keyword Scoring</h3>
 <p>BM25 measures term frequency saturation and document length normalization:</p>
 <div class="math-formula">
 $$\text{BM25}(q, D) = \sum_{i=1}^{n} \text{IDF}(q_i) \cdot \frac{f(q_i, D) \cdot (k_1 + 1)}{f(q_i, D) + k_1 \cdot \left(1 - b + b \cdot \frac{|D|}{\text{avgdl}}\right)}$$
 </div>
 <p style="font-size: 0.85rem; color: var(--text-secondary);">Where k<sub>1</sub>=1.5 controls term frequency saturation, b=0.75 controls document length penalty, and IDF(q<sub>i</sub>) = ln((N - n(q<sub>i</sub>) + 0.5) / (n(q<sub>i</sub>) + 0.5) + 1).</p>
 </div>
<!-- Formula 2: FAISS L2 to Similarity -->
 <div class="math-card">
 <h3>2. FAISS Dense L2 Distance to Similarity</h3>
 <p>FAISS <code>IndexFlatL2</code> computes Euclidean distance d<sub>L2</sub>. I convert distance to similarity score in (0, 1]:</p>
 <div class="math-formula">
 $$S_{\text{dense}} = \frac{1}{1 + d_{\text{L2}}}$$
 </div>
 </div>
<!-- Formula 3: Score Normalization Methods -->
 <div class="math-card">
 <h3>3. Corpus-Level Quantile CDF Calibration</h3>
 <p>Per-query min-max destroys global scale information. Instead, I map score s to its percentile rank in the 1.7M corpus score distribution:</p>
 <div class="math-formula">
 $$S_{\text{cdf}}(s) = \frac{\text{searchsorted}(\text{Corpus\_CDF}, s)}{N_{\text{corpus}}}$$
 </div>
 <p style="font-size: 0.85rem; color: var(--text-secondary);">Maps any raw score distribution to approximately uniform [0, 1] quantile space while preserving per-query confidence variation.</p>
 </div>
<!-- Formula 4: Shannon Entropy -->
 <div class="math-card">
 <h3>4. Shannon Entropy of Score Distributions</h3>
 <p>Normalizes scores into a probability mass p<sub>i</sub> = S<sub>i</sub> / Σ S<sub>j</sub>, then calculates Shannon entropy (base 2):</p>
 <div class="math-formula">
 $$H = -\sum_{i=1}^{k} p_i \log_2(p_i)$$
 </div>
 <p style="font-size: 0.85rem; color: var(--text-secondary);"><i>Note:</i> For negative-valued z-scores, softmax is applied: p<sub>i</sub> = e<sup>S<sub>i</sub> - max(S)</sup> / Σ e<sup>S<sub>j</sub> - max(S)</sup> to prevent affine collapse.</p>
 </div>
<!-- Formula 5: Adaptive Alpha -->
 <div class="math-card">
 <h3>5. Bayesian Precision-Weighted Fusion Weight (α)</h3>
 <p>Per-query dynamic weight allocation based on retriever entropy (uncertainty):</p>
 <div class="math-formula">
 $$\alpha = \frac{H_{\text{dense}}}{H_{\text{dense}} + H_{\text{sparse}} + \varepsilon}$$
 $$\text{Fused Score} = \alpha \cdot S_{\text{sparse, cal}} + (1 - \alpha) \cdot S_{\text{dense, cal}}$$
 </div>
 <p style="font-size: 0.85rem; color: var(--text-secondary);">If FAISS dense retrieval has high entropy (uncertainty), α $\to$ 1 (system relies on BM25). If BM25 has high entropy, α $\to$ 0 (system relies on FAISS dense vectors).</p>
 </div>
 </section>
<!-- Interactive Simulator Section -->
 <section id="simulator" class="section">
 <div class="section-header">
 <span class="section-tag">Interactive Playground</span>
 <h2 class="section-title">Live Entropy & Adaptive α Weight Simulator</h2>
 <p class="section-subtitle">Adjust dense and sparse entropy sliders to watch how the system dynamically shifts trust!</p>
 </div>
<div class="simulator-card">
 <div class="grid-2">
 <div class="slider-group">
 <div class="slider-label">
 <span>Dense Retriever Entropy (H<sub>dense</sub>)</span>
 <span id="h-dense-val" class="slider-value">4.20</span>
 </div>
 <input type="range" id="h-dense-slider" min="0" max="6" step="0.05" value="4.20">
 <p style="font-size: 0.75rem; color: var(--text-muted); margin-top: 0.3rem;">Higher value = FAISS vector search is uncertain/confused.</p>
 </div>
<div class="slider-group">
 <div class="slider-label">
 <span>Sparse Retriever Entropy (H<sub>sparse</sub>)</span>
 <span id="h-sparse-val" class="slider-value">4.20</span>
 </div>
 <input type="range" id="h-sparse-slider" min="0" max="6" step="0.05" value="4.20">
 <p style="font-size: 0.75rem; color: var(--text-muted); margin-top: 0.3rem;">Higher value = BM25 keyword search is uncertain/confused.</p>
 </div>
 </div>
<div class="gauge-container">
 <div>
 <div style="font-size: 0.75rem; color: var(--text-muted); text-transform: uppercase;">Calculated Alpha (α)</div>
 <div id="alpha-display" class="alpha-display">0.500</div>
 </div>
<div style="flex: 1;">
 <div style="display: flex; justify-content: space-between; font-size: 0.8rem; margin-bottom: 0.4rem; font-weight: 600;">
 <span style="color: var(--accent-amber);">BM25 Weight: <span id="bm25-percent">50.0%</span></span>
 <span style="color: var(--accent-purple);">FAISS Weight: <span id="dense-percent">50.0%</span></span>
 </div>
 <div class="progress-bar-bg">
 <div id="bar-bm25" class="progress-bar-bm25" style="width: 50%;"></div>
 <div id="bar-dense" class="progress-bar-dense" style="width: 50%;"></div>
 </div>
 </div>
 </div>
<div id="system-state-text" style="margin-top: 1rem; font-size: 0.85rem; color: var(--text-secondary); text-align: center;">
 <strong>Balanced Confidence:</strong> Equal weighting between lexical keywords and vector semantics.
 </div>
 </div>
 </section>
<!-- File & Function Registry Section -->
 <section id="codebase" class="section">
 <div class="section-header">
 <span class="section-tag">Codebase Directory</span>
 <h2 class="section-title">What Each File & Function Does</h2>
 <p class="section-subtitle">Exhaustive guide to all 9 source files and their underlying functions.</p>
 </div>
<input type="text" id="code-search" class="search-box" placeholder="🔎 Search functions or files (e.g. calibrate_cdf, rrf_fuse, DocumentStore)...">
<div class="accordion">
<!-- File 1: app/main.py -->
 <div class="accordion-item">
 <div class="accordion-header">
 <div class="accordion-title-box">
 <span>📄</span>
 <span>app/main.py</span>
 </div>
 <span class="function-tag">FastAPI Endpoints & Lifespan</span>
 </div>
 <div class="accordion-content">
 <p style="margin-bottom: 0.75rem;">FastAPI application defining REST API endpoints and startup handlers.</p>
 <ul style="margin-left: 1.2rem; display: flex; flex-direction: column; gap: 0.5rem;">
 <li><code>lifespan(app)</code>: Async startup context manager that creates upload directory and triggers <code>store.load()</code>.</li>
 <li><code>GET /health</code>: Returns index status, loaded files, and document counts.</li>
 <li><code>GET /modes</code>: Lists all 7 available retrieval modes with descriptions.</li>
 <li><code>POST /upload</code>: Accepts PDF, TXT, MD files, saves to <code>data/uploads/</code>, and indexes chunks.</li>
 <li><code>POST /query</code>: Main search endpoint accepting question, top_k, and retrieval mode.</li>
 </ul>
 </div>
 </div>
<!-- File 2: app/retriever.py -->
 <div class="accordion-item">
 <div class="accordion-header">
 <div class="accordion-title-box">
 <span>📄</span>
 <span>app/retriever.py</span>
 </div>
 <span class="function-tag">Pipeline Orchestrator & DocumentStore</span>
 </div>
 <div class="accordion-content">
 <p style="margin-bottom: 0.75rem;">Central document manager and mode router combining FAISS, BM25, and Cross-Encoder.</p>
 <ul style="margin-left: 1.2rem; display: flex; flex-direction: column; gap: 0.5rem;">
 <li><code>DocumentStore.load()</code>: Loads FAISS vectorstore, BM25 pickle index, CDF arrays, and text lookup map from disk.</li>
 <li><code>DocumentStore.add_file(path)</code>: Extracts text from PDF/TXT/MD, chunks text, and builds indices.</li>
 <li><code>DocumentStore._search_dense(query, top_k)</code>: Runs FAISS similarity search and converts L2 distance to [0, 1] similarity.</li>
 <li><code>DocumentStore._search_sparse(query, top_k)</code>: Runs BM25 tokenized keyword search.</li>
 <li><code>DocumentStore.search(query, top_k, mode)</code>: Routes search to one of 7 modes and records latency telemetry.</li>
 <li><code>extract_text(path)</code>: Extracts raw text from PDF using <code>pypdf.PdfReader</code> or UTF-8 text files.</li>
 <li><code>chunk_text(text, source)</code>: Splits text with <code>RecursiveCharacterTextSplitter(800, 120)</code>.</li>
 </ul>
 </div>
 </div>
<!-- File 3: app/calibration.py -->
 <div class="accordion-item">
 <div class="accordion-header">
 <div class="accordion-title-box">
 <span>📄</span>
 <span>app/calibration.py</span>
 </div>
 <span class="function-tag">Score Calibration, Entropy & Statistics</span>
 </div>
 <div class="accordion-content">
 <p style="margin-bottom: 0.75rem;">Research math module handling distribution transforms, Shannon entropy, QPP baselines, and bootstrap significance testing.</p>
 <ul style="margin-left: 1.2rem; display: flex; flex-direction: column; gap: 0.5rem;">
 <li><code>calibrate_raw(scores)</code>: Identity baseline.</li>
 <li><code>calibrate_minmax(scores)</code>: Scales scores to [0, 1] per query.</li>
 <li><code>calibrate_zscore(scores)</code>: Standardizes scores to mean=0, std=1.</li>
 <li><code>calibrate_cdf(scores, corpus_cdf)</code>: Maps scores to corpus percentile ranks using <code>np.searchsorted</code>.</li>
 <li><code>compute_entropy(scores)</code>: Computes Shannon entropy H = -Σ p log2 p with softmax handling for negative z-scores.</li>
 <li><code>compute_alpha(h_dense, h_sparse)</code>: Calculates dynamic precision weight α = H_dense / (H_dense + H_sparse + ε).</li>
 <li><code>build_corpus_cdfs(...)</code>: Scores sample queries against ALL corpus docs to create offline CDF arrays.</li>
 <li><code>clarity_score(top_k_texts, corpus_freqs)</code>: Computes KL divergence of top-k language model vs corpus language model.</li>
 <li><code>nqc(scores, corpus_mean)</code>: Normalized Query Commitment std(top_k) / |mean(corpus)|.</li>
 <li><code>bootstrap_ci(a, b, n_resamples=1000)</code>: Performs paired bootstrap resampling for 95% confidence intervals and p-value significance.</li>
 </ul>
 </div>
 </div>
<!-- File 4: app/fusion.py -->
 <div class="accordion-item">
 <div class="accordion-header">
 <div class="accordion-title-box">
 <span>📄</span>
 <span>app/fusion.py</span>
 </div>
 <span class="function-tag">RRF & Linear Score Fusion</span>
 </div>
 <div class="accordion-content">
 <p style="margin-bottom: 0.75rem;">Implements reciprocal rank fusion, fixed linear fusion, and calibrated entropy-weighted fusion.</p>
 <ul style="margin-left: 1.2rem; display: flex; flex-direction: column; gap: 0.5rem;">
 <li><code>rrf_fuse(sparse, dense, k=60)</code>: Combines candidate ranks via RRF(d) = Σ 1 / (60 + r(d)).</li>
 <li><code>linear_fuse(sparse, dense, alpha=0.5)</code>: Merges candidates with fixed α=0.5 linear combination.</li>
 <li><code>entropy_fuse(sparse, dense, calibration="cdf")</code>: Calibrates scores, computes Shannon entropy per retriever, calculates dynamic α, and merges candidates.</li>
 </ul>
 </div>
 </div>
<!-- File 5: app/sparse_retriever.py -->
 <div class="accordion-item">
 <div class="accordion-header">
 <div class="accordion-title-box">
 <span>📄</span>
 <span>app/sparse_retriever.py</span>
 </div>
 <span class="function-tag">BM25 Index & Search</span>
 </div>
 <div class="accordion-content">
 <p style="margin-bottom: 0.75rem;">Manages BM25Okapi index and full-corpus scoring.</p>
 <ul style="margin-left: 1.2rem; display: flex; flex-direction: column; gap: 0.5rem;">
 <li><code>BM25Index.tokenize(text)</code>: Lowercases, strips punctuation, and splits text into tokens.</li>
 <li><code>BM25Index.add_documents(docs)</code>: Tokenizes documents and builds <code>BM25Okapi</code> model.</li>
 <li><code>BM25Index.score_all(query)</code>: Returns BM25 scores across all corpus documents (used for offline CDF building).</li>
 </ul>
 </div>
 </div>
<!-- File 6: app/reranker.py -->
 <div class="accordion-item">
 <div class="accordion-header">
 <div class="accordion-title-box">
 <span>📄</span>
 <span>app/reranker.py</span>
 </div>
 <span class="function-tag">Cross-Encoder Transformer</span>
 </div>
 <div class="accordion-content">
 <p style="margin-bottom: 0.75rem;">Lazy-loads <code>cross-encoder/ms-marco-MiniLM-L6-v2</code> for pairwise (query, document) scoring.</p>
 <ul style="margin-left: 1.2rem; display: flex; flex-direction: column; gap: 0.5rem;">
 <li><code>CrossEncoderReranker.rerank(query, candidates, top_k)</code>: Runs pairwise cross-attention prediction and sorts candidates descending by score.</li>
 </ul>
 </div>
 </div>
<!-- File 7: app/datasets.py -->
 <div class="accordion-item">
 <div class="accordion-header">
 <div class="accordion-title-box">
 <span>📄</span>
 <span>app/datasets.py</span>
 </div>
 <span class="function-tag">SciFact Dataset Loader</span>
 </div>
 <div class="accordion-content">
 <p style="margin-bottom: 0.75rem;">Loads BEIR SciFact benchmark corpus (5,183 PubMed abstracts), test claims (300 queries), and qrels relevancy labels.</p>
 <ul style="margin-left: 1.2rem; display: flex; flex-direction: column; gap: 0.5rem;">
 <li><code>load_scifact_corpus()</code>: Parses <code>corpus.jsonl</code> into document records.</li>
 <li><code>load_scifact_queries()</code>: Parses <code>queries.jsonl</code> mapping query IDs to claim text.</li>
 <li><code>load_scifact_qrels()</code>: Loads ground-truth relevance pairs from <code>qrels/test.tsv</code>.</li>
 </ul>
 </div>
 </div>
<!-- File 8: scripts/build_index.py -->
 <div class="accordion-item">
 <div class="accordion-header">
 <div class="accordion-title-box">
 <span>📄</span>
 <span>scripts/build_index.py</span>
 </div>
 <span class="function-tag">Offline Index & CDF Generator</span>
 </div>
 <div class="accordion-content">
 <p style="margin-bottom: 0.75rem;">Offline script executing indexing step, generating FAISS vectors, BM25 indices, unigram language models, document lookup tables, and corpus-level CDF numpy files.</p>
 </div>
 </div>
<!-- File 9: scripts/evaluate.py -->
 <div class="accordion-item">
 <div class="accordion-header">
 <div class="accordion-title-box">
 <span>📄</span>
 <span>scripts/evaluate.py</span>
 </div>
 <span class="function-tag">Scientific Evaluation & H1/H2 Benchmark</span>
 </div>
 <div class="accordion-content">
 <p style="margin-bottom: 0.75rem;">Evaluates 7 retrieval modes across 300 test claims, computes Pearson correlations for H1, runs 1,000 paired bootstrap resamples for H2, and deduplicates chunk results by document ID.</p>
 </div>
 </div>
</div>
 </section>
<!-- Research Findings & Benchmarks Section -->
 <section id="benchmarks" class="section">
 <div class="section-header">
 <span class="section-tag">Scientific Evidence</span>
 <h2 class="section-title">BEIR SciFact Benchmark Results & Research Hypotheses</h2>
 <p class="section-subtitle">Evaluating 7 retrieval modes across 300 test claims (5,183 PubMed abstracts).</p>
 </div>
<table class="data-table">
 <thead>
 <tr>
 <th>Mode</th>
 <th>NDCG@10</th>
 <th>MRR</th>
 <th>P@3</th>
 <th>P@5</th>
 <th>R@5</th>
 <th>p95 Latency</th>
 </tr>
 </thead>
 <tbody>
 <tr>
 <td><code>dense</code></td>
 <td>0.6715</td>
 <td>0.6329</td>
 <td>0.2466</td>
 <td>0.1647</td>
 <td>0.7495</td>
 <td>33 ms</td>
 </tr>
 <tr>
 <td><code>sparse</code></td>
 <td>0.6151</td>
 <td>0.5804</td>
 <td>0.2233</td>
 <td>0.1520</td>
 <td>0.7089</td>
 <td>130 ms</td>
 </tr>
 <tr>
 <td><code>rrf</code></td>
 <td>0.7028</td>
 <td>0.6713</td>
 <td>0.2555</td>
 <td>0.1680</td>
 <td>0.7744</td>
 <td>249 ms</td>
 </tr>
 <tr>
 <td><code>hybrid_fixed</code></td>
 <td>0.6829</td>
 <td>0.6527</td>
 <td>0.2522</td>
 <td>0.1647</td>
 <td>0.7594</td>
 <td>232 ms</td>
 </tr>
 <tr>
 <td><code>hybrid_calibrated</code></td>
 <td>0.6981</td>
 <td>0.6672</td>
 <td>0.2489</td>
 <td>0.1660</td>
 <td>0.7661</td>
 <td>241 ms</td>
 </tr>
 <tr>
 <td><code>hybrid_fixed_rerank</code></td>
 <td>0.7006</td>
 <td>0.6653</td>
 <td>0.2578</td>
 <td>0.1700</td>
 <td>0.7714</td>
 <td>2432 ms</td>
 </tr>
 <tr class="highlight-row">
 <td><strong>hybrid_calibrated_rerank</strong></td>
 <td><strong>0.7071</strong></td>
 <td><strong>0.6719</strong></td>
 <td><strong>0.2622</strong></td>
 <td><strong>0.1720</strong></td>
 <td><strong>0.7838</strong></td>
 <td>2116 ms</td>
 </tr>
 </tbody>
 </table>
<div class="grid-2" style="margin-top: 1.5rem;">
 <div class="card">
 <h3 class="card-title">🔬 H1: Entropy Correlation Analysis</h3>
 <div class="card-body">
 <p><strong>Hypothesis:</strong> CDF entropy has the strongest negative correlation with retrieval quality.</p>
 <p style="margin-top: 0.5rem; color: var(--accent-rose);">❌ <strong>Verdict: Not Supported.</strong></p>
 <p style="margin-top: 0.4rem; font-size: 0.85rem;">Z-score entropy turned out to be the strongest negative predictor (r = -0.4065, p &lt; 10<sup>-6</sup>) for dense retrieval on SciFact, whereas CDF entropy showed a weaker correlation (r = -0.1151).</p>
 </div>
 </div>
<div class="card">
 <h3 class="card-title">📊 H2: Paired Bootstrap Significance</h3>
 <div class="card-body">
 <p><strong>Hypothesis:</strong> Calibrated entropy fusion significantly outperforms fixed-alpha and RRF baselines.</p>
 <p style="margin-top: 0.5rem; color: var(--accent-amber);">⚠️ <strong>Verdict: Partially Supported.</strong></p>
 <p style="margin-top: 0.4rem; font-size: 0.85rem;">CDF entropy fusion significantly beats fixed-alpha fusion (+0.0152 NDCG@10 gain, p = 0.012), but ties with RRF. With cross-encoder reranking, the full pipeline achieves the best overall NDCG@10 (0.7071).</p>
 </div>
 </div>
 </div>
 </section>
<!-- How to Run & Deploy Section -->
 <section id="how-to-run" class="section">
 <div class="section-header">
 <span class="section-tag">Deployment & Setup Guide</span>
 <h2 class="section-title">How to Run & Deploy</h2>
 <p class="section-subtitle">Commands to build indices, run FastAPI, execute pytest, and evaluate performance.</p>
 </div>
<div class="card" style="margin-bottom: 1.5rem;">
 <h3 class="card-title">1. Local Setup & Installation</h3>
 <pre><code># Create virtual environment
python -m venv venv
# Activate venv (Windows PowerShell)
.\venv\Scripts\Activate.ps1
# Install requirements
pip install -r requirements.txt</code></pre>
 </div>
<div class="card" style="margin-bottom: 1.5rem;">
 <h3 class="card-title">2. Build Retrieval Artifacts & Index SciFact</h3>
 <pre><code># Builds FAISS, BM25, Corpus CDFs, and Document Lookup tables
python scripts/build_index.py --dataset scifact</code></pre>
 </div>
<div class="card" style="margin-bottom: 1.5rem;">
 <h3 class="card-title">3. Start the FastAPI Server & Interactive Swagger Docs</h3>
 <pre><code># Launch server at http://127.0.0.1:8000
uvicorn app.main:app --reload
# Open Swagger Docs in your browser: http://127.0.0.1:8000/docs</code></pre>
 </div>
<div class="card" style="margin-bottom: 1.5rem;">
 <h3 class="card-title">4. Run Pytest Test Suite</h3>
 <pre><code># Executes all 9 test cases covering API endpoints, telemetry, and calibration
pytest tests/ -v</code></pre>
 </div>
<div class="card">
 <h3 class="card-title">5. Run Benchmark Evaluation & Significance Tests</h3>
 <pre><code># Evaluates all 7 modes on SciFact test claims & runs bootstrap tests
python scripts/evaluate.py</code></pre>
 </div>
 </section>
</main>
</div>
<!-- JS Engine -->
<script src="understanding of project/app.js"></script>
</body>
</html>