Spaces:
Running
Running
File size: 46,523 Bytes
ed65693 b8a611f ed65693 b8a611f ed65693 b8a611f ed65693 b8a611f ed65693 b8a611f ed65693 b8a611f ed65693 b8a611f ed65693 b8a611f ed65693 b8a611f ed65693 b8a611f ed65693 b8a611f f28ebaf b8a611f ed65693 b8a611f ed65693 b8a611f ed65693 b8a611f ed65693 b8a611f ed65693 b8a611f ed65693 b8a611f ed65693 b8a611f ed65693 b8a611f ed65693 b8a611f ed65693 b8a611f ed65693 b8a611f ed65693 b8a611f ed65693 b8a611f ed65693 b8a611f ed65693 b8a611f ed65693 b8a611f ed65693 b8a611f ed65693 b8a611f ed65693 b8a611f ed65693 b8a611f ed65693 b8a611f ed65693 b8a611f ed65693 b8a611f ed65693 b8a611f ed65693 b8a611f ed65693 b8a611f ed65693 b8a611f ed65693 b8a611f ed65693 b8a611f ed65693 b8a611f ed65693 b8a611f ed65693 b8a611f ed65693 b8a611f ed65693 b8a611f ed65693 b8a611f ed65693 b8a611f ed65693 b8a611f ed65693 b8a611f ed65693 b8a611f ed65693 b8a611f ed65693 b8a611f ed65693 b8a611f ed65693 b8a611f ed65693 b8a611f ed65693 b8a611f ed65693 b8a611f ed65693 b8a611f ed65693 b8a611f ed65693 b8a611f ed65693 b8a611f ed65693 b8a611f ed65693 b8a611f ed65693 b8a611f ed65693 b8a611f ed65693 b8a611f ed65693 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 | <!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Calibrated Entropy-Weighted Hybrid Retrieval — Technical Guide & Operational Engine</title>
<link rel="preconnect" href="https://fonts.googleapis.com">
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
<link href="https://fonts.googleapis.com/css2?family=Inter:wght@400;500;600;700;800&family=JetBrains+Mono:wght@400;500;700&display=swap" rel="stylesheet">
<link rel="stylesheet" href="style.css">
<!-- MathJax for LaTeX Rendering -->
<script src="https://polyfill.io/v3/polyfill.min.js?features=es6"></script>
<script id="MathJax-script" async src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js"></script>
</head>
<body>
<div class="app-container">
<!-- Sidebar Navigation -->
<nav class="sidebar">
<div class="sidebar-brand">
<div class="logo-icon">🔬</div>
<div class="sidebar-brand-text">
<h1>Hybrid RAG System</h1>
<p>Project Technical Guide</p>
</div>
</div>
<div class="nav-section-label">Navigation</div>
<ul class="nav-menu">
<li><a href="#overview" class="nav-link active"><span class="icon">🚀</span> Overview</a></li>
<li><a href="#live-demo" class="nav-link"><span class="icon">🎮</span> Operational Retrieval Engine</a></li>
<li><a href="#techstack" class="nav-link"><span class="icon">⚡</span> Tech Stack</a></li>
<li><a href="#architecture" class="nav-link"><span class="icon">🏗️</span> Pipeline & Data Flow</a></li>
<li><a href="#data-journey" class="nav-link"><span class="icon">📦</span> Data State & Chunking</a></li>
<li><a href="#mathematics" class="nav-link"><span class="icon">📐</span> Mathematics Deep-Dive</a></li>
<li><a href="#simulator" class="nav-link"><span class="icon">🎛️</span> Interactive Simulator</a></li>
<li><a href="#codebase" class="nav-link"><span class="icon">📁</span> File & Function Registry</a></li>
<li><a href="#benchmarks" class="nav-link"><span class="icon">📊</span> Research Results & H1/H2</a></li>
<li><a href="#how-to-run" class="nav-link"><span class="icon">🛠️</span> How to Run & Deploy</a></li>
</ul>
</nav>
<!-- Main Content -->
<main class="main-content">
<!-- Hero Header -->
<header id="overview" class="hero section">
<div class="hero-badge">🔬 Calibrated Hybrid RAG Architecture</div>
<h1 class="hero-title">Calibrated Entropy-Weighted Hybrid Retrieval</h1>
<p class="hero-description">
I built this research-oriented hybrid retrieval system to test whether score calibration makes retrieval uncertainty measurable and useful for search performance. This interactive masterclass explains every architectural decision, mathematical algorithm, and implementation detail I built into the system.
</p>
<div class="hero-stats">
<div class="stat-item">
<h3>7 Modes</h3>
<p>Retrieval Ablation</p>
</div>
<div class="stat-item">
<h3>Computer Science</h3>
<p>& Science Data</p>
</div>
<div class="stat-item">
<h3>Strict Guard</h3>
<p>Low-Confidence Rejection</p>
</div>
<div class="stat-item">
<h3>0.7071</h3>
<p>Peak NDCG@10 Score</p>
</div>
</div>
</header>
<!-- OPERATIONAL HYBRID RETRIEVAL ENGINE DEMO -->
<section id="live-demo" class="section">
<div class="section-header">
<span class="section-tag">Operational Engine</span>
<h2 class="section-title">🎮 Operational Retrieval Engine & Live Queries</h2>
<p class="section-subtitle">Test the actual hybrid retrieval pipeline I built across Computer Science & Biomedical corpora. If top candidates fall below minimum confidence (Score < 0.15), the system explicitly rejects the result to prevent hallucination.</p>
</div>
<div class="simulator-card">
<div class="grid-2">
<div>
<label style="font-weight: 600; font-size: 0.85rem; display: block; margin-bottom: 0.4rem;">Search Question / Claim</label>
<input type="text" id="demo-query-input" class="search-box" value="Transformer models rely on multi-head self-attention mechanisms." style="margin-bottom: 0;">
</div>
<div>
<label style="font-weight: 600; font-size: 0.85rem; display: block; margin-bottom: 0.4rem;">Retrieval Mode</label>
<select id="demo-mode-select" class="search-box" style="margin-bottom: 0; cursor: pointer; background: var(--bg-card);">
<option value="hybrid_calibrated_rerank" selected>hybrid_calibrated_rerank (Full Calibrated Pipeline + Reranker)</option>
<option value="hybrid_calibrated">hybrid_calibrated (CDF Calibration + Dynamic Entropy Alpha)</option>
<option value="hybrid_fixed_rerank">hybrid_fixed_rerank (Min-Max Calibration + Reranker)</option>
<option value="hybrid_fixed">hybrid_fixed (Min-Max Calibration + Fixed α=0.5)</option>
<option value="rrf">rrf (Reciprocal Rank Fusion k=60)</option>
<option value="sparse">sparse (BM25 Lexical Keyword Search)</option>
<option value="dense">dense (FAISS Dense Vector Similarity)</option>
</select>
</div>
</div>
<div style="margin-top: 0.75rem; display: flex; gap: 0.5rem; flex-wrap: wrap;">
<span style="font-size: 0.75rem; color: var(--text-muted); font-weight: 600;">Sample Queries:</span>
<button class="function-tag" onclick="document.getElementById('demo-query-input').value='B-Tree indexing and cost-based query optimizer'; document.getElementById('demo-mode-select').value='sparse'; document.getElementById('run-demo-btn').click();" style="cursor: pointer; background: rgba(56, 189, 248, 0.15); color: var(--accent-cyan);">🔍 BM25 Lexical Match Example</button>
<button class="function-tag" onclick="document.getElementById('demo-query-input').value='How do neural agents learn from experience replay?'; document.getElementById('demo-mode-select').value='dense'; document.getElementById('run-demo-btn').click();" style="cursor: pointer; background: rgba(168, 85, 247, 0.15); color: var(--accent-purple);">📐 FAISS Dense Vector Example</button>
<button class="function-tag" onclick="document.getElementById('demo-query-input').value='Transformer models rely on multi-head self-attention mechanisms.'; document.getElementById('demo-mode-select').value='hybrid_calibrated_rerank'; document.getElementById('run-demo-btn').click();" style="cursor: pointer; background: rgba(52, 211, 153, 0.15); color: var(--accent-emerald);">⚡ Full Hybrid + Reranker</button>
<button class="function-tag" onclick="document.getElementById('demo-query-input').value='Random irrelevant cooking recipe.'; document.getElementById('demo-mode-select').value='hybrid_calibrated_rerank'; document.getElementById('run-demo-btn').click();" style="cursor: pointer; background: rgba(244, 63, 94, 0.15); color: var(--accent-rose);">🚫 Test Low Confidence Rejection</button>
</div>
<button id="run-demo-btn" style="margin-top: 1.25rem; width: 100%; padding: 0.85rem; background: linear-gradient(135deg, var(--accent-cyan), var(--accent-blue)); border: none; border-radius: 8px; color: white; font-weight: 700; font-size: 0.95rem; cursor: pointer; box-shadow: 0 4px 12px rgba(56, 189, 248, 0.3);">
⚡ Execute Retrieval & Compute Telemetry
</button>
<!-- Live Telemetry Display -->
<div id="demo-telemetry-box" style="margin-top: 1.5rem; padding: 1rem; background: rgba(0, 0, 0, 0.4); border: 1px solid var(--border-color); border-radius: 8px; font-family: var(--font-mono); font-size: 0.85rem;">
<!-- Populated by JavaScript -->
</div>
<!-- Ranked Results Output -->
<div id="demo-results-container" style="margin-top: 1.5rem;">
<!-- Populated by JavaScript -->
</div>
</div>
</section>
<!-- Tech Stack Section -->
<section id="techstack" class="section">
<div class="section-header">
<span class="section-tag">Technology Stack</span>
<h2 class="section-title">What I Built & Core Technologies</h2>
<p class="section-subtitle">Every library, framework, and model I integrated into the system.</p>
</div>
<div class="tech-grid">
<div class="tech-card">
<div class="tech-icon">🐍</div>
<div class="tech-info">
<h4>Python 3.11</h4>
<p>Core runtime language, typing system, async context management, and fast scientific computation.</p>
</div>
</div>
<div class="tech-card">
<div class="tech-icon">⚡</div>
<div class="tech-info">
<h4>FastAPI & Uvicorn</h4>
<p>High-performance web API framework with automatic Pydantic request validation and interactive Swagger UI docs.</p>
</div>
</div>
<div class="tech-card">
<div class="tech-icon">🔍</div>
<div class="tech-info">
<h4>BM25Okapi (rank_bm25)</h4>
<p>Lexical keyword retriever calculating term frequency / inverse document frequency scores across tokenized corpora.</p>
</div>
</div>
<div class="tech-card">
<div class="tech-icon">📐</div>
<div class="tech-info">
<h4>FAISS & Sentence Transformers</h4>
<p>Dense vector search using <code>all-MiniLM-L6-v2</code> embeddings (384 dimensions) and IndexFlatL2 distance indexing.</p>
</div>
</div>
<div class="tech-card">
<div class="tech-icon">🎯</div>
<div class="tech-info">
<h4>Cross-Encoder Reranker</h4>
<p>High-precision reranking using <code>cross-encoder/ms-marco-MiniLM-L6-v2</code> (22M parameter pairwise cross-attention model).</p>
</div>
</div>
<div class="tech-card">
<div class="tech-icon">📊</div>
<div class="tech-info">
<h4>NumPy & SciPy</h4>
<p>Empirical CDF quantile mapping via <code>np.searchsorted</code>, Pearson correlation r with p-values, and bootstrap resampling.</p>
</div>
</div>
</div>
</section>
<!-- Architecture & Data Flow Section (INLINE PIPELINE DETAILS) -->
<section id="architecture" class="section">
<div class="section-header">
<span class="section-tag">System Architecture</span>
<h2 class="section-title">End-to-End Pipeline & Background Data Flow</h2>
<p class="section-subtitle">Click any pipeline node to expand and inspect its detailed execution logic right beneath that step.</p>
</div>
<div class="pipeline-container">
<div class="pipeline-steps">
<!-- Node 1 -->
<div class="pipeline-node active" data-step="1">
<div class="node-left">
<div class="node-number">1</div>
<div class="node-text">
<h4>Query Receipt & Tokenization</h4>
<p>User query hits POST /query endpoint</p>
</div>
</div>
<span class="function-tag">app/main.py</span>
<div class="node-inline-details" style="display: block; margin-top: 1rem; width: 100%;">
<div class="node-details-box" style="margin-top: 0.5rem;">
<h3>Step 1: User Query Input & Tokenization</h3>
<p>The user submits a natural language search query (e.g. "Transformer models rely on multi-head self-attention mechanisms.") to the POST /query endpoint. The query is cleaned and tokenized separately for lexical and dense search streams.</p>
</div>
</div>
</div>
<!-- Node 2 -->
<div class="pipeline-node" data-step="2">
<div class="node-left">
<div class="node-number">2</div>
<div class="node-text">
<h4>Dual Parallel Retrieval (BM25 + FAISS)</h4>
<p>Retrieves candidate_k=20 chunks from both engines</p>
</div>
</div>
<span class="function-tag">app/retriever.py</span>
<div class="node-inline-details" style="display: none; margin-top: 1rem; width: 100%;">
<div class="node-details-box" style="margin-top: 0.5rem;">
<h3>Step 2: Dual Parallel Retrieval (Sparse BM25 + Dense FAISS)</h3>
<p>The query runs through two independent engines in parallel:<br>• <b>BM25 (Sparse)</b>: Scores documents based on term frequency and inverse document frequency (BM25Okapi).<br>• <b>FAISS (Dense)</b>: Encodes query into a 384-dimensional vector using <code>all-MiniLM-L6-v2</code> and performs exhaustive L2 similarity search over chunk vectors.</p>
</div>
</div>
</div>
<!-- Node 3 -->
<div class="pipeline-node" data-step="3">
<div class="node-left">
<div class="node-number">3</div>
<div class="node-text">
<h4>Corpus-Level CDF Quantile Calibration</h4>
<p>Maps raw scores to corpus percentiles via np.searchsorted</p>
</div>
</div>
<span class="function-tag">app/calibration.py</span>
<div class="node-inline-details" style="display: none; margin-top: 1rem; width: 100%;">
<div class="node-details-box" style="margin-top: 0.5rem;">
<h3>Step 3: Corpus-Level CDF Calibration</h3>
<p>Raw scores from BM25 (unbounded, e.g. 0–35) and FAISS (similarity, e.g. 0.3–0.9) cannot be directly compared. I look up each score's empirical quantile rank in offline pre-computed corpus CDF arrays (<code>corpus_cdf_bm25.npy</code> and <code>corpus_cdf_dense.npy</code>) built from 1.7M scores. This maps both distributions to a common [0, 1] probability percentile space.</p>
</div>
</div>
</div>
<!-- Node 4 -->
<div class="pipeline-node" data-step="4">
<div class="node-left">
<div class="node-number">4</div>
<div class="node-text">
<h4>Shannon Entropy Computation (H_sparse & H_dense)</h4>
<p>Measures retrieval uncertainty across score distributions</p>
</div>
</div>
<span class="function-tag">app/calibration.py</span>
<div class="node-inline-details" style="display: none; margin-top: 1rem; width: 100%;">
<div class="node-details-box" style="margin-top: 0.5rem;">
<h3>Step 4: Shannon Entropy Computation & Softmax Handling</h3>
<p>For each retriever's calibrated score distribution over candidate documents, I compute Shannon entropy H = -Σ p_i log2(p_i). High entropy means flat, ambiguous scores (low confidence); low entropy means a sharp peak for top documents (high confidence).</p>
</div>
</div>
</div>
<!-- Node 5 -->
<div class="pipeline-node" data-step="5">
<div class="node-left">
<div class="node-number">5</div>
<div class="node-text">
<h4>Adaptive Dynamic Alpha Weight Calculation</h4>
<p>Computes precision-weighted balance: α = H_dense / (H_dense + H_sparse + ε)</p>
</div>
</div>
<span class="function-tag">app/fusion.py</span>
<div class="node-inline-details" style="display: none; margin-top: 1rem; width: 100%;">
<div class="node-details-box" style="margin-top: 0.5rem;">
<h3>Step 5: Adaptive Precision-Weighted Alpha Calculation</h3>
<p>I calculate the dynamic weighting parameter α:<br><code>α = H_dense / (H_dense + H_sparse + ε)</code><br>If FAISS has high entropy (uncertainty), α increases, giving more weight to lexical BM25. If BM25 has high entropy, α decreases, placing trust in FAISS.</p>
</div>
</div>
</div>
<!-- Node 6 -->
<div class="pipeline-node" data-step="6">
<div class="node-left">
<div class="node-number">6</div>
<div class="node-text">
<h4>Weighted Linear Fusion</h4>
<p>Score = α · BM25_cal + (1-α) · FAISS_cal</p>
</div>
</div>
<span class="function-tag">app/fusion.py</span>
<div class="node-inline-details" style="display: none; margin-top: 1rem; width: 100%;">
<div class="node-details-box" style="margin-top: 0.5rem;">
<h3>Step 6: Weighted Linear Score Fusion</h3>
<p>Candidate document scores are combined: <code>Score = α · S_sparse,cal + (1 - α) · S_dense,cal</code>. The resulting merged candidate list is sorted descending by fused score.</p>
</div>
</div>
</div>
<!-- Node 7 -->
<div class="pipeline-node" data-step="7">
<div class="node-left">
<div class="node-number">7</div>
<div class="node-text">
<h4>Cross-Encoder Pairwise Reranking</h4>
<p>Full cross-attention scoring using ms-marco-MiniLM-L6-v2</p>
</div>
</div>
<span class="function-tag">app/reranker.py</span>
<div class="node-inline-details" style="display: none; margin-top: 1rem; width: 100%;">
<div class="node-details-box" style="margin-top: 0.5rem;">
<h3>Step 7: Cross-Encoder Transformer Reranking</h3>
<p>Top candidate pairs <code>(query, doc_text)</code> are fed into a 22M parameter Cross-Encoder model (<code>ms-marco-MiniLM-L6-v2</code>). Unlike dual-encoders, Cross-Encoders apply full pairwise self-attention across query and document tokens for maximum precision.</p>
</div>
</div>
</div>
<!-- Node 8 -->
<div class="pipeline-node" data-step="8">
<div class="node-left">
<div class="node-number">8</div>
<div class="node-text">
<h4>Document ID Deduplication</h4>
<p>Keeps top-scoring chunk per source document</p>
</div>
</div>
<span class="function-tag">scripts/evaluate.py</span>
<div class="node-inline-details" style="display: none; margin-top: 1rem; width: 100%;">
<div class="node-details-box" style="margin-top: 0.5rem;">
<h3>Step 8: Source Document Deduplication</h3>
<p>Retrieved text chunks are deduplicated by source document ID (keeping the highest-scoring chunk per document) so metrics and final output represent distinct source documents rather than repeating chunks.</p>
</div>
</div>
</div>
<!-- Node 9 -->
<div class="pipeline-node" data-step="9">
<div class="node-left">
<div class="node-number">9</div>
<div class="node-text">
<h4>Final Output & Telemetry</h4>
<p>Returns top-k snippets, sources, scores, latency & entropy metrics</p>
</div>
</div>
<span class="function-tag">app/main.py</span>
<div class="node-inline-details" style="display: none; margin-top: 1rem; width: 100%;">
<div class="node-details-box" style="margin-top: 0.5rem;">
<h3>Step 9: Answer Generation & Telemetry Output</h3>
<p>Returns top-k document snippets, source IDs, relevance scores, cross-encoder scores, and full execution telemetry (latency, α, H_sparse, H_dense, calibration mode).</p>
</div>
</div>
</div>
</div>
</div>
</section>
<!-- Data State & Chunking Journey Section -->
<section id="data-journey" class="section">
<div class="section-header">
<span class="section-tag">Data Lifecycle</span>
<h2 class="section-title">How Data Was Before, How It's Split, & Persistence</h2>
<p class="section-subtitle">From raw text to FAISS vectors, BM25 inverted indices, and corpus CDF lookup tables.</p>
</div>
<div class="grid-2">
<div class="card">
<h3 class="card-title">📄 1. Input Data Structure</h3>
<div class="card-body">
<p><strong>Raw Sources:</strong> Uploaded PDF files, plain text (<code>.txt</code>), markdown (<code>.md</code>), Computer Science documents, or the benchmark SciFact corpus (5,183 PubMed scientific abstracts).</p>
<p style="margin-top: 0.5rem;"><strong>Before Chunking:</strong> Long documents with full title, abstract, and body text. Raw scores across different length documents would introduce severe length bias without chunking.</p>
</div>
</div>
<div class="card">
<h3 class="card-title">✂️ 2. Recursive Chunking Strategy</h3>
<div class="card-body">
<p>Using <code>RecursiveCharacterTextSplitter</code>:</p>
<ul style="margin-left: 1.2rem; margin-top: 0.4rem;">
<li><code>chunk_size = 800</code> characters</li>
<li><code>chunk_overlap = 120</code> characters</li>
</ul>
<p style="margin-top: 0.5rem;"><strong>Metadata Attached:</strong> Each chunk is assigned a tuple key <code>(source_doc_id, chunk_index)</code>, e.g. <code>"CS-101:1"</code>.</p>
</div>
</div>
</div>
<div class="grid-3" style="margin-top: 1.5rem;">
<div class="card">
<h3 class="card-title">🗂️ 3. FAISS Vector Store</h3>
<div class="card-body">
<p>Each chunk is embedded into a 384-dimensional vector using <code>all-MiniLM-L6-v2</code> and saved to <code>data/faiss_index/index.faiss</code>.</p>
</div>
</div>
<div class="card">
<h3 class="card-title">📖 4. BM25 Tokenized Corpus</h3>
<div class="card-body">
<p>Chunks are lowercased, stripped of punctuation, whitespace-split, and pickled in <code>bm25_index.pkl</code> for <code>BM25Okapi</code> keyword matching.</p>
</div>
</div>
<div class="card">
<h3 class="card-title">📈 5. Corpus CDF Score Arrays</h3>
<div class="card-body">
<p>100 sample queries are scored against ALL 17,243 corpus chunks offline (1,724,300 scores per retriever). Sorted arrays saved to <code>corpus_cdf_bm25.npy</code> and <code>corpus_cdf_dense.npy</code>.</p>
</div>
</div>
</div>
</section>
<!-- Mathematics Deep-Dive Section -->
<section id="mathematics" class="section">
<div class="section-header">
<span class="section-tag">Mathematical Foundations</span>
<h2 class="section-title">Exact Formulas & Algorithm Mechanics</h2>
<p class="section-subtitle">Understanding score calibration, entropy, precision-weighting, and statistical testing.</p>
</div>
<!-- Formula 1: BM25 -->
<div class="math-card">
<h3>1. BM25 Okapi Keyword Scoring</h3>
<p>BM25 measures term frequency saturation and document length normalization:</p>
<div class="math-formula">
$$\text{BM25}(q, D) = \sum_{i=1}^{n} \text{IDF}(q_i) \cdot \frac{f(q_i, D) \cdot (k_1 + 1)}{f(q_i, D) + k_1 \cdot \left(1 - b + b \cdot \frac{|D|}{\text{avgdl}}\right)}$$
</div>
<p style="font-size: 0.85rem; color: var(--text-secondary);">Where k<sub>1</sub>=1.5 controls term frequency saturation, b=0.75 controls document length penalty, and IDF(q<sub>i</sub>) = ln((N - n(q<sub>i</sub>) + 0.5) / (n(q<sub>i</sub>) + 0.5) + 1).</p>
</div>
<!-- Formula 2: FAISS L2 to Similarity -->
<div class="math-card">
<h3>2. FAISS Dense L2 Distance to Similarity</h3>
<p>FAISS <code>IndexFlatL2</code> computes Euclidean distance d<sub>L2</sub>. I convert distance to similarity score in (0, 1]:</p>
<div class="math-formula">
$$S_{\text{dense}} = \frac{1}{1 + d_{\text{L2}}}$$
</div>
</div>
<!-- Formula 3: Score Normalization Methods -->
<div class="math-card">
<h3>3. Corpus-Level Quantile CDF Calibration</h3>
<p>Per-query min-max destroys global scale information. Instead, I map score s to its percentile rank in the 1.7M corpus score distribution:</p>
<div class="math-formula">
$$S_{\text{cdf}}(s) = \frac{\text{searchsorted}(\text{Corpus\_CDF}, s)}{N_{\text{corpus}}}$$
</div>
<p style="font-size: 0.85rem; color: var(--text-secondary);">Maps any raw score distribution to approximately uniform [0, 1] quantile space while preserving per-query confidence variation.</p>
</div>
<!-- Formula 4: Shannon Entropy -->
<div class="math-card">
<h3>4. Shannon Entropy of Score Distributions</h3>
<p>Normalizes scores into a probability mass p<sub>i</sub> = S<sub>i</sub> / Σ S<sub>j</sub>, then calculates Shannon entropy (base 2):</p>
<div class="math-formula">
$$H = -\sum_{i=1}^{k} p_i \log_2(p_i)$$
</div>
<p style="font-size: 0.85rem; color: var(--text-secondary);"><i>Note:</i> For negative-valued z-scores, softmax is applied: p<sub>i</sub> = e<sup>S<sub>i</sub> - max(S)</sup> / Σ e<sup>S<sub>j</sub> - max(S)</sup> to prevent affine collapse.</p>
</div>
<!-- Formula 5: Adaptive Alpha -->
<div class="math-card">
<h3>5. Bayesian Precision-Weighted Fusion Weight (α)</h3>
<p>Per-query dynamic weight allocation based on retriever entropy (uncertainty):</p>
<div class="math-formula">
$$\alpha = \frac{H_{\text{dense}}}{H_{\text{dense}} + H_{\text{sparse}} + \varepsilon}$$
$$\text{Fused Score} = \alpha \cdot S_{\text{sparse, cal}} + (1 - \alpha) \cdot S_{\text{dense, cal}}$$
</div>
<p style="font-size: 0.85rem; color: var(--text-secondary);">If FAISS dense retrieval has high entropy (uncertainty), α $\to$ 1 (system relies on BM25). If BM25 has high entropy, α $\to$ 0 (system relies on FAISS dense vectors).</p>
</div>
</section>
<!-- Interactive Simulator Section -->
<section id="simulator" class="section">
<div class="section-header">
<span class="section-tag">Interactive Playground</span>
<h2 class="section-title">Live Entropy & Adaptive α Weight Simulator</h2>
<p class="section-subtitle">Adjust dense and sparse entropy sliders to watch how the system dynamically shifts trust!</p>
</div>
<div class="simulator-card">
<div class="grid-2">
<div class="slider-group">
<div class="slider-label">
<span>Dense Retriever Entropy (H<sub>dense</sub>)</span>
<span id="h-dense-val" class="slider-value">4.20</span>
</div>
<input type="range" id="h-dense-slider" min="0" max="6" step="0.05" value="4.20">
<p style="font-size: 0.75rem; color: var(--text-muted); margin-top: 0.3rem;">Higher value = FAISS vector search is uncertain/confused.</p>
</div>
<div class="slider-group">
<div class="slider-label">
<span>Sparse Retriever Entropy (H<sub>sparse</sub>)</span>
<span id="h-sparse-val" class="slider-value">4.20</span>
</div>
<input type="range" id="h-sparse-slider" min="0" max="6" step="0.05" value="4.20">
<p style="font-size: 0.75rem; color: var(--text-muted); margin-top: 0.3rem;">Higher value = BM25 keyword search is uncertain/confused.</p>
</div>
</div>
<div class="gauge-container">
<div>
<div style="font-size: 0.75rem; color: var(--text-muted); text-transform: uppercase;">Calculated Alpha (α)</div>
<div id="alpha-display" class="alpha-display">0.500</div>
</div>
<div style="flex: 1;">
<div style="display: flex; justify-content: space-between; font-size: 0.8rem; margin-bottom: 0.4rem; font-weight: 600;">
<span style="color: var(--accent-amber);">BM25 Weight: <span id="bm25-percent">50.0%</span></span>
<span style="color: var(--accent-purple);">FAISS Weight: <span id="dense-percent">50.0%</span></span>
</div>
<div class="progress-bar-bg">
<div id="bar-bm25" class="progress-bar-bm25" style="width: 50%;"></div>
<div id="bar-dense" class="progress-bar-dense" style="width: 50%;"></div>
</div>
</div>
</div>
<div id="system-state-text" style="margin-top: 1rem; font-size: 0.85rem; color: var(--text-secondary); text-align: center;">
<strong>Balanced Confidence:</strong> Equal weighting between lexical keywords and vector semantics.
</div>
</div>
</section>
<!-- File & Function Registry Section -->
<section id="codebase" class="section">
<div class="section-header">
<span class="section-tag">Codebase Directory</span>
<h2 class="section-title">What Each File & Function Does</h2>
<p class="section-subtitle">Exhaustive guide to all 9 source files and their underlying functions.</p>
</div>
<input type="text" id="code-search" class="search-box" placeholder="🔎 Search functions or files (e.g. calibrate_cdf, rrf_fuse, DocumentStore)...">
<div class="accordion">
<!-- File 1: app/main.py -->
<div class="accordion-item">
<div class="accordion-header">
<div class="accordion-title-box">
<span>📄</span>
<span>app/main.py</span>
</div>
<span class="function-tag">FastAPI Endpoints & Lifespan</span>
</div>
<div class="accordion-content">
<p style="margin-bottom: 0.75rem;">FastAPI application defining REST API endpoints and startup handlers.</p>
<ul style="margin-left: 1.2rem; display: flex; flex-direction: column; gap: 0.5rem;">
<li><code>lifespan(app)</code>: Async startup context manager that creates upload directory and triggers <code>store.load()</code>.</li>
<li><code>GET /health</code>: Returns index status, loaded files, and document counts.</li>
<li><code>GET /modes</code>: Lists all 7 available retrieval modes with descriptions.</li>
<li><code>POST /upload</code>: Accepts PDF, TXT, MD files, saves to <code>data/uploads/</code>, and indexes chunks.</li>
<li><code>POST /query</code>: Main search endpoint accepting question, top_k, and retrieval mode.</li>
</ul>
</div>
</div>
<!-- File 2: app/retriever.py -->
<div class="accordion-item">
<div class="accordion-header">
<div class="accordion-title-box">
<span>📄</span>
<span>app/retriever.py</span>
</div>
<span class="function-tag">Pipeline Orchestrator & DocumentStore</span>
</div>
<div class="accordion-content">
<p style="margin-bottom: 0.75rem;">Central document manager and mode router combining FAISS, BM25, and Cross-Encoder.</p>
<ul style="margin-left: 1.2rem; display: flex; flex-direction: column; gap: 0.5rem;">
<li><code>DocumentStore.load()</code>: Loads FAISS vectorstore, BM25 pickle index, CDF arrays, and text lookup map from disk.</li>
<li><code>DocumentStore.add_file(path)</code>: Extracts text from PDF/TXT/MD, chunks text, and builds indices.</li>
<li><code>DocumentStore._search_dense(query, top_k)</code>: Runs FAISS similarity search and converts L2 distance to [0, 1] similarity.</li>
<li><code>DocumentStore._search_sparse(query, top_k)</code>: Runs BM25 tokenized keyword search.</li>
<li><code>DocumentStore.search(query, top_k, mode)</code>: Routes search to one of 7 modes and records latency telemetry.</li>
<li><code>extract_text(path)</code>: Extracts raw text from PDF using <code>pypdf.PdfReader</code> or UTF-8 text files.</li>
<li><code>chunk_text(text, source)</code>: Splits text with <code>RecursiveCharacterTextSplitter(800, 120)</code>.</li>
</ul>
</div>
</div>
<!-- File 3: app/calibration.py -->
<div class="accordion-item">
<div class="accordion-header">
<div class="accordion-title-box">
<span>📄</span>
<span>app/calibration.py</span>
</div>
<span class="function-tag">Score Calibration, Entropy & Statistics</span>
</div>
<div class="accordion-content">
<p style="margin-bottom: 0.75rem;">Research math module handling distribution transforms, Shannon entropy, QPP baselines, and bootstrap significance testing.</p>
<ul style="margin-left: 1.2rem; display: flex; flex-direction: column; gap: 0.5rem;">
<li><code>calibrate_raw(scores)</code>: Identity baseline.</li>
<li><code>calibrate_minmax(scores)</code>: Scales scores to [0, 1] per query.</li>
<li><code>calibrate_zscore(scores)</code>: Standardizes scores to mean=0, std=1.</li>
<li><code>calibrate_cdf(scores, corpus_cdf)</code>: Maps scores to corpus percentile ranks using <code>np.searchsorted</code>.</li>
<li><code>compute_entropy(scores)</code>: Computes Shannon entropy H = -Σ p log2 p with softmax handling for negative z-scores.</li>
<li><code>compute_alpha(h_dense, h_sparse)</code>: Calculates dynamic precision weight α = H_dense / (H_dense + H_sparse + ε).</li>
<li><code>build_corpus_cdfs(...)</code>: Scores sample queries against ALL corpus docs to create offline CDF arrays.</li>
<li><code>clarity_score(top_k_texts, corpus_freqs)</code>: Computes KL divergence of top-k language model vs corpus language model.</li>
<li><code>nqc(scores, corpus_mean)</code>: Normalized Query Commitment std(top_k) / |mean(corpus)|.</li>
<li><code>bootstrap_ci(a, b, n_resamples=1000)</code>: Performs paired bootstrap resampling for 95% confidence intervals and p-value significance.</li>
</ul>
</div>
</div>
<!-- File 4: app/fusion.py -->
<div class="accordion-item">
<div class="accordion-header">
<div class="accordion-title-box">
<span>📄</span>
<span>app/fusion.py</span>
</div>
<span class="function-tag">RRF & Linear Score Fusion</span>
</div>
<div class="accordion-content">
<p style="margin-bottom: 0.75rem;">Implements reciprocal rank fusion, fixed linear fusion, and calibrated entropy-weighted fusion.</p>
<ul style="margin-left: 1.2rem; display: flex; flex-direction: column; gap: 0.5rem;">
<li><code>rrf_fuse(sparse, dense, k=60)</code>: Combines candidate ranks via RRF(d) = Σ 1 / (60 + r(d)).</li>
<li><code>linear_fuse(sparse, dense, alpha=0.5)</code>: Merges candidates with fixed α=0.5 linear combination.</li>
<li><code>entropy_fuse(sparse, dense, calibration="cdf")</code>: Calibrates scores, computes Shannon entropy per retriever, calculates dynamic α, and merges candidates.</li>
</ul>
</div>
</div>
<!-- File 5: app/sparse_retriever.py -->
<div class="accordion-item">
<div class="accordion-header">
<div class="accordion-title-box">
<span>📄</span>
<span>app/sparse_retriever.py</span>
</div>
<span class="function-tag">BM25 Index & Search</span>
</div>
<div class="accordion-content">
<p style="margin-bottom: 0.75rem;">Manages BM25Okapi index and full-corpus scoring.</p>
<ul style="margin-left: 1.2rem; display: flex; flex-direction: column; gap: 0.5rem;">
<li><code>BM25Index.tokenize(text)</code>: Lowercases, strips punctuation, and splits text into tokens.</li>
<li><code>BM25Index.add_documents(docs)</code>: Tokenizes documents and builds <code>BM25Okapi</code> model.</li>
<li><code>BM25Index.score_all(query)</code>: Returns BM25 scores across all corpus documents (used for offline CDF building).</li>
</ul>
</div>
</div>
<!-- File 6: app/reranker.py -->
<div class="accordion-item">
<div class="accordion-header">
<div class="accordion-title-box">
<span>📄</span>
<span>app/reranker.py</span>
</div>
<span class="function-tag">Cross-Encoder Transformer</span>
</div>
<div class="accordion-content">
<p style="margin-bottom: 0.75rem;">Lazy-loads <code>cross-encoder/ms-marco-MiniLM-L6-v2</code> for pairwise (query, document) scoring.</p>
<ul style="margin-left: 1.2rem; display: flex; flex-direction: column; gap: 0.5rem;">
<li><code>CrossEncoderReranker.rerank(query, candidates, top_k)</code>: Runs pairwise cross-attention prediction and sorts candidates descending by score.</li>
</ul>
</div>
</div>
<!-- File 7: app/datasets.py -->
<div class="accordion-item">
<div class="accordion-header">
<div class="accordion-title-box">
<span>📄</span>
<span>app/datasets.py</span>
</div>
<span class="function-tag">SciFact Dataset Loader</span>
</div>
<div class="accordion-content">
<p style="margin-bottom: 0.75rem;">Loads BEIR SciFact benchmark corpus (5,183 PubMed abstracts), test claims (300 queries), and qrels relevancy labels.</p>
<ul style="margin-left: 1.2rem; display: flex; flex-direction: column; gap: 0.5rem;">
<li><code>load_scifact_corpus()</code>: Parses <code>corpus.jsonl</code> into document records.</li>
<li><code>load_scifact_queries()</code>: Parses <code>queries.jsonl</code> mapping query IDs to claim text.</li>
<li><code>load_scifact_qrels()</code>: Loads ground-truth relevance pairs from <code>qrels/test.tsv</code>.</li>
</ul>
</div>
</div>
<!-- File 8: scripts/build_index.py -->
<div class="accordion-item">
<div class="accordion-header">
<div class="accordion-title-box">
<span>📄</span>
<span>scripts/build_index.py</span>
</div>
<span class="function-tag">Offline Index & CDF Generator</span>
</div>
<div class="accordion-content">
<p style="margin-bottom: 0.75rem;">Offline script executing indexing step, generating FAISS vectors, BM25 indices, unigram language models, document lookup tables, and corpus-level CDF numpy files.</p>
</div>
</div>
<!-- File 9: scripts/evaluate.py -->
<div class="accordion-item">
<div class="accordion-header">
<div class="accordion-title-box">
<span>📄</span>
<span>scripts/evaluate.py</span>
</div>
<span class="function-tag">Scientific Evaluation & H1/H2 Benchmark</span>
</div>
<div class="accordion-content">
<p style="margin-bottom: 0.75rem;">Evaluates 7 retrieval modes across 300 test claims, computes Pearson correlations for H1, runs 1,000 paired bootstrap resamples for H2, and deduplicates chunk results by document ID.</p>
</div>
</div>
</div>
</section>
<!-- Research Findings & Benchmarks Section -->
<section id="benchmarks" class="section">
<div class="section-header">
<span class="section-tag">Scientific Evidence</span>
<h2 class="section-title">BEIR SciFact Benchmark Results & Research Hypotheses</h2>
<p class="section-subtitle">Evaluating 7 retrieval modes across 300 test claims (5,183 PubMed abstracts).</p>
</div>
<table class="data-table">
<thead>
<tr>
<th>Mode</th>
<th>NDCG@10</th>
<th>MRR</th>
<th>P@3</th>
<th>P@5</th>
<th>R@5</th>
<th>p95 Latency</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>dense</code></td>
<td>0.6715</td>
<td>0.6329</td>
<td>0.2466</td>
<td>0.1647</td>
<td>0.7495</td>
<td>33 ms</td>
</tr>
<tr>
<td><code>sparse</code></td>
<td>0.6151</td>
<td>0.5804</td>
<td>0.2233</td>
<td>0.1520</td>
<td>0.7089</td>
<td>130 ms</td>
</tr>
<tr>
<td><code>rrf</code></td>
<td>0.7028</td>
<td>0.6713</td>
<td>0.2555</td>
<td>0.1680</td>
<td>0.7744</td>
<td>249 ms</td>
</tr>
<tr>
<td><code>hybrid_fixed</code></td>
<td>0.6829</td>
<td>0.6527</td>
<td>0.2522</td>
<td>0.1647</td>
<td>0.7594</td>
<td>232 ms</td>
</tr>
<tr>
<td><code>hybrid_calibrated</code></td>
<td>0.6981</td>
<td>0.6672</td>
<td>0.2489</td>
<td>0.1660</td>
<td>0.7661</td>
<td>241 ms</td>
</tr>
<tr>
<td><code>hybrid_fixed_rerank</code></td>
<td>0.7006</td>
<td>0.6653</td>
<td>0.2578</td>
<td>0.1700</td>
<td>0.7714</td>
<td>2432 ms</td>
</tr>
<tr class="highlight-row">
<td><strong>hybrid_calibrated_rerank</strong></td>
<td><strong>0.7071</strong></td>
<td><strong>0.6719</strong></td>
<td><strong>0.2622</strong></td>
<td><strong>0.1720</strong></td>
<td><strong>0.7838</strong></td>
<td>2116 ms</td>
</tr>
</tbody>
</table>
<div class="grid-2" style="margin-top: 1.5rem;">
<div class="card">
<h3 class="card-title">🔬 H1: Entropy Correlation Analysis</h3>
<div class="card-body">
<p><strong>Hypothesis:</strong> CDF entropy has the strongest negative correlation with retrieval quality.</p>
<p style="margin-top: 0.5rem; color: var(--accent-rose);">❌ <strong>Verdict: Not Supported.</strong></p>
<p style="margin-top: 0.4rem; font-size: 0.85rem;">Z-score entropy turned out to be the strongest negative predictor (r = -0.4065, p < 10<sup>-6</sup>) for dense retrieval on SciFact, whereas CDF entropy showed a weaker correlation (r = -0.1151).</p>
</div>
</div>
<div class="card">
<h3 class="card-title">📊 H2: Paired Bootstrap Significance</h3>
<div class="card-body">
<p><strong>Hypothesis:</strong> Calibrated entropy fusion significantly outperforms fixed-alpha and RRF baselines.</p>
<p style="margin-top: 0.5rem; color: var(--accent-amber);">⚠️ <strong>Verdict: Partially Supported.</strong></p>
<p style="margin-top: 0.4rem; font-size: 0.85rem;">CDF entropy fusion significantly beats fixed-alpha fusion (+0.0152 NDCG@10 gain, p = 0.012), but ties with RRF. With cross-encoder reranking, the full pipeline achieves the best overall NDCG@10 (0.7071).</p>
</div>
</div>
</div>
</section>
<!-- How to Run & Deploy Section -->
<section id="how-to-run" class="section">
<div class="section-header">
<span class="section-tag">Deployment & Setup Guide</span>
<h2 class="section-title">How to Run & Deploy</h2>
<p class="section-subtitle">Commands to build indices, run FastAPI, execute pytest, and evaluate performance.</p>
</div>
<div class="card" style="margin-bottom: 1.5rem;">
<h3 class="card-title">1. Local Setup & Installation</h3>
<pre><code># Create virtual environment
python -m venv venv
# Activate venv (Windows PowerShell)
.\venv\Scripts\Activate.ps1
# Install requirements
pip install -r requirements.txt</code></pre>
</div>
<div class="card" style="margin-bottom: 1.5rem;">
<h3 class="card-title">2. Build Retrieval Artifacts & Index SciFact</h3>
<pre><code># Builds FAISS, BM25, Corpus CDFs, and Document Lookup tables
python scripts/build_index.py --dataset scifact</code></pre>
</div>
<div class="card" style="margin-bottom: 1.5rem;">
<h3 class="card-title">3. Start the FastAPI Server & Interactive Swagger Docs</h3>
<pre><code># Launch server at http://127.0.0.1:8000
uvicorn app.main:app --reload
# Open Swagger Docs in your browser: http://127.0.0.1:8000/docs</code></pre>
</div>
<div class="card" style="margin-bottom: 1.5rem;">
<h3 class="card-title">4. Run Pytest Test Suite</h3>
<pre><code># Executes all 9 test cases covering API endpoints, telemetry, and calibration
pytest tests/ -v</code></pre>
</div>
<div class="card">
<h3 class="card-title">5. Run Benchmark Evaluation & Significance Tests</h3>
<pre><code># Evaluates all 7 modes on SciFact test claims & runs bootstrap tests
python scripts/evaluate.py</code></pre>
</div>
</section>
</main>
</div>
<!-- JS Engine -->
<script src="understanding of project/app.js"></script>
</body>
</html>
|