RAG retreival updates

README.md

@@ -22,38 +22,44 @@ Visualize and compare **5 chunking strategies** side-by-side:
 | **Parent-Child** | Two-level nested chunking — large parent windows with smaller child chunks inside |
 | **Semantic**     | Detects topic shifts using embedding similarity + adaptive thresholding           |
 
-- **Document X-Ray Viewer** — Original text with color-coded chunk boundaries and overlap regions
-- **Chunk Inspector** — Stats panel showing total chunks, average token count, and per-chunk metadata
+- **Document X-Ray Viewer** — Original text with color-coded chunk boundaries and overlap regions.
+- **Chunk Inspector** — Stats panel showing total chunks, average token count, and per-chunk metadata.
+- **File Uploader** — Attach and parse custom text or markdown documents directly in the configuration panel.
 
 ### 🌌 Phase 2 — Embedding Lab
 
-- Generate embeddings using **3 local Ollama embedding models** (Nomic Embed Text, Embedding Gemma, Qwen3 Embedding)
-- **UMAP dimensionality reduction** projects high-dimensional embeddings down to 2D
-- **Interactive Canvas** with pan, zoom, hover tooltips, and click-to-select
-- Parent-child connection lines visualized in vector space
+- Generate embeddings using **3 local Ollama embedding models** (Nomic Embed Text, Embedding Gemma, Qwen3 Embedding).
+- **UMAP dimensionality reduction** projects high-dimensional embeddings down to 2D.
+- **Interactive Canvas** with pan, zoom, hover tooltips, and click-to-select. Drag-panning is isolated from clicks to ensure smooth navigation without losing focus.
+- Parent-child connection lines visualized in vector space.
 
-### 🔍 Phase 3 — Retrieval
+### 🔍 Phase 3 — Advanced Retrieval & Reranking
 
-- **ChromaDB** persistent vector store — chunks are indexed on every run
-- **Sonar Query Simulator** — type a natural language query and watch the retrieval happen in real time
-- Retrieved chunks render as ranked result cards with distance scores
-- **Sonar Probe** — click anywhere on the canvas to find the nearest chunks by 2D proximity
-- **Document X-Ray Highlighting** — retrieved chunks glow in the original text with rank-based styling (gold for Rank 1, dashed for Rank 2, dotted for Rank 3)
+- **ChromaDB** persistent vector store — chunks are indexed on every run.
+- **Flexible Retrieval Modes** — Switch dynamically between **Dense** (vector similarity), **Sparse** (BM25 lexical search), or **Hybrid** (RRF fusion) search paths.
+- **Sonar Query Simulator** — Type a natural language query and watch the sonar ping animate across the canvas in real time.
+- **Sonar Probe** — Click anywhere on the 2D canvas to retrieve the nearest chunks in that region.
+- **Document X-Ray Highlighting** — Retrieved chunks glow dynamically in the document viewer with rank-based styling (gold for Rank 1, dashed for Rank 2, dotted for Rank 3).
+- **Metadata Level Filtering** — Filter your context pool on the fly (retrieve *Only Parents*, *Only Children*, or *All Levels*).
+- **Cross-Encoder Reranking** — Run a local FlashRank (`ms-marco-MiniLM-L-12-v2`) engine to rerank search results.
+- **Rank Shift Badges** — Visual indicators showing exactly how much chunks moved after reranking (`▲ +3`, `▼ -1`, or `• Unchanged`).
+- **Normalized Match Strength** — Converts raw vector distances into intuitive similarity percentages (e.g. `Match: 87.7%`).
+- **Reranking Lineage** — Displays the pre-reranked retrieval score for comparison (e.g. `Match: 95.0% (was Match: 87.7%)`).
 
 ### ⚔️ Phase 3.2 — LLM-as-a-Judge (The Grand Arena)
 
-- **Side-by-Side Comparison** — Compare retrieval results from two different models/strategies in a split-screen arena
-- **AI Referee** — Call upon a local Ollama model to evaluate, rank, and score retrieved contexts
-- **Multi-Dimensional Scorecard** — Referee grades chunks on Relevance, Completeness, Factual Plausibility, and Clarity
-- **Pydantic Validator Guardrails** — Validates the referee's output to catch and override arithmetic lies and position bias
+- **Side-by-Side Comparison** — Compare retrieval results from two different models/strategies in a split-screen arena.
+- **AI Referee** — Call upon a local Ollama model to evaluate, rank, and score retrieved contexts.
+- **Multi-Dimensional Scorecard** — Referee grades chunks on Relevance, Completeness, Factual Plausibility, and Clarity.
+- **Pydantic Validator Guardrails** — Validates the referee's output to catch and override arithmetic lies and position bias.
 
 ![Grand Arena Comparison](assets/arena_comparison.gif)
 
 ### 📐 Phase 4 — Adaptive Thresholding (Gradient Fix)
 
-- Semantic chunking uses an **adaptive gradient derivative / peak detection** algorithm instead of a static threshold split
-- Computes the dynamic threshold based on document-wide mean and standard deviation of inter-sentence embedding distances
-- Uses local maxima peak detection to prevent fragmenting paragraphs, ensuring splits only happen at true topic shift peaks
+- Semantic chunking uses an **adaptive gradient derivative / peak detection** algorithm instead of a static threshold split.
+- Computes the dynamic threshold based on document-wide mean and standard deviation of inter-sentence embedding distances.
+- Uses local maxima peak detection to prevent fragmenting paragraphs, ensuring splits only happen at true topic shift peaks.
 
 ---
 
@@ -260,7 +266,7 @@ Chunks input text, generates embeddings, reduces to 2D, and stores in ChromaDB.
 
 ### `POST /api/retrieve`
 
-Embeds a query and retrieves the top-K most similar chunks from ChromaDB.
+Embeds a query and retrieves the top-K most similar chunks from ChromaDB (with optional reranking, HyDE expansion, and metadata filtering).
 
 **Request Body:**
 
@@ -269,11 +275,15 @@ Embeds a query and retrieves the top-K most similar chunks from ChromaDB.
   "search_text": "What is gradient descent?",
   "embedding_model": "nomic-embed-text",
   "strategy": "fixed_size",
-  "top_k": 3
+  "top_k": 3,
+  "retrieval_mode": "dense",
+  "use_hyde": false,
+  "use_reranking": true,
+  "metadata": { "level": 1 }
 }
 ```
 
-**Response:** `QueryResponse` with query coordinates, retrieved chunks, and distance scores.
+**Response:** `QueryResponse` with query coordinates, retrieved chunks (with original ranks and original scores populated if reranked), and hypothetical answer text if HyDE is used.
 
 ### `POST /api/compare`
 
@@ -288,7 +298,11 @@ Compares retrieval results from two different configurations side-by-side.
   "model_a": "nomic-embed-text",
   "strategy_a": "fixed_size",
   "model_b": "EmbeddingGemma",
-  "strategy_b": "semantic"
+  "strategy_b": "semantic",
+  "retrieval_mode": "dense",
+  "use_hyde": false,
+  "use_reranking": true,
+  "metadata": null
 }
 ```
 

backend/engines/{BM25Enginer.py → bm25_engine.py}

@@ -1,7 +1,7 @@
 import math
-from typing import Any, Dict, List
+from typing import Any, Dict, List, Optional
 
-from backend.routers.retrieval_router import get_keywords
+from backend.utils import get_keywords
 
 
 class BM25Engine:
@@ -58,10 +58,21 @@ class BM25Engine:
 
         return total_score
 
-    def search(self, query: str, top_k: int = 3) -> list:
+    def search(
+        self, query: str, top_k: int = 3, where: Optional[Dict[str, Any]] = None
+    ) -> list:
         scores = []
         # Score all docs
         for idx in range(self.corpus_size):
+            if where:
+                match = True
+                doc_meta = self.metadatas[idx] or {}
+                for key, val in where.items():
+                    if doc_meta.get(key) != val:
+                        match = False
+                        break
+                if not match:
+                    continue
             s = self.score(query, idx)
             scores.append((s, idx))
 

backend/engines/re_ranker_engine.py

@@ -0,0 +1,37 @@
+from typing import List
+from flashrank import Ranker, RerankRequest
+from backend.models.schemas import RetrievedChunk
+
+
+class RerankerEngine:
+    _ranker = None
+
+    def __init__(self):
+        if RerankerEngine._ranker is None:
+            RerankerEngine._ranker = Ranker(model_name="ms-marco-MiniLM-L-12-v2")
+        self.ranker = RerankerEngine._ranker
+
+    def rerank(self, query: str, chunks: List[RetrievedChunk], top_k: int):
+        formatted_list = []
+        for idx, chunk in enumerate(chunks):
+            chunk.original_score = chunk.score
+            chunk.original_rank = idx + 1
+            formatted_list.append(
+                {
+                    "id": chunk.id,
+                    "text": chunk.text,
+                    "score": chunk.score,
+                }
+            )
+
+        request = RerankRequest(query=query, passages=formatted_list)
+        results = self.ranker.rerank(request)
+        chunk_map = {chunk.id: chunk for chunk in chunks}
+        reranked_chunks = []
+
+        for result in results[:top_k]:
+            chunk = chunk_map[result["id"]]
+            chunk.score = float(result["score"])
+            reranked_chunks.append(chunk)
+
+        return reranked_chunks

backend/models/schemas.py

@@ -86,6 +86,7 @@ class QueryRequest(BaseModel):
     retrieval_mode: RetrievalMode = RetrievalMode.DENSE
     use_hyde: bool = False
     use_reranking: bool = False
+    metadata: Optional[Dict[str, Any]] = None
 
 
 class RetrievedChunk(BaseModel):
@@ -97,6 +98,8 @@ class RetrievedChunk(BaseModel):
     parent_id: Optional[str] = None
     level: int = 0
     text_highlighted: Optional[str] = None
+    original_score: Optional[float] = None
+    original_rank: Optional[int] = None
 
 
 class QueryResponse(BaseModel):
@@ -117,6 +120,7 @@ class CompareRequest(BaseModel):
     retrieval_mode: RetrievalMode = RetrievalMode.DENSE
     use_hyde: bool = False
     use_reranking: bool = False
+    metadata: Optional[Dict[str, Any]] = None
 
 
 class CompareResponse(BaseModel):

backend/routers/retrieval_router.py

@@ -1,12 +1,11 @@
-from backend.constants import hyde_prompt
-from backend.engines import llm_client
+from backend.engines.re_ranker_engine import RerankerEngine
+from backend.engines.bm25_engine import BM25Engine
 from backend.engines.llm_client import OllamaClient
 from backend.constants import system_instructions
 import asyncio
-from typing import Any, List, Optional
+from typing import Any, Dict, List, Optional
 from fastapi import APIRouter
 import json
-import re
 from backend.engines.embedding import EmbeddingEngine
 from backend.engines.reducer import ReducerEngine
 from backend.models.schemas import (
@@ -20,17 +19,11 @@ from backend.models.schemas import (
     RetrievedChunk,
 )
 from backend.storage.vector_store import VectorStore
-
-from nltk.tokenize import RegexpTokenizer
-from nltk.corpus import stopwords
+from backend.utils import get_hyde_text, get_keywords, get_text_highlights, rrf
 
 router = APIRouter(prefix="/api", tags=["retrieval"])
 
 
-tokenizer = RegexpTokenizer(r"\w+")
-stop_words = set(stopwords.words("english"))
-
-
 @router.post("/retrieve", response_model=QueryResponse)
 async def retrieve(request: QueryRequest):
 
@@ -40,18 +33,24 @@ async def retrieve(request: QueryRequest):
         else request.search_text
     )
 
-    if request.use_reranking == True:
-        "" ""
+    limit = request.top_k * 4 if request.use_reranking else request.top_k
 
     retrieved_chunks, embeddings = await process_retrieval(
         request.embedding_model,
         search_text,
         request.strategy,
-        request.top_k,
-        request.search_text,
+        limit,
         request.retrieval_mode,
+        request.search_text,
+        request.metadata,
     )
 
+    if request.use_reranking == True:
+        re_ranker = RerankerEngine()
+        retrieved_chunks = re_ranker.rerank(
+            query=request.search_text, chunks=retrieved_chunks, top_k=request.top_k
+        )
+
     query_coords = [0.0, 0.0]
     if ReducerEngine._last_fitted_reducer is not None:
         projected: Any = ReducerEngine._last_fitted_reducer.transform(embeddings)
@@ -76,6 +75,7 @@ async def compare(request: CompareRequest):
         if request.use_hyde
         else request.search_text
     )
+    limit = request.top_k * 4 if request.use_reranking else request.top_k
     (
         (retrieved_chunks_a, _),
         (retrieved_chunks_b, _),
@@ -84,18 +84,35 @@ async def compare(request: CompareRequest):
             request.model_a,
             retrieval_text,
             request.strategy_a,
-            request.top_k,
+            limit,
+            request.retrieval_mode,
             request.search_text,
+            request.metadata,
         ),
         process_retrieval(
             request.model_b,
             retrieval_text,
             request.strategy_b,
-            request.top_k,
+            limit,
+            request.retrieval_mode,
             request.search_text,
+            request.metadata,
         ),
     )
 
+    if request.use_reranking == True:
+        re_ranker = RerankerEngine()
+        retrieved_chunks_a = re_ranker.rerank(
+            query=request.search_text,
+            chunks=retrieved_chunks_a,
+            top_k=request.top_k,
+        )
+        retrieved_chunks_b = re_ranker.rerank(
+            query=request.search_text,
+            chunks=retrieved_chunks_b,
+            top_k=request.top_k,
+        )
+
     response_kwargs = {
         "search_text": request.search_text,
         "results_a": retrieved_chunks_a,
@@ -130,25 +147,87 @@ async def process_retrieval(
     top_k,
     retrieval_mode,
     original_query: Optional[str] = None,
+    metadata: Optional[Dict[str, Any]] = None,
 ):
     vector_store = VectorStore()
     embedding_engine = EmbeddingEngine(model.value)
     embeddings = await embedding_engine.generate_embeddings([search_text])
     collection_name = f"{model.value}_{strategy.value}".replace(":", "-")
-    retrieval_response: Any
+    result: Any = None
     if retrieval_mode == RetrievalMode.DENSE:
-        """"""
+        result: Any = await vector_store.retrieve(
+            collection_name=collection_name,
+            embeddings=embeddings,
+            n_results=top_k,
+            where=metadata,
+        )
     elif retrieval_mode == RetrievalMode.SPARSE:
-        """"""
+        all_data = await vector_store.get_all_documents(collection_name=collection_name)
+        docs = all_data.get("documents") or []
+        doc_ids = all_data.get("ids") or []
+        meta_datas: List[Any] = all_data.get("metadatas", []) or []
+
+        if not docs:
+            result = {
+                "ids": [[]],
+                "documents": [[]],
+                "distances": [[]],
+                "metadatas": [[]],
+            }
+        else:
+            bm25 = BM25Engine(documents=docs, doc_ids=doc_ids, metadatas=meta_datas)
+            sparse_results = bm25.search(query=search_text, top_k=top_k, where=metadata)
+            result = {
+                "ids": [[res["id"] for res in sparse_results]],
+                "documents": [[res["text"] for res in sparse_results]],
+                "distances": [[res["score"] for res in sparse_results]],
+                "metadatas": [[res["metadata"] for res in sparse_results]],
+            }
     elif retrieval_mode == RetrievalMode.HYBRID:
-        """"""
-
-    result: Any = await vector_store.retrieve(
-        collection_name=collection_name,
-        embeddings=embeddings,
-        n_results=top_k,
-    )
-
+        dense_res = await vector_store.retrieve(
+            collection_name=collection_name,
+            embeddings=embeddings,
+            n_results=top_k * 2,
+            where=metadata,
+        )
+        dense_ids = (
+            dense_res.get("ids", [[]])[0] if dense_res and "ids" in dense_res else []
+        )
+        all_data = await vector_store.get_all_documents(collection_name)
+        docs = all_data.get("documents", []) or []
+        ids = all_data.get("ids", []) or []
+        metadatas: List[Any] = all_data.get("metadatas", []) or []
+
+        if not docs:
+            result = {
+                "ids": [[]],
+                "documents": [[]],
+                "distances": [[]],
+                "metadatas": [[]],
+            }
+        else:
+            bm25 = BM25Engine(documents=docs, doc_ids=ids, metadatas=metadatas)
+            sparse_results = bm25.search(search_text, top_k=top_k * 2, where=metadata)
+            sparse_ids = [res["id"] for res in sparse_results]
+
+            # 2. Build a lookup map of the document contents
+            doc_lookup = {
+                ids[i]: {"text": docs[i], "metadata": metadatas[i]}
+                for i in range(len(ids))
+            }
+
+            # 3. Fuse rankings
+            fused_ranks = rrf(dense_ids, sparse_ids, k=60)[:top_k]
+
+            # 4. Standardize output
+            result = {
+                "ids": [[item[0] for item in fused_ranks]],
+                "documents": [[doc_lookup[item[0]]["text"] for item in fused_ranks]],
+                "distances": [[item[1] for item in fused_ranks]],  # RRF score
+                "metadatas": [
+                    [doc_lookup[item[0]]["metadata"] for item in fused_ranks]
+                ],
+            }
     retrieved_chunks = []
 
     highlight_text = original_query if original_query is not None else search_text
@@ -177,41 +256,3 @@ async def process_retrieval(
             )
 
     return retrieved_chunks, embeddings
-
-
-def get_text_highlights(original_text, search_text):
-    query_keywords = get_keywords(search_text)
-    if not query_keywords:
-        return original_text
-
-    query_keywords.sort(key=len, reverse=True)
-
-    safe_keywords = [f"{re.escape(w)}(?:'s)?" for w in query_keywords]
-
-    pattern_string = r"\b(" + "|".join(safe_keywords) + r")\b"
-    pattern = re.compile(pattern_string, re.IGNORECASE)
-
-    return pattern.sub(r"<mark>\1</mark>", original_text)
-
-
-def get_keywords(text: str):
-    token = tokenizer.tokenize(text.lower())
-    return [word for word in token if word not in stop_words]
-
-
-async def get_hyde_text(search_text):
-    llm_client = OllamaClient()
-    hyde_prompt.format(search_text=search_text)
-    return await llm_client.generate(hyde_prompt, response_format=None)
-
-
-def rrf(dense_ranks: List[str], sparse_ranks: List[str], k: int = 60) -> List[tuple]:
-    rrf_scores = {}
-
-    for rank, doc_id in enumerate(dense_ranks):
-        rrf_scores[doc_id] = rrf_scores.get(doc_id, 0.0) + 1.0 / (k + rank + 1)
-
-    for rank, doc_id in enumerate(sparse_ranks):
-        rrf_scores[doc_id] = rrf_scores.get(doc_id, 0.0) + 1.0 / (k + rank + 1)
-
-    return sorted(rrf_scores.items(), key=lambda x: x[1], reverse=True)

backend/storage/vector_store.py

@@ -25,11 +25,12 @@ class VectorStore:
             metadatas=metadatas,
         )
 
-    async def retrieve(self, collection_name, embeddings, n_results=3):
+    async def retrieve(self, collection_name, embeddings, where, n_results=3):
         collection = self.get_collection(collection_name)
 
         return await asyncio.to_thread(
             collection.query,
+            where=where,
             query_embeddings=embeddings,
             n_results=n_results,
         )

backend/utils.py

@@ -0,0 +1,47 @@
+import re
+from typing import List
+from backend.constants import hyde_prompt
+from backend.engines.llm_client import OllamaClient
+from nltk.tokenize import RegexpTokenizer
+from nltk.corpus import stopwords
+
+tokenizer = RegexpTokenizer(r"\w+")
+stop_words = set(stopwords.words("english"))
+
+
+def get_keywords(text: str):
+    token = tokenizer.tokenize(text.lower())
+    return [word for word in token if word not in stop_words]
+
+
+async def get_hyde_text(search_text):
+    llm_client = OllamaClient()
+    formatted_prompt = hyde_prompt.format(search_text=search_text)
+    return await llm_client.generate(formatted_prompt, response_format=None)
+
+
+def rrf(dense_ranks: List[str], sparse_ranks: List[str], k: int = 60) -> List[tuple]:
+    rrf_scores = {}
+
+    for rank, doc_id in enumerate(dense_ranks):
+        rrf_scores[doc_id] = rrf_scores.get(doc_id, 0.0) + 1.0 / (k + rank + 1)
+
+    for rank, doc_id in enumerate(sparse_ranks):
+        rrf_scores[doc_id] = rrf_scores.get(doc_id, 0.0) + 1.0 / (k + rank + 1)
+
+    return sorted(rrf_scores.items(), key=lambda x: x[1], reverse=True)
+
+
+def get_text_highlights(original_text, search_text):
+    query_keywords = get_keywords(search_text)
+    if not query_keywords:
+        return original_text
+
+    query_keywords.sort(key=len, reverse=True)
+
+    safe_keywords = [f"{re.escape(w)}(?:'s)?" for w in query_keywords]
+
+    pattern_string = r"\b(" + "|".join(safe_keywords) + r")\b"
+    pattern = re.compile(pattern_string, re.IGNORECASE)
+
+    return pattern.sub(r"<mark>\1</mark>", original_text)

frontend/app.js

@@ -173,6 +173,29 @@ dom.textInput.addEventListener("input", () => {
   dom.charCount.textContent = `${state.text.length} chars`;
 });
 
+const btnUploadFile = document.getElementById("btn-upload-file");
+const fileInput = document.getElementById("file-input");
+
+if (btnUploadFile && fileInput) {
+  btnUploadFile.addEventListener("click", () => fileInput.click());
+
+  fileInput.addEventListener("change", (e) => {
+    const file = e.target.files[0];
+    if (!file) return;
+
+    const reader = new FileReader();
+    reader.onload = function (event) {
+      const text = event.target.result;
+      dom.textInput.value = text;
+      state.text = text;
+      dom.charCount.textContent = `${text.length} chars`;
+      fileInput.value = "";
+    };
+    reader.readAsText(file);
+  });
+}
+
+
 // ============================================================
 // Strategy Selector
 // ============================================================
@@ -492,7 +515,8 @@ function renderChunkList(chunks) {
 function highlightChunk(chunkId) {
   // Highlight in X-ray viewer
   dom.xrayText.querySelectorAll(".chunk-highlight").forEach((el) => {
-    el.classList.toggle("active", el.dataset.chunkId === chunkId);
+    const ownerList = (el.dataset.allChunks || "").split(",");
+    el.classList.toggle("active", ownerList.includes(chunkId));
   });
 
   // Highlight in chunk list
@@ -544,6 +568,49 @@ function escapeHtml(str) {
   return div.innerHTML;
 }
 
+function getRankShiftBadgeHtml(originalRank, currentRank) {
+  if (originalRank === undefined || originalRank === null) {
+    return "";
+  }
+  const shift = originalRank - currentRank;
+  if (shift > 0) {
+    return `<span class="rank-shift-badge rank-shift-up">▲ +${shift}</span>`;
+  } else if (shift < 0) {
+    return `<span class="rank-shift-badge rank-shift-down">▼ ${shift}</span>`;
+  } else {
+    return `<span class="rank-shift-badge rank-shift-unchanged">• Unchanged</span>`;
+  }
+}
+
+function formatScoreHtml(score, originalScore, retrievalMode, useReranking) {
+  let label = "Score";
+  let formattedScore = score.toFixed(3);
+  
+  if (useReranking) {
+    label = "Rerank";
+    formattedScore = `${(score * 100).toFixed(1)}%`;
+  } else if (retrievalMode === "sparse") {
+    label = "BM25";
+    formattedScore = score.toFixed(2);
+  } else if (retrievalMode === "hybrid") {
+    label = "RRF";
+    formattedScore = score.toFixed(4);
+  } else {
+    label = "Dist";
+    formattedScore = score.toFixed(3);
+  }
+
+  let html = `<span class="score-badge-val">${label}: ${formattedScore}</span>`;
+  
+  if (useReranking && originalScore !== null && originalScore !== undefined) {
+    let origLabel = retrievalMode === "sparse" ? "BM25" : (retrievalMode === "hybrid" ? "RRF" : "Dist");
+    let origFormatted = originalScore.toFixed(3);
+    html += `<span class="score-badge-original" title="Score before reranking"> (was ${origLabel}: ${origFormatted})</span>`;
+  }
+  
+  return html;
+}
+
 document.addEventListener("keydown", (e) => {
   // Ctrl/Cmd + Enter to run
   if ((e.ctrlKey || e.metaKey) && e.key === "Enter") {
@@ -940,6 +1007,7 @@ function setupCanvasListeners(canvas) {
   canvas.addEventListener("mousedown", (e) => {
     const pos = getMousePos(e);
     vectorState.isDragging = true;
+    vectorState.dragStartPos = { x: pos.x, y: pos.y };
     vectorState.dragStart = {
       x: pos.x - vectorState.pan.x,
       y: pos.y - vectorState.pan.y,
@@ -1029,8 +1097,16 @@ function setupCanvasListeners(canvas) {
   );
 
   canvas.addEventListener("click", (e) => {
-    if (vectorState.isDragging) return;
     const pos = getMousePos(e);
+    if (vectorState.dragStartPos) {
+      const dx = pos.x - vectorState.dragStartPos.x;
+      const dy = pos.y - vectorState.dragStartPos.y;
+      const dist = Math.hypot(dx, dy);
+      if (dist > 5) {
+        return; // It was a drag, not a click!
+      }
+    }
+    if (vectorState.isDragging) return;
     const width = canvas.width / window.devicePixelRatio;
     const height = canvas.height / window.devicePixelRatio;
 
@@ -1193,6 +1269,16 @@ async function runQuerySimulator() {
       <div class="skeleton-loader"></div>
     `;
 
+    const useHyde = document.getElementById("query-use-hyde") ? document.getElementById("query-use-hyde").checked : false;
+    const useReranking = document.getElementById("query-use-reranking") ? document.getElementById("query-use-reranking").checked : false;
+    const filterLevel = document.getElementById("query-filter-level") ? document.getElementById("query-filter-level").value : "all";
+    let whereClause = null;
+    if (filterLevel === "parent") {
+      whereClause = { "level": 0 };
+    } else if (filterLevel === "child") {
+      whereClause = { "level": 1 };
+    }
+
     // Make the actual API call to the backend
     const res = await fetch("/api/retrieve", {
       method: "POST", // Needs to be POST to send a JSON body!
@@ -1203,6 +1289,9 @@ async function runQuerySimulator() {
         strategy: state.strategy,
         top_k: 3,
         retrieval_mode: dom.queryRetrievalMode ? dom.queryRetrievalMode.value : "dense",
+        use_hyde: useHyde,
+        use_reranking: useReranking,
+        metadata: whereClause,
       }),
     });
 
@@ -1242,6 +1331,8 @@ async function runQuerySimulator() {
         chunk: chunk,
         dist: result.score,
         text_highlighted: result.text_highlighted,
+        original_rank: result.original_rank,
+        original_score: result.original_score,
       };
     });
 
@@ -1259,20 +1350,43 @@ async function runQuerySimulator() {
     };
 
     // Render results in the drawer
-    dom.queryResultsList.innerHTML = topK
+    let htmlContent = "";
+    if (data.hypothetical_answer) {
+      htmlContent += `
+        <div class="query-result-item" style="border-left-color: var(--superman-blue); background-color: var(--superman-blue-muted); margin-bottom: 12px; border-radius: 6px; padding: 10px;">
+          <div class="query-result-header" style="margin-bottom: 4px;">
+            <span class="rank-badge" style="color: var(--superman-blue); font-weight: bold;">🔮 Generated Hypothetical Answer (HyDE)</span>
+          </div>
+          <div class="query-result-text" style="font-style: italic; color: var(--text-primary); font-size: 0.85rem;">"${escapeHtml(data.hypothetical_answer)}"</div>
+        </div>
+      `;
+    }
+
+    htmlContent += topK
       .map((item, index) => {
+        const retrievalMode = dom.queryRetrievalMode ? dom.queryRetrievalMode.value : "dense";
         return `
         <div class="query-result-item" style="border-left-color: ${index === 0 ? "var(--warning-color)" : "var(--info-color)"}">
           <div class="query-result-header">
-            <span class="rank-badge" style="color: ${index === 0 ? "var(--warning-color)" : "var(--info-color)"}">Rank ${index + 1} (${item.chunk.id})</span>
-            <span class="dist-badge">Dist: ${item.dist.toFixed(3)}</span>
+            <span class="rank-badge" style="color: ${index === 0 ? "var(--warning-color)" : "var(--info-color)"}">
+              Rank ${index + 1}
+              ${getRankShiftBadgeHtml(item.original_rank, index + 1)}
+            </span>
+            <span class="dist-badge">
+              ${formatScoreHtml(item.dist, item.original_score, retrievalMode, useReranking)}
+            </span>
           </div>
           <div class="query-result-text">${item.text_highlighted || escapeHtml(item.chunk.text.substring(0, 100)) + "..."}</div>
+          <div class="query-result-meta">
+            <span>ID: ${item.chunk.id}</span>
+          </div>
         </div>
       `;
       })
       .join("");
 
+    dom.queryResultsList.innerHTML = htmlContent;
+
     // --- TASK 3.4: X-RAY DOCUMENT HIGHLIGHTING ---
     // 1. Activate the search mode on the X-Ray text to dim non-retrieved chunks
     dom.xrayText.classList.add("search-active");
@@ -1290,14 +1404,15 @@ async function runQuerySimulator() {
     topK.forEach((item, index) => {
       const rank = index + 1;
       const chunkId = item.chunk.id;
-      const span = dom.xrayText.querySelector(`[data-chunk-id="${chunkId}"]`);
+      const spans = dom.xrayText.querySelectorAll(`[data-all-chunks*="${chunkId}"]`);
 
-      if (span) {
+      spans.forEach((span) => {
         span.classList.add(`retrieved-rank-${rank}`);
-        // If it's the #1 hit, scroll the Document Viewer straight to it!
-        if (rank === 1) {
-          span.scrollIntoView({ behavior: "smooth", block: "center" });
-        }
+      });
+
+      // If it's the #1 hit, scroll the Document Viewer straight to it!
+      if (rank === 1 && spans.length > 0) {
+        spans[0].scrollIntoView({ behavior: "smooth", block: "center" });
       }
     });
     // ----------------------------------------------
@@ -1376,12 +1491,27 @@ if (domArena.btnFight) {
       domReferee.triggerBox.style.display = "block";
     }
 
+    const hydeContainer = document.getElementById("arena-hyde-container");
+    if (hydeContainer) {
+      hydeContainer.style.display = "none";
+    }
+
     domArena.resultsA.innerHTML =
       '<div class="skeleton-loader"></div><div class="skeleton-loader"></div>';
     domArena.resultsB.innerHTML =
       '<div class="skeleton-loader"></div><div class="skeleton-loader"></div>';
 
     try {
+      const useHyde = document.getElementById("arena-use-hyde") ? document.getElementById("arena-use-hyde").checked : false;
+      const useReranking = document.getElementById("arena-use-reranking") ? document.getElementById("arena-use-reranking").checked : false;
+      const filterLevel = document.getElementById("arena-filter-level") ? document.getElementById("arena-filter-level").value : "all";
+      let whereClause = null;
+      if (filterLevel === "parent") {
+        whereClause = { "level": 0 };
+      } else if (filterLevel === "child") {
+        whereClause = { "level": 1 };
+      }
+
       const res = await fetch("/api/compare", {
         method: "POST",
         headers: { "Content-Type": "application/json" },
@@ -1393,6 +1523,9 @@ if (domArena.btnFight) {
           model_b: domArena.modelB.value,
           strategy_b: domArena.strategyB.value,
           retrieval_mode: domArena.retrievalMode ? domArena.retrievalMode.value : "dense",
+          use_hyde: useHyde,
+          use_reranking: useReranking,
+          metadata: whereClause,
         }),
       });
 
@@ -1403,16 +1536,35 @@ if (domArena.btnFight) {
 
       const data = await res.json();
 
+      // Render HyDE box if present
+      const hydeText = document.getElementById("arena-hyde-text");
+      if (hydeContainer && hydeText) {
+        if (data.hypothetical_answer) {
+          hydeText.textContent = `"${data.hypothetical_answer}"`;
+          hydeContainer.style.display = "block";
+        } else {
+          hydeContainer.style.display = "none";
+        }
+      }
+
       // Render Column A
       domArena.resultsA.innerHTML = data.results_a
         .map(
           (chunk, i) => `
         <div class="query-result-item" style="border-left-color: var(--warning-color)">
           <div class="query-result-header">
-            <span class="rank-badge" style="color: var(--warning-color)">Rank ${i + 1}</span>
-            <span class="dist-badge">Dist: ${chunk.score.toFixed(3)}</span>
+            <span class="rank-badge" style="color: var(--warning-color)">
+              Rank ${i + 1}
+              ${getRankShiftBadgeHtml(chunk.original_rank, i + 1)}
+            </span>
+            <span class="dist-badge">
+              ${formatScoreHtml(chunk.score, chunk.original_score, domArena.retrievalMode ? domArena.retrievalMode.value : "dense", useReranking)}
+            </span>
           </div>
           <div class="query-result-text">${chunk.text_highlighted || escapeHtml(chunk.text)}</div>
+          <div class="query-result-meta">
+            <span>ID: ${chunk.id}</span>
+          </div>
         </div>
       `,
         )
@@ -1424,10 +1576,18 @@ if (domArena.btnFight) {
           (chunk, i) => `
         <div class="query-result-item" style="border-left-color: var(--info-color)">
           <div class="query-result-header">
-            <span class="rank-badge" style="color: var(--info-color)">Rank ${i + 1}</span>
-            <span class="dist-badge">Dist: ${chunk.score.toFixed(3)}</span>
+            <span class="rank-badge" style="color: var(--info-color)">
+              Rank ${i + 1}
+              ${getRankShiftBadgeHtml(chunk.original_rank, i + 1)}
+            </span>
+            <span class="dist-badge">
+              ${formatScoreHtml(chunk.score, chunk.original_score, domArena.retrievalMode ? domArena.retrievalMode.value : "dense", useReranking)}
+            </span>
           </div>
           <div class="query-result-text">${chunk.text_highlighted || escapeHtml(chunk.text)}</div>
+          <div class="query-result-meta">
+            <span>ID: ${chunk.id}</span>
+          </div>
         </div>
       `,
         )

frontend/index.html

@@ -30,8 +30,12 @@
 
           <!-- Text Input -->
           <div class="config-section">
-            <div class="config-section-title">Input Text</div>
+            <div class="config-section-title" style="display: flex; justify-content: space-between; align-items: center;">
+              <span>Input Text</span>
+              <button type="button" id="btn-upload-file" class="btn-sm" style="cursor: pointer; background: var(--superman-blue-muted); border: 1px solid var(--superman-blue); color: var(--superman-blue); padding: 2px 8px; border-radius: 4px; font-size: 0.75rem; font-weight: bold;">📎 Attach File</button>
+            </div>
             <div class="text-input-wrapper">
+              <input type="file" id="file-input" style="display: none;" />
               <textarea
                 id="text-input"
                 class="text-input"
@@ -243,7 +247,7 @@
                 <button id="btn-open-arena" class="btn-sm" style="background: var(--superman-red); color: white; border: none; padding: 4px 12px; border-radius: 4px; cursor: pointer; font-weight: bold; font-family: var(--font-sans);">⚔️ Arena Comparison</button>
               </div>
               <div class="query-input-wrapper">
-                <select id="query-retrieval-mode" class="select-input" style="width: 110px; flex-shrink: 0; display: none;">
+                <select id="query-retrieval-mode" class="select-input" style="width: 110px; flex-shrink: 0;">
                   <option value="dense" selected>Dense</option>
                   <option value="sparse">Sparse</option>
                   <option value="hybrid">Hybrid</option>
@@ -251,6 +255,24 @@
                 <input type="text" id="query-input" placeholder="Type a concept query to fetch chunks (e.g. 'linear regression')..." />
                 <button id="btn-query" class="btn-query">🔍 Query</button>
               </div>
+              <div class="query-options-row" style="display: flex; gap: var(--space-md); margin-top: var(--space-sm); font-size: 0.85rem; color: var(--text-secondary); align-items: center; flex-wrap: wrap;">
+                <label style="display: flex; align-items: center; gap: var(--space-xs); cursor: pointer; user-select: none;">
+                  <input type="checkbox" id="query-use-hyde" style="cursor: pointer; width: 14px; height: 14px;" />
+                  🔮 Use HyDE
+                </label>
+                <label style="display: flex; align-items: center; gap: var(--space-xs); cursor: pointer; user-select: none;">
+                  <input type="checkbox" id="query-use-reranking" style="cursor: pointer; width: 14px; height: 14px;" />
+                  ⚡ Use Re-Ranking
+                </label>
+                <div style="display: flex; align-items: center; gap: var(--space-xs); margin-left: auto;">
+                  <span style="font-size: 0.8rem; color: var(--text-tertiary);">Filter Level:</span>
+                  <select id="query-filter-level" class="select-input" style="padding: 4px 8px; font-size: 0.78rem;">
+                    <option value="all">All Levels</option>
+                    <option value="parent">Only Parents (Lvl 0)</option>
+                    <option value="child">Only Children (Lvl 1)</option>
+                  </select>
+                </div>
+              </div>
               <div class="query-results-drawer" id="query-results-drawer" style="display: none;">
                 <div class="drawer-header">
                   <h3>Retrieved Context</h3>
@@ -310,7 +332,7 @@
           </div>
           
           <div class="arena-query-bar">
-            <select id="arena-retrieval-mode" class="select-input" style="width: 110px; flex-shrink: 0; display: none;">
+            <select id="arena-retrieval-mode" class="select-input" style="width: 110px; flex-shrink: 0;">
               <option value="dense" selected>Dense</option>
               <option value="sparse">Sparse</option>
               <option value="hybrid">Hybrid</option>
@@ -319,6 +341,31 @@
             <button class="btn-run" id="btn-arena-fight" style="width: 150px; flex-shrink: 0;">🔥 FIGHT!</button>
           </div>
 
+          <div class="arena-options-row" style="display: flex; gap: var(--space-md); margin: 0 0 var(--space-md) 0; padding: 0 var(--space-xl); font-size: 0.85rem; color: var(--text-secondary); justify-content: flex-start; align-items: center; flex-wrap: wrap;">
+            <label style="display: flex; align-items: center; gap: var(--space-xs); cursor: pointer; user-select: none;">
+              <input type="checkbox" id="arena-use-hyde" style="cursor: pointer; width: 14px; height: 14px;" />
+              🔮 Use HyDE
+            </label>
+            <label style="display: flex; align-items: center; gap: var(--space-xs); cursor: pointer; user-select: none;">
+              <input type="checkbox" id="arena-use-reranking" style="cursor: pointer; width: 14px; height: 14px;" />
+              ⚡ Use Re-Ranking
+            </label>
+            <div style="display: flex; align-items: center; gap: var(--space-xs); margin-left: auto;">
+              <span style="font-size: 0.8rem; color: var(--text-tertiary);">Filter Level:</span>
+              <select id="arena-filter-level" class="select-input" style="padding: 4px 8px; font-size: 0.78rem;">
+                <option value="all">All Levels</option>
+                <option value="parent">Only Parents (Lvl 0)</option>
+                <option value="child">Only Children (Lvl 1)</option>
+              </select>
+            </div>
+          </div>
+
+          <!-- Dynamic HyDE box in Arena -->
+          <div id="arena-hyde-container" style="display: none; margin: 0 var(--space-lg) var(--space-md) var(--space-lg); padding: var(--space-sm) var(--space-md); border-left: 4px solid var(--superman-blue); background-color: var(--superman-blue-muted); border-radius: var(--radius-sm);">
+            <div style="font-weight: bold; color: var(--superman-blue); font-size: 0.85rem; margin-bottom: 4px;">🔮 Generated Hypothetical Answer (HyDE)</div>
+            <div id="arena-hyde-text" style="font-style: italic; color: var(--text-primary); font-size: 0.85rem;"></div>
+          </div>
+
           <div class="arena-body">
             <div class="arena-columns">
               <!-- COLUMN A -->

frontend/styles.css

@@ -750,6 +750,75 @@ input[type="range"]::-moz-range-thumb {
   position: relative;
 }
 
+/* Query Result Footer Metadata */
+.query-result-meta {
+  display: flex;
+  gap: var(--space-md);
+  margin-top: 6px;
+  font-size: 0.68rem;
+  color: var(--text-tertiary);
+  font-family: var(--font-mono);
+  border-top: 1px dashed var(--border-default);
+  padding-top: 4px;
+}
+
+/* Distance and Score Badges */
+.dist-badge {
+  display: inline-flex;
+  align-items: center;
+  font-family: var(--font-mono);
+  font-size: 0.72rem;
+  background: var(--bg-surface-raised);
+  border: 1px solid var(--border-default);
+  padding: 2px 8px;
+  border-radius: 99px;
+  color: var(--text-secondary);
+}
+
+.score-badge-val {
+  font-weight: 600;
+  color: var(--superman-blue);
+}
+
+.score-badge-original {
+  font-size: 0.65rem;
+  color: var(--text-tertiary);
+  font-style: italic;
+  margin-left: 4px;
+}
+
+/* Rank Shift Badges */
+.rank-shift-badge {
+  display: inline-flex;
+  align-items: center;
+  gap: 2px;
+  font-size: 0.72rem;
+  font-weight: 700;
+  padding: 2px 6px;
+  border-radius: 4px;
+  margin-left: 8px;
+  text-transform: uppercase;
+  letter-spacing: 0.02em;
+}
+
+.rank-shift-up {
+  color: #16a34a;
+  background-color: rgba(22, 163, 74, 0.08);
+  border: 1px solid rgba(22, 163, 74, 0.2);
+}
+
+.rank-shift-down {
+  color: #dc2626;
+  background-color: rgba(220, 38, 38, 0.08);
+  border: 1px solid rgba(220, 38, 38, 0.2);
+}
+
+.rank-shift-unchanged {
+  color: var(--text-tertiary);
+  background-color: var(--bg-surface-raised);
+  border: 1px solid var(--border-default);
+}
+
 /* Lexical Heatmap Highlights */
 .query-result-text mark,
 .retrieved-chunk-text mark {

pyproject.toml

@@ -16,4 +16,5 @@ dependencies = [
     "umap-learn>=0.5.12",
     "numpy>=2.4.4",
     "ollama>=0.6.2",
+    "flashrank>=0.2.10",
 ]

rag_test_corpus.md

@@ -0,0 +1,93 @@
+# Dataset 1: Superhero Origins and Vulnerabilities
+
+Born Kal-El on the dying planet Krypton, the child was sent to Earth by his father Jor-El just before planetary destruction. He was raised as Clark Kent in Smallville, Kansas. Under Earth's yellow sun, his Kryptonian cells function like solar batteries, granting him superpowers: flight, heat vision, and invulnerability. However, he is vulnerable to radioactive fragments of his home planet.
+
+Green Kryptonite is the most common form; it causes immediate physical pain, weakens his muscles, and will eventually kill him if exposure is prolonged. Red Kryptonite does not weaken him physically but alters his psychology, inducing wild mood swings, aggression, and apathy. Blue Kryptonite affects only Bizarro clones, stripping them of their powers, but is harmless to normal Kryptonians. Gold Kryptonite is highly dangerous, as it permanently strips a Kryptonian of their super-abilities by destroying their cellular solar absorption capability.
+
+In his civilian identity, Clark Kent works as an investigative reporter in Metropolis. He works at the Daily Planet, a major metropolitan newspaper known for its giant golden globe symbol atop its building. His editor-in-chief is the hot-tempered Perry White, who demands hard-hitting journalistic integrity. Clark shares a desk office space with Lois Lane, a star reporter whom he eventually marries, and Jimmy Olsen, a young photographer famous for his signature signal-watch that emits a high-frequency sound only Superman can hear.
+
+Alexander Joseph Luthor, CEO of LexCorp, is a brilliant billionaire scientist obsessed with exposing Superman as a threat to humanity. LexCorp occupies a high-tech corporate campus in downtown Metropolis, featuring sub-level research laboratories.
+
+Luthor's primary weapon development project is the "Xenon-based Solar Radiation Simulator" (Project XSRS). This high-power apparatus emits synthetic red solar radiation, mimicking Krypton's red giant star, Rao. When Superman enters the simulator's projection field, his yellow sun charge is neutralized, temporarily rendering him as weak as an ordinary human. This allows Luthor's armored strike forces to engage him directly without facing super-strength.
+
+# Dataset 2: Machine Learning and Linear Regression
+
+Linear regression is a foundational supervised learning algorithm used to model the relationship between a dependent scalar variable Y and one or more explanatory variables denoted X. When there is one explanatory variable, it is called simple linear regression; for more than one, it is called multiple linear regression. The algorithm assumes a linear relationship and estimates coefficients to construct a predictive line.
+
+The model is defined by the equation Y = beta_0 + beta_1 * X + epsilon, where beta_0 represents the Y-intercept, beta_1 represents the slope coefficient (indicating the change in Y per unit change in X), and epsilon represents the residual error. The goal of fitting a regression line is to find the values of beta that minimize the differences between the predicted values and the actual observed data points.
+
+To find the optimal coefficients, the algorithm utilizes Ordinary Least Squares (OLS). OLS minimizes the Sum of Squared Residuals (SSR), which represents the vertical distances between the observed points and the fitted line. By squaring the residuals, the algorithm penalizes larger errors more heavily and prevents positive and negative errors from canceling each other out.
+
+Gradient Descent is an alternative optimization algorithm used to find the coefficients, particularly in high-dimensional settings. It starts with random values for beta and iteratively adjusts them in the direction of the steepest descent of the loss function (the negative gradient). The size of the steps taken in each iteration is determined by the learning rate parameter, which must be tuned carefully to prevent overshooting or extremely slow convergence.
+
+Overfitting occurs when a linear model fits the training data too closely, capturing random noise rather than the underlying pattern. This leads to poor generalization on unseen data. To mitigate this, regularization techniques are introduced: Ridge Regression (L2 regularization) adds a penalty term proportional to the square of the coefficients, while Lasso Regression (L1 regularization) adds a penalty proportional to the absolute values of the coefficients, effectively driving some coefficient weights to zero to perform automatic feature selection.
+
+# Dataset 3: Corporate Remote Work and Cybersecurity Policy
+
+This policy defines the security configurations and operational guidelines required for all employees working remotely. Remote access to the corporate intranet is permitted only through the Unified Endpoint Security VPN (UES-VPN), utilizing mandatory multi-factor authentication (MFA). Employees are strictly prohibited from using public, unsecured Wi-Fi networks unless a secondary virtual private network connection is active.
+
+All corporate laptops must run the designated Enterprise Threat Detection agent (ETD v4.2) at all times. The agent performs real-time heuristic scans of local file storage and intercepts unauthorized external network requests. System patches and security updates are pushed automatically every Wednesday at 03:00 UTC, and endpoints must remain powered on to receive these critical updates.
+
+Data classification guidelines dictate that any document containing personally identifiable information (PII) or proprietary algorithm designs must be tagged as "Confidential - Level 3." These files must be stored exclusively in encrypted, access-controlled SharePoint repositories. Sharing Level 3 documents via public channels, standard email attachments, or external messaging platforms is a high-severity violation subject to disciplinary action.
+
+Incident response protocols require that any suspected endpoint compromise or malware detection event be reported to the security operations center (SOC) within 15 minutes of discovery. Endpoints flagged as potentially compromised will be automatically isolated from the internal network by the active directory controller, preventing lateral movement across corporate segments while forensic teams analyze the logs.
+
+# Dataset 4: Roman History and Infrastructure
+
+The Roman Empire's expansion and stability relied heavily on its mastery of engineering and physical infrastructure. Key to Roman connectivity was the construction of paved stone roads. These roadways featured deep gravel beds for drainage, stone borders, and milestone markers indicating the distance to the local regional capital.
+
+Aqueducts were built to transport clean mountain water directly to metropolitan bathhouses and public fountains. These conduits operated solely on gravity, requiring precise calculations of gradient channels. When crossing deep valleys, Roman engineers constructed multi-tier stone arch bridges, such as the Pont du Gard, to maintain the steady flow of water without needing mechanical pumps.
+
+Concrete, or "opus caementicium," was a revolutionary Roman invention. By combining volcanic ash (pozzolana) with lime and volcanic tuff, builders created a fast-setting, water-resistant binding mortar. This concrete allowed for the construction of massive domes, such as the Pantheon in Rome, which features a coffered unreinforced concrete ceiling spanning 43 meters.
+
+Military legions were responsible for constructing fortified outposts, or "castra," along the empire's borders. These camps followed a standardized grid layout, containing central barracks, silos, and defensive palisades. Along the northern borders, walls like Hadrian's Wall stretched across miles of open terrain to isolate Roman territories from tribal raids.
+
+# Dataset 5: Molecular Biology and DNA Replication
+
+Deoxyribonucleic acid (DNA) is the double-stranded helical molecule storing the genetic blueprint of cellular life. The structure consists of nucleotide bases (adenine, thymine, cytosine, and guanine) bound to a deoxyribose-phosphate backbone. DNA replication is semiconservative, meaning each daughter molecule contains one original parent strand and one newly synthesized strand.
+
+The replication process begins at specific chromosomal regions known as the origin of replication. The enzyme DNA Helicase unwinds the double helix, breaking hydrogen bonds between complementary base pairs. This creates a Y-shaped replication fork, exposing single strands of template DNA for synthesis.
+
+To prevent the exposed single strands from re-annealing, Single-Strand Binding Proteins (SSBs) coat the open DNA. An enzyme called RNA Primase then synthesizes short RNA primers, providing the free 3'-hydroxyl group required for DNA Polymerase III to begin adding complementary DNA nucleotides.
+
+DNA replication is directional, occurring only in the 5'-to-3' direction. On the leading strand, synthesis occurs continuously toward the replication fork. On the lagging strand, synthesis runs discontinuously away from the fork, generating short segments called Okazaki fragments. These fragments are later sealed into a continuous strand by the enzyme DNA Ligase.
+
+# Dataset 6: Monetary Policy and Financial Markets
+
+Monetary policy refers to the actions taken by a nation's central bank to control the money supply and influence macroeconomic objectives, such as price stability and maximum employment. The primary tool used is the adjustment of the target interbank lending rate (e.g., the Federal Funds Rate in the United States).
+
+When inflation rises above target levels, the central bank implements contractionary monetary policy. By raising interest rates, borrowing costs increase for commercial banks, which in turn raises interest rates for business loans and mortgages. This higher cost of capital dampens consumer spending and business investment, cooling the overall economic expansion.
+
+Conversely, during economic recessions, expansionary policy is deployed. The central bank cuts interest rates to lower borrowing costs, encouraging consumer credit usage and business expansion. In extreme scenarios where rates approach zero, central banks use Quantitative Easing (QE), purchasing large quantities of long-term government bonds to inject liquidity directly into the financial system.
+
+Financial markets react dynamically to monetary decisions. Bond yields generally rise in lockstep with central bank rate hikes, causing bond prices to fall. Equity markets often experience volatility during contractionary cycles, as higher discount rates reduce the present value of future corporate earnings, making stocks less attractive compared to fixed-income assets.
+
+# Dataset 7: Classical Culinary Arts and Sourdough Fermentation
+
+The art of baking sourdough bread relies entirely on natural wild fermentation rather than commercial baker's yeast. A sourdough starter, or "levain," is a symbiotic culture of wild yeasts (primarily Candida humilis) and lactic acid bacteria (such as Lactobacillus sanfranciscensis) cultivated from flour and water.
+
+Sourdough preparation begins with autolyse, the mixing of flour and water to hydrate proteins and activate naturally occurring amylase enzymes. This resting phase initiates the development of gluten strands before the addition of salt and the levain culture.
+
+During bulk fermentation, the dough undergoes folding cycles to build structural elasticity and trap carbon dioxide gas. The lactic acid bacteria produce organic acids, giving the dough its signature sour flavor profile and lowering the pH, which improves gluten strength and increases shelf stability.
+
+The baking phase requires a high-heat environment, typically inside a preheated cast-iron Dutch oven. The trapped steam inside the closed vessel keeps the dough's surface moist, allowing the loaf to expand fully (the "oven spring") before the cover is removed to let the crust brown through the Maillard reaction.
+
+# Dataset 8: Astronomy and Star Lifecycles
+
+Stars form inside dense interstellar clouds of gas and dust known as molecular clouds or stellar nurseries. Gravity causes localized pockets in these clouds to collapse under their own mass, heating the core until nuclear fusion begins, transitioning the object into a main-sequence star.
+
+Main-sequence stars generate energy by fusing hydrogen atoms into helium in their cores. The outbound radiation pressure from this fusion process perfectly balances the inbound gravitational force, establishing a state of hydrostatic equilibrium that keeps the star stable for millions or billions of years.
+
+When a low-mass star runs out of hydrogen, its core contracts and heats up, causing the outer layers to expand and cool, transforming the star into a red giant. Eventually, the outer layers are ejected, forming a planetary nebula, leaving behind a hot, dense core called a white dwarf that slowly cools over time.
+
+Massive stars end their lives in dramatic supernova explosions. When their cores can no longer support nuclear fusion against gravity, they collapse in milliseconds, creating a shockwave that ejects the star's outer envelope. The remaining compressed core becomes either a highly magnetic neutron star or a gravitational black hole.
+
+# Dataset 9: Psychology, Memory, and Sleep Cycles
+
+Human memory is categorized into distinct stages: sensory memory, working memory, and long-term memory. Working memory holds transient information for immediate processing, while long-term memory stores vast amounts of information indefinitely through structural changes in synaptic connections, a process known as consolidation.
+
+Sleep plays a critical role in memory consolidation. During sleep, the brain actively replays neural activity from the day, transferring information from the temporary storage of the hippocampus to the permanent storage of the neocortex.
+
+The sleep cycle alternates between Non-Rapid Eye Movement (NREM) and Rapid Eye Movement (REM) sleep. NREM sleep is divided into light sleep and slow-wave sleep (SWS). Slow-wave sleep is characterized by synchronized delta brain waves, which are crucial for consolidating declarative memories (facts and events).
+
+REM sleep is characterized by rapid eye movements, high brain activity, and muscle atonia. This stage is associated with intense dreaming and is critical for consolidating procedural memories (skills and tasks) and processing emotional experiences.

uv.lock

@@ -326,6 +326,22 @@ wheels = [
     { url = "https://files.pythonhosted.org/packages/81/47/dd9a212ef6e343a6857485ffe25bba537304f1913bdbed446a23f7f592e1/filelock-3.29.0-py3-none-any.whl", hash = "sha256:96f5f6344709aa1572bbf631c640e4ebeeb519e08da902c39a001882f30ac258", size = 39812, upload-time = "2026-04-19T15:39:08.752Z" },
 ]
 
+[[package]]
+name = "flashrank"
+version = "0.2.10"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+    { name = "numpy" },
+    { name = "onnxruntime" },
+    { name = "requests" },
+    { name = "tokenizers" },
+    { name = "tqdm" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/55/1f/176cb4a857a70c3538f637e19389ab6aed21548a1ba1d1424fccc8bba108/FlashRank-0.2.10.tar.gz", hash = "sha256:f8f82a25c32fdfc668a09dc4089421d6aab8e7f71308424b541f40bb3f01d9db", size = 18905, upload-time = "2025-01-06T13:33:01.657Z" }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/ec/99/72639cc1c9221c5bc77a2df1c2d352fe11965553bdf7d3e0856e7fcc8fd6/FlashRank-0.2.10-py3-none-any.whl", hash = "sha256:5d3272ae657d793c132d1e7917ed9e2adf49e0e1c60735583a67b051c6f0434a", size = 14511, upload-time = "2025-01-06T13:32:59.42Z" },
+]
+
 [[package]]
 name = "flatbuffers"
 version = "25.12.19"
@@ -1622,6 +1638,7 @@ source = { virtual = "." }
 dependencies = [
     { name = "chromadb" },
     { name = "fastapi" },
+    { name = "flashrank" },
     { name = "httpx" },
     { name = "langchain-text-splitters" },
     { name = "nltk" },
@@ -1637,6 +1654,7 @@ dependencies = [
 requires-dist = [
     { name = "chromadb", specifier = ">=1.0.0" },
     { name = "fastapi", specifier = ">=0.115.0" },
+    { name = "flashrank", specifier = ">=0.2.10" },
     { name = "httpx", specifier = ">=0.28.0" },
     { name = "langchain-text-splitters", specifier = ">=1.1.2" },
     { name = "nltk", specifier = ">=3.9.4" },