docs/learning.md

# Prism — Learning Concepts

## Concept Index by Product Workflow

Every concept in this file maps to a specific stage in Prism's request lifecycle. Use this to understand *when* each idea becomes relevant, not just *what* it is.

```
┌─────────────────────────────────────────────────────────────────┐
│  USER UPLOADS A DOCUMENT                                        │
│                                                                 │
│  PDF/TXT/CSV → chunk → embed → store → briefing                │
│                                                                 │
│  Concept 11  RecursiveCharacterTextSplitter                     │
│              How text is split into chunks (paragraph → line    │
│              → word → char priority). Overlap bridges splits.   │
│                                                                 │
│  Concept 1   Parent-Child Chunking                              │
│              Design pattern: small child chunks for retrieval,  │
│              large parent chunks sent to LLM.                   │
│                                                                 │
│  Concept 2   InMemoryStore                                      │
│              Where parent chunks live (RAM only, dies on        │
│              restart). ChromaDB stores child vectors on disk.   │
│                                                                 │
│  Concept 14  LLM at Ingest Time (Briefing)                      │
│              After ingest, LLM auto-generates 5-bullet summary  │
│              + 3 suggested questions. Runs once, not per query. │
│                                                                 │
│  Concept 21  Contextual Retrieval                               │
│              Prepend 2-sentence situating context to each chunk │
│              before embedding. Fixes decontextualized chunks.   │
│              Measured: +18% recall (0.51→0.60), +6.6pp P@5.    │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│  USER SENDS A QUESTION                                          │
│                                                                 │
│  question → (optionally) multi-query expand → (optionally)     │
│  HyDE expand → retrieve → fuse → rerank → top-5 chunks         │
│                                                                 │
│  Concept 17  Multi-Query Retrieval                              │
│              LLM generates 3 phrasings of query → retrieve for │
│              each → pool + deduplicate → RRF → rerank.          │
│              Widens candidate pool. ON by default.              │
│                                                                 │
│  Concept 7   HyDE                                               │
│              LLM generates fake answer → embed fake answer      │
│              instead of query → closes question/answer vector   │
│              space gap → higher context recall.                 │
│              Measured: +21pp recall (0.51→0.72). ON by default. │
│                                                                 │
│  Concept 4   BM25Okapi                                          │
│              Sparse keyword retrieval. Catches exact terms       │
│              (section numbers, ₹ amounts) that dense misses.    │
│              TF saturation prevents high-frequency term bias.   │
│                                                                 │
│  Concept 3   RRF + Cross-Encoder Pipeline                       │
│              Dense (0.7) + BM25 (0.3) merged via weighted RRF. │
│              Cross-encoder reranks top-10 jointly → top-5.      │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│  WEB SEARCH (always on)                                         │
│                                                                 │
│  question → condense with history → Tavily → web results        │
│                                                                 │
│  Concept 5   ConversationalRetrievalChain Condensation Trap     │
│              Chain's internal condense step strips prepended    │
│              web context. Fix: bypass chain entirely for web    │
│              queries. Direct LLM call preserves all context.    │
│                                                                 │
│  Concept 12  ConversationBufferWindowMemory                     │
│              Sliding k=10 window of chat history injected into  │
│              condense_question() before Tavily search, so       │
│              follow-up queries have full context.               │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│  LLM GENERATES ANSWER                                           │
│                                                                 │
│  [doc chunks] + [web results] + [chat history] → LLM → answer  │
│                                                                 │
│  Concept 12  ConversationBufferWindowMemory (output_key trap)   │
│              Memory saves this turn for next question.          │
│              output_key="answer" required — chain returns       │
│              multiple keys, memory needs to know which to save. │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│  EVALUATION (offline, run via scripts/run_eval_versioned.py)    │
│                                                                 │
│  50 eval pairs → retrieve → answer → score → versioned JSON    │
│                                                                 │
│  Concept 8   Eval Metric Design (Faithfulness Is Circular)      │
│              Why faithfulness 1.0 was meaningless. How          │
│              answer_correctness (vs ground truth) is honest.    │
│                                                                 │
│  Concept 9   Precision@K vs Recall Diagnostic                   │
│              P@5=0.89 + recall=0.51 = pool too narrow.          │
│              The combination tells you exactly what to fix.     │
│                                                                 │
│  Concept 15  All 7 Eval Metrics — Full Reference                │
│              Answer Correctness, Answer Relevancy, Context      │
│              Recall, Precision@5, Latency p50/p95/p99.          │
│              Includes computation steps, failure modes,         │
│              Prism v1.0.0 results, and metric interaction map.  │
│                                                                 │
│  Concept 20  Semantic Chunking Tradeoff                         │
│              Topic-boundary splits improve recall but hurt      │
│              precision when eval pairs are aligned to fixed     │
│              chunk boundaries. Measured: +9.3pp recall,         │
│              -27.3pp P@5, 5× latency (v1.4.0 ablation).        │
└─────────────────────────────────────────────────────────────────┘
```

---

## 1. Parent-Child Chunking — The Retrieval-Faithfulness Tradeoff

### The Problem with One Chunk Size

| Chunk Size | Retrieval | LLM Answer Quality |
| --- | --- | --- |
| Small (200 chars) | Precise — matches exact phrase | Bad — too little context, answer is fragmented |
| Large (800 chars) | Imprecise — embedding averages over too much text | Good — LLM sees full context |

**Key insight:** Embeddings of large chunks get "diluted" — the vector represents the average meaning of 800 chars. Small chunks have sharper, more focused vectors that match queries better.

### How ParentDocumentRetriever Works

**INGEST:**

- 800-char parent → stored in `InMemoryStore` (keyed by ID)
- 200-char children → embedded → stored in ChromaDB

**QUERY:**

1. Query hits ChromaDB → finds best matching 200-char child chunk
2. Look up `parent_id` from child metadata
3. Return the full 800-char parent to LLM

### Why This Works

- **Child chunk** = precise retrieval target (dense vector = focused meaning)
- **Parent chunk** = rich answer context (LLM gets surrounding sentences)

### Concrete Example — RBI Circular PDF

**Raw text (one paragraph):**

> "The Reserve Bank of India has mandated that all UPI transactions above ₹2,000 must undergo additional authentication from January 2025. This includes biometric verification or OTP-based second factor. Non-compliant PSPs will face penalties up to ₹10 lakh per violation."
> 

**After chunking:**

- **Parent (800 chars) → InMemoryStore:** Full paragraph above
- **Child A (200 chars) → ChromaDB:** "The Reserve Bank of India has mandated that all UPI transactions above ₹2,000 must undergo additional authentication from January 2025."
- **Child B:** "This includes biometric verification or OTP-based second factor."
- **Child C:** "Non-compliant PSPs will face penalties up to ₹10 lakh per violation."

**Query:** `"What is the UPI transaction authentication limit?"`

→ ChromaDB finds **Child A** (score: 0.91) — focused vector on UPI + ₹2000 + authentication

→ Fetches parent_id → returns **full parent paragraph** to LLM

**Comparison Table:**

| Approach | LLM Gets | Problem |
| --- | --- | --- |
| Dense-only, large chunks | Full paragraph (good) | Match was imprecise — wrong paragraph might score higher |
| Dense-only, small chunks | Child A only (38 words) | Misses penalty info — incomplete answer |
| ParentDocumentRetriever | Full paragraph (precise match + rich context) | ✅ Best of both |

**LLM final answer:** "UPI transactions above ₹2,000 require additional authentication (biometric or OTP) from Jan 2025. Non-compliant PSPs face up to ₹10 lakh penalty."

---

## 2. InMemoryStore — What It Actually Is

**Key point: InMemoryStore is NOT part of ChromaDB. It's a separate, RAM-only store.**

Under the hood it's essentially a plain Python dict wrapped in a LangChain class:

```python
store = {}
store["parent_id_abc123"] = "Full 800-char parent chunk text..."
store["parent_id_def456"] = "Another parent chunk..."
```

### Two Separate Stores in Prism

| ChromaDB (on disk) | InMemoryStore (RAM only) |
| --- | --- |
| Child chunk vectors | Parent chunk text |
| [vector, metadata] → find similar | parent_id → full text |
| ✅ Survives restart | ❌ Dies on restart |

### Flow

```
Child A metadata = { "parent_id": "abc123", "text": "short child..." }
                            ↓
             InMemoryStore["abc123"]
                            ↓
             "Full 800-char parent text" → LLM
```

### ⚠️ Known Limitation in Prism

Render free tier cold-starts → InMemoryStore is wiped → must **re-ingest PDFs on every cold start**.

ChromaDB persists to disk so child vectors survive, but parent text is gone.

### ⚠️ Architecture Note — Intended vs Current Implementation

`architecture.md` and Concept 1 describe ParentDocumentRetriever (child 200 / parent 800). The current `ingest.py` uses a single-pass `RecursiveCharacterTextSplitter` at 500 chars — no separate parent store. The ParentDocumentRetriever was the original design and is documented as such. Both chunking approaches teach the same tradeoff; the parent-child concept remains valid as a pattern even if the current code simplified to single-pass chunking.

---

## 3. RRF + Cross-Encoder Reranker Pipeline

### 3a. RRF — Reciprocal Rank Fusion

**Problem:** Dense retrieval returns a ranked list. BM25 returns a ranked list. Scores are on different scales (BM25: 0–15, cosine: 0–1) — can't add them directly.

**RRF Solution:** Ignore raw scores. Use only rank position.

```
Standard RRF formula:
score(doc) = Σ  1 / (k + rank_in_list)
             k = 60  (constant, dampens top-rank advantage)

Prism's weighted RRF (applied inside the formula per list):
score(doc) += dense_weight  / (k + rank_in_dense_list)   # 0.7 × contribution
score(doc) += sparse_weight / (k + rank_in_sparse_list)  # 0.3 × contribution
```

**Example:**

| Doc | Dense (ChromaDB) | BM25 | RRF Score |
| --- | --- | --- | --- |
| Doc A | Rank 1 (0.91) | Rank 2 (9.1) | 1/61 + 1/62 = **0.0325** ✅ |
| Doc C | Rank 3 (0.71) | Rank 1 (12.3) | 1/63 + 1/61 = **0.0320** |
| Doc B | Rank 2 (0.87) | — | 1/62 = 0.0161 |
| Doc D | — | Rank 3 (7.4) | 1/63 = 0.0159 |

**Doc A wins** — appeared high in BOTH lists → signals true relevance.

> In Prism: `dense_weight=0.7`, `sparse_weight=0.3` applied **inside** the RRF formula — each list's contribution is multiplied by its weight before summing. Not "before RRF" as a pre-filter, but as a per-list scaling factor within fusion (see Concept 4 for the actual code).
> 

### 3b. Cross-Encoder Reranker

**Problem:** RRF gives top-10 candidates. Bi-encoder (ChromaDB) encodes query and doc *separately* → approximate similarity.

**Cross-Encoder:** Feeds query + doc *together* into BERT → full attention across both → much more accurate relevance score.

|  | Bi-Encoder (ChromaDB) | Cross-Encoder (TinyBERT) |
| --- | --- | --- |
| Method | embed(query) + embed(doc) → cosine(q,d) | BERT([query][SEP][doc]) → single score 0–1 |
| Speed | Fast | Slow |
| Accuracy | Approximate | Accurate |
| Encoding | Independent | Joint |

**Example — after RRF top-10:**

| Pair | Cross-Encoder Score |
| --- | --- |
| (query, Doc A) | 0.94 ✅ |
| (query, Doc B) | 0.88 ✅ |
| (query, Doc C) | 0.61 |
| (query, Doc D) | 0.23 |

Top-5 by cross-encoder score → LLM

### Full Retrieval Pipeline

```
Query
  |
  ├→ ChromaDB dense  → top-10 ranked docs
  ├→ BM25 sparse     → top-10 ranked docs
  |
  ↓
RRF fusion → merged top-10 (rank-based, scale-agnostic)
  |
  ↓
Cross-encoder → re-scores all 10 jointly with query
  |
  ↓
Top-5 parent chunks → LLM
```

---

---

## 4. BM25Okapi — Why Keywords Beat Embeddings for Exact Terms

### The Problem with Dense Retrieval on Regulatory Text

Embeddings capture *meaning*. But regulatory text has exact identifiers — section numbers, policy codes, rupee amounts — where the exact token matters, not the meaning.

**Example:**

Query: `"What is the penalty for UPI non-compliance?"`

A dense embedding model reads this as: *"something about UPI and consequences"*.

Two chunks in corpus:
- Chunk A: `"Non-compliant PSPs will face penalties up to ₹10 lakh per violation."` ← exact answer
- Chunk B: `"UPI has transformed digital payments in India with over 100 billion transactions."` ← semantically close (UPI topic) but wrong

Dense retrieval might rank Chunk B high because it's heavily UPI-themed. BM25 ranks Chunk A high because "penalt" and "non-compli" are rare, high-signal tokens.

### How BM25Okapi Works

Plain TF-IDF problem: a doc that says "UPI" 50 times gets 50× the score. That's unfair — one mention of "₹10 lakh" in the right context should beat 50 mentions of "UPI" in a generic overview.

BM25 fixes this with **term frequency saturation**:

```
BM25 score = IDF(term) × [ tf × (k1 + 1) ] / [ tf + k1 × (1 - b + b × dl/avgdl) ]

Where:
  tf    = how many times term appears in this chunk
  IDF   = how rare the term is across all chunks (log scale)
  k1    = saturation constant (~1.5) — controls how fast TF saturates
  b     = length normalization (~0.75)
  dl    = this chunk's length
  avgdl = average chunk length in corpus
```

**Saturation in plain English:**

| tf (term count in chunk) | TF-IDF score | BM25 score (k1=1.5) |
| --- | --- | --- |
| 1 | 1.0 | 1.0 |
| 5 | 5.0 | 1.56 ← plateaus |
| 20 | 20.0 | 1.79 ← barely grows |
| 50 | 50.0 | 1.88 ← effectively capped |

A chunk mentioning "UPI" 50 times scores almost the same as one mentioning it 5 times. But "₹10 lakh" appearing once in a chunk that has it = high IDF (rare token) × full TF benefit.

### In Prism

```python
# bm25_index.py
from rank_bm25 import BM25Okapi

# .lower() matters — "UPI" and "upi" are different tokens without it
corpus = [doc["content"].lower().split() for doc in all_chunks]
bm25 = BM25Okapi(corpus)

# At query time — query also lowercased to match corpus tokenization:
scores = bm25.get_scores(query.lower().split())
top_k_idx = sorted(range(len(scores)), key=lambda i: scores[i], reverse=True)[:k]
```

BM25 operates on raw tokens (word split). No embeddings. No GPU. Rebuilds in ~1s at startup from corpus.

**Weight in Prism:** `sparse_weight=0.3` in RRF — BM25 is complementary, not dominant. Dense handles semantic meaning; BM25 catches exact matches dense misses.

---

## 5. ConversationalRetrievalChain — The Silent Context Killer

### What the Chain Does Internally

`ConversationalRetrievalChain` has two internal steps most people don't know about:

```
User question + chat history
        ↓
[STEP 1] Condense question
        LLM rewrites "Is the price level good?" 
        → "Is Bajaj Finance stock ₹908-924 a good buy in 2026?"
        (standalone question for retrieval)
        ↓
[STEP 2] Retrieve + Answer
        Standalone question → retriever → top-5 chunks → LLM answer
```

Step 1 exists so follow-up questions work without full history context. Good design for RAG.

### The Bug: Web Context Gets Stripped

When Prism added Tavily web search, the naive approach was:

```python
web_results = tavily.search(question)
augmented_question = f"Web context: {web_results}\n\nQuestion: {question}"
chain.invoke({"question": augmented_question})  # ← WRONG
```

What actually happens:

```
augmented_question (with Tavily content prepended)
        ↓
[STEP 1] Chain's condense LLM:
        "Rewrite this as a standalone retrieval query."
        Output: "What is Bajaj Finance stock price?"
        ← ALL TAVILY CONTENT STRIPPED. LLM never sees it.
        ↓
[STEP 2] Answer LLM gets: corpus chunks only. No web context.
```

The chain's condensation step rewrites the question for retrieval quality — and throws away everything else.

### The Fix in Prism

Bypass the chain entirely for web queries. Direct LLM call:

```python
# chain.py
def run_query_with_web(question, rag_docs, web_sources, memory):
    history = memory.load_memory_variables({})["history"]
    
    prompt = f"""
    Chat history: {history}
    
    Web search results:
    {web_sources}
    
    Document context:
    {rag_docs}
    
    Question: {question}
    Answer:"""
    
    answer = llm.invoke(prompt)
    memory.save_context({"input": question}, {"answer": answer})
    return answer
```

No condensation step. Web context reaches the LLM guaranteed.

**Lesson:** LangChain abstractions are powerful but opaque. When something doesn't work, read what the chain actually does internally — don't assume the abstraction is transparent.

---

## 7. HyDE — Hypothetical Document Embeddings

### The Problem with Embedding a Question

When you search ChromaDB, you embed the *query* and find similar vectors. But your corpus contains *answers*, not questions. They live in different regions of vector space.

**Concrete example:**

Query: `"What is the UPI transaction limit for P2P transfers?"`

This embeds as a *question vector* — the model has seen millions of questions, it knows this pattern.

The answer in your corpus: `"P2P UPI transfers are capped at ₹1 lakh per transaction per day as per NPCI guidelines."`

This embeds as an *answer vector* — declarative sentence, factual tone, different region in 1536-dimensional space.

They're semantically related, but the vector distance is larger than it should be.

### HyDE's Solution

Before searching, ask the LLM to hallucinate an answer:

```
Step 1: LLM generates a fake answer to the query
  Query: "What is the UPI transaction limit for P2P transfers?"
  Fake answer: "The UPI transaction limit for P2P transfers is typically 
                set by NPCI and varies by bank, generally around ₹1 lakh 
                per day for most PSPs."

Step 2: Embed the FAKE ANSWER (not the query)
  vector = embed("The UPI transaction limit for P2P transfers is...")

Step 3: Search ChromaDB with this vector
  → Finds real answer chunks that are semantically close to a fake answer
  → Much closer in vector space than the original question was
```

### Why "Hypothetical" Works

The fake answer and the real corpus chunk are both declarative, factual, answer-shaped text. They live in the same region of vector space. The query (a question) lives elsewhere.

```
Vector space (simplified):

[Question zone]          [Answer zone]
"What is UPI limit?" --- ... --- "P2P capped at ₹1 lakh..."
       ↑                                    ↑
  far from corpus                    close to corpus chunk
  
HyDE:
"UPI limit is ~₹1 lakh..." ← fake answer → close to real chunk ✅
```

### In Prism

```python
# retriever.py
def _hyde_expand(self, query: str) -> str:
    prompt = f"Write a 2-sentence factual answer to: {query}"
    return self.llm.invoke(prompt).content

def _get_relevant_documents(self, query: str):
    dense_query = self._hyde_expand(query) if self.use_hyde else query
    
    # Dense search uses fake answer embedding
    dense_docs = self.vectorstore.similarity_search(dense_query, k=10)
    
    # BM25 still uses original query (keyword matching needs real terms)
    sparse_docs = self.bm25_retrieve(query, k=10)
    
    return self.rrf_and_rerank(dense_docs, sparse_docs)
```

**ON by default** (`config.yaml` `hyde_enabled: true`) — adds one Groq call per query (~200ms). Latency cost accepted.

**Measured lift (v1.1.0, 18 samples):** context_recall 0.51 → 0.72 (+21pp). Latency 2029ms → 4018ms p50 (2×). HyDE specifically helps recall — it finds chunks that keyword/direct-embed matching misses.

---

## 8. Eval Metric Design — Why Faithfulness Is Circular

### Four RAGAS Metrics and What They Actually Measure

| Metric | Judge compares | Question it answers |
| --- | --- | --- |
| **faithfulness** | LLM answer vs retrieved chunks | "Did the LLM stick to what was in the retrieved docs?" |
| **answer_correctness** | LLM answer vs ground_truth reference | "Is the answer actually right?" |
| **answer_relevancy** | LLM answer vs original question | "Did the answer address what was asked?" |
| **context_recall** | Retrieved chunks vs ground_truth | "Did retrieval find the chunks needed to answer?" |

### The Faithfulness Trap

In Prism v1 eval, faithfulness scored 1.0. Seemed great. Was meaningless.

Here's why:

```
Eval pair designed alongside corpus:
  Question: "What is the UPI P2P transaction limit?"
  Ground truth: "₹1 lakh per day"
  
  Corpus chunk (also written by us): 
  "P2P UPI transfers are capped at ₹1 lakh per day per NPCI guidelines."

Flow:
  1. LLM retrieves that chunk (of course — it's perfectly matched)
  2. LLM answers: "The UPI P2P limit is ₹1 lakh per day."
  3. RAGAS faithfulness judge: "Does answer match retrieved chunk?" → YES → score: 1.0
```

The judge is comparing the answer to the chunk that was *designed to produce that answer*. Circular. Score tells you the retrieval worked, not whether the answer is correct.

### answer_correctness Is the Honest Metric

```
Flow:
  1. LLM answers: "The UPI P2P limit is ₹1 lakh per day."
  2. Ground truth (human-written reference): "₹1 lakh per transaction per day"
  3. RAGAS judge: "Does answer match ground truth?" → mostly yes → score: 0.82
```

Judge compares to an *independent human reference*. No circular dependency on retrieved chunks. Harder to game.

**Prism v1.0.0 Violet results:**

| Metric | Score | Interpretation |
| --- | --- | --- |
| answer_correctness | 0.82 | 82% of answers match ground truth — honest signal |
| answer_relevancy | 0.62 | RAGAS penalizes verbosity — Groq 70B tends to over-explain |
| context_recall | 0.51 | Only 51% of needed chunks retrieved — target for Multi-Query |
| P@5 | 0.89 | When chunks are retrieved, 89% are correct — precision is fine |

### The P@5 + Recall Diagnostic

P@5=0.89 + recall=0.51 tells you something specific:

```
Retrieval pool is too narrow.

"When we retrieve something, it's usually right (0.89)."
"But we're missing ~half the relevant chunks (0.51)."

Root cause: single-phrasing query misses chunks phrased differently.
Fix: Multi-Query Retrieval — generate 3 phrasings, retrieve for each, pool candidates.
```

This is exactly how production ML teams diagnose retrieval systems. The combination of metrics points to the specific fix.

---

## 9. Precision@K vs Recall — The Retrieval Diagnostic Pair

### Definitions in Plain English

Imagine your corpus has **5 chunks** that are genuinely relevant to a query. Your retriever returns **5 chunks** (K=5).

```
Ground truth relevant chunks: [A, B, C, D, E]
Retrieved chunks:              [A, B, X, Y, Z]

Precision@5 = correct retrieved / K = 2/5 = 0.40
              "Of what I returned, how much was right?"

Recall@5    = correct retrieved / total relevant = 2/5 = 0.40
              "Of all the right chunks, how many did I find?"
```

### The Four Diagnostic Combinations

| P@5 | Recall | Diagnosis | Fix |
| --- | --- | --- | --- |
| High | High | ✅ Retrieval working well | Ship it |
| High | Low | Pool too narrow — finding right chunks but missing others | Multi-Query Retrieval, HyDE |
| Low | High | Too much noise — finding relevant chunks but also junk | Better reranking, stricter K |
| Low | Low | Retrieval fundamentally broken | Check embeddings, chunking strategy |

### Prism v1.0.0 Violet: High P, Low Recall

```
P@5 = 0.89 → "When Prism retrieves a chunk, 89% of the time it's relevant."
Recall = 0.51 → "But Prism only finds 51% of the relevant chunks total."

Example query: "What are the RBI guidelines on UPI merchant limits?"

Relevant chunks in corpus: [merchant_limit_2024, merchant_kyc, limit_circular_2023, payment_cap, psp_obligations]

Prism retrieved: [merchant_limit_2024, merchant_kyc, some_unrelated_chunk, another_unrelated, payment_cap]

Precision@5 = 3/5 = 0.60 (found 3 right ones, 2 noise)
Recall = 3/5 = 0.60 (missed limit_circular_2023 and psp_obligations)
```

Why did it miss them? `limit_circular_2023` might use different phrasing: *"The ceiling for merchant UPI collections was revised..."* — no overlap with "merchant limits". Single-phrasing retrieval misses it.

Multi-Query generates: `"UPI merchant payment ceiling"`, `"RBI merchant collection limit"`, `"PSP merchant UPI cap"` → retrieves from all three → pools candidates → recall rises.

---

---

## 11. RecursiveCharacterTextSplitter — How Text Gets Chunked

### The Problem with Fixed Splits

Splitting every 500 characters naively:

```
"...comply with Section 7(b) of the Act. |SPLIT| Payment systems must maintain..."
```

Cut mid-sentence. "Section 7(b)" separated from "Payment systems must maintain" — the context that makes "Section 7(b)" meaningful is now in a different chunk. The embedding for that chunk is weak.

### How RecursiveCharacterTextSplitter Works

It tries a priority list of separators, from most preferred to least:

```python
separators = ["\n\n", "\n", " ", ""]
# Priority: paragraph break > line break > word break > character break
```

For a given `chunk_size=500`:

1. **Try `\n\n` first** — split at paragraph boundaries. If resulting piece ≤ 500 chars: done.
2. **If piece still >500** — try `\n` (line breaks within the paragraph).
3. **If still >500** — try spaces (word boundaries).
4. **Last resort** — split at exact character position.

**Result:** Chunks break at natural language boundaries when possible, hard character cuts only when forced.

### Overlap — Why Adjacent Chunks Share Text

```python
splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,
    chunk_overlap=50,  # last 50 chars of chunk N = first 50 chars of chunk N+1
)
```

Without overlap:
```
Chunk 1: "...UPI transaction limits are set by NPCI guidelines for"
Chunk 2: "each payment service provider based on risk assessment."
```
Query `"UPI limit by PSP"` — the full sentence is split. Neither chunk alone has a strong embedding.

With overlap=50:
```
Chunk 1: "...UPI transaction limits are set by NPCI guidelines for each"
Chunk 2: "for each payment service provider based on risk assessment."
```
The bridging phrase appears in both chunks. One of them will have a strong vector for the full concept.

### In Prism

```python
# ingest.py
splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,   # from config.yaml
    chunk_overlap=50,
)
chunks = splitter.split_documents(documents)
```

**Note:** Prism originally used ParentDocumentRetriever (child 200 / parent 800). The current code uses a single-pass splitter at 500 chars. The architecture.md documents the intended design; the code reflects the current implementation. Both teach the same concept — chunk size is a precision-vs-context tradeoff.

---

## 12. ConversationBufferWindowMemory — Sliding Window Chat History

### Why You Need Memory in a Chat System

Each `POST /api/chat` is a stateless HTTP request. The LLM has no memory of what was said 10 seconds ago. Without memory:

```
User: "Tell me about UPI transaction limits."
Prism: "UPI P2P limit is ₹1 lakh per day..."

User: "What about merchant payments?"   ← no context
LLM sees: question = "What about merchant payments?"  ← what is "about"? who knows?
Prism: "Merchant payments are a type of..."  ← wrong, generic answer
```

### How ConversationBufferWindowMemory Works

Keeps last `k` conversation turns (human + AI) as a rolling window:

```python
# memory.py
memory = ConversationBufferWindowMemory(
    memory_key="chat_history",
    return_messages=True,
    output_key="answer",   # ← critical — explained below
    k=10,                  # keep last 10 turns
)
```

Turn 1: `[Human: "Tell me about UPI limits", AI: "UPI P2P is ₹1 lakh..."]`
Turn 2: `[Human: "Tell me about UPI limits", AI: "...", Human: "What about merchant payments?", AI: "Merchant UPI limit is ₹5 lakh..."]`

At turn 11: Turn 1 is evicted. Window always has last 10.

The chain sees:
```
Chat history:
  Human: Tell me about UPI limits
  AI: UPI P2P is ₹1 lakh per day...

Current question: What about merchant payments?
```
→ LLM understands "What about" refers to UPI limits. Answers correctly.

### The output_key Trap

`ConversationalRetrievalChain` returns a dict with multiple keys:

```python
result = chain.invoke({"question": "..."})
# result = {
#   "answer": "UPI P2P limit is ₹1 lakh...",
#   "source_documents": [...],
#   "question": "..."
# }
```

Memory's `save_context` call needs to know which key is the "output" to save:

```python
# WRONG — KeyError because chain returns multiple output keys
memory = ConversationBufferWindowMemory(memory_key="chat_history")

# RIGHT — tell memory exactly which key to save
memory = ConversationBufferWindowMemory(
    memory_key="chat_history",
    output_key="answer",   # save result["answer"], not the whole dict
)
```

Without `output_key="answer"`: LangChain tries to infer the output key, sees multiple candidates, raises `ValueError: Multiple keys returned`. This was a real bug hit during Prism development — subtle because the chain runs fine; the crash happens on the `save_context` call after.

---

## 14. LLM at Ingest Time — The Briefing Pattern

### Two Places LLMs Can Run in a RAG System

Most tutorials show LLM running only at **query time**: user asks → retrieve → LLM answers.

Prism also runs LLM at **ingest time**: document uploaded → LLM summarizes → shown to user immediately.

```
INGEST TIME (once, when doc uploaded):
  PDF → chunks → embed → store
                ↓ (also)
            LLM reads first 6 chunks
            → generates 5-bullet summary
            → generates 3 suggested questions
            → returned in upload response

QUERY TIME (every user message):
  question → retrieve → LLM answers
```

### Why This Is Useful

When a user uploads an unfamiliar document, they don't know what to ask. The briefing gives them:

1. **Orientation** — "what is this document about?"
2. **Starting questions** — clickable prompts that immediately work

```python
# briefing.py
def generate_briefing(doc_name: str, text_sample: str) -> dict:
    prompt = f"""
    Analyze this document excerpt and respond with valid JSON only.
    Document: {doc_name}
    Content: {text_sample[:3000]}
    
    Return exactly: {{"summary": ["5 bullet strings"], "suggested_questions": ["3 question strings"]}}
    """
    response = llm.invoke([HumanMessage(content=prompt)])
    # Parse JSON from response → return to frontend
```

```python
# routes/upload.py — after ingestion completes:
briefing = generate_briefing(doc_name, sample_text)
return {
    "uploaded": [filename],
    "documents": docs,
    "briefing": briefing,   # ← frontend shows this immediately
}
```

### Design Decisions

**Non-critical path** — briefing failure doesn't block upload:
```python
try:
    briefing = generate_briefing(...)
except Exception as e:
    logger.warning("Briefing skipped: %s", e)
    briefing = None  # upload still succeeds
```

**JSON extraction from LLM output** — LLMs often wrap JSON in markdown fences (` ```json`). Prism strips these with regex before `json.loads()`:
```python
raw = re.sub(r"^```(?:json)?\s*", "", raw)  # strip opening fence
raw = re.sub(r"\s*```$", "", raw)           # strip closing fence
match = re.search(r"\{.*\}", raw, re.DOTALL)  # extract JSON object
data = json.loads(match.group())
```

**When to use LLM at ingest time vs query time:**

| | Ingest time | Query time |
| --- | --- | --- |
| Runs | Once per document | Once per user message |
| Cost | Fixed (paid once) | Scales with usage |
| Best for | Document-level metadata, summaries, question suggestions | Answering specific user questions |
| Examples | Briefing, contextual retrieval, chunk tagging | RAG answer, faithfulness eval, HyDE |

The most powerful use of ingest-time LLM is **contextual retrieval** (roadmap): for every chunk, ask LLM "given this full document, write 2 sentences situating this chunk" — then prepend that context before embedding. Anthropic reports ~49% reduction in retrieval failures. Same LLM-at-ingest-time pattern, much higher impact.

---

---

## 15. Eval Dashboard Metrics — Full Reference

Prism's eval dashboard shows 7 numbers per run. Each answers a different question about the system. Understanding *what* each metric measures, *how* it's computed, and *what can fool it* is the difference between blindly running evals and actually improving the system.

---

### Overview — What Each Metric Diagnoses

```
Query → [Retrieval] → top-5 chunks → [LLM] → answer
           ↑                ↑                 ↑
      Precision@5     Context Recall     Answer Correctness
                                         Answer Relevancy
                                         Latency p50/p95/p99
```

| Metric | What it measures | Requires ground truth? | LLM call? |
| --- | --- | --- | --- |
| Answer Correctness | Is the answer factually right? | Yes | Yes (judge) |
| Answer Relevancy | Does answer address the question? | No | Yes + embed |
| Context Recall | Did retrieval find all needed chunks? | Yes | Yes (RAGAS) |
| Precision@5 | Are the top-5 chunks relevant? | Yes (keywords/sources) | No |
| Latency p50/p95/p99 | How fast is the system? | No | No |

---

### Metric 1 — Answer Correctness (primary)

**Question it answers:** "Is Prism's answer actually correct?"

**Why it's primary:** The only metric that directly measures answer quality vs a human-authored reference. Everything else is a proxy.

**How it's computed:**

```python
CORRECTNESS_PROMPT = """
You are an evaluation judge. Score the generated answer against the reference answer.

Use a 1–5 integer scale:
5 — All key facts present and correct
4 — Most key facts correct, minor omissions or imprecision
3 — Some key facts correct, moderate gaps
2 — Few facts correct, significant errors or hallucinations
1 — Completely wrong, irrelevant, or contradicts reference

Reference answer: {ground_truth}
Generated answer: {answer}

Respond with JSON: {"score": 4, "reason": "one sentence"}
"""

# Normalize 1–5 → 0–1
score_normalized = (raw_score - 1) / 4
```

**Example:**

```
Ground truth: "UPI P2P limit is ₹1 lakh per transaction per day per NPCI guidelines."
Answer:       "You can transfer up to ₹1 lakh daily on UPI for person-to-person transfers."

Judge: score=4 (correct amount + correct type, slight imprecision on "per transaction")
Normalized: (4-1)/4 = 0.75
```

**What the judge model sees:** Only the answer and ground truth. NOT the retrieved chunks. This is intentional — if the answer is wrong but the chunks were correct, that's an LLM reasoning failure. If the answer is right but the chunks were irrelevant, that's lucky hallucination.

**What fools it:**
- 8B judge is lenient — "approximately correct" often scores 4/5
- Rephrased correct answers may score 3 if judge doesn't recognize equivalence
- Very long answers may confuse the judge — score correct facts buried in padding lower

**Prism v1.0.0 result:** `0.82` — 82% of answers have most key facts correct. Target: >0.85 via better retrieval (Multi-Query → more context → better answers).

---

### Metric 2 — Answer Relevancy

**Question it answers:** "Did Prism answer the question that was actually asked, or did it go off on a tangent?"

**How it's computed (RAGAS):**

```
Step 1: Given the generated answer, LLM generates N reverse questions
        Answer: "UPI P2P limit is ₹1 lakh per day..."
        Reverse Q1: "What is the UPI transaction limit?"
        Reverse Q2: "How much can you transfer via UPI P2P daily?"
        Reverse Q3: "What is the per-day UPI P2P cap?"

Step 2: Embed original question + all N reverse questions
        embed("What is the UPI transaction limit for P2P transfers?") → vector_q
        embed("What is the UPI transaction limit?") → vector_r1
        embed("How much can you transfer via UPI P2P daily?") → vector_r2

Step 3: Cosine similarity between original question and each reverse question
        sim(vector_q, vector_r1) = 0.94
        sim(vector_q, vector_r2) = 0.91
        sim(vector_q, vector_r3) = 0.89

Step 4: Answer relevancy = mean similarity = (0.94 + 0.91 + 0.89) / 3 = 0.91
```

**Intuition:** If the answer actually addresses the question, reverse-engineered questions from that answer will be similar to the original question. If the answer went off-topic, the reverse questions will diverge.

**What fools it:**
- **Verbosity penalty:** Groq 70B tends to over-explain. A padded answer like "Great question! UPI P2P limit is ₹1 lakh. UPI has transformed payments in India, with many PSPs offering..." generates reverse questions about UPI history, not the limit → lower similarity → lower score.
- Doesn't check factual correctness — a confident wrong answer can score high if it's "on topic".

**Prism v1.0.0 result:** `0.62` — lower than expected. Direct cause: Groq 70B verbose responses. Fix: tighter system prompt (`"Be concise. One paragraph maximum."`).

---

### Metric 3 — Context Recall

**Question it answers:** "Did the retriever surface all the chunks that were needed to construct a correct answer?"

**How it's computed (RAGAS):**

```
Step 1: Take ground truth reference answer
        "UPI P2P limit is ₹1 lakh per day. Merchant UPI payments are capped at ₹5 lakh."

Step 2: RAGAS LLM breaks it into individual factual sentences (claims)
        Claim A: "UPI P2P limit is ₹1 lakh per day."
        Claim B: "Merchant UPI payments are capped at ₹5 lakh."

Step 3: For each claim, check: can this claim be attributed to any retrieved chunk?
        Claim A → search retrieved chunks → Chunk 3 contains "P2P cap ₹1 lakh" → ATTRIBUTED ✓
        Claim B → search retrieved chunks → no chunk mentions merchant cap → NOT ATTRIBUTED ✗

Step 4: Context recall = attributed claims / total claims = 1/2 = 0.50
```

**Intuition:** Recall measures completeness of the retrieval pool. Low recall = the retriever missed relevant chunks → LLM can't answer completely even if it tries → answer will be incomplete or hallucinated.

**Why this is the most important metric to improve:**
- Low recall (0.51 in v1.0.0) means ~half the relevant information is missing from context
- LLM can only work with what it's given — perfect LLM + bad retrieval = bad answer
- Improving recall directly improves answer quality

**What fools it:**
- Ground truth phrasing matters — if GT says "₹1 lakh" and the chunk says "100,000 rupees", RAGAS LLM may or may not link them
- Long GT answers with many claims → denominator grows → harder to achieve high recall

**Prism v1.0.0 result:** `0.51` — retriever finds correct chunks (P@5=0.89) but misses ~half the relevant ones. **Directly motivates Multi-Query Retrieval.**

---

### Metric 4 — Precision@5

**Question it answers:** "Of the 5 chunks Prism returned, how many were actually relevant?"

**How it's computed (custom, deterministic — no LLM):**

```python
def compute_precision_at_k(query, retrieved_chunks, ground_truth, k=5):
    relevant_sources = ground_truth.get("relevant_sources", [])   # expected source filenames
    keywords = ground_truth.get("relevant_chunk_keywords", [])     # expected keywords

    top_k = retrieved_chunks[:k]
    relevant_count = 0

    for chunk in top_k:
        source_match = chunk["source"] in relevant_sources
        keyword_match = any(kw.lower() in chunk["content"].lower() for kw in keywords)
        
        if source_match or keyword_match:    # OR — either condition counts
            relevant_count += 1

    return relevant_count / k   # 0.0 – 1.0
```

**Example eval pair:**
```json
{
  "query": "What is the UPI merchant payment limit?",
  "relevant_sources": ["npci_merchant_guidelines.pdf"],
  "relevant_chunk_keywords": ["merchant", "₹5 lakh", "payment cap", "PSP ceiling"]
}
```

Retrieved chunks:
```
Chunk 1 (source: npci_merchant_guidelines.pdf) → source_match ✓ → relevant
Chunk 2 (content: "...merchant PSP ceiling is ₹5 lakh...") → keyword_match ✓ → relevant
Chunk 3 (source: rbi_circular.pdf, content: "UPI transaction monitoring...") → neither → irrelevant
Chunk 4 (source: npci_merchant_guidelines.pdf) → source_match ✓ → relevant
Chunk 5 (content: "...digital payment systems in India...") → neither → irrelevant

P@5 = 3/5 = 0.60
```

**Why deterministic (no LLM):**
- Fast — runs per-query without API calls
- Reproducible — same query always same score
- No rate limits — can evaluate 50 pairs without hitting Groq TPD

**What fools it:**
- Source OR keyword: a chunk from the wrong document that happens to mention a keyword counts as relevant
- Keyword list quality matters — too broad → everything matches, score inflates → too narrow → correct chunks missed, score deflates

**Prism v1.0.0 result:** `0.89` — 89% of retrieved chunks are relevant. High precision + low recall = pool is accurate but too narrow. The retriever is finding the right kind of chunks, just not enough of them.

---

### Metric 5 — Latency p50 / p95 / p99

**Question it answers:** "How fast is Prism in practice?"

**Why three numbers instead of average:**

```
Query latencies (ms): [1800, 1950, 2100, 1900, 2050, 2200, 1850, 3800, 2000, 1950]
                                                                  ↑ one slow outlier

Mean: 2160ms  ← pulled up by the outlier, misleading
p50:  1975ms  ← 50% of queries faster than this (typical experience)
p95:  3230ms  ← 95% of queries faster than this (worst normal case)
p99:  3740ms  ← 99% of queries faster than this (absolute worst)
```

**How it's computed:**

```python
import numpy as np

# Measure per query: retrieval + LLM call (no Tavily, no web search)
t0 = time.time()
answer, docs, chunks = _answer_query(llm, retriever, query)
latency_ms = int((time.time() - t0) * 1000)
latencies.append(latency_ms)

# After all queries:
p50 = int(np.percentile(latencies, 50))   # median
p95 = int(np.percentile(latencies, 95))   # 95th percentile
p99 = int(np.percentile(latencies, 99))   # 99th percentile
```

**What's included in the latency measurement:**
- ChromaDB dense retrieval (cosine ANN search)
- BM25 sparse retrieval (in-memory token scoring)
- RRF fusion (pure Python, negligible)
- TinyBERT cross-encoder reranking (BERT forward pass × 10 pairs)
- Groq LLM call (network + inference)

**What's NOT included:**
- Tavily web search (separate, always adds 500–1000ms)
- HyDE expansion (adds one extra Groq call ~200ms)
- Python GC (`gc.collect()` after response)

**Prism results across versions:**

| Version | Stack | p50 | p95 | p99 |
| --- | --- | --- | --- | --- |
| v1.0.0 | baseline | 2029ms | — | — |
| v1.1.0 | HyDE | 4018ms | 6452ms | 6455ms |
| v1.2.0 | HyDE+MQ | 1812ms | 6136ms | 6744ms |
| v1.3.0 | HyDE+MQ+CTX | 2610ms | 6720ms | 6959ms |

HyDE adds one Groq call per query → p50 2×. Contextual chunks are longer → LLM processes more tokens → higher p50 vs baseline. Wide p95/p99 gap = occasional Groq congestion spikes, not retrieval.

**Production benchmarks for context:**

| System | p50 target | Notes |
| --- | --- | --- |
| Search engine | <200ms | Pre-indexed, no LLM |
| RAG (GPU cloud) | 500–800ms | GPU reranker + fast LLM |
| Prism (Render free CPU) | ~2000ms | CPU reranker + network LLM |
| Prism + streaming | ~300ms to first token | Same total time, perceived faster |

---

### How the Metrics Interact — Reading the Dashboard

```
High correctness + high recall + high P@5 = system is working ✅

High P@5 + low recall:
→ Retriever finds right chunks but misses others
→ Fix: Multi-Query Retrieval, HyDE

Low P@5 + high recall:
→ Retriever finds many chunks but most are noise
→ Fix: stricter reranking, smaller retrieve_k

High recall + low correctness:
→ Retriever gave good context, LLM failed to use it
→ Fix: better system prompt, larger LLM model

Low relevancy + high correctness:
→ LLM is verbose/padded but factually right
→ Fix: tighter prompt ("be concise, one paragraph maximum")

High latency p99 >> p50:
→ Occasional slow queries (complex retrieval or Groq congestion)
→ Monitor, add timeout handling
```

**Metric evolution across Prism versions:**

```
              v1.0.0   v1.1.0   v1.2.0   v1.3.0
              (base)   (HyDE)   (HyDE+MQ)(+CTX)
correctness:  0.82     0.75     0.77     0.78
relevancy:    0.62     0.845    0.890    0.799
recall:       0.51     0.721    0.645    0.768   ← primary target
P@5:          0.89     0.911    0.904    0.984
p50:          2029ms   4018ms   1812ms   2610ms

Verdict (v1.0.0): recall is the primary bottleneck.
Fix: HyDE (+21pp recall), Multi-Query (wider pool), Contextual Retrieval (+18% recall).
Best production stack: v1.3.0 — HyDE + Multi-Query + Contextual Retrieval.
```

---

## 17. Multi-Query Retrieval — Wider Net for Higher Recall

### The Problem: Single Phrasing Has Blind Spots

A query hits ChromaDB + BM25 with one set of tokens. Corpus chunks that express the same concept differently never surface.

```
Query: "What is the UPI merchant limit?"

Corpus has:
  Chunk A: "merchant UPI transaction cap is ₹5 lakh"      ← found ✓ (keyword match)
  Chunk B: "ceiling for PSP collections was revised..."    ← MISSED (different vocab)
  Chunk C: "payment service providers may not exceed..."   ← MISSED (no "merchant" token)
```

Chunk B and C are relevant. Single-phrasing retrieval misses them. This is why Prism v1.0.0 Violet had context_recall = 0.51 — only half the needed chunks were retrieved.

### The Fix: Generate 3 Phrasings, Pool Results

```
Step 1: LLM generates 3 paraphrases of the original query
  Original:    "What is the UPI merchant limit?"
  Phrasing 2:  "What is the PSP payment ceiling for UPI collections?"
  Phrasing 3:  "How much can merchants accept via UPI transactions?"

Step 2: Run retrieval for EACH phrasing
  Original     → [Chunk A, Chunk D, Chunk E, ...]  (retrieve_k per list)
  Phrasing 2   → [Chunk B, Chunk A, Chunk F, ...]  ← Chunk B appears now
  Phrasing 3   → [Chunk C, Chunk B, Chunk G, ...]  ← Chunk C appears now

Step 3: Pool + deduplicate by content key (keep best rank per chunk)
  Combined unique pool: [A, B, C, D, E, F, G, ...]

Step 4: RRF fuse the deduplicated pool → rerank top-5
  Reranker sees 30 candidates instead of 10 → picks best 5 from wider pool
```

### Why Deduplication Uses Best Rank

If Chunk A appears at rank 1 in phrasing 1 and rank 3 in phrasing 2, keep it at rank 1 — its highest-confidence rank. The merged ranked list is then sorted by best rank before RRF fusion.

```python
# retriever.py — deduplication loop
for rank, doc in enumerate(d_results):
    key = doc["content"][:120]
    if key not in dense_seen or rank < dense_seen[key][0]:
        dense_seen[key] = (rank, doc)  # keep best (lowest) rank

dense_pool = [doc for _, doc in sorted(dense_seen.values(), key=lambda x: x[0])]
```

### In Prism

```python
# retriever.py
def _multi_query_expand(self, query: str) -> list[str]:
    prompt = (
        "Generate 3 different phrasings of the following question for document retrieval. "
        "Each phrasing must use different vocabulary but seek the same information. "
        "Return ONLY the 3 questions, one per line, no numbering, no preamble.\n\n"
        f"Question: {query}"
    )
    # returns [original_query, phrasing_2, phrasing_3, phrasing_4]

def _get_relevant_documents(self, query, ...):
    queries = self._multi_query_expand(query) if self.use_multi_query else [query]
    
    dense_seen, sparse_seen = {}, {}
    for q in queries:
        # retrieve for each phrasing, keep best rank per unique chunk
        ...
    
    dense_pool = sorted_by_best_rank(dense_seen)
    sparse_pool = sorted_by_best_rank(sparse_seen)
    
    fused = self._rrf_fuse(dense_pool, sparse_pool)   # wider pool
    reranked = rerank(query, fused[:retrieve_k], top_k=rerank_k)  # reranker uses original query
```

**ON by default** (`config.yaml` `multi_query_enabled: true`) — adds one Groq call per query (~200ms).

**Measured result (v1.2.0, 25 samples, HyDE+MQ):** recall=0.645, P@5=0.904. Multi-Query alone provided smaller-than-expected recall lift. Root cause: query-side reformulation cannot fix decontextualized chunk embeddings — chunks with weak vectors rank low regardless of how the query is phrased. The larger recall gain came from Contextual Retrieval (Concept 21), which fixes chunk quality at ingest time.

**Cost:** 1 extra Groq call + 3× retrieval calls (fast, in-memory) + 3× BM25 (negligible).

**Reranker still uses original query** — not the phrasings. The phrasings widen the candidate pool; the reranker judges relevance against what the user actually asked.

---

## Summary — Key Concepts Cheatsheet

| Concept | What it solves | Where in Prism |
| --- | --- | --- |
| ParentDocumentRetriever | Retrieval precision + answer faithfulness | Ingestion + retrieval layer |
| InMemoryStore | Fast parent text lookup by ID | RAM store (lost on restart) |
| ChromaDB | Vector similarity search for child chunks | Persistent disk store |
| BM25Okapi | Exact keyword matching with TF saturation | Sparse retrieval (weight 0.3) |
| RRF | Merging dense + sparse rankings (scale-agnostic) | Post-retrieval fusion |
| Cross-Encoder | Accurate joint query-doc relevance scoring | Final reranking step |
| Chain condensation trap | Why LangChain strips web context silently | Web query bypass in chain.py |
| HyDE | Closes question-answer vector space gap; +21pp recall | Dense retrieval (hyde_enabled: true) |
| Faithfulness is circular | Why eval metrics designed alongside corpus lie | answer_correctness chosen as primary |
| P@5 + Recall diagnostic pair | Identifies whether retrieval pool is narrow or noisy | v1.0.0 Violet: P=0.89, R=0.51 |
| RecursiveCharacterTextSplitter | Splits at paragraph/line/word/char in priority order | ingest.py chunking |
| ConversationBufferWindowMemory | Sliding k-window of chat history + output_key trap | memory.py, used in chain |
| LLM at ingest time | Briefing pattern — run LLM once per doc, not per query | briefing.py on upload |
| Contextual Retrieval | Fix decontextualized chunks at ingest; +18% recall | `contextualize_chunks_async()` in ingest.py |
| Answer Correctness | LLM judge 1–5 vs ground truth, normalized → 0–1 | Primary metric, independent of retrieved chunks |
| Answer Relevancy | Reverse-question cosine similarity — penalizes verbose LLM | RAGAS, no ground truth needed |
| Context Recall | GT sentences attributed to retrieved chunks ÷ total | RAGAS, measures retrieval completeness |
| Precision@5 | Relevant chunks in top-5 ÷ 5, source + keyword match | Custom deterministic, no LLM call |
| Latency p50/p95/p99 | Percentile timing: median / worst-normal / absolute-worst | numpy.percentile over per-query ms measurements |
| Multi-Query Retrieval | Wider candidate pool; best rank dedup before RRF | retriever.py `_multi_query_expand()` (multi_query_enabled: true) |
| Semantic Chunking Tradeoff | Recall vs precision when eval is fixed-chunk-aligned | v1.4.0 ablation: +9.3pp recall, −27.3pp P@5, 5× latency |

---


---

## 20. Semantic Chunking Tradeoff — Recall vs Precision

### The Problem with Fixed-Size Splits

`RecursiveCharacterTextSplitter` at 500 chars cuts mid-sentence, mid-table, mid-list. Embedding a truncated sentence returns a weak vector — the chunk doesn't represent a complete idea. Semantic chunking fixes this by splitting at topic boundaries (where cosine similarity between adjacent sentences drops below a threshold), producing variable-size chunks that each represent one coherent thought.

### What the Ablation Showed (Prism v1.4.0, 25 samples)

| Stack | recall | P@5 | latency p50 |
|-------|--------|-----|-------------|
| v1.3.0 HyDE+MQ+CTX (fixed 500-char) | 0.768 | **0.984** | 2610ms |
| v1.4.0 + Semantic chunking | **0.861** | 0.711 | 12952ms |
| Delta | +9.3pp | **−27.3pp** | 5× slower |

### Why Recall Went Up

Semantic chunks contain complete sentences and complete ideas. The embedding captures the full meaning — better vector → higher chance of matching the relevant query → more reference content covered → higher context recall.

### Why Precision Collapsed

P@5 measures overlap between retrieved chunks and the ground-truth `relevant_chunks` field in `eval_pairs.json`. Those reference chunks were defined against fixed 500-char splits. Semantic chunks have different, larger boundaries — they contain the relevant content but as part of a bigger unit, so exact overlap with the reference drops. The reranker also receives a wider, noisier candidate pool (variable-size chunks = less uniform scoring surface).

**Key insight:** P@5 is eval-alignment-sensitive. If ground truth was labelled against fixed chunks, semantic chunks will always score lower on P@5 even when they retrieve better content. For a production system with human-labelled relevance judgements (not self-aligned eval pairs), the gap would be smaller.

### Why Latency Was 5×

Semantic chunks are longer on average than 500-char fixed chunks. Longer chunks = more tokens fed to the LLM per answer generation call. Retrieval time is unchanged (semantic chunking is an ingest-time decision), but inference time scales with context length.

### Decision

Semantic chunking rejected for Prism production. v1.3.0 (HyDE + Multi-Query + Contextual) confirmed as best stack: P@5=0.984, recall=0.768, p50=2610ms.

Semantic chunking would make more sense when:
- Ground truth eval labels are created after chunking (aligned to the actual chunk boundaries)
- LLM context window is not a bottleneck
- Recall is the primary metric (e.g. legal/compliance: never miss a relevant clause)

---

## 21. Contextual Retrieval — Fixing Decontextualized Chunks at Ingest

### The Problem: Fixed-Size Chunks Lose Context

`RecursiveCharacterTextSplitter` at 500 chars cuts documents into fragments. Many fragments are decontextualized — they lack the surrounding information that gives them meaning:

```
Chunk from NPCI merchant guidelines PDF:
"The limit was revised to ₹2 lakh."

Problems:
- "The limit" — which limit? Not in this chunk.
- "revised" — from what? Not in this chunk.
- "₹2 lakh" — for what transaction type? Not in this chunk.

Embedding of this chunk → weak, generic vector.
Query "merchant UPI transaction cap" → this chunk may not surface.
```

No query-side technique (HyDE, Multi-Query) can fix this — the chunk embedding is weak regardless of how the query is phrased.

### The Fix: Anthropic's Contextual Retrieval

At ingest time, before embedding, ask the LLM to prepend 2 sentences situating each chunk in its document:

```
Prompt:
  "Given this document: [full doc or representative sample]
   Write 2 sentences situating this chunk in context.
   
   Chunk: 'The limit was revised to ₹2 lakh.'"

LLM output:
  "In the NPCI UPI merchant guidelines (2024), Section 4.3 covers
   transaction ceiling revisions for PSPs. The limit was revised to ₹2 lakh."
```

The contextual prefix is **prepended to the chunk** before embedding:

```python
# ingest.py
chunk.page_content = f"{context_prefix}\n\n{chunk.page_content}"
# then embed this augmented text
```

The chunk stored for retrieval is now specific and rich. Same chunk now surfaces for "merchant UPI transaction cap" queries.

### Why This Works

| | Original chunk | Contextual chunk |
|---|---|---|
| Text | "The limit was revised to ₹2 lakh." | "NPCI UPI merchant guidelines 2024, PSP ceiling. The limit was revised to ₹2 lakh." |
| Embedding | Generic "revision" vector | Specific "merchant UPI PSP ceiling" vector |
| Retrieval | Misses merchant-related queries | Surfaces correctly |

The embedding now represents a complete, specific idea instead of a floating fragment.

### Design Pattern: Ingest-Time vs Query-Time

| | Ingest-time LLM (Contextual Retrieval) | Query-time LLM (HyDE, Multi-Query) |
|---|---|---|
| Runs | Once per chunk, at upload | Every query |
| Cost | Paid once; benefit on every future query | Paid per query |
| What it fixes | Bad chunk embeddings (quality problem) | Query-corpus vocabulary gap (coverage problem) |
| Latency impact | Upload slower (~40s for 30 chunks) | Query slower (+200ms per technique) |

**Key insight:** Query-side techniques improve how well a query matches existing chunk vectors. Contextual retrieval improves the chunk vectors themselves. Both are needed for maximum recall.

### Measured Results — Prism v1.3.0 (25 samples)

| Metric | v1.0.0 baseline | v1.3.0 (HyDE+MQ+CTX) | Delta |
|--------|-----------------|----------------------|-------|
| context_recall | 0.51 | 0.768 | **+25.8pp** |
| precision_at_5 | 0.89 | 0.984 | **+9.4pp** |
| answer_relevancy | 0.62 | 0.799 | +17.9pp |
| latency p50 | 2029ms | 2610ms | +28% |

Contextual retrieval contributes ~+18% recall (v1.3.0 vs v1.2.0 without CTX: 0.768 vs 0.645).

### Production Complication: 40s Upload

30 chunks × 1 Groq call × ~1.3s/call (sequential) = ~40s blocking. Solution: parallel calls with `asyncio.Semaphore(3)` — 3 concurrent Groq calls at ~3000 TPM burst (under Groq's 6000 TPM limit). Reduces to ~15s. Two-phase upload: sync non-contextual embed first (user can query in <3s), contextual replacement in background.

### Distinction from Briefing (Concept 14)

| | Briefing | Contextual Retrieval |
|---|---|---|
| What | Document-level 5-bullet summary + 3 questions | Chunk-level 2-sentence situating context |
| Shown to user | Yes (in upload response) | No (prepended to chunk text, invisible) |
| Purpose | User orientation | Embedding quality improvement |
| LLM calls | 1 per document | 1 per chunk (~30 per doc) |