docs/learning.md

Prism — Learning Concepts

Concept Index by Product Workflow

Every concept in this file maps to a specific stage in Prism's request lifecycle. Use this to understand when each idea becomes relevant, not just what it is.

┌─────────────────────────────────────────────────────────────────┐
│  USER UPLOADS A DOCUMENT                                        │
│                                                                 │
│  PDF/TXT/CSV → chunk → embed → store → briefing                │
│                                                                 │
│  Concept 11  RecursiveCharacterTextSplitter                     │
│              How text is split into chunks (paragraph → line    │
│              → word → char priority). Overlap bridges splits.   │
│                                                                 │
│  Concept 1   Parent-Child Chunking                              │
│              Design pattern: small child chunks for retrieval,  │
│              large parent chunks sent to LLM.                   │
│                                                                 │
│  Concept 2   InMemoryStore                                      │
│              Where parent chunks live (RAM only, dies on        │
│              restart). ChromaDB stores child vectors on disk.   │
│                                                                 │
│  Concept 14  LLM at Ingest Time (Briefing)                      │
│              After ingest, LLM auto-generates 5-bullet summary  │
│              + 3 suggested questions. Runs once, not per query. │
│                                                                 │
│  Concept 21  Contextual Retrieval                               │
│              Prepend 2-sentence situating context to each chunk │
│              before embedding. Fixes decontextualized chunks.   │
│              Measured: +18% recall (0.51→0.60), +6.6pp P@5.    │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│  USER SENDS A QUESTION                                          │
│                                                                 │
│  question → (optionally) multi-query expand → (optionally)     │
│  HyDE expand → retrieve → fuse → rerank → top-5 chunks         │
│                                                                 │
│  Concept 17  Multi-Query Retrieval                              │
│              LLM generates 3 phrasings of query → retrieve for │
│              each → pool + deduplicate → RRF → rerank.          │
│              Widens candidate pool. ON by default.              │
│                                                                 │
│  Concept 7   HyDE                                               │
│              LLM generates fake answer → embed fake answer      │
│              instead of query → closes question/answer vector   │
│              space gap → higher context recall.                 │
│              Measured: +21pp recall (0.51→0.72). ON by default. │
│                                                                 │
│  Concept 4   BM25Okapi                                          │
│              Sparse keyword retrieval. Catches exact terms       │
│              (section numbers, ₹ amounts) that dense misses.    │
│              TF saturation prevents high-frequency term bias.   │
│                                                                 │
│  Concept 3   RRF + Cross-Encoder Pipeline                       │
│              Dense (0.7) + BM25 (0.3) merged via weighted RRF. │
│              Cross-encoder reranks top-10 jointly → top-5.      │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│  WEB SEARCH (always on)                                         │
│                                                                 │
│  question → condense with history → Tavily → web results        │
│                                                                 │
│  Concept 5   ConversationalRetrievalChain Condensation Trap     │
│              Chain's internal condense step strips prepended    │
│              web context. Fix: bypass chain entirely for web    │
│              queries. Direct LLM call preserves all context.    │
│                                                                 │
│  Concept 12  ConversationBufferWindowMemory                     │
│              Sliding k=10 window of chat history injected into  │
│              condense_question() before Tavily search, so       │
│              follow-up queries have full context.               │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│  LLM GENERATES ANSWER                                           │
│                                                                 │
│  [doc chunks] + [web results] + [chat history] → LLM → answer  │
│                                                                 │
│  Concept 12  ConversationBufferWindowMemory (output_key trap)   │
│              Memory saves this turn for next question.          │
│              output_key="answer" required — chain returns       │
│              multiple keys, memory needs to know which to save. │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│  EVALUATION (offline, run via scripts/run_eval_versioned.py)    │
│                                                                 │
│  50 eval pairs → retrieve → answer → score → versioned JSON    │
│                                                                 │
│  Concept 8   Eval Metric Design (Faithfulness Is Circular)      │
│              Why faithfulness 1.0 was meaningless. How          │
│              answer_correctness (vs ground truth) is honest.    │
│                                                                 │
│  Concept 9   Precision@K vs Recall Diagnostic                   │
│              P@5=0.89 + recall=0.51 = pool too narrow.          │
│              The combination tells you exactly what to fix.     │
│                                                                 │
│  Concept 15  All 7 Eval Metrics — Full Reference                │
│              Answer Correctness, Answer Relevancy, Context      │
│              Recall, Precision@5, Latency p50/p95/p99.          │
│              Includes computation steps, failure modes,         │
│              Prism v1.0.0 results, and metric interaction map.  │
│                                                                 │
│  Concept 20  Semantic Chunking Tradeoff                         │
│              Topic-boundary splits improve recall but hurt      │
│              precision when eval pairs are aligned to fixed     │
│              chunk boundaries. Measured: +9.3pp recall,         │
│              -27.3pp P@5, 5× latency (v1.4.0 ablation).        │
└─────────────────────────────────────────────────────────────────┘

---

1. Parent-Child Chunking — The Retrieval-Faithfulness Tradeoff

The Problem with One Chunk Size

Chunk Size	Retrieval	LLM Answer Quality
Small (200 chars)	Precise — matches exact phrase	Bad — too little context, answer is fragmented
Large (800 chars)	Imprecise — embedding averages over too much text	Good — LLM sees full context

Key insight: Embeddings of large chunks get "diluted" — the vector represents the average meaning of 800 chars. Small chunks have sharper, more focused vectors that match queries better.

How ParentDocumentRetriever Works

INGEST:

• 800-char parent → stored in InMemoryStore (keyed by ID)
• 200-char children → embedded → stored in ChromaDB

QUERY:

1. Query hits ChromaDB → finds best matching 200-char child chunk
2. Look up parent_id from child metadata
3. Return the full 800-char parent to LLM

Why This Works

• Child chunk = precise retrieval target (dense vector = focused meaning)
• Parent chunk = rich answer context (LLM gets surrounding sentences)

Concrete Example — RBI Circular PDF

Raw text (one paragraph):

"The Reserve Bank of India has mandated that all UPI transactions above ₹2,000 must undergo additional authentication from January 2025. This includes biometric verification or OTP-based second factor. Non-compliant PSPs will face penalties up to ₹10 lakh per violation."

After chunking:

• Parent (800 chars) → InMemoryStore: Full paragraph above
• Child A (200 chars) → ChromaDB: "The Reserve Bank of India has mandated that all UPI transactions above ₹2,000 must undergo additional authentication from January 2025."
• Child B: "This includes biometric verification or OTP-based second factor."
• Child C: "Non-compliant PSPs will face penalties up to ₹10 lakh per violation."

Query: "What is the UPI transaction authentication limit?"

→ ChromaDB finds Child A (score: 0.91) — focused vector on UPI + ₹2000 + authentication

→ Fetches parent_id → returns full parent paragraph to LLM

Comparison Table:

Approach	LLM Gets	Problem
Dense-only, large chunks	Full paragraph (good)	Match was imprecise — wrong paragraph might score higher
Dense-only, small chunks	Child A only (38 words)	Misses penalty info — incomplete answer
ParentDocumentRetriever	Full paragraph (precise match + rich context)	✅ Best of both

LLM final answer: "UPI transactions above ₹2,000 require additional authentication (biometric or OTP) from Jan 2025. Non-compliant PSPs face up to ₹10 lakh penalty."

---

2. InMemoryStore — What It Actually Is

Key point: InMemoryStore is NOT part of ChromaDB. It's a separate, RAM-only store.

Under the hood it's essentially a plain Python dict wrapped in a LangChain class:

store = {}
store["parent_id_abc123"] = "Full 800-char parent chunk text..."
store["parent_id_def456"] = "Another parent chunk..."

Two Separate Stores in Prism

ChromaDB (on disk)	InMemoryStore (RAM only)
Child chunk vectors	Parent chunk text
[vector, metadata] → find similar	parent_id → full text
✅ Survives restart	❌ Dies on restart

Flow

Child A metadata = { "parent_id": "abc123", "text": "short child..." }
                            ↓
             InMemoryStore["abc123"]
                            ↓
             "Full 800-char parent text" → LLM

⚠️ Known Limitation in Prism

Render free tier cold-starts → InMemoryStore is wiped → must re-ingest PDFs on every cold start.

ChromaDB persists to disk so child vectors survive, but parent text is gone.

⚠️ Architecture Note — Intended vs Current Implementation

architecture.md and Concept 1 describe ParentDocumentRetriever (child 200 / parent 800). The current ingest.py uses a single-pass RecursiveCharacterTextSplitter at 500 chars — no separate parent store. The ParentDocumentRetriever was the original design and is documented as such. Both chunking approaches teach the same tradeoff; the parent-child concept remains valid as a pattern even if the current code simplified to single-pass chunking.

---

3. RRF + Cross-Encoder Reranker Pipeline

3a. RRF — Reciprocal Rank Fusion

Problem: Dense retrieval returns a ranked list. BM25 returns a ranked list. Scores are on different scales (BM25: 0–15, cosine: 0–1) — can't add them directly.

RRF Solution: Ignore raw scores. Use only rank position.

Standard RRF formula:
score(doc) = Σ  1 / (k + rank_in_list)
             k = 60  (constant, dampens top-rank advantage)

Prism's weighted RRF (applied inside the formula per list):
score(doc) += dense_weight  / (k + rank_in_dense_list)   # 0.7 × contribution
score(doc) += sparse_weight / (k + rank_in_sparse_list)  # 0.3 × contribution

Example:

Doc	Dense (ChromaDB)	BM25	RRF Score
Doc A	Rank 1 (0.91)	Rank 2 (9.1)	1/61 + 1/62 = 0.0325 ✅
Doc C	Rank 3 (0.71)	Rank 1 (12.3)	1/63 + 1/61 = 0.0320
Doc B	Rank 2 (0.87)	—	1/62 = 0.0161
Doc D	—	Rank 3 (7.4)	1/63 = 0.0159

Doc A wins — appeared high in BOTH lists → signals true relevance.

In Prism: dense_weight=0.7, sparse_weight=0.3 applied inside the RRF formula — each list's contribution is multiplied by its weight before summing. Not "before RRF" as a pre-filter, but as a per-list scaling factor within fusion (see Concept 4 for the actual code).

3b. Cross-Encoder Reranker

Problem: RRF gives top-10 candidates. Bi-encoder (ChromaDB) encodes query and doc separately → approximate similarity.

Cross-Encoder: Feeds query + doc together into BERT → full attention across both → much more accurate relevance score.

	Bi-Encoder (ChromaDB)	Cross-Encoder (TinyBERT)
Method	embed(query) + embed(doc) → cosine(q,d)	BERT([query][SEP][doc]) → single score 0–1
Speed	Fast	Slow
Accuracy	Approximate	Accurate
Encoding	Independent	Joint

Example — after RRF top-10:

Pair	Cross-Encoder Score
(query, Doc A)	0.94 ✅
(query, Doc B)	0.88 ✅
(query, Doc C)	0.61
(query, Doc D)	0.23

Top-5 by cross-encoder score → LLM

Full Retrieval Pipeline

Query
  |
  ├→ ChromaDB dense  → top-10 ranked docs
  ├→ BM25 sparse     → top-10 ranked docs
  |
  ↓
RRF fusion → merged top-10 (rank-based, scale-agnostic)
  |
  ↓
Cross-encoder → re-scores all 10 jointly with query
  |
  ↓
Top-5 parent chunks → LLM

---

---

4. BM25Okapi — Why Keywords Beat Embeddings for Exact Terms

The Problem with Dense Retrieval on Regulatory Text

Embeddings capture meaning. But regulatory text has exact identifiers — section numbers, policy codes, rupee amounts — where the exact token matters, not the meaning.

Example:

Query: "What is the penalty for UPI non-compliance?"

A dense embedding model reads this as: "something about UPI and consequences".

Two chunks in corpus:

• Chunk A: "Non-compliant PSPs will face penalties up to ₹10 lakh per violation." ← exact answer
• Chunk B: "UPI has transformed digital payments in India with over 100 billion transactions." ← semantically close (UPI topic) but wrong

Dense retrieval might rank Chunk B high because it's heavily UPI-themed. BM25 ranks Chunk A high because "penalt" and "non-compli" are rare, high-signal tokens.

How BM25Okapi Works

Plain TF-IDF problem: a doc that says "UPI" 50 times gets 50× the score. That's unfair — one mention of "₹10 lakh" in the right context should beat 50 mentions of "UPI" in a generic overview.

BM25 fixes this with term frequency saturation:

BM25 score = IDF(term) × [ tf × (k1 + 1) ] / [ tf + k1 × (1 - b + b × dl/avgdl) ]

Where:
  tf    = how many times term appears in this chunk
  IDF   = how rare the term is across all chunks (log scale)
  k1    = saturation constant (~1.5) — controls how fast TF saturates
  b     = length normalization (~0.75)
  dl    = this chunk's length
  avgdl = average chunk length in corpus

Saturation in plain English:

tf (term count in chunk)	TF-IDF score	BM25 score (k1=1.5)
1	1.0	1.0
5	5.0	1.56 ← plateaus
20	20.0	1.79 ← barely grows
50	50.0	1.88 ← effectively capped

A chunk mentioning "UPI" 50 times scores almost the same as one mentioning it 5 times. But "₹10 lakh" appearing once in a chunk that has it = high IDF (rare token) × full TF benefit.

In Prism

# bm25_index.py
from rank_bm25 import BM25Okapi

# .lower() matters — "UPI" and "upi" are different tokens without it
corpus = [doc["content"].lower().split() for doc in all_chunks]
bm25 = BM25Okapi(corpus)

# At query time — query also lowercased to match corpus tokenization:
scores = bm25.get_scores(query.lower().split())
top_k_idx = sorted(range(len(scores)), key=lambda i: scores[i], reverse=True)[:k]

BM25 operates on raw tokens (word split). No embeddings. No GPU. Rebuilds in ~1s at startup from corpus.

Weight in Prism: sparse_weight=0.3 in RRF — BM25 is complementary, not dominant. Dense handles semantic meaning; BM25 catches exact matches dense misses.

---

5. ConversationalRetrievalChain — The Silent Context Killer

What the Chain Does Internally

ConversationalRetrievalChain has two internal steps most people don't know about:

User question + chat history
        ↓
[STEP 1] Condense question
        LLM rewrites "Is the price level good?" 
        → "Is Bajaj Finance stock ₹908-924 a good buy in 2026?"
        (standalone question for retrieval)
        ↓
[STEP 2] Retrieve + Answer
        Standalone question → retriever → top-5 chunks → LLM answer

Step 1 exists so follow-up questions work without full history context. Good design for RAG.

The Bug: Web Context Gets Stripped

When Prism added Tavily web search, the naive approach was:

web_results = tavily.search(question)
augmented_question = f"Web context: {web_results}\n\nQuestion: {question}"
chain.invoke({"question": augmented_question})  # ← WRONG

What actually happens:

augmented_question (with Tavily content prepended)
        ↓
[STEP 1] Chain's condense LLM:
        "Rewrite this as a standalone retrieval query."
        Output: "What is Bajaj Finance stock price?"
        ← ALL TAVILY CONTENT STRIPPED. LLM never sees it.
        ↓
[STEP 2] Answer LLM gets: corpus chunks only. No web context.

The chain's condensation step rewrites the question for retrieval quality — and throws away everything else.

The Fix in Prism

Bypass the chain entirely for web queries. Direct LLM call:

# chain.py
def run_query_with_web(question, rag_docs, web_sources, memory):
    history = memory.load_memory_variables({})["history"]
    
    prompt = f"""
    Chat history: {history}
    
    Web search results:
    {web_sources}
    
    Document context:
    {rag_docs}
    
    Question: {question}
    Answer:"""
    
    answer = llm.invoke(prompt)
    memory.save_context({"input": question}, {"answer": answer})
    return answer

No condensation step. Web context reaches the LLM guaranteed.

Lesson: LangChain abstractions are powerful but opaque. When something doesn't work, read what the chain actually does internally — don't assume the abstraction is transparent.

---

7. HyDE — Hypothetical Document Embeddings

The Problem with Embedding a Question

When you search ChromaDB, you embed the query and find similar vectors. But your corpus contains answers, not questions. They live in different regions of vector space.

Concrete example:

Query: "What is the UPI transaction limit for P2P transfers?"

This embeds as a question vector — the model has seen millions of questions, it knows this pattern.

The answer in your corpus: "P2P UPI transfers are capped at ₹1 lakh per transaction per day as per NPCI guidelines."

This embeds as an answer vector — declarative sentence, factual tone, different region in 1536-dimensional space.

They're semantically related, but the vector distance is larger than it should be.

HyDE's Solution

Before searching, ask the LLM to hallucinate an answer:

Step 1: LLM generates a fake answer to the query
  Query: "What is the UPI transaction limit for P2P transfers?"
  Fake answer: "The UPI transaction limit for P2P transfers is typically 
                set by NPCI and varies by bank, generally around ₹1 lakh 
                per day for most PSPs."

Step 2: Embed the FAKE ANSWER (not the query)
  vector = embed("The UPI transaction limit for P2P transfers is...")

Step 3: Search ChromaDB with this vector
  → Finds real answer chunks that are semantically close to a fake answer
  → Much closer in vector space than the original question was

Why "Hypothetical" Works

The fake answer and the real corpus chunk are both declarative, factual, answer-shaped text. They live in the same region of vector space. The query (a question) lives elsewhere.

Vector space (simplified):

[Question zone]          [Answer zone]
"What is UPI limit?" --- ... --- "P2P capped at ₹1 lakh..."
       ↑                                    ↑
  far from corpus                    close to corpus chunk
  
HyDE:
"UPI limit is ~₹1 lakh..." ← fake answer → close to real chunk ✅

In Prism

# retriever.py
def _hyde_expand(self, query: str) -> str:
    prompt = f"Write a 2-sentence factual answer to: {query}"
    return self.llm.invoke(prompt).content

def _get_relevant_documents(self, query: str):
    dense_query = self._hyde_expand(query) if self.use_hyde else query
    
    # Dense search uses fake answer embedding
    dense_docs = self.vectorstore.similarity_search(dense_query, k=10)
    
    # BM25 still uses original query (keyword matching needs real terms)
    sparse_docs = self.bm25_retrieve(query, k=10)
    
    return self.rrf_and_rerank(dense_docs, sparse_docs)

ON by default (config.yaml hyde_enabled: true) — adds one Groq call per query (~200ms). Latency cost accepted.

Measured lift (v1.1.0, 18 samples): context_recall 0.51 → 0.72 (+21pp). Latency 2029ms → 4018ms p50 (2×). HyDE specifically helps recall — it finds chunks that keyword/direct-embed matching misses.

---

8. Eval Metric Design — Why Faithfulness Is Circular

Four RAGAS Metrics and What They Actually Measure

Metric	Judge compares	Question it answers
faithfulness	LLM answer vs retrieved chunks	"Did the LLM stick to what was in the retrieved docs?"
answer_correctness	LLM answer vs ground_truth reference	"Is the answer actually right?"
answer_relevancy	LLM answer vs original question	"Did the answer address what was asked?"
context_recall	Retrieved chunks vs ground_truth	"Did retrieval find the chunks needed to answer?"

The Faithfulness Trap

In Prism v1 eval, faithfulness scored 1.0. Seemed great. Was meaningless.

Here's why:

Eval pair designed alongside corpus:
  Question: "What is the UPI P2P transaction limit?"
  Ground truth: "₹1 lakh per day"
  
  Corpus chunk (also written by us): 
  "P2P UPI transfers are capped at ₹1 lakh per day per NPCI guidelines."

Flow:
  1. LLM retrieves that chunk (of course — it's perfectly matched)
  2. LLM answers: "The UPI P2P limit is ₹1 lakh per day."
  3. RAGAS faithfulness judge: "Does answer match retrieved chunk?" → YES → score: 1.0

The judge is comparing the answer to the chunk that was designed to produce that answer. Circular. Score tells you the retrieval worked, not whether the answer is correct.

answer_correctness Is the Honest Metric

Flow:
  1. LLM answers: "The UPI P2P limit is ₹1 lakh per day."
  2. Ground truth (human-written reference): "₹1 lakh per transaction per day"
  3. RAGAS judge: "Does answer match ground truth?" → mostly yes → score: 0.82

Judge compares to an independent human reference. No circular dependency on retrieved chunks. Harder to game.

Prism v1.0.0 Violet results:

Metric	Score	Interpretation
answer_correctness	0.82	82% of answers match ground truth — honest signal
answer_relevancy	0.62	RAGAS penalizes verbosity — Groq 70B tends to over-explain
context_recall	0.51	Only 51% of needed chunks retrieved — target for Multi-Query
P@5	0.89	When chunks are retrieved, 89% are correct — precision is fine

The P@5 + Recall Diagnostic

P@5=0.89 + recall=0.51 tells you something specific:

Retrieval pool is too narrow.

"When we retrieve something, it's usually right (0.89)."
"But we're missing ~half the relevant chunks (0.51)."

Root cause: single-phrasing query misses chunks phrased differently.
Fix: Multi-Query Retrieval — generate 3 phrasings, retrieve for each, pool candidates.

This is exactly how production ML teams diagnose retrieval systems. The combination of metrics points to the specific fix.

---

9. Precision@K vs Recall — The Retrieval Diagnostic Pair

Definitions in Plain English

Imagine your corpus has 5 chunks that are genuinely relevant to a query. Your retriever returns 5 chunks (K=5).

Ground truth relevant chunks: [A, B, C, D, E]
Retrieved chunks:              [A, B, X, Y, Z]

Precision@5 = correct retrieved / K = 2/5 = 0.40
              "Of what I returned, how much was right?"

Recall@5    = correct retrieved / total relevant = 2/5 = 0.40
              "Of all the right chunks, how many did I find?"

The Four Diagnostic Combinations

P@5	Recall	Diagnosis	Fix
High	High	✅ Retrieval working well	Ship it
High	Low	Pool too narrow — finding right chunks but missing others	Multi-Query Retrieval, HyDE
Low	High	Too much noise — finding relevant chunks but also junk	Better reranking, stricter K
Low	Low	Retrieval fundamentally broken	Check embeddings, chunking strategy

Prism v1.0.0 Violet: High P, Low Recall

P@5 = 0.89 → "When Prism retrieves a chunk, 89% of the time it's relevant."
Recall = 0.51 → "But Prism only finds 51% of the relevant chunks total."

Example query: "What are the RBI guidelines on UPI merchant limits?"

Relevant chunks in corpus: [merchant_limit_2024, merchant_kyc, limit_circular_2023, payment_cap, psp_obligations]

Prism retrieved: [merchant_limit_2024, merchant_kyc, some_unrelated_chunk, another_unrelated, payment_cap]

Precision@5 = 3/5 = 0.60 (found 3 right ones, 2 noise)
Recall = 3/5 = 0.60 (missed limit_circular_2023 and psp_obligations)

Why did it miss them? limit_circular_2023 might use different phrasing: "The ceiling for merchant UPI collections was revised..." — no overlap with "merchant limits". Single-phrasing retrieval misses it.

Multi-Query generates: "UPI merchant payment ceiling", "RBI merchant collection limit", "PSP merchant UPI cap" → retrieves from all three → pools candidates → recall rises.

---

---

11. RecursiveCharacterTextSplitter — How Text Gets Chunked

The Problem with Fixed Splits

Splitting every 500 characters naively:

"...comply with Section 7(b) of the Act. |SPLIT| Payment systems must maintain..."

Cut mid-sentence. "Section 7(b)" separated from "Payment systems must maintain" — the context that makes "Section 7(b)" meaningful is now in a different chunk. The embedding for that chunk is weak.

How RecursiveCharacterTextSplitter Works

It tries a priority list of separators, from most preferred to least:

separators = ["\n\n", "\n", " ", ""]
# Priority: paragraph break > line break > word break > character break

For a given chunk_size=500:

1. Try \n\n first — split at paragraph boundaries. If resulting piece ≤ 500 chars: done.
2. If piece still >500 — try \n (line breaks within the paragraph).
3. If still >500 — try spaces (word boundaries).
4. Last resort — split at exact character position.

Result: Chunks break at natural language boundaries when possible, hard character cuts only when forced.

Overlap — Why Adjacent Chunks Share Text

splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,
    chunk_overlap=50,  # last 50 chars of chunk N = first 50 chars of chunk N+1
)

Without overlap:

Chunk 1: "...UPI transaction limits are set by NPCI guidelines for"
Chunk 2: "each payment service provider based on risk assessment."

Query "UPI limit by PSP" — the full sentence is split. Neither chunk alone has a strong embedding.

With overlap=50:

Chunk 1: "...UPI transaction limits are set by NPCI guidelines for each"
Chunk 2: "for each payment service provider based on risk assessment."

The bridging phrase appears in both chunks. One of them will have a strong vector for the full concept.

In Prism

# ingest.py
splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,   # from config.yaml
    chunk_overlap=50,
)
chunks = splitter.split_documents(documents)

Note: Prism originally used ParentDocumentRetriever (child 200 / parent 800). The current code uses a single-pass splitter at 500 chars. The architecture.md documents the intended design; the code reflects the current implementation. Both teach the same concept — chunk size is a precision-vs-context tradeoff.

---

12. ConversationBufferWindowMemory — Sliding Window Chat History

Why You Need Memory in a Chat System

Each POST /api/chat is a stateless HTTP request. The LLM has no memory of what was said 10 seconds ago. Without memory:

User: "Tell me about UPI transaction limits."
Prism: "UPI P2P limit is ₹1 lakh per day..."

User: "What about merchant payments?"   ← no context
LLM sees: question = "What about merchant payments?"  ← what is "about"? who knows?
Prism: "Merchant payments are a type of..."  ← wrong, generic answer

How ConversationBufferWindowMemory Works

Keeps last k conversation turns (human + AI) as a rolling window:

# memory.py
memory = ConversationBufferWindowMemory(
    memory_key="chat_history",
    return_messages=True,
    output_key="answer",   # ← critical — explained below
    k=10,                  # keep last 10 turns
)

Turn 1: [Human: "Tell me about UPI limits", AI: "UPI P2P is ₹1 lakh..."] Turn 2: [Human: "Tell me about UPI limits", AI: "...", Human: "What about merchant payments?", AI: "Merchant UPI limit is ₹5 lakh..."]

At turn 11: Turn 1 is evicted. Window always has last 10.

The chain sees:

Chat history:
  Human: Tell me about UPI limits
  AI: UPI P2P is ₹1 lakh per day...

Current question: What about merchant payments?

→ LLM understands "What about" refers to UPI limits. Answers correctly.

The output_key Trap

ConversationalRetrievalChain returns a dict with multiple keys:

result = chain.invoke({"question": "..."})
# result = {
#   "answer": "UPI P2P limit is ₹1 lakh...",
#   "source_documents": [...],
#   "question": "..."
# }

Memory's save_context call needs to know which key is the "output" to save:

# WRONG — KeyError because chain returns multiple output keys
memory = ConversationBufferWindowMemory(memory_key="chat_history")

# RIGHT — tell memory exactly which key to save
memory = ConversationBufferWindowMemory(
    memory_key="chat_history",
    output_key="answer",   # save result["answer"], not the whole dict
)

Without output_key="answer": LangChain tries to infer the output key, sees multiple candidates, raises ValueError: Multiple keys returned. This was a real bug hit during Prism development — subtle because the chain runs fine; the crash happens on the save_context call after.

---

14. LLM at Ingest Time — The Briefing Pattern

Two Places LLMs Can Run in a RAG System

Most tutorials show LLM running only at query time: user asks → retrieve → LLM answers.

Prism also runs LLM at ingest time: document uploaded → LLM summarizes → shown to user immediately.

INGEST TIME (once, when doc uploaded):
  PDF → chunks → embed → store
                ↓ (also)
            LLM reads first 6 chunks
            → generates 5-bullet summary
            → generates 3 suggested questions
            → returned in upload response

QUERY TIME (every user message):
  question → retrieve → LLM answers

Why This Is Useful

When a user uploads an unfamiliar document, they don't know what to ask. The briefing gives them:

1. Orientation — "what is this document about?"
2. Starting questions — clickable prompts that immediately work

# briefing.py
def generate_briefing(doc_name: str, text_sample: str) -> dict:
    prompt = f"""
    Analyze this document excerpt and respond with valid JSON only.
    Document: {doc_name}
    Content: {text_sample[:3000]}
    
    Return exactly: {{"summary": ["5 bullet strings"], "suggested_questions": ["3 question strings"]}}
    """
    response = llm.invoke([HumanMessage(content=prompt)])
    # Parse JSON from response → return to frontend

# routes/upload.py — after ingestion completes:
briefing = generate_briefing(doc_name, sample_text)
return {
    "uploaded": [filename],
    "documents": docs,
    "briefing": briefing,   # ← frontend shows this immediately
}

Design Decisions

Non-critical path — briefing failure doesn't block upload:

try:
    briefing = generate_briefing(...)
except Exception as e:
    logger.warning("Briefing skipped: %s", e)
    briefing = None  # upload still succeeds

JSON extraction from LLM output — LLMs often wrap JSON in markdown fences ( ```json). Prism strips these with regex before json.loads():

raw = re.sub(r"^```(?:json)?\s*", "", raw)  # strip opening fence
raw = re.sub(r"\s*```$", "", raw)           # strip closing fence
match = re.search(r"\{.*\}", raw, re.DOTALL)  # extract JSON object
data = json.loads(match.group())

When to use LLM at ingest time vs query time:

	Ingest time	Query time
Runs	Once per document	Once per user message
Cost	Fixed (paid once)	Scales with usage
Best for	Document-level metadata, summaries, question suggestions	Answering specific user questions
Examples	Briefing, contextual retrieval, chunk tagging	RAG answer, faithfulness eval, HyDE

The most powerful use of ingest-time LLM is contextual retrieval (roadmap): for every chunk, ask LLM "given this full document, write 2 sentences situating this chunk" — then prepend that context before embedding. Anthropic reports ~49% reduction in retrieval failures. Same LLM-at-ingest-time pattern, much higher impact.

---

---

15. Eval Dashboard Metrics — Full Reference

Prism's eval dashboard shows 7 numbers per run. Each answers a different question about the system. Understanding what each metric measures, how it's computed, and what can fool it is the difference between blindly running evals and actually improving the system.

---

Overview — What Each Metric Diagnoses

Query → [Retrieval] → top-5 chunks → [LLM] → answer
           ↑                ↑                 ↑
      Precision@5     Context Recall     Answer Correctness
                                         Answer Relevancy
                                         Latency p50/p95/p99

Metric	What it measures	Requires ground truth?	LLM call?
Answer Correctness	Is the answer factually right?	Yes	Yes (judge)
Answer Relevancy	Does answer address the question?	No	Yes + embed
Context Recall	Did retrieval find all needed chunks?	Yes	Yes (RAGAS)
Precision@5	Are the top-5 chunks relevant?	Yes (keywords/sources)	No
Latency p50/p95/p99	How fast is the system?	No	No

---

Metric 1 — Answer Correctness (primary)

Question it answers: "Is Prism's answer actually correct?"

Why it's primary: The only metric that directly measures answer quality vs a human-authored reference. Everything else is a proxy.

How it's computed:

CORRECTNESS_PROMPT = """
You are an evaluation judge. Score the generated answer against the reference answer.

Use a 1–5 integer scale:
5 — All key facts present and correct
4 — Most key facts correct, minor omissions or imprecision
3 — Some key facts correct, moderate gaps
2 — Few facts correct, significant errors or hallucinations
1 — Completely wrong, irrelevant, or contradicts reference

Reference answer: {ground_truth}
Generated answer: {answer}

Respond with JSON: {"score": 4, "reason": "one sentence"}
"""

# Normalize 1–5 → 0–1
score_normalized = (raw_score - 1) / 4

Example:

Ground truth: "UPI P2P limit is ₹1 lakh per transaction per day per NPCI guidelines."
Answer:       "You can transfer up to ₹1 lakh daily on UPI for person-to-person transfers."

Judge: score=4 (correct amount + correct type, slight imprecision on "per transaction")
Normalized: (4-1)/4 = 0.75

What the judge model sees: Only the answer and ground truth. NOT the retrieved chunks. This is intentional — if the answer is wrong but the chunks were correct, that's an LLM reasoning failure. If the answer is right but the chunks were irrelevant, that's lucky hallucination.

What fools it:

• 8B judge is lenient — "approximately correct" often scores 4/5
• Rephrased correct answers may score 3 if judge doesn't recognize equivalence
• Very long answers may confuse the judge — score correct facts buried in padding lower

Prism v1.0.0 result: 0.82 — 82% of answers have most key facts correct. Target: >0.85 via better retrieval (Multi-Query → more context → better answers).

---

Metric 2 — Answer Relevancy

Question it answers: "Did Prism answer the question that was actually asked, or did it go off on a tangent?"

How it's computed (RAGAS):

Step 1: Given the generated answer, LLM generates N reverse questions
        Answer: "UPI P2P limit is ₹1 lakh per day..."
        Reverse Q1: "What is the UPI transaction limit?"
        Reverse Q2: "How much can you transfer via UPI P2P daily?"
        Reverse Q3: "What is the per-day UPI P2P cap?"

Step 2: Embed original question + all N reverse questions
        embed("What is the UPI transaction limit for P2P transfers?") → vector_q
        embed("What is the UPI transaction limit?") → vector_r1
        embed("How much can you transfer via UPI P2P daily?") → vector_r2

Step 3: Cosine similarity between original question and each reverse question
        sim(vector_q, vector_r1) = 0.94
        sim(vector_q, vector_r2) = 0.91
        sim(vector_q, vector_r3) = 0.89

Step 4: Answer relevancy = mean similarity = (0.94 + 0.91 + 0.89) / 3 = 0.91

Intuition: If the answer actually addresses the question, reverse-engineered questions from that answer will be similar to the original question. If the answer went off-topic, the reverse questions will diverge.

What fools it:

• Verbosity penalty: Groq 70B tends to over-explain. A padded answer like "Great question! UPI P2P limit is ₹1 lakh. UPI has transformed payments in India, with many PSPs offering..." generates reverse questions about UPI history, not the limit → lower similarity → lower score.
• Doesn't check factual correctness — a confident wrong answer can score high if it's "on topic".

Prism v1.0.0 result: 0.62 — lower than expected. Direct cause: Groq 70B verbose responses. Fix: tighter system prompt ("Be concise. One paragraph maximum.").

---

Metric 3 — Context Recall

Question it answers: "Did the retriever surface all the chunks that were needed to construct a correct answer?"

How it's computed (RAGAS):

Step 1: Take ground truth reference answer
        "UPI P2P limit is ₹1 lakh per day. Merchant UPI payments are capped at ₹5 lakh."

Step 2: RAGAS LLM breaks it into individual factual sentences (claims)
        Claim A: "UPI P2P limit is ₹1 lakh per day."
        Claim B: "Merchant UPI payments are capped at ₹5 lakh."

Step 3: For each claim, check: can this claim be attributed to any retrieved chunk?
        Claim A → search retrieved chunks → Chunk 3 contains "P2P cap ₹1 lakh" → ATTRIBUTED ✓
        Claim B → search retrieved chunks → no chunk mentions merchant cap → NOT ATTRIBUTED ✗

Step 4: Context recall = attributed claims / total claims = 1/2 = 0.50

Intuition: Recall measures completeness of the retrieval pool. Low recall = the retriever missed relevant chunks → LLM can't answer completely even if it tries → answer will be incomplete or hallucinated.

Why this is the most important metric to improve:

• Low recall (0.51 in v1.0.0) means ~half the relevant information is missing from context
• LLM can only work with what it's given — perfect LLM + bad retrieval = bad answer
• Improving recall directly improves answer quality

What fools it:

• Ground truth phrasing matters — if GT says "₹1 lakh" and the chunk says "100,000 rupees", RAGAS LLM may or may not link them
• Long GT answers with many claims → denominator grows → harder to achieve high recall

Prism v1.0.0 result: 0.51 — retriever finds correct chunks (P@5=0.89) but misses ~half the relevant ones. Directly motivates Multi-Query Retrieval.

---

Metric 4 — Precision@5

Question it answers: "Of the 5 chunks Prism returned, how many were actually relevant?"

How it's computed (custom, deterministic — no LLM):

def compute_precision_at_k(query, retrieved_chunks, ground_truth, k=5):
    relevant_sources = ground_truth.get("relevant_sources", [])   # expected source filenames
    keywords = ground_truth.get("relevant_chunk_keywords", [])     # expected keywords

    top_k = retrieved_chunks[:k]
    relevant_count = 0

    for chunk in top_k:
        source_match = chunk["source"] in relevant_sources
        keyword_match = any(kw.lower() in chunk["content"].lower() for kw in keywords)
        
        if source_match or keyword_match:    # OR — either condition counts
            relevant_count += 1

    return relevant_count / k   # 0.0 – 1.0

Example eval pair:

{
  "query": "What is the UPI merchant payment limit?",
  "relevant_sources": ["npci_merchant_guidelines.pdf"],
  "relevant_chunk_keywords": ["merchant", "₹5 lakh", "payment cap", "PSP ceiling"]
}

Retrieved chunks:

Chunk 1 (source: npci_merchant_guidelines.pdf) → source_match ✓ → relevant
Chunk 2 (content: "...merchant PSP ceiling is ₹5 lakh...") → keyword_match ✓ → relevant
Chunk 3 (source: rbi_circular.pdf, content: "UPI transaction monitoring...") → neither → irrelevant
Chunk 4 (source: npci_merchant_guidelines.pdf) → source_match ✓ → relevant
Chunk 5 (content: "...digital payment systems in India...") → neither → irrelevant

P@5 = 3/5 = 0.60

Why deterministic (no LLM):

• Fast — runs per-query without API calls
• Reproducible — same query always same score
• No rate limits — can evaluate 50 pairs without hitting Groq TPD

What fools it:

• Source OR keyword: a chunk from the wrong document that happens to mention a keyword counts as relevant
• Keyword list quality matters — too broad → everything matches, score inflates → too narrow → correct chunks missed, score deflates

Prism v1.0.0 result: 0.89 — 89% of retrieved chunks are relevant. High precision + low recall = pool is accurate but too narrow. The retriever is finding the right kind of chunks, just not enough of them.

---

Metric 5 — Latency p50 / p95 / p99

Question it answers: "How fast is Prism in practice?"

Why three numbers instead of average:

Query latencies (ms): [1800, 1950, 2100, 1900, 2050, 2200, 1850, 3800, 2000, 1950]
                                                                  ↑ one slow outlier

Mean: 2160ms  ← pulled up by the outlier, misleading
p50:  1975ms  ← 50% of queries faster than this (typical experience)
p95:  3230ms  ← 95% of queries faster than this (worst normal case)
p99:  3740ms  ← 99% of queries faster than this (absolute worst)

How it's computed:

import numpy as np

# Measure per query: retrieval + LLM call (no Tavily, no web search)
t0 = time.time()
answer, docs, chunks = _answer_query(llm, retriever, query)
latency_ms = int((time.time() - t0) * 1000)
latencies.append(latency_ms)

# After all queries:
p50 = int(np.percentile(latencies, 50))   # median
p95 = int(np.percentile(latencies, 95))   # 95th percentile
p99 = int(np.percentile(latencies, 99))   # 99th percentile

What's included in the latency measurement:

• ChromaDB dense retrieval (cosine ANN search)
• BM25 sparse retrieval (in-memory token scoring)
• RRF fusion (pure Python, negligible)
• TinyBERT cross-encoder reranking (BERT forward pass × 10 pairs)
• Groq LLM call (network + inference)

What's NOT included:

• Tavily web search (separate, always adds 500–1000ms)
• HyDE expansion (adds one extra Groq call ~200ms)
• Python GC (gc.collect() after response)

Prism results across versions:

Version	Stack	p50	p95	p99
v1.0.0	baseline	2029ms	—	—
v1.1.0	HyDE	4018ms	6452ms	6455ms
v1.2.0	HyDE+MQ	1812ms	6136ms	6744ms
v1.3.0	HyDE+MQ+CTX	2610ms	6720ms	6959ms

HyDE adds one Groq call per query → p50 2×. Contextual chunks are longer → LLM processes more tokens → higher p50 vs baseline. Wide p95/p99 gap = occasional Groq congestion spikes, not retrieval.

Production benchmarks for context:

System	p50 target	Notes
Search engine	<200ms	Pre-indexed, no LLM
RAG (GPU cloud)	500–800ms	GPU reranker + fast LLM
Prism (Render free CPU)	~2000ms	CPU reranker + network LLM
Prism + streaming	~300ms to first token	Same total time, perceived faster

---

How the Metrics Interact — Reading the Dashboard

High correctness + high recall + high P@5 = system is working ✅

High P@5 + low recall:
→ Retriever finds right chunks but misses others
→ Fix: Multi-Query Retrieval, HyDE

Low P@5 + high recall:
→ Retriever finds many chunks but most are noise
→ Fix: stricter reranking, smaller retrieve_k

High recall + low correctness:
→ Retriever gave good context, LLM failed to use it
→ Fix: better system prompt, larger LLM model

Low relevancy + high correctness:
→ LLM is verbose/padded but factually right
→ Fix: tighter prompt ("be concise, one paragraph maximum")

High latency p99 >> p50:
→ Occasional slow queries (complex retrieval or Groq congestion)
→ Monitor, add timeout handling

Metric evolution across Prism versions:

              v1.0.0   v1.1.0   v1.2.0   v1.3.0
              (base)   (HyDE)   (HyDE+MQ)(+CTX)
correctness:  0.82     0.75     0.77     0.78
relevancy:    0.62     0.845    0.890    0.799
recall:       0.51     0.721    0.645    0.768   ← primary target
P@5:          0.89     0.911    0.904    0.984
p50:          2029ms   4018ms   1812ms   2610ms

Verdict (v1.0.0): recall is the primary bottleneck.
Fix: HyDE (+21pp recall), Multi-Query (wider pool), Contextual Retrieval (+18% recall).
Best production stack: v1.3.0 — HyDE + Multi-Query + Contextual Retrieval.

---

17. Multi-Query Retrieval — Wider Net for Higher Recall

The Problem: Single Phrasing Has Blind Spots

A query hits ChromaDB + BM25 with one set of tokens. Corpus chunks that express the same concept differently never surface.

Query: "What is the UPI merchant limit?"

Corpus has:
  Chunk A: "merchant UPI transaction cap is ₹5 lakh"      ← found ✓ (keyword match)
  Chunk B: "ceiling for PSP collections was revised..."    ← MISSED (different vocab)
  Chunk C: "payment service providers may not exceed..."   ← MISSED (no "merchant" token)

Chunk B and C are relevant. Single-phrasing retrieval misses them. This is why Prism v1.0.0 Violet had context_recall = 0.51 — only half the needed chunks were retrieved.

The Fix: Generate 3 Phrasings, Pool Results

Step 1: LLM generates 3 paraphrases of the original query
  Original:    "What is the UPI merchant limit?"
  Phrasing 2:  "What is the PSP payment ceiling for UPI collections?"
  Phrasing 3:  "How much can merchants accept via UPI transactions?"

Step 2: Run retrieval for EACH phrasing
  Original     → [Chunk A, Chunk D, Chunk E, ...]  (retrieve_k per list)
  Phrasing 2   → [Chunk B, Chunk A, Chunk F, ...]  ← Chunk B appears now
  Phrasing 3   → [Chunk C, Chunk B, Chunk G, ...]  ← Chunk C appears now

Step 3: Pool + deduplicate by content key (keep best rank per chunk)
  Combined unique pool: [A, B, C, D, E, F, G, ...]

Step 4: RRF fuse the deduplicated pool → rerank top-5
  Reranker sees 30 candidates instead of 10 → picks best 5 from wider pool

Why Deduplication Uses Best Rank

If Chunk A appears at rank 1 in phrasing 1 and rank 3 in phrasing 2, keep it at rank 1 — its highest-confidence rank. The merged ranked list is then sorted by best rank before RRF fusion.

# retriever.py — deduplication loop
for rank, doc in enumerate(d_results):
    key = doc["content"][:120]
    if key not in dense_seen or rank < dense_seen[key][0]:
        dense_seen[key] = (rank, doc)  # keep best (lowest) rank

dense_pool = [doc for _, doc in sorted(dense_seen.values(), key=lambda x: x[0])]

In Prism

# retriever.py
def _multi_query_expand(self, query: str) -> list[str]:
    prompt = (
        "Generate 3 different phrasings of the following question for document retrieval. "
        "Each phrasing must use different vocabulary but seek the same information. "
        "Return ONLY the 3 questions, one per line, no numbering, no preamble.\n\n"
        f"Question: {query}"
    )
    # returns [original_query, phrasing_2, phrasing_3, phrasing_4]

def _get_relevant_documents(self, query, ...):
    queries = self._multi_query_expand(query) if self.use_multi_query else [query]
    
    dense_seen, sparse_seen = {}, {}
    for q in queries:
        # retrieve for each phrasing, keep best rank per unique chunk
        ...
    
    dense_pool = sorted_by_best_rank(dense_seen)
    sparse_pool = sorted_by_best_rank(sparse_seen)
    
    fused = self._rrf_fuse(dense_pool, sparse_pool)   # wider pool
    reranked = rerank(query, fused[:retrieve_k], top_k=rerank_k)  # reranker uses original query

ON by default (config.yaml multi_query_enabled: true) — adds one Groq call per query (~200ms).

Measured result (v1.2.0, 25 samples, HyDE+MQ): recall=0.645, P@5=0.904. Multi-Query alone provided smaller-than-expected recall lift. Root cause: query-side reformulation cannot fix decontextualized chunk embeddings — chunks with weak vectors rank low regardless of how the query is phrased. The larger recall gain came from Contextual Retrieval (Concept 21), which fixes chunk quality at ingest time.

Cost: 1 extra Groq call + 3× retrieval calls (fast, in-memory) + 3× BM25 (negligible).

Reranker still uses original query — not the phrasings. The phrasings widen the candidate pool; the reranker judges relevance against what the user actually asked.

---

Summary — Key Concepts Cheatsheet

Concept	What it solves	Where in Prism
ParentDocumentRetriever	Retrieval precision + answer faithfulness	Ingestion + retrieval layer
InMemoryStore	Fast parent text lookup by ID	RAM store (lost on restart)
ChromaDB	Vector similarity search for child chunks	Persistent disk store
BM25Okapi	Exact keyword matching with TF saturation	Sparse retrieval (weight 0.3)
RRF	Merging dense + sparse rankings (scale-agnostic)	Post-retrieval fusion
Cross-Encoder	Accurate joint query-doc relevance scoring	Final reranking step
Chain condensation trap	Why LangChain strips web context silently	Web query bypass in chain.py
HyDE	Closes question-answer vector space gap; +21pp recall	Dense retrieval (hyde_enabled: true)
Faithfulness is circular	Why eval metrics designed alongside corpus lie	answer_correctness chosen as primary
P@5 + Recall diagnostic pair	Identifies whether retrieval pool is narrow or noisy	v1.0.0 Violet: P=0.89, R=0.51
RecursiveCharacterTextSplitter	Splits at paragraph/line/word/char in priority order	ingest.py chunking
ConversationBufferWindowMemory	Sliding k-window of chat history + output_key trap	memory.py, used in chain
LLM at ingest time	Briefing pattern — run LLM once per doc, not per query	briefing.py on upload
Contextual Retrieval	Fix decontextualized chunks at ingest; +18% recall	contextualize_chunks_async() in ingest.py
Answer Correctness	LLM judge 1–5 vs ground truth, normalized → 0–1	Primary metric, independent of retrieved chunks
Answer Relevancy	Reverse-question cosine similarity — penalizes verbose LLM	RAGAS, no ground truth needed
Context Recall	GT sentences attributed to retrieved chunks ÷ total	RAGAS, measures retrieval completeness
Precision@5	Relevant chunks in top-5 ÷ 5, source + keyword match	Custom deterministic, no LLM call
Latency p50/p95/p99	Percentile timing: median / worst-normal / absolute-worst	numpy.percentile over per-query ms measurements
Multi-Query Retrieval	Wider candidate pool; best rank dedup before RRF	retriever.py _multi_query_expand() (multi_query_enabled: true)
Semantic Chunking Tradeoff	Recall vs precision when eval is fixed-chunk-aligned	v1.4.0 ablation: +9.3pp recall, −27.3pp P@5, 5× latency

---

---

20. Semantic Chunking Tradeoff — Recall vs Precision

The Problem with Fixed-Size Splits

RecursiveCharacterTextSplitter at 500 chars cuts mid-sentence, mid-table, mid-list. Embedding a truncated sentence returns a weak vector — the chunk doesn't represent a complete idea. Semantic chunking fixes this by splitting at topic boundaries (where cosine similarity between adjacent sentences drops below a threshold), producing variable-size chunks that each represent one coherent thought.

What the Ablation Showed (Prism v1.4.0, 25 samples)

Stack	recall	P@5	latency p50
v1.3.0 HyDE+MQ+CTX (fixed 500-char)	0.768	0.984	2610ms
v1.4.0 + Semantic chunking	0.861	0.711	12952ms
Delta	+9.3pp	−27.3pp	5× slower

Why Recall Went Up

Semantic chunks contain complete sentences and complete ideas. The embedding captures the full meaning — better vector → higher chance of matching the relevant query → more reference content covered → higher context recall.

Why Precision Collapsed

P@5 measures overlap between retrieved chunks and the ground-truth relevant_chunks field in eval_pairs.json. Those reference chunks were defined against fixed 500-char splits. Semantic chunks have different, larger boundaries — they contain the relevant content but as part of a bigger unit, so exact overlap with the reference drops. The reranker also receives a wider, noisier candidate pool (variable-size chunks = less uniform scoring surface).

Key insight: P@5 is eval-alignment-sensitive. If ground truth was labelled against fixed chunks, semantic chunks will always score lower on P@5 even when they retrieve better content. For a production system with human-labelled relevance judgements (not self-aligned eval pairs), the gap would be smaller.

Why Latency Was 5×

Semantic chunks are longer on average than 500-char fixed chunks. Longer chunks = more tokens fed to the LLM per answer generation call. Retrieval time is unchanged (semantic chunking is an ingest-time decision), but inference time scales with context length.

Decision

Semantic chunking rejected for Prism production. v1.3.0 (HyDE + Multi-Query + Contextual) confirmed as best stack: P@5=0.984, recall=0.768, p50=2610ms.

Semantic chunking would make more sense when:

• Ground truth eval labels are created after chunking (aligned to the actual chunk boundaries)
• LLM context window is not a bottleneck
• Recall is the primary metric (e.g. legal/compliance: never miss a relevant clause)

---

21. Contextual Retrieval — Fixing Decontextualized Chunks at Ingest

The Problem: Fixed-Size Chunks Lose Context

RecursiveCharacterTextSplitter at 500 chars cuts documents into fragments. Many fragments are decontextualized — they lack the surrounding information that gives them meaning:

Chunk from NPCI merchant guidelines PDF:
"The limit was revised to ₹2 lakh."

Problems:
- "The limit" — which limit? Not in this chunk.
- "revised" — from what? Not in this chunk.
- "₹2 lakh" — for what transaction type? Not in this chunk.

Embedding of this chunk → weak, generic vector.
Query "merchant UPI transaction cap" → this chunk may not surface.

No query-side technique (HyDE, Multi-Query) can fix this — the chunk embedding is weak regardless of how the query is phrased.

The Fix: Anthropic's Contextual Retrieval

At ingest time, before embedding, ask the LLM to prepend 2 sentences situating each chunk in its document:

Prompt:
  "Given this document: [full doc or representative sample]
   Write 2 sentences situating this chunk in context.
   
   Chunk: 'The limit was revised to ₹2 lakh.'"

LLM output:
  "In the NPCI UPI merchant guidelines (2024), Section 4.3 covers
   transaction ceiling revisions for PSPs. The limit was revised to ₹2 lakh."

The contextual prefix is prepended to the chunk before embedding:

# ingest.py
chunk.page_content = f"{context_prefix}\n\n{chunk.page_content}"
# then embed this augmented text

The chunk stored for retrieval is now specific and rich. Same chunk now surfaces for "merchant UPI transaction cap" queries.

Why This Works

	Original chunk	Contextual chunk
Text	"The limit was revised to ₹2 lakh."	"NPCI UPI merchant guidelines 2024, PSP ceiling. The limit was revised to ₹2 lakh."
Embedding	Generic "revision" vector	Specific "merchant UPI PSP ceiling" vector
Retrieval	Misses merchant-related queries	Surfaces correctly

The embedding now represents a complete, specific idea instead of a floating fragment.

Design Pattern: Ingest-Time vs Query-Time

	Ingest-time LLM (Contextual Retrieval)	Query-time LLM (HyDE, Multi-Query)
Runs	Once per chunk, at upload	Every query
Cost	Paid once; benefit on every future query	Paid per query
What it fixes	Bad chunk embeddings (quality problem)	Query-corpus vocabulary gap (coverage problem)
Latency impact	Upload slower (~40s for 30 chunks)	Query slower (+200ms per technique)

Key insight: Query-side techniques improve how well a query matches existing chunk vectors. Contextual retrieval improves the chunk vectors themselves. Both are needed for maximum recall.

Measured Results — Prism v1.3.0 (25 samples)

Metric	v1.0.0 baseline	v1.3.0 (HyDE+MQ+CTX)	Delta
context_recall	0.51	0.768	+25.8pp
precision_at_5	0.89	0.984	+9.4pp
answer_relevancy	0.62	0.799	+17.9pp
latency p50	2029ms	2610ms	+28%

Contextual retrieval contributes ~+18% recall (v1.3.0 vs v1.2.0 without CTX: 0.768 vs 0.645).

Production Complication: 40s Upload

30 chunks × 1 Groq call × ~1.3s/call (sequential) = ~40s blocking. Solution: parallel calls with asyncio.Semaphore(3) — 3 concurrent Groq calls at ~3000 TPM burst (under Groq's 6000 TPM limit). Reduces to ~15s. Two-phase upload: sync non-contextual embed first (user can query in <3s), contextual replacement in background.

Distinction from Briefing (Concept 14)

	Briefing	Contextual Retrieval
What	Document-level 5-bullet summary + 3 questions	Chunk-level 2-sentence situating context
Shown to user	Yes (in upload response)	No (prepended to chunk text, invisible)
Purpose	User orientation	Embedding quality improvement
LLM calls	1 per document	1 per chunk (~30 per doc)