docs/evolution.md

Prism — Project Evolution

End-to-end record of what was broken at each stage, what was built to fix it, and what is planned next. Updated as the project evolves. Last updated: 2026-07-05 (Stage 18).

---

Table of Contents

1. Stage 0 — v1 Baseline
2. Stage 1 — v2 Hybrid Retrieval Architecture
3. Stage 2 — Chain Scores + RAGAS Endpoint
4. Stage 3 — Groq Migration + Web Search
5. Stage 4 — OOM Hell on Render
6. Stage 5 — TinyBERT + RAGAS Removal + Benchmark JSON
7. Stage 6 — Multi-Workspace
8. Stage 7 — Singleton Cache + URL Guard
9. Stage 8 — Eval Dashboard + Rigorous Metrics
10. Stage 9 — Multi-Query Retrieval
11. Stage 10 — Contextual Retrieval (Eval)
12. Stage 14 — Briefing Fix + HyDE Re-eval
13. Stage 12 — HF Spaces Migration
14. Stage 11 — Contextual Retrieval in Production + Dashboard Polish
15. Stage 15 — Semantic Chunking Ablation + Retrieval Stack Finalized
16. Stage 16 — Citation Highlighting
17. Stage 17 — Metadata Filtering
18. Current State Snapshot
19. Roadmap — Retrieval & Answer Quality
20. Roadmap — New Features

---

Stage 0 — v1 Baseline

What existed

• Dense-only ChromaDB vector retrieval
• Single global document collection
• Basic chat with ConversationalRetrievalChain
• No evaluation framework
• No web search
• No logging

What was wrong

Problem	Impact
Dense-only retrieval	Misses exact keyword matches — regulatory text has section numbers, policy codes, specific terms that semantic search fails on
No evaluation	No way to measure if answers were correct or grounded
No web search	Static corpus only — cannot answer questions about current stock prices, recent news
Single collection	No topic isolation — all documents mixed in one retrieval pool
No logging	Impossible to debug production failures

This was the starting point. No fixes yet.

---

Stage 1 — v2 Hybrid Retrieval Architecture (2026-05-17)

What was wrong before building

• retriever.py was dense-only ChromaDB — v2 was documented but not implemented
• ragas_eval.py missing entirely; RAGAS eval endpoint not wired
• No BM25, no reranker, no score visibility

What we built

File	What changed
server/bm25_index.py	BM25Okapi singleton; module-level (not app.state) so importable anywhere; rebuilt on startup + after upload
server/reranker.py	CrossEncoder singleton; pre-loaded at startup to avoid cold-start latency on first query
server/retriever.py	Full rewrite as HybridRetriever(BaseRetriever) — RRF fusion of dense (weight 0.7) + sparse (weight 0.3)
server/main.py	BM25 build + reranker load wired into lifespan startup
server/routes/upload.py	BM25 rebuild triggered after each upload

Key design decisions

• HybridRetriever as BaseRetriever subclass — ConversationalRetrievalChain expects a BaseRetriever; subclassing means chain.py needs zero changes
• BM25 as module-level singleton — avoids threading state through lifespan → constructor; get_index() importable anywhere
• Reranker pre-loaded at startup — ~0.5s load from disk cache; better to pay at startup than add latency to first user query
• MiniLM-L-6-v2 chosen as reranker (~85MB) — best ranking quality available at the time

What was still missing

• Chain score extraction (similarity/BM25/RRF/rerank not returned in API response)
• RAGAS eval endpoint
• Groq LLM (still on Euron)

---

Stage 2 — Chain Scores + RAGAS Endpoint (2026-05-23)

What was wrong

• API response had no retrieval scores — no way to show per-source similarity/BM25/RRF/rerank scores
• RAGAS eval endpoint not wired; ragas_eval.py missing
• data/ground_truth/eval_pairs.json had only keyword hints, no ground_truth answers → context_precision and context_recall always returned null

What we built

File	What changed
server/chain.py	Score extraction — similarity/bm25/rrf/rerank scores passed through to API response per source
server/routes/chat.py	Added retrieval_method field; stores contexts in eval_log for downstream RAGAS eval
server/config.yaml	Hybrid retrieval params: dense_weight, sparse_weight, retrieve_k, rerank_k
requirements.txt	Added rank_bm25, sentence-transformers, ragas, datasets
server/eval/ragas_eval.py	RAGAS faithfulness + answer_relevancy via LangchainLLMWrapper
server/routes/eval.py	POST /api/eval/ragas endpoint wired
server/main.py	/health endpoint added

What was still broken

• RAGAS not installed in venv (added to requirements.txt; installs at Docker build only)
• eval_pairs.json still had no ground_truth → 2 of 4 RAGAS metrics null
• LLM still on Euron gpt-4.1-mini

---

Stage 3 — Groq Migration + Web Search Fixes (2026-05-24)

What was wrong

Problem	Root cause
LLM on Euron (gpt-4.1-mini)	Closed model, slower, weaker interview story vs open-weight
Web search silently broken	tavily-python in requirements.txt but never pip-installed
Tavily returned shallow results	search_depth="basic" — not enough content from financial sites
Web context never reached LLM	ConversationalRetrievalChain's condensation step rewrote the question and stripped prepended Tavily context before LLM ever saw it
Follow-up web queries returned garbage	Raw follow-up ("Is the price level good?") sent to Tavily with no chat history context
No request logging	Production failures undebuggable

What we built

Component	Change
LLM	Migrated Euron → Groq llama-3.3-70b-versatile via langchain-groq. Euron kept for embeddings (Groq has no embeddings endpoint)
server/chain.py	run_query_with_web() — bypasses chain condensation; direct LLM call with RAG + Tavily context + memory
server/chain.py	condense_question() — rewrites follow-up queries using chat history before Tavily search
Tavily	search_depth="advanced", max_results=3 (2× credits but richer content)
server/main.py	Request logging middleware — logs METHOD /path STATUS Xms per request
server/utils.py	Centralised logging — root logger + logs/finrag.log (5MB×3 rotation), noisy libs silenced
frontend/.../MessageBubble.jsx	WebSourcesList component — Tavily URLs as clickable green pill links
.env.example	Fixed — real keys had been committed; replaced with placeholders

Key discoveries

• ConversationalRetrievalChain condensation = silent context killer for web queries. Only fix: bypass the chain entirely for web path.
• Memory's output_key="answer" — save_context must use {"answer": answer} not {"output": answer} or KeyError.

---

Stage 4 — OOM Hell on Render (2026-05-24, four sub-issues)

Render free tier: 512MB RAM. This stage was four separate OOM root causes discovered in sequence.

---

4a — CUDA torch OOM (startup crash)

Problem: sentence-transformers pulled CUDA torch (~2GB) by default. OOM before uvicorn bound to port → Render showed "No open ports detected" timeout. Zero server stdout — invisible failure.

Diagnosis clue: Build log showed cuda-toolkit-13.0.2, nvidia-cublas being installed. Port scan timeout = uvicorn crash at import time (not lifespan — lifespan runs after port bind).

Fix:

# Install CPU-only torch BEFORE requirements.txt
RUN pip install torch --index-url https://download.pytorch.org/whl/cpu
RUN pip install -r requirements.txt

ENV HF_HUB_OFFLINE=1
ENV TRANSFORMERS_OFFLINE=1

---

4b — ragas startup ImportError

Problem: ragas 0.4.3 imports langchain_community.chat_models.vertexai at package __init__ level. That module was removed in langchain-community 0.4.x. Crash propagated: eval.py → ragas_eval.py → ragas.__init__ → ImportError before uvicorn bound port.

Fix: All ragas imports moved inside run_ragas_eval() function body (lazy import).

---

4c — CrossEncoder OOM during chat

Problem: CrossEncoder.predict(20 pairs) = BERT forward pass on 20 pairs → ~200–400MB spike on top of base ~250MB → OOM on first web query.

Fix:

• config.yaml: retrieve_k: 20 → 10
• reranker.py: model.predict(pairs, batch_size=4) — limits how many pairs processed at once

---

4d — Web search OOM (post-GC headroom)

Problem: After first query, Python retained chain/LLM objects at ~490MB. Web query added ~9KB Tavily content + condense_question LLM call + run_query_with_web LLM call → OOM.

Fix:

• Tavily content truncated to 800 chars per result (was up to ~3000)
• max_results: 3 → 2
• gc.collect() after each chat request in routes/chat.py

---

Stage 5 — TinyBERT + RAGAS Removal + Benchmark JSON (2026-05-30)

What was wrong

Problem	Root cause
MiniLM-L-6-v2 (~85MB) still OOMing on web queries	Too close to 512MB ceiling even after GC
Live RAGAS eval always 500 on Render	nest_asyncio.apply() (called at ragas import time) cannot patch uvloop — the event loop uvicorn uses on Linux. ValueError: Can't patch loop of type uvloop.Loop. Permanently unfixable without replacing uvicorn's event loop.
Groq 70B exhausted 100k daily tokens in one RAGAS run	RAGAS makes ~10 LLM calls per sample for statement decomposition. 10 samples × 10 calls = 100k tokens gone.
context_precision and context_recall always null	eval_pairs.json had no ground_truth answers — only keyword hints

What we built

Component	Change
server/reranker.py	Switched to cross-encoder/ms-marco-TinyBERT-L-2-v2 (~17MB vs 85MB). Saves 68MB permanently.
server/routes/eval.py	Removed POST /api/eval/ragas endpoint
requirements.txt	Removed ragas
scripts/run_ragas_local.py	Local RAGAS runner: ingest corpus → generate answers → run eval → write JSON. Uses llama-3.1-8b-instant as judge (500k TPD vs 70B's 100k TPD)
frontend/src/data/ragas_benchmark.json	Static scores — Vercel builds dashboard from file
frontend/.../EvalPanel.jsx	Replaced live run button with static benchmark panel
data/ground_truth/eval_pairs.json	Added ground_truth field to all 20 pairs → unlocked context_precision + context_recall
UI	i-button tooltips on faithfulness badge + all 4 RAGAS metric cards

Real scores committed

faithfulness:      1.0   (note: likely inflated — see below)
answer_relevancy:  0.90
context_precision: TBD   (pending fresh run)
context_recall:    TBD   (pending fresh run)

Key discoveries

• results["metric_name"] returns None in ragas 0.2.x — must use results.to_pandas()["metric_name"].mean()
• TinyBERT loads with harmless UNEXPECTED key bert.embeddings.position_ids warning
• Faithfulness 1.0 is likely inflated — eval queries were designed alongside the corpus, and 8B judge is lenient. Scores are directional, not absolute. Run on held-out queries for honest numbers.

---

Stage 5.5 — Rebranding: FinRAG → Prism (2026-06-13)

What changed

The project was originally named FinRAG — a fintech-specific RAG demo. As the architecture matured (multi-workspace, URL ingestion, domain-agnostic retrieval), it became clear the tool was no longer fintech-specific. Any corpus — legal, HR, medical, research — could be loaded and queried.

Decision: Rebrand to Prism. Name reflects the core idea: feed any document set in, get clear structured answers out. One engine, any domain.

Before	After
FinRAG	Prism
Fintech-specific framing	Domain-agnostic positioning
finrag-v2.onrender.com	prism.onrender.com
README pitched at fintech analysts	README pitched at any knowledge-worker

What stayed the same

All retrieval architecture, eval framework, and deployment stack unchanged. Rebrand is naming and framing only — the engine is identical.

What was wrong with the old name

• "FinRAG" implied fintech-only → narrowed the demo audience
• Interviewers at non-fintech MNCs (Adobe, Atlassian, Intuit) would dismiss it as domain-locked
• The actual retrieval engine is domain-agnostic — the name should match

---

Stage 6 — Multi-Workspace (2026-06 early)

What was wrong

• Single ChromaDB collection — no isolation between document sets
• Switching topics meant re-ingesting and overwriting previous docs
• list_collections() broke on chromadb ≥0.5.4 (returns list[str], not list[Collection])
• Non-web chat path used stale global chain's source_documents instead of workspace-specific retriever → wrong docs shown after workspace switch

What we built

File	Change
server/routes/workspaces.py	Workspace CRUD — one ChromaDB collection per workspace
server/routes/chat.py	Always resolves workspace-specific retriever before branching on web_search
server/routes/workspaces.py	list_collections() normalised with isinstance check — works on chromadb ≥0.5.4 (list[str]) and <0.5 (list[Collection])
frontend/src/components/Sidebar.jsx	Workspace switcher UI; per-workspace doc list
frontend/src/App.jsx, api.js, ChatArea.jsx, FileUpload.jsx	workspace_id passed on all requests

Key discovery

• Non-web path was relying on stale global chain's source_documents rather than workspace-specific retriever. After switching workspaces, the wrong collection's docs were being cited.

---

Stage 7 — Singleton Cache + URL Guard (2026-06-14) ← Current

What was wrong

• Every POST /api/chat called get_or_create_collection() + built a new HybridRetriever = full embedding reload per request → OOM after 2–3 queries in the same workspace
• React component state (message list) persisted across workspace switch — showed previous workspace's chat history
• External URL ingestion had no size guard → large pages (news articles, regulatory filings) caused OOM during embed

What we built

File	Change
server/retriever.py	Module-level Dict[workspace_id, (vectorstore, retriever)] cache. Cache invalidated after ingest. routes/chat.py reuses cached retriever.
server/routes/chat.py	Eliminated double retrieval on non-web path; fixed stray print statement
server/url_loader.py	Max content size guard before embedding external URL content
frontend/src/App.jsx	key={workspaceId} on <ChatArea> → remounts component on workspace switch → clears stale messages and state

Commits

529675f  fix: singleton vectorstore/retriever cache to prevent OOM on repeated queries
6c9f809  fix: URL size guard for OOM prevention, eliminate double retrieval in chat
521a27a  fix: remount ChatArea on workspace switch to clear stale messages

Additional fix (2026-06-16) — HyDE (Hypothetical Document Embeddings)

What: Before dense ChromaDB search, LLM generates a hypothetical 2-sentence answer. That answer (not raw query) is embedded for ANN search. BM25 + reranker still use original query.

File	Change
server/retriever.py	_hyde_expand() method; use_hyde: bool field on HybridRetriever; dense path uses expanded query when enabled
config.yaml	retrieval.hyde_enabled: false — toggle without code change

Why off by default: Adds one Groq call per query (~200ms). Enable to measure RAGAS context_recall lift, then decide.

Commit: 8945b43

---

Additional fix (2026-06-17) — Mandatory web search

Problem: Web search was opt-in toggle. Users querying corpus-only got hallucinated answers from irrelevant documents (e.g., Singapore visa question grounded in random passport-mentioning corpus doc, faithfulness 4/5).

Fix:

File	Change
frontend/src/components/ChatArea.jsx	Removed toggle button; const webSearch = true hardcoded; placeholder always says "docs + web"
server/routes/chat.py	web_search: bool = True as default in ChatRequest

Every query now hits Tavily + RAG corpus. run_query_with_web always called with both rag_docs + web_sources.

---

Stage 8 — Eval Dashboard + Rigorous Metrics (2026-06-17)

What was wrong

• faithfulness 1.0 and context_precision 1.0 artificially inflated — eval pairs designed alongside corpus, 8B judge lenient. Meaningless scores.
• Per-message faithfulness badge cluttered user UI. Users don't care about LLM judge scores.
• 10 samples — not statistically meaningful.
• Single flat JSON, no versioning — no way to track metric evolution across architecture changes.

What we built

Component	Change
eval-dashboard/	Separate Vite + React static site (own Vercel project). Reads versioned JSON run files.
eval-dashboard/src/components/	MetricCard (score + delta vs prev), EvolutionChart (Recharts line chart across versions), RunTable (per-query expandable rows with answer vs ground_truth), LatencyStats (p50/p95 bars)
eval-dashboard/public/data/index.json	Run registry — list of all versioned eval runs
scripts/run_eval_versioned.py	New eval script. Args: --version, --tag, --n. Computes answer_correctness (LLM judge vs ground_truth), answer_relevancy + context_recall (RAGAS), precision@5, latency p50/p95/p99. Writes versioned JSON + updates index.
data/ground_truth/eval_pairs.json	Expanded 20 → 50 pairs. Added multi-hop, comparative, negative, numeric, and edge-case questions.
frontend/src/components/MessageBubble.jsx	Removed FaithfulnessBadge component and rendering block.
server/routes/chat.py	Removed score_faithfulness() call. One fewer Groq API call per query → faster responses.

Metrics before vs after

Metric	Before	After
faithfulness	1.0 (inflated)	Removed from prod path
context_precision	1.0 (inflated)	Replaced by answer_correctness (LLM judge vs ground_truth)
answer_relevancy	0.88	Kept (RAGAS)
context_recall	0.83	Kept (RAGAS)
precision@5	tracked separately	Now in main eval dashboard
latency p50/p95	not tracked	Now tracked per eval run
sample_count	10	50 (5× improvement)

---

Stage 9 — Multi-Query Retrieval (2026-06-19)

What was wrong

• context_recall = 0.51 in v1.0.0 Violet — retriever missed ~half the relevant chunks
• Single-phrasing retrieval only surfaces chunks whose vocabulary matches the query tokens
• Chunks expressing same concept with different words (e.g. "PSP ceiling" vs "merchant limit") never entered the candidate pool

What we built

File	Change
server/retriever.py	_multi_query_expand() — Groq LLM generates 3 phrasings (temperature=0.3). _get_relevant_documents() iterates all phrasings, deduplicates by content key keeping best rank, RRF fuses pooled results, reranks with original query.
config.yaml	retrieval.multi_query_enabled: false toggle
docs/learning.md	Concept 17 — Multi-Query Retrieval

Key design decisions

• Deduplication keeps best rank — a chunk at rank 1 in one phrasing and rank 8 in another enters RRF at rank 1, not 8
• Reranker uses original query — phrasings widen the pool; the reranker judges relevance against what the user actually asked
• Off by default — adds one Groq call per query (~200ms). Enable → run v1.1.0 "Indigo" eval → measure delta → decide
• retrieve_k cap maintained — reranker input capped at retrieve_k even with wider pool, preserving RAM budget on Render

Expected outcome

• context_recall: 0.51 → measurably higher (target: >0.65)
• P@5: ~0.89 (no degradation expected — reranker filters noise from wider pool)
• Latency: +200–300ms per query (one extra Groq call for phrasing generation)
• Next eval run: scripts/run_eval_versioned.py --version v1.1.0 --tag "Indigo" --n 50

---

Stage 10 — Contextual Retrieval (Eval) (2026-06-20)

What was wrong

Phase 1 (HyDE + Multi-Query) left context_recall at ~0.51. Root cause confirmed: fixed-size 500-char splits produce decontextualized chunks. "The limit was revised to ₹2 lakh." has no document name, no section, no subject — weak embedding that misses ~half relevant content. Query-side techniques cannot fix bad chunk quality.

What we built

File	Change
server/ingest.py	contextualize_chunks(chunks, documents, model, sleep_between_calls) — calls Groq 8B per chunk, prepends 2-sentence situating context to page_content before embedding. Fallback to original text on any failure.
config.yaml	contextual_retrieval.enabled: false, contextual_retrieval.model: llama-3.1-8b-instant
scripts/run_eval_versioned.py	--contextual flag + --data-dir arg. When set: clears eval_ctx collection, re-ingests with contextualize_chunks, evaluates against eval_ctx. Production upload untouched.
tests/test_ingest.py	3 tests: context prepended, fallback on failure, empty chunks skipped

Results — v1.3.0 "Violet"

Metric	v1.0.0 baseline	v1.3.0 contextual	Delta
context_recall	0.510	0.601	+9.1pp (+18%)
precision_at_5	0.890	0.956	+6.6pp
answer_relevancy	0.620	0.633	+1.3pp
answer_correctness	0.820	0.815	-0.5pp (noise)
latency p50	2029ms	4161ms	+2× ⚠️
latency p95	—	6122ms	—

Key discoveries

• Biggest single lift across all Phase 1+2 experiments: recall +9.1pp absolute
• Precision also improved significantly (0.890→0.956) — wider context gives reranker stronger signal
• Latency 2× because contextualized chunks are longer (~150 extra tokens per chunk) → LLM processes more tokens per answer generation call. Zero retrieval-time overhead (as designed), but query-time cost is real.
• Recall target was 0.65 — hit 0.60. Gap remains; next candidate is semantic chunking (Phase 2b)
• Production path: if shipping contextual retrieval, need FastAPI BackgroundTask for async contextualization at upload time (otherwise user waits 40s+ per doc upload)

---

Stage 11 — Contextual Retrieval in Production + Dashboard Polish (2026-06-20)

What was wrong

• Contextual retrieval proven in eval (recall +18%) but never shipped to production — users got non-contextual chunks
• eval-dashboard X-axis showed raw version strings (v1.0.0) with no dates
• No version badge visible in main Prism UI
• Encrypted PDFs caused 500 Internal Server Error instead of a clean user-facing message
• max_concurrent=20 for parallel Groq calls → 20k token burst → 429 TPM limit on free tier (6000 TPM)
• Render free tier ephemeral filesystem: docs lost on every cold start (known limitation)
• Upload blocking for ~52s while Euron embedding API processes chunks sequentially

What we built

File	Change
server/ingest.py	contextualize_chunks_async() — parallel Groq calls via asyncio.gather + Semaphore(max_concurrent). ~10× faster than sequential. Retry parses suggested wait time from 429 error message.
server/routes/upload.py	Two-phase upload: sync non-contextual embed first (user queryable immediately), then _contextual_refresh_bg() BackgroundTask replaces non-contextual chunks with contextual versions.
config.yaml	contextual_retrieval.enabled: true, max_concurrent: 3 (3 × ~1000 tokens = 3000 TPM — safe under 6000 limit)
server/ingest.py	load_documents_from_paths(): catches FileNotDecryptedError → raises ValueError with user-friendly message
server/routes/upload.py	Catches ValueError from loader → returns HTTP 422 instead of 500
eval-dashboard/src/components/EvolutionChart.jsx	Custom XAxisTick: stacked version name + short date (e.g. Violet (v1.3) / 20 Jun 26)
eval-dashboard/src/App.jsx	VERSION_NOTES constant with bullet notes per version; release notes panel shown below run meta
frontend/src/components/Sidebar.jsx	Violet v1.3 badge (indigo pill) in sidebar footer
frontend/src/config.js	New file — MAINTENANCE_MODE + MAINTENANCE_MESSAGE config flags
frontend/src/App.jsx	Maintenance banner driven by config.js; hidden when MAINTENANCE_MODE = false

Key discoveries

• asyncio.gather with Semaphore(3) keeps burst under 3000 TPM — safe on Groq free tier (6000 TPM limit)
• Groq 429 errors include "Please try again in X.Xs" — parse this for accurate retry sleep instead of hardcoded 2s
• Render free tier: ephemeral filesystem. Every cold start wipes ./chroma_db. Docs must be re-uploaded. Fix: Render persistent disk ($0.25/GB/month)
• Euron embedding API sequential calls: 30 chunks × ~1.7s/call = ~52s blocking upload. Next optimization: move embed to background too (return 202 immediately, notify when ready)
• max_concurrent=20 was the OOM trigger in the previous session — 20 async coroutines each holding ~10MB response + retry state saturated 512MB

---

Stage 15 — Semantic Chunking Ablation + Retrieval Stack Finalized (2026-06-26)

What was wrong

Ablation study incomplete — semantic chunking (v1.4.0) was blocked by Groq rate limits in the prior session. Best production stack unconfirmed.

What we built / ran

Version	Config	recall	P@5	relevancy	correctness	p50
v1.1.0	HyDE	0.721	0.911	0.845	0.750	4018ms
v1.2.0	HyDE+MQ	0.645	0.904	0.890	0.770	1812ms
v1.3.0	HyDE+MQ+CTX	0.768	0.984	0.799	0.780	2610ms
v1.4.0	HyDE+MQ+CTX+Semantic	0.861	0.711	0.885	0.750	12952ms

Decision: semantic chunking rejected

Semantic chunking raises recall +9.3pp (0.768→0.861) but P@5 collapses -27.3pp (0.984→0.711) and latency is 5× worse (2610ms→12952ms p50).

Root cause of P@5 collapse: SemanticChunker produces variable-size, topic-boundary chunks. These don't align with the fixed ground-truth keyword spans used for precision@5 scoring. The reranker receives a wider but noisier candidate pool — recall expands while precision degrades.

Best stack confirmed: v1.3.0 — HyDE + Multi-Query + Contextual Retrieval.

Key discoveries

• MQ alone hurts recall (-7.6pp vs HyDE-only) but recovers fully when combined with CTX
• CTX is highest-leverage single addition: +8pp P@5, recall recovery, at 2× query latency cost
• Semantic chunking is a double-edged sword — better chunk boundaries for recall, worse alignment with precision evaluation
• Ablation study is the interview story: systematic metric-driven elimination of techniques

---

Stage 16 — Citation Highlighting (2026-06-26)

What was wrong

Sources listed below each answer as truncated 200-char snippets. LLM already outputs [1], [2] inline citations but they rendered as plain unclickable text. Users couldn't see which passage in the answer corresponded to which source.

What we built

File	Change
frontend/src/components/CitationPopover.jsx	New — viewport-aware popover (fixed-position). Shows: source type badge (pdf/web/file), filename/title, page badge, full chunk content (scrollable), rerank score, "Open page N →" for PDF / "Open source →" for web
frontend/src/components/MessageBubble.jsx	Parse [N] markers in answer text → clickable <sup> superscripts. openCitation state ({ idx, rect } | null). Toggle on same click. onMouseDown stopPropagation fix (prevents document mousedown from immediately re-opening after close).
frontend/src/components/SourceExpander.jsx	Removed 200-char content truncation — full chunk text shown
server/main.py	Added GET /api/files/{filename} → FileResponse from data/raw/. Path traversal blocked via is_relative_to(). UPLOAD_DIR made absolute (Path(__file__).resolve().parent.parent / "data" / "raw").
server/routes/upload.py	UPLOAD_DIR made absolute (Path(__file__).resolve().parent.parent.parent / "data" / "raw")

Key discoveries

• mousedown on document fires before click — without e.stopPropagation() on the <sup> mousedown, clicking an open citation closes then immediately reopens it (toggle broken)
• startswith() on raw path strings has prefix-confusion bug (/data/rawevil passes /data/raw check) — replaced with Path.is_relative_to() (Python 3.9+)
• Relative Path("data/raw") resolves against process CWD — if uvicorn starts from non-project-root directory, file serving breaks. Absolute __file__-relative path fixes this.
• anchorRect captured at click time via el.getBoundingClientRect() — stored in state as plain object, no ref needed in popover

Interview story

"The LLM cites [1], [2] in its answer. Clicking one opens a popover showing the exact passage retrieved — full text, source file, page number, and rerank score. For PDFs it links directly to that page in the browser."

---

Stage 17 — Metadata Filtering (2026-06-27)

What was wrong

All documents in a workspace were always searched together. A user with 10 docs spanning 5 years had no way to scope a query to a specific doc or subset. Corpus-wide retrieval diluted precision when the relevant content was known to be in one file.

What we built

File	Change
server/ingest.py	source_type metadata field (pdf/txt/csv) added to all chunks at load time via SOURCE_TYPE_MAP. Both load_documents and load_documents_from_paths patched.
server/url_loader.py	source_type: "url" added to URL-ingested doc metadata.
server/bm25_index.py	BM25Index.search() gets filter_sources: set[str] | None = None. When set, scores using full-corpus BM25 index (stable IDF) but restricts candidate pool to matching docs.
server/retriever.py	filter_docs: list[str] | None = None field on HybridRetriever. Wired into _dense_retrieve (ChromaDB where={"source": {"$in": filter_docs}}) and _get_relevant_documents (BM25 filter_sources). New get_retriever_filtered(workspace_id, filter_docs) helper — one-off instance reusing cached vectorstore, not added to singleton cache.
server/routes/chat.py	filter_docs: list[str] | None = None on ChatRequest. Guard: empty list → None. When truthy: get_retriever_filtered(workspace, active_filter). Log includes filter=%s.
frontend/src/api.js	streamChat gets filterDocs = null as 4th arg; sends filter_docs: filterDocs?.length ? filterDocs : null.
frontend/src/App.jsx	filterDocs: string[] state. useEffect resets to [] on workspace change. handleFilterChange toggles doc in/out. Props forwarded to Sidebar + ChatArea.
frontend/src/components/Sidebar.jsx	Doc list items clickable — toggle filter on click. Selected: ring-2 ring-indigo-500 bg-indigo-50. Unselected during active filter: opacity-50. "Clear filter" button in section header when any selected. Delete button: e.stopPropagation() + deselects deleted doc from filter.
frontend/src/components/ChatArea.jsx	Filter badge above input when filterDocs.length > 0 (shows scoped doc names + × clear). Placeholder: "Searching N selected doc(s)..." when filter active. streamChat called with filterDocs.length > 0 ? filterDocs : null.
tests/test_bm25_filter.py	6 tests: no-filter returns all, filter restricts by source, empty set returns empty, nonexistent source returns empty, multiple sources, unbuilt index returns empty.
tests/test_source_type.py	4 tests: pdf/txt/csv/url each gets correct source_type.

Key design decisions

• Full-corpus BM25 for filtering: spec suggested rebuilding BM25 on filtered subset; implementation uses full-corpus index to score + restricts candidate pool by source. Stable IDF — correct IR semantics. Accepted as superior to spec.
• New retriever instance per filtered request: get_retriever_filtered() creates a one-off HybridRetriever; singleton cache (_retriever_cache) untouched. Thread-safe: heavy vectorstore stays cached, lightweight retriever is cheap.
• Empty filter = no filter: backend guard body.filter_docs if body.filter_docs else None — empty array from frontend treated as no filter.
• Filter resets on workspace switch: useEffect(() => setFilterDocs([]), [currentWorkspace]) — stale filter from workspace A doesn't carry to workspace B.
• Delete deselects: handleDelete calls onFilterChange(docName) if deleted doc was selected — prevents badge showing "Scoped to: [deleted]" with zero results.

Interview story

"Within a workspace, users can click any doc chip in the sidebar to scope retrieval. Dense retrieval passes where={"source": {"$in": selected_docs}} to ChromaDB; BM25 pre-filters its candidate pool. Zero selection = full-corpus behavior unchanged. Filter badge above the input makes the scope visible."

---

Stage 18 — Free-Tier Stability + App Restored to Live (2026-07-05)

What was wrong

• Maintenance banner left ON after Cerebras migration attempt (2026-06-29) failed and was reverted
• Config.yaml still had hyde_enabled: true + multi_query_enabled: true — each query burned 4 Groq calls
• Free tier limit: 6000 TPM → 429 storms under concurrent use with HyDE + MQ + contextual all on
• Eval dashboard had no indication which version is live or why best stack (v1.3.0) isn't deployed

What we built

File	Change
config.yaml	hyde_enabled: false, multi_query_enabled: false — reduces query-time Groq calls 4 → 1–2
frontend/src/config.js	MAINTENANCE_MODE: false — app live
eval-dashboard/public/data/index.json	is_live: true + live_note on v1.1.0 (closest proxy); blocked_by constraint on v1.3.0 + v1.4.0
eval-dashboard/src/App.jsx	Green LIVE badge + prod config note on v1.1.0; amber "not in production" warning on v1.3.0/v1.4.0

Key design decisions

• Contextual retrieval kept ON — uses openai/gpt-oss-20b via Euron API, zero Groq TPM impact at query time. Ingest-time only.
• HyDE + MQ disabled, not removed — toggles in config.yaml; re-enable instantly when on paid tier
• v1.1.0 as live proxy in eval dashboard — no eval run exists for "CTX-only, no HyDE, no MQ" config. v1.1.0 (recall=0.721) is an overestimate; actual live recall ≈ 0.55–0.65 given contextual index without HyDE query expansion
• Upgrade path documented in eval dashboard — v1.3.0 blocked_by note explains exactly what to fix

Groq call budget (current vs best)

Config	Calls/query	TPM risk
Current (CTX only)	1–2	Safe
v1.3.0 (HyDE+MQ+CTX)	4	429 on free tier

Upgrade path to v1.3.0

1. Switch to paid Groq tier (or find higher-TPM free provider)
2. Set hyde_enabled: true + multi_query_enabled: true in config.yaml
3. Push → HF Spaces rebuilds → run scripts/run_eval_versioned.py --version v1.3.1 --tag "Violet" --n 50 to confirm metrics

---

Current State Snapshot

Retrieval:    Hybrid BM25 (0.3) + ChromaDB dense (0.7) → RRF → TinyBERT rerank top-10→5
LLM:          Groq llama-3.3-70b-versatile
Embeddings:   Euron API text-embedding-3-small (sequential, ~1.7s/chunk — bottleneck)
Chunking:     RecursiveCharacterTextSplitter 500-char, overlap 50
Memory:       ConversationBufferWindowMemory k=10
Web search:   Tavily advanced, 800-char truncation, max 2 results — MANDATORY (always on)
HyDE:         DISABLED (hyde_enabled=false). Best measured: +21pp recall but costs 1 Groq call/query.
              Re-enable when on paid Groq tier or higher-TPM provider.
Multi-Query:  DISABLED (multi_query_enabled=false). Costs 1 Groq call/query — free tier cannot sustain.
              Re-enable with HyDE together (v1.3.0 config) on paid tier.
Contextual:   ENABLED (contextual_retrieval.enabled=true). Uses Euron model (openai/gpt-oss-20b) —
              zero Groq TPM impact. Two-phase upload: sync non-contextual embed (<3s queryable),
              BackgroundTask replaces with contextual chunks. max_concurrent=3, max_chunks=50 gate.
Semantic:     DISABLED (semantic_enabled=false). Ablation showed recall +9.3pp but P@5 -27.3pp and 5× latency.
              Rejected — v1.3.0 (HyDE+MQ+CTX) is the confirmed best stack when TPM allows.
Groq calls/query (current): 1–2 (condense_question if follow-up + answer). Safe under 6000 TPM free tier.
Groq calls/query (v1.3.0):  4 (condense + HyDE + Multi-Query + answer) → 429 storms on free tier.
Eval:         Separate eval-dashboard/ static site → https://askprism-eval.vercel.app/
              v1.1.0 marked LIVE (closest proxy). v1.3.0 and v1.4.0 show amber "not in production" warning.
              Best measured: v1.3.0 recall=0.768, P@5=0.984, p50=2610ms
              Versioning: MAJOR.MINOR.PATCH — name changes on MAJOR only (v1.x.x=Violet, v2.x.x=Indigo)
Citation:     [N] markers in LLM answers → clickable <sup> → CitationPopover (fixed-position, viewport-aware).
              Shows full chunk text, source name, page, rerank score. PDF: "Open page N →" link via GET /api/files/{filename}.
              Web: "Open source →". Toggle, click-away, above/below flip at 60% viewport height.
              SourceExpander: full content shown (200-char truncation removed).
Frontend:     Violet v1.3 badge in sidebar footer. Maintenance banner config-driven (frontend/src/config.js).
              MAINTENANCE_MODE=false — app is live as of 2026-07-05.
Filter:       Sidebar doc chips toggleable. Selected: indigo ring. Badge above chat input shows scoped docs + clear ×.
              POST /api/chat accepts filter_docs: string[] | null. Empty = no filter. Resets on workspace switch.
              Backend: get_retriever_filtered() creates one-off HybridRetriever; singleton cache untouched.
              ChromaDB where={"source": {"$in": filter_docs}}. BM25 filters candidate pool, scores with full-corpus IDF.
Workspaces:   Per-workspace ChromaDB collection, singleton retriever cache
Infra:        HF Spaces CPU Basic (backend, 16GB RAM, ephemeral FS — re-upload required after cold start) +
              https://askprism.vercel.app/ (frontend) + https://askprism-eval.vercel.app/ (eval)
              Backend URL: https://benroshan-prism.hf.space
Known limits: Euron embed ~5s/chunk sequential — 30 chunks = ~150s total contextualization in background.
              HF Spaces ephemeral FS: chroma_db lost on cold start. Fix: mount HF persistent storage bucket.
              HyDE + MQ disabled for free-tier stability. Best stack (v1.3.0) needs paid Groq or alt provider.
Observability: LangSmith traces all LLM + retrieval calls (optional, env var)
Streaming:    POST /api/chat returns SSE stream. token events per LLM chunk, done event with
              sources + retrieval_method. Frontend streams tokens into pre-placed assistant
              bubble. Bouncing dots while condense+search runs, blinking cursor during generation.

---

Stage 12 — HF Spaces Migration (2026-06-22)

What was wrong

Render free tier (512MB RAM) caused repeated OOM crashes under contextual retrieval:

• Base RSS after upload = 524MB (over the 512MB limit)
• gc.collect() had no effect — ChromaDB HNSW index + torch runtime held by native allocators, not Python heap
• Contextual refresh (3 async Groq coroutines) + simultaneous chat (Tavily + LLM + CrossEncoder) = peak exceeded 512MB
• Workarounds (RSS guard skipping contextual retrieval, web search suppression during refresh) negated the +18% recall improvement

What we built

File	Change
Dockerfile	Port 8000 → 7860 (HF convention). Add useradd -m -u 1000 user + chown -R user /app (HF runs containers as UID 1000). Set HF_HOME=/app/.cache/huggingface BEFORE pre-download so user 1000 owns cached weights. Set HF_HUB_OFFLINE=1 AFTER download.
README.md	Added HF Spaces frontmatter (sdk: docker, app_port: 7860). Updated deploy instructions.
server/routes/chat.py	Removed is_contextualizing web search suppression guard (Render-specific).
server/routes/upload.py	Removed RSS > 460MB contextual retrieval skip guard (Render-specific).
docs/, decisions.md	Render → HF Spaces across all infra references.

Key discoveries

• HF_HUB_OFFLINE must be set AFTER the pre-download RUN step — setting it before blocks the download itself
• Docker build runs pre-download as root by default; must USER 1000 first then set HF_HOME under /app so runtime user 1000 can read the cached weights
• HF Spaces free CPU Basic: 2 vCPUs, 16GB RAM — resolves all Render OOM issues permanently
• Contextual retrieval now runs fully in production (was silently skipped by RSS guard on Render)

---

Stage 13 — Async Embed Upload (2026-06-23)

What was wrong

embed_and_store() blocked POST /api/upload for ~150s (30 chunks × ~5s/chunk via Euron API). User saw spinner, could not query, could not cancel. Upload timeout was 300s.

What we built

File	Change
server/main.py	app.state.upload_jobs = {} initialized in lifespan
server/routes/upload.py	POST /api/upload returns 202 + job_id in <1s. Parse+chunk sync; embed+contextual in _embed_and_contextualize_bg() BackgroundTask. New GET /api/upload/status/{job_id} endpoint.
frontend/src/api.js	Added getUploadStatus(jobId); reduced uploadFiles timeout 300s → 30s
frontend/src/components/FileUpload.jsx	Polls status every 2s; shows stage label under spinner; fires callbacks on ready. Defensive `

Key discoveries

• Old Vercel frontend receiving new 202 response before redeploy → data.documents undefined → React crash. Fix: defensive docs?.documents || [] guard.
• Groq TPM 429s at max_concurrent=3 still hit (~5/30 chunks fall back to original text) — some chunks are larger than average. Retry logic handles gracefully.
• Briefing fails with JSON parse error (pre-existing bug in generate_briefing — separate fix).

---

Stage 14 — Briefing Fix + HyDE Re-eval (2026-06-24)

What was wrong

• generate_briefing() crashed with JSONDecodeError when Groq LLM returned control characters (ASCII 0x00–0x1f) or Python dict syntax (single quotes) instead of valid JSON.
• Old eval runs (v1.0.0–v1.4.0) accumulated across multiple sessions; stale runs cluttered the dashboard.
• HyDE recall measurement from prior session (v1.1.0_20260619, recall=0.545) was based on 50-sample run that hit Groq 429s mid-run — partial results, unreliable numbers.

What we built

File	Change
server/briefing.py	Strip control chars [\x00-\x08\x0b\x0c\x0e-\x1f] before JSON parse. Fall back to ast.literal_eval() on JSONDecodeError to handle Python dict syntax from LLM. Added import ast.
config.yaml	hyde_enabled: true, multi_query_enabled: true, contextual_retrieval.enabled: false (contextual off — 429s at 30-chunk scale even with Semaphore(3))
eval-dashboard/public/data/runs/	Deleted stale runs (v1.0.0_20260618, v1.1.0_20260619, v1.2.0_20260619, v1.3.0_20260620, v1.3.0_20260623, v1.4.0_20260623). Added v1.1.0_20260624.json — fresh HyDE-only run.
eval-dashboard/public/data/index.json	Updated to single clean run registry.

HyDE re-eval results — v1.1.0_20260624 (18 samples, hyde=true, multi_query=false)

Metric	v1.0.0 baseline	v1.1.0 HyDE	Delta
answer_correctness	0.820	0.750	-0.070
answer_relevancy	0.620	0.845	+0.225
context_recall	0.510	0.721	+0.211
precision_at_5	0.890	0.911	+0.021
latency p50	2029ms	4018ms	+2×

Key discoveries

• HyDE gives +21pp recall (0.51→0.72) on this 18-sample run — much larger than previously measured (+3.5pp on 50 samples with 429s). Smaller sample set; repeat at 50 samples to confirm.
• answer_correctness flat at 0.75 for all 18 samples — 8B judge giving uniform score, not differentiating. May indicate judge calibration issue, not actual correctness plateau.
• Latency 2× (2029ms→4018ms) — HyDE adds one Groq call per query for hypothetical expansion.
• Briefing fix unblocks document upload → briefing flow end-to-end.

---

Roadmap — Retrieval & Answer Quality

Phase 1 — Quick wins (no infra change, measurable RAGAS lift)

HyDE (Hypothetical Document Embeddings) ✅ Done (Stage 7, commit 8945b43)

• Implemented in server/retriever.py. Toggle: config.yaml hyde_enabled (default: false).
• Enable + re-run eval to measure context_recall lift vs v2.0 baseline (0.70).

Multi-Query Retrieval ✅ Done (Stage 9, 2026-06-19)

• Implemented in server/retriever.py. Toggle: config.yaml multi_query_enabled (default: false).
• Enable + run scripts/run_eval_versioned.py --version v1.1.0 --tag "Indigo" --n 50 to measure context_recall lift vs 0.51.

---

Phase 2 — Ingest pipeline (requires re-ingest of all docs)

Contextual Retrieval ✅ Done + Shipped to Production (Stage 10+11, 2026-06-20)

• contextualize_chunks_async() + BackgroundTask in routes/upload.py. Two-phase: sync non-contextual embed (queryable <3s) → background contextual replacement.
• v1.3.0 results: recall 0.510→0.601 (+18%), P@5 0.890→0.956. Latency 2× at query time (longer chunks → more LLM tokens).
• max_concurrent=3 in config.yaml — safe under Groq 6000 TPM limit.

Semantic Chunking

• Problem: Fixed 200-char splits cut mid-sentence, mid-table, mid-list. Embedding a truncated sentence returns a weak vector.
• How: Replace RecursiveCharacterTextSplitter with LangChain's SemanticChunker — splits at sentence boundaries where cosine similarity between adjacent sentences drops below a threshold (topic shift).
• Effort: Medium. Config change in ingest.py + re-ingest. Tune breakpoint_threshold_type.
• Expected lift: Fewer nonsensical chunks in top-5. Most noticeable on regulatory PDFs with section headers and numbered lists.

---

Phase 3 — UX + trust

Streaming Responses

• Problem: User submits question → 8–15s wait → full answer appears. Feels broken even on fast hardware.
• How: Backend: chain.astream_events() → StreamingResponse yielding SSE tokens. Frontend: EventSource or fetch + ReadableStream — append tokens as they arrive. Faithfulness scoring runs as background task after full answer assembled.
• Effort: High — both backend and frontend change. ConversationalRetrievalChain supports astream_events() in LangChain ≥0.2.
• Impact: Perceived latency drops from 10s to ~1s. Single biggest UX improvement.

Citation Highlighting ✅ Done (Stage 16, 2026-06-26)

• [N] markers clickable → CitationPopover with full chunk text, page badge, rerank score. PDF "Open page N →" link. Zero new npm deps.
• Works for all source types: PDF, URL, TXT, CSV. No PDF viewer library needed — page link uses browser's built-in viewer.

---

Phase 4 — Differentiation

Metadata Filtering

• Problem: Multi-workspace isolates by collection, but within a workspace (10 docs across 5 years) no way to scope retrieval to year=2024 or doc_type=rbi_circular.
• How: Tag chunks with {source_type, year, doc_name} at ingest. Pass optional filter param in /api/chat request. ChromaDB where clause on dense retrieval; BM25 pre-filters corpus to matching chunk IDs.
• Impact: Precision boost on time-scoped or source-scoped queries.

Document Comparison Mode

• Problem: No way to ask "What changed between RBI circular 2023 and 2024?"
• How: Frontend sends two doc IDs + comparison query. Backend retrieves relevant chunks from each collection separately, synthesises a structured diff answer.
• Impact: Killer fintech feature. Unique demo moment. Differentiates from generic RAG.

Agentic Mode (LangGraph)

• Problem: Single-shot RAG cannot handle multi-step reasoning: retrieve → compute → web search → synthesise.
• How: Replace ConversationalRetrievalChain with a LangGraph graph. Nodes: retriever, web_search, calculator, synthesiser. LLM decides which tool to call.
• Impact: Separates Prism from basic RAG — becomes a research agent. Strongest interview story.

---

Roadmap Priority Matrix

HIGH impact × LOW effort  → Build first
  HyDE
  Multi-query retrieval
  Metadata filtering

HIGH impact × MEDIUM effort → Build second
  Contextual retrieval (+ re-ingest)
  Semantic chunking (+ re-ingest)
  Streaming responses

HIGH impact × HIGH effort → Build last
  Citation highlighting
  Document comparison
  Agentic mode (LangGraph)

---

Interview Story Arc

v1  → Dense-only retrieval. No eval. No baseline.
v2  → Hybrid BM25+dense, cross-encoder rerank. Measured with RAGAS.
     → faithfulness=1.0, answer_relevancy=0.90 on 20-pair eval set.
+HyDE → context_recall 0.51→0.72 (+21pp). Hypothetical answer embedding closes vocabulary gap.
+Contextual → context_recall 0.60 (+18% vs baseline). Ingest-time LLM chunk augmentation.
+Agentic  → Multi-step reasoning. Not RAG anymore — research agent.

Each step has a metric. That is the complete RAG engineering narrative for MNC DS interviews.