docs/architecture.md

Architecture — Prism

Problem

Fintech analysts spend hours manually reading RBI circulars, NPCI reports, and earnings transcripts. Standard dense-only RAG fails silently and misses exact keyword matches in regulatory text (section numbers, policy codes).

Architecture overview

Upload → chunk (500-char) → optional contextual augmentation (LLM prepends 2-sentence context per chunk) → embed via Euron API → store in ChromaDB + BM25 index. Query → optional HyDE expand → optional multi-query expand → hybrid retrieval (ChromaDB dense + BM25 sparse) → weighted RRF fusion → cross-encoder rerank (top-10 → top-5) → Tavily web search → LLM answer via SSE stream → LangSmith trace. Multi-workspace: each workspace has its own ChromaDB collection; vectorstore + retriever cached per workspace. Upload returns 202 immediately; embed + contextualize run as background task polled via GET /api/upload/status/{job_id}. Eval runs offline via scripts/run_eval_versioned.py; results served by a separate eval-dashboard/ static site.

Component breakdown

Component	Technology	Purpose
Vector store	ChromaDB (persistent)	Dense embedding storage and retrieval
Sparse retrieval	rank_bm25 (BM25Okapi)	Keyword-match retrieval for regulatory text
Hybrid fusion	Weighted RRF (dense 0.7 + sparse 0.3)	Merge dense + sparse result lists
Reranker	cross-encoder/ms-marco-TinyBERT-L-2-v2	Re-score top-10 → return top-5 (~17MB; chosen over MiniLM-L-6-v2 ~85MB for lower memory footprint)
Embeddings	Euron API (text-embedding-3-small)	API-based; Groq has no embeddings endpoint.
LLM	Groq (llama-3.3-70b-versatile) via langchain-groq	Fast open-weight inference; OpenAI-compatible
Chunking	RecursiveCharacterTextSplitter (500-char, overlap 50)	Single-pass split; semantic chunking available but disabled (ablation: +9.3pp recall, −27.3pp P@5, 5× latency)
Contextual retrieval	LLM (openai/gpt-oss-20b) at ingest time	Prepends 2-sentence situating context per chunk before embedding; +18% recall. Async via Semaphore(3).
HyDE	Groq LLM generates hypothetical answer before dense search	Closes question/answer vector space gap; +21pp recall. ON by default.
Multi-Query	Groq LLM generates 3 query phrasings	Widens candidate pool before RRF; best-rank dedup. ON by default.
Web search	Tavily (advanced, max 2 results, 800-char truncation)	Mandatory on every query; grounded answers for open-domain questions
Memory	ConversationBufferWindowMemory (k=10)	Last 10 conversation turns
Chain	stream_query_with_web() — direct LLM call bypassing ConversationalRetrievalChain	Bypasses chain to prevent condensation step stripping web context; yields SSE token stream
Streaming	FastAPI StreamingResponse + SSE	Token events during generation; done event with sources + retrieval_method
Async upload	202 + job_id; background _embed_and_contextualize_bg	User queryable in <3s; embed + contextualize run in background; poll GET /api/upload/status/{job_id}
File serving	GET /api/files/{filename} → FileResponse	Serves uploaded docs for citation popover "Open page N →" links; path traversal blocked via is_relative_to()
Citation	[N] markers in LLM answer → CitationPopover (React)	Clickable superscripts show full chunk text, page, rerank score; PDF page link via file serving route
Workspace	ChromaDB collection per workspace	Isolated document sets; switcher in frontend sidebar
Retriever cache	Module-level dict keyed by workspace	Singleton vectorstore+retriever per workspace; invalidate on ingest
Eval	Separate eval-dashboard/ Vite+React static site	Reads versioned JSON run files; metrics: answer_correctness, answer_relevancy, context_recall, precision@5, latency p50/p95/p99
Eval script	scripts/run_eval_versioned.py	Runs offline against 50-pair ground truth; writes versioned JSON + updates index.json
Eval ground truth	data/ground_truth/eval_pairs.json (50 pairs)	Multi-hop, comparative, negative, numeric, edge-case questions with reference answers
Observability	LangSmith	Traces all LLM + retrieval calls via LANGCHAIN_TRACING_V2=true
Document parsing	LlamaParse (primary), pypdf (fallback)	PDF extraction
Backend	FastAPI + Uvicorn	REST API
Frontend	React 19 + Vite + Tailwind CSS v4	Chat / Upload tabs
Deployment	HF Spaces (Docker backend, 16GB RAM) + Vercel (frontend) + Vercel (eval-dashboard)	Production

Data flow

Ingestion

1. POST /api/upload → parse PDFs/TXT/CSV → returns 202 + job_id immediately
2. Background task _embed_and_contextualize_bg starts: a. RecursiveCharacterTextSplitter (500-char, overlap 50) → chunks b. embed_and_store(): Euron API embeds chunks → store in ChromaDB (workspace collection) c. Rebuild BM25 index from new corpus d. generate_briefing(): LLM summarises first 6 chunks → 5 bullets + 3 suggested questions e. If contextual_retrieval.enabled: contextualize_chunks_async() — Groq 8B prepends 2-sentence context per chunk (Semaphore(3) → max 3000 TPM burst); replace old chunk IDs in ChromaDB with contextual versions
3. Frontend polls GET /api/upload/status/{job_id} every 2s → stages: embedding → contextualizing → ready

Query

1. POST /api/chat receives question → returns StreamingResponse (text/event-stream)
2. condense_question(): LLM rewrites follow-up question using chat history → standalone query for search
3. Tavily web search (advanced, max 2 results, 800-char/result) runs in parallel with retrieval
4. HybridRetriever._get_relevant_documents(): a. If multi_query_enabled: LLM generates 3 phrasings; retrieve for each; pool + best-rank dedup b. If hyde_enabled: LLM generates hypothetical answer; embed fake answer for dense search c. dense_retrieve: ChromaDB top-10 per query phrasing (cosine similarity) d. sparse_retrieve: BM25 top-10 per query phrasing e. reciprocal_rank_fusion: merge → deduplicate → weighted RRF (dense 0.7, sparse 0.3) f. Reranker.rerank: cross-encoder scores all candidates jointly → top-5 chunks
5. stream_query_with_web(): direct LLM call with RAG chunks + Tavily results + chat history → streams tokens
6. SSE events: {"type": "token", "content": "..."} per chunk; {"type": "done", "sources": [...], "retrieval_method": "..."} at end
7. Sources include per-chunk scores: similarity, bm25, rrf, rerank

Key design decisions

• API embeddings over local: Euron API embeddings ~0MB RAM; Groq has no embeddings endpoint so Euron is retained for embeddings.
• RecursiveCharacterTextSplitter 500-char: single-pass chunking; semantic chunking ablation (v1.4.0) showed +9.3pp recall but −27.3pp P@5 and 5× latency — rejected.
• Cross-encoder reranker: bi-encoder (ChromaDB) is fast but approximate; cross-encoder is slower but more accurate on top-10 pool.
• BM25 weight 0.3: regulatory text has exact keyword matches (section numbers); sparse retrieval catches what dense misses.
• HyDE ON by default: hypothetical answer embedding closes question/answer vector space gap. Measured +21pp recall (0.51→0.72, v1.1.0). Adds one Groq call per query (~200ms latency).
• Multi-Query ON by default: 3 phrasings widen candidate pool before RRF. Best-rank dedup ensures highest-confidence rank carried into fusion. Adds one Groq call per query.
• Contextual retrieval: LLM prepends 2-sentence situating context to each chunk at ingest before embedding. +18% recall (v1.3.0). asyncio.Semaphore(3) caps parallel Groq calls at 3000 TPM — safe under 6000 TPM free limit.
• Mandatory web search: always-on Tavily + RAG prevents hallucination on open-domain queries. Toggle removed after opt-in caused wrong corpus docs to be cited with high faithfulness score.
• Streaming SSE: stream_query_with_web() yields token events via FastAPI StreamingResponse. Bypasses ConversationalRetrievalChain condensation step (which strips web context). Direct LLM call with full context.
• Async upload (202 pattern): parse+chunk synchronous (<1s) → return 202 + job_id → embed+contextualize in BackgroundTask. Frontend polls GET /api/upload/status/{job_id}. User queryable in <3s without waiting ~40s for contextualization.
• Citation popover: [N] markers in LLM output → clickable <sup> → CitationPopover shows full chunk text, source, page, rerank score. PDF sources get "Open page N →" link via GET /api/files/{filename}. Path.is_relative_to() guards against traversal.
• TinyBERT-L-2-v2 reranker: MiniLM-L-6-v2 (85MB) vs TinyBERT-L-2-v2 (17MB) — same ranking quality at demo corpus scale with lower memory footprint.
• Singleton vectorstore/retriever cache: each workspace caches its Chroma vectorstore + HybridRetriever in a module-level dict. Without cache, every chat request created a new Chroma instance (full embedding reload). Cache is invalidated after ingest.
• Multi-workspace isolation: each workspace maps to one ChromaDB collection. Frontend workspace switcher passes workspace_id on every request; backend resolves the correct collection before retrieval.
• URL size guard: url_loader.py enforces a max content size before embedding URL content, preventing memory spikes from large external pages.
• Eval dashboard separate site: eval runs offline via scripts/run_eval_versioned.py; results versioned as JSON. No live eval endpoint on prod. Per-message faithfulness badge removed — eval moved to dedicated dashboard.
• answer_correctness over faithfulness: faithfulness (LLM judge vs retrieved chunks) is circular — inflates when eval pairs are corpus-aligned. answer_correctness (LLM judge vs ground_truth reference) is an independent signal.
• HF Spaces Docker (UID 1000): model weights baked into image under HF_HOME=/app/.cache/huggingface as user 1000 at build time; HF_HUB_OFFLINE=1 set after download to block runtime network calls.

Known limitations

• BM25 index rebuilt in memory on each startup (not persisted to disk)
• HF Spaces free tier: ephemeral filesystem — chroma_db lost on cold start (re-upload required)
• Euron embedding API sequential: ~1.7s/chunk — 30 chunks = ~52s in background (user unblocked via 202, but contextual refresh still takes ~15s with Semaphore(3))
• Groq free tier: contextual retrieval 429s frequent at large doc scale (>30 chunks) even at max_concurrent=3; some chunks fall back to non-contextual text
• anchorRect in CitationPopover stale after page scroll (acceptable for demo)

Future improvements

• Persist BM25 index to disk (pickle) — eliminates ~1s rebuild on startup
• Mount HF persistent storage bucket — eliminate chroma_db loss on cold start
• Metadata filtering — shipped Stage 17 (sidebar doc chips → filter_docs on /api/chat → ChromaDB where + BM25 pool filter)
• Document comparison mode: retrieve from two collections, synthesise structured diff answer
• Agentic mode (LangGraph): replace ConversationalRetrievalChain with graph — nodes for retrieval, web search, calculator, synthesiser