docs: update architecture.md to reflect current shipped state (v1.3.0 stack)

docs/architecture.md

@@ -4,7 +4,7 @@
 Fintech analysts spend hours manually reading RBI circulars, NPCI reports, and earnings transcripts. Standard dense-only RAG fails silently and misses exact keyword matches in regulatory text (section numbers, policy codes).
 
 ## Architecture overview
-Query → hybrid retrieval (ChromaDB dense + BM25 sparse) → weighted RRF fusion → cross-encoder rerank (top-10 → top-5) → LLM answer → LangSmith trace. ParentDocumentRetriever stores 200-char child chunks for retrieval but returns 800-char parent chunks to LLM. Multi-workspace: each workspace has its own ChromaDB collection; vectorstore + retriever cached per workspace to prevent OOM on repeated queries. Eval runs offline via `scripts/run_eval_versioned.py`; results served by a separate `eval-dashboard/` static site.
+Upload → chunk (500-char) → optional contextual augmentation (LLM prepends 2-sentence context per chunk) → embed via Euron API → store in ChromaDB + BM25 index. Query → optional HyDE expand → optional multi-query expand → hybrid retrieval (ChromaDB dense + BM25 sparse) → weighted RRF fusion → cross-encoder rerank (top-10 → top-5) → Tavily web search → LLM answer via SSE stream → LangSmith trace. Multi-workspace: each workspace has its own ChromaDB collection; vectorstore + retriever cached per workspace. Upload returns 202 immediately; embed + contextualize run as background task polled via `GET /api/upload/status/{job_id}`. Eval runs offline via `scripts/run_eval_versioned.py`; results served by a separate `eval-dashboard/` static site.
 
 ## Component breakdown
 
@@ -16,9 +16,17 @@ Query → hybrid retrieval (ChromaDB dense + BM25 sparse) → weighted RRF fusio
 | Reranker | cross-encoder/ms-marco-TinyBERT-L-2-v2 | Re-score top-10 → return top-5 (~17MB; chosen over MiniLM-L-6-v2 ~85MB for lower memory footprint) |
 | Embeddings | Euron API (text-embedding-3-small) | API-based; Groq has no embeddings endpoint. |
 | LLM | Groq (llama-3.3-70b-versatile) via langchain-groq | Fast open-weight inference; OpenAI-compatible |
-| Chunking | LangChain ParentDocumentRetriever | Child 200-char indexed, parent 800-char sent to LLM |
+| Chunking | RecursiveCharacterTextSplitter (500-char, overlap 50) | Single-pass split; semantic chunking available but disabled (ablation: +9.3pp recall, −27.3pp P@5, 5× latency) |
+| Contextual retrieval | LLM (llama-3.1-8b-instant) at ingest time | Prepends 2-sentence situating context per chunk before embedding; +18% recall. Async via Semaphore(3). |
+| HyDE | Groq LLM generates hypothetical answer before dense search | Closes question/answer vector space gap; +21pp recall. ON by default. |
+| Multi-Query | Groq LLM generates 3 query phrasings | Widens candidate pool before RRF; best-rank dedup. ON by default. |
+| Web search | Tavily (advanced, max 2 results, 800-char truncation) | Mandatory on every query; grounded answers for open-domain questions |
 | Memory | ConversationBufferWindowMemory (k=10) | Last 10 conversation turns |
-| Chain | ConversationalRetrievalChain | LangChain orchestration |
+| Chain | `stream_query_with_web()` — direct LLM call bypassing ConversationalRetrievalChain | Bypasses chain to prevent condensation step stripping web context; yields SSE token stream |
+| Streaming | FastAPI `StreamingResponse` + SSE | Token events during generation; `done` event with sources + retrieval_method |
+| Async upload | 202 + job_id; background `_embed_and_contextualize_bg` | User queryable in <3s; embed + contextualize run in background; poll `GET /api/upload/status/{job_id}` |
+| File serving | `GET /api/files/{filename}` → FileResponse | Serves uploaded docs for citation popover "Open page N →" links; path traversal blocked via `is_relative_to()` |
+| Citation | `[N]` markers in LLM answer → CitationPopover (React) | Clickable superscripts show full chunk text, page, rerank score; PDF page link via file serving route |
 | Workspace | ChromaDB collection per workspace | Isolated document sets; switcher in frontend sidebar |
 | Retriever cache | Module-level dict keyed by workspace | Singleton vectorstore+retriever per workspace; invalidate on ingest |
 | Eval | Separate `eval-dashboard/` Vite+React static site | Reads versioned JSON run files; metrics: answer_correctness, answer_relevancy, context_recall, precision@5, latency p50/p95/p99 |
@@ -33,42 +41,60 @@ Query → hybrid retrieval (ChromaDB dense + BM25 sparse) → weighted RRF fusio
 ## Data flow
 
 ### Ingestion
-1. `run_ingest.py` or `POST /api/upload` → load PDFs/TXT/CSV
-2. `ParentDocumentRetriever`: split into 800-char parent + 200-char child chunks
-3. Embed child chunks via Euron API (text-embedding-3-small) → store in ChromaDB
-4. Store parent chunks in `InMemoryStore`
-5. Build BM25 index over child chunk corpus
+1. `POST /api/upload` → parse PDFs/TXT/CSV → returns 202 + `job_id` immediately
+2. Background task `_embed_and_contextualize_bg` starts:
+   a. `RecursiveCharacterTextSplitter` (500-char, overlap 50) → chunks
+   b. `embed_and_store()`: Euron API embeds chunks → store in ChromaDB (workspace collection)
+   c. Rebuild BM25 index from new corpus
+   d. `generate_briefing()`: LLM summarises first 6 chunks → 5 bullets + 3 suggested questions
+   e. If `contextual_retrieval.enabled`: `contextualize_chunks_async()` — Groq 8B prepends 2-sentence context per chunk (Semaphore(3) → max 3000 TPM burst); replace old chunk IDs in ChromaDB with contextual versions
+3. Frontend polls `GET /api/upload/status/{job_id}` every 2s → stages: embedding → contextualizing → ready
 
 ### Query
-1. `POST /api/chat` receives question
-2. `dense_retrieve`: ChromaDB top-10 by cosine similarity (workspace-specific collection)
-3. `sparse_retrieve`: BM25 top-10 by keyword score
-4. `reciprocal_rank_fusion`: merge → deduplicate → RRF score
-5. `Reranker.rerank`: cross-encoder score → return top-5 parent chunks
-6. `ConversationalRetrievalChain`: LLM answers with context + memory
-7. Response includes: answer, sources (with scores), retrieval_method
+1. `POST /api/chat` receives question → returns `StreamingResponse` (text/event-stream)
+2. `condense_question()`: LLM rewrites follow-up question using chat history → standalone query for search
+3. Tavily web search (advanced, max 2 results, 800-char/result) runs in parallel with retrieval
+4. `HybridRetriever._get_relevant_documents()`:
+   a. If `multi_query_enabled`: LLM generates 3 phrasings; retrieve for each; pool + best-rank dedup
+   b. If `hyde_enabled`: LLM generates hypothetical answer; embed fake answer for dense search
+   c. `dense_retrieve`: ChromaDB top-10 per query phrasing (cosine similarity)
+   d. `sparse_retrieve`: BM25 top-10 per query phrasing
+   e. `reciprocal_rank_fusion`: merge → deduplicate → weighted RRF (dense 0.7, sparse 0.3)
+   f. `Reranker.rerank`: cross-encoder scores all candidates jointly → top-5 chunks
+5. `stream_query_with_web()`: direct LLM call with RAG chunks + Tavily results + chat history → streams tokens
+6. SSE events: `{"type": "token", "content": "..."}` per chunk; `{"type": "done", "sources": [...], "retrieval_method": "..."}` at end
+7. Sources include per-chunk scores: similarity, bm25, rrf, rerank
 
 ## Key design decisions
 - **API embeddings over local**: Euron API embeddings ~0MB RAM; Groq has no embeddings endpoint so Euron is retained for embeddings.
-- **ParentDocumentRetriever**: small chunks improve retrieval precision; large parent chunks improve answer faithfulness
-- **Cross-encoder reranker**: bi-encoder (ChromaDB) is fast but approximate; cross-encoder is slower but more accurate on top-20 pool
-- **BM25 weight 0.3**: regulatory text has exact keyword matches (section numbers); sparse retrieval catches what dense misses
-- **RAGAS benchmark pre-computed locally**: `nest_asyncio` cannot patch `uvloop` (used by uvicorn on Linux), making live RAGAS eval impossible on prod. Run `scripts/run_ragas_local.py` locally, commit JSON results, Vercel builds dashboard from file.
+- **RecursiveCharacterTextSplitter 500-char**: single-pass chunking; semantic chunking ablation (v1.4.0) showed +9.3pp recall but −27.3pp P@5 and 5× latency — rejected.
+- **Cross-encoder reranker**: bi-encoder (ChromaDB) is fast but approximate; cross-encoder is slower but more accurate on top-10 pool.
+- **BM25 weight 0.3**: regulatory text has exact keyword matches (section numbers); sparse retrieval catches what dense misses.
+- **HyDE ON by default**: hypothetical answer embedding closes question/answer vector space gap. Measured +21pp recall (0.51→0.72, v1.1.0). Adds one Groq call per query (~200ms latency).
+- **Multi-Query ON by default**: 3 phrasings widen candidate pool before RRF. Best-rank dedup ensures highest-confidence rank carried into fusion. Adds one Groq call per query.
+- **Contextual retrieval**: LLM prepends 2-sentence situating context to each chunk at ingest before embedding. +18% recall (v1.3.0). `asyncio.Semaphore(3)` caps parallel Groq calls at 3000 TPM — safe under 6000 TPM free limit.
+- **Mandatory web search**: always-on Tavily + RAG prevents hallucination on open-domain queries. Toggle removed after opt-in caused wrong corpus docs to be cited with high faithfulness score.
+- **Streaming SSE**: `stream_query_with_web()` yields token events via FastAPI `StreamingResponse`. Bypasses `ConversationalRetrievalChain` condensation step (which strips web context). Direct LLM call with full context.
+- **Async upload (202 pattern)**: parse+chunk synchronous (<1s) → return 202 + job_id → embed+contextualize in `BackgroundTask`. Frontend polls `GET /api/upload/status/{job_id}`. User queryable in <3s without waiting ~40s for contextualization.
+- **Citation popover**: `[N]` markers in LLM output → clickable `<sup>` → `CitationPopover` shows full chunk text, source, page, rerank score. PDF sources get "Open page N →" link via `GET /api/files/{filename}`. `Path.is_relative_to()` guards against traversal.
 - **TinyBERT-L-2-v2 reranker**: MiniLM-L-6-v2 (~85MB) vs TinyBERT-L-2-v2 (~17MB) — same ranking quality at demo corpus scale with lower memory footprint.
 - **Singleton vectorstore/retriever cache**: each workspace caches its Chroma vectorstore + HybridRetriever in a module-level dict. Without cache, every chat request created a new Chroma instance (full embedding reload). Cache is invalidated after ingest.
 - **Multi-workspace isolation**: each workspace maps to one ChromaDB collection. Frontend workspace switcher passes `workspace_id` on every request; backend resolves the correct collection before retrieval.
 - **URL size guard**: `url_loader.py` enforces a max content size before embedding URL content, preventing memory spikes from large external pages.
-- **Eval dashboard separate site**: eval runs offline, results versioned as JSON. Separates eval tooling from user-facing app; no live eval endpoint on prod backend. Per-message faithfulness badge removed from UI — moved to dedicated dashboard.
+- **Eval dashboard separate site**: eval runs offline via `scripts/run_eval_versioned.py`; results versioned as JSON. No live eval endpoint on prod. Per-message faithfulness badge removed — eval moved to dedicated dashboard.
 - **answer_correctness over faithfulness**: faithfulness (LLM judge vs retrieved chunks) is circular — inflates when eval pairs are corpus-aligned. answer_correctness (LLM judge vs ground_truth reference) is an independent signal.
 - **HF Spaces Docker (UID 1000)**: model weights baked into image under `HF_HOME=/app/.cache/huggingface` as user 1000 at build time; `HF_HUB_OFFLINE=1` set after download to block runtime network calls.
 
 ## Known limitations
-- InMemoryStore for parent chunks: does not survive server restart (re-ingest required)
 - BM25 index rebuilt in memory on each startup (not persisted to disk)
 - HF Spaces free tier: ephemeral filesystem — chroma_db lost on cold start (re-upload required)
-- Euron embedding API sequential: ~5s/chunk — 30 chunks = ~150s blocking upload
+- Euron embedding API sequential: ~1.7s/chunk — 30 chunks = ~52s in background (user unblocked via 202, but contextual refresh still takes ~15s with Semaphore(3))
+- Groq free tier: contextual retrieval 429s frequent at large doc scale (>30 chunks) even at max_concurrent=3; some chunks fall back to non-contextual text
+- `anchorRect` in CitationPopover stale after page scroll (acceptable for demo)
 
 ## Future improvements
-- Persist BM25 index to disk (pickle)
-- Move Euron embedding to background task (return 202, poll for ready)
-- Streaming responses from LLM
+- Persist BM25 index to disk (pickle) — eliminates ~1s rebuild on startup
+- Mount HF persistent storage bucket — eliminate chroma_db loss on cold start
+- Metadata filtering: tag chunks with `{source_type, year, doc_name}` at ingest; pass `filter` param in `/api/chat` for scoped retrieval
+- Document comparison mode: retrieve from two collections, synthesise structured diff answer
+- Agentic mode (LangGraph): replace ConversationalRetrievalChain with graph — nodes for retrieval, web search, calculator, synthesiser