docs/decisions.md

Technical Decisions — Prism

Decision log

Date	Decision	Rationale	Status
2026-05	Euron API for embeddings (not local sentence-transformers)	Groq has no embeddings endpoint; Euron API keeps RAM footprint near zero vs local model	Active
2026-05-24	Groq (llama-3.3-70b-versatile) for LLM; Euron retained for embeddings	Groq: faster inference, open-weight model. Euron kept for embeddings — Groq has no embeddings endpoint.	Active
2026-05	Render (Docker) over Railway for backend	Free tier RAM fit confirmed: embeddings ~150MB + reranker ~17MB = ~230MB total	Superseded 2026-06-22
2026-06-22	HF Spaces Docker over Render for backend	Render 512MB free tier caused repeated OOM under contextual refresh + concurrent chat. HF Spaces CPU Basic = 16GB RAM free. No memory guards needed. Port 7860, user UID 1000, HF_HOME=/app/.cache/huggingface baked at build time.	Active
2026-05	ParentDocumentRetriever (child 200 / parent 800)	Better faithfulness: small chunks retrieved precisely, large chunks give LLM full context	Active
2026-05	BM25 weight 0.3 in RRF fusion	Regulatory text has exact keyword matches; sparse recall is complementary, not dominant	Active
2026-05-30	RAGAS pre-computed locally, dashboard JSON-driven	nest_asyncio cannot patch uvloop on Render — live eval endpoint always 500s. Run scripts/run_ragas_local.py (8B judge model) → commit ragas_benchmark.json → Vercel builds dashboard.	Active
2026-05-30	RAGAS judge uses llama-3.1-8b-instant not llama-3.3-70b	70B model exhausts Groq free-tier 100k TPD in one eval run. 8B has 500k TPD and is sufficient for statement-level faithfulness checks. Answer generation still uses 70B.	Active
2026-05	LangSmith tracing via env var (no code changes)	LangChain reads LANGCHAIN_TRACING_V2 automatically; zero instrumentation cost	Active
2026-05-30	Reranker switched to TinyBERT-L-2-v2 (17MB) from MiniLM-L-6-v2 (85MB)	Lower memory footprint with acceptable ranking quality at demo corpus scale	Active
2026-05	Cross-encoder reranker pre-downloaded at Docker build time	Avoids cold-start latency on first request in production	Active
2026-05	Idempotent ingestion via md5(source+page+text) chunk IDs	Re-running ingest does not duplicate chunks in ChromaDB	Active
2026-06-14	Multi-workspace: one ChromaDB collection per workspace	Isolated document sets per workspace; workspace_id passed on every request; list_collections() normalised for chromadb ≥0.5.4 (returns list[str]) and <0.5 (returns list[Collection])	Active
2026-06-14	Multi-workspace frontend: workspace switcher + per-workspace doc list and chat	Sidebar shows all workspaces; switching remounts ChatArea via React key prop to clear stale messages and state	Active
2026-06-14	Singleton vectorstore/retriever cache keyed by workspace_id	Every chat request was creating a new Chroma instance (full embedding reload) on top of the existing one → OOM on repeated queries. Cache dict in retriever.py reuses instances; invalidated after ingest.	Active
2026-06-14	URL size guard in url_loader.py before embedding	Large external pages (news, filings) could exhaust 512MB RAM during URL ingest. Guard truncates/rejects oversized content before embed call.	Active
2026-06-16	HyDE for dense retrieval; toggled via config.yaml hyde_enabled	Hypothetical answer embedding lands closer to real answer chunks in vector space than raw query. BM25 + reranker still use original query. Off by default — adds one Groq call (~200ms); enable to measure RAGAS lift before committing.	Active
2026-06-17	Web search mandatory (not toggle)	Opt-in toggle caused users to get hallucinated answers grounded in wrong corpus docs when web search was off. Always-on Tavily + RAG gives grounded answers for both corpus and open-domain queries.	Active
2026-06-17	Separate eval-dashboard as own Vercel project	Eval tooling is not user-facing; separating avoids bloating the main frontend and lets eval dashboard evolve independently	Active
2026-06-17	answer_correctness replaces faithfulness as primary metric	faithfulness (judge vs retrieved chunks) is circular — inflates when eval pairs were designed alongside corpus. answer_correctness (judge vs ground_truth reference) is independent signal	Active
2026-06-17	Per-message faithfulness badge removed from user UI	Badge added noise without value to end users; saves one Groq call per query (~200ms latency reduction); eval moved to dedicated dashboard	Active
2026-06-17	eval_pairs.json expanded 20 → 50 pairs	20 samples not statistically meaningful; added multi-hop, comparative, negative, numeric, edge-case question types	Active
2026-06-17	Versioned eval JSON runs + index.json registry	Single flat JSON had no history; versioned runs let dashboard show metric evolution across architecture changes	Active

| 2026-06-20 | Contextual retrieval shipped to production via BackgroundTask | Eval proved +18% recall. Two-phase: sync non-contextual embed first (user queryable <3s), background replaces with contextual. Avoids blocking upload on ~52s Groq calls. | Active | | 2026-06-20 | contextualize_chunks_async() with Semaphore(3) | Parallel Groq calls 10× faster than sequential. max_concurrent=3 keeps burst at ~3000 TPM — safe under Groq free tier 6000 TPM limit. max_concurrent=20 caused OOM + 429 storm. | Active | | 2026-06-20 | Retry parses wait time from 429 error message | Groq 429 includes "Please try again in X.Xs". Parsing gives accurate sleep duration. Hardcoded 2s was too short for 10s rate limit windows. | Active | | 2026-06-20 | 422 for encrypted PDF upload instead of 500 | pypdf.FileNotDecryptedError propagated as unhandled 500. Now caught in load_documents_from_paths(), re-raised as ValueError, caught in upload route → HTTP 422 with clear message. | Active | | 2026-06-20 | Config-driven maintenance banner in frontend/src/config.js | Single file to toggle MAINTENANCE_MODE + MAINTENANCE_MESSAGE. Edit + push → Vercel redeploys in ~30s. No hardcoded HTML. | Active | | 2026-06-24 | ast.literal_eval fallback + control char strip in generate_briefing() | Groq LLM sometimes returns Python dict syntax (single quotes) or embeds ASCII control chars (0x00–0x1f) that break json.loads. Strip control chars first; fall back to ast.literal_eval on JSONDecodeError. Both are safe — input is already extracted from LLM regex match. | Active | | 2026-06-24 | HyDE enabled by default in config.yaml | Re-eval (v1.1.0_20260624, 18 samples) shows +21pp recall (0.51→0.72) with HyDE. Latency cost 2× (p50 4018ms vs 2029ms) accepted — recall gain outweighs latency. | Active | | 2026-06-26 | Citation highlighting via popover, not PDF viewer pane | Sources are PDF/URL/TXT/CSV — a PDF-only viewer fails for most. Popover with full chunk text works for all types. PDF sources get bonus "Open page N →" link via browser's built-in viewer. Zero new npm deps. | Active | | 2026-06-26 | DOMRect snapshot at click time (not live ref) for popover positioning | Simpler than passing anchorRef into CitationPopover — captures position once at click, no ref forwarding complexity. Stale after scroll (acceptable for demo). | Active | | 2026-06-26 | onMouseDown stopPropagation on citation <sup> | Document-level mousedown in CitationPopover fires before onClick on the marker. Without stopPropagation, clicking an open citation closes (mousedown) then immediately reopens (click). stopPropagation on mousedown lets toggle logic in onClick run correctly. | Active | | 2026-06-26 | Path.is_relative_to() over startswith() for file serving guard | startswith() on raw strings has prefix-confusion bug: /data/rawevil passes /data/raw check. is_relative_to() (Python 3.9+) is separator-aware and correct. | Active | | 2026-06-26 | UPLOAD_DIR as absolute __file__-relative path | Relative Path("data/raw") resolves against process CWD — breaks if uvicorn started from non-project-root directory. Path(__file__).resolve().parent.../ "data" / "raw" is stable regardless of CWD. | Active |

| 2026-06-27 | Metadata filtering: full-corpus BM25 scoring with candidate pool filter | Spec suggested rebuilding BM25 on filtered subset. Full-corpus IDF is correct IR — rare terms don't get inflated weight in a 2-doc subset. Filter which indices enter candidate pool; score with global index. | Active | | 2026-06-27 | Metadata filtering: one-off retriever per filtered request | get_retriever_filtered() creates new HybridRetriever with filter_docs set; reuses cached vectorstore. Singleton cache (_retriever_cache) untouched. Thread-safe: vectorstore (heavy) stays shared, retriever (cheap) is ephemeral. | Active | | 2026-06-27 | Empty filter_docs treated as no filter | Backend guard: body.filter_docs if body.filter_docs else None. Frontend sends null when array is empty. Prevents zero-result queries from empty selection. | Active | | 2026-06-27 | source_type metadata added to all chunks at ingest | PDF/TXT/CSV tagged at load time in ingest.py; URL-ingested docs tagged in url_loader.py. Enables citation badge display and potential future source-type filtering. | Active | | 2026-06-27 | Filter state resets on workspace switch | useEffect(() => setFilterDocs([]), [currentWorkspace]) in App.jsx. Stale filter from workspace A cannot pollute queries in workspace B. | Active | | 2026-07-05 | HyDE + Multi-Query disabled in production (free-tier stability) | HyDE + MQ = 2 extra Groq calls/query on top of condense + answer = 4 total. Free tier 6000 TPM → 429 storms. Disabled in config.yaml; contextual stays ON (Euron model, no Groq impact). Re-enable on paid tier. | Active | | 2026-07-05 | Eval dashboard marks live version and blocked versions explicitly | index.json gets is_live: true + live_note on closest proxy run; blocked_by on runs not deployable. Dashboard renders LIVE badge + constraint warnings. Honest representation of prod vs best-measured gap. | Active |

Rejected alternatives

Alternative	Why rejected
sentence-transformers local embeddings	~400MB RAM → OOM on Render free tier
Pinecone / Weaviate vector store	Adds external dependency and cost; ChromaDB sufficient for demo scale
LlamaIndex instead of LangChain	LangChain has better ParentDocumentRetriever and ConversationalRetrievalChain support
Streaming LLM responses	Adds frontend complexity; acceptable latency at demo scale
BM25 persisted to disk	Not required for demo; rebuild on startup is fast enough (~1s for sample corpus)
Cerebras migration (2026-06-29)	Free tier only has gpt-oss-120b + zai-glm-4.7 at 5 req/min — Llama 3.3 70B not available. Worse rate limit than Groq. All 8 commits hard-reset.
HyDE + Multi-Query in production on free Groq tier	4 Groq calls/query → 429 storms. Disabled in config.yaml, not removed — zero code change to re-enable on paid tier.