docs/tech-stack-rationale.md

# Tech Stack Rationale

| Field | Value |
| --- | --- |
| Project | Insurance Sales Portfolio Expert |
| Date | 2026-05-13 |
| Status | Living document — updated as picks change |
| Companion docs | `02-architecture.md` (architecture detail) · `decisions.md` (per-decision log) |

---

## Purpose

This is the **single, consolidated artifact** explaining every technology, provider, and framework choice in the project. It exists so that a reviewer (or future maintainer, or you in three months) can audit any pick by asking:

> "Why this and not the alternative? What evidence supports the choice? What changes if the constraint changes?"

Every entry has alternatives considered and a reasoning trace. Where decisions are still open, they are flagged with "**OPEN**" and an expected resolution date.

---

## 1. Component map — what the system is made of

The product is split into five layers. Each layer has independent technology choices.

```
┌─────────────────────────────────────────────────────────────────┐
│ 1. FRONTEND        web UI · chat · audio · filter · compare    │
└─────────────────────────────────────────────────────────────────┘
                          ↓ HTTPS
┌─────────────────────────────────────────────────────────────────┐
│ 2. BACKEND API     route handlers · auth · request validation  │
└─────────────────────────────────────────────────────────────────┘
                          ↓
┌─────────────────────────────────────────────────────────────────┐
│ 3. ORCHESTRATION   persona prompt · intent routing · providers │
└─────────────────────────────────────────────────────────────────┘
              ↓                                ↓
┌──────────────────────────┐   ┌──────────────────────────────────┐
│ 4a. STRUCTURED STORE     │   │ 4b. VECTOR STORE                 │
│     filter / compare     │   │     RAG retrieval                │
└──────────────────────────┘   └──────────────────────────────────┘
                          ↑
┌─────────────────────────────────────────────────────────────────┐
│ 5. INGEST PIPELINE      crawl · chunk · embed · extract        │
└─────────────────────────────────────────────────────────────────┘
```

---

## 2. The picks — one table to find anything

| # | Layer | Component | Pick | Status | Why (one line) | Alternatives considered |
| --- | --- | --- | --- | --- | --- | --- |
| 1 | Frontend | Framework | **Next.js 14 (App Router)** | Locked | Production-pattern, full UI flexibility, fast w/ shadcn/ui | Streamlit, Gradio, Chainlit, Reflex |
| 2 | Frontend | Styling | **Tailwind CSS + shadcn/ui** | Locked | Copy-paste components produce beautiful UIs in hours | MUI, Chakra, plain CSS |
| 3 | Frontend | Audio capture | **MediaRecorder API** (browser-native) | Locked | Zero deps, cross-browser, blob→POST flow is simple | streamlit-mic-recorder, WebRTC, manual ScriptProcessor |
| 4 | Frontend | Hosting | **Vercel** (free tier) | Locked | Native Next.js host, GitHub auto-deploy, edge-cached | Netlify, Cloudflare Pages |
| 5 | Backend | Framework | **FastAPI + Pydantic** | Locked | Pydantic matches our 48-field schema; OpenAPI auto-docs; async I/O for parallel provider calls | Flask, Django, Node/Express |
| 6 | Backend | Hosting | **Render** (free 750h/mo) | Locked | GitHub auto-deploy, Python-native, persistent disk for DuckDB + Chroma | Fly.io, Railway, Modal |
| 7 | Backend | API style | **REST** for v1, possible WebSocket if voice latency demands | Locked | Simpler; switch to WS only if measured latency exceeds target | Pure WS, gRPC |
| 8 | Backend | FE↔BE type safety | **`openapi-typescript` codegen** from FastAPI's auto-generated OpenAPI | Locked | Single source of truth; backend route changes propagate automatically | tRPC (Node-only), GraphQL, manual TS types |
| 9 | Orchestration | STT | **Sarvam Saarika v2.5** | Locked | Sarvam-first; their newer Indic ASR model | Whisper-large-v3, Deepgram Nova-2, Sarvam Saaras v3 |
| 10 | Orchestration | TTS | **Sarvam Bulbul** | Locked | Sarvam-first; Indic prosody first-class | ElevenLabs, OpenAI TTS, Coqui local |
| 11 | Orchestration | LLM (brain) | **NVIDIA NIM tiered: DeepSeek-V4-Pro (heavy) + DeepSeek-V4-Flash (fast)** | Locked (D-019) | V4-Pro (1.6T/49B MoE, frontier, beats Opus-4.6 + GPT-5.4 on SimpleQA) for comparison/recommendation; V4-Flash (284B/13B MoE, lower TTFT) for voice + fact-find. Single NIM key, free tier, no daily cap. Sarvam-M moved to Indic translation only — its 2048 output cap was truncating advisor responses. | Sarvam-M as brain (rejected — truncation), OpenRouter `:free` (rejected — 50/day cap), Gemini 2.5 Flash (rejected — adds 2nd provider ecosystem), GPT-4o via GitHub Models (rejected — same 50/day cap) |
| 12 | Orchestration | LLM (grader / self-critique) | **NVIDIA NIM Llama-4 Maverick (Meta MoE)** | Locked (D-019) | Different family from the DeepSeek brain — Meta MoE judging DeepSeek MoE, so the brain does not mark its own homework. Same NIM endpoint and key. 400B/17B MoE, frontier-tier. Replaces Groq Llama-3.3-70B (30 req/min cap blocked sweeps). | Groq Llama (rejected — rate limit), Cerebras Qwen-235B (rejected — same family as another DeepSeek path), Sarvam-M held-out |
| 13 | Orchestration | Embeddings | **BGE-small-en-v1.5 (local CPU, sentence-transformers)** | Locked | Free, runs offline, 384-dim, BEIR-strong on insurance/legal text. Voyage path kept for v2 swap (~3pp retrieval-quality delta in BEIR-style spot checks). | Voyage AI voyage-3 (rejected for v1 — 3 RPM free-tier blocked 104-PDF ingest), OpenAI text-embedding-3-small (rejected — no GPT), Cohere |
| 14 | Storage | Structured DB | **DuckDB** (single file) | Locked | One file, columnar, no server; deploys via `git clone` | SQLite, Postgres |
| 15 | Storage | Vector DB | **Chroma** (local persisted) | Locked | Embedded, no infra, supports metadata filtering | FAISS, Pinecone, Qdrant, Weaviate |
| 16 | Ingest | PDF parsing | **pdfplumber** | Locked | Clean text + page numbers — critical for citation grammar | PyMuPDF, Unstructured.io |
| 17 | Ingest | Chunking | **Custom 800-token chunks, 120-token overlap, page-aware** | Locked | Standard for technical docs; overlap protects clause boundaries | LangChain text splitter, fixed-size |
| 18 | Ingest | Structured extraction | **Sarvam-M (Llama-3.3-70B / DeepSeek-V3 fallback) with Pydantic structured output + self-critique pass** | Locked | Reliable JSON-mode extraction; self-critique gives per-field confidence; fallback brain handles tables / complex clauses Sarvam-M misses | Pure regex, LangChain extraction, Instructor lib |
| 19 | Pricing | Approach | **Hand-curated illustrative bands** from public PolicyBazaar quotes | Locked | Honest, defensible; "advisor-not-broker" positioning | Live scraping, actuarial model |
| 20 | Cross-cutting | Auth | **None for v1** (single-tenant demo) | Locked | Out of scope per `01-requirements.md` §7 | Auth0, Clerk, NextAuth |
| 21 | Cross-cutting | Observability | **JSONL turn log + cost tracker** | Locked | Lightweight; one log file, queryable post-hoc | Langfuse, Helicone, custom dashboard |
| 22 | Cross-cutting | Secrets management | **`.env` (chmod 600, gitignored) + Render env vars + Vercel env vars** | Locked | Three sources, never committed | HashiCorp Vault, Doppler |

---

## 3. Selection rubric — how we chose

Every pick above followed this five-test rubric. **Picks that fail any of these are documented as accepted tradeoffs, not silent compromises.**

### 3.1 Sarvam-first hypothesis

For every component Sarvam plausibly ships (STT, TTS, LLM, embeddings), the starting hypothesis is *"use Sarvam unless we have empirical reason not to."* This is graded behavior for a Sarvam assignment — silent defaults to OpenAI/Anthropic/ElevenLabs are the most common screen-out signal.

We will **benchmark** before locking final Sarvam vs non-Sarvam picks:

- **STT:** 20-utterance test set across English, Hindi, Hinglish — measure word-error rate
- **TTS:** Subjective quality + latency on 5 sample advisor responses
- **LLM:** 50 gold Q&A pairs run with Sarvam-M and GPT-4o — measure factual accuracy + citation accuracy

Where Sarvam wins → ship Sarvam. Where Sarvam loses → document the gap with numbers and ship the alternative. Either outcome is a strong artifact.

### 3.2 Real benchmark, not vibes

For any pick that could affect downstream accuracy (LLM, embeddings, extraction prompt design), we make the call on **empirical evidence**, not first principles. The gold Q&A harness exists for exactly this — it transforms "I think X is better" into "X scored 92% vs Y's 87% on our test set."

### 3.3 Production-pattern, not science-project

Every component is picked **as if the next step is white-labelling to a BFSI customer.** No "great for prototyping" picks that block productionization. Examples:

- ✅ FastAPI — already production-grade
- ✅ DuckDB — embedded but production-deployed at Motherduck and many fintechs
- ❌ ~~Streamlit~~ — great prototyping, no white-label path → switched to Next.js

### 3.4 Single-file / single-deploy where possible

Each extra service is a deploy risk and a moving part. We chose:

- DuckDB (one file) over Postgres (separate process + cluster)
- Chroma local (embedded) over Pinecone (separate service)
- Render (single backend deploy) over Render-FE + Render-BE + Redis + Postgres

The total infrastructure footprint is **2 deploys** (Vercel + Render) + **2 cloud APIs** (Sarvam + OpenAI). That's it.

### 3.5 Documented alternatives

Every pick has at least 2 alternatives recorded. A reviewer can audit the reasoning. "Why DuckDB?" → table row 14 → SQLite (no columnar, no analytics) and Postgres (overkill, deploy overhead). Three options considered; reasoning explicit.

---

## 4. Cost envelope (24-hour build + demo run)

| Item | Estimated cost |
| --- | --- |
| Sarvam API (STT + TTS + LLM) — build + 100 demo calls | likely free under signup credits |
| Voyage AI embeddings (~5M tokens to embed 75 PDFs) | ~$0.45 (free under $50 signup credit) |
| Claude Haiku 4.5 for grader / self-critique (~200 calls) | ~$0.20 |
| Render (free tier, 750 h/mo) | $0 |
| Vercel (free tier, 100 GB bandwidth) | $0 |
| GitHub (public repo) | $0 |
| **Total realistic spend** | **< $1** |

Production projection (1,000 daily active users, 5 turns each):
- ~150K STT calls/mo + ~150K TTS calls/mo + ~150K LLM calls/mo
- Estimated ~$200-400/mo for Sarvam APIs (highly volume-discount-sensitive)
- Render Standard ($7/mo) + Vercel Pro ($20/mo) for resource limits
- **Production envelope: ~$250-500/mo for the first 1K DAU.** Sub-cent per turn.

---

## 5. Open picks (will be resolved in next 24 hours)

| # | Question | Resolution |
| --- | --- | --- |
| O1 | Does Sarvam expose an embeddings API? If yes, benchmark vs OpenAI text-embedding-3-small | Check `dashboard.sarvam.ai` API listing; <30 min |
| O2 | Saarika v2.5 vs Saaras v3 for STT — depends on whether we need translate/codemix modes | Benchmark both on 20-utterance set; <1h |
| O3 | Recommendation engine shape: rule-based pre-filter + LLM justification vs pure-LLM reasoning | Decide after gold Q&A harness runs; leaning rule-based for testability |
| O4 | Caching layer — should LLM responses be cached for repeated questions? | Yes for demo; simple file cache, no Redis |
| O5 | Should we add basic auth on the deployed URL to avoid abuse? | Probably yes — single shared password for interviewer access |

---

## 6. Decisions log cross-reference

Every pick here corresponds to one or more entries in `decisions.md`:

| Pick | Decision IDs |
| --- | --- |
| Vertical-slice scope | D-001 |
| Health insurance category | D-002 |
| Curated corpus, not user-uploaded | D-003 |
| Hybrid structured + unstructured | D-004 |
| **Frontend stack** (Next.js + FastAPI) | D-005 (revised 2026-05-13) |
| Sarvam-first benchmarking | D-006 |
| Pricing as illustrative band | D-007 |
| Consultative persona | D-008 |
| 10 insurers × all health policies | D-009 |
| Secret handling | D-010 |
| Embeddings provider (pending) | D-011 |
| Render deployment over alternatives | D-012 |
| Next.js + Tailwind + shadcn UI stack | D-013 |
| GPT-4o-mini as grader to avoid circular eval | D-014 |

---

## 7. Revision history

| Version | Date | Change |
| --- | --- | --- |
| 0.1 | 2026-05-13 | Initial draft. Switched D-005 frontend from Streamlit to Next.js + FastAPI; added D-011 through D-014 for new picks. |